Evaluation of AI Agents (Preview)

To evaluate the performance of an AI agent, we used the Microsoft.Extensions.AI.Evaluation library.

See Evaluation libraries for more information.

Getting Started

To begin, you need to configure an LLM model to use for the evaluation.

In the cluster YAML configuration file, you can must add the following configuration to the AI section:

AI:
  Evaluation:
    Provider: [Provider]
    ModelId: [ModelId]

For example, to use gpt-5.6-luna, you can add the following configuration:

AI:
  Evaluation:
    Provider: OpenAI
    ModelId: gpt-5.6-luna

The supported providers are :

OpenAI
Anthropic

Then you must configure the secrets for the provider you are using:

dotnet user-secrets set "AI:Evaluation:ApiKey" "[ApiKey]" --id [ClusterRootNamespace].AspNetCore
dotnet user-secrets set "AI:Evaluation:ApiEndpoint" "[ApiEndpoint]" --id [ClusterRootNamespace].AspNetCore

The API key is always required.
The endpoint is optional when using the provider's official endpoint, but required when using an alternative endpoint, such as Microsoft Foundry.
The cluster root namespace can be found in the cluster configuration file.

Evaluators

The library provides several evaluators to evaluate the performance of the AI agent. The following evaluators are available:

GroundednessEvaluator: This evaluator checks if the answer is relevant and true. It uses the evalution context to check if the answer is correct.
RelevanceTruthAndCompletenessEvaluator: Evaluates the extent to which a given response contains all necessary and relevant information with respect to the provided ground truth. It doesn't use the evalution context to check if the answer is correct.
CoherenceEvaluator: Measures the grammatical proficiency of a generative AI's predicted answer. It doesn't use the evalution context to check if the answer is correct.
FluencyEvaluator: Evaluates the fluency of a given response or a multi-turn conversation, including reasoning. It doesn't use the evalution context to check if the answer is correct.

Warning

The Microsoft.Extensions.AI.Evaluation library is in preview, so these evaluators may not be available in future versions of Neos.

Create the xUnit test project

In Neos Studio, in the AI Agent view, you can create an xUnit test project by clicking the "Create Evaluation Project" button in the toolbar. If the project doesn't exist, it will be created; otherwise, it will be updated. This action will create an xUnit class for the current agent. Each time you want to evaluate an agent, you must add a class using this action.

You can then write your tests in the generated class.

Note

A partial "Startup" class is created in the test project. This class is used to configure the test project's services. You can add your own services or replace some services with a mock. The "Startup.generated.cs" file containing the generated class part is generated by Neos Studio and should not be modified.

Create a test to evaluate a textual answer

In this case, you can use the EvaluateAsync method of the base class AIAgentEvaluation to evaluate the agent and you choose evaluators you want to use among this list:

RelevanceTruthAndCompletenessEvaluator
CoherenceEvaluator
FluencyEvaluator
GroundednessEvaluator

Note

The GroundednessEvaluator evaluator requires an evaluation context to check if the answer is relevant and true.
The context should contain the information needed to check if the answer is correct and not be only a simple expected answer.
This evaluator is useful when the agent uses an external source to answer the question (e.g. using a plugin like RAG's technique or a database).

If your agent use an external source (e.g. using a plugin like RAG's technique), you should use the EvaluateAsync method of the AIAgentEvaluation base class to evaluate the agent.

Here's an example test:

using FluentAssertions;
using GroupeIsa.Neos.Application.XUnit.AI;
using GroupeIsa.Neos.Designer.Application.AIAgents;
using GroupeIsa.Neos.Designer.Domain.Persistence;
using GroupeIsa.Neos.Shared.Linq;
using Microsoft.Extensions.AI.Evaluation;
using Microsoft.Extensions.AI.Evaluation.Quality;
using Xunit;
using Xunit.Abstractions;

namespace GroupeIsa.Neos.Designer.Application.Tests.AIAgentEvaluations
{
    /// <inheritdoc/>
    public class EntityExplorerTests : AIAgentEvaluation
    {
        private readonly Mock<IProductRepository>  _productRepositoryMock;
        private readonly Mock<IOrderDetailRepository>  _orderDetailMock;

        public EntityExplorerTests(ITestOutputHelper output, IServiceProvider serviceProvider, Mock<IProductRepository> productRepositoryMock, Mock<IOrderDetailRepository> orderDetailMock)
            : base(output, serviceProvider)
        {
            _productRepositoryMock = productRepositoryMock;
            _orderDetailMock = orderDetailMock;
        }

        [Fact]
        public async Task EntityExplorerAgent_ShouldAnswerTo_UIViewMethodDefinitionLocation()
        {
            // Arrange
            _productRepositoryMock
                .Setup(x => x.GetQuery())
                .Returns(new[] { new Product { Id = 78, Name = "Original Frankfurter grüne Soße" }}.AsQueryable());
            _orderDetailMock
                .Setup(x => x.GetQuery())
                .Returns(new[] { new OrderDetails { ProductId = 10 }}.AsQueryable());

            AIEvaluationOptions options = new AIEvaluationOptions("Which products were never ordered ?")
                .WithEvaluator(AIEvaluator.Groundedness)
                .WithContext("""The products that are never ordered is:
                                    - Product ID: 78, Name: Original Frankfurter grüne Soße
                                    _ Product ID: 10, Name: Ikura""");

            // Act
            EvaluationResult evalResult = await EvaluateAsync<EntityExplorerAgent>(options, CancellationToken.None);

            // Assert
            NumericMetric groundedness = evalResult.Get<NumericMetric>(GroundednessEvaluator.GroundednessMetricName);
            groundedness.Interpretation!.Failed.Should().BeFalse(groundedness.Reason);
        }
    }
}

The question to ask to the agent is defined in the AIEvaluationOptions object.

The response format is text
Product ID: 78 is determined by executing a query in a database in a AI function

Then you must call the EvaluateAsync method to evaluate the agent.
And you must use the GroundednessEvaluator evaluator to check if the answer is relevant, as this evaluator is the only one that can check the relevance of the response based on the expected response.

In evaluation tests the services are configured with their real implementation. So the repositories use the real implementation to query the database configured in the cluster YAML file.
You can replace the real implementation with a mock implementation if you want to test the agent with a specific data set or to verify a call to a specific method of a particular service.

To do this, in the startup class of the test project, you can add the following code to configure the services:

 public partial class Startup
 {
     /// <summary>
     /// Configures the services.
     /// </summary>
     /// <param name="services">Services.</param>
     public void ConfigureServices(IServiceCollection services)
     {
         _ = InternalConfigureServices(services);

         // Replace the real implementation with a mock
         var productRepositoryMock = new Mock<IProductRepository>();
         services.AddSingleton(productRepositoryMock);
         services.Replace(ServiceDescriptor.Scoped(sp => productRepositoryMock.Object));

         var orderDetailRepositoryMock = new Mock<IOrderDetailRepository>();
         services.AddSingleton(orderDetailRepositoryMock);
         services.Replace(ServiceDescriptor.Scoped(sp => orderDetailRepositoryMock.Object));
     }
 }

The service IProductRepository is replaced by a mock implementation and the Mock<IProductRepository> is configured as singleton in the service collection so it can be retrieved in the test class constructor.

Create a test to evaluate an exact answer

In this case, you should use the PromptAsync method of the AIAgentEvaluation base class to evaluate the agent and check if the response is correct like a classic unit test.

Here's an example test:

[Theory]
[InlineData("Order of March 27, 2025", "OrderView", null, "en", "OrderDate eq 2025-03-27")]
[InlineData("Order with one detail having the item named 'Book'", "OrderView", null, "en", "OrderDetails/any(od: contains(toupper(od/Product/ProductName), 'BOOK'))")]
public async Task ShouldCreate_ODataFilter(string question, string entityViewName, string? uiViewName, string language, string expectedODataFilter)
{
    // Arrange
    AIEvaluationOptions options = new AIEvaluationOptions(
        AiOdataQueryGenerator.GenerateFullPrompt(new EntityViewTextualContext(), entityViewName, question, language, uiViewName),
        expectedODataFilter)
        .WithPromptOptions(new PromptOptions().WithResponseFormat(ResponseFormat.Json));

    // Act
    string answer = await PromptAsync<ODataQueryGeneratorAgent>(options, CancellationToken.None);

    // Assert
    var json = JsonDocument.Parse(answer);
    json.RootElement.GetProperty("filter").GetString().Should().Be(expectedODataFilter);
}

The agent transform a filter in natural language into an OData filter, the expected answer is a JSON response with the filter in the filter property.

How to get the LLM answer

To get the LLM answer, you can use the Answer property of the AIEvaluationBase class. This property contains the current answer of the LLM model.

The answer and the expected answer are also logged in the xUnit output.

Generate reports

First you need to install the dotnet tool :

dotnet tool install --local Microsoft.Extensions.AI.Evaluation.Console --version 9.5.0

Then you can generate the reports using the following command in the root of your cluster after running the tests:

dotnet tool run aieval report --path c:\temp --output report.html

Upgrade your test project

After upgrading Neos, you may need to upgrade your existing test projects.
This is necessary to update the test project to the new version of Neos.
This command updates the version of the Microsoft.Extensions.AI.Evaluation library to the version used in the new version of Neos and updates the startup.generated.cs file if necessary.
You can do this by running the following command in the root of your clusters:

neos upgrade-ai-test

Table of Contents

Evaluation of AI Agents (Preview)

Getting Started

Evaluators

Warning

Create the xUnit test project

Note

Create a test to evaluate a textual answer

Note

Create a test to evaluate an exact answer

How to get the LLM answer

Generate reports

Upgrade your test project