Evaluation of AI Agents (Preview)

The evaluation tests check the behavior of the AI agents against the real models: they detect the regressions caused by a change of prompt, of skill, of model or of AI library version. They rely on the Microsoft.Extensions.AI.Evaluation libraries and on the AIAgentEvaluation base class of the GroupeIsa.Neos.Application.XUnit package.

See Evaluation libraries for more information.

Configuration

An evaluation uses two kinds of models:

the tested models: the models of the evaluated agents, configured as in the application (AI section of the cluster and its secrets);
the judge model: the model which evaluates the answers, configured in the AI:Evaluation section.

Every configuration key can be supplied by the cluster configuration, by the user secrets or by an environment variable, using __ as the section separator (for example AI__Evaluation__TargetModel).

Key	Description
`AI:Evaluation:Provider`	Provider of the judge model: `OpenAI`, `AzureOpenAI` or `Anthropic`.
`AI:Evaluation:ModelId`	Identifier of the judge model (for example `gpt-6-luna`).
`AI:Evaluation:ApiKey`	API key of the judge model.
`AI:Evaluation:ApiEndpoint`	Endpoint of the judge model, required for an alternative endpoint such as Azure OpenAI or Microsoft Foundry.
`AI:Evaluation:SupportsTemperature`	`true` when the judge model accepts a temperature. `false` by default, because the reasoning models reject it.
`AI:Evaluation:TargetModel`	Name of the tested model, among the models of the application. When set, it replaces the model of the agents.
`AI:Evaluation:StorageRootPath`	Folder where the evaluation results are stored (current directory by default).
`AI:Evaluation:ExecutionName`	Name of the execution in the report (date and time by default, build number in the pipeline).
`AI:Evaluation:EnableResponseCaching`	`false` disables the caching of the judge model responses. `true` by default, which avoids paying the same judgement twice locally.

The evaluation project reads, in this order, the application settings of the cluster, the user secrets of the cluster, the user secrets dedicated to the evaluations (GroupeIsa.Neos.AI.Evaluation) and the environment variables: the last one wins. The dedicated user secrets are therefore the right place to configure the evaluations once for all the clusters of a computer:

dotnet user-secrets set "AI:Evaluation:Provider" "AzureOpenAI" --id GroupeIsa.Neos.AI.Evaluation
dotnet user-secrets set "AI:Evaluation:ModelId" "gpt-6-luna" --id GroupeIsa.Neos.AI.Evaluation
dotnet user-secrets set "AI:Evaluation:ApiEndpoint" "[ApiEndpoint]" --id GroupeIsa.Neos.AI.Evaluation
dotnet user-secrets set "AI:Evaluation:ApiKey" "[ApiKey]" --id GroupeIsa.Neos.AI.Evaluation

The tested models can be configured the same way, for example AI:ApiEndpoint and AI:ApiKey for the models using the default endpoint of the application.

Note

The internal agents of the framework (smart search, translations, input suggestions) read their model from a setting such as AI:SmartSearchModel rather than from their prompt. To evaluate them with AI:Evaluation:TargetModel, the Startup of the evaluation project applies the tested model to these settings (see the Northwind evaluation project, which evaluates them with AzureGpt54, the gpt-5.4 model, when AI:Evaluation:TargetModel is not set).

Evaluators

The judge model evaluates the answers with the following evaluators, chosen with WithEvaluator:

Evaluator	What is evaluated
`Groundedness`	The answer is grounded on the evaluation context given with `WithContext`.
`RelevanceTruthAndCompleteness`	The answer is relevant, true and complete regarding the question.
`Coherence`	The answer is coherent and well written.
`Fluency`	The answer is fluent.
`ToolCallAccuracy`	The functions (skills, tools) called by the agent are the right ones, with the right arguments.
`TaskAdherence`	The agent adheres to the task it was given, including the functions it called.
`IntentResolution`	The agent understood and resolved the intent of the user.

The functions called by the agent and their results are given to the judge model with the answer, as well as the definitions of all the functions the agent could have called.

Warning

The Microsoft.Extensions.AI.Evaluation libraries are in preview, so these evaluators may change in future versions.

Create the evaluation project

In Neos Studio, in the AI agent view, the "Create Evaluation Project" button creates the projects/AI.AgentEvaluation project of the cluster, or updates it, and adds a test class for the current agent. Each time you want to evaluate a new agent, you add its class with this action.

The generated test class carries the [AIEvaluation("<AgentId>")] attribute and contains two example scenarios, skipped until you replace their placeholders: a repeated evaluation by the judge model, and a question outside the scope of the agent which must not call any function.

Note

A partial Startup class is created in the test project, to configure its services: you can add your own services or replace some of them with a mock. The Startup.Generated.cs file is generated and must not be modified.

The generated services include the OpenAPI document of the application, built as in the application but outside of the web host: the skills creating or proposing data read the JSON schema of the entity views in this document, and can therefore be evaluated.

In the evaluation tests, the services are configured with their real implementation. Replace in the Startup class the services that would write data or need an infrastructure the evaluation does not have, for example:

public void ConfigureServices(IServiceCollection services)
{
    _ = InternalConfigureServices(services);

    // The user interface does not exist: the interactions are only recorded as function calls.
    Mock<IUIInteraction> uiInteractionMock = new();
    services.AddSingleton(uiInteractionMock);
    services.Replace(ServiceDescriptor.Scoped(sp => uiInteractionMock.Object));

    // Without any authenticated user, the agents would not see any view: every function resource is allowed.
    Mock<IFunctionResourceAuthorizer> functionResourceAuthorizerMock = new();
    functionResourceAuthorizerMock
        .Setup(s => s.IsAllowedAsync(It.IsAny<FunctionResourceType>(), It.IsAny<string>()))
        .ReturnsAsync(true);
    functionResourceAuthorizerMock
        .Setup(s => s.IsAllowedAsync(It.IsAny<FunctionResourceType>(), It.IsAny<string>(), It.IsAny<FunctionOperation?>()))
        .ReturnsAsync(true);
    services.Replace(ServiceDescriptor.Scoped(sp => functionResourceAuthorizerMock.Object));

    // The saves succeed without writing anything.
    Mock<IUnitOfWork> unitOfWorkMock = new() { DefaultValue = DefaultValue.Mock };
    unitOfWorkMock.Setup(u => u.SaveAsync(It.IsAny<CancellationToken>())).ReturnsAsync(Result.Ok());
    services.Replace(ServiceDescriptor.Scoped(sp => unitOfWorkMock.Object));
}

The mocks are shared by the tests, which may run in parallel: set them up once in the Startup class, and check what the agent did through its function calls rather than by verifying the mocks.

Write an evaluation test

Check an exact answer

When the answer can be checked without any judge model, use PromptAsync and assert the answer as in a classic unit test. PromptAsync<TAgent, TResult> prompts the agent for a typed answer, as the application does with a JSON schema:

[Fact]
public async Task ShouldCreateTheDescribedImage()
{
    AIEvaluationOptions options = new AIEvaluationOptions("A red circle on a white background").WithRepetitions(3, 2);

    await PromptRepeatedlyAsync<ImageCreatorAgent, GeneratedSvgImage>(
        options,
        image =>
        {
            image.FileName.Should().MatchRegex("^[A-Z][A-Za-z0-9]*\\.svg$");
            XElement.Parse(image.Content).Name.LocalName.Should().Be("svg");
        },
        TestContext.Current.CancellationToken);
}

Absorb the non-determinism of the model

The same question can get different answers. WithRepetitions(3, 2) runs a case up to three times and requires two successes. PromptRepeatedlyAsync, EvaluateRepeatedlyAsync and RepeatAsync (for a service of the application) stop as soon as the required number of successes is reached, or can no longer be reached, and fail the test with the reason of each failed attempt.

List all the right answers rather than a single one: several translations or corrections can be right.

Check the functions called by the agent

FunctionCalls contains the functions (skills, tools) called during the last prompt, with their arguments, results and errors, and PromptInvocations the prompts answered, sub-agents included, with their token usage:

[Fact]
public async Task ShouldNotOpenAnyView_WhenTheRequestIsNotANavigation()
{
    AIEvaluationOptions options = new AIEvaluationOptions("What is the capital of Australia?").WithRepetitions(3, 2);

    await PromptRepeatedlyAsync<NeosNavigateAgent>(
        options,
        _ => FunctionCalls.Should().NotContain(call => call.FunctionName == "OpenUIViewName"),
        TestContext.Current.CancellationToken);
}

The test output lists the answer, each function call (and its error, if any) and the token usage of each model used by the agent and its sub-agents.

For each agent, write at least a nominal case, a case in French, a case outside the scope of the agent and, for the agents with skills, a case checking that nothing is saved without the confirmation of the user.

Evaluate a textual answer with the judge model

When the answer is free text, use EvaluateAsync or EvaluateRepeatedlyAsync with the evaluators you need:

AIEvaluationOptions options = new AIEvaluationOptions(
        "In which UI view is the \"ShowChatbot\" method defined?",
        "The \"ShowChatbot\" method is defined in the \"DesignerUI\" UI view.")
    .WithEvaluator(AIEvaluator.Groundedness)
    .WithRepetitions(3, 2);

await EvaluateRepeatedlyAsync<EntityExplorerAgent>(
    options,
    result =>
    {
        NumericMetric groundedness = result.Get<NumericMetric>(GroundednessEvaluator.GroundednessMetricName);
        groundedness.Interpretation!.Failed.Should().BeFalse(groundedness.Reason);
    },
    TestContext.Current.CancellationToken);

The Groundedness evaluator needs an evaluation context: the information allowing to check the answer, not only an expected answer. A test fails with an explicit message when the judge model cannot evaluate the answer (wrong endpoint, unknown deployment...).

Compare OData filters

The agents building OData filters can give several filters with the same meaning. ODataFilterNormalizer.AreEquivalent compares them by meaning: the order of the conditions, the parentheses, the names of the lambda variables, the side of the literals and the simple negations do not matter.

ODataFilterNormalizer.AreEquivalent("not(toupper(City) ne 'LYON')", "'LYON' eq toupper(City)").Should().BeTrue();

Run the evaluation tests

Every evaluation test carries the Category=AIEvaluation trait, and the Agent=<AgentId> trait when its class carries the [AIEvaluation("<AgentId>")] attribute. Run the tests of a single agent with a filter:

dotnet test projects/AI.AgentEvaluation/AI.AgentEvaluation.slnx --filter "Category=AIEvaluation&Agent=NeosNavigate"

Note

In a deep folder, such as a git worktree, run the built test assembly with the xUnit runner, which accepts the same traits: dotnet exec AI.AgentEvaluation.dll -trait "Agent=NeosNavigate".

Read the evaluation report

The results of the judge model are stored in the AI:Evaluation:StorageRootPath folder. Install the dotnet tool once, then generate the report after running the tests:

dotnet tool install --local Microsoft.Extensions.AI.Evaluation.Console --version 10.9.0
dotnet tool run aieval report --path [StorageRootPath] --output report.html

The report shows, for each scenario and iteration, the metrics, the reasons given by the judge model and its token usage.

Limitations

The agents depending on real data or on a vector index are evaluated with mocked skills returning a fixed context.

Upgrade your test project

After upgrading Neos, upgrade the evaluation projects of your cluster by running the following command at its root. It updates the version of the evaluation libraries, the project file and the Startup.Generated.cs file:

neos upgrade-ai-test

Generate the cluster as well (neos generate): the generated Startup.Generated.cs builds the OpenAPI document through the Startup class of the cluster, which must be generated with the same version of Neos.

Table of Contents

Evaluation of AI Agents (Preview)

Configuration

Note

Evaluators

Warning

Create the evaluation project

Note

Write an evaluation test

Check an exact answer

Absorb the non-determinism of the model

Check the functions called by the agent

Evaluate a textual answer with the judge model

Compare OData filters

Run the evaluation tests

Note

Read the evaluation report

Limitations

Upgrade your test project