Upgrading cluster test projects to xUnit 4

Neos 3.3 moves its test dependencies from xUnit 3 to xUnit 4. xUnit 4 changes several APIs that cluster test projects use, so a test project left on xUnit 3 constructs no longer compiles against the Neos test packages. The neos migrate-metadata command upgrades the test projects of the cluster automatically and lists in its output what it could not upgrade on its own.

What changes with xUnit 4

xUnit 3 xUnit 4
xunit.v3 package xunit.v3.mtp-off package, version 4.0.0
xunit.runner.visualstudio 3.x xunit.runner.visualstudio 4.0.0
Xunit.DependencyInjection 11.x Xunit.DependencyInjection 12.0.1
ITestCaseOrderer, [TestCaseOrderer] ITestMethodOrderer, [TestMethodOrderer]
[assembly: CollectionBehavior(DisableTestParallelization = true)] [assembly: Parallelization(Mode = ParallelMode.None)]
ParallelMode.Off ParallelMode.None

The PriorityOrderer class provided by GroupeIsa.Neos.Shared.XUnit now implements ITestMethodOrderer, so it must be used with [TestMethodOrderer]. The xunit.v3.mtp-off package keeps the VSTest execution used by dotnet test and the Neos pipelines.

What neos migrate-metadata upgrades

Run the command once after installing Neos 3.3, before neos generate:

neos migrate-metadata

For each local module of the cluster, the command upgrades the xUnit 3 test projects: business assembly unit tests, end-to-end tests, API tests, and any other test project of the module. A project is an xUnit 3 test project when it, or a file it imports, references an xunit.v3 package. It:

  • Upgrades the xUnit package references, declared with PackageReference Include or overridden with PackageReference Update, of the projects and of the .props files they import, and deletes the packages.lock.json file of each upgraded project and of every business assembly unit test project. The next restore recreates it. A .props file is upgraded only when it references an xunit.v3 package itself, so that a file shared with xUnit 2 projects keeps their xunit.runner.visualstudio version.
  • Takes into account the nearest Directory.Build.props file, like MSBuild, between the project and the module folder. Only the imports written as relative paths are followed: an import built from MSBuild properties, such as $(MSBuildThisFileDirectory), is not. A project whose xUnit references only come through such an import is therefore not detected and must be migrated by hand. Every .props file of the module that references an xunit.v3 package itself is still upgraded.
  • Upgrades in the same way the shared test helper projects that these test projects reference, directly or not, even outside the modules folder, as long as they stay inside the cluster folder. The nearest Directory.Build.props file of such a helper is looked up to the cluster folder.
  • Replaces [TestCaseOrderer(typeof(PriorityOrderer))] with [TestMethodOrderer(typeof(PriorityOrderer))], CollectionBehavior(DisableTestParallelization = true) with Parallelization(Mode = ParallelMode.None), and ParallelMode.Off with ParallelMode.None, and adds the Xunit.Sdk and Xunit.v3 using directives when they are missing.
  • Upgrades the code of the business assembly unit tests even before neos generate has recreated their generated references file. The next generation moves their packages to xUnit 4.

The command keeps the encoding and line endings of the files it rewrites. It never modifies xUnit 2 projects, the modules restored as dependencies, files outside the cluster folder, or the bin, obj, and node_modules folders.

Warnings to handle manually

The command logs a warning, file by file, for every construct it cannot upgrade safely. The project must then be fixed by hand before it compiles with xUnit 4.

Warning What to do
A class implements ITestCaseOrderer Implement ITestMethodOrderer (method OrderTestMethods) instead, then use it with [TestMethodOrderer].
[TestCaseOrderer] uses an orderer other than PriorityOrderer Replace it with [TestMethodOrderer] once the orderer implements ITestMethodOrderer.
CollectionBehavior uses other parallelization options Replace them with the Parallelization attribute.
A package version comes from an MSBuild property or a version range Set the version of the package to the value listed in the table above.
A project or .props file cannot be read or written Fix the file or its permissions, then upgrade it by hand. Such a project is not migrated.
A referenced project is outside the cluster folder Upgrade that project by hand if it uses xUnit 3.

Projects kept on xUnit 3

A test project that references the Neos test packages (GroupeIsa.Neos.Shared.XUnit, GroupeIsa.Neos.EndToEndTests.*) with an explicit version older than the Neos version running the migration still relies on xUnit 3 through these packages. Only the numeric part of the versions is compared: 3.3.0-alpha.5 counts as 3.3.0, and a version that is not a literal, such as $(NeosTestVersion) or a version range, is not taken into account. The command leaves such a project untouched, together with every xUnit 3 project of the cluster that shares a project reference, in either direction, or an imported .props file referencing an xUnit package with a kept project, directly or through other kept projects. A warning names each of these projects once.

To move them to xUnit 4 later:

  1. Upgrade the Neos test packages they reference to the current Neos version. For end-to-end test projects, neos create-ui-tests-project regenerates the references file of the project.
  2. Apply the conversions listed in What changes with xUnit 4 by hand.

Other projects to upgrade manually

  • AI agent evaluation projects are not part of the modules. Run neos upgrade-ai-test to regenerate them with the xUnit 4 packages.
  • Test projects outside the modules that no module test project references are not upgraded. Apply the conversions above by hand.
  • Files above the module folder: a Directory.Build.props file above the module folder, and a .props file imported through an MSBuild property from outside the module, are not upgraded. Update their xUnit versions by hand.
  • Central package management: the xUnit versions declared in a Directory.Packages.props file are not upgraded. Update them by hand.

The xUnit 4 analyzers also report the xUnit1051 warning when an asynchronous call in a test does not receive TestContext.Current.CancellationToken. It does not prevent compilation, and the command leaves it to be fixed at your own pace.

See also