Performance of running a large number of source-generated tests

[hainguyenthanh5-oss]

Please refer to the attached sample solution, which simulates 10,000 unit tests using NUnit and TUnit.

TestRunnerPerformance.zip

Compile the solution in Release configuration, enabling or disabling the source generator for the Calculator.Tests.TUnit project by setting <EnableTUnitSourceGeneration>true</EnableTUnitSourceGeneration> or <EnableTUnitSourceGeneration>false</EnableTUnitSourceGeneration>.

Below are the commands to execute the unit tests:

# NUnit
dotnet run --project ./Calculator.Tests.NUnit/Calculator.Tests.NUnit.csproj --no-build --configuration Release -- --no-progress

# TUnit
dotnet run --project ./Calculator.Tests.TUnit/Calculator.Tests.TUnit.csproj --no-build --configuration Release -- --disable-logo --no-progress

The following table shows the results from my machine:

NUnit	TUnit (source generator DISABLED)	TUnit (source generator ENABLED)
~2s	~2s	~11s

These results indicate that test execution is significantly slower when the source generator is enabled, which is unexpected since source-generated tests are generally considered more performant than reflected tests. Although TUnit supports Ahead-of-Time (AOT) compilation, we cannot use it in our current project.

Is this the expected behavior for the TUnit source generator?

[thomhurst]

Can you try on 1.18 please?

[hainguyenthanh5-oss]

@thomhurst Thank you for your prompt support. The performance of source-generated tests in TUnit v1.18 was significantly improved. However, it's still not as good as the reflection-based test running performance using --reflection. Moreover, reflection test running is still not as fast as the NUnit runner. Could you please review and optimize the reflection test runner more?