Skip to content

🤖 Update Benchmark Results #5249

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
thomhurst merged 1 commit into from
Mar 26, 2026
Merged

🤖 Update Benchmark Results #5249

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 9 additions & 9 deletions docs/docs/benchmarks/AsyncTests.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ sidebar_position: 2
# AsyncTests Benchmark

:::info Last Updated
This benchmark was automatically generated on **2026-03-25** from the latest CI run.
This benchmark was automatically generated on **2026-03-26** from the latest CI run.

**Environment:** Ubuntu Latest • .NET SDK 10.0.201
:::
Expand All @@ -16,11 +16,11 @@ This benchmark was automatically generated on **2026-03-25** from the latest CI

| Framework | Version | Mean | Median | StdDev |
|-----------|---------|------|--------|--------|
| **TUnit** | 1.21.20 | 538.0 ms | 534.4 ms | 7.45 ms |
| NUnit | 4.5.1 | 695.5 ms | 694.5 ms | 9.98 ms |
| MSTest | 4.1.0 | 615.5 ms | 614.0 ms | 6.96 ms |
| xUnit3 | 3.2.2 | 749.3 ms | 741.9 ms | 14.70 ms |
| **TUnit (AOT)** | 1.21.20 | 122.9 ms | 122.9 ms | 0.41 ms |
| **TUnit** | 1.21.24 | 538.0 ms | 536.7 ms | 6.62 ms |
| NUnit | 4.5.1 | 695.9 ms | 692.2 ms | 11.73 ms |
| MSTest | 4.1.0 | 615.1 ms | 613.9 ms | 4.36 ms |
| xUnit3 | 3.2.2 | 742.7 ms | 741.6 ms | 4.43 ms |
| **TUnit (AOT)** | 1.21.24 | 122.2 ms | 122.2 ms | 0.13 ms |

## 📈 Visual Comparison

Expand Down Expand Up @@ -58,8 +58,8 @@ This benchmark was automatically generated on **2026-03-25** from the latest CI
xychart-beta
title "AsyncTests Performance Comparison"
x-axis ["TUnit", "NUnit", "MSTest", "xUnit3", "TUnit_AOT"]
y-axis "Time (ms)" 0 --> 900
bar [538, 695.5, 615.5, 749.3, 122.9]
y-axis "Time (ms)" 0 --> 892
bar [538, 695.9, 615.1, 742.7, 122.2]
```

## 🎯 Key Insights
Expand All @@ -72,4 +72,4 @@ This benchmark compares TUnit's performance against NUnit, MSTest, xUnit3 using
View the [benchmarks overview](/docs/benchmarks) for methodology details and environment information.
:::

*Last generated: 2026-03-25T00:37:35.045Z*
*Last generated: 2026-03-26T00:39:42.200Z*
16 changes: 8 additions & 8 deletions docs/docs/benchmarks/BuildTime.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ sidebar_position: 8
# Build Performance Benchmark

:::info Last Updated
This benchmark was automatically generated on **2026-03-25** from the latest CI run.
This benchmark was automatically generated on **2026-03-26** from the latest CI run.

**Environment:** Ubuntu Latest • .NET SDK 10.0.201
:::
Expand All @@ -18,10 +18,10 @@ Compilation time comparison across frameworks:

| Framework | Version | Mean | Median | StdDev |
|-----------|---------|------|--------|--------|
| **TUnit** | 1.21.20 | 2.406 s | 2.411 s | 0.0323 s |
| Build_NUnit | 4.5.1 | 2.221 s | 2.220 s | 0.0204 s |
| Build_MSTest | 4.1.0 | 2.312 s | 2.314 s | 0.0147 s |
| Build_xUnit3 | 3.2.2 | 2.223 s | 2.223 s | 0.0220 s |
| **TUnit** | 1.21.24 | 2.661 s | 2.668 s | 0.0538 s |
| Build_NUnit | 4.5.1 | 2.430 s | 2.423 s | 0.0251 s |
| Build_MSTest | 4.1.0 | 2.517 s | 2.512 s | 0.0206 s |
| Build_xUnit3 | 3.2.2 | 2.438 s | 2.426 s | 0.0342 s |

## 📈 Visual Comparison

Expand Down Expand Up @@ -59,8 +59,8 @@ Compilation time comparison across frameworks:
xychart-beta
title "Build Time Comparison"
x-axis ["Build_TUnit", "Build_NUnit", "Build_MSTest", "Build_xUnit3"]
y-axis "Time (s)" 0 --> 3
bar [2.406, 2.221, 2.312, 2.223]
y-axis "Time (s)" 0 --> 4
bar [2.661, 2.43, 2.517, 2.438]
```

---
Expand All @@ -69,4 +69,4 @@ xychart-beta
View the [benchmarks overview](/docs/benchmarks) for methodology details and environment information.
:::

*Last generated: 2026-03-25T00:37:35.047Z*
*Last generated: 2026-03-26T00:39:42.202Z*
18 changes: 9 additions & 9 deletions docs/docs/benchmarks/DataDrivenTests.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ sidebar_position: 3
# DataDrivenTests Benchmark

:::info Last Updated
This benchmark was automatically generated on **2026-03-25** from the latest CI run.
This benchmark was automatically generated on **2026-03-26** from the latest CI run.

**Environment:** Ubuntu Latest • .NET SDK 10.0.201
:::
Expand All @@ -16,11 +16,11 @@ This benchmark was automatically generated on **2026-03-25** from the latest CI

| Framework | Version | Mean | Median | StdDev |
|-----------|---------|------|--------|--------|
| **TUnit** | 1.21.20 | 469.09 ms | 467.59 ms | 3.922 ms |
| NUnit | 4.5.1 | 553.19 ms | 551.93 ms | 6.252 ms |
| MSTest | 4.1.0 | 461.75 ms | 458.30 ms | 7.502 ms |
| xUnit3 | 3.2.2 | 588.54 ms | 585.95 ms | 6.872 ms |
| **TUnit (AOT)** | 1.21.20 | 24.19 ms | 24.26 ms | 0.928 ms |
| **TUnit** | 1.21.24 | 491.76 ms | 490.95 ms | 4.691 ms |
| NUnit | 4.5.1 | 607.71 ms | 606.74 ms | 6.281 ms |
| MSTest | 4.1.0 | 597.35 ms | 596.47 ms | 10.647 ms |
| xUnit3 | 3.2.2 | 651.92 ms | 649.88 ms | 7.973 ms |
| **TUnit (AOT)** | 1.21.24 | 24.26 ms | 24.10 ms | 0.427 ms |

## 📈 Visual Comparison

Expand Down Expand Up @@ -58,8 +58,8 @@ This benchmark was automatically generated on **2026-03-25** from the latest CI
xychart-beta
title "DataDrivenTests Performance Comparison"
x-axis ["TUnit", "NUnit", "MSTest", "xUnit3", "TUnit_AOT"]
y-axis "Time (ms)" 0 --> 707
bar [469.09, 553.19, 461.75, 588.54, 24.19]
y-axis "Time (ms)" 0 --> 783
bar [491.76, 607.71, 597.35, 651.92, 24.26]
```

## 🎯 Key Insights
Expand All @@ -72,4 +72,4 @@ This benchmark compares TUnit's performance against NUnit, MSTest, xUnit3 using
View the [benchmarks overview](/docs/benchmarks) for methodology details and environment information.
:::

*Last generated: 2026-03-25T00:37:35.045Z*
*Last generated: 2026-03-26T00:39:42.200Z*
18 changes: 9 additions & 9 deletions docs/docs/benchmarks/MassiveParallelTests.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ sidebar_position: 4
# MassiveParallelTests Benchmark

:::info Last Updated
This benchmark was automatically generated on **2026-03-25** from the latest CI run.
This benchmark was automatically generated on **2026-03-26** from the latest CI run.

**Environment:** Ubuntu Latest • .NET SDK 10.0.201
:::
Expand All @@ -16,11 +16,11 @@ This benchmark was automatically generated on **2026-03-25** from the latest CI

| Framework | Version | Mean | Median | StdDev |
|-----------|---------|------|--------|--------|
| **TUnit** | 1.21.20 | 662.5 ms | 662.0 ms | 2.79 ms |
| NUnit | 4.5.1 | 1,215.1 ms | 1,215.0 ms | 6.52 ms |
| MSTest | 4.1.0 | 2,947.3 ms | 2,948.9 ms | 5.35 ms |
| xUnit3 | 3.2.2 | 3,076.8 ms | 3,077.3 ms | 7.28 ms |
| **TUnit (AOT)** | 1.21.20 | 226.7 ms | 226.8 ms | 0.51 ms |
| **TUnit** | 1.21.24 | 711.6 ms | 713.6 ms | 7.42 ms |
| NUnit | 4.5.1 | 1,281.9 ms | 1,283.2 ms | 11.10 ms |
| MSTest | 4.1.0 | 3,026.4 ms | 3,027.7 ms | 7.74 ms |
| xUnit3 | 3.2.2 | 3,155.3 ms | 3,153.8 ms | 12.34 ms |
| **TUnit (AOT)** | 1.21.24 | 231.1 ms | 231.6 ms | 1.46 ms |

## 📈 Visual Comparison

Expand Down Expand Up @@ -58,8 +58,8 @@ This benchmark was automatically generated on **2026-03-25** from the latest CI
xychart-beta
title "MassiveParallelTests Performance Comparison"
x-axis ["TUnit", "NUnit", "MSTest", "xUnit3", "TUnit_AOT"]
y-axis "Time (ms)" 0 --> 3693
bar [662.5, 1215.1, 2947.3, 3076.8, 226.7]
y-axis "Time (ms)" 0 --> 3787
bar [711.6, 1281.9, 3026.4, 3155.3, 231.1]
```

## 🎯 Key Insights
Expand All @@ -72,4 +72,4 @@ This benchmark compares TUnit's performance against NUnit, MSTest, xUnit3 using
View the [benchmarks overview](/docs/benchmarks) for methodology details and environment information.
:::

*Last generated: 2026-03-25T00:37:35.045Z*
*Last generated: 2026-03-26T00:39:42.200Z*
18 changes: 9 additions & 9 deletions docs/docs/benchmarks/MatrixTests.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ sidebar_position: 5
# MatrixTests Benchmark

:::info Last Updated
This benchmark was automatically generated on **2026-03-25** from the latest CI run.
This benchmark was automatically generated on **2026-03-26** from the latest CI run.

**Environment:** Ubuntu Latest • .NET SDK 10.0.201
:::
Expand All @@ -16,11 +16,11 @@ This benchmark was automatically generated on **2026-03-25** from the latest CI

| Framework | Version | Mean | Median | StdDev |
|-----------|---------|------|--------|--------|
| **TUnit** | 1.21.20 | 612.8 ms | 613.0 ms | 2.96 ms |
| NUnit | 4.5.1 | 1,644.4 ms | 1,644.6 ms | 6.71 ms |
| MSTest | 4.1.0 | 1,548.6 ms | 1,548.7 ms | 8.61 ms |
| xUnit3 | 3.2.2 | 1,692.8 ms | 1,694.6 ms | 5.66 ms |
| **TUnit (AOT)** | 1.21.20 | 129.0 ms | 128.9 ms | 0.42 ms |
| **TUnit** | 1.21.24 | 590.0 ms | 589.0 ms | 6.13 ms |
| NUnit | 4.5.1 | 1,582.9 ms | 1,573.2 ms | 16.18 ms |
| MSTest | 4.1.0 | 1,491.7 ms | 1,488.6 ms | 7.93 ms |
| xUnit3 | 3.2.2 | 1,619.5 ms | 1,617.8 ms | 7.77 ms |
| **TUnit (AOT)** | 1.21.24 | 126.1 ms | 126.0 ms | 0.50 ms |

## 📈 Visual Comparison

Expand Down Expand Up @@ -58,8 +58,8 @@ This benchmark was automatically generated on **2026-03-25** from the latest CI
xychart-beta
title "MatrixTests Performance Comparison"
x-axis ["TUnit", "NUnit", "MSTest", "xUnit3", "TUnit_AOT"]
y-axis "Time (ms)" 0 --> 2032
bar [612.8, 1644.4, 1548.6, 1692.8, 129]
y-axis "Time (ms)" 0 --> 1944
bar [590, 1582.9, 1491.7, 1619.5, 126.1]
```

## 🎯 Key Insights
Expand All @@ -72,4 +72,4 @@ This benchmark compares TUnit's performance against NUnit, MSTest, xUnit3 using
View the [benchmarks overview](/docs/benchmarks) for methodology details and environment information.
:::

*Last generated: 2026-03-25T00:37:35.046Z*
*Last generated: 2026-03-26T00:39:42.201Z*
18 changes: 9 additions & 9 deletions docs/docs/benchmarks/ScaleTests.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ sidebar_position: 6
# ScaleTests Benchmark

:::info Last Updated
This benchmark was automatically generated on **2026-03-25** from the latest CI run.
This benchmark was automatically generated on **2026-03-26** from the latest CI run.

**Environment:** Ubuntu Latest • .NET SDK 10.0.201
:::
Expand All @@ -16,11 +16,11 @@ This benchmark was automatically generated on **2026-03-25** from the latest CI

| Framework | Version | Mean | Median | StdDev |
|-----------|---------|------|--------|--------|
| **TUnit** | 1.21.20 | 514.60 ms | 512.97 ms | 6.663 ms |
| NUnit | 4.5.1 | 681.33 ms | 679.38 ms | 8.723 ms |
| MSTest | 4.1.0 | 683.19 ms | 683.67 ms | 10.074 ms |
| xUnit3 | 3.2.2 | 693.17 ms | 693.08 ms | 6.816 ms |
| **TUnit (AOT)** | 1.21.20 | 36.08 ms | 36.18 ms | 2.507 ms |
| **TUnit** | 1.21.24 | 480.44 ms | 480.40 ms | 2.438 ms |
| NUnit | 4.5.1 | 627.92 ms | 628.14 ms | 5.839 ms |
| MSTest | 4.1.0 | 592.94 ms | 591.30 ms | 8.837 ms |
| xUnit3 | 3.2.2 | 638.96 ms | 637.65 ms | 7.348 ms |
| **TUnit (AOT)** | 1.21.24 | 33.91 ms | 33.31 ms | 2.604 ms |

## 📈 Visual Comparison

Expand Down Expand Up @@ -58,8 +58,8 @@ This benchmark was automatically generated on **2026-03-25** from the latest CI
xychart-beta
title "ScaleTests Performance Comparison"
x-axis ["TUnit", "NUnit", "MSTest", "xUnit3", "TUnit_AOT"]
y-axis "Time (ms)" 0 --> 832
bar [514.6, 681.33, 683.19, 693.17, 36.08]
y-axis "Time (ms)" 0 --> 767
bar [480.44, 627.92, 592.94, 638.96, 33.91]
```

## 🎯 Key Insights
Expand All @@ -72,4 +72,4 @@ This benchmark compares TUnit's performance against NUnit, MSTest, xUnit3 using
View the [benchmarks overview](/docs/benchmarks) for methodology details and environment information.
:::

*Last generated: 2026-03-25T00:37:35.046Z*
*Last generated: 2026-03-26T00:39:42.201Z*
18 changes: 9 additions & 9 deletions docs/docs/benchmarks/SetupTeardownTests.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ sidebar_position: 7
# SetupTeardownTests Benchmark

:::info Last Updated
This benchmark was automatically generated on **2026-03-25** from the latest CI run.
This benchmark was automatically generated on **2026-03-26** from the latest CI run.

**Environment:** Ubuntu Latest • .NET SDK 10.0.201
:::
Expand All @@ -16,11 +16,11 @@ This benchmark was automatically generated on **2026-03-25** from the latest CI

| Framework | Version | Mean | Median | StdDev |
|-----------|---------|------|--------|--------|
| **TUnit** | 1.21.20 | 536.7 ms | 536.3 ms | 4.18 ms |
| NUnit | 4.5.1 | 1,154.6 ms | 1,154.1 ms | 6.62 ms |
| MSTest | 4.1.0 | 1,096.5 ms | 1,096.9 ms | 3.67 ms |
| xUnit3 | 3.2.2 | 1,220.2 ms | 1,219.5 ms | 8.73 ms |
| **TUnit (AOT)** | 1.21.20 | NA | NA | NA |
| **TUnit** | 1.21.24 | 579.1 ms | 579.3 ms | 4.36 ms |
| NUnit | 4.5.1 | 1,188.3 ms | 1,185.1 ms | 11.11 ms |
| MSTest | 4.1.0 | 1,112.1 ms | 1,110.8 ms | 5.90 ms |
| xUnit3 | 3.2.2 | 1,242.5 ms | 1,244.2 ms | 6.78 ms |
| **TUnit (AOT)** | 1.21.24 | NA | NA | NA |

## 📈 Visual Comparison

Expand Down Expand Up @@ -58,8 +58,8 @@ This benchmark was automatically generated on **2026-03-25** from the latest CI
xychart-beta
title "SetupTeardownTests Performance Comparison"
x-axis ["TUnit", "NUnit", "MSTest", "xUnit3", "TUnit_AOT"]
y-axis "Time (ms)" 0 --> 1465
bar [536.7, 1154.6, 1096.5, 1220.2, 0]
y-axis "Time (ms)" 0 --> 1491
bar [579.1, 1188.3, 1112.1, 1242.5, 0]
```

## 🎯 Key Insights
Expand All @@ -72,4 +72,4 @@ This benchmark compares TUnit's performance against NUnit, MSTest, xUnit3 using
View the [benchmarks overview](/docs/benchmarks) for methodology details and environment information.
:::

*Last generated: 2026-03-25T00:37:35.046Z*
*Last generated: 2026-03-26T00:39:42.201Z*
10 changes: 5 additions & 5 deletions docs/docs/benchmarks/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ sidebar_position: 1
# Performance Benchmarks

:::info Last Updated
These benchmarks were automatically generated on **2026-03-25** from the latest CI run.
These benchmarks were automatically generated on **2026-03-26** from the latest CI run.

**Environment:** Ubuntu Latest • .NET SDK 10.0.201
:::
Expand Down Expand Up @@ -37,7 +37,7 @@ These benchmarks compare TUnit against the most popular .NET testing frameworks:

| Framework | Version Tested |
|-----------|----------------|
| **TUnit** | 1.21.20 |
| **TUnit** | 1.21.24 |
| **xUnit v3** | 3.2.2 |
| **NUnit** | 4.5.1 |
| **MSTest** | 4.1.0 |
Expand All @@ -56,10 +56,10 @@ The benchmarks measure real-world testing patterns:
### Environment

- **OS**: Ubuntu Latest (GitHub Actions)
- **Runtime**: .NET 10.0.5 (10.0.5, 10.0.526.15411), X64 RyuJIT x86-64-v4
- **Runtime**: .NET 10.0.5 (10.0.5, 10.0.526.15411), X64 RyuJIT x86-64-v3
- **SDK**: .NET SDK 10.0.201
- **Hardware**: GitHub Actions Standard Runner (Ubuntu)
- **Tool**: BenchmarkDotNet v0.15.8, Linux Ubuntu 24.04.3 LTS (Noble Numbat)
- **Tool**: BenchmarkDotNet v0.15.8, Linux Ubuntu 24.04.4 LTS (Noble Numbat)

### Why These Numbers Matter

Expand All @@ -80,4 +80,4 @@ These benchmarks run automatically daily via [GitHub Actions](https://github.com
Each benchmark runs multiple iterations with statistical analysis to ensure accuracy. Results may vary based on hardware and test characteristics.
:::

*Last generated: 2026-03-25T00:37:35.047Z*
*Last generated: 2026-03-26T00:39:42.202Z*
Loading
Loading