Skip to content

Fix DiskSpd parser for v2.2 intermediate CPU-topology headers (exactly-64-vCPU case) #737

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ericavella merged 1 commit into from
Jun 26, 2026
Merged

Fix DiskSpd parser for v2.2 intermediate CPU-topology headers (exactly-64-vCPU case) #737

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion VERSION
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
3.3.12
3.3.13
75 changes: 75 additions & 0 deletions src/VirtualClient/VirtualClient.Actions.UnitTests/DiskSpd/DiskSpdMetricsParserTests.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -330,5 +330,80 @@ public void DiskSpdParserVerifyForCoreCountGreaterThan64WhichAddsProcessorGroupi
MetricAssert.Exists(metrics, "total latency 75th", 2.819, "ms");
MetricAssert.Exists(metrics, "total latency 90th", 7.472, "ms");
}

[Test]
public void DiskSpdParserVerifyV220FormatOnMultiSocketSingleProcessorGroupSystem()
{
// Authentic DiskSpd 2.2 output captured from a 64-vCPU Azure VM (Standard_E64ds_v5:
// 2 sockets, 2 NUMA nodes, 1 processor group). Because DiskSpd emits each topology column
// (Socket/Node/Group/Core/Class) only when the system has more than one of that unit, this
// exactly-64-vCPU system produces the intermediate header "Socket | Node | Core | CPU" -
// it has the Socket and Node columns but NO Group column (64 vCPUs fit in one group). This
// is the customer's "exactly 64 vCPUs fails" case: it matches neither the full
// "Socket | Node | Group | Core | CPU" header nor the bare "Core | CPU" header that earlier
// fixes special-cased, so before NormalizeCpuTable it threw "The given key 'CPU' was not
// present in the dictionary" (the title was inserted mid-line, keying the section
// "Socket | Node | CPU").
string results = File.ReadAllText(MockFixture.GetDirectory(typeof(DiskSpdMetricsParserTests), "Examples", "DiskSpd", "DiskSpdExample-WriteOnly-v2.2.0-MultiSocketSingleGroup.txt"));
var parser = new DiskSpdMetricsParser(results, "diskspd.exe -c64M -b4K -r4K -t4 -o4 -w100 -d5 -Suw -W2 -D -L -Rtext C:\\dskspd\\testfile.dat");

IList<Metric> metrics = parser.Parse();

// cpu metrics - the Socket, Node and Core columns must all be stripped so rows are keyed by
// the CPU number (0..5). The User/Kernel values (e.g. cpu 1) prove the columns are not
// shifted after dropping three leading topology columns.
MetricAssert.Exists(metrics, "cpu usage 0", 1.25, "percentage");
MetricAssert.Exists(metrics, "cpu usage 1", 3.12, "percentage");
MetricAssert.Exists(metrics, "cpu usage 2", 0.62, "percentage");
MetricAssert.Exists(metrics, "cpu usage 3", 0.94, "percentage");
MetricAssert.Exists(metrics, "cpu usage average", 0.09, "percentage");
MetricAssert.Exists(metrics, "cpu user 1", 0.31, "percentage");
MetricAssert.Exists(metrics, "cpu kernel 1", 2.81, "percentage");
MetricAssert.Exists(metrics, "cpu kernel average", 0.09, "percentage");

// Total IO + Write IO + latency must still parse end-to-end.
MetricAssert.Exists(metrics, "total bytes total", 69226496, "bytes");
MetricAssert.Exists(metrics, "total throughput total", 13.18, "MiB/s");
MetricAssert.Exists(metrics, "write iops total", 3374.97, "iops");
MetricAssert.Exists(metrics, "write latency 50th", 4.738, "ms");
MetricAssert.Exists(metrics, "total latency max", 46.704, "ms");
}

[Test]
public void DiskSpdParserVerifyV220FormatOnMultiNumaMultiProcessorGroupSystem()
{
// Authentic DiskSpd 2.2 output captured from a 96-vCPU Azure VM (Standard_D96as_v5:
// 1 socket, 2 NUMA nodes, 2 processor groups). This produces the intermediate header
// "Node | Group | Core | CPU" - it has the Node, Group and Core columns but no Socket
// column. The Node and Core columns must be dropped while the Group column is RETAINED, so
// that the group-relative CPU number is mapped to a unique processor id across the group
// boundary (group 1's CPU 0 becomes id 64 = 64*group + cpu). Without retaining Group, the
// group-relative CPU numbers (0,1,2,.. repeated per group) would collide. Before the
// dynamic fix this header threw "The given key 'CPU' was not present in the dictionary".
string results = File.ReadAllText(MockFixture.GetDirectory(typeof(DiskSpdMetricsParserTests), "Examples", "DiskSpd", "DiskSpdExample-WriteOnly-v2.2.0-MultiNumaMultiGroup.txt"));
var parser = new DiskSpdMetricsParser(results, "diskspd.exe -c64M -b4K -r4K -t4 -o4 -w100 -d5 -Suw -W2 -D -L -Rtext C:\\dskspd\\testfile.dat");

IList<Metric> metrics = parser.Parse();

// Group 0 CPUs are keyed 0..3 (64*0 + cpu); the User/Kernel values prove the Node and Core
// columns were stripped without shifting the value columns.
MetricAssert.Exists(metrics, "cpu usage 0", 0.31, "percentage");
MetricAssert.Exists(metrics, "cpu usage 1", 4.08, "percentage");
MetricAssert.Exists(metrics, "cpu usage 2", 3.13, "percentage");
MetricAssert.Exists(metrics, "cpu user 1", 1.57, "percentage");
MetricAssert.Exists(metrics, "cpu kernel 1", 2.51, "percentage");

// Group 1 CPUs are offset to 64..67 (64*1 + cpu) so they do not collide with group 0.
MetricAssert.Exists(metrics, "cpu usage 64", 0, "percentage");
MetricAssert.Exists(metrics, "cpu usage 65", 0, "percentage");
MetricAssert.Exists(metrics, "cpu usage average", 0.72, "percentage");

// Total IO + Write IO + latency must still parse end-to-end.
MetricAssert.Exists(metrics, "total bytes total", 24952832, "bytes");
MetricAssert.Exists(metrics, "total throughput total", 4.75, "MiB/s");
MetricAssert.Exists(metrics, "write iops total", 1215.24, "iops");
MetricAssert.Exists(metrics, "write latency 50th", 6.402, "ms");
MetricAssert.Exists(metrics, "total latency max", 53.872, "ms");
}
}
}
76 changes: 76 additions & 0 deletions ...ctions.UnitTests/Examples/DiskSpd/DiskSpdExample-WriteOnly-v2.2.0-MultiNumaMultiGroup.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@

Command Line: C:\dskspd\amd64\diskspd.exe -c64M -b4K -r4K -t4 -o4 -w100 -d5 -Suw -W2 -D -L -Rtext C:\dskspd\testfile.dat

System information:

computer name: alexwill-amd96
start time: 2026/06/26 17:40:00 UTC

cpu count: 96
core count: 48
group count: 2
node count: 2
socket count: 1
heterogeneous cores: n

Results for timespan 1:
*******************************************************************************

actual test time: 5.00s
thread count: 4

Node | Group | Core | CPU | Usage | User | Kernel | Idle
------------------------------------------------------------------
0| 0| 0| 0| 0.31%| 0.00%| 0.31%| 99.69%
0| 0| 0| 1| 4.08%| 1.57%| 2.51%| 95.92%
0| 0| 1| 2| 3.13%| 1.88%| 1.25%| 96.87%
0| 0| 1| 3| 1.57%| 0.63%| 0.94%| 98.43%
1| 1| 0| 0| 0.00%| 0.00%| 0.00%| 100.00%
1| 1| 0| 1| 0.00%| 0.00%| 0.00%| 100.00%
1| 1| 1| 2| 0.00%| 0.00%| 0.00%| 100.00%
1| 1| 1| 3| 0.00%| 0.00%| 0.00%| 100.00%
------------------------------------------------------------------
avg.| 0.72%| 0.48%| 0.24%| 99.28%

Total IO
thread | bytes | I/Os | MiB/s | I/O per s | AvgLat | IopsStdDev | LatStdDev | file
------------------------------------------------------------------------------------------------------------------
0 | 12554240 | 3065 | 2.39 | 611.41 | 6.491 | 280.11 | 13.500 | C:\dskspd\testfile.dat (64MiB)
1 | 12398592 | 3027 | 2.36 | 603.83 | 6.572 | 281.21 | 13.668 | C:\dskspd\testfile.dat (64MiB)
------------------------------------------------------------------------------------------------------------------
total: 24952832 | 6092 | 4.75 | 1215.24 | 6.531 | 396.90 | 13.584

Read IO
thread | bytes | I/Os | MiB/s | I/O per s | AvgLat | IopsStdDev | LatStdDev | file
------------------------------------------------------------------------------------------------------------------
0 | 0 | 0 | 0.00 | 0.00 | 0.000 | 0.00 | N/A | C:\dskspd\testfile.dat (64MiB)
1 | 0 | 0 | 0.00 | 0.00 | 0.000 | 0.00 | N/A | C:\dskspd\testfile.dat (64MiB)
------------------------------------------------------------------------------------------------------------------
total: 0 | 0 | 0.00 | 0.00 | 0.000 | 0.00 | N/A

Write IO
thread | bytes | I/Os | MiB/s | I/O per s | AvgLat | IopsStdDev | LatStdDev | file
------------------------------------------------------------------------------------------------------------------
0 | 12554240 | 3065 | 2.39 | 611.41 | 6.491 | 280.11 | 13.500 | C:\dskspd\testfile.dat (64MiB)
1 | 12398592 | 3027 | 2.36 | 603.83 | 6.572 | 281.21 | 13.668 | C:\dskspd\testfile.dat (64MiB)
------------------------------------------------------------------------------------------------------------------
total: 24952832 | 6092 | 4.75 | 1215.24 | 6.531 | 396.90 | 13.584

Total latency distribution:
%-ile | Read (ms) | Write (ms) | Total (ms)
----------------------------------------------
min | N/A | 1.100 | 1.100
25th | N/A | 4.214 | 4.214
50th | N/A | 6.402 | 6.402
75th | N/A | 8.298 | 8.298
90th | N/A | 10.887 | 10.887
95th | N/A | 12.994 | 12.994
99th | N/A | 18.342 | 18.342
3-nines | N/A | 44.981 | 44.981
4-nines | N/A | 52.115 | 52.115
5-nines | N/A | 53.872 | 53.872
6-nines | N/A | 53.872 | 53.872
7-nines | N/A | 53.872 | 53.872
8-nines | N/A | 53.872 | 53.872
9-nines | N/A | 53.872 | 53.872
max | N/A | 53.872 | 53.872
74 changes: 74 additions & 0 deletions ...ons.UnitTests/Examples/DiskSpd/DiskSpdExample-WriteOnly-v2.2.0-MultiSocketSingleGroup.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@

Command Line: C:\dskspd\amd64\diskspd.exe -c64M -b4K -r4K -t4 -o4 -w100 -d5 -Suw -W2 -D -L -Rtext C:\dskspd\testfile.dat

System information:

computer name: alexwill-int64
start time: 2026/06/26 17:55:00 UTC

cpu count: 64
core count: 32
group count: 1
node count: 2
socket count: 2
heterogeneous cores: n

Results for timespan 1:
*******************************************************************************

actual test time: 5.00s
thread count: 4

Socket | Node | Core | CPU | Usage | User | Kernel | Idle
-------------------------------------------------------------------
0| 0| 0| 0| 1.25%| 0.00%| 1.25%| 98.75%
0| 0| 0| 1| 3.12%| 0.31%| 2.81%| 96.88%
0| 0| 1| 2| 0.62%| 0.00%| 0.62%| 99.38%
0| 0| 1| 3| 0.94%| 0.00%| 0.94%| 99.06%
0| 0| 2| 4| 0.00%| 0.00%| 0.00%| 100.00%
0| 0| 2| 5| 0.00%| 0.00%| 0.00%| 100.00%
-------------------------------------------------------------------
avg.| 0.09%| 0.00%| 0.09%| 99.91%

Total IO
thread | bytes | I/Os | MiB/s | I/O per s | AvgLat | IopsStdDev | LatStdDev | file
------------------------------------------------------------------------------------------------------------------
0 | 17256448 | 4213 | 3.29 | 841.30 | 4.753 | 45.39 | 9.686 | C:\dskspd\testfile.dat (64MiB)
1 | 17440768 | 4258 | 3.32 | 850.28 | 4.703 | 49.64 | 9.635 | C:\dskspd\testfile.dat (64MiB)
------------------------------------------------------------------------------------------------------------------
total: 69226496 | 16901 | 13.18 | 3374.97 | 4.739 | 176.61 | 9.670

Read IO
thread | bytes | I/Os | MiB/s | I/O per s | AvgLat | IopsStdDev | LatStdDev | file
------------------------------------------------------------------------------------------------------------------
0 | 0 | 0 | 0.00 | 0.00 | 0.000 | 0.00 | N/A | C:\dskspd\testfile.dat (64MiB)
1 | 0 | 0 | 0.00 | 0.00 | 0.000 | 0.00 | N/A | C:\dskspd\testfile.dat (64MiB)
------------------------------------------------------------------------------------------------------------------
total: 0 | 0 | 0.00 | 0.00 | 0.000 | 0.00 | N/A

Write IO
thread | bytes | I/Os | MiB/s | I/O per s | AvgLat | IopsStdDev | LatStdDev | file
------------------------------------------------------------------------------------------------------------------
0 | 17256448 | 4213 | 3.29 | 841.30 | 4.753 | 45.39 | 9.686 | C:\dskspd\testfile.dat (64MiB)
1 | 17440768 | 4258 | 3.32 | 850.28 | 4.703 | 49.64 | 9.635 | C:\dskspd\testfile.dat (64MiB)
------------------------------------------------------------------------------------------------------------------
total: 69226496 | 16901 | 13.18 | 3374.97 | 4.739 | 176.61 | 9.670

Total latency distribution:
%-ile | Read (ms) | Write (ms) | Total (ms)
----------------------------------------------
min | N/A | 1.000 | 1.000
25th | N/A | 5.877 | 5.877
50th | N/A | 4.738 | 4.738
75th | N/A | 6.114 | 6.114
90th | N/A | 8.291 | 8.291
95th | N/A | 10.435 | 10.435
99th | N/A | 14.864 | 14.864
3-nines | N/A | 38.880 | 38.880
4-nines | N/A | 45.213 | 45.213
5-nines | N/A | 46.704 | 46.704
6-nines | N/A | 46.704 | 46.704
7-nines | N/A | 46.704 | 46.704
8-nines | N/A | 46.704 | 46.704
9-nines | N/A | 46.704 | 46.704
max | N/A | 46.704 | 46.704
Loading
Loading