Skip to content

👷 ci(perf): measure CodSpeed with macro walltime#222

Merged
gaborbernat merged 1 commit into
tox-dev:mainfrom
gaborbernat:fix/codspeed-walltime
Jul 13, 2026
Merged

👷 ci(perf): measure CodSpeed with macro walltime#222
gaborbernat merged 1 commit into
tox-dev:mainfrom
gaborbernat:fix/codspeed-walltime

Conversation

@gaborbernat

Copy link
Copy Markdown
Member

CodSpeed produced incompatible regression sets for PR #213 although the benchmarked OCI and PyPI production sources did not change; one run reported six regressions up to 99.79%, while two reruns replaced them with different sets. Compare all three measurements in #221.

CodSpeed documents cache and allocator variance when CPU simulation runs multiple benchmarks in one binary, and recommends macro-runner walltime for system and I/O suites. The workflow now builds and runs both operation suites in walltime mode while retaining pinned codspeed-macro hardware.

Walltime repeats each case and includes file-system behavior used by the end-to-end HTTP benchmarks. Performance regressions still fail the required CodSpeed analysis.

CPU simulation produced incompatible regression sets for unchanged OCI and
PyPI benchmark code. PR tox-dev#213 changed one search test between measured heads;
the reported PyPI regressions changed names and magnitudes.

Run the mixed microbenchmark and end-to-end operation suites with walltime on
the existing pinned macro runner. CodSpeed recommends this mode for I/O and
system-level work; repeated sampling avoids one-shot cache and allocator swings.

Fixes tox-dev#221
@gaborbernat gaborbernat added the bug Something isn't working label Jul 12, 2026
@read-the-docs-community

Copy link
Copy Markdown

@codspeed-hq

codspeed-hq Bot commented Jul 12, 2026

Copy link
Copy Markdown

Merging this PR will create unknown performance changes

🆕 29 new benchmarks
⏩ 62 skipped benchmarks1

Performance Changes

Mode Benchmark BASE HEAD Efficiency
🆕 WallTime oci_blob_serve N/A 161 µs N/A
🆕 WallTime oci_manifest_by_digest N/A 23.6 µs N/A
🆕 WallTime oci_tags_list N/A 18.6 µs N/A
🆕 WallTime oci_version_check N/A 5.5 µs N/A
🆕 WallTime normalize_name N/A 267 ns N/A
🆕 WallTime parse_version N/A 655 ns N/A
🆕 WallTime parse_version_specifiers N/A 678 ns N/A
🆕 WallTime sorted_desc N/A 178.6 µs N/A
🆕 WallTime detail_html[large] N/A 4.6 ms N/A
🆕 WallTime detail_html[small] N/A 37.8 µs N/A
🆕 WallTime detail_json[large] N/A 1.1 ms N/A
🆕 WallTime detail_json[small] N/A 6.8 µs N/A
🆕 WallTime distribution_filename N/A 843 ns N/A
🆕 WallTime index_html N/A 298.2 µs N/A
🆕 WallTime index_json N/A 117.9 µs N/A
🆕 WallTime metadata N/A 6 µs N/A
🆕 WallTime detail_html[large] N/A 305.2 µs N/A
🆕 WallTime detail_html[small] N/A 2.3 µs N/A
🆕 WallTime index_html N/A 56.7 µs N/A
🆕 WallTime legacy_json[large] N/A 7.6 ms N/A
... ... ... ... ... ...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.


Comparing gaborbernat:fix/codspeed-walltime (9e1c414) with main (d29b7fe)

Open in CodSpeed

Footnotes

  1. 62 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@gaborbernat gaborbernat merged commit 218fe1e into tox-dev:main Jul 13, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant