👷 ci(perf): measure CodSpeed with macro walltime#222
Merged
Conversation
CPU simulation produced incompatible regression sets for unchanged OCI and PyPI benchmark code. PR tox-dev#213 changed one search test between measured heads; the reported PyPI regressions changed names and magnitudes. Run the mixed microbenchmark and end-to-end operation suites with walltime on the existing pinned macro runner. CodSpeed recommends this mode for I/O and system-level work; repeated sampling avoids one-shot cache and allocator swings. Fixes tox-dev#221
Merging this PR will create unknown performance changes
Performance Changes
Comparing Footnotes
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CodSpeed produced incompatible regression sets for PR #213 although the benchmarked OCI and PyPI production sources did not change; one run reported six regressions up to 99.79%, while two reruns replaced them with different sets. Compare all three measurements in #221.
CodSpeed documents cache and allocator variance when CPU simulation runs multiple benchmarks in one binary, and recommends macro-runner walltime for system and I/O suites. The workflow now builds and runs both operation suites in walltime mode while retaining pinned
codspeed-macrohardware.Walltime repeats each case and includes file-system behavior used by the end-to-end HTTP benchmarks. Performance regressions still fail the required CodSpeed analysis.