Skip to content

benches: reduce codspeed flakiness#470

Merged
prestwich merged 3 commits intorecmo:mainfrom
DaniPopes:codspeed-flaky
May 26, 2025
Merged

benches: reduce codspeed flakiness#470
prestwich merged 3 commits intorecmo:mainfrom
DaniPopes:codspeed-flaky

Conversation

@DaniPopes
Copy link
Contributor

@DaniPopes DaniPopes commented May 26, 2025

Some smaller benchmarks change results on every run; this is due to either:

  • using rand::rng which is not deterministic
  • so few instructions that it was benchmarking loading/storing to memory

This PR fixes both, and all benchmarks should be stable; the benchmarks still contain a lot of copies but that's out of our control: CodSpeedHQ/codspeed-rust#98

@DaniPopes DaniPopes requested a review from prestwich as a code owner May 26, 2025 16:22
@DaniPopes DaniPopes marked this pull request as draft May 26, 2025 16:22
@codspeed-hq
Copy link

codspeed-hq bot commented May 26, 2025

CodSpeed Performance Report

Merging #470 will degrade performances by 15.5%

Comparing DaniPopes:codspeed-flaky (1a3b792) with main (767f07d)

Summary

⚡ 72 improvements
❌ 3 regressions
✅ 150 untouched benchmarks
⁉️ 1 dropped benchmarks

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Benchmarks breakdown

Benchmark BASE HEAD Change
add/4096 3.9 µs 3.3 µs +18.07%
add/512 537.2 ns 440.6 ns +21.94%
add_mod/192 719.4 ns 564.2 ns +27.52%
add_mod/384 2.7 µs 2.3 µs +13.84%
add_mod/4096 10.3 µs 8.6 µs +19.01%
add_mod/64 630.3 ns 567.8 ns +11.01%
⁉️ algo/addmul_n/0 < 1 ns N/A N/A
algo/div/3x2/ref 245.6 ns 273.6 ns -10.25%
algo/div/reciprocal/ref 154.4 ns 182.8 ns -15.5%
algorithms/gcd/matrix/from_u64 331.7 ns 301.4 ns +10.05%
algorithms/gcd/matrix/from_u64_prefix 578.1 ns 664.4 ns -13%
cmp/128 91.7 ns 29.7 ns ×3.1
cmp/192 152.2 ns 29.7 ns ×5.1
cmp/256 154.4 ns 29.7 ns ×5.2
cmp/384 275.6 ns 29.7 ns ×9.3
cmp/4096 2.9 µs 2.1 µs +41.32%
cmp/512 280 ns 29.7 ns ×9.4
cmp/64 61.1 ns 29.7 ns ×2.1
div_rem/4096/4096 5.6 µs 5.1 µs +11.13%
div_rem/4096/64 5.7 µs 5.1 µs +11.62%
... ... ... ... ...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

@DaniPopes DaniPopes force-pushed the codspeed-flaky branch 3 times, most recently from 9c9d9c1 to 2e98fc3 Compare May 26, 2025 17:17
Reduces CI time by avoiding having to build unused dev-deps and the
ruint test suite.
@DaniPopes DaniPopes marked this pull request as ready for review May 26, 2025 18:37
@DaniPopes DaniPopes requested a review from recmo as a code owner May 26, 2025 18:37
@prestwich prestwich merged commit 7a75e0e into recmo:main May 26, 2025
18 of 19 checks passed
@DaniPopes DaniPopes deleted the codspeed-flaky branch May 26, 2025 19:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants