Parallelize microbenchmarks and run them more times

[igoragoli]

⚠️ The PR description was modified to describe new changes.

• TODO: Resolve all TODOs before merging to use production branches/images instead of testing ones.

What does this PR do?

• Runs benchmarks 6 times (configurable by REPETITIONS on benchmarks/execution.yml) to reduce inter-run variability.
  • I had initially tested with 10 repetitions. That would make the CI job go from 30 minutes (current state, without repetitions) to 50 minutes. I brought it down to 6 repetitions, so the CI job duration is kept as 30 minutes.
  • If flakiness is still high, we can bump the number of repetitions or configure the threshold on the analysis step over at benchmarking-platform at the dd-trace-rb branch.
• To prevent extremely long CI jobs, runs benchmarks in parallel with CPU isolation.
  • Since we have 24 available CPUs, the 13 benchmarks were split into two arbitrary groups. Each group is defined on benchmarks/execution.yml
  • Every benchmark is run with 2 CPUs (configurable by CPUS_PER_BENCHMARK on the benchmarks/execution.yml job).

Motivation:

https://datadoghq.atlassian.net/browse/APMSP-2544

Change log entry

None.

Additional Notes:

How to test the change?

Execution and reporting

• CI pipeline runs successfully with parallel benchmark execution
  • "other" group and "profiling" group.
• Artifacts collected correctly for all benchmarks (including di_instrument.rb with legacy naming fallback)
  • Artifacts on the microbenchmarks-pr-comment job, which has access to all artifacts for candidates and baselines on all repetitions
• Results on the BP UI
  • Results for "profiling - Allocations" clearly showing six different runs. This means runs are being combined correctly.
• Results posted as PR comment

Reducing flakiness
The effect of multiple repetitions and CPU isolation on result variability was tested and reported in this document: https://datadoghq.atlassian.net/wiki/x/egJ3cAE

25 out of ~45 scenarios were flaky before fixes, 0 are flaky after fixes.

These tests used 10 repetitions. While this PR introduces 6 repetitions, it should already bring the flakiness down.

[github-actions]

Thank you for updating Change log entry section 👏

^{Visited at: 2026-02-20 09:48:24 UTC}

[datadog-official]

✅ Tests

🎉 All green!

❄️ No new flaky tests detected
🧪 All tests passed

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 95.10% (-0.00%)

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 61a7f68 | Docs | Datadog PR Page | Was this helpful? React with 👍/👎 or give us feedback!}

[pr-commenter]

Benchmarks

Benchmark execution time: 2026-03-03 11:20:54

Comparing candidate commit 61a7f68 in PR branch augusto/2544.flakiness with baseline commit 158e037 in branch master.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 46 metrics, 0 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

• 🟩 = significantly better candidate vs. baseline
• 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

[ivoanjo]

I think the PR seems in good shape? My only question is -- I see the new jobs coming up in GitLab -- is there a way to check that the way we're exposing the files is still correct?

E.g. how do we test that we haven't broken our benchmark reporting code?

[igoragoli]

Hey @ivoanjo and @p-datadog, thank you for the reviews!

I'll answer Oleg on the conversation thread. To answer Ivo:

E.g. how do we test that we haven't broken our benchmark reporting code?

Great point, and while reports on the BP UI are working as expected, PR comments from one microbenchmarking job will overwrite the other. Results have to be combined somehow.

[igoragoli]

Hi! For visibility, I re-requested reviews from all that have reviewed, since you have pointed out towards different fixes.

[ivoanjo]

👍 LGTM

We're in the middle of a release so merging to master is blocked, it should be unblocked later today/by Monday.

At this point I don't see any reason to not to give this a try and then if there's some extra adjustment needed we'll do at as a follow-up PR.

[igoragoli]

#5313 (comment) is awesome. We're comparing benchmarks from this branch against master. Since there were no code changes that should impact performance, we would indeed expect to see 0 improvements/regression.

Corroborates with Ruby microbenchmark stability experiments showing that 10 reps took out all flakiness. 6 reps seems to be sufficient.

[p-datadog]

I asked claude to review this PR and it came up with a couple of pages of feedback, I DM'd @igoragoli to go over it since I don't understand for about half of it whether it addresses real issues or just theoretical ones.

I wanted to see about DRYing the file names (which take up a large part of the diff) but claude also spotted some missing validation which seems to be at least plausible and improvements in error reporting.

[ivoanjo]

I asked claude to review this PR and it came up with a couple of pages of feedback, I DM'd @igoragoli to go over it since I don't understand for about half of it whether it addresses real issues or just theoretical ones.

I wanted to see about DRYing the file names (which take up a large part of the diff) but claude also spotted some missing validation which seems to be at least plausible and improvements in error reporting.

To be clear, is any of that feedback blocking? In this age of "feedback from AI is free" I think it's even more important for the line between "hey maybe this would a cool improvement" and "hey we should really fix this, it's a confusing/dangerous/weird footgun" to be clear for folks :)

[p-datadog]

I am not sure - that's what i wanted to find out.

I think we are all in the summit anyway at the moment.