From c66b399e3c7738b3d87b6bc90b7168d144e39567 Mon Sep 17 00:00:00 2001 From: not-matthias Date: Wed, 20 May 2026 18:42:37 +0200 Subject: [PATCH 1/3] docs(agents): propose macOS-only fib benchmark for flamegraph testing --- ...26-05-20-macos-fib-flamegraph-benchmark.md | 109 ++++++++++++++++++ 1 file changed, 109 insertions(+) create mode 100644 .agents/docs/2026-05-20-macos-fib-flamegraph-benchmark.md diff --git a/.agents/docs/2026-05-20-macos-fib-flamegraph-benchmark.md b/.agents/docs/2026-05-20-macos-fib-flamegraph-benchmark.md new file mode 100644 index 0000000..ac6218c --- /dev/null +++ b/.agents/docs/2026-05-20-macos-fib-flamegraph-benchmark.md @@ -0,0 +1,109 @@ +# Proposal: macOS-only Fib benchmark for flamegraph testing + +## Goal + +Add a benchmark whose only purpose is to exercise CodSpeed's flamegraph +symbolization pipeline on macOS (codspeed-macro runners). It should: + +- run on macOS (locally and on `codspeed-macro` CI) +- be **skipped** on Linux + Windows so it doesn't pollute the regular walltime + results or fail on platforms where the flamegraph path isn't applicable +- have a stack shape that produces an obviously-recognisable flamegraph (deep + recursion = `fib`) + +## Findings + +- `examples/example-gradle/src/jmh/java/bench/FibBenchmark.java` already + defines a recursive `fib(30)` benchmark. Reusing the same shape (but a + dedicated class) keeps the flamegraph trivially recognisable. +- The Gradle JMH plugin (`me.champeau.jmh` v0.7.2) exposes `includes` / + `excludes` regex filters. We're already using `includes` to curate the CI + subset in `examples/example-gradle/build.gradle.kts:32`. +- CI is matrixed in `.github/workflows/ci.yml`. The flamegraph-relevant job + is `walltime-benchmarks` on `codspeed-macro` — macOS-only by definition. + The cross-platform jobs (`build-and-run-gradle`, `build-and-run-maven`) run + the JMH JAR on Linux + Windows + macOS, so a naive new benchmark would also + execute there. +- JMH itself has no `@SkipOnPlatform`-style annotation. The two reasonable + gating mechanisms are: + 1. Filter at the runner level (Gradle `excludes`) + 2. Throw / `Assume.assumeTrue(...)` inside `@Setup` (JMH treats a thrown + exception in setup as a benchmark error, not a skip — so this is + **not** suitable). + +## Options + +### 1. Gate via Gradle `excludes` based on `OperatingSystem` (recommended) + +Add `FibFlamegraphBenchmark` next to the existing `FibBenchmark`, and in +`examples/example-gradle/build.gradle.kts` exclude it everywhere except macOS: + +```kotlin +import org.gradle.internal.os.OperatingSystem + +jmh { + // …existing config… + if (!OperatingSystem.current().isMacOsX) { + excludes.add(".*FibFlamegraphBenchmark.*") + } +} +``` + +The `CODSPEED_ENV` branch already filters down to a curated set on CI; add +`FibFlamegraphBenchmark` to that alternation so it runs on `codspeed-macro` +but stays out of the way on the regular Linux/Windows matrix. + +Pros: +- One source of truth (the build script). No per-benchmark scaffolding. +- `OperatingSystem.current()` is the same API Gradle uses internally and + doesn't require a new dependency. +- Easy to mirror in the Maven example (`` block). + +Cons: +- Excludes are regex-based; the benchmark class name must stay matchable. + Mitigation: keep `Flamegraph` in the class name and exclude on that token. + +### 2. Put macOS-only benchmarks in a dedicated package + filter + +E.g. `bench.macos.FibFlamegraphBenchmark`, then exclude `bench\\.macos\\..*` +when `!isMacOsX`. + +Pros: groups future macOS-only benchmarks naturally; no per-class regex. +Cons: more moving parts up front for a single benchmark. + +### 3. Use a System property check + early `return` + +Make the benchmark body a no-op on non-macOS. JMH still runs it, so the run +shows up in results as a near-zero benchmark. + +Pros: no build-script changes. +Cons: pollutes results, defeats the "skip on Linux" goal. **Rejected.** + +## Recommendation + +Go with **Option 1**. Smallest diff, no new package, and the gating lives in +the same place as the existing `CODSPEED_ENV` filter so future readers +discover it immediately. + +### Concrete change set (not yet applied) + +1. New file `examples/example-gradle/src/jmh/java/bench/FibFlamegraphBenchmark.java` + — copy of `FibBenchmark` with a distinct class name and a `@Param` value + that pushes recursion deep enough to be visually interesting in the + flamegraph (e.g. `35`). +2. Edit `examples/example-gradle/build.gradle.kts`: + - import `org.gradle.internal.os.OperatingSystem` + - add `excludes.add(".*FibFlamegraphBenchmark.*")` under + `if (!OperatingSystem.current().isMacOsX)` + - extend the `CODSPEED_ENV` include alternation to include + `FibFlamegraphBenchmark` +3. No CI changes needed: the existing `walltime-benchmarks` job on + `codspeed-macro` will pick it up automatically. + +## Open questions + +- Should the Maven example mirror this? The Maven example doesn't currently + run under `codspeed-macro`, so probably no — but worth confirming. +- Do we want to lock the `@Fork` / iteration counts down for the flamegraph + benchmark specifically (a single long iteration produces a cleaner + flamegraph than many short ones)? From 5e8f25388c668ae4f413b8f487a9e05386777ab9 Mon Sep 17 00:00:00 2001 From: not-matthias Date: Wed, 20 May 2026 18:44:34 +0200 Subject: [PATCH 2/3] feat(bench): add macOS-only FibFlamegraphBenchmark Exists solely to exercise CodSpeed's flamegraph symbolization on codspeed-macro runners. Gated to macOS via a Gradle excludes rule so the Linux/Windows matrix jobs skip it. --- examples/example-gradle/build.gradle.kts | 11 +++++++- .../java/bench/FibFlamegraphBenchmark.java | 27 +++++++++++++++++++ 2 files changed, 37 insertions(+), 1 deletion(-) create mode 100644 examples/example-gradle/src/jmh/java/bench/FibFlamegraphBenchmark.java diff --git a/examples/example-gradle/build.gradle.kts b/examples/example-gradle/build.gradle.kts index 756b5f7..25ebc5b 100644 --- a/examples/example-gradle/build.gradle.kts +++ b/examples/example-gradle/build.gradle.kts @@ -1,3 +1,5 @@ +import org.gradle.internal.os.OperatingSystem + plugins { java id("me.champeau.jmh") version "0.7.2" @@ -30,9 +32,16 @@ jmh { // regex passed to JMH, so multiple entries collapse to one pattern with // literal commas and match nothing. Use a single alternation instead. includes.set(listOf( - ".*(SleepBenchmark|BacktrackingBenchmark|FibBenchmark).*", + ".*(SleepBenchmark|BacktrackingBenchmark|FibBenchmark|FibFlamegraphBenchmark).*", )) } + + // FibFlamegraphBenchmark exists only to exercise the macOS flamegraph + // pipeline on codspeed-macro runners. Skip it on Linux/Windows so the + // cross-platform jobs don't run a redundant fib variant. + if (!OperatingSystem.current().isMacOsX) { + excludes.add(".*FibFlamegraphBenchmark.*") + } } sourceSets { diff --git a/examples/example-gradle/src/jmh/java/bench/FibFlamegraphBenchmark.java b/examples/example-gradle/src/jmh/java/bench/FibFlamegraphBenchmark.java new file mode 100644 index 0000000..669483a --- /dev/null +++ b/examples/example-gradle/src/jmh/java/bench/FibFlamegraphBenchmark.java @@ -0,0 +1,27 @@ +package bench; + +import java.util.concurrent.TimeUnit; +import org.openjdk.jmh.annotations.*; + +/** + * macOS-only benchmark whose recursion shape produces an easily-recognisable flamegraph. Gated to + * macOS via Gradle excludes in {@code build.gradle.kts}. + */ +@BenchmarkMode(Mode.AverageTime) +@OutputTimeUnit(TimeUnit.NANOSECONDS) +@State(Scope.Benchmark) +public class FibFlamegraphBenchmark { + + @Param({"35"}) + private int n; + + @Benchmark + public long fib() { + return fib(n); + } + + private static long fib(int n) { + if (n <= 1) return n; + return fib(n - 1) + fib(n - 2); + } +} From 8846f210dd7af9055cd3303c518b02ec9f3731cc Mon Sep 17 00:00:00 2001 From: not-matthias Date: Fri, 22 May 2026 19:07:12 +0200 Subject: [PATCH 3/3] fixup! feat(bench): add macOS-only FibFlamegraphBenchmark Switch benchmark selection from OS detection to a CODSPEED_BENCH_GROUP env var, and split the walltime job in two on codspeed-macro: the existing matrix runs Sleep/Backtracking/Fib across JDK distributions, and a new walltime-benchmarks-flamegraph job runs FibFlamegraphBenchmark with CODSPEED_BENCH_GROUP=flamegraph. --- .github/workflows/ci.yml | 20 ++++++++++++++++++++ examples/example-gradle/build.gradle.kts | 22 ++++++++++------------ 2 files changed, 30 insertions(+), 12 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index a293771..df9461d 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -102,6 +102,25 @@ jobs: mode: walltime run: ./gradlew :examples:example-gradle:jmh + walltime-benchmarks-flamegraph: + runs-on: codspeed-macro + steps: + - uses: actions/checkout@v4 + with: + submodules: true + - uses: actions/setup-java@v4 + with: + distribution: temurin + java-version: 21 + cache: 'gradle' + - name: Run FibFlamegraphBenchmark + uses: CodSpeedHQ/action@main + env: + CODSPEED_BENCH_GROUP: flamegraph + with: + mode: walltime + run: ./gradlew :examples:example-gradle:jmh + check: runs-on: ubuntu-latest if: always() @@ -111,6 +130,7 @@ jobs: - build-and-run-gradle - build-and-run-maven - walltime-benchmarks + - walltime-benchmarks-flamegraph steps: - uses: re-actors/alls-green@release/v1 with: diff --git a/examples/example-gradle/build.gradle.kts b/examples/example-gradle/build.gradle.kts index 25ebc5b..1cdb38a 100644 --- a/examples/example-gradle/build.gradle.kts +++ b/examples/example-gradle/build.gradle.kts @@ -1,5 +1,3 @@ -import org.gradle.internal.os.OperatingSystem - plugins { java id("me.champeau.jmh") version "0.7.2" @@ -31,16 +29,16 @@ jmh { // me.champeau.jmh joins `includes` with commas into a single positional // regex passed to JMH, so multiple entries collapse to one pattern with // literal commas and match nothing. Use a single alternation instead. - includes.set(listOf( - ".*(SleepBenchmark|BacktrackingBenchmark|FibBenchmark|FibFlamegraphBenchmark).*", - )) - } - - // FibFlamegraphBenchmark exists only to exercise the macOS flamegraph - // pipeline on codspeed-macro runners. Skip it on Linux/Windows so the - // cross-platform jobs don't run a redundant fib variant. - if (!OperatingSystem.current().isMacOsX) { - excludes.add(".*FibFlamegraphBenchmark.*") + // + // CODSPEED_BENCH_GROUP=flamegraph selects the macOS-only + // FibFlamegraphBenchmark used to exercise the flamegraph pipeline. + // Default is the regular benchmark set. + val pattern = if (System.getenv("CODSPEED_BENCH_GROUP") == "flamegraph") { + ".*FibFlamegraphBenchmark.*" + } else { + ".*(SleepBenchmark|BacktrackingBenchmark|FibBenchmark).*" + } + includes.set(listOf(pattern)) } }