From c66b399e3c7738b3d87b6bc90b7168d144e39567 Mon Sep 17 00:00:00 2001 From: not-matthias Date: Wed, 20 May 2026 18:42:37 +0200 Subject: [PATCH 1/2] docs(agents): propose macOS-only fib benchmark for flamegraph testing --- ...26-05-20-macos-fib-flamegraph-benchmark.md | 109 ++++++++++++++++++ 1 file changed, 109 insertions(+) create mode 100644 .agents/docs/2026-05-20-macos-fib-flamegraph-benchmark.md diff --git a/.agents/docs/2026-05-20-macos-fib-flamegraph-benchmark.md b/.agents/docs/2026-05-20-macos-fib-flamegraph-benchmark.md new file mode 100644 index 0000000..ac6218c --- /dev/null +++ b/.agents/docs/2026-05-20-macos-fib-flamegraph-benchmark.md @@ -0,0 +1,109 @@ +# Proposal: macOS-only Fib benchmark for flamegraph testing + +## Goal + +Add a benchmark whose only purpose is to exercise CodSpeed's flamegraph +symbolization pipeline on macOS (codspeed-macro runners). It should: + +- run on macOS (locally and on `codspeed-macro` CI) +- be **skipped** on Linux + Windows so it doesn't pollute the regular walltime + results or fail on platforms where the flamegraph path isn't applicable +- have a stack shape that produces an obviously-recognisable flamegraph (deep + recursion = `fib`) + +## Findings + +- `examples/example-gradle/src/jmh/java/bench/FibBenchmark.java` already + defines a recursive `fib(30)` benchmark. Reusing the same shape (but a + dedicated class) keeps the flamegraph trivially recognisable. +- The Gradle JMH plugin (`me.champeau.jmh` v0.7.2) exposes `includes` / + `excludes` regex filters. We're already using `includes` to curate the CI + subset in `examples/example-gradle/build.gradle.kts:32`. +- CI is matrixed in `.github/workflows/ci.yml`. The flamegraph-relevant job + is `walltime-benchmarks` on `codspeed-macro` — macOS-only by definition. + The cross-platform jobs (`build-and-run-gradle`, `build-and-run-maven`) run + the JMH JAR on Linux + Windows + macOS, so a naive new benchmark would also + execute there. +- JMH itself has no `@SkipOnPlatform`-style annotation. The two reasonable + gating mechanisms are: + 1. Filter at the runner level (Gradle `excludes`) + 2. Throw / `Assume.assumeTrue(...)` inside `@Setup` (JMH treats a thrown + exception in setup as a benchmark error, not a skip — so this is + **not** suitable). + +## Options + +### 1. Gate via Gradle `excludes` based on `OperatingSystem` (recommended) + +Add `FibFlamegraphBenchmark` next to the existing `FibBenchmark`, and in +`examples/example-gradle/build.gradle.kts` exclude it everywhere except macOS: + +```kotlin +import org.gradle.internal.os.OperatingSystem + +jmh { + // …existing config… + if (!OperatingSystem.current().isMacOsX) { + excludes.add(".*FibFlamegraphBenchmark.*") + } +} +``` + +The `CODSPEED_ENV` branch already filters down to a curated set on CI; add +`FibFlamegraphBenchmark` to that alternation so it runs on `codspeed-macro` +but stays out of the way on the regular Linux/Windows matrix. + +Pros: +- One source of truth (the build script). No per-benchmark scaffolding. +- `OperatingSystem.current()` is the same API Gradle uses internally and + doesn't require a new dependency. +- Easy to mirror in the Maven example (`` block). + +Cons: +- Excludes are regex-based; the benchmark class name must stay matchable. + Mitigation: keep `Flamegraph` in the class name and exclude on that token. + +### 2. Put macOS-only benchmarks in a dedicated package + filter + +E.g. `bench.macos.FibFlamegraphBenchmark`, then exclude `bench\\.macos\\..*` +when `!isMacOsX`. + +Pros: groups future macOS-only benchmarks naturally; no per-class regex. +Cons: more moving parts up front for a single benchmark. + +### 3. Use a System property check + early `return` + +Make the benchmark body a no-op on non-macOS. JMH still runs it, so the run +shows up in results as a near-zero benchmark. + +Pros: no build-script changes. +Cons: pollutes results, defeats the "skip on Linux" goal. **Rejected.** + +## Recommendation + +Go with **Option 1**. Smallest diff, no new package, and the gating lives in +the same place as the existing `CODSPEED_ENV` filter so future readers +discover it immediately. + +### Concrete change set (not yet applied) + +1. New file `examples/example-gradle/src/jmh/java/bench/FibFlamegraphBenchmark.java` + — copy of `FibBenchmark` with a distinct class name and a `@Param` value + that pushes recursion deep enough to be visually interesting in the + flamegraph (e.g. `35`). +2. Edit `examples/example-gradle/build.gradle.kts`: + - import `org.gradle.internal.os.OperatingSystem` + - add `excludes.add(".*FibFlamegraphBenchmark.*")` under + `if (!OperatingSystem.current().isMacOsX)` + - extend the `CODSPEED_ENV` include alternation to include + `FibFlamegraphBenchmark` +3. No CI changes needed: the existing `walltime-benchmarks` job on + `codspeed-macro` will pick it up automatically. + +## Open questions + +- Should the Maven example mirror this? The Maven example doesn't currently + run under `codspeed-macro`, so probably no — but worth confirming. +- Do we want to lock the `@Fork` / iteration counts down for the flamegraph + benchmark specifically (a single long iteration produces a cleaner + flamegraph than many short ones)? From 5e8f25388c668ae4f413b8f487a9e05386777ab9 Mon Sep 17 00:00:00 2001 From: not-matthias Date: Wed, 20 May 2026 18:44:34 +0200 Subject: [PATCH 2/2] feat(bench): add macOS-only FibFlamegraphBenchmark Exists solely to exercise CodSpeed's flamegraph symbolization on codspeed-macro runners. Gated to macOS via a Gradle excludes rule so the Linux/Windows matrix jobs skip it. --- examples/example-gradle/build.gradle.kts | 11 +++++++- .../java/bench/FibFlamegraphBenchmark.java | 27 +++++++++++++++++++ 2 files changed, 37 insertions(+), 1 deletion(-) create mode 100644 examples/example-gradle/src/jmh/java/bench/FibFlamegraphBenchmark.java diff --git a/examples/example-gradle/build.gradle.kts b/examples/example-gradle/build.gradle.kts index 756b5f7..25ebc5b 100644 --- a/examples/example-gradle/build.gradle.kts +++ b/examples/example-gradle/build.gradle.kts @@ -1,3 +1,5 @@ +import org.gradle.internal.os.OperatingSystem + plugins { java id("me.champeau.jmh") version "0.7.2" @@ -30,9 +32,16 @@ jmh { // regex passed to JMH, so multiple entries collapse to one pattern with // literal commas and match nothing. Use a single alternation instead. includes.set(listOf( - ".*(SleepBenchmark|BacktrackingBenchmark|FibBenchmark).*", + ".*(SleepBenchmark|BacktrackingBenchmark|FibBenchmark|FibFlamegraphBenchmark).*", )) } + + // FibFlamegraphBenchmark exists only to exercise the macOS flamegraph + // pipeline on codspeed-macro runners. Skip it on Linux/Windows so the + // cross-platform jobs don't run a redundant fib variant. + if (!OperatingSystem.current().isMacOsX) { + excludes.add(".*FibFlamegraphBenchmark.*") + } } sourceSets { diff --git a/examples/example-gradle/src/jmh/java/bench/FibFlamegraphBenchmark.java b/examples/example-gradle/src/jmh/java/bench/FibFlamegraphBenchmark.java new file mode 100644 index 0000000..669483a --- /dev/null +++ b/examples/example-gradle/src/jmh/java/bench/FibFlamegraphBenchmark.java @@ -0,0 +1,27 @@ +package bench; + +import java.util.concurrent.TimeUnit; +import org.openjdk.jmh.annotations.*; + +/** + * macOS-only benchmark whose recursion shape produces an easily-recognisable flamegraph. Gated to + * macOS via Gradle excludes in {@code build.gradle.kts}. + */ +@BenchmarkMode(Mode.AverageTime) +@OutputTimeUnit(TimeUnit.NANOSECONDS) +@State(Scope.Benchmark) +public class FibFlamegraphBenchmark { + + @Param({"35"}) + private int n; + + @Benchmark + public long fib() { + return fib(n); + } + + private static long fib(int n) { + if (n <= 1) return n; + return fib(n - 1) + fib(n - 2); + } +}