docs(benchmark): point the README at results instead of stating them#69
Open
bburda wants to merge 1 commit into
Open
docs(benchmark): point the README at results instead of stating them#69bburda wants to merge 1 commit into
bburda wants to merge 1 commit into
Conversation
…d numbers The example-results, what-to-optimize, and reference-point sections quoted host-specific figures from a seed run that drift out of sync with the committed baseline. Replace them with pointers to where results actually live - the committed baseline (benchmark/baseline/ci.json) and the per-run report.md / summary.json - plus a per-lane description of what each lane reports. Keeps the method and the qualitative findings, drops the absolute numbers that go stale.
There was a problem hiding this comment.
Pull request overview
Updates the benchmark README to avoid embedding host-specific benchmark numbers that drift over time, and instead directs readers to the committed baseline and per-run artifacts that contain current measurements.
Changes:
- Replaces the “Example results” section with “Reading the results”, pointing to
benchmark/baseline/ci.jsonandbenchmark/results/...outputs. - Removes hard-coded magnitudes from “What to optimize” while keeping the actionable optimization targets and lane references.
- Updates “Reference point” to reference recorded host/SHA metadata rather than a fixed host and commit.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+105
to
+107
| - **Per-run output** - `benchmark/results/<timestamp>/<lane>/`: `report.md` (median + | ||
| IQR table), `summary.json` (machine-readable numbers and a one-line verdict), `*.png` | ||
| (the chart), and `run_metadata.json` (host, allocator, gateway version). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The benchmark README's
Example results,What to optimize, andReference pointsections quoted host-specific numbers from an early seed run (USS, threads, scaling table, sweep/heap/load/fault figures, a fixed host + gateway SHA). Those figures drift out of sync with the committed baseline over time - the prose already disagreed withbenchmark/baseline/ci.json.This change makes the README point to where the results live rather than restate them:
Example results->Reading the results: directs the reader to the committed baseline (benchmark/baseline/ci.json) and the per-run output (report.md,summary.json,*.png,run_metadata.json), then describes what each lane reports - without hard-coding absolute numbers. The scaling chart stays as a labelled example.What to optimize: keeps the three optimization targets and their lane pointers; drops the specific magnitudes (re-runload/fault/sweepfor current numbers).Reference point: points to the host/SHA recorded inci.jsonandrun_metadata.jsoninstead of a hard-coded host and SHA.Net: the method, lane descriptions, and qualitative findings stay; the per-host numbers that go stale are gone.
Checklist
Example resultsanchor