Skip to content

Add resume benchmark harness and record baseline#31

Merged
hacktivist123 merged 1 commit into
hacktivist123:mainfrom
uncoooloj:tools/benchmark-baseline-harness
Jun 19, 2026
Merged

Add resume benchmark harness and record baseline#31
hacktivist123 merged 1 commit into
hacktivist123:mainfrom
uncoooloj:tools/benchmark-baseline-harness

Conversation

@uncoooloj

Copy link
Copy Markdown
Contributor

docs/Benchmarking.md defined standards but no numbers were ever recorded, so token-efficiency claims were unmeasurable.

  • New scripts/benchmark-resume.py (stdlib-only, self-contained): measures bytes-that-enter-context per fixture under three strategies — raw full read, message-only projection (bounded tool summaries), keyword/evidence digest. Markdown table + --json output, tokens approximated as bytes/4.
  • docs/Benchmarking.md gains 'Baseline measurements (2026-06-10)': on current fixtures, projection is 68.9% of raw and digest 30.9% overall; on noisy JSONL fixtures projection alone is 25–26% and the wrong-newest codex fixture drops to 12.6%. Fixtures are ~0.5–1.7KB while real transcripts run 1–70MB, so these ratios understate real-world reduction.

🤖 Generated with Claude Code

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@hacktivist123

Copy link
Copy Markdown
Owner

Nice baseline — really helpful to have real numbers on record, and the honesty about tiny fixtures vs real 1–70MB transcripts is appreciated.

One thing worth tracking: benchmark-resume.py re-implements the projection/digest logic inline rather than importing from session-digest.py. That's a reasonable choice for a self-contained measurement harness, but the two implementations can drift over time — if the live digest strategy changes and the benchmark doesn't, the recorded ratios will stop reflecting real behavior.

Suggestion: either add a one-line note in the docstring / Benchmarking.md that the inline projection is a measurement copy and must stay in sync with session-digest.py, or add a small parity check that runs both on a fixture and asserts they agree within a tolerance. Not blocking — just flagging for future maintainers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants