Skip to content

feat(runner): zero-trust cost re-metering + crash-safe whole-repo output#39

Open
isPANN wants to merge 1 commit into
mainfrom
feat/zero-trust-cost-and-crash-safe-output
Open

feat(runner): zero-trust cost re-metering + crash-safe whole-repo output#39
isPANN wants to merge 1 commit into
mainfrom
feat/zero-trust-cost-and-crash-safe-output

Conversation

@isPANN

@isPANN isPANN commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator

What

Two related runner improvements: meter cost from token usage × declared price (not a self-reported total), and make whole-repo runs durable so an early-stop/crash keeps the bugs already found.

Changes

Zero-trust cost metering (mirrors the existing zero-trust bug re-verification)

  • Submissions now carry usage_totals (the 4 billed token buckets: input / output / cache_read / cache_write) + the declared prices snapshot.
  • The backend recomputes total_cost_usd and the efficiency metrics from usage_totals × prices and ignores the self-reported total. Legacy submissions without these fields fall back to the reported figure.
  • whole-repo now surfaces its session-level 4-bucket usage (was dropped on the floor).

Whole-repo durability

  • The agent appends each certificate to TRAJECTORY_DIR/certs.txt the moment it finds it (new {{certs_file}} prompt slot); the runner harvests certificates from both the trajectory and that disk log (deduped by rule+source).
  • Fix trajectory loss: output_path was unset and trajectory_dir was never passed, so the --rm'd container dropped the whole trajectory. Both are now wired, and output_path makes mini-swe-agent persist the trajectory after every step (crash-proof). TRAJECTORY_DIR is exposed as a config knob.

Versioned output

  • submission.json stays the stable "latest" pointer (prb submit reads it); every run also writes submission-<model>-<timestamp>.json beside it, so runs don't clobber each other.
  • Envelope self-describes its run: adds usage_totals, prices, agent_mode.

Docs / schema / tests

  • submission.env.example, the run-benchmark skill (field table + SKILL.md), and both JSON schemas updated.
  • Added tests for cost re-metering (a bogus self-reported total is overridden), (de)serialization round-trips, and disk-log cert recovery. 176 passed.

Notes

  • Backwards compatible: all new envelope fields are additive/optional; the budget-cap enforcement path is unchanged.
  • Requires a runner image rebuild to take effect (benchmark/ + config_repo.yaml are baked in).

🤖 Generated with Claude Code

Meter cost from token usage × declared price (not a self-reported total), and
make whole-repo runs durable so an early-stop/crash keeps the found bugs.

- cost: submissions carry usage_totals (4 token buckets) + prices; the backend
  recomputes total_cost_usd/efficiency from them and ignores the self-reported
  figure (mirrors the zero-trust bug re-verification). whole-repo now surfaces
  its session-level 4-bucket usage (was dropped).
- whole-repo durability: the agent appends each certificate to
  TRAJECTORY_DIR/certs.txt the moment it finds it; the trajectory is persisted
  every step (output_path); the runner harvests certs from BOTH the trajectory
  and the disk log (deduped).
- fix trajectory loss: output_path was unset and trajectory_dir was never
  passed, so the --rm'd container dropped everything. Both are now wired, with
  TRAJECTORY_DIR exposed as a config knob.
- versioned output: submission.json stays the stable "latest" pointer; a
  submission-<model>-<timestamp>.json archive is written beside it so runs
  don't clobber each other.
- envelope self-describes: adds usage_totals, prices, agent_mode.
- docs (submission.env.example, skill field table + SKILL.md), schemas, and
  tests updated.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant