Skip to content

feat(audit): cost-trace truthfulness audit script (Theme 5 Layer 5)#325

Draft
mitwilli-create wants to merge 1 commit into
mainfrom
feat/theme5-l5-cost-trace-audit-2026-05-29
Draft

feat(audit): cost-trace truthfulness audit script (Theme 5 Layer 5)#325
mitwilli-create wants to merge 1 commit into
mainfrom
feat/theme5-l5-cost-trace-audit-2026-05-29

Conversation

@mitwilli-create
Copy link
Copy Markdown
Owner

Summary

Theme 5 Layer 5 from the 2026-05-29 task-audit. Standalone script that statistically audits the polish-cost-trace JSONL ledgers for plausibility, sum coherence, and outlier detection. Closes the bug-class catalog entry cost-tracker-implausible-entry-poisoning-pre-flight-check by giving Mitchell a recurring audit lens, not just a write-time check.

Smoke validation

$ node scripts/audit-cost-trace.mjs --dry-run
[load] 897 entries from 7 files
[D1] per-day totals: INFO (grand=$125.6)
[D2] cost bands: PASS (outlier_pct=0)
[D3] outliers: PASS (count=0)
[D4] model plausibility: PASS (flagged=0)
[D5] suspicious file: INFO

Healthy state across the board: no >$5 single-call outliers, no model-cost mismatches (Haiku >$0.50 / Sonnet >$2 / Opus >$5), no entries in data/cost-trace-suspicious.jsonl. Total tracked spend across 7 trace files is $125.60.

Dimensions

# Dimension Verdict logic
D1 Per-day totals — last 7 days INFO (informational)
D2 Cost bands — % calls in [0, $0.50] / [$0.50, $2] / [$2, $5] / >$5 PASS if 0 outliers
D3 Outliers (>$5) — top 20 entries flagged for review PASS if 0 outliers
D4 Model-cost plausibility — Haiku >$0.50, Sonnet >$2, Opus >$5 flagged PASS if 0 flagged
D5 Suspicious file — count from data/cost-trace-suspicious.jsonl PASS if empty

Tunable model bands

MODEL_BANDS constant in the script. Adjust if a vendor changes pricing:

const MODEL_BANDS = {
  'haiku': 0.50, 'flash': 0.50, 'sonnet': 2.00, 'opus': 5.00,
  'gpt-5': 3.00, 'gemini': 1.00, 'grok': 1.00, 'sonar': 0.50,
};

Test plan

  • node --check passes
  • --dry-run produces audit report in .claude/audit/<DATE>/cost-trace-audit-<DATE>.md
  • Default mode invokes Sonnet 4.6 synthesis (~$0.05-0.15)
  • --council triggers 7-model adjudication
  • Output is gitignored

🤖 Generated with Claude Code

Closes Theme 5 Layer 5 from the 2026-05-29 task-audit. New standalone
script that statistically audits the polish-cost-trace JSONL ledgers
for plausibility, sum coherence, and outlier detection.

scripts/audit-cost-trace.mjs (~210 LOC)
  Reads:
    data/polish-cost-trace-*.json (one per day, append-only JSONL)
    data/cost-trace-suspicious.jsonl (if present)

  Dimensions:
    D1 per-day totals       — sums per cost-trace file (last 7 days)
    D2 cost bands           — % calls in [0, $0.50] / [$0.50, $2] / [$2, $5] / >$5
    D3 outliers (>$5)       — top 20 entries flagged for review
    D4 model-cost plausibility — Haiku >$0.50 / Sonnet >$2 / Opus >$5 flagged
    D5 suspicious file      — count + summary of already-flagged entries

Smoke validation:
  --dry-run scanned 897 entries from 7 files. Grand total $125.60.
  Zero outliers, zero model-band violations, zero pre-flagged entries.
  All dimensions PASS or INFO. Healthy cost-trace state confirmed.

Cross-fork-leak hardening:
  Output frontmatter: [PERSONAL — DO NOT PUBLISH]
  assertNoLeak() guard at render time.

Closes the bug-class entry:
  "cost-tracker-implausible-entry-poisoning-pre-flight-check"
  in ~/.claude/knowledge/brain/bug-class-catalog.md by giving Mitchell
  a recurring audit lens, not just a write-time check.

Defaults: ~$0.05-0.15 per run (Sonnet 4.6 narrative synthesis).
--council adds 7-model adjudication for ~$20-40.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant