Skip to content

fix(miner): wire real dollar-cost tracking into the loop's budgetSpent#5356

Merged
JSONbored merged 1 commit into
mainfrom
feat/miner-cost-tracking-5135
Jul 12, 2026
Merged

fix(miner): wire real dollar-cost tracking into the loop's budgetSpent#5356
JSONbored merged 1 commit into
mainfrom
feat/miner-cost-tracking-5135

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

Summary

Follow-up to #5303/#5320: loop-cli.js's GovernorCapUsage.budgetSpent was never incremented (only turnsTaken/elapsedMs used real signals), so a capLimits.budget dollar-cost dimension could never trip from the loop's own spend.

Investigation found a real, already-flowing cost signal was being silently dropped: the Agent SDK's own result message declares total_cost_usd: number unconditionally (both SDKResultSuccess and SDKResultError carry it — a session is billed whether or not it succeeds), but agent-sdk-driver.ts only ever read num_turns.

Changes

  • CodingAgentDriverResult gains an optional costUsd field.
  • agent-sdk-driver.ts extracts the SDK's real total_cost_usd whenever a result message arrived, on both the success and the "did not complete successfully" paths (never fabricated on a thrown/no-result stream, where no billing signal exists at all).
  • iterate-loop.ts sums it into a new totalCostUsd on IterateLoopResult, mirroring the existing totalTurnsUsed accumulator exactly (same 4 return points).
  • attempt-cli.js surfaces it through options.onResult alongside totalTurnsUsed/iterationsUsed.
  • loop-cli.js increments usage.budgetSpent with the real value before calling governor-state.js's saveCapUsage.

Documented, deliberate gap

The CLI-subprocess providers (claude-cli/codex-cli) report no cost signal today — their production argv never passes --output-format json, so there's nothing to parse cost from without a separate, larger change to how those providers invoke the real CLI binaries (a real scope decision, not a simple additive fix — I did not make that change here). budgetSpent is real for agent-sdk and an honest 0 (not fabricated) for the CLI-subprocess providers; a capLimits.budget dimension only ever meaningfully trips against agent-sdk spend today.

Test plan

  • New regression: okDriverResult's real costUsd flows through runMinerAttemptresult.loopResult.totalCostUsd
  • New regression: a mid-loop abandon (real iteration ran and was billed before the ceiling forced abandon) still reports real totalTurnsUsed/totalCostUsd
  • New regression: loop-cli.js's budgetSpent reflects the real totalCostUsd from a submitted attempt
  • New regression: a provider reporting no cost signal (mirrors claude-cli/codex-cli) leaves budgetSpent honestly at 0 across multiple real attempts
  • coding-agent-driver-parity.test.ts: documented-divergence tests locking in "CLI driver never reports cost" vs. "Agent-SDK driver reports the real total_cost_usd"
  • npx vitest run test/unit — 735 files / 14,582 tests pass
  • npm run test:coverage — verified patch coverage of every new/touched line via lcov (including the mid-loop abandon path, previously untested at the root level)
  • npx tsc --noEmit -p . --incremental false (fresh, no cache) — clean
  • npm run test:engine-parity, npm run test:driver-parity, npm run test --workspace @jsonbored/gittensory-engine (520 node:test cases) — all pass
  • npm run --prefix packages/gittensory-miner build — pass
  • git diff --check — clean

Advances #5135

loop-cli.js's GovernorCapUsage.budgetSpent was never incremented --
only turnsTaken/elapsedMs used real signals -- so a capLimits.budget
dollar-cost dimension could never trip from the loop's own spend.
Investigation found a real, already-flowing cost signal was being
silently dropped: the Agent SDK's own result message declares
`total_cost_usd: number` unconditionally (SDKResultSuccess and
SDKResultError both carry it -- a session is billed whether or not it
succeeds), but agent-sdk-driver.ts only ever read `num_turns`.

Wires it through the whole real pipeline:
- CodingAgentDriverResult gains an optional costUsd field.
- agent-sdk-driver.ts extracts the SDK's real total_cost_usd whenever
  a result message arrived, on both the success and the "did not
  complete successfully" paths (never fabricated on a thrown/no-result
  stream, where no billing signal exists).
- iterate-loop.ts sums it into a new totalCostUsd on IterateLoopResult,
  mirroring the existing totalTurnsUsed accumulator exactly.
- attempt-cli.js surfaces it through options.onResult alongside
  totalTurnsUsed/iterationsUsed.
- loop-cli.js increments usage.budgetSpent with the real value before
  calling governor-state.js's saveCapUsage.

Documented, deliberate gap: the CLI-subprocess providers (claude-cli/
codex-cli) report no cost signal today -- their production argv never
passes --output-format json, so there is nothing to parse cost from
without a separate, larger change to how those providers invoke the
real CLI binaries. budgetSpent is real for agent-sdk and an honest 0
(not fabricated) for the CLI-subprocess providers; a capLimits.budget
dimension only ever meaningfully trips against agent-sdk spend.
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
gittensory-ui 7c4221c Commit Preview URL

Branch Preview URL
Jul 12 2026, 02:51 PM

@codecov

codecov Bot commented Jul 12, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 66.66667% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 94.42%. Comparing base (181cc0f) to head (7c4221c).
⚠️ Report is 5 commits behind head on main.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
...ckages/gittensory-engine/src/miner/iterate-loop.ts 60.00% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #5356      +/-   ##
==========================================
+ Coverage   94.40%   94.42%   +0.01%     
==========================================
  Files         550      550              
  Lines       44117    44120       +3     
  Branches    14631    14633       +2     
==========================================
+ Hits        41650    41660      +10     
+ Misses       1792     1785       -7     
  Partials      675      675              
Flag Coverage Δ
shard-1 44.00% <0.00%> (-0.01%) ⬇️
shard-2 34.66% <66.66%> (+0.02%) ⬆️
shard-3 31.60% <0.00%> (-0.01%) ⬇️
shard-4 31.41% <0.00%> (-0.01%) ⬇️
shard-5 33.15% <0.00%> (-0.01%) ⬇️
shard-6 43.35% <0.00%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
...es/gittensory-engine/src/miner/agent-sdk-driver.ts 100.00% <100.00%> (ø)
...gittensory-engine/src/miner/coding-agent-driver.ts 100.00% <ø> (ø)
packages/gittensory-miner/lib/attempt-cli.js 96.55% <ø> (ø)
packages/gittensory-miner/lib/loop-cli.js 93.23% <ø> (ø)
...ckages/gittensory-engine/src/miner/iterate-loop.ts 64.28% <60.00%> (+10.58%) ⬆️

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@gittensory-orb gittensory-orb Bot added the gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. label Jul 12, 2026
@gittensory-orb

gittensory-orb Bot commented Jul 12, 2026

Copy link
Copy Markdown

Caution

🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥

🛑 Gittensory review result - fixes required

Review updated: 2026-07-12 15:24:12 UTC

10 files · 1 AI reviewer · 2 blockers · readiness 93/100 · CI failing · unstable

🛑 Suggested Action - Manual Review

  • No linked issue detected — If this PR is intended to solve an issue, link it explicitly in the PR body.
  • Maintainer requires a linked issue — Link the relevant issue (for example Closes #123) before opening the PR.

Review summary
This diff wires the Agent SDK's real total_cost_usd through CodingAgentDriverResult -> IterateLoopResult.totalCostUsd -> attempt-cli's onResult -> loop-cli's usage.budgetSpent accumulator, mirroring the existing totalTurnsUsed pattern exactly at all 4 iterate-loop return points. The change is correct and honest about scope: costUsd is genuinely absent (undefined, not fabricated) for CLI-subprocess providers and for the thrown/no-result driver path, and present on both success and failure result-message branches per the SDK's actual type contract. Tests cover the accumulation across handoff/abandon paths and lock in the CLI-driver-reports-no-cost divergence.

Nits — 5 non-blocking
  • packages/gittensory-miner/lib/loop-cli.js:353 — verify governor-state.js's saveCapUsage/capLimits.budget comparison logic actually treats a fractional budgetSpent correctly (e.g. no integer/rounding assumption), since this is the first place a non-integer usage field flows through that path.
  • codecov/patch flags 66.66% patch coverage — confirm the uncovered lines aren't a real gap in loop-cli.js's new budgetSpent line or agent-sdk-driver.ts's costUsd branches, since only 2 of the ~7 changed source files have visible new tests in this diff excerpt.
  • Add a direct unit test for agent-sdk-driver.ts asserting costUsd is read from total_cost_usd on the `is_error: true` / non-success subtype path too (currently only the success-subtype SDK test at test/contract/coding-agent-driver-parity.test.ts:242 exercises this field).
  • nit: `packages/gittensory-engine/src/miner/agent-sdk-driver.ts:174` now promises cost is preserved on the non-success result path, but the visible tests only assert the success result frame; add a regression where `subtype !== "success"` or `is_error === true` still returns `costUsd`.
  • nit: `packages/gittensory-engine/src/miner/iterate-loop.ts:277` sums raw provider numbers without rejecting `NaN`, `Infinity`, or negative values; if the SDK contract is the reason this is safe, leave that explicit or clamp to finite non-negative values before adding.

Why this is blocked

  • No linked issue detected — If this PR is intended to solve an issue, link it explicitly in the PR body.
  • Maintainer requires a linked issue — Link the relevant issue (for example Closes #123) before opening the PR.

CI checks failing

  • codecov/patch — 66.66% of diff hit (target 99.00%)
Signal Result Evidence
Code review ❌ 2 blockers 1 reviewer
Linked issue ⚠️ Missing No linked issue or no-issue rationale found.
Related work ✅ No active overlap found No same-issue or scoped active PR overlap found.
Change scope ✅ 20/20 Low review scope from cached public metadata (no linked issue context).
Validation posture ✅ 25/25 PR body includes validation/test evidence.
Contributor workload ✅ 10/10 Author activity: 44 registered-repo PR(s), 36 merged, 476 issue(s).
Contributor context ✅ Confirmed Gittensor contributor JSONbored; Gittensor profile; 44 PR(s), 476 issue(s).
Gate result ❌ Blocking Repo-configured hard blocker found.
Improvement ✅ Minor risk: clean · value: minor — Code changes are accompanied by test evidence.
Review context
  • Author: JSONbored
  • Role context: owner (maintainer lane)
  • Public audience mode: oss maintainer
  • Lane context: Repository is configured for direct PR review.
  • Public profile languages: not available
  • Official Gittensor activity: 44 PR(s), 476 issue(s).
  • PR-specific overlap: none found.
Contributor next steps
  • Treat this as maintainer-lane context rather than normal contributor-lane activity.
  • Explain no-issue PR.
  • Link the issue being solved, or explicitly explain why this is a no-issue PR.
Signal definitions
  • Related work = same linked issue, overlapping active PRs, or title/path similarity.
  • Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
  • Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
  • Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
  • Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.
[BETA] Chat with Gittensory

Ask Gittensory a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.

  • @gittensory ask &lt;question&gt; answers contribution-quality Q&A with source citations and freshness.
  • @gittensory chat &lt;question&gt; answers in natural prose from cached decision-pack facts via local inference (maintainer/collaborator; read-only).
  • A plain-language @gittensory mention with a real question is routed to the closest matching read-only command automatically -- no exact syntax required.

Full command reference: https://gittensory.aethereal.dev/docs/gittensory-commands

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed


💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by Gittensory, a quiet PR intelligence layer for OSS maintainers.

  • Re-run Gittensory review

@gittensory-orb gittensory-orb Bot added the manual-review Gittensor contributor context label Jul 12, 2026
@JSONbored JSONbored merged commit 7935bb4 into main Jul 12, 2026
18 of 19 checks passed
@JSONbored JSONbored deleted the feat/miner-cost-tracking-5135 branch July 12, 2026 15:35
JSONbored added a commit that referenced this pull request Jul 12, 2026
Advances #5135

defaultCliSubprocessArgs built ["--max-turns", N, "--acceptance-criteria",
path, instructions] -- neither --max-turns nor --acceptance-criteria is
a real flag on the claude or codex CLI (verified against `claude --help`
and `codex exec --help`, plus a live invocation), and the argv never
included -p/--print (claude) or the exec subcommand (codex) at all.
Without those, a spawned claude/codex process starts an INTERACTIVE
session against a subprocess whose stdin is closed -- meaning the
claude-cli/codex-cli providers had no real non-interactive argv shape
in production.

Splits the shared builder into defaultClaudeCliArgs (--print
--output-format json --permission-mode acceptEdits) and
defaultCodexCliArgs (exec --json --sandbox workspace-write), matching
each CLI's own real, verified interface. --output-format json / --json
also finally activate claudeErrorStatus/codexErrorFromStdout's existing
JSON(L) parsing, which was already written for exactly this shape but
never actually triggered since the flag was never passed.

Surfaces two more real, previously-latent gaps this exposed:
- claude's own documented behavior ("--output-format json sometimes
  exits non-zero") implies it can also exit 0 while the JSON envelope
  reports is_error: true -- confirmed empirically via a live invocation.
  claudeErrorStatus was already written to detect this exact shape but
  was only ever checked on the code!==0 branch; now also checked on
  success exit codes.
- Adds real cost parsing (ported from src/selfhost/ai.ts's own
  COST_KEYS/extractCliUsage best-effort pattern) so claude-cli/codex-cli
  now report costUsd from their own real JSON(L) output too, closing
  the "CLI providers report no cost" gap #5356 documented.

driver-factory.ts's buildCliArgsWithConfiguredModel is now command-aware:
codex's own -m/--model flag is scoped to the exec subcommand (per codex
exec --help) and must land after "exec", not prefixed before everything
the way claude's top-level --model flag can be -- a single shared
prefix-everything scheme would have silently misparsed for codex.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. manual-review Gittensor contributor context

Projects

Development

Successfully merging this pull request may close these issues.

1 participant