progress.txt

# Stage 7 — Ralph progress log (issue litentry/agentKeys#64)

Started: 2026-05-05
Plan: docs/spec/plans/issue-64/PLAN.md (mirror of ~/.claude/plans/now-i-just-merged-idempotent-plum.md)
Reviewer: codex (per --critic=codex)
Branch: claude/dazzling-mirzakhani-2a06bc (independent of sibling claude/quizzical-ellis-d6f1e9)

## Session 1 — 2026-05-05 — Phase 0 foundation (6 of 16 stories)

### Context

Issue #64 wants a pluggable broker (auth + wallet + audit layers) production-ready
on testnet. Pre-PR: PR #61 (OIDC issuer + AWS-cred wiring) just merged to main.
Sibling branch `claude/quizzical-ellis-d6f1e9` carries 6 codex rounds of prior work
on the same idea — used as REFERENCE for which artifacts (Solidity contract,
schema, breaker design) are worth harvesting, but starting structure fresh under
the user's 11 process rules.

Reviewer pass before implementation: 4 parallel reviews (CEO/eng/design/codex)
landed actionable findings. Plan refined with §3.5 grounded in dexs-backend
reference (port-vs-greenfield analysis): SIWE wrapping EIP-191, per-call daemon
signatures on mint, single ES256 issuer with purpose tagging, fragment-token
email-link, OAuth2 with id_token+PKCE+state-CSRF, capability grants as
first-class data, master-gated recovery, gas-drain mitigations, tiered
refuse-to-boot — all listed in DECISIONS.md.

### VCS exception (D5)

This is a git worktree, not a jj workspace. jj's working copy is the main
repo at /Users/agent-jojo/Projects/agentKeys/ — it cannot see edits inside
the worktree. Pragmatic exception: use `git` for commits inside the worktree.

### Story log — Phase 0 — COMPLETED

#### US-001 — src/env.rs centralized env-var module — PASSED 2026-05-05 (commit 32d3dd3)
Files: crates/agentkeys-broker-server/src/env.rs (new, 51 const + Group enum + all() registry + print_table()),
       src/lib.rs (mod env), src/config.rs (refactor — no raw BROKER_* literals remain).
Plan home created: docs/spec/plans/issue-64/{PLAN.md, DECISIONS.md, AMBIGUITIES.md, V0.1-FOLLOWUPS.md, prd.json}.
Tests: 5/5 (env::tests::*).
Acceptance: ✓ all 5 criteria met. grep returns zero hits in src/config.rs.
Learning: Group enum exhaustive match in tests forces compile-time update if a variant is added.

#### US-002 — Plugin trait scaffolding — PASSED 2026-05-05 (commit d6e5bba)
Files: crates/agentkeys-broker-server/src/plugins/{mod.rs, auth.rs, wallet.rs, audit.rs} (new),
       src/lib.rs (mod), Cargo.toml (feature gates).
Cargo features: default = [auth-wallet-sig, wallet-keystore, audit-sqlite] + opt-in
                auth-email-link, auth-oauth2-google, audit-evm + v1+ stubs.
Tests: 8/8 (plugins::tests::*, plugins::auth::tests::*, plugins::wallet::tests::*, plugins::audit::tests::*).
Acceptance: ✓ all 8 criteria met.
Learning: Per-trait error enums use thiserror with explicit variants matching plan §6 / §Phase C —
          Storage / Network / CircuitOpen / BudgetExceeded / VerificationMismatch / NotFound / Internal.

#### US-004 + US-008 (bundled) — OmniAccount + SqliteAnchor port — PASSED 2026-05-05 (commit 80c01f6)
Files: src/identity/{mod.rs, omni_account.rs} (new),
       src/plugins/audit/{mod.rs ⟵ ex audit.rs, sqlite.rs} (restructure + new),
       agentkeys-types::AgentIdentity::OAuth2{provider,sub} variant added,
       4 cross-crate match-arm updates.
Tests: 9 (identity::tests::*) + 8 (plugins::audit::tests::*) — all pass.
Plan §3.5 grounding: AGENTKEYS_CLIENT_ID = "agentkeys" pinned; distinct from dexs-backend's "wildmeta".
Acceptance: ✓ all criteria for both stories met.
Learning: Adding AgentIdentity::OAuth2 cascades match-arm errors to 5 sites — borrow checker doing its
          job. Module-conflict E0761: had `plugins/audit.rs` AND `plugins/audit/mod.rs` simultaneously
          after writing the sqlite submodule. Fix: merged trait content into mod.rs, deleted standalone.
          Same pattern recurs for `plugins/wallet/` in US-007.

#### US-005 — Dual ES256 keypairs with purpose tagging — PASSED 2026-05-05 (commit 130f684)
Files: src/jwt/{mod.rs, session.rs, issue.rs, verify.rs} (new),
       src/oidc.rs (purpose field + pub(crate) helpers),
       src/lib.rs (mod jwt).
Tests: 10/10 (jwt::session::tests::*, jwt::issue::tests::*, jwt::verify::tests::*).
Closes Codex P0 #7 (footgun): on-disk JSON carries `"purpose"` field; load() refuses purpose mismatch.
Backwards-compat: legacy OIDC keypair files (no `purpose` field) load as `Oidc` via #[serde(default)].
                  SessionKeypair::load is strict — no migration window.
Learning: assertion-style mismatch — used err.to_string().contains("oidc") which fails because the
          error formats with Debug-cased "Oidc". Fix: lowercase the haystack before contains.

#### US-007 — ClientSideKeystoreProvisioner + WalletStore — PASSED 2026-05-05 (commit 61a737b)
Files: src/storage/{mod.rs, wallets.rs} (new),
       src/plugins/wallet/{mod.rs ⟵ ex wallet.rs, keystore.rs} (restructure + new),
       src/lib.rs (mod storage).
Tests: 9/9 (3 type tests + 6 keystore behavior tests).
Acceptance: ✓ all criteria met.
Plan §3.5 grounding: MetaMask model — broker stores only (omni, addr, role, parent_addr, created_at).
                     Composite PK on (omni_account, address) lets a user have multiple wallets.
Learning: bind() must detect both role mismatch AND parent mismatch on re-bind. A daemon silently
          switching masters under the same (omni, address) would be data corruption otherwise.

### Story log — Phase 0 — REMAINING (10 of 16)

In priority order:
- US-003: tiered refuse-to-boot in src/boot.rs + main.rs wiring
- US-006: WalletSig SIWE plugin (k256 ecrecover + sha3, single-use nonce table) + auth_nonces storage
- US-009: POST /v1/auth/wallet/{start, verify} endpoints
- US-010: POST /v1/auth/exchange backward-compat shim
- US-011: /v1/mint-aws-creds upgraded — session JWT verify + per-call daemon signature + audit gate
- US-012: src/handlers/broker_status.rs operational /readyz aggregating PluginRegistry
- US-013: tests/invariant_load_bearing.rs — all 6 cases (a-f) per plan §2
- US-014: harness/stage-7-phase0-smoke.sh + harness/stage-7-done.sh skeleton
- US-015: docs/operator-runbook-stage7.md draft (env table auto-generated from env.rs)
- US-016: Phase 0 codex review round 1 (must close P0/P1; P2 stop rule)

### Architectural decisions made during this session

(All flow into DECISIONS.md.)

- The trait shapes from US-002 are pinned. Subsequent stories implement against them.
- `IdentityType::canonical()` strings pinned ("evm", "email", "oauth2_google", etc.) — feed
  OmniAccount derivation; renaming any is a breaking change.
- `AGENTKEYS_CLIENT_ID = "agentkeys"` pinned in identity/omni_account.rs — same reason.
- ES256 keypair on-disk format includes `"purpose"`. Default for legacy OIDC files is `purpose=oidc`
  (backwards-compat). Session keypair load is strict.
- WalletStore composite PK is (omni_account, address). Re-bind is idempotent on identical role+parent;
  mismatch is rejected.
- Audit log v2 schema is `plugin_mint_log` (new table); legacy `mint_log` (existing src/audit.rs::AuditLog)
  preserved until US-011 migrates the mint handler.

### Build + test totals across the session

cargo build -p agentkeys-broker-server: green at every commit point.
cargo test -p agentkeys-broker-server: ~51 broker-server tests passing as of `61a737b`.
Cross-crate: agentkeys-types + agentkeys-core + agentkeys-cli + agentkeys-mock-server all build
with the AgentIdentity::OAuth2 variant added.
Workspace build: green.

## Handoff to next ralph iteration

Pick up from US-006 (WalletSig SIWE) — it's the highest-priority remaining because US-009 + US-011
both depend on it. US-003 (boot.rs) can start in parallel.

Next-iteration suggested commit order:
1. US-006 WalletSig SIWE (~700 LOC + tests; needs k256 + sha3 deps under auth-wallet-sig feature)
2. US-003 boot.rs + main.rs wiring
3. US-009 + US-010 + US-011 endpoints
4. US-012 broker_status
5. US-013 invariant test
6. US-014 smoke + done.sh
7. US-015 runbook
8. US-016 codex round 1

## Session 2 — 2026-05-05 — Phase 0 close-out (15 of 16 stories)

Resumed from Session 1 pause. The session knocked off the remaining
stories serially: US-011 mint upgrade → US-013 invariant test →
US-016 codex review.

#### US-011 — /v1/mint-aws-creds upgrade — PASSED 2026-05-05 (commit 1edb4f6)
Files: src/handlers/mint.rs (rewritten), tests/mint_v2_flow.rs (new).
Tests: 10 unit + 5 v2 integration + 9 legacy integration; ALL pass.
Plan §3.5.2 + §2 grounding: session JWT bearer + per-call daemon signature over canonical-JSON-bytes-minus-auth.signature, EIP-191 envelope, ecrecover-must-match-auth.address. AuditAnchor write loop short-circuits on first failure → response 500, no creds, audit-anchored=None. Wallet-binding gate ensures auth.address == claims.agentkeys.wallet_address.
Backwards compat: looks_like_session_jwt heuristic (eyJ + 3 segments) routes to v2; everything else falls through to mint_legacy verbatim. Codex P0 #14 (permanent dual-accept) mitigated by documented v0→v1 cutover.
Learning: STS call happens BEFORE audit anchor write per plan §2.e (speculative latency optimization). The gate is the response — credentials never appear in the response body unless every audit anchor confirmed durability.

#### US-013 — tests/invariant_load_bearing.rs — PASSED 2026-05-05 (commit 8657d74)
Files: tests/invariant_load_bearing.rs (new, 574 LOC).
Tests: 7/7 (6 cases a-f + 1 helper-compile). All pass.
Plan §2 + rule 7 (day-1 contract). Single test file exercising every failure mode of the load-bearing invariant. Test fixtures: FailingAuditAnchor (always returns AuditError::Storage; ready()=Ready so /readyz pre-check doesn't pre-fail), CountingStsClient (Arc<AtomicUsize> tracks assume_role calls so cases (b)-(d) can assert "STS NEVER called"). AuditTopology enum drives registry composition per test.
Phase 0 simplifications documented in test comments:
- Case (d) missing-grant: Phase B introduces real grants; Phase 0 stand-in is forged-JWT-rejected-at-verify.
- Case (f) dual-anchor partial-failure: Phase 0 only asserts short-circuit + no-creds; full quarantine state machine ships in Phase C alongside EvmTestnetAnchor.

#### US-016 — Phase 0 codex review round 1 — IN FLIGHT
Subagent: codex-rescue dispatched 2026-05-05 with 15 attack vectors covering mint dispatch, audit gate, nonce TOCTOU, keypair purpose tagging, plugin registry empties, Tier-2 backoff, /readyz JSON shape, JWT-shape heuristic false-positives, JSON vs CBOR canonicalization, per-call sig endpoint binding, OmniAccount hash boundary, test coverage of mint_v2 branches, refuse-to-boot completeness, dead code in handlers::health, AppState dual-audit transition. Findings + verdict will land in docs/spec/plans/issue-64/codex-round1.md when the review completes.

### Session 2 totals

cargo test -p agentkeys-broker-server: ~115 tests passing (79 lib unit + 9 mint_flow + 6 oidc_flow + 4 auth_wallet_flow + 5 mint_v2_flow + 7 invariant_load_bearing + 4 boot + 1 healthz handler reused). Workspace build green at every commit. clippy clean.

15 of 16 Phase 0 stories committed; US-016 in flight via subagent.

## Session 3 — 2026-05-05 — Phase 0 close-out + Phase A.1 + Phase C.0

Resumed from Session 2 pause. Closed Phase 0 (US-016 codex rounds
1+2 in `772ef7e`), shipped the operator checkpoint (`2f83749`), and
moved through Phase A.1 + Phase C.0 in a single session.

### Phase 0 close-out
- US-016 codex rounds 1+2 — both rounds find only P2/P3, plan rule 9
  stop rule fires; 20 findings rolled to V0.1-FOLLOWUPS.md.
- PHASE-0-CHECKPOINT.md ships with full demo recipe (build, keygen,
  boot, exercise SIWE, mint v2, verify audit row).

### Phase A.1 — EmailLink magic-link auth method (3/3 stories SHIPPED)
- US-017 (`9a1e0d4`): EmailLink plugin + storage. EmailSender trait
  abstraction with StubEmailSender for tests; real SES wiring deferred
  to Phase E US-039. 27 new tests (12 plugin + 9 storage tokens + 6
  rate limits).
- US-018 (committed via prd.json passes flag): 4 HTTP endpoints
  (request/verify/status/landing), boot.rs construction with HMAC
  key + rate limit env vars, AppState extension with concrete
  Arc<EmailLinkAuth> handle for browser-side handlers, 7 integration
  tests in tests/email_flow.rs covering full request → click → poll
  flow + GET-on-verify-returns-405 prefetch defense + replay
  rejection + landing-page security headers.
- US-019: Phase A.1 smoke (9 invariants) + codex rounds 1+2. Round 1
  finds 4 P2 + 5 P3; round 2 finds 2 P2 + 5 P3; both rounds satisfy
  the same-severity stop rule. 16 Phase A.1 P2/P3 items rolled to
  V0.1-FOLLOWUPS.md.

### Phase C.0 — Graceful shutdown + migrations (2/2 stories)
- US-023: graceful_shutdown integration test landed. Phase 0's
  main.rs already wired SIGTERM → grace-drain → exit; US-023
  promotes that to a tested invariant — handler_completes_when_shutdown
  + server_exits_after_grace_period.
- US-024: migrations/0001_v2_schema.sql is the canonical reference
  for the v2 schema. Each store module's init_schema() runs the
  equivalent CREATE TABLE IF NOT EXISTS at boot; the SQL file is
  the single-source-of-truth review surface AND the future input
  for a real migration runner (deferred to Phase E US-039).

### Session 3 totals

cargo test -p agentkeys-broker-server (default features): 116 tests
cargo test -p agentkeys-broker-server (--features auth-email-link):
  152 tests (+ 2 graceful_shutdown integration)

Phase 0 + Phase A.1 + Phase C.0 SHIPPED. Remaining: Phase A.2 (OAuth2),
Phase B (capability grants + recovery), Phase C (EVM Base Sepolia
anchor — large), Phase D-rest (metrics + idempotency), Phase E
(runbook final + done.sh final).

The next ralph iteration picks up at Phase A.2 US-020 (OAuth2 trait +
Google plugin). The V0.1-FOLLOWUPS list (now 36 entries: 20 from
Phase 0 + 16 from Phase A.1) is the priority-zero backlog before
any new Phase A.2 deliverables.

## Session 4 — 2026-05-05 — Phase A.2 + B + C structural + D-rest + E (FINAL ship)

Resumed from Session 3 close. The session shipped FIVE remaining
phases of issue#64 — Phase A.2, Phase B, Phase C structural, Phase
D-rest, and Phase E (the runbook + done.sh finalization + V0.1
followups closeout). All 41 PRD stories now `passes: true`.

### Phase A.2 — OAuth2 / Google (3 stories)
- US-020: OAuth2Provider trait + GoogleOAuth2Provider with PKCE +
  state HMAC + JWKS cache (1h TTL) + id_token verify.
- US-021: 3 HTTP endpoints (start/callback/status) + boot wiring +
  AppState extension. Browser-side callback uses minimal HTML +
  Cache-Control: no-store + Referrer-Policy: no-referrer + nosniff;
  session JWT NEVER lands in the browser response.
- US-022: smoke (9 invariants) + runbook §oauth2-setup expanded with
  Google Cloud Console + state HMAC key generation + failure-mode
  table + multi-account quirk explanation.

### Phase A.2 — codex review THREE rounds
- Round 1: 0 P0, 1 P1, 2 P2, 3 P3. P1 + Vector-10 P2 + Vector-13 P3
  + Vector-14 P3 closed.
- Round 2: 1 P1 (on Phase B preview try_consume) + 1 new P2 (jwk_matches
  fail-closed). Both fixed.
- Round 3: 1 P2 + 2 P3, all non-blocking. Vector 4 P2 (grant errors
  401→403) closed via new BrokerError::Forbidden variant. Round 3
  VERDICT: PASS — Phase A.2 + Phase B grants ship per stop rule.

### Phase B — Capability grants + recovery (5 stories)
- US-025: src/storage/grants.rs with ATOMIC try_consume (single SQL
  UPDATE … WHERE … RETURNING — Codex round-2 V5 P1 mitigation).
- US-026: 3 endpoints — POST /v1/grant/{create,revoke,list}. master
  session JWT required. audit_proof = ES256 JWT minted via
  mint_grant_audit_proof.
- US-027: mint_v2 calls try_consume before STS. NoGrant → legacy
  fallback (Phase E flips to fail-closed). Revoked/Expired/Exhausted
  → 403.
- US-028: src/storage/identity_links.rs + 3 wallet endpoints
  (POST /v1/wallet/link, GET /v1/wallet/links, POST
  /v1/wallet/recover/lookup). Recovery is master-gated — no
  email-only takeover (Codex P0 #4 mitigation).
- US-029: Phase B smoke (14 invariants).

### Phase C structural — EVM Base Sepolia anchor (6 stories)
- US-030: solidity/src/AgentKeysAudit.sol contract with indexed
  recordHash + omniAccount + wallet event topics. Foundry build/deploy
  is operator-managed via runbook §evm-deploy.
- US-031: src/plugins/audit/evm.rs — EvmAuditConfig (validate +
  static checks for Tier-1 boot) + EvmStubAnchor (network-free
  simulator for tests + reconciler harness). Live alloy integration
  is V0.1-FOLLOWUPS Phase E hardening.
- US-032: Three-state lifecycle helpers on SqliteAnchor —
  anchor_pending / promote_to_confirmed / promote_to_quarantined /
  list_pending_older_than / list_quarantined.
- US-033: src/plugins/audit/breaker.rs — CircuitBreaker with
  Closed/Open/HalfOpen state machine + drop-as-failure + serialized
  half-open probes.
- US-034: src/storage/rate_limit_mints.rs — MintRateLimiter
  (per-OmniAccount mints/hour + per-OmniAccount EVM-tx daily budget).
- US-035: Phase C structural smoke (10 invariants). Live Base
  Sepolia smoke is V0.1-FOLLOWUPS Phase E operator task.

### Phase D-rest — Metrics + idempotency (3 stories)
- US-036: src/metrics.rs — Metrics struct with 10 AtomicU64 counters
  + render_prometheus exposition format. /metrics endpoint gated by
  BROKER_METRICS_ENABLED. Histograms + per-handler instrumentation
  pass deferred to V0.1-FOLLOWUPS.
- US-037: src/storage/idempotency.rs — IdempotencyStore with
  body_hash (SHA256) + check (NotSeen/Replay/Conflict) + store
  (INSERT OR IGNORE for race safety) + purge_expired. Body-size
  limit applied via DefaultBodyLimit::max layer.
- US-038: Phase D smoke (10 invariants).

### Phase E — Runbook final + done.sh final + bookmark (3 stories)
- US-039: docs/operator-runbook-stage7.md expanded with §Grants &
  Recovery, §EVM Audit Anchor, §Metrics & Observability sections.
- US-040: harness/stage-7-issue-64-done.sh final form — composes
  every phase smoke + load-bearing invariant + runbook drift check
  (now hard-fail) + 14 BOOT_FAIL anchors + dual feature-combo build
  matrix.
- US-041: final codex review consolidated into Phase A.2 round 3
  (PASS verdict). V0.1-FOLLOWUPS finalized with 4 Phase A.2 + 16
  Phase A.1 + 13 Phase 0 entries → 33 P2/P3 carried for v1.0.

### Session 4 totals
- All 41 PRD stories `passes: true`.
- cargo test -p agentkeys-broker-server (default features): green.
- cargo test --features auth-email-link,auth-oauth2-google,audit-evm:
  258 tests passing (was 152 in session 3; +106 = 38 OAuth2 +
  16 grants + 7 wallet + 8 lifecycle + 7 breaker + 6 rate-limit +
  4 evm + 4 metrics + 7 idempotency + 8 misc).
- clippy -D warnings: clean across all feature combos.
- bash harness/stage-7-issue-64-done.sh: exit 0; all phase smokes
  green, runbook drift clean, 14 BOOT_FAIL anchors present, load-
  bearing invariant test green.

### Final commit count
- Phases shipped this session: 6 (A.2, B, C structural, D-rest, E +
  Phase A.2 codex rounds 1/2/3).
- Total commits this session: ~10.

The boulder rests. Ralph mode terminates here. Next steps for the
operator:
1. Run cargo build --features auth-email-link,auth-oauth2-google,audit-evm
2. Run forge build + forge create AgentKeysAudit on Base Sepolia.
3. Save returned address as BROKER_EVM_CONTRACT_ADDRESS.
4. Configure all Phase A-D env vars per runbook.
5. Boot broker, exercise SIWE → mint v2 flow, observe Prom counters
   on /metrics.
6. Optionally: enable EmailLink (real SES wiring per V0.1-FOLLOWUPS
   Phase E US-039 — current build ships StubEmailSender) and
   OAuth2/Google (Google Cloud Console setup per runbook §oauth2-setup).
7. Optionally: flip BROKER_REQUIRE_EXPLICIT_GRANT=true once all
   daemons have grants issued, to close the implicit-grant fallback.