DPI production hardening: grants service, TAP runtime, BITE store (FATFD C0-C7, v1.0.0)

.audit/GUARDRAILS.md

@@ -0,0 +1,27 @@
+# Pancake Guardrails
+
+Human-owned invariants. The system may NEVER modify this file without explicit human approval.
+
+## Architecture invariants
+
+1. **The AR node never stores ownership.** Nodes verify presented grant credentials; they do not keep grantee lists.
+2. **The AR hub never stores field ownership or "my fields" lists.** The hub is the trust anchor (accounts, issuer accreditation, revocation registry) — not the ownership ledger.
+3. **Pancake never stores field geometry** beyond GeoID list membership. Geometry lives in AR nodes.
+4. **Permissions travel as credentials, not database rows.** Any relying party (TerraPipe included) verifies a presented credential; no service holds another service's ACLs.
+5. **GeoIDs are the only spatial join key** across services.
+
+## Credential-security checklist (applied at every Phase 2.5 review)
+
+1. The issuer private key is never logged, serialized into responses, or committed.
+2. Every issuance path requires an authenticated owner.
+3. Every credential has an expiry (`exp`) set.
+4. Revocation is recorded before the revoke API returns success.
+5. No endpoint returns another user's fieldlists or grants.
+6. SQL is always parameterized (ORM or bound parameters only).
+7. Vendor credentials (TAP) come from environment variables only.
+
+## Process invariants
+
+8. Tests are written before code for every new endpoint.
+9. Test count never decreases (zero-regression rule, enforced by validator check D).
+10. No direct commits to main; every cycle lands via a feature branch.

.audit/README.md

@@ -0,0 +1,74 @@
+# FATFD — Feedback, Audit, Test & Fix, Deploy
+
+The quality harness for the Pancake DPI services.
+
+## What is FATFD?
+
+FATFD is a structured development lifecycle that ensures every change to Pancake is audited, tested, verified, and documented before it ships. Each unit of work runs as a **cycle** through the phases below, and every cycle leaves a written trace in this directory.
+
+## Directory Structure
+
+```
+.audit/
+├── README.md              <- You are here
+├── GUARDRAILS.md          <- Human-owned invariants (NEVER auto-modified)
+├── fatfd_state.json       <- Harness state (version, cycle history, test counts)
+├── knowledge.json         <- All findings: fixed, open, learned, deferred
+├── validate_fatfd.py      <- Integrity validator (quality gate)
+├── hooks/                 <- Git hooks
+│   ├── pre-commit         <- Blocks commits that violate FATFD integrity
+│   └── install.sh         <- Symlink installer
+└── reports/               <- Cycle reports (timestamped)
+```
+
+## Lifecycle Phases
+
+| Phase | Name | What it does |
+|-------|------|-------------|
+| F | Collect Feedback | Ingest blockers, grant acceptance criteria, GitHub issues |
+| 1 | Audit | Static analysis (ruff), pattern grep for last finding's root-cause class, knowledge regression scan |
+| 2 | Test & Fix | Per-finding fix loop; tests written before code for every new endpoint |
+| 2.5 | Adversarial Review | Self-review against the credential-security checklist in GUARDRAILS.md |
+| 3 | Docs | README + spec docs updated in the same change |
+| 4 | Re-Audit | Full test suite + end-to-end demo script |
+| 5 | Version | Semver bump in `services/pyproject.toml`; minor bump per cycle |
+| 6 | Git | Feature branch, descriptive commits; no direct pushes to main |
+| 7 | Deploy | Staging bring-up (compose or local uvicorn); demo script green |
+| 8 | Learn | Append findings to `knowledge.json` (IDs `PC-2026-NNNN`) |
+| 9 | State | Update `fatfd_state.json`; validator must pass |
+
+## Validator
+
+Run: `python3 .audit/validate_fatfd.py`
+
+| Check | What it verifies |
+|-------|-----------------|
+| A | `pending_work` count >= knowledge.json open count |
+| B | State file freshness (updated within current cycle) |
+| C | `findings.open_count` matches knowledge.json |
+| D | Test count never decreases between cycles (zero-regression rule) |
+| E | No open critical findings when a cycle is marked complete |
+
+Exit code 0 = all checks pass; 1 = integrity failure (pre-commit hook blocks the commit).
+
+## Knowledge Base
+
+Every finding ever discovered lives in `knowledge.json` with:
+- Unique ID (`PC-2026-NNNN`)
+- Severity (critical, high, medium, low)
+- Status (fixed, open, partial, learned, deferred)
+- Area (grants, meal, tap, store, harness, docs)
+- Description and fix version
+
+## Getting Started
+
+```bash
+# Install git hooks
+bash .audit/hooks/install.sh
+
+# Run the validator
+python3 .audit/validate_fatfd.py
+
+# Run the full test suite
+.venv/bin/python -m pytest services/tests -q
+```

.audit/fatfd_state.json

@@ -0,0 +1,88 @@
+{
+  "schema_version": "1.0.0",
+  "last_updated": "2026-07-07T17:30:00-07:00",
+  "version": "1.0.0",
+  "current_cycle": {
+    "id": "C7",
+    "name": "Packaging, docs, final audit",
+    "status": "complete"
+  },
+  "findings": {
+    "total": 5,
+    "fixed": 4,
+    "open": 0,
+    "deferred": 0
+  },
+  "pending_work": [],
+  "testing": {
+    "test_count": 99,
+    "last_run": "2026-07-07T17:28:00-07:00",
+    "result": "99 passed (services/tests 95 + root legacy regression 4); ruff clean; e2e demo green; OpenAPI exported (15 paths)"
+  },
+  "cycle_history": [
+    {
+      "id": "C0",
+      "name": "Harness bootstrap",
+      "status": "complete",
+      "finished": "2026-07-07",
+      "test_count": 0,
+      "notes": "Validator (checks A-E), guardrails, knowledge base seeded (3 findings), pre-commit hook installed, feature branch feat/dpi-production-hardening created. Legacy tests/functional/test_intake.py fixed to skip when no app package exists (was 4 collection errors)."
+    },
+    {
+      "id": "C1",
+      "name": "Specs + Merkle + test-issuer kit",
+      "status": "complete",
+      "finished": "2026-07-07",
+      "test_count": 43,
+      "notes": "CREDENTIAL_PROFILE.md (SD-JWT VC claim schema, ODRL, StatusList2021, verifier rules) and MERKLE_LISTID.md (construction + 4 verified known-answer vectors) written. merkle.py, sdjwt.py, statuslist.py, issuer.py implemented. Test kit mints the 5 verifier-development credentials; dev keys generated at runtime, never committed."
+    },
+    {
+      "id": "C2",
+      "name": "Grants service skeleton + hub-delegated auth",
+      "status": "complete",
+      "finished": "2026-07-07",
+      "test_count": 51,
+      "notes": "FastAPI app factory, SQLAlchemy models (sqlite dev / postgres prod), JWKS-cached RS256 hub-token verification, create-on-first-seen user mirroring, /healthz + /healthz/me. Adversarial checks: non-access tokens, expired tokens, rogue-signer tokens all rejected (8 auth tests)."
+    },
+    {
+      "id": "C3",
+      "name": "Ownership + grants (grant-critical cycle)",
+      "status": "complete",
+      "finished": "2026-07-07",
+      "test_count": 71,
+      "notes": "POST/GET /fieldlists (idempotent by Merkle construction, owner-scoped 404s), inclusion-proof endpoint, POST /grants/issue (SD-JWT VC per CREDENTIAL_PROFILE.md), GET /grants/received (DPI-account delivery, no OTP), POST /grants/revoke (bit flipped + audit packet before 200; hub report when HUB_URL set), public /grants/status-list, POST /grants/verify relying-party endpoint. Security review against GUARDRAILS checklist: issuance requires authenticated owner; revoke issuer-only; cross-account access returns 404; every credential has exp; key never serialized."
+    },
+    {
+      "id": "C4",
+      "name": "MEAL hardening + audit API",
+      "status": "complete",
+      "finished": "2026-07-07",
+      "test_count": 83,
+      "notes": "MealStore: persistent per-ListID MEAL ledger, SHA-256 hash chain, Ed25519 packet signatures, verify_chain with signature checks (closes PC-2026-0001). Lifecycle events logged: fieldlist.created, grant.issued, grant.retrieved, grant.revoked. Audit API: GET /audit/{geoid} (resolves member GeoID -> containing ListIDs), /audit/{geoid}/report with chain integrity, /audit/meals/{id}/verify. Tamper test proves detection. Legacy implementation/meal.py fixed (PC-2026-0004) with regression tests."
+    },
+    {
+      "id": "C5",
+      "name": "TAP runtime",
+      "status": "complete",
+      "finished": "2026-07-07",
+      "test_count": 91,
+      "notes": "Canonical TAP adapter base (pancake_services.tap.adapter_base) + runtime with per-vendor interval scheduler and executor. Retry policy lives in the runtime (exponential backoff, max_retries), adapters stay dumb. FROZEN ingest interface for adapter workstreams: sink(bite) -> None. YAML vendor config with ${VAR} env interpolation; missing env var is a hard startup error (guardrail 7)."
+    },
+    {
+      "id": "C6",
+      "name": "BITE store",
+      "status": "complete",
+      "finished": "2026-07-07",
+      "test_count": 99,
+      "notes": "BiteStore: persistence with content-hash dedupe, envelope validation at the boundary, GeoID/type/vendor/time-range querying with pagination. GET /bites API (authenticated). Write-read round-trip test enforced. End-to-end test: TAP runtime -> BiteStore.save -> queryable by GeoID. pgvector column deferred by design (no RAG this sprint)."
+    },
+    {
+      "id": "C7",
+      "name": "Packaging, docs, final audit",
+      "status": "complete",
+      "finished": "2026-07-07",
+      "test_count": 99,
+      "notes": "services/Dockerfile + docker-compose.yml (postgres 16) + .env.example. CI rewritten: ruff + validator + both test suites + e2e demo + testkit smoke on Python 3.12. sprints/SPRINT_4_DATA_WALLETS.md rewritten from Indy/Aries to the shipped SD-JWT design (closes PC-2026-0003). services/README.md (API surface, config, deployment guide), root README updated. export_openapi.py (15 paths). demo/end_to_end_demo.py: issue -> retrieve -> verify -> revoke -> audit, all green. Cycle report in .audit/reports/cycle_report_20260707.md."
+    }
+  ]
+}

.audit/hooks/install.sh

@@ -0,0 +1,7 @@
+#!/bin/sh
+# Symlink FATFD hooks into .git/hooks
+HOOKS_DIR="$(cd "$(dirname "$0")" && pwd)"
+REPO_ROOT="$(git -C "$HOOKS_DIR" rev-parse --show-toplevel)"
+ln -sf "$HOOKS_DIR/pre-commit" "$REPO_ROOT/.git/hooks/pre-commit"
+chmod +x "$HOOKS_DIR/pre-commit"
+echo "FATFD pre-commit hook installed."

.audit/hooks/pre-commit

@@ -0,0 +1,11 @@
+#!/bin/sh
+# FATFD pre-commit gate: block commits that break harness integrity.
+REPO_ROOT="$(git rev-parse --show-toplevel)"
+
+if git diff --cached --name-only | grep -qE '^(\.audit/|services/)'; then
+    python3 "$REPO_ROOT/.audit/validate_fatfd.py" || exit 1
+    if [ -x "$REPO_ROOT/.venv/bin/ruff" ]; then
+        "$REPO_ROOT/.venv/bin/ruff" check "$REPO_ROOT/services" || exit 1
+    fi
+fi
+exit 0

.audit/knowledge.json

@@ -0,0 +1,55 @@
+{
+  "schema_version": "1.0.0",
+  "findings": [
+    {
+      "id": "PC-2026-0001",
+      "severity": "high",
+      "status": "fixed",
+      "area": "meal",
+      "title": "MEAL packet signatures unimplemented",
+      "description": "implementation/meal.py left cryptographic.signature as None (TODO) and sequence.previous_packet_id hardcoded to 'prev-id'. Fixed: previous_packet_id now tracked through append_packet, and the persistence layer (services/pancake_services/grants/mealstore.py) signs every packet_hash with the instance Ed25519 key and verifies signatures in verify_chain.",
+      "discovered": "2026-07-07",
+      "fix_version": "0.4.0"
+    },
+    {
+      "id": "PC-2026-0002",
+      "severity": "critical",
+      "status": "fixed",
+      "area": "grants",
+      "title": "No service or auth layer exists",
+      "description": "Pancake was POC-grade: no FastAPI service, no authentication, no persistence wiring. Fixed: services/pancake_services/grants is a FastAPI service with hub-delegated RS256/JWKS auth, fieldlists (Merkle ListIDs), SD-JWT VC grant issuance/retrieval/revocation, StatusList2021 revocation, signed MEAL audit ledger, and audit API.",
+      "discovered": "2026-07-07",
+      "fix_version": "0.4.0"
+    },
+    {
+      "id": "PC-2026-0003",
+      "severity": "medium",
+      "status": "fixed",
+      "area": "docs",
+      "title": "Sprint-4 wallet docs contradict SD-JWT VC decision",
+      "description": "sprints/SPRINT_4_DATA_WALLETS.md specified Hyperledger Indy/Aries, contradicting the adopted SD-JWT VC + StatusList2021 + did:web design. Fixed: page rewritten as a superseded/shipped notice mapping the old proposal to the shipped design with links to the normative specs.",
+      "discovered": "2026-07-07",
+      "fix_version": "1.0.0"
+    },
+    {
+      "id": "PC-2026-0004",
+      "severity": "high",
+      "status": "fixed",
+      "area": "meal",
+      "title": "Legacy MEAL hash computed before content_hash stored; verify_chain could never pass",
+      "description": "implementation/meal.py create_packet computed packet_hash while cryptographic.content_hash was still empty, so verify_chain always failed with a hash mismatch; MEAL.create/append_packet also read packet['packet_hash'] instead of packet['cryptographic']['packet_hash'] (KeyError). Both fixed; regression tests in tests/unit/test_meal_legacy.py.",
+      "discovered": "2026-07-07",
+      "fix_version": "0.4.0"
+    },
+    {
+      "id": "PC-2026-0005",
+      "severity": "medium",
+      "status": "learned",
+      "area": "grants",
+      "title": "Issuer key custody is env-var interim",
+      "description": "PANCAKE_ISSUER_KEY (Ed25519) is loaded from the environment; the service refuses to start without it and the key is never logged or serialized. Decision: env var for the sprint, vault (or KMS) before production. Track as a production-readiness gate.",
+      "discovered": "2026-07-07",
+      "fix_version": null
+    }
+  ]
+}

.audit/reports/cycle_report_20260707.md

@@ -0,0 +1,42 @@
+# FATFD Cycle Report — 2026-07-07 (C0 through C7)
+
+**Branch:** `feat/dpi-production-hardening` · **Version:** 0.1.0 → 1.0.0
+
+## Summary
+
+Pancake went from POC (loose scripts, no service, no auth, unverifiable MEAL chains) to a
+tested DPI service in seven FATFD cycles executed back-to-back:
+
+| Cycle | Delivered | Tests after |
+|---|---|---|
+| C0 | Harness: validator (A–E), guardrails, knowledge base, pre-commit hook | 0 |
+| C1 | CREDENTIAL_PROFILE.md + MERKLE_LISTID.md (normative), merkle/sdjwt/statuslist/issuer modules, 5-credential test-issuer kit | 43 |
+| C2 | FastAPI grants service, hub-delegated RS256/JWKS auth, ORM models | 51 |
+| C3 | FieldLists (Merkle ListIDs), grant issue/receive/revoke/verify, public status list | 71 |
+| C4 | Signed MEAL ledger (Ed25519 + hash chain), OpenScience audit API, legacy meal.py fixes | 83 |
+| C5 | TAP runtime: scheduler, executor, retry policy, env-interpolated config, frozen ingest interface | 91 |
+| C6 | BITE store: dedupe, validation, GeoID/type/time queries, /bites API, end-to-end runtime→store test | 99 |
+| C7 | Docker/compose packaging, .env.example, CI rewrite, sprint-4 doc rewrite, OpenAPI export, e2e demo script | 99 |
+
+## Verification evidence
+
+- `pytest services/tests` — 95 passed; `pytest tests` — 4 passed, 1 skipped (legacy POC intake suite skips by design).
+- `ruff check services` — clean.
+- `python .audit/validate_fatfd.py` — checks A–E pass.
+- `services/demo/end_to_end_demo.py` — issue → retrieve (DPI account) → verify → revoke → audit, all assertions green.
+- `services/export_openapi.py` — 15 API paths exported.
+
+## Findings ledger
+
+- PC-2026-0001 (high) MEAL signatures/linkage — **fixed** (mealstore + legacy fixes).
+- PC-2026-0002 (critical) no service/auth layer — **fixed** (grants service).
+- PC-2026-0003 (medium) Sprint-4 Indy/Aries doc drift — **fixed** (rewritten).
+- PC-2026-0004 (high) legacy MEAL hash-order bug, verify_chain never passable — **fixed** + regression tests.
+- PC-2026-0005 (medium) issuer key custody env-var interim — **learned**; vault/KMS is a production gate.
+
+## Deferred by design
+
+- pgvector/RAG columns on the BITE store (no RAG this sprint).
+- TerraPipe vendor adapters (separate workstream; frozen ingest interface published).
+- Wallet holder-binding (`cnf`) for TraceFoodChain presentation flows (credential profile already reserves the claim).
+- OTP-to-VC bridge: intentionally not built (DPI-account delivery decision); flagged for grant-owner confirmation.