Add TOKEN_ECONOMY_GUIDANCE.md for budgeting AI spend and model choice

README.md

@@ -196,6 +196,7 @@ Detailed SpecFlow harness instructions: [QUICKSTART.md](QUICKSTART.md)
 | [QUICKSTART.md](QUICKSTART.md)                                           | Local setup and first run                   |
 | [CONTRIBUTING.md](docs/CONTRIBUTING.md)                                  | How to contribute — workflow and PR checklist |
 | [CLAUDE.md](CLAUDE.md)                                                   | Development protocol and STEEL commandments |
+| [TOKEN_ECONOMY_GUIDANCE.md](TOKEN_ECONOMY_GUIDANCE.md)                   | Token economics and model choice — budgeting AI spend predictably |
 | [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md)                             | System design and data flow                 |
 | [docs/mcp/API_REFERENCE.md](docs/mcp/API_REFERENCE.md)                   | MCP tool reference                          |
 | [docs/backend/DEVELOPMENT.md](docs/backend/DEVELOPMENT.md)               | Backend development guide                   |

TOKEN_ECONOMY_GUIDANCE.md

@@ -0,0 +1,187 @@
+# Token Economy & Model Choice Guidance
+
+**The point of this document is predictability, not a single "true cost."** 
+
+AI spend varies run to run. What a budget needs is a *repeatable way to bound it* — a small number of levers you can read off a plan before you spend anything, plus a way to calibrate those levers from your own history.
+
+---
+
+## TL;DR for budgeting
+
+For an engineer who **actively** uses AI in their SDLC — coding agents for day-to-day PRs **plus** SpecFlow for larger, spec-driven builds — plan for:
+
+> **≈ $1,600 – $2,400 / month per active engineer, all-in.**
+
+Broken down:
+
+
+| Component                         | Monthly range       | What it is                                         |
+| --------------------------------- | ------------------- | -------------------------------------------------- |
+| Baseline coding-agent usage (`B`) | ~$800               | PRs, refactors, reviews, day-to-day IDE agent work |
+| SpecFlow runs                     | ~$800 – $1,600      | 2–3 spec-driven generation runs per week           |
+| **Total (active engineer)**       | **$1,600 – $2,400** |                                                    |
+
+
+These are planning numbers for an *active* user. An engineer who runs SpecFlow occasionally sits well below this; the value of the model below is that you can place any given engineer on the scale from their actual usage.
+
+---
+
+
+
+## The prediction model
+
+Monthly AI spend for one engineer:
+
+```
+Monthly = B  +  (runs_per_month × cost_per_run)
+
+cost_per_run ≈ P × W × C
+```
+
+(A month ≈ 4 weeks, so 2–3 runs/week ≈ 8–12 runs/month.)
+
+
+| Symbol          | Meaning                        | Typical value        | Where it comes from                                                                |
+| --------------- | ------------------------------ | -------------------- | ---------------------------------------------------------------------------------- |
+| `B`             | Baseline coding-agent spend    | ~$800 / mo           | Your existing IDE-agent bill (independent of SpecFlow)                             |
+| `runs_per_month` | SpecFlow generation runs      | 8 – 12               | Team behavior (≈ 2–3 / week × 4 weeks)                                             |
+| `P`             | Phases in the plan             | 10 – 25              | **Known for free** from `run_planning` (see below)                                 |
+| `W`             | Parallel variants (workspaces) | **3** (default; 1–3) | `WORKSPACE_COUNT`; each variant runs a different model in parallel                 |
+| `C`             | Cost per phase-session         | ~$3 – $8             | **Calibrate from your telemetry** — SpecFlow records per-model token + cache usage |
+
+
+The one number you should not take on faith is `C`. Everything else is either a fact about your plan (`P`, `W`) or your existing bill (`B`). Calibrate `C` from your first few runs (SpecFlow reports per-model input/output/cache tokens and cost) and the formula becomes a genuine forecast rather than a guess.
+
+---
+
+
+
+## Why this is predictable: SpecFlow's structure
+
+Three properties of the harness make the formula above hold up.
+
+### 1. You know the size before you spend — plan first, for free
+
+`check_specification_completeness` and `run_planning` run **locally in your IDE** against your specs. They cost nothing on the backend, are re-runnable, and `run_planning` produces `IMPLEMENTATION_PLAN.md` with a concrete **phase count** (surfaced as `phase_count` when the plan is converted, and as `total_phases` on the run). The phase is SpecFlow's unit of work — so before a single generation token is spent, you already know `P`, the primary size driver of the run.
+
+If a plan comes back with 40 phases, that is your early-warning signal that the spec is large (or under-decomposed) — decide *then*, not after an 8-hour run.
+
+### 2. Each phase is a bounded session
+
+The planning skill deliberately produces **small, single-concern phases** and enforces hard limits per phase:
+
+- max **2–3 tasks**
+- max **3–5 commits**
+- max **8–10 files** created or modified
+- "completable in a single focused session"
+
+At runtime each phase is one agent session with a turn cap (`max_turns`, default 200). Bounding tasks/commits/files/turns bounds the token volume a phase can consume. This is *why* `C` is stable enough to calibrate: phases are designed not to sprawl. (Note: these are structural caps, not a fixed token budget — SpecFlow does not impose a hardcoded per-phase token limit.)
+
+### 3. Parallel variants are a known multiplier
+
+A run executes across **W parallel workspaces** (default **3**, configurable 1–3). Each workspace is assigned a *different* model, round-robined from the configured codegen tier, and builds the whole app independently. This is the mechanism behind SpecFlow's core idea — multiple SOTA models build the same spec, and agreement in their complexity scores proves the spec is complete.
+
+For budgeting it means the run cost is multiplied by `W`. Three variants ≈ 3× the single-variant cost. You choose this trade-off explicitly: more variants → stronger completeness signal, proportionally more spend.
+
+### 4. Prompt caching keeps per-phase cost down and stable
+
+Agent sessions re-read a large, mostly-unchanging context (specs, locked decisions, prior files) on every turn. With prompt caching, that re-read context is billed at the **cache-read rate (~0.1× of input price)** rather than full input price; the one-time cache write is ~1.25× input. In practice the great majority of a phase's input tokens are cache reads, which is what keeps `C` in the low single-digit dollars instead of an order of magnitude higher. SpecFlow tracks `cache_read_tokens` / `cache_write_tokens` per model and prices them from live OpenRouter rates, so your calibrated `C` already reflects your real cache behavior.
+
+---
+
+
+
+## Worked example
+
+A full-stack app (frontend + backend), planned at **22 phases**, run with the **default 3-variant fleet** — three comparable, current-generation models building in parallel (SpecFlow's configured codegen default is `anthropic/claude-sonnet-4.6`, `openai/gpt-5.5`, `z-ai/glm-5.2`; any comparable trio, e.g. a Gemini Flash-class model, is a valid substitute).
+
+```
+phase-sessions in the run = P × W = 22 phases × 3 variants = 66 sessions
+cost_per_run             ≈ 66 sessions × C
+```
+
+**Budgeting assumption (not a tool constant):** model each phase-session as ~160k tokens of effective context, re-read across its agentic loop, with ~90% of input tokens served from cache.
+
+```
+Nominal context per variant  = 22 × 160k ≈ 3.5M tokens
+Nominal context, all 3 variants ≈ 10.6M tokens
+```
+
+Billed cost is dominated by cache reads, not that nominal figure. With realistic agentic token volumes per phase (cumulative input in the low millions once tool use, file reads, and retries are counted, ~90% cached; ~50–100k output), a mid-tier phase-session lands around **C ≈ $6**:
+
+```
+cost_per_run ≈ 66 × $6 ≈ $400
+```
+
+which matches the observed range of **$200 – $400 per run** for a medium-to-large full-stack build on current-generation mid-tier models. This is a large, 3-variant run at the top of the per-run range; a typical week mixes it with smaller and single-variant runs, so the *average* run costs less. Across ~8–12 runs/month that averages out to roughly **$800 – $1,600 / month** of SpecFlow spend, on top of the ~$800 baseline → the **$1,600 – $2,400 / month** headline.
+
+Scale the same way for other sizes: halve the phase count → roughly halve the run; drop to 1 variant → roughly a third of the cost (and a weaker completeness signal).
+
+---
+
+
+
+## Model choice & due diligence
+
+Predictability depends as much on discipline as on the harness. Before you commit to a run:
+
+**1. Keep the variant fleet to one product generation with comparable size and capability.**
+The whole point of multi-variant generation is that model *disagreement* signals spec incompleteness. That signal is only valid if the models are peers. If one variant runs a current flagship reasoning model and another a small, older, or cheaper model, you can't tell whether divergence came from the spec or from the capability gap between the models — and per-variant cost stops sitting in a predictable band. So the codegen fleet (`LLM_MEDIUM`) must be **models of the same product generation, within roughly a 2× price band, and of similar coding capability** — the defaults (Sonnet-4.6 / GPT-5.5 / GLM-5.2 class) are chosen exactly this way. Practical rule of thumb: if you wouldn't hand the same production ticket to both models and expect comparable work, don't put them in the same fleet.
+
+SpecFlow exposes three tiers. Match this discipline within each:
+
+| Tier   | Env var      | Default                                                     | Used for                                        |
+| ------ | ------------ | ----------------------------------------------------------- | ----------------------------------------------- |
+| HIGH   | `LLM_HIGH`   | `anthropic/claude-opus-4.8`                                 | Planning, KB init                               |
+| MEDIUM | `LLM_MEDIUM` | `anthropic/claude-sonnet-4.6, openai/gpt-5.5, z-ai/glm-5.2` | Code generation (one per workspace), estimation |
+| LOW    | `LLM_LOW`    | `anthropic/claude-haiku-4.5`                                | Mechanical steps (indexing, markdown→JSON)      |
+
+### Which models to use for LOW / MEDIUM / HIGH
+
+The three tiers are set independently. `HIGH` and `LOW` take a single model (the first entry is used); `MEDIUM` takes a **comma-separated list** — each entry runs in one parallel workspace, so the list length effectively sets your fleet. Choose by the job each tier does, not by picking the "best" model everywhere:
+
+| Tier | Job | What to pick | Why | Current default |
+|------|-----|--------------|-----|-----------------|
+| **HIGH** | Planning & KB init — runs a handful of times per run, but the plan it writes drives every downstream phase | Your single **strongest reasoning model** | Cheapest place to spend for the largest leverage: a better plan → right-sized phases → less MEDIUM spend. Do **not** economize here. | `anthropic/claude-opus-4.8` |
+| **MEDIUM** | Code generation — one model per parallel workspace; **dominates cost** (`P × W`) | **1–3 capable, mutually comparable current-gen coding models** (see rule above) | This is where the app is actually built and where the completeness signal is produced; peers keep both the signal valid and per-variant cost in a band | `claude-sonnet-4.6`, `gpt-5.5`, `glm-5.2` |
+| **LOW** | Mechanical, high-volume, low-judgment steps (spec indexing, markdown→JSON) | The **cheapest fast model** that reliably follows structured instructions | No judgment required, so a larger model is pure wasted cost | `anthropic/claude-haiku-4.5` |
+
+Guidelines when swapping in your own models:
+- **Do not mix generations within a tier.** Replace all of `MEDIUM` together when a new generation lands — don't leave one old model beside two new ones.
+- **`HIGH` should be ≥ every `MEDIUM` model in capability.** It plans the work the MEDIUM fleet executes; a weaker planner caps the whole run.
+- **Fewer MEDIUM models = proportionally cheaper, weaker signal.** One model → cheapest, no cross-model agreement to read. Three → strongest completeness signal at ~3× the codegen cost. This is the same `W` dial as point 3.
+- **Provider format.** Values follow OpenRouter's `provider/model` convention (e.g. `anthropic/claude-opus-4.8`); with only `ANTHROPIC_API_KEY` set, SpecFlow routes to Anthropic directly instead.
+
+
+**2. Do the due diligence on the spec and the plan — it is free and it is the biggest cost lever.**
+`run_planning` is local and re-runnable. Read the plan. Confirm the phase count is proportionate to the work, that phases are single-concern, and that the spec is actually complete (`check_specification_completeness`). Every phase you remove by tightening the spec is `W ×` its cost removed from the run. A run started against a vague spec is the most expensive kind of run — it produces divergent variants (spend without a clean signal) and often needs a redo. Spend the free local iterations until the plan is *exactly* what you need before calling `run_generation`.
+
+**3. Right-size** `W` **to the decision you're making.**
+Use all 3 variants when the completeness signal matters (new product surface, estimation you'll commit budget against). Drop to 1 variant for a quick build where you don't need cross-model agreement. This is a direct, linear cost dial.
+
+---
+
+
+
+## Calibrating for your team
+
+Do this once per team, then re-check quarterly:
+
+1. Run 3–5 representative SpecFlow generations.
+2. Read the per-model token + cache usage SpecFlow records for each run (surfaced in the run report and API).
+3. Divide run cost by `P × W` to get your real `C`.
+4. Plug `B` (your existing coding-agent bill), your teams' `runs_per_month`, typical `P`, and chosen `W` into the formula.
+
+You now have a per-engineer monthly forecast grounded in your own models, specs, and cache behavior — defensible in a budget review, not a vendor estimate.
+
+---
+
+
+
+## Assumptions & caveats
+
+- `160k tokens/phase` **and** `90% cache rate` **are budgeting assumptions introduced in this document**, chosen to produce round, conservative planning numbers. They are **not** limits or constants enforced by SpecFlow. The harness bounds a phase by `max_turns` (default 200) and the 2–3 task / 3–5 commit / 8–10 file caps, and prices tokens from live per-model rates. Always prefer your calibrated `C` over these placeholders.
+- Prices move. Provider pricing (fetched live from OpenRouter, or Anthropic when `ANTHROPIC_API_KEY` is set) and model availability change; re-calibrate after any model or fleet change.
+- Deploy + E2E phases (when your analysis is `INTEGRATION_TESTS_READY`) add their own phase-sessions on the same `P × W × C` basis — count them in `P`.
+- These figures describe *active* usage. Occasional users cost proportionally less; the formula scales down cleanly.
+