Audit remediation: quality gates with teeth, pinned toolchain, content and runtime fixes

.gitattributes

@@ -1,7 +1,11 @@
-# `src/asset_manifest.py` is generated by `scripts/fingerprint_assets.py`.
+# `src/asset_manifest.py` is generated by `scripts/fingerprint_assets.py`,
+# `src/example_sources_data.py` by `scripts/embed_example_sources.py`, and
+# `src/editorial_registry_data.py` by `scripts/embed_editorial_registry.py`.
 # On merge/rebase, keep our side of the conflict — the post-merge and
-# post-rewrite hooks regenerate the file deterministically afterwards.
+# post-rewrite hooks regenerate these files deterministically afterwards.
 # This works once `scripts/install-git-hooks.sh` has been run locally,
 # which registers `merge.ours.driver = true` and points `core.hooksPath`
 # at `.githooks/`.
 src/asset_manifest.py merge=ours
+src/example_sources_data.py merge=ours
+src/editorial_registry_data.py merge=ours

.githooks/post-merge

@@ -1,9 +1,10 @@
 #!/usr/bin/env bash
-# Regenerate the asset manifest after a merge or pull so the digest
-# reflects the merged tree, not whichever parent won the conflict.
+# Regenerate generated files after a merge or pull so embedded data,
+# asset manifests, fingerprints, and prototype pages reflect the merged
+# tree, not whichever parent won the conflict.
 set -e
 cd "$(git rev-parse --show-toplevel)"
-uv run python scripts/fingerprint_assets.py >/dev/null
-if ! git diff --quiet src/asset_manifest.py public/_headers; then
-  echo "post-merge: asset manifest regenerated; stage and amend if needed"
+make build >/dev/null
+if ! git diff --quiet src/example_sources_data.py src/editorial_registry_data.py src/asset_manifest.py public/_headers public/prototyping; then
+  echo "post-merge: generated files regenerated; stage and amend if needed"
 fi

.githooks/post-rewrite

@@ -1,9 +1,10 @@
 #!/usr/bin/env bash
-# Regenerate the asset manifest after rebase/amend so the digest matches
-# the rewritten history, not whichever commit happened to win each step.
+# Regenerate generated files after rebase/amend so embedded data,
+# asset manifests, fingerprints, and prototype pages match the rewritten
+# history, not whichever commit happened to win each step.
 set -e
 cd "$(git rev-parse --show-toplevel)"
-uv run python scripts/fingerprint_assets.py >/dev/null
-if ! git diff --quiet src/asset_manifest.py public/_headers; then
-  echo "post-rewrite: asset manifest regenerated; stage and amend if needed"
+make build >/dev/null
+if ! git diff --quiet src/example_sources_data.py src/editorial_registry_data.py src/asset_manifest.py public/_headers public/prototyping; then
+  echo "post-rewrite: generated files regenerated; stage and amend if needed"
 fi

.github/pull_request_template.md

@@ -5,7 +5,6 @@
 ## Verification
 
 - [ ] `make verify`
-- [ ] `scripts/check_example_migration_parity.py`
 - [ ] `scripts/format_examples.py --check`
 - [ ] `make verify-python-version VERSION=3.13`
 - [ ] `git diff --check`

.github/workflows/preview.yml

@@ -38,21 +38,29 @@ jobs:
         env:
           CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
           CLOUDFLARE_ACCOUNT_ID: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}
-        run: npx --yes wrangler whoami
+        run: npx --yes wrangler@4.90.0 whoami
       - name: Sync Python Workers vendor
         run: uv run pywrangler sync
+      # Workflow inputs are passed through env vars rather than interpolated
+      # into the script, so a crafted input cannot inject shell commands into
+      # steps that hold the Cloudflare API token.
       - name: Upload Cloudflare Preview
         env:
           CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
           CLOUDFLARE_ACCOUNT_ID: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}
+          PREVIEW_NAME: ${{ inputs.name || 'preview' }}
+          PREVIEW_MESSAGE: ${{ github.sha }}
         run: |
           set -x
           uv run pywrangler preview \
-            --name "${{ inputs.name || 'preview' }}" \
-            --message "${{ github.sha }}" \
+            --name "$PREVIEW_NAME" \
+            --message "$PREVIEW_MESSAGE" \
             --json
       - name: Smoke test deployed Preview
-        run: scripts/smoke_deployment.py "https://${{ inputs.name || 'preview' }}-pythonbyexample.adewale-883.workers.dev"
+        env:
+          PREVIEW_NAME: ${{ inputs.name || 'preview' }}
+          PBE_SMOKE_BYPASS_SECRET: ${{ secrets.PBE_SMOKE_BYPASS_SECRET }}
+        run: scripts/smoke_deployment.py "https://${PREVIEW_NAME}-pythonbyexample.adewale-883.workers.dev"
       - name: Dump wrangler logs on failure
         if: failure()
         run: |

.github/workflows/regenerate-generated-files.yml

@@ -1,11 +1,12 @@
 name: Regenerate generated files
 
 # Merges made through the GitHub UI bypass the local git hooks that keep
-# src/asset_manifest.py and friends in sync, so a merge to main can land with
-# stale generated files (and a red Verify run). This workflow regenerates them
-# and pushes a fix-up commit. Pushes made with GITHUB_TOKEN do not trigger
-# other workflows, so the fix-up commit is not re-verified by CI; it only ever
-# contains deterministic `make build` output.
+# src/asset_manifest.py, src/example_sources_data.py, and
+# src/editorial_registry_data.py in sync, so a merge to main can land with
+# stale generated files (and a red Verify run). This
+# workflow regenerates them and pushes a fix-up commit. Pushes made with
+# GITHUB_TOKEN do not trigger other workflows, so the fix-up commit is not
+# re-verified by CI; it only ever contains deterministic `make build` output.
 
 on:
   push:
@@ -24,20 +25,37 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v5
+        with:
+          enable-cache: false
       - uses: actions/setup-python@v5
         with:
           python-version: '3.13'
-      - name: Regenerate embedded examples and fingerprinted assets
-        run: make build
+      # Pull before building so the regenerated output reflects the latest
+      # main, not the triggering SHA — building first then rebasing could
+      # push stale generated files when another commit lands mid-run.
+      # merge.ours.driver backs the merge=ours attributes on generated files
+      # so the rebase never hits conflict markers in them.
       - name: Commit and push regenerated files if they drifted
         run: |
-          git add -A src/example_sources_data.py src/asset_manifest.py public
-          if git diff --cached --quiet; then
-            echo "Generated files are current."
-            exit 0
-          fi
           git config user.name "github-actions[bot]"
           git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
-          git commit -m "Regenerate fingerprinted assets"
-          git pull --rebase origin main
-          git push origin main
+          git config merge.ours.driver true
+          for attempt in 1 2 3; do
+            git pull --rebase origin main
+            make build
+            git add -A src/example_sources_data.py src/editorial_registry_data.py src/asset_manifest.py public
+            if git diff --cached --quiet; then
+              echo "Generated files are current."
+              exit 0
+            fi
+            git commit -m "Regenerate fingerprinted assets"
+            if git push origin main; then
+              exit 0
+            fi
+            echo "Push raced with another commit; retrying (attempt $attempt)."
+            git reset --soft HEAD~1
+            git restore --staged .
+          done
+          echo "Failed to push regenerated files after 3 attempts." >&2
+          exit 1

.github/workflows/verify.yml

@@ -2,8 +2,17 @@ name: Verify
 
 on:
   push:
+    branches:
+      - main
   pull_request:
 
+permissions:
+  contents: read
+
+concurrency:
+  group: verify-${{ github.ref }}
+  cancel-in-progress: true
+
 jobs:
   verify:
     runs-on: ubuntu-latest
@@ -42,7 +51,6 @@ jobs:
           CHROME_PATH: /usr/bin/google-chrome
         run: |
           make verify
-          scripts/check_example_migration_parity.py
           scripts/format_examples.py --check
           make verify-python-version VERSION=3.13
           git diff --check

CHANGELOG.md

@@ -6,6 +6,36 @@ The format is inspired by [Keep a Changelog](https://keepachangelog.com/en/1.1.0
 
 ## Unreleased
 
+### Added
+
+- Content gates: `check_program_covers_cells.py` (every executable cell anchored in the `:::program` block, with a visible `standalone_cells` opt-out), `check_prose_duplication.py` (verbatim paragraph/note copy-paste residue), and `check_inline_links.py` (prose links must resolve to real internal pages) — all wired into `make quality-checks`.
+- Inline `[text](/examples/slug)` and `[text](/journeys/slug)` links render as anchors via the shared `src/textfmt.py` renderer; figure captions render backtick spans as code through the same path.
+- A `while-backedge` figure for the while-loops banner that actually draws the back-edge its caption describes, alongside the corrected loop-shape comparison caption.
+- Figure contracts: XML well-formedness and text-escaping (Contract 11), mechanical caption rules (Contract 5c), reverse section-figure containment, and an end-to-end check that attached figures appear in rendered pages.
+- The marginalia gestalt shows one card per attachment with the production figcaption, so caption/figure disagreements are visible during review; prototype pages regenerate in `make build` and are covered by `make check-generated`.
+- Behavioral test suite for Turnstile clearance signing, challenge modes, cookie attributes, and HTML cache keys.
+- Security headers on Worker HTML responses and static assets (nosniff, referrer-policy, frame protections, and a full CSP); POST body size cap with a friendly 413 page.
+- An end-to-end "adding a new example" workflow and a complete quality-check table in `CONTRIBUTING.md`.
+
+### Changed
+
+- The Makefile pins every Python invocation to 3.13 via `uv run`, so `make verify` matches CI regardless of the system interpreter; `make dev`/`deploy` use the workers dependency group.
+- CI workflows: deduplicated PR runs with explicit permissions and concurrency; preview workflow passes inputs through env vars (no shell interpolation next to the Cloudflare token), pins wrangler, and passes the smoke-bypass secret; the regenerate workflow pulls before building and retries with a rebuild on push races.
+- Quality gates got teeth: curated scores are bounded against the criterion heuristic (`--max-delta`), waivers must carry future ISO expiry dates and are flagged when stale, the journey-section average floor is enforced, confusable-pair tokens must appear inside teaching cells and cannot be satisfied inside a longer sibling token, every `:::note` block is checked, `paired_pages` requires both `see_also` discoverability and registry-named cell evidence, scope-first-pass pages must link registry-named focused neighbors, and the rubric-audit snapshot computes its findings from live registries instead of hardcoding PASS verdicts.
+- Editorial metadata moved from Python literals into `docs/quality-registries.toml`: journeys, see-also edge labels, figure attachments, captions, and curated scores now load through `src/editorial_registry.py`.
+- Removed the migration-era golden fixture and parity/refresh scripts after the Markdown-backed release window; ordinary content edits now rely on verifiers, content gates, generated diffs, and browser/runtime checks rather than snapshot refreshes.
+- ASGI bridge: WebSocket close events reach the close handler (`onclose` was assigned to `onopen`), headers use latin-1 per the ASGI spec, and application lifespan runs once per Worker instead of per request.
+- Unknown `TURNSTILE_CHALLENGE_MODE` values fail closed to requiring a challenge; the smoke-bypass header compares in constant time.
+- Template substitution is single-pass, so submitted code containing `__TOKEN__` text can no longer corrupt the rendered page; embedded JSON escapes `<` so `</script>` in example code cannot break out of the inline script.
+- Worker HTML cache keys ignore ordinary query strings (routes ignore them too), so arbitrary `?utm=` or cache-busting params cannot fragment the rendered-page cache.
+- FastAPI GET handlers now call renderers directly instead of delegating to the legacy pure-Python `route()` helper and reparsing URLs.
+- Oversized request bodies are capped in the ASGI bridge before the FastAPI app reads them, so the Worker no longer buffers unbounded POST bodies before rejecting them.
+- Deployment smoke refuses to send the Turnstile smoke-bypass secret to non-HTTPS origins.
+- Worker and static responses now include HSTS alongside CSP, frame, referrer, and nosniff headers.
+- Check-script boilerplate is consolidated through `scripts/_common.py`; quality gates share the canonical loader, registry reader, and frontmatter parser.
+- Small accent-colored text and the primary Run button use WCAG-AA contrast colors; the editor textarea has an accessible label.
+- Content corrections across 29 example pages: dict-iteration RuntimeError scoped to key insertion/removal, PEP 695 `type` statement teaching, NotImplemented-returning operator methods, visible positional-only/keyword-only TypeError demos, divergent broad-except demonstration, honest sandbox-boundary framing for subprocesses/threads/networking, cell-specific prose replacing copied intros, and four broken inline links now rendering as real anchors.
+
 ## 2026-05-16
 
 ### Added

CONTRIBUTING.md

@@ -49,7 +49,6 @@ Then run:
 
 ```bash
 make verify
-scripts/check_example_migration_parity.py
 scripts/format_examples.py --check
 make verify-python-version VERSION=3.13
 git diff --check
@@ -64,17 +63,38 @@ make check-generated
 
 ## Quality checks
 
-Five scripts enforce the catalog-level rules from `docs/example-quality-rubric.md`. Run them together with `make quality-checks`.
+These scripts enforce the catalog-level rules from `docs/example-quality-rubric.md`. Run them together with `make quality-checks`.
 
 | Script | What it gates |
 | --- | --- |
-| `scripts/check_registry_integrity.py` | Every owner slug in `docs/quality-registries.toml` exists in `manifest.toml`; tokens are present. |
-| `scripts/check_confusable_pairs.py` | Each confusable pair's owning page contains every token that signals the contrast. |
-| `scripts/check_broad_surface_tours.py` | Each broad-title page either covers every required form or sets `scope_first_pass = true` with `see_also` links to focused neighbors. |
+| `scripts/check_registry_integrity.py` | Every owner slug in `docs/quality-registries.toml` exists in `manifest.toml`; tokens are present; each `paired_pages` pair is discoverable through `see_also` and has registry-named cell tokens demonstrated inside a teaching cell. |
+| `scripts/check_confusable_pairs.py` | Each confusable pair's owning page contains every token inside teaching cells; a token shadowed inside a longer sibling token (plain `def` inside `async def`) does not count. |
+| `scripts/check_broad_surface_tours.py` | Each broad-title page either covers every required form or sets `scope_first_pass = true` with registry-named `focused_neighbors` linked through `see_also`. |
 | `scripts/check_footgun_coverage.py` | Each canonical Python footgun has a page that contains both broken-form and fixed-form tokens. |
-| `scripts/check_notes_supported.py` | Every `:::note` bullet shares at least one keyword with the page body, so notes cannot assert behavior the page never demonstrates. |
-
-The single source of truth for the registries is `docs/quality-registries.toml`. Add a new pair, broad tour, or footgun there, then update the owning page so the tokens appear in cells or prose.
+| `scripts/check_notes_supported.py` | Every `:::note` bullet (across all note blocks) shares at least one keyword with the page body outside the notes, so notes cannot assert behavior the page never demonstrates. |
+| `scripts/check_program_covers_cells.py` | Every executable cell shares substantive code with the `:::program` block, so the editor reproduces what the cells teach; `standalone_cells = true` opts out visibly. |
+| `scripts/check_prose_duplication.py` | No verbatim repeated paragraphs, no cell prose copied from the intro, no duplicate note bullets. |
+| `scripts/check_inline_links.py` | Inline `[text](target)` links in prose resolve to real `/examples` or `/journeys` pages. |
+| `scripts/score_example_criteria.py` | Heuristic criterion scores per example; fails when a curated score exceeds the heuristic by more than the delta bound, so the score registry cannot inflate without the page changing. |
+| `scripts/check_quality_scores.py` | Curated scores meet the 9.0 target / 8.5 hard minimum, waivers carry future expiry dates and are dropped when stale, and the journey-section average stays above its floor. |
+| `scripts/check_no_figure_rationales.py` | No-figure opt-outs carry a reason and an unexpired `review_after` date. |
+| `scripts/check_journey_outcomes.py` | Every journey section declares learner outcomes. |
+| `scripts/audit_example_graph.py --check` | The `see_also` graph has no broken targets, self-links, over-linked pages, or orphaned examples. |
+
+The single source of truth for the registries is `docs/quality-registries.toml`. Add confusable pairs, broad tours, footguns, journey metadata, figure attachments, captions, and curated scores there, then update the owning page so the tokens appear in cells or prose.
+
+## Adding a new example end to end
+
+1. Write `src/example_sources/<slug>.md` (frontmatter, intro prose, one `:::program`, cells, notes) and add the slug to `manifest.toml`'s `order`.
+2. Add `see_also` links in both directions — `scripts/audit_example_graph.py --check` fails orphaned pages.
+3. Score the page against `docs/example-quality-rubric.md` and add the `[[example_quality_scores]]` entry to `docs/quality-registries.toml`.
+4. Attach a figure by adding `[[figure_attachments]]` and `[[example_figure_scores]]` entries to `docs/quality-registries.toml` (or add a `no_figure_rationales` entry with a review date); keep only new paint functions in `src/marginalia.py`. Review the result on `public/prototyping/marginalia-gestalt.html`, which shows the production caption under every figure.
+5. Run `make build`, `make verify-examples`, `make quality-checks`, and `scripts/format_examples.py --check`.
+6. Run the full `make verify` with a local Worker running, then commit — including the regenerated `src/example_sources_data.py`, `src/editorial_registry_data.py`, `src/asset_manifest.py`, and fingerprinted assets.
+
+## Secrets and deploy configuration
+
+Deployment uses repository secrets: `CLOUDFLARE_API_TOKEN` and `CLOUDFLARE_ACCOUNT_ID` (Preview workflow), plus optional `PBE_SMOKE_BYPASS_SECRET` so deploy smoke tests can run edited-code POSTs past the Turnstile challenge. Runtime Worker secrets (`TURNSTILE_SECRET_KEY`, `TURNSTILE_CLEARANCE_SECRET`, `PBE_SMOKE_BYPASS_SECRET`) are managed with `wrangler secret put`; see `docs/turnstile-runner-protection-spec.md`.
 
 ## Style expectations