Add per-PR visual diff view (Playwright + Chromatic)

[RisingOrange]

Adds a per-PR visual diff view: every PR gets side-by-side before/after renders of the site's pages, so reviewers can see the visual impact of a change without clicking through Netlify previews page-by-page. Under the hood this is visual regression testing via Playwright + Chromatic.

Non-blocking / informational. The diff view is a signal reviewers can glance at; it never blocks a merge. The workflow exits 0 regardless of diffs, and nothing in branch protection gates on it. Adopting this shouldn't change the merge flow, just surface information.

Why

• Visual regressions slip past manual QA (e.g. a past /outcomes layout bug shipped unnoticed — /outcomes is in the covered set, so this setup would have surfaced it).
• Netlify previews only enable manual inspection; the reviewer has to open each affected page themselves and remember what it looked like before. A diff view flips that: reviewers see exactly what changed, automatically.
• Works for catching regressions and reviewing intentional changes. Example: #702 changed the post banner aspect ratio across all posts — with this setup, a reviewer would have seen the change applied to /if-anyone-builds-it-campaign at a glance, rather than spot-checking via Netlify.

How it works

1. On each PR, GitHub Actions runs pnpm build && pnpm preview and navigates Playwright through the covered routes.
2. @chromatic-com/playwright captures HTML + asset archives of each page.
3. chromaui/action uploads archives; Chromatic re-renders them server-side in a controlled browser and diffs against the baseline.
4. A sticky PR comment (live example) summarizes what the run covered — counts, per-category ratios, exclusions with reasons, a link to the Chromatic review UI, and a ⚠️ warning section when any un-fixtured server-side request falls through.
5. Chromatic posts a separate "UI Review" check showing the live diff status (N changes to accept, accepted baselines, etc.).
6. Reviewers click the Chromatic link to accept/deny per snapshot. Accepted snapshots become the new baseline once main moves.

No baselines in git — Chromatic stores them per-branch and promotes on merge. No regeneration workflow, no Linux-vs-macOS font-parity problems, no PR churn from binary PNGs.

Coverage

22 routes × 2 viewports = 44 snapshots per run. The covered set is discovered by walking src/routes/; each page controls its own inclusion via a @visualDiffEnabled annotation right in its source file.

Routes and posts opt in or out via a @visualDiffEnabled annotation in the source file itself:

<!-- @visualDiffEnabled: false — admin-only tool -->

---
title: My Post
---

<!-- @visualDiffEnabled: true — post with a banner image -->

• Pages under src/routes/ default to included. Opt out with @visualDiffEnabled: false in +page.svelte or +page.ts.
• Posts under src/posts/ (rendered via [slug]) default to excluded, since they share a layout; only representatives of distinct variants (banner image, link-heavy, long-form, embedded Svelte components) opt in.

Renames carry the annotation with the file, so the test set can't silently drift from reality. tests/visual/routes.ts and tests/visual/scope-comment.ts share one scanner (tests/visual/annotations.ts) — what the test actually runs and what the PR comment reports can't diverge.

Current opt-outs (grep @visualDiffEnabled: false):

• /submitted — redirects on mount
• /verify — token-dependent
• /quotes — ~28M-pixel desktop snapshot, over Chromatic's 25M cap

Deterministic rendering — how snapshots are kept stable

A visual-regression check is only useful if the same code produces the same screenshot every run. Several measures together keep snapshots stable:

External data (the biggest source of noise) is intercepted at two boundaries:

• MSW-node (tests/visual/msw-setup.ts) intercepts outbound HTTP from the Node process during pnpm build (prerender, remote prerender functions) and pnpm preview (on-demand SSR). Notion, Airtable, and Substack RSS are served from pinned fixtures in tests/visual/fixtures/. Catch-all handlers return empty results for un-fixtured Airtable/Notion endpoints; when one is hit (or a wholly uncovered host is bypassed), the scope comment's ⚠️ section surfaces it.
• Playwright page.route() default-denies any cross-origin iframe, XHR, or fetch request from the browser. Tally, Luma, Mapbox, segment.io, and any new third-party widget someone adds in a future PR render as an empty container — no live third-party state leaks in. Cross-origin resources (Cloudinary images, fonts, inlang CDN scripts) pass through unchanged.

Non-HTTP sources of drift are neutralized in playwright.config.ts and the test harness:

• reducedMotion: 'reduce' — animations and CSS transitions complete instantly (the site already honors prefers-reduced-motion in src/styles/reset.css).
• locale: 'en-US' and timezoneId: 'UTC' pinned at the browser context level, so any Intl / date formatting is deterministic.
• img[loading="lazy"] is flipped to eager before each screenshot, and the run waits for document.fonts.ready plus networkidle, so fonts and below-the-fold images are fully resolved at capture time.

Sizing — the mobile project uses deviceScaleFactor: 1 (overriding Pixel 7's default DPR of 2.625) so long-page snapshots stay under Chromatic's 25M-pixel limit without losing layout fidelity.

A green diff on /about means "no layout regression given the fixture people list," not "the Airtable integration works." The PR comment spells out this distinction.

Scope comment

Posted on each PR by a companion workflow (.github/workflows/visual-diff-comment.yml), updated in-place on re-runs. Baseline state and warning state on fork PRs:

• RisingOrange/pauseai-website#3 — baseline comment, no warnings.
• RisingOrange/pauseai-website#4 — PR opts a new post in via frontmatter annotation; comment reflects the updated count.
• RisingOrange/pauseai-website#5 — PR introduces a deliberately un-fixtured outbound call; comment's ⚠️ section surfaces it.

Sample comment body (baseline)

**Visual diff coverage**
_commit [`abc1234`](https://…/commit/abc1234def) · [run #12345](https://…/actions/runs/12345)_

[Review snapshots in Chromatic](https://www.chromatic.com/build?appId=…&number=66)

- **Pages:** 16/19 covered. _Included_ by default.
- **Posts:** 6/101 covered. _Excluded_ by default (posts share a layout).

<details><summary>Route breakdown</summary>

- **Pages excluded:**
  - `/quotes`: ~28M-pixel desktop snapshot is over Chromatic's 25M cap
  - `/submitted`: post-form state
  - `/verify`: token-dependent
- **Posts covered:**
  - `/action`
  - `/funding`
  - `/if-anyone-builds-it-campaign`
  - `/join`
  - `/learn`
  - `/values`

</details>

<details><summary>Not exercised</summary>

- Pages that fetch from external APIs render against pinned fixtures, not live data.
- Cross-origin iframes and API calls (Tally, Luma, Mapbox, analytics, newly added widgets) are aborted at the browser boundary. Containers render as empty. Check the Netlify preview deploy to verify their real rendering.
- Only `en` locale is built.

</details>

[`tests/visual/README.md`](…) explains the annotation syntax and the coverage model.

Sample warning section (when an un-fixtured request is hit)

<details open><summary>⚠️ 1 external request not covered by fixtures</summary>

- **Endpoints:**
  - 1× `GET https://api.example.com/v1/data` (no handler; allowed through)
- **Fix:** add an explicit handler + fixture in `tests/visual/msw-setup.ts` (so the page renders its populated layout against stable data), or exclude the page with `@visualDiffEnabled: false`.
- **Why it matters:** without a fixture, CI gets either a 401 (endpoint needs auth) or drifting live data, so the snapshot won't match production.

</details>

Safety of the comment pipeline

The posting workflow (visual-diff-comment.yml) triggers on workflow_run and runs in the base-repo context with pull-requests: write. Fork code never executes in the trusted context — the trusted workflow only consumes the artifact produced by the fork's run and posts its Markdown as a sticky comment. The PR number is derived from workflow_run.head_sha + head_branch + head_repository.id via the trusted GitHub API, not from anything the fork artifact could tamper with. Annotation-reason text is escaped before rendering so a malicious reason can't close the wrapping <details>, ping users, or inject links. The residual surface — fork-authored Markdown posted verbatim — is explicitly an accepted tradeoff given the contributor model (known volunteers, worst case = visible-and-recoverable comment spam).

CI behavior + fork PRs

• Runs on PRs and pushes to main.
• Non-blocking. exitZeroOnChanges: true — workflow exits 0 even when Chromatic detects diffs. Chromatic's check is informational; branch protection is not configured to require it.
• exitOnceUploaded: true — CI exits after archive upload. Warm-cache runs: ~2:45–3:00 (scope comment + MSW overhead add ~30s to the original ~2:15 baseline). Cold-cache first run: ~5:30.
• concurrency group cancels stale runs on new pushes.
• Playwright browsers and imagetools transforms are cached.
• playwright-report/ uploaded as a GHA artifact on every run (14d retention) for inspection without needing the Chromatic UI.

Fork PRs work end-to-end, including the scope comment. GitHub Actions can't pass repo secrets to fork-PR runs, so the Chromatic project token is inlined as plaintext in the workflow — this is Chromatic's own recommended pattern for fork-PR support. The scope comment posts via the workflow_run-based companion workflow described above.

Setup required for upstream adoption

1. Create a Chromatic project linked to this repo (free tier: 5k snapshots/month — expected usage ~1–2k/mo based on current PR volume).
2. Replace the inlined project token in .github/workflows/visual-diff.yml with the new project's token.
3. Merge this PR — the first push to main uploads initial baselines; accept them once in Chromatic and the pipeline is live.

Any repo collaborator can review and accept baselines via GitHub OAuth login — no separate Chromatic invites needed.

Local screenshot workflow — using the Playwright suite for manual before/after

The Playwright suite doubles as a local before/after screenshot tool:

pnpm exec playwright test
pnpm exec playwright show-report

Runs the suite against a fresh preview build and opens an HTML report with per-route screenshots. For before/after: run on the base branch, save playwright-report/ somewhere, switch branches, re-run, eyeball both reports. No Chromatic quota, no PR check side effects.

Known limitations

• Transient mid-hydration flashes aren't caught. Static snapshots capture the settled state.
• Chromium-only. Firefox- or Safari-specific rendering bugs won't surface. Adding more browsers is possible but multiplies the snapshot count.
• en only — locale-dependent layout regressions are invisible.
• Fixture data, not live. The diff exercises the layout path given the fixture shape, not the external integration.
• Free-tier ceiling is 5,000 snapshots/month; we're well under, but worth watching as the route count grows.

Possible follow-ups — out of scope for this PR

Natural extensions once the base setup is adopted:

• More browsers. Firefox and/or WebKit to catch vendor-specific rendering bugs (e.g. #664 was a Firefox-only hero color mismatch).
• More viewports. A portrait desktop viewport would catch bugs like #667 (large portrait desktop incorrectly getting the mobile layout).
• Banner-visible snapshots. The campaign banner is disabled in layout, so banner-styling regressions (e.g. #677) aren't surfaced. Optionally force a banner on in visual-test mode.

[netlify]

👷 Deploy request for pauseai pending review.

Visit the deploys page to approve it

Name	Link
🔨 Latest commit	6133410

[Wituareard]

I like the idea but setting up a project requires running the Chromatic CLI and that doesn't work on my machine because of a space in the directory path. @anthonybailey Could you set this up maybe?

[anthonybailey]

Probably. I'll try. Nag me.

My understanding is that pnpm build actions are executed in a new place (a machine owned by GitHub) without any secrets in environment settings.

So LLM access to localize new source content will not occur (a build without keys is en-only.) This is fine and sensible as a default.

We will continue to move as much static content generation from remote sources as possible into build. But again, it should function sensibly without keys. The basic model here seems OK. Any "but in production we do this instead" qualms are mostly answered by "at the moment we have zero diff support so this is simply new information". We do need reviewers to not be misled re what has been exercised though.

Any thoughts there @RisingOrange ? What prevents misunderstanding this? Day to day CR needs some clues future users see so they know to read some subset of this PR.

The only other thing I dislike is specifying routes separately in the test. We will rename things, accidentally changing test coverage.

Any way to have the non-default sample/excluded coverage choices specified in the source code for the individual cases?

[RisingOrange]

Good points!

1. "Reviewers should not be misled about what has been exercised"

Addressed by three things working together:

• Intercept external requests and serve fixtures. Prevents live-data drift between runs and lets snapshots actually exercise populated-layout paths. Details: Deterministic rendering.
• Scope comment on each PR. A sticky comment tells reviewers exactly what was covered — counts, exclusions with reasons, caveats in the "Not exercised" section, a link into Chromatic, and a ⚠️ flag for any un-fixtured request so a new integration can't silently pass. Details: Scope comment. Live samples on fork PRs: #3 (baseline), #4 (post opt-in), #5 (warning).
• Coverage is broad. 14/17 non-post pages and 9 representative posts; the 3 non-post opt-outs are Chromatic size cap (/quotes), post-form redirect (/submitted), and token-gated (/verify).

Residual caveats are honest: CI is en-only, fixture data ≠ live data, browser-side third-party widgets are not covered. These are called out in the comment's "Not exercised" section and documented in tests/visual/README.md.

2. "Specify routes in the source instead of separately in the test"

Done via a colocated @visualDiffEnabled comment on each page — pages default to included and opt out, posts default to excluded and opt in. Renaming a file carries its opt-in/out with it; there's no central route list to keep in sync. The scope comment uses the same scanner as the test, so what runs and what the comment reports can't drift. Details: Coverage.

What do you think?