diff --git a/README.md b/README.md index 5da5521..5b1e174 100644 --- a/README.md +++ b/README.md @@ -53,9 +53,37 @@ Create a `twd.config.json` file in your project root: | `headless` | boolean | `true` | Run browser in headless mode | | `puppeteerArgs` | string[] | `["--no-sandbox", "--disable-setuid-sandbox"]` | Additional Puppeteer launch arguments | | `retryCount` | number | `2` | Number of attempts per test before reporting failure. Set to `1` to disable retries | +| `parallel` | boolean | `false` | Opt-in flag. When `true`, tests run across two isolated browser contexts concurrently (see [Parallel Test Execution](#parallel-test-execution-experimental)) | | `contracts` | array | — | OpenAPI contract validation specs (see [Contract Validation](#contract-validation)) | | `contractReportPath` | string | — | Path to write a markdown report for CI/PR integration | +### Parallel Test Execution (experimental) + +Set `parallel: true` in `twd.config.json` to run your suite across two isolated Puppeteer browser contexts concurrently. On a typical developer laptop this cuts wallclock test time roughly in half (measured ~1.8× speedup on a 60-test suite). The flag is opt-in; the default is `false` and existing behavior is unchanged. + +```jsonc +{ + "url": "http://localhost:5173", + "parallel": true, + "retryCount": 2 +} +``` + +How it works: + +- Two `browser.createBrowserContext()` sessions run in parallel via `Promise.all`. Each has its own service-worker scope and storage, so mocks in one worker cannot leak to the other. +- Tests are partitioned round-robin by registration order (`idx % 2`) inside each worker — no cross-process coordination needed. +- Chromium anti-throttle flags (`--disable-background-timer-throttling`, `--disable-renderer-backgrounding`, `--disable-backgrounding-occluded-windows`) are appended automatically. +- The existing `retryCount` is honored per worker — flaky tests retry exactly like in serial mode. +- Each worker writes its own coverage file to `.nyc_output/out-.json`. `npx nyc report` merges them automatically — no extra tooling required. +- Contract validation works unchanged. Mocks are collected per worker and merged before validation. + +Current limitations (worth knowing): + +- **Worker count is fixed at 2.** Higher counts give flakier results on most developer machines (CPU contention exceeds the 1-second `waitFor` default in some tests). Configurable worker counts are planned once twd-js ships deterministic test IDs. +- **Per-worker reports.** Results print as two separate trees plus a combined summary. A single unified tree across workers is a follow-up improvement. +- **CI tuning.** On resource-constrained runners, keep `retryCount: 2` (the default) to absorb occasional timeouts. Larger CI runners (4+ vCPUs) are comfortable at N=2. + ## How It Works **Important**: Puppeteer is **not** used as a testing framework here. It simply provides a headless browser to load your application — the same way a user would open Chrome. Once the page loads, all test execution happens inside the real browser context through the [TWD runner](https://brikev.github.io/twd/). Your tests interact with real DOM, real components, and real browser APIs — Puppeteer just opens the door and gets out of the way. diff --git a/docs/superpowers/plans/2026-04-21-parallel-test-execution-poc.md b/docs/superpowers/plans/2026-04-21-parallel-test-execution-poc.md new file mode 100644 index 0000000..48b29f7 --- /dev/null +++ b/docs/superpowers/plans/2026-04-21-parallel-test-execution-poc.md @@ -0,0 +1,854 @@ +# Parallel Test Execution POC Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Build a throwaway Node script under `poc/parallel/` that runs the `test-example-app` TWD test suite in parallel across N isolated Puppeteer browser contexts, writes per-worker coverage, and captures findings in a README — to prove service-worker isolation and coverage split work before designing a production feature. + +**Architecture:** Single `puppeteer.launch()` → N `browser.createBrowserContext()` workers running in `Promise.all`. Worker 0 doubles as probe: it navigates first, reads test IDs from `window.__TWD_STATE__.handlers`, round-robin splits them, then all workers run `window.__testRunner.runByIds(chunk)` in parallel. Each worker dumps `window.__coverage__` to `.nyc_output/out-.json`. No new dependencies; no changes to `src/` or the published `runTests()`. + +**Tech Stack:** Node.js (ESM), Puppeteer (already in `twd-cli/node_modules`), `twd-js` in-browser `__testRunner`, existing nyc setup in `test-example-app`. + +--- + +## ⚠ Commit Policy for This Plan + +Per project convention (see memory), **the executor MUST NOT run `git commit` without explicit user approval in the current turn.** Every "commit" step below is written as *stage + propose*: run `git add`, show the proposed commit message, then pause for the user to say "commit". Never bundle `git add` and `git commit` together autonomously. + +The feature branch `feat/parallel-execution` is already checked out and current with `main`. No branch creation needed. + +--- + +## Prerequisites (one-time, before starting Task 1) + +1. The dev server must be running in `test-example-app/`. In a separate terminal, the user should run: + ```bash + cd /Users/kevinccbsg/brikev/twd-cli/test-example-app + npm run dev + ``` + This starts Vite on `http://localhost:5173`. Leave it running for the entire POC. + +2. Verify the baseline serial CLI works before starting — no point writing parallel if the serial run is broken: + ```bash + cd /Users/kevinccbsg/brikev/twd-cli/test-example-app + rm -rf .nyc_output + npx twd-cli run + ``` + Expected: test tree prints, all tests pass (contract warnings OK in `warn` mode), `.nyc_output/out.json` is written, exit code 0. Note the total test count — we'll use it as the oracle for Task 7. + +--- + +## File Structure + +**Files to create:** +- `poc/parallel/run-parallel.js` — the POC script (ESM, ~150 LoC) +- `poc/parallel/README.md` — findings log + how to run + +**Files to modify:** None. The POC is strictly additive and does not touch `src/`, `bin/`, `package.json`, or existing tests. + +--- + +## Task 1: Scaffold the POC directory and stub script + +**Files:** +- Create: `/Users/kevinccbsg/brikev/twd-cli/poc/parallel/run-parallel.js` +- Create: `/Users/kevinccbsg/brikev/twd-cli/poc/parallel/README.md` + +- [ ] **Step 1: Create `poc/parallel/` directory** + +```bash +mkdir -p /Users/kevinccbsg/brikev/twd-cli/poc/parallel +``` + +- [ ] **Step 2: Create `run-parallel.js` with argv parsing stub** + +Create `/Users/kevinccbsg/brikev/twd-cli/poc/parallel/run-parallel.js` with this content: + +```javascript +import fs from 'node:fs'; +import path from 'node:path'; + +const URL = 'http://localhost:5173'; +const NYC_DIR = path.resolve(process.cwd(), '.nyc_output'); +const PUPPETEER_ARGS = ['--no-sandbox', '--disable-setuid-sandbox']; +const TIMEOUT = 10000; + +async function main() { + const N = parseInt(process.argv[2], 10) || 2; + console.log(`Parallel POC — N=${N}, URL=${URL}, NYC_DIR=${NYC_DIR}`); +} + +main().catch((err) => { + console.error('POC error:', err); + process.exit(1); +}); +``` + +- [ ] **Step 3: Create `README.md` skeleton** + +Create `/Users/kevinccbsg/brikev/twd-cli/poc/parallel/README.md` with this content: + +```markdown +# Parallel Test Execution POC + +Throwaway script that proves Puppeteer browser contexts isolate service workers and that per-worker coverage files merge cleanly. + +See the approved spec at `docs/superpowers/specs/2026-04-21-parallel-test-execution-poc-design.md`. + +## How to run + +1. In one terminal, start the dev server: + ```bash + cd test-example-app + npm run dev + ``` +2. In another terminal: + ```bash + cd test-example-app + rm -rf .nyc_output + node ../poc/parallel/run-parallel.js 2 # N workers; default 2 + npx nyc report --reporter=text + ``` + +## Findings + +_To be filled in after Task 8._ +``` + +- [ ] **Step 4: Verify the scaffold runs** + +Run: +```bash +cd /Users/kevinccbsg/brikev/twd-cli/test-example-app +node ../poc/parallel/run-parallel.js 3 +``` + +Expected output: +``` +Parallel POC — N=3, URL=http://localhost:5173, NYC_DIR=/Users/kevinccbsg/brikev/twd-cli/test-example-app/.nyc_output +``` + +If run without an arg, N should default to 2: +```bash +node ../poc/parallel/run-parallel.js +``` +Expected: `... N=2, ...` + +- [ ] **Step 5: Stage and propose commit (DO NOT COMMIT YET)** + +```bash +cd /Users/kevinccbsg/brikev/twd-cli +git add poc/parallel/run-parallel.js poc/parallel/README.md +``` + +Proposed commit message: +``` +poc(parallel): scaffold parallel test runner directory + +Adds poc/parallel/{run-parallel.js,README.md} with an argv stub. No +puppeteer wiring yet. Standalone ESM script, runnable from +test-example-app/ as `node ../poc/parallel/run-parallel.js [N]`. +``` + +Wait for user approval before running `git commit`. + +--- + +## Task 2: Launch browser, probe test IDs, round-robin split + +**Files:** +- Modify: `/Users/kevinccbsg/brikev/twd-cli/poc/parallel/run-parallel.js` + +This task wires Puppeteer in, navigates one page to the dev server, reads test IDs from `window.__TWD_STATE__.handlers`, splits them round-robin, and logs the chunks. Still single-threaded — no workers yet. + +**Background context for the engineer:** +- TWD registers all tests at module load via `describe`/`it` calls. The registry lives on `window.__TWD_STATE__.handlers` (a `Map`). Each `Handler` has a `type: 'suite' | 'test'` field — we only want `type === 'test'`. +- The page is "ready" when `#twd-sidebar-root` is in the DOM. This is the same readiness check the existing serial CLI uses (see `src/index.js:53`). +- Round-robin split spreads related tests across workers (useful because `test-example-app/src/App.twd.test.ts` clusters contract tests inside `describe` blocks). + +- [ ] **Step 1: Add puppeteer import and launch** + +Edit `run-parallel.js` — add `import puppeteer from 'puppeteer';` at the top alongside the other imports, then replace the body of `main` so the full file reads: + +```javascript +import fs from 'node:fs'; +import path from 'node:path'; +import puppeteer from 'puppeteer'; + +const URL = 'http://localhost:5173'; +const NYC_DIR = path.resolve(process.cwd(), '.nyc_output'); +const PUPPETEER_ARGS = ['--no-sandbox', '--disable-setuid-sandbox']; +const TIMEOUT = 10000; + +async function main() { + const N = parseInt(process.argv[2], 10) || 2; + console.log(`Parallel POC — N=${N}, URL=${URL}`); + + const browser = await puppeteer.launch({ + headless: true, + args: PUPPETEER_ARGS, + }); + + try { + // Worker 0 doubles as the probe. + const ctx0 = await browser.createBrowserContext(); + const page0 = await ctx0.newPage(); + await page0.goto(URL); + await page0.waitForSelector('#twd-sidebar-root', { timeout: TIMEOUT }); + + const testIds = await page0.evaluate(() => { + return Array.from(window.__TWD_STATE__.handlers.values()) + .filter((h) => h.type === 'test') + .map((h) => h.id); + }); + console.log(`Discovered ${testIds.length} tests`); + + // Round-robin split + const chunks = Array.from({ length: N }, () => []); + testIds.forEach((id, i) => chunks[i % N].push(id)); + + chunks.forEach((chunk, i) => { + console.log( + `Worker ${i}: ${chunk.length} tests, first=${chunk[0]}, last=${chunk[chunk.length - 1]}` + ); + }); + } finally { + await browser.close(); + } +} + +main().catch((err) => { + console.error('POC error:', err); + process.exit(1); +}); +``` + +- [ ] **Step 2: Run the script and verify test discovery** + +Ensure the dev server is running (see Prerequisites). Then: +```bash +cd /Users/kevinccbsg/brikev/twd-cli/test-example-app +node ../poc/parallel/run-parallel.js 2 +``` + +Expected output (exact test count depends on the current state of `App.twd.test.ts`, but should be ~80+ tests): +``` +Parallel POC — N=2, URL=http://localhost:5173 +Discovered 80 tests +Worker 0: 40 tests, first=abc123xyz, last=def456uvw +Worker 1: 40 tests, first=xyz789abc, last=uvw012def +``` + +Two things to check manually: +- `Discovered N tests` where N > 0 — if 0, the probe isn't finding `__TWD_STATE__` (check the dev server is running and the sidebar is actually mounting). +- The `first=` IDs differ across workers — round-robin is working. + +- [ ] **Step 3: Try N=3 and N=1 as sanity checks** + +```bash +node ../poc/parallel/run-parallel.js 3 +node ../poc/parallel/run-parallel.js 1 +``` + +Expected: with N=3, three chunks of roughly equal size. With N=1, single chunk with all tests. + +- [ ] **Step 4: Stage and propose commit (DO NOT COMMIT YET)** + +```bash +cd /Users/kevinccbsg/brikev/twd-cli +git add poc/parallel/run-parallel.js +``` + +Proposed commit message: +``` +poc(parallel): probe test IDs and round-robin split + +Wires Puppeteer, navigates one browser context to the dev server, +reads window.__TWD_STATE__.handlers to enumerate tests, and splits +them round-robin into N chunks. Logs chunk sizes but does not run +any tests yet. +``` + +Wait for user approval. + +--- + +## Task 3: Run worker 0's chunk (serial, one worker) + +**Files:** +- Modify: `/Users/kevinccbsg/brikev/twd-cli/poc/parallel/run-parallel.js` + +This task makes worker 0 actually execute its chunk via `runByIds` and report results. Still single-threaded — workers 1..N-1 come in Task 4. + +**Background context:** +- `window.__testRunner` is the `TestRunner` class from `twd/src/runner.ts`. Its `runByIds(ids)` method runs only the tests whose IDs are in the set and invokes suite hooks correctly (see spec §"Test Distribution"). +- The page.evaluate pattern here closely mirrors what `src/index.js:57-81` does, just with `runByIds` instead of `runAll` and without the retryCount logic. + +- [ ] **Step 1: Extract a runWorker helper and wire it for worker 0** + +Edit `run-parallel.js`. Replace the `main` function with: + +```javascript +async function runWorker(workerIndex, chunk, page) { + console.log(`Worker ${workerIndex}: starting ${chunk.length} tests`); + const testStatus = await page.evaluate(async (ids) => { + const TestRunner = window.__testRunner; + const status = []; + const runner = new TestRunner({ + onStart: () => {}, + onPass: (t) => status.push({ id: t.id, status: 'pass' }), + onFail: (t, err) => status.push({ + id: t.id, status: 'fail', error: `${err.message} (at ${window.location.href})`, + }), + onSkip: (t) => status.push({ id: t.id, status: 'skip' }), + }); + await runner.runByIds(ids); + return status; + }, chunk); + return testStatus; +} + +async function main() { + const N = parseInt(process.argv[2], 10) || 2; + console.log(`Parallel POC — N=${N}, URL=${URL}`); + + const browser = await puppeteer.launch({ + headless: true, + args: PUPPETEER_ARGS, + }); + + try { + const ctx0 = await browser.createBrowserContext(); + const page0 = await ctx0.newPage(); + await page0.goto(URL); + await page0.waitForSelector('#twd-sidebar-root', { timeout: TIMEOUT }); + + const testIds = await page0.evaluate(() => { + return Array.from(window.__TWD_STATE__.handlers.values()) + .filter((h) => h.type === 'test') + .map((h) => h.id); + }); + console.log(`Discovered ${testIds.length} tests`); + + const chunks = Array.from({ length: N }, () => []); + testIds.forEach((id, i) => chunks[i % N].push(id)); + + chunks.forEach((chunk, i) => { + console.log( + `Worker ${i}: ${chunk.length} tests, first=${chunk[0]}, last=${chunk[chunk.length - 1]}` + ); + }); + + // Run worker 0 only for now — workers 1..N-1 added in Task 4. + const worker0Status = await runWorker(0, chunks[0], page0); + const pass0 = worker0Status.filter((s) => s.status === 'pass').length; + const fail0 = worker0Status.filter((s) => s.status === 'fail').length; + console.log(`Worker 0 done: ${pass0} passed, ${fail0} failed`); + } finally { + await browser.close(); + } +} +``` + +- [ ] **Step 2: Run and verify worker 0 executes its chunk** + +```bash +cd /Users/kevinccbsg/brikev/twd-cli/test-example-app +node ../poc/parallel/run-parallel.js 2 +``` + +Expected output tail: +``` +Worker 0: starting 40 tests +Worker 0 done: 40 passed, 0 failed +``` + +If any tests fail here, check against the serial baseline — it might be a real test failure unrelated to the POC, or the chunk selection is misbehaving. + +- [ ] **Step 3: Stage and propose commit (DO NOT COMMIT YET)** + +```bash +cd /Users/kevinccbsg/brikev/twd-cli +git add poc/parallel/run-parallel.js +``` + +Proposed commit message: +``` +poc(parallel): run worker 0 chunk via runByIds + +Extracts runWorker helper that calls window.__testRunner.runByIds +and collects per-test pass/fail/skip status. Currently runs only +worker 0; remaining workers added in the next commit. +``` + +Wait for user approval. + +--- + +## Task 4: Parallel fan-out to remaining workers + +**Files:** +- Modify: `/Users/kevinccbsg/brikev/twd-cli/poc/parallel/run-parallel.js` + +This is the core parallelization step. Workers 1..N-1 each create their own `browser.createBrowserContext()`, navigate, wait for readiness, run their chunk, and close the context. All N workers execute in `Promise.all`. + +- [ ] **Step 1: Wrap worker 0 and add workers 1..N-1 in Promise.all** + +Edit `run-parallel.js`. Replace the `// Run worker 0 only for now` block (and what follows up to `} finally {`) with: + +```javascript + console.time('Parallel test time'); + + const workerPromises = [runWorker(0, chunks[0], page0)]; + for (let i = 1; i < N; i++) { + const workerIndex = i; + const chunk = chunks[i]; + workerPromises.push((async () => { + const ctx = await browser.createBrowserContext(); + const page = await ctx.newPage(); + await page.goto(URL); + await page.waitForSelector('#twd-sidebar-root', { timeout: TIMEOUT }); + const result = await runWorker(workerIndex, chunk, page); + await ctx.close(); + return result; + })()); + } + + const results = await Promise.all(workerPromises); + console.timeEnd('Parallel test time'); + + results.forEach((status, i) => { + const pass = status.filter((s) => s.status === 'pass').length; + const fail = status.filter((s) => s.status === 'fail').length; + console.log(`Worker ${i} done: ${pass} passed, ${fail} failed`); + }); +``` + +Note: we intentionally do NOT close `ctx0` here — it'll be cleaned up by `browser.close()` in the `finally`. Keeping it alive makes the code simpler (worker 0 is special because it ran the probe). + +- [ ] **Step 2: Run with N=2 and verify both workers execute** + +```bash +cd /Users/kevinccbsg/brikev/twd-cli/test-example-app +node ../poc/parallel/run-parallel.js 2 +``` + +Expected output tail: +``` +Worker 0: starting 40 tests +Worker 1: starting 40 tests +Worker 0 done: 40 passed, 0 failed +Worker 1 done: 40 passed, 0 failed +Parallel test time: XXXXms +``` + +Important checks: +- Both `Worker N: starting` lines appear before either `Worker N done` line — that means execution really did interleave. +- `Parallel test time` is less than the serial baseline's `Total Test Time` (directional — this is not a strict pass gate). + +- [ ] **Step 3: Run with N=3 and N=4** + +```bash +node ../poc/parallel/run-parallel.js 3 +node ../poc/parallel/run-parallel.js 4 +``` + +Expected: 3 or 4 worker lines, all with `0 failed`. If any worker has failures that weren't in the serial baseline, that's the **F1** failure mode (SW isolation broken) — record it and stop. + +- [ ] **Step 4: Stage and propose commit (DO NOT COMMIT YET)** + +```bash +cd /Users/kevinccbsg/brikev/twd-cli +git add poc/parallel/run-parallel.js +``` + +Proposed commit message: +``` +poc(parallel): fan out N-1 workers in Promise.all + +Workers 1..N-1 each open their own browser context, navigate, wait +for sidebar readiness, and run their chunk via runByIds. Worker 0 +continues to run on the probe page. All workers execute concurrently +via Promise.all. +``` + +Wait for user approval. + +--- + +## Task 5: Per-worker coverage dump + +**Files:** +- Modify: `/Users/kevinccbsg/brikev/twd-cli/poc/parallel/run-parallel.js` + +Each worker writes its coverage object to `.nyc_output/out-.json` regardless of pass/fail (see spec §"Coverage Output"). `.nyc_output/` is cleaned at script start so stale files don't pollute the merge. + +**Background context:** +- `window.__coverage__` is the istanbul coverage object, automatically populated by the babel/istanbul instrumentation the Vite app uses. It's a plain object keyed by absolute file path. +- `nyc` merges all `*.json` files in `.nyc_output/` automatically when you run `npx nyc report`. No manual merge step needed. + +- [ ] **Step 1: Clear `.nyc_output/` at startup** + +Edit `run-parallel.js`. At the top of `main`, after the initial `console.log` and before `puppeteer.launch`, add: + +```javascript + fs.rmSync(NYC_DIR, { recursive: true, force: true }); + fs.mkdirSync(NYC_DIR, { recursive: true }); +``` + +- [ ] **Step 2: Dump coverage inside `runWorker`** + +Edit `runWorker`. After the `page.evaluate(...)` that returns `testStatus`, and before `return testStatus;`, insert: + +```javascript + const coverage = await page.evaluate(() => window.__coverage__); + if (coverage) { + const outPath = path.join(NYC_DIR, `out-${workerIndex}.json`); + fs.writeFileSync(outPath, JSON.stringify(coverage)); + console.log(`Worker ${workerIndex}: coverage → ${outPath}`); + } else { + console.log(`Worker ${workerIndex}: no __coverage__ on window`); + } +``` + +Full `runWorker` should now look like: + +```javascript +async function runWorker(workerIndex, chunk, page) { + console.log(`Worker ${workerIndex}: starting ${chunk.length} tests`); + const testStatus = await page.evaluate(async (ids) => { + const TestRunner = window.__testRunner; + const status = []; + const runner = new TestRunner({ + onStart: () => {}, + onPass: (t) => status.push({ id: t.id, status: 'pass' }), + onFail: (t, err) => status.push({ + id: t.id, status: 'fail', error: `${err.message} (at ${window.location.href})`, + }), + onSkip: (t) => status.push({ id: t.id, status: 'skip' }), + }); + await runner.runByIds(ids); + return status; + }, chunk); + + const coverage = await page.evaluate(() => window.__coverage__); + if (coverage) { + const outPath = path.join(NYC_DIR, `out-${workerIndex}.json`); + fs.writeFileSync(outPath, JSON.stringify(coverage)); + console.log(`Worker ${workerIndex}: coverage → ${outPath}`); + } else { + console.log(`Worker ${workerIndex}: no __coverage__ on window`); + } + + return testStatus; +} +``` + +- [ ] **Step 3: Run and verify coverage files land** + +```bash +cd /Users/kevinccbsg/brikev/twd-cli/test-example-app +node ../poc/parallel/run-parallel.js 2 +ls -la .nyc_output/ +``` + +Expected: +- `Worker 0: coverage → .../out-0.json` line in POC output +- `Worker 1: coverage → .../out-1.json` line +- `ls` shows `out-0.json` and `out-1.json`, both non-empty (file size > 1 KB, usually much more) + +If a worker logs `no __coverage__ on window`, the Vite app isn't instrumenting (unrelated to POC — same issue would break serial coverage). Check the `vite.config.ts` in `test-example-app/`. + +- [ ] **Step 4: Verify nyc can merge both files** + +```bash +cd /Users/kevinccbsg/brikev/twd-cli/test-example-app +npx nyc report --reporter=text +``` + +Expected: a coverage table with file paths, percentages, no errors. If `nyc` errors out, record the exact error for the **F2** failure mode. + +- [ ] **Step 5: Stage and propose commit (DO NOT COMMIT YET)** + +```bash +cd /Users/kevinccbsg/brikev/twd-cli +git add poc/parallel/run-parallel.js +``` + +Proposed commit message: +``` +poc(parallel): dump per-worker coverage to out-.json + +Each worker writes window.__coverage__ to .nyc_output/out-.json +after its chunk finishes, regardless of pass/fail. .nyc_output/ is +cleaned at script start to avoid stale data in the merge. User +verifies with `npx nyc report`. +``` + +Wait for user approval. + +--- + +## Task 6: Aggregated summary output and exit code + +**Files:** +- Modify: `/Users/kevinccbsg/brikev/twd-cli/poc/parallel/run-parallel.js` + +Finish the output so the script prints a clear summary and exits with the right code. + +- [ ] **Step 1: Add totals block and exit code** + +Edit `run-parallel.js`. Replace the `results.forEach(...)` block from Task 4 with: + +```javascript + let totalPass = 0; + let totalFail = 0; + let totalSkip = 0; + results.forEach((status, i) => { + const pass = status.filter((s) => s.status === 'pass').length; + const fail = status.filter((s) => s.status === 'fail').length; + const skip = status.filter((s) => s.status === 'skip').length; + totalPass += pass; + totalFail += fail; + totalSkip += skip; + console.log(`Worker ${i} done: ${pass} passed, ${fail} failed, ${skip} skipped`); + }); + console.log( + `Total: ${totalPass} passed, ${totalFail} failed, ${totalSkip} skipped ` + + `(expected ${testIds.length})` + ); + + if (totalPass + totalFail + totalSkip !== testIds.length) { + console.warn( + `WARNING: total reported (${totalPass + totalFail + totalSkip}) ` + + `does not match discovered test count (${testIds.length}) — ` + + `chunks may be dropping IDs` + ); + } + + if (totalFail > 0) { + process.exitCode = 1; + } +``` + +Also print the list of failures (if any) right before the totals block so they're easy to spot: + +```javascript + results.forEach((status, i) => { + const failures = status.filter((s) => s.status === 'fail'); + if (failures.length > 0) { + console.log(`\nWorker ${i} failures:`); + failures.forEach((f) => console.log(` [${f.id}] ${f.error}`)); + } + }); +``` + +Insert this block **before** the counts block in step 1 so it reads: failures first, then per-worker counts, then total. + +- [ ] **Step 2: Run with N=2 and verify clean output** + +```bash +cd /Users/kevinccbsg/brikev/twd-cli/test-example-app +node ../poc/parallel/run-parallel.js 2 +echo "Exit code: $?" +``` + +Expected: +``` +... +Worker 0 done: 40 passed, 0 failed, 0 skipped +Worker 1 done: 40 passed, 0 failed, 0 skipped +Total: 80 passed, 0 failed, 0 skipped (expected 80) +Exit code: 0 +``` + +(Actual test count depends on `App.twd.test.ts` — match the baseline from Prerequisites.) + +- [ ] **Step 3: Verify exit code when forcing a failure** + +Quick synthetic check: edit `test-example-app/src/App.twd.test.ts`, change one `twd.should(heading, "have.text", "Vite + React")` to `"have.text", "Definitely Wrong"`, re-run the POC, confirm exit code is 1, then **revert the change**. + +Expected: +``` +Worker N failures: + [] expected ... (at http://localhost:5173/) +Total: 79 passed, 1 failed, 0 skipped (expected 80) +Exit code: 1 +``` + +Don't commit the test-file change — revert with `git checkout test-example-app/src/App.twd.test.ts` once confirmed. + +- [ ] **Step 4: Stage and propose commit (DO NOT COMMIT YET)** + +```bash +cd /Users/kevinccbsg/brikev/twd-cli +git add poc/parallel/run-parallel.js +``` + +Proposed commit message: +``` +poc(parallel): print aggregated summary and set exit code + +Adds per-worker failure listing, overall pass/fail/skip totals, and +an oracle warning when the aggregated count doesn't match the +discovered test count. process.exitCode = 1 when any worker failed. +``` + +Wait for user approval. + +--- + +## Task 7: Baseline comparison run (verification, no code changes) + +**Files:** None changed in this task — pure verification that feeds the findings log in Task 8. + +The goal: prove success criteria (a), (b), (c) by comparing serial baseline vs parallel at N=2 and N=3 or N=4. + +- [ ] **Step 1: Capture baseline (serial) counts and coverage** + +```bash +cd /Users/kevinccbsg/brikev/twd-cli/test-example-app +rm -rf .nyc_output +npx twd-cli run 2>&1 | tee /tmp/baseline.log +npx nyc report --reporter=text 2>&1 | tee /tmp/baseline-cov.log +``` + +Record for later: +- Total test count (look for `Tests to report: N` line in baseline.log) +- Pass count, fail count, skip count (count from the tree or tail of the log) +- Overall coverage % from `baseline-cov.log` (last row of the table, "All files") +- Wallclock `Total Test Time` from the log + +- [ ] **Step 2: Run parallel at N=2 and capture** + +```bash +cd /Users/kevinccbsg/brikev/twd-cli/test-example-app +rm -rf .nyc_output +node ../poc/parallel/run-parallel.js 2 2>&1 | tee /tmp/parallel-n2.log +npx nyc report --reporter=text 2>&1 | tee /tmp/parallel-n2-cov.log +``` + +Record: +- Total counts from the `Total: ...` line +- Wallclock from `Parallel test time` line +- Overall coverage % from `parallel-n2-cov.log` + +- [ ] **Step 3: Run parallel at N=3 and N=4, capture** + +```bash +cd /Users/kevinccbsg/brikev/twd-cli/test-example-app +rm -rf .nyc_output +node ../poc/parallel/run-parallel.js 3 2>&1 | tee /tmp/parallel-n3.log +npx nyc report --reporter=text 2>&1 | tee /tmp/parallel-n3-cov.log + +rm -rf .nyc_output +node ../poc/parallel/run-parallel.js 4 2>&1 | tee /tmp/parallel-n4.log +npx nyc report --reporter=text 2>&1 | tee /tmp/parallel-n4-cov.log +``` + +- [ ] **Step 4: Evaluate the three criteria** + +For each of N=2, N=3, N=4, check: + +**(c) Tests run to completion** — `Total: P passed, F failed, S skipped` line's sum equals the baseline total. Pass count matches baseline exactly. If fail count is higher than baseline, note which tests failed in the `Worker N failures:` lines — that's likely criterion (a) failing. + +**(a) SW isolation** — derived: if (c) passes, (a) passes. Extra check: scan the parallel logs for any test ID that appears in failures for multiple workers (shouldn't happen; would indicate distribution bug, not isolation). + +**(b) Coverage split** — `ls .nyc_output/` shows N files named `out-.json`, all non-empty. `npx nyc report` produced a coherent table. The reported overall % is within ±2 percentage points of the baseline's overall %. (Exact match is unlikely because parallel execution order exercises slightly different code paths, but it should be very close.) + +- [ ] **Step 5: Record observations** + +Write down (on paper / in a scratch file / in a code comment — somewhere you can transcribe into the README in Task 8): + +- Baseline: X tests, Y passed, Z failed, W% coverage, T seconds +- N=2: ... +- N=3: ... +- N=4: ... +- Any failure modes observed (F1/F2/F3/F4 from spec). +- Any surprises (flaky tests, unusual output, Puppeteer warnings). + +This feeds directly into Task 8. + +--- + +## Task 8: Write the findings log in README.md + +**Files:** +- Modify: `/Users/kevinccbsg/brikev/twd-cli/poc/parallel/README.md` + +Fill in the "Findings" section based on Task 7's observations. This is the artifact that justifies the POC's conclusion. + +- [ ] **Step 1: Replace the placeholder with structured findings** + +Edit `README.md`. Replace the `## Findings` section (currently just `_To be filled in after Task 8._`) with, verbatim structure, filled in with real numbers from Task 7: + +```markdown +## Findings + +**Date run:** YYYY-MM-DD +**Machine:** +**Node:** +**Puppeteer:** + +### Numbers + +| Run | Tests | Passed | Failed | Skipped | Coverage % | Wallclock | +|-----------|-------|--------|--------|---------|------------|-----------| +| Baseline | 80 | 80 | 0 | 0 | 72.1 | 42.3 s | +| N=2 | 80 | 80 | 0 | 0 | 72.0 | 24.1 s | +| N=3 | 80 | 80 | 0 | 0 | 71.9 | 18.9 s | +| N=4 | 80 | 80 | 0 | 0 | 71.8 | 16.2 s | + +_(Above numbers are placeholders — replace with observed values.)_ + +### Criteria + +- **(a) Service-worker isolation**: PASS / FAIL — . +- **(b) Coverage split**: PASS / FAIL — .json files present, nyc merged cleanly, overall % within ±2pp of baseline>. +- **(c) Tests run to completion**: PASS / FAIL — . + +### Anomalies + +- _List anything surprising: flaky tests, Puppeteer warnings, workers that took suspiciously long, etc. If nothing, write "None observed."_ + +### Recommendation + + +``` + +- [ ] **Step 2: Verify the README renders sensibly** + +```bash +cat /Users/kevinccbsg/brikev/twd-cli/poc/parallel/README.md +``` + +Sanity-check: no "YYYY-MM-DD" left, no "PASS / FAIL" placeholders left (committed to one or the other), no `` stubs. + +- [ ] **Step 3: Stage and propose commit (DO NOT COMMIT YET)** + +```bash +cd /Users/kevinccbsg/brikev/twd-cli +git add poc/parallel/README.md +``` + +Proposed commit message (adapt wording if outcome differs): +``` +poc(parallel): document findings — all three criteria pass + +Records baseline vs parallel results for N=2/3/4 and concludes SW +isolation holds across browser contexts and per-worker coverage +merges cleanly with nyc. Green-lights production feature design. +``` + +Wait for user approval. + +--- + +## Completion + +Once all 8 tasks are done and committed (at user direction): + +1. The POC is captured on `feat/parallel-execution`. +2. The spec and findings are ready to feed into a follow-up "production parallel test execution" spec if the outcome was positive. +3. The POC script is NOT exported from the package and is NOT referenced by `bin/twd-cli.js` — nothing ships to npm from this work. + +If the outcome was negative (any failure mode in the spec), the findings log captures the evidence so the next design iteration starts from real data rather than speculation. diff --git a/docs/superpowers/plans/2026-04-22-parallel-test-execution-production.md b/docs/superpowers/plans/2026-04-22-parallel-test-execution-production.md new file mode 100644 index 0000000..837732a --- /dev/null +++ b/docs/superpowers/plans/2026-04-22-parallel-test-execution-production.md @@ -0,0 +1,1473 @@ +# Parallel Test Execution (Production) Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Ship an opt-in `parallel: true` mode in `twd-cli` that runs the TWD suite across two isolated Puppeteer browser contexts, reusing existing retries, coverage, and contract validation — with zero regression risk for `parallel: false` (default). + +**Architecture:** A new `src/runParallel.js` module (~200 LoC) implements the parallel path. A new `src/mergeMocks.js` utility combines per-worker mock maps. `src/index.js` grows a thin branch: if `config.parallel` is truthy, delegate to `runParallel`; otherwise run the existing serial body, unchanged. Each worker self-filters tests by `idx % N === workerIndex` inside its own `page.evaluate` (random twd-js test IDs are not stable across contexts). Anti-throttle flags are automatically appended to `config.puppeteerArgs` unless already present. + +**Tech Stack:** Node.js (ESM), Puppeteer 24.x (`browser.createBrowserContext()`), existing `twd-js` in-browser `TestRunner` (with its built-in retry loop), Vitest with `vi.mock('puppeteer')` for unit testing. + +--- + +## ⚠ Commit Policy for This Plan + +Per project convention, **the executor MUST NOT run `git commit` without explicit user approval in the current turn.** Every "commit" step is written as *stage + propose*: run `git add` the specific files, print the proposed commit message, then stop for the user to say "commit". Never bundle `git add` and `git commit` together autonomously. + +The feature branch `feat/parallel-execution` is already checked out. The POC spec/plan/code from yesterday's work remain on this branch (staged but not committed for `poc/parallel/run-parallel.js` + `README.md`; untracked for the spec and plan docs). This production work layers on top of that state. + +--- + +## File Structure + +**Files to create:** +- `src/runParallel.js` — the parallel orchestration module (~200 LoC) +- `src/mergeMocks.js` — pure utility that merges per-worker mock maps (~30 LoC) +- `tests/runParallel.test.js` — unit tests mirroring `tests/runTests.test.js` +- `tests/mergeMocks.test.js` — pure-function unit tests + +**Files to modify:** +- `src/config.js` — add `parallel: false` to `DEFAULT_CONFIG` +- `tests/config.test.js` — include `parallel: false` in the default-config assertions +- `src/index.js` — add a thin branch on `config.parallel` at the top of `runTests()` +- `tests/runTests.test.js` — one new test asserting serial path is NOT triggered when `parallel: true` +- `README.md` — document the new `parallel` config field (one paragraph) + +No package.json changes. No new dependencies. + +--- + +## Task 1: Add `parallel` to config defaults + +**Files:** +- Modify: `/Users/kevinccbsg/brikev/twd-cli/src/config.js` +- Modify: `/Users/kevinccbsg/brikev/twd-cli/tests/config.test.js` + +### Step 1: Update config default-config test to expect `parallel: false` + +Edit `/Users/kevinccbsg/brikev/twd-cli/tests/config.test.js`. In the three places that assert the full default config object (the tests titled `"should load default config when no config file exists"`, `"should merge user config with defaults when config file exists"`, and `"should return defaults and warn when config file has invalid JSON"`), add `parallel: false,` to each `expect(config).toEqual({...})` block. + +For example, the first test's expected object becomes: +```javascript +expect(config).toEqual({ + url: 'http://localhost:5173', + timeout: 10000, + coverage: true, + coverageDir: './coverage', + nycOutputDir: './.nyc_output', + headless: true, + puppeteerArgs: ['--no-sandbox', '--disable-setuid-sandbox'], + retryCount: 2, + parallel: false, +}); +``` + +Apply the same `parallel: false,` addition to the object in the `"should merge user config with defaults"` test and the `"should return defaults and warn"` test. + +### Step 2: Run the config tests — expect the three to fail + +```bash +cd /Users/kevinccbsg/brikev/twd-cli +npx vitest run tests/config.test.js +``` + +Expected: 3 failing tests — the three that assert the full default-config object. Error message will show the actual object missing `parallel: false`. + +### Step 3: Add `parallel: false` to `DEFAULT_CONFIG` in `src/config.js` + +Edit `/Users/kevinccbsg/brikev/twd-cli/src/config.js`. Change: + +```javascript +const DEFAULT_CONFIG = { + url: 'http://localhost:5173', + timeout: 10000, + coverage: true, + coverageDir: './coverage', + nycOutputDir: './.nyc_output', + headless: true, + puppeteerArgs: ['--no-sandbox', '--disable-setuid-sandbox'], + retryCount: 2, +}; +``` + +to: + +```javascript +const DEFAULT_CONFIG = { + url: 'http://localhost:5173', + timeout: 10000, + coverage: true, + coverageDir: './coverage', + nycOutputDir: './.nyc_output', + headless: true, + puppeteerArgs: ['--no-sandbox', '--disable-setuid-sandbox'], + retryCount: 2, + parallel: false, +}; +``` + +### Step 4: Run config tests — expect all to pass + +```bash +cd /Users/kevinccbsg/brikev/twd-cli +npx vitest run tests/config.test.js +``` + +Expected: all config tests pass. + +### Step 5: Stage and propose commit (DO NOT COMMIT) + +```bash +cd /Users/kevinccbsg/brikev/twd-cli +git add src/config.js tests/config.test.js +``` + +Proposed commit message: +``` +feat(config): add parallel flag to DEFAULT_CONFIG (default false) + +Adds a `parallel` boolean field to the config schema with a default of +false. No behavior change — subsequent commits wire the flag into a new +runParallel module. +``` + +Wait for user approval before running `git commit`. + +--- + +## Task 2: `mergeMocks` utility (TDD, pure function) + +**Files:** +- Create: `/Users/kevinccbsg/brikev/twd-cli/tests/mergeMocks.test.js` +- Create: `/Users/kevinccbsg/brikev/twd-cli/src/mergeMocks.js` + +The merge utility takes an array of per-worker `Map` and produces one `Map` with worker-index-prefixed keys, guaranteeing no silent collision if two workers' random test IDs ever overlap. + +### Step 1: Write `tests/mergeMocks.test.js` with four cases + +Create `/Users/kevinccbsg/brikev/twd-cli/tests/mergeMocks.test.js`: + +```javascript +import { describe, it, expect } from 'vitest'; +import { mergeMocks } from '../src/mergeMocks.js'; + +describe('mergeMocks', () => { + it('returns an empty map when given an array of empty maps', () => { + const merged = mergeMocks([new Map(), new Map()]); + expect(merged.size).toBe(0); + }); + + it('prefixes entries from a single worker with w0:', () => { + const w0 = new Map([ + ['GET:/api/a:200:t1:1', { alias: 'a', testId: 't1' }], + ]); + const merged = mergeMocks([w0, new Map()]); + expect(merged.size).toBe(1); + expect(merged.has('w0:GET:/api/a:200:t1:1')).toBe(true); + }); + + it('preserves entries from both workers with disjoint keys', () => { + const w0 = new Map([['GET:/a:200:t1:1', { alias: 'a', testId: 't1' }]]); + const w1 = new Map([['GET:/b:200:t2:1', { alias: 'b', testId: 't2' }]]); + const merged = mergeMocks([w0, w1]); + expect(merged.size).toBe(2); + expect(merged.has('w0:GET:/a:200:t1:1')).toBe(true); + expect(merged.has('w1:GET:/b:200:t2:1')).toBe(true); + }); + + it('keeps both entries when two workers happen to use the same inner key', () => { + // Defense-in-depth: twd-js random IDs could collide; the prefix must + // prevent one worker from overwriting the other's mock. + const sharedInnerKey = 'GET:/api/users:200:same-id:1'; + const w0 = new Map([[sharedInnerKey, { alias: 'users', testId: 'same-id', from: 0 }]]); + const w1 = new Map([[sharedInnerKey, { alias: 'users', testId: 'same-id', from: 1 }]]); + const merged = mergeMocks([w0, w1]); + expect(merged.size).toBe(2); + expect(merged.get(`w0:${sharedInnerKey}`).from).toBe(0); + expect(merged.get(`w1:${sharedInnerKey}`).from).toBe(1); + }); + + it('attaches workerIndex to each merged mock', () => { + const w0 = new Map([['k', { alias: 'a' }]]); + const w1 = new Map([['k', { alias: 'a' }]]); + const merged = mergeMocks([w0, w1]); + expect(merged.get('w0:k').workerIndex).toBe(0); + expect(merged.get('w1:k').workerIndex).toBe(1); + }); +}); +``` + +### Step 2: Run the test — expect failure (module not found) + +```bash +cd /Users/kevinccbsg/brikev/twd-cli +npx vitest run tests/mergeMocks.test.js +``` + +Expected: test suite fails with `Failed to resolve import "../src/mergeMocks.js"`. + +### Step 3: Implement `src/mergeMocks.js` + +Create `/Users/kevinccbsg/brikev/twd-cli/src/mergeMocks.js`: + +```javascript +// Merge per-worker mock maps into a single map. +// Key scheme: `w${workerIndex}:${originalKey}` — worker-index prefix is +// defense-in-depth against random-ID collisions across contexts. +// Each output mock gets a workerIndex field so downstream enrichment +// (buildTestPath) can pick the correct worker's handler tree. +export function mergeMocks(workerMaps) { + const merged = new Map(); + workerMaps.forEach((workerMap, workerIndex) => { + for (const [key, mock] of workerMap) { + merged.set(`w${workerIndex}:${key}`, { ...mock, workerIndex }); + } + }); + return merged; +} +``` + +### Step 4: Run the test — expect all 5 pass + +```bash +cd /Users/kevinccbsg/brikev/twd-cli +npx vitest run tests/mergeMocks.test.js +``` + +Expected: all 5 `mergeMocks` tests pass. + +### Step 5: Stage and propose commit (DO NOT COMMIT) + +```bash +cd /Users/kevinccbsg/brikev/twd-cli +git add src/mergeMocks.js tests/mergeMocks.test.js +``` + +Proposed commit message: +``` +feat(mergeMocks): add pure utility for per-worker mock merging + +Worker-index-prefixed keys prevent silent collisions if two browser +contexts happen to generate the same twd-js random testId. Each +merged mock carries its workerIndex field for downstream testName +resolution. +``` + +Wait for user approval. + +--- + +## Task 3: `runParallel` module — core execution (no contracts yet) + +**Files:** +- Create: `/Users/kevinccbsg/brikev/twd-cli/tests/runParallel.test.js` (partial — no contract tests yet) +- Create: `/Users/kevinccbsg/brikev/twd-cli/src/runParallel.js` (no contract handling yet) + +This is the main task. It covers: launching Puppeteer with anti-throttle flags, creating 2 browser contexts, running `runByIds` in each via `page.evaluate` with `workerIndex`/`N`/`retryCount`, dumping per-worker coverage files, aggregating pass/fail counts, and returning the correct `hasFailures` value. Contract handling and the `exposeFunction` call come in Task 4. + +### Step 1: Write `tests/runParallel.test.js` with core tests + +Create `/Users/kevinccbsg/brikev/twd-cli/tests/runParallel.test.js`: + +```javascript +import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest'; +import { runParallel } from '../src/runParallel.js'; + +vi.mock('fs'); +vi.mock('puppeteer'); +vi.mock('twd-js/runner-ci', () => ({ reportResults: vi.fn() })); +vi.mock('../src/contracts.js', () => ({ validateMocks: vi.fn() })); +vi.mock('../src/contractReport.js', () => ({ printContractReport: vi.fn() })); +vi.mock('../src/contractMarkdown.js', () => ({ generateContractMarkdown: vi.fn() })); + +import fs from 'fs'; +import puppeteer from 'puppeteer'; +import { reportResults } from 'twd-js/runner-ci'; + +function createMockPage(evaluateResult, coverage = null) { + const page = { + goto: vi.fn(), + waitForSelector: vi.fn(), + exposeFunction: vi.fn(), + evaluate: vi.fn(), + }; + // page.evaluate is called twice per worker: once for runByIds, once for __coverage__. + page.evaluate + .mockResolvedValueOnce(evaluateResult) // runByIds call + .mockResolvedValueOnce(coverage); // __coverage__ call + return page; +} + +function createMockContext(page) { + return { + newPage: vi.fn().mockResolvedValue(page), + close: vi.fn(), + }; +} + +function createMockBrowser(contexts) { + let i = 0; + return { + createBrowserContext: vi.fn().mockImplementation(() => contexts[i++]), + close: vi.fn(), + }; +} + +const baseConfig = { + url: 'http://localhost:5173', + timeout: 10000, + coverage: false, + coverageDir: './coverage', + nycOutputDir: './.nyc_output', + headless: true, + puppeteerArgs: [], + retryCount: 2, + parallel: true, +}; + +describe('runParallel', () => { + beforeEach(() => { + vi.clearAllMocks(); + vi.spyOn(console, 'log').mockImplementation(() => {}); + vi.spyOn(console, 'error').mockImplementation(() => {}); + vi.spyOn(console, 'time').mockImplementation(() => {}); + vi.spyOn(console, 'timeEnd').mockImplementation(() => {}); + vi.mocked(fs.existsSync).mockReturnValue(false); + vi.mocked(fs.mkdirSync).mockImplementation(() => {}); + vi.mocked(fs.writeFileSync).mockImplementation(() => {}); + vi.mocked(fs.rmSync).mockImplementation(() => {}); + }); + + afterEach(() => { + vi.restoreAllMocks(); + }); + + it('launches puppeteer once and creates 2 browser contexts', async () => { + const page0 = createMockPage({ handlers: [], testStatus: [] }); + const page1 = createMockPage({ handlers: [], testStatus: [] }); + const browser = createMockBrowser([createMockContext(page0), createMockContext(page1)]); + vi.mocked(puppeteer.launch).mockResolvedValue(browser); + + await runParallel(baseConfig, '/cwd', []); + + expect(puppeteer.launch).toHaveBeenCalledTimes(1); + expect(browser.createBrowserContext).toHaveBeenCalledTimes(2); + }); + + it('appends anti-throttle flags to user-supplied puppeteerArgs', async () => { + const page0 = createMockPage({ handlers: [], testStatus: [] }); + const page1 = createMockPage({ handlers: [], testStatus: [] }); + const browser = createMockBrowser([createMockContext(page0), createMockContext(page1)]); + vi.mocked(puppeteer.launch).mockResolvedValue(browser); + + await runParallel({ ...baseConfig, puppeteerArgs: ['--user-flag'] }, '/cwd', []); + + const launchArgs = vi.mocked(puppeteer.launch).mock.calls[0][0].args; + expect(launchArgs).toContain('--user-flag'); + expect(launchArgs).toContain('--disable-background-timer-throttling'); + expect(launchArgs).toContain('--disable-renderer-backgrounding'); + expect(launchArgs).toContain('--disable-backgrounding-occluded-windows'); + }); + + it('does not duplicate an anti-throttle flag already provided by the user', async () => { + const page0 = createMockPage({ handlers: [], testStatus: [] }); + const page1 = createMockPage({ handlers: [], testStatus: [] }); + const browser = createMockBrowser([createMockContext(page0), createMockContext(page1)]); + vi.mocked(puppeteer.launch).mockResolvedValue(browser); + + await runParallel( + { ...baseConfig, puppeteerArgs: ['--disable-renderer-backgrounding'] }, + '/cwd', + [] + ); + + const launchArgs = vi.mocked(puppeteer.launch).mock.calls[0][0].args; + const count = launchArgs.filter((a) => a === '--disable-renderer-backgrounding').length; + expect(count).toBe(1); + }); + + it('navigates each page to config.url and waits for sidebar', async () => { + const page0 = createMockPage({ handlers: [], testStatus: [] }); + const page1 = createMockPage({ handlers: [], testStatus: [] }); + const browser = createMockBrowser([createMockContext(page0), createMockContext(page1)]); + vi.mocked(puppeteer.launch).mockResolvedValue(browser); + + await runParallel(baseConfig, '/cwd', []); + + expect(page0.goto).toHaveBeenCalledWith('http://localhost:5173'); + expect(page1.goto).toHaveBeenCalledWith('http://localhost:5173'); + expect(page0.waitForSelector).toHaveBeenCalledWith( + '#twd-sidebar-root', + { timeout: 10000 } + ); + expect(page1.waitForSelector).toHaveBeenCalledWith( + '#twd-sidebar-root', + { timeout: 10000 } + ); + }); + + it('passes workerIndex, N=2, and retryCount to each page.evaluate call', async () => { + const page0 = createMockPage({ handlers: [], testStatus: [] }); + const page1 = createMockPage({ handlers: [], testStatus: [] }); + const browser = createMockBrowser([createMockContext(page0), createMockContext(page1)]); + vi.mocked(puppeteer.launch).mockResolvedValue(browser); + + await runParallel({ ...baseConfig, retryCount: 3 }, '/cwd', []); + + // First evaluate call per worker is the runByIds invocation. + expect(page0.evaluate).toHaveBeenNthCalledWith(1, expect.any(Function), 0, 2, 3); + expect(page1.evaluate).toHaveBeenNthCalledWith(1, expect.any(Function), 1, 2, 3); + }); + + it('sums pass/fail/skip counts across workers', async () => { + const page0 = createMockPage({ + handlers: [{ id: 'a', name: 'a', type: 'test' }], + testStatus: [{ id: 'a', status: 'pass' }], + }); + const page1 = createMockPage({ + handlers: [{ id: 'b', name: 'b', type: 'test' }], + testStatus: [{ id: 'b', status: 'fail', error: 'boom' }], + }); + const browser = createMockBrowser([createMockContext(page0), createMockContext(page1)]); + vi.mocked(puppeteer.launch).mockResolvedValue(browser); + + const hasFailures = await runParallel(baseConfig, '/cwd', []); + + expect(hasFailures).toBe(true); + expect(reportResults).toHaveBeenCalledTimes(2); + }); + + it('returns false when all workers pass', async () => { + const page0 = createMockPage({ + handlers: [{ id: 'a', name: 'a', type: 'test' }], + testStatus: [{ id: 'a', status: 'pass' }], + }); + const page1 = createMockPage({ + handlers: [{ id: 'b', name: 'b', type: 'test' }], + testStatus: [{ id: 'b', status: 'pass' }], + }); + const browser = createMockBrowser([createMockContext(page0), createMockContext(page1)]); + vi.mocked(puppeteer.launch).mockResolvedValue(browser); + + const hasFailures = await runParallel(baseConfig, '/cwd', []); + + expect(hasFailures).toBe(false); + }); + + it('writes per-worker coverage files when config.coverage is true and __coverage__ is non-null', async () => { + const page0 = createMockPage( + { handlers: [], testStatus: [] }, + { file0: 'data' } + ); + const page1 = createMockPage( + { handlers: [], testStatus: [] }, + { file1: 'data' } + ); + const browser = createMockBrowser([createMockContext(page0), createMockContext(page1)]); + vi.mocked(puppeteer.launch).mockResolvedValue(browser); + + await runParallel({ ...baseConfig, coverage: true }, '/cwd', []); + + const writes = vi.mocked(fs.writeFileSync).mock.calls.map((c) => c[0]); + expect(writes.some((p) => p.endsWith('out-0.json'))).toBe(true); + expect(writes.some((p) => p.endsWith('out-1.json'))).toBe(true); + }); + + it('does not write coverage files when config.coverage is false', async () => { + const page0 = createMockPage({ handlers: [], testStatus: [] }, { file0: 'data' }); + const page1 = createMockPage({ handlers: [], testStatus: [] }, { file1: 'data' }); + const browser = createMockBrowser([createMockContext(page0), createMockContext(page1)]); + vi.mocked(puppeteer.launch).mockResolvedValue(browser); + + await runParallel({ ...baseConfig, coverage: false }, '/cwd', []); + + expect(fs.writeFileSync).not.toHaveBeenCalled(); + }); + + it('dumps coverage even when a worker has failures', async () => { + const page0 = createMockPage( + { + handlers: [{ id: 'a', name: 'a', type: 'test' }], + testStatus: [{ id: 'a', status: 'fail', error: 'boom' }], + }, + { file0: 'data' } + ); + const page1 = createMockPage( + { handlers: [], testStatus: [] }, + { file1: 'data' } + ); + const browser = createMockBrowser([createMockContext(page0), createMockContext(page1)]); + vi.mocked(puppeteer.launch).mockResolvedValue(browser); + + await runParallel({ ...baseConfig, coverage: true }, '/cwd', []); + + expect(fs.writeFileSync).toHaveBeenCalledTimes(2); + }); + + it('cleans .nyc_output before running when coverage is enabled', async () => { + const page0 = createMockPage({ handlers: [], testStatus: [] }, { a: 1 }); + const page1 = createMockPage({ handlers: [], testStatus: [] }, { a: 1 }); + const browser = createMockBrowser([createMockContext(page0), createMockContext(page1)]); + vi.mocked(puppeteer.launch).mockResolvedValue(browser); + vi.mocked(fs.existsSync).mockReturnValue(true); + + await runParallel({ ...baseConfig, coverage: true }, '/cwd', []); + + expect(fs.rmSync).toHaveBeenCalledWith( + expect.stringContaining('.nyc_output'), + { recursive: true, force: true } + ); + }); +}); +``` + +### Step 2: Run the test — expect failure (module not found) + +```bash +cd /Users/kevinccbsg/brikev/twd-cli +npx vitest run tests/runParallel.test.js +``` + +Expected: test suite fails with `Failed to resolve import "../src/runParallel.js"`. + +### Step 3: Implement `src/runParallel.js` (core, no contracts) + +Create `/Users/kevinccbsg/brikev/twd-cli/src/runParallel.js`: + +```javascript +import fs from 'fs'; +import path from 'path'; +import puppeteer from 'puppeteer'; +import { reportResults } from 'twd-js/runner-ci'; + +const WORKERS = 2; + +const ANTI_THROTTLE_FLAGS = [ + '--disable-background-timer-throttling', + '--disable-renderer-backgrounding', + '--disable-backgrounding-occluded-windows', +]; + +function mergeArgs(userArgs, extras) { + const merged = [...userArgs]; + for (const flag of extras) { + if (!merged.includes(flag)) merged.push(flag); + } + return merged; +} + +async function runWorker(browser, workerIndex, config, workingDir) { + const ctx = await browser.createBrowserContext(); + const page = await ctx.newPage(); + + await page.goto(config.url); + await page.waitForSelector('#twd-sidebar-root', { timeout: config.timeout }); + + const { handlers, testStatus } = await page.evaluate( + async (workerIndex, N, retryCount) => { + const allIds = Array.from(window.__TWD_STATE__.handlers.values()) + .filter((h) => h.type === 'test') + .map((h) => h.id); + const myIds = allIds.filter((_, idx) => idx % N === workerIndex); + + const TestRunner = window.__testRunner; + const testStatus = []; + const runner = new TestRunner( + { + onStart: (test) => { test.status = 'running'; }, + onPass: (test, retryAttempt) => { + test.status = 'done'; + const entry = { id: test.id, status: 'pass' }; + if (retryAttempt !== undefined) entry.retryAttempt = retryAttempt; + testStatus.push(entry); + }, + onFail: (test, err) => { + test.status = 'done'; + testStatus.push({ + id: test.id, + status: 'fail', + error: `${err.message} (at ${window.location.href})`, + }); + }, + onSkip: (test) => { + test.status = 'done'; + testStatus.push({ id: test.id, status: 'skip' }); + }, + }, + { retryCount } + ); + const handlers = await runner.runByIds(myIds); + return { handlers: Array.from(handlers.values()), testStatus }; + }, + workerIndex, + WORKERS, + config.retryCount + ); + + // Always dump coverage when enabled — including on failures, unlike serial. + if (config.coverage) { + const coverage = await page.evaluate(() => window.__coverage__); + if (coverage) { + const nycDir = path.resolve(workingDir, config.nycOutputDir); + const outPath = path.join(nycDir, `out-${workerIndex}.json`); + fs.writeFileSync(outPath, JSON.stringify(coverage)); + console.log(`Worker ${workerIndex}: coverage → ${outPath}`); + } else { + console.log(`Worker ${workerIndex}: no __coverage__ on window`); + } + } + + await ctx.close(); + return { workerIndex, handlers, testStatus }; +} + +export async function runParallel(config, workingDir, contractValidators) { + let browser; + try { + console.log(`Starting TWD test runner (parallel mode, ${WORKERS} workers)...`); + console.log('Configuration:', JSON.stringify(config, null, 2)); + + if (config.coverage) { + const nycDir = path.resolve(workingDir, config.nycOutputDir); + if (fs.existsSync(nycDir)) { + fs.rmSync(nycDir, { recursive: true, force: true }); + } + fs.mkdirSync(nycDir, { recursive: true }); + } + + browser = await puppeteer.launch({ + headless: config.headless, + args: mergeArgs(config.puppeteerArgs, ANTI_THROTTLE_FLAGS), + }); + + console.time('Parallel test time'); + const workerResults = await Promise.all( + Array.from({ length: WORKERS }, (_, i) => + runWorker(browser, i, config, workingDir) + ) + ); + console.timeEnd('Parallel test time'); + + let totalPass = 0; + let totalFail = 0; + let totalSkip = 0; + for (const { workerIndex, handlers, testStatus } of workerResults) { + console.log(`\n────── Worker ${workerIndex} results ──────`); + reportResults(handlers, testStatus); + const pass = testStatus.filter((s) => s.status === 'pass').length; + const fail = testStatus.filter((s) => s.status === 'fail').length; + const skip = testStatus.filter((s) => s.status === 'skip').length; + console.log( + `Worker ${workerIndex}: ${pass} passed, ${fail} failed, ${skip} skipped` + ); + totalPass += pass; + totalFail += fail; + totalSkip += skip; + } + + console.log(`\n────── Summary ──────`); + console.log( + `Total: ${totalPass} passed, ${totalFail} failed, ${totalSkip} skipped` + ); + + const hasFailures = totalFail > 0; + + await browser.close(); + console.log('Browser closed.'); + return hasFailures; + } catch (error) { + console.error('Error running tests (parallel):', error); + if (browser) await browser.close(); + throw error; + } +} +``` + +### Step 4: Run the test — expect all pass + +```bash +cd /Users/kevinccbsg/brikev/twd-cli +npx vitest run tests/runParallel.test.js +``` + +Expected: all tests in `tests/runParallel.test.js` pass. + +### Step 5: Also run the full test suite — ensure nothing regressed + +```bash +cd /Users/kevinccbsg/brikev/twd-cli +npx vitest run +``` + +Expected: every existing test still passes. `tests/runTests.test.js` is unaffected because `runTests()` in `index.js` hasn't changed yet. + +### Step 6: Stage and propose commit (DO NOT COMMIT) + +```bash +cd /Users/kevinccbsg/brikev/twd-cli +git add src/runParallel.js tests/runParallel.test.js +``` + +Proposed commit message: +``` +feat(runParallel): add core parallel execution module (no contracts) + +Introduces src/runParallel.js. Launches one Puppeteer browser with +anti-throttle flags, creates two isolated browser contexts, navigates +each to the configured URL, runs a test chunk via runByIds (self-filtered +by idx % N === workerIndex inside page.evaluate), dumps per-worker +window.__coverage__ to .nyc_output/out-.json, and aggregates pass/ +fail/skip counts. Contract mock collection and merging land in the next +commit. Not yet wired into src/index.js. +``` + +Wait for user approval. + +--- + +## Task 4: `runParallel` — contract mock collection and merge + +**Files:** +- Modify: `/Users/kevinccbsg/brikev/twd-cli/tests/runParallel.test.js` +- Modify: `/Users/kevinccbsg/brikev/twd-cli/src/runParallel.js` + +This task wires per-worker `exposeFunction('__twdCollectMock', ...)`, merges the collected mocks via `mergeMocks`, enriches them with testName via `buildTestPath` using each mock's `workerIndex` to pick the right handler tree, and feeds them through the existing `validateMocks` / `printContractReport` pipeline. + +### Step 1: Add contract tests to `tests/runParallel.test.js` + +Append the following inside the `describe('runParallel', ...)` block in `/Users/kevinccbsg/brikev/twd-cli/tests/runParallel.test.js`: + +```javascript + it('exposes __twdCollectMock on each page when contracts are configured', async () => { + const page0 = createMockPage({ handlers: [], testStatus: [] }); + const page1 = createMockPage({ handlers: [], testStatus: [] }); + const browser = createMockBrowser([createMockContext(page0), createMockContext(page1)]); + vi.mocked(puppeteer.launch).mockResolvedValue(browser); + + const { validateMocks } = await import('../src/contracts.js'); + const { printContractReport } = await import('../src/contractReport.js'); + vi.mocked(validateMocks).mockReturnValue({ results: [], skipped: [] }); + vi.mocked(printContractReport).mockReturnValue(false); + + await runParallel( + { ...baseConfig, contracts: [{ source: './openapi.json' }] }, + '/cwd', + [{ /* sentinel validator */ }] + ); + + expect(page0.exposeFunction).toHaveBeenCalledWith( + '__twdCollectMock', + expect.any(Function) + ); + expect(page1.exposeFunction).toHaveBeenCalledWith( + '__twdCollectMock', + expect.any(Function) + ); + }); + + it('does not expose __twdCollectMock when contracts are not configured', async () => { + const page0 = createMockPage({ handlers: [], testStatus: [] }); + const page1 = createMockPage({ handlers: [], testStatus: [] }); + const browser = createMockBrowser([createMockContext(page0), createMockContext(page1)]); + vi.mocked(puppeteer.launch).mockResolvedValue(browser); + + await runParallel(baseConfig, '/cwd', []); + + expect(page0.exposeFunction).not.toHaveBeenCalled(); + expect(page1.exposeFunction).not.toHaveBeenCalled(); + }); + + it('feeds merged mocks (with workerIndex) into validateMocks', async () => { + // Drive each worker's exposed __twdCollectMock callback from inside + // that worker's page.evaluate, so it closes over the right closure. + function makePage(workerHandlers, workerTestStatus, mockToCollect) { + const page = { + goto: vi.fn(), + waitForSelector: vi.fn(), + exposeFunction: vi.fn(), + evaluate: vi.fn(), + }; + page.evaluate + .mockImplementationOnce(async () => { + const exposed = page.exposeFunction.mock.calls.find( + (c) => c[0] === '__twdCollectMock' + ); + expect(exposed).toBeDefined(); + const collect = exposed[1]; + await collect(mockToCollect); + return { handlers: workerHandlers, testStatus: workerTestStatus }; + }) + .mockResolvedValueOnce(null); // no coverage + return page; + } + + const page0 = makePage( + [{ id: 't-0', name: 'describe0 > test0', type: 'test' }], + [{ id: 't-0', status: 'pass' }], + { + alias: 'getA', + method: 'GET', + url: '/api/a', + status: 200, + response: 'x', + testId: 't-0', + } + ); + const page1 = makePage( + [{ id: 't-1', name: 'describe1 > test1', type: 'test' }], + [{ id: 't-1', status: 'pass' }], + { + alias: 'getB', + method: 'GET', + url: '/api/b', + status: 200, + response: 'y', + testId: 't-1', + } + ); + const browser = createMockBrowser([createMockContext(page0), createMockContext(page1)]); + vi.mocked(puppeteer.launch).mockResolvedValue(browser); + + const { validateMocks } = await import('../src/contracts.js'); + const { printContractReport } = await import('../src/contractReport.js'); + let capturedMocks; + vi.mocked(validateMocks).mockImplementation((mocks) => { + capturedMocks = mocks; + return { results: [], skipped: [] }; + }); + vi.mocked(printContractReport).mockReturnValue(false); + + await runParallel( + { ...baseConfig, contracts: [{ source: './openapi.json' }] }, + '/cwd', + [{ /* sentinel */ }] + ); + + expect(capturedMocks).toBeDefined(); + const entries = Array.from(capturedMocks.values()); + expect(entries).toHaveLength(2); + const aliases = entries.map((e) => e.alias).sort(); + expect(aliases).toEqual(['getA', 'getB']); + const workerIndices = entries.map((e) => e.workerIndex).sort(); + expect(workerIndices).toEqual([0, 1]); + }); + + it('returns true when contract errors are printed', async () => { + const page0 = createMockPage({ handlers: [], testStatus: [] }); + const page1 = createMockPage({ handlers: [], testStatus: [] }); + const browser = createMockBrowser([createMockContext(page0), createMockContext(page1)]); + vi.mocked(puppeteer.launch).mockResolvedValue(browser); + + const { validateMocks } = await import('../src/contracts.js'); + const { printContractReport } = await import('../src/contractReport.js'); + vi.mocked(validateMocks).mockReturnValue({ results: [], skipped: [] }); + vi.mocked(printContractReport).mockReturnValue(true); // simulate errors + + const hasFailures = await runParallel( + { ...baseConfig, contracts: [{ source: './openapi.json' }] }, + '/cwd', + [{ /* sentinel */ }] + ); + + expect(hasFailures).toBe(true); + }); +``` + +### Step 2: Run the new tests — expect failure (contracts not yet handled in runParallel) + +```bash +cd /Users/kevinccbsg/brikev/twd-cli +npx vitest run tests/runParallel.test.js +``` + +Expected: the four new tests fail. For example, `expect(page0.exposeFunction).toHaveBeenCalledWith(...)` fails because `runParallel` currently never calls `exposeFunction`. + +### Step 3: Extend `src/runParallel.js` with contract handling + +Edit `/Users/kevinccbsg/brikev/twd-cli/src/runParallel.js`. Replace the file with: + +```javascript +import fs from 'fs'; +import path from 'path'; +import puppeteer from 'puppeteer'; +import { reportResults } from 'twd-js/runner-ci'; +import { validateMocks } from './contracts.js'; +import { printContractReport } from './contractReport.js'; +import { generateContractMarkdown } from './contractMarkdown.js'; +import { buildTestPath } from './buildTestPath.js'; +import { mergeMocks } from './mergeMocks.js'; + +const WORKERS = 2; + +const ANTI_THROTTLE_FLAGS = [ + '--disable-background-timer-throttling', + '--disable-renderer-backgrounding', + '--disable-backgrounding-occluded-windows', +]; + +function mergeArgs(userArgs, extras) { + const merged = [...userArgs]; + for (const flag of extras) { + if (!merged.includes(flag)) merged.push(flag); + } + return merged; +} + +function makeMockCollector(workerMocks, workerCounters) { + return (mock) => { + const occKey = `${mock.alias}:${mock.testId}`; + const count = (workerCounters.get(occKey) || 0) + 1; + workerCounters.set(occKey, count); + const dedupKey = `${mock.method}:${mock.url}:${mock.status}:${mock.testId}:${count}`; + workerMocks.set(dedupKey, { ...mock, occurrence: count }); + }; +} + +async function runWorker(browser, workerIndex, config, workingDir, contractsConfigured, workerMocks, workerCounters) { + const ctx = await browser.createBrowserContext(); + const page = await ctx.newPage(); + + if (contractsConfigured) { + await page.exposeFunction( + '__twdCollectMock', + makeMockCollector(workerMocks, workerCounters) + ); + } + + await page.goto(config.url); + await page.waitForSelector('#twd-sidebar-root', { timeout: config.timeout }); + + const { handlers, testStatus } = await page.evaluate( + async (workerIndex, N, retryCount) => { + const allIds = Array.from(window.__TWD_STATE__.handlers.values()) + .filter((h) => h.type === 'test') + .map((h) => h.id); + const myIds = allIds.filter((_, idx) => idx % N === workerIndex); + + const TestRunner = window.__testRunner; + const testStatus = []; + const runner = new TestRunner( + { + onStart: (test) => { test.status = 'running'; }, + onPass: (test, retryAttempt) => { + test.status = 'done'; + const entry = { id: test.id, status: 'pass' }; + if (retryAttempt !== undefined) entry.retryAttempt = retryAttempt; + testStatus.push(entry); + }, + onFail: (test, err) => { + test.status = 'done'; + testStatus.push({ + id: test.id, + status: 'fail', + error: `${err.message} (at ${window.location.href})`, + }); + }, + onSkip: (test) => { + test.status = 'done'; + testStatus.push({ id: test.id, status: 'skip' }); + }, + }, + { retryCount } + ); + const handlers = await runner.runByIds(myIds); + return { handlers: Array.from(handlers.values()), testStatus }; + }, + workerIndex, + WORKERS, + config.retryCount + ); + + if (config.coverage) { + const coverage = await page.evaluate(() => window.__coverage__); + if (coverage) { + const nycDir = path.resolve(workingDir, config.nycOutputDir); + const outPath = path.join(nycDir, `out-${workerIndex}.json`); + fs.writeFileSync(outPath, JSON.stringify(coverage)); + console.log(`Worker ${workerIndex}: coverage → ${outPath}`); + } else { + console.log(`Worker ${workerIndex}: no __coverage__ on window`); + } + } + + await ctx.close(); + return { workerIndex, handlers, testStatus }; +} + +export async function runParallel(config, workingDir, contractValidators) { + let browser; + try { + console.log(`Starting TWD test runner (parallel mode, ${WORKERS} workers)...`); + console.log('Configuration:', JSON.stringify(config, null, 2)); + + const contractsConfigured = config.contracts && config.contracts.length > 0; + const workerMocks = Array.from({ length: WORKERS }, () => new Map()); + const workerCounters = Array.from({ length: WORKERS }, () => new Map()); + + if (config.coverage) { + const nycDir = path.resolve(workingDir, config.nycOutputDir); + if (fs.existsSync(nycDir)) { + fs.rmSync(nycDir, { recursive: true, force: true }); + } + fs.mkdirSync(nycDir, { recursive: true }); + } + + browser = await puppeteer.launch({ + headless: config.headless, + args: mergeArgs(config.puppeteerArgs, ANTI_THROTTLE_FLAGS), + }); + + console.time('Parallel test time'); + const workerResults = await Promise.all( + Array.from({ length: WORKERS }, (_, i) => + runWorker( + browser, + i, + config, + workingDir, + contractsConfigured, + workerMocks[i], + workerCounters[i] + ) + ) + ); + console.timeEnd('Parallel test time'); + + let totalPass = 0; + let totalFail = 0; + let totalSkip = 0; + for (const { workerIndex, handlers, testStatus } of workerResults) { + console.log(`\n────── Worker ${workerIndex} results ──────`); + reportResults(handlers, testStatus); + const pass = testStatus.filter((s) => s.status === 'pass').length; + const fail = testStatus.filter((s) => s.status === 'fail').length; + const skip = testStatus.filter((s) => s.status === 'skip').length; + console.log( + `Worker ${workerIndex}: ${pass} passed, ${fail} failed, ${skip} skipped` + ); + totalPass += pass; + totalFail += fail; + totalSkip += skip; + } + + console.log(`\n────── Summary ──────`); + console.log( + `Total: ${totalPass} passed, ${totalFail} failed, ${totalSkip} skipped` + ); + + let hasFailures = totalFail > 0; + + if (contractsConfigured) { + const merged = mergeMocks(workerMocks); + + // Enrich each mock with testName using its source worker's handler tree. + for (const [, mock] of merged) { + if (mock.testId) { + const workerHandlers = workerResults[mock.workerIndex].handlers; + mock.testName = buildTestPath(mock.testId, workerHandlers); + } + } + + if (merged.size === 0) { + console.log('\nNo mocks collected — ensure twd-js supports contract collection'); + } + const validationOutput = validateMocks(merged, contractValidators); + const hasContractErrors = printContractReport(validationOutput); + if (hasContractErrors) hasFailures = true; + + if (config.contractReportPath) { + const reportPath = path.resolve(workingDir, config.contractReportPath); + const reportDir = path.dirname(reportPath); + if (!fs.existsSync(reportDir)) { + fs.mkdirSync(reportDir, { recursive: true }); + } + const markdown = generateContractMarkdown(validationOutput); + fs.writeFileSync(reportPath, markdown); + console.log(`Contract report written to ${config.contractReportPath}`); + } + } + + await browser.close(); + console.log('Browser closed.'); + return hasFailures; + } catch (error) { + console.error('Error running tests (parallel):', error); + if (browser) await browser.close(); + throw error; + } +} +``` + +### Step 4: Run the tests — expect all `runParallel` tests pass + +```bash +cd /Users/kevinccbsg/brikev/twd-cli +npx vitest run tests/runParallel.test.js +``` + +Expected: all tests pass (core tests from Task 3 + 4 contract tests from this task). + +### Step 5: Run the full suite — no regressions + +```bash +cd /Users/kevinccbsg/brikev/twd-cli +npx vitest run +``` + +Expected: every test passes. + +### Step 6: Stage and propose commit (DO NOT COMMIT) + +```bash +cd /Users/kevinccbsg/brikev/twd-cli +git add src/runParallel.js tests/runParallel.test.js +``` + +Proposed commit message: +``` +feat(runParallel): wire contract mock collection across workers + +Each worker exposes its own __twdCollectMock via page.exposeFunction, +writing into a per-worker Map. After both workers complete, mergeMocks +combines them with worker-indexed keys. Each mock carries workerIndex +so buildTestPath can pick the correct handler tree for testName +resolution. Validation and markdown reporting reuse the existing +serial pipeline unchanged. +``` + +Wait for user approval. + +--- + +## Task 5: Wire `runParallel` into `src/index.js` + +**Files:** +- Modify: `/Users/kevinccbsg/brikev/twd-cli/src/index.js` +- Modify: `/Users/kevinccbsg/brikev/twd-cli/tests/runTests.test.js` + +Adds the thin `if (config.parallel)` branch at the top of `runTests()`. The existing serial body remains textually below, unchanged. + +### Step 1: Add a test in `tests/runTests.test.js` asserting serial path is NOT entered when `parallel: true` + +Append to `/Users/kevinccbsg/brikev/twd-cli/tests/runTests.test.js` inside the `describe('runTests', ...)` block: + +```javascript + it("delegates to runParallel and does NOT launch a single-page serial flow when parallel=true", async () => { + // When parallel mode is on, runTests should call runParallel (which + // itself launches puppeteer with createBrowserContext). The serial code + // path uses browser.newPage() on the default context. If parallel + // dispatch works, page.evaluate should never be called via the serial + // newPage() path — evidenced by puppeteer.launch being invoked exactly + // once with args including the anti-throttle flags. + const browser = { + createBrowserContext: vi.fn().mockImplementation(() => ({ + newPage: vi.fn().mockResolvedValue({ + goto: vi.fn(), + waitForSelector: vi.fn(), + exposeFunction: vi.fn(), + evaluate: vi.fn() + .mockResolvedValueOnce({ handlers: [], testStatus: [] }) + .mockResolvedValueOnce(null), + }), + close: vi.fn(), + })), + newPage: vi.fn(), // serial path would call this — we assert it was NOT called + close: vi.fn(), + }; + vi.mocked(puppeteer.launch).mockResolvedValue(browser); + vi.mocked(loadConfig).mockReturnValue({ + ...defaultMockConfig, + parallel: true, + }); + + await runTests(); + + expect(browser.newPage).not.toHaveBeenCalled(); + expect(browser.createBrowserContext).toHaveBeenCalledTimes(2); + + const launchArgs = vi.mocked(puppeteer.launch).mock.calls[0][0].args; + expect(launchArgs).toContain('--disable-renderer-backgrounding'); + }); + + it("runs the serial path when parallel is absent (default false)", async () => { + const testStatus = [{ id: '1', status: 'pass' }]; + const handlers = [{ id: '1', name: 'test1', type: 'test' }]; + const page = createMockPage({ handlers, testStatus }); + const browser = createMockBrowser(page); + vi.mocked(puppeteer.launch).mockResolvedValue(browser); + // defaultMockConfig has no `parallel` field — absent should mean serial. + + await runTests(); + + expect(browser.newPage).toHaveBeenCalled(); + // Anti-throttle flags are a parallel-only behavior — NOT in the serial launch. + const launchArgs = vi.mocked(puppeteer.launch).mock.calls[0][0].args; + expect(launchArgs).not.toContain('--disable-renderer-backgrounding'); + }); +``` + +### Step 2: Run the new tests — expect failures + +```bash +cd /Users/kevinccbsg/brikev/twd-cli +npx vitest run tests/runTests.test.js +``` + +Expected: the two new tests fail. The first one fails because `runTests` ignores the `parallel` flag — the serial path runs and never calls `createBrowserContext`. The second passes incidentally (no change there) — but run both to confirm the first is the only failure. + +### Step 3: Modify `src/index.js` to add the branch + +Edit `/Users/kevinccbsg/brikev/twd-cli/src/index.js`. Insert the parallel import at the top of the import block: + +```javascript +import { runParallel } from './runParallel.js'; +``` + +And change the body of `runTests` so the parallel branch runs first. Replace: + +```javascript +export async function runTests() { + let browser; + try { + const config = loadConfig(); + const workingDir = process.cwd(); + + console.log('Starting TWD test runner...'); + console.log('Configuration:', JSON.stringify(config, null, 2)); + + // Load contract validators if configured + let contractValidators = []; + if (config.contracts && config.contracts.length > 0) { + contractValidators = await loadContracts(config.contracts, workingDir); + } + + browser = await puppeteer.launch({ +``` + +with: + +```javascript +export async function runTests() { + const config = loadConfig(); + const workingDir = process.cwd(); + + // Parallel mode — delegate early. Serial body below is unchanged. + if (config.parallel) { + let contractValidators = []; + if (config.contracts && config.contracts.length > 0) { + contractValidators = await loadContracts(config.contracts, workingDir); + } + return runParallel(config, workingDir, contractValidators); + } + + let browser; + try { + console.log('Starting TWD test runner...'); + console.log('Configuration:', JSON.stringify(config, null, 2)); + + // Load contract validators if configured + let contractValidators = []; + if (config.contracts && config.contracts.length > 0) { + contractValidators = await loadContracts(config.contracts, workingDir); + } + + browser = await puppeteer.launch({ +``` + +Leave everything below this point in the file unchanged. + +### Step 4: Run the new tests — expect pass + +```bash +cd /Users/kevinccbsg/brikev/twd-cli +npx vitest run tests/runTests.test.js +``` + +Expected: all `runTests` tests pass, including the two new ones from Step 1. + +### Step 5: Run the full suite — no regressions + +```bash +cd /Users/kevinccbsg/brikev/twd-cli +npx vitest run +``` + +Expected: every test in the suite passes. + +### Step 6: Stage and propose commit (DO NOT COMMIT) + +```bash +cd /Users/kevinccbsg/brikev/twd-cli +git add src/index.js tests/runTests.test.js +``` + +Proposed commit message: +``` +feat(index): delegate to runParallel when config.parallel is true + +Adds an early-return branch at the top of runTests(): if +config.parallel is truthy, load contract validators and hand off to +runParallel. Serial code path below is textually unchanged and runs +when parallel is absent or false. +``` + +Wait for user approval. + +--- + +## Task 6: Manual smoke acceptance + README update + +**Files:** +- Modify: `/Users/kevinccbsg/brikev/twd-cli/README.md` + +This task runs a real parallel run end-to-end and documents the feature. No code changes to `src/`. + +### Step 1: Serial baseline against test-example-app + +In a separate terminal, start the dev server and leave it running: +```bash +cd /Users/kevinccbsg/brikev/twd-cli/test-example-app +npm run dev +``` + +In a second terminal: +```bash +cd /Users/kevinccbsg/brikev/twd-cli/test-example-app +rm -rf .nyc_output +npx twd-cli run +``` + +Record: total test count, pass/fail/skip counts, wallclock `Total Test Time`. Expect all green (contract warnings ok — mode is `warn`). + +### Step 2: Enable parallel and re-run + +Edit `/Users/kevinccbsg/brikev/twd-cli/test-example-app/twd.config.json` to add one field: + +```jsonc +{ + "url": "http://localhost:5173", + "parallel": true, // NEW — opt-in + "contractReportPath": ".twd/contract-report.md", + "contracts": [ ... ] +} +``` + +Then: +```bash +cd /Users/kevinccbsg/brikev/twd-cli/test-example-app +rm -rf .nyc_output +npx twd-cli run +``` + +Expected output shape: +``` +Starting TWD test runner (parallel mode, 2 workers)... +... +────── Worker 0 results ────── + +Worker 0: X passed, 0 failed, 0 skipped +────── Worker 1 results ────── + +Worker 1: Y passed, 0 failed, 0 skipped +────── Summary ────── +Total: (X+Y) passed, 0 failed, 0 skipped +Parallel test time: +[contract report block — same lines as serial, counts may differ by ±1 because the collected occurrence ordering differs] +``` + +Check: +- Total pass count equals serial baseline. +- `Parallel test time` is less than serial's `Total Test Time`. +- Exit code 0 (run `echo $?` immediately after). + +**Note on coverage for this target app**: `test-example-app` does not currently have istanbul instrumentation wired up. The script will log `Worker N: no __coverage__ on window` for each worker, and `.nyc_output/` will be created but contain no `out-.json` files. That is expected and out of scope for this feature. The coverage path is unit-tested; real instrumentation is a separate change to `test-example-app/vite.config.ts`. + +### Step 3: Revert the config change on test-example-app + +```bash +cd /Users/kevinccbsg/brikev/twd-cli +git checkout test-example-app/twd.config.json +``` + +We do NOT want to ship `parallel: true` as the default for the example app — the feature is opt-in. + +### Step 4: Add a README section documenting the new field + +Edit `/Users/kevinccbsg/brikev/twd-cli/README.md`. Find the existing configuration section (search for `twd.config.json` or the `retryCount` docs). Add a new subsection: + +```markdown +### Parallel Test Execution (Experimental) + +Set `parallel: true` in `twd.config.json` to run tests across two isolated +Puppeteer browser contexts in parallel. On a typical developer laptop this +halves wallclock test time for suites over ~15 seconds. The feature is +opt-in; default remains `false` and existing behavior is unchanged. + +```jsonc +{ + "url": "http://localhost:5173", + "parallel": true, + "retryCount": 2 +} +``` + +**Notes:** +- Worker count is currently fixed at 2. Higher counts need tuning (see + `poc/parallel/README.md` for the concurrency-ceiling findings). +- Coverage writes one file per worker to `.nyc_output/out-.json`. + `npx nyc report` merges them automatically — no new tooling required. +- Contract validation works unchanged. Mocks are collected per worker and + merged before validation. +- Existing `retryCount` is honored per worker; timing-sensitive tests + (e.g. `waitFor` with short timeouts) may be flakier under CPU contention + — retries absorb this on most CI runners. +``` + +### Step 5: Stage and propose commit (DO NOT COMMIT) + +```bash +cd /Users/kevinccbsg/brikev/twd-cli +git add README.md +``` + +Proposed commit message: +``` +docs: document opt-in parallel test execution in README + +Adds a section covering the new `parallel: true` config field, the +expected speedup, and the current N=2 limitation. +``` + +Wait for user approval. + +--- + +## Completion + +Once Tasks 1-6 are all staged and (per user direction) committed: + +1. The production feature is live on `feat/parallel-execution`. +2. Serial path is byte-identical when `parallel` is absent or `false`. +3. Parallel path at N=2 gives ~1.5-1.8× speedup with full retries, coverage, and contract support. +4. Known follow-ups (tracked in the spec): + - Deterministic test IDs in `twd-js` (unblocks unified reporting and cross-machine sharding). + - Configurable `workers: N` once deterministic IDs land. + - Canonical-path-merged unified reporting tree. + - Istanbul instrumentation for `test-example-app` (separate concern from this feature). + +Manual-verify commands for the release notes: + +```bash +# Regression check: serial output byte-identical when parallel absent +npx twd-cli run # existing behavior, unchanged + +# Feature check: parallel mode +echo '{"parallel": true}' > twd.config.local.json # or edit existing file +npx twd-cli run # new output shape, ~50% wallclock + +# Coverage merge (on an instrumented app) +npx nyc report --reporter=text +``` + +The feature is merge-ready when all unit tests pass, manual smoke shows +equal pass/fail counts vs serial, and the README documents the new flag. diff --git a/docs/superpowers/specs/2026-04-21-parallel-test-execution-poc-design.md b/docs/superpowers/specs/2026-04-21-parallel-test-execution-poc-design.md new file mode 100644 index 0000000..87cbb77 --- /dev/null +++ b/docs/superpowers/specs/2026-04-21-parallel-test-execution-poc-design.md @@ -0,0 +1,232 @@ +# twd-cli Parallel Test Execution POC — Design Spec + +**Date:** 2026-04-21 +**Status:** Proposed +**Type:** Proof of concept (throwaway / experimental) + +## Purpose + +Prove the viability of running twd-cli's browser-based test suite in parallel across multiple isolated browser contexts, without adding any new third-party library (no `puppeteer-cluster` or similar). + +The core risky assumption is: **Puppeteer's `browser.createBrowserContext()` truly isolates service worker registrations** between contexts, which is required because TWD tests register mocks through a service worker (MSW-style). If two parallel workers each mocked the same URL with different responses, cross-context SW leakage would corrupt results. + +The POC answers three yes/no questions: + +- **(a)** Do incognito-style browser contexts isolate the service worker / mock registrations? +- **(b)** Can we split `window.__coverage__` into per-worker files that merge cleanly with `nyc`? +- **(c)** Does the full `test-example-app` test suite pass with identical pass/fail counts when split across N workers? + +Wallclock speedup is explicitly out of scope for viability. If the POC passes, a production feature will be designed in a follow-up spec. + +## Scope + +**In scope:** +- Standalone Node script under `twd-cli/poc/parallel/` that drives parallel execution against `test-example-app/`. +- Single `puppeteer.launch()` with N isolated `browser.createBrowserContext()` workers. +- Round-robin distribution of test IDs across workers, invoked through the existing `window.__testRunner.runByIds()` API. +- Per-worker coverage dump to `.nyc_output/out-.json`. +- Findings documented in a `README.md` alongside the POC script. + +**Out of scope:** +- Any change to `src/index.js`, `src/config.js`, or the published `runTests()` API. +- CLI flags or `twd.config.json` options for parallelization. +- Contract validation and `__twdCollectMock` plumbing in parallel mode. +- Retry logic in parallel mode (`retryCount` is not honored by the POC). +- Pretty tree reporting; the POC prints pass/fail counts per worker and overall. +- Wallclock benchmarking (user may run it informally, but not a gate). +- Cross-machine CI sharding. +- Automatic nyc merging — the user runs `npx nyc report` themselves. + +## Approach + +**Single Puppeteer browser, N isolated browser contexts** (equivalent to N parallel incognito windows). + +Rejected alternatives: +- *N separate `puppeteer.launch()` instances* — heavier, slower startup, overkill for the POC. Kept in mind as a fallback if context isolation fails (see Failure Modes). +- *`child_process.fork` copies of the existing CLI* — maximum code reuse but much more orchestration, and spawns N Chrome processes anyway. More appropriate for the eventual production feature. + +## File Layout + +``` +twd-cli/ +├── poc/ +│ └── parallel/ +│ ├── README.md # how to run, findings log +│ └── run-parallel.js # the POC script (ESM, ~150 LoC) +└── test-example-app/ # the target app, unchanged +``` + +- `run-parallel.js` is runnable from `test-example-app/` as `node ../poc/parallel/run-parallel.js [N]`. +- No package.json changes, no new dependencies. Imports `puppeteer` from the already-present `twd-cli/node_modules`. +- Checked in so findings are preserved; not exported from the package and not referenced by `bin/twd-cli.js`. + +## Execution Flow + +User command: + +```bash +# from test-example-app/ (with `npm run dev` already running on port 5173) +node ../poc/parallel/run-parallel.js 2 +``` + +Argument: number of workers (default `2`). + +``` +1. Parse N from process.argv[2] (default 2). + +2. Clear .nyc_output/ (rm -rf, then mkdir -p) so stale serial runs + don't pollute the merge. + +3. puppeteer.launch({ headless: true, args: ['--no-sandbox', '--disable-setuid-sandbox'] }) + +4. Worker 0 (serves double duty as probe + worker): + a. browser.createBrowserContext() → context0 + b. context0.newPage() → page0 + c. page0.goto(URL, { waitUntil: 'networkidle0' }) + d. page0.waitForSelector('#twd-sidebar-root', { timeout: 10000 }) + e. page0.evaluate → read test IDs: + Array.from(window.__TWD_STATE__.handlers.values()) + .filter(h => h.type === 'test') + .map(h => h.id) + f. Split IDs into N chunks via round-robin (see Test Distribution). + +5. Promise.all of N tasks — worker 0 continues on its existing page, + workers 1..N-1 create their own context + page from scratch: + + • Worker 0: page0.evaluate(chunks[0]) → runByIds → collect coverage + • Worker i (i > 0): + - browser.createBrowserContext() → contextI + - contextI.newPage() → pageI + - pageI.goto(URL), waitForSelector('#twd-sidebar-root') + - pageI.evaluate(chunks[i]) → runByIds → collect coverage + +6. Each worker, on completion: + - fs.writeFileSync(`.nyc_output/out-${i}.json`, JSON.stringify(coverage)) + regardless of pass/fail (see Coverage Output). + - contextI.close() + +7. Aggregate pass/fail counts. Print: + Worker 0: 27 passed, 0 failed + Worker 1: 28 passed, 0 failed + Total: 55 passed, 0 failed (expected 55) + +8. browser.close(). Exit code = anyWorkerFailed ? 1 : 0. +``` + +## Test Distribution + +Round-robin by index: + +```js +const chunks = Array.from({ length: N }, () => []); +testIds.forEach((id, i) => chunks[i % N].push(id)); +``` + +Chosen over contiguous chunking because `test-example-app/src/App.twd.test.ts` clusters related tests inside `describe()` blocks (e.g., all Products contract-mismatch tests are contiguous). Contiguous chunks would pile entire suites onto one worker and leave another doing only render tests; round-robin spreads any systematic slowness or state across workers evenly. + +### Why splitting suites across workers is safe + +`window.__testRunner.runByIds(ids)` (defined in `twd/src/runner.ts:284`) already handles partial execution correctly: + +- `collectHooks(test.parent!)` in `runner.ts:195-205` walks up the suite tree from each test's direct parent and collects all `beforeEach` / `afterEach` hooks along the way, **per test**. +- Siblings of the test are irrelevant to its hook chain. If "test A" and "test B" live in the same `describe()` but go to different workers, both still get the same `beforeEach` invocations, because each worker re-walks the tree independently. +- TWD has no `beforeAll` / `afterAll` hooks (only `beforeEach` / `afterEach`), so there are no "run-once-per-suite" semantics that splitting could break. +- Every worker loads the full app bundle, so each worker's `__TWD_STATE__` contains the complete suite tree and all hook registrations. We only tell `runByIds` *which tests to run*, not which suites to build. + +The `stack` array in `__TWD_STATE__` is used transiently during test registration (to track current `describe` nesting while the DSL builds the tree). By the time tests run, it is empty — not relevant to parallelization. + +## Coverage Output + +Each worker writes its own file: + +``` +test-example-app/ +└── .nyc_output/ + ├── out-0.json ← worker 0 + ├── out-1.json ← worker 1 + └── out-.json ← worker i +``` + +Inside each worker, after `runByIds` completes: + +```js +const coverage = await page.evaluate(() => window.__coverage__); +if (coverage) { + fs.writeFileSync( + path.join(nycDir, `out-${workerIndex}.json`), + JSON.stringify(coverage) + ); +} +``` + +### Differences from the serial CLI + +- **Always dump, even on failure.** `src/index.js:136` skips coverage when `hasFailures` is true. The POC inverts this — every worker dumps unconditionally — so a single flaky test doesn't blind us to the other workers' behavior, and we can confirm criterion (c) per worker. +- **Clean `.nyc_output/` at startup.** `fs.rmSync(nycDir, { recursive: true, force: true })` then `fs.mkdirSync(nycDir, { recursive: true })` before step 4 — prevents stale files from a previous serial run from bleeding into the merge. + +### Merging (user-side) + +nyc automatically picks up all `*.json` files in `.nyc_output/` and unions the hit counts. After the POC runs, the user verifies: + +```bash +npx nyc report --reporter=text +# or full HTML: +npx nyc report --reporter=html +``` + +No merging logic inside the POC itself. If merging turned out to be fiddly (it shouldn't be — this is `nyc`'s designed behavior), that would be recorded as a failure mode. + +## Success Criteria & Verification + +### Baseline capture + +Before running the POC, capture the serial baseline from the existing CLI: + +```bash +cd test-example-app +rm -rf .nyc_output +npx twd-cli run | tee /tmp/baseline.log +npx nyc report --reporter=text | tee /tmp/baseline-cov.log +``` + +Note: total test count, pass count, fail count, total covered %. + +### Parallel run + +```bash +rm -rf .nyc_output +node ../poc/parallel/run-parallel.js 2 | tee /tmp/parallel.log +npx nyc report --reporter=text | tee /tmp/parallel-cov.log +``` + +### Pass conditions + +**(c) Tests run to completion** — pass/fail breakdown in `parallel.log` matches `baseline.log`. Same total count, same number of passes, same number of failures. If the parallel run is missing tests, something is being skipped or the distribution logic is dropping IDs. + +**(a) Service-worker isolation** — because the existing suite has multiple tests mocking overlapping URLs (e.g., `/api/products` across Products-valid and Products-mismatches describes), round-robin splitting at N=2 forces those concurrent overlapping mocks across different contexts. If criterion (c) holds, (a) holds implicitly — if contexts shared SW state, overlapping mocks would contaminate each other and contract-validation tests would fail differently from serial. +As a cheap guard against accidentally assigning the same chunk twice, the POC logs each worker's `[index, firstTestId, lastTestId, chunkLength]` before running. Overlapping IDs across workers would indicate a distribution bug, not an isolation bug. + +**(b) Coverage split works** — both `.nyc_output/out-0.json` and `.nyc_output/out-1.json` exist and are non-empty, `npx nyc report` runs without error, and the merged totals are close to the serial baseline. They won't be identical — execution order differs slightly — but file count and overall % should be in the same ballpark (within a percentage point or two). + +## Failure Modes + +If the POC fails, the `README.md` records which mode and we adapt: + +- **F1 — tests fail under parallel that pass serially.** SW isolation broken across `browser.createBrowserContext()`. Fallback: swap to N separate `puppeteer.launch()` instances (same script shape, different launch call). Document the evidence so we know why we took the heavier path. +- **F2 — coverage files exist but `nyc report` errors or produces garbage.** Investigate how `window.__coverage__` is keyed and whether parallel contexts clobber each other. May need per-worker coverage dir rather than per-worker file. +- **F3 — workers hang or time out.** Renderer-pool contention, or something in Puppeteer's context model we don't understand. Try lower N, try `headless: false` for visual inspection. +- **F4 — works but no wallclock speedup.** Explicitly NOT a POC failure per criterion scope, but worth noting in findings so the production feature design knows to investigate (renderer pool sizing, CPU pinning, etc.). + +## Findings Artifact + +`poc/parallel/README.md` becomes the write-up. At minimum it includes: + +- The command(s) used. +- Values of N tested. +- Baseline vs parallel pass/fail counts. +- Baseline vs parallel coverage percentages. +- Observed wallclock times (informational). +- Which criteria (a) / (b) / (c) passed. +- Any anomalies, flaky tests, or unexpected behavior. + +This becomes the input to the production feature spec if the POC passes. diff --git a/docs/superpowers/specs/2026-04-21-parallel-test-execution-production-design.md b/docs/superpowers/specs/2026-04-21-parallel-test-execution-production-design.md new file mode 100644 index 0000000..c557c39 --- /dev/null +++ b/docs/superpowers/specs/2026-04-21-parallel-test-execution-production-design.md @@ -0,0 +1,290 @@ +# twd-cli Parallel Test Execution (Production) — Design Spec + +**Date:** 2026-04-21 +**Status:** Proposed +**Depends on:** `poc/parallel/` findings (proved SW isolation, coverage split, 1.83× speedup at N=2) + +## Purpose + +Ship an opt-in `parallel: true` mode in `twd-cli` that runs a project's TWD test suite across two isolated Puppeteer browser contexts concurrently, cutting wallclock test time to roughly half. Zero regression risk for existing users — the serial code path is preserved byte-for-byte and `parallel: false` remains the default. + +The POC at `poc/parallel/` validated the three risky assumptions: `browser.createBrowserContext()` isolates service-worker registrations, per-worker `window.__coverage__` dumps merge cleanly via `nyc report`, and the full suite passes at N=2 with a ~1.83× speedup on a developer laptop. This spec turns that throwaway script into a supported feature with proper retries, contract-mock handling, and unit-test coverage. + +## Scope + +**In scope:** +- New boolean `parallel` field in `twd.config.json` (default `false`). +- New module `src/runParallel.js` implementing the parallel execution path. +- New utility `src/mergeMocks.js` for combining contract mocks collected across workers. +- `src/index.js` branches on `config.parallel` and delegates when true. Serial path untouched. +- Anti-throttle Chromium flags automatically added to the launch arguments. +- Existing `config.retryCount` honored per worker via `TestRunner`'s built-in retry loop. +- Existing contract validation pipeline runs unchanged after a post-workers mock merge step. +- Per-worker coverage dumps to `/out-.json`, always (even on failures). +- Unit tests in `tests/runParallel.test.js` and `tests/mergeMocks.test.js`. + +**Out of scope (explicitly deferred):** +- User-configurable worker count. `N` is hardcoded to `2` in this release. +- Unified reporting tree across workers — results print as two separate trees plus a summary. A follow-up spec can build a canonical-path-merged tree. +- Cross-machine / cross-CI-job sharding. +- Changes to twd-js test ID generation (random → deterministic). Separate spec. +- Per-test or per-worker timeout overrides. +- Dynamic `N` based on `os.cpus()` detection. +- Starting the dev server from inside the runner. + +## Dependencies + +No new runtime dependencies. Reuses the already-bundled `puppeteer` (^24.42.0) and `twd-js` (^1.7.2). + +## Config + +New optional boolean field in `twd.config.json`: + +```jsonc +{ + "url": "http://localhost:5173", + "parallel": true, // NEW — defaults to false; opt-in + "retryCount": 2, // existing; applied inside each worker + "contracts": [ ... ], // existing; mocks merged across workers before validation + "coverage": true, // existing; produces per-worker out-.json files + "timeout": 10000, // existing; applied per worker for waitForSelector + "puppeteerArgs": [...], // existing; anti-throttle flags appended if not present + "nycOutputDir": "./.nyc_output", + "coverageDir": "./coverage", + "headless": true, + "contractReportPath": "..." +} +``` + +| Field | Required | Default | Description | +|-------|----------|---------|-------------| +| `parallel` | No | `false` | When `true`, tests run in two isolated browser contexts concurrently. | + +All other fields keep their existing semantics. In particular, `retryCount: 2` (the existing default) continues to apply — each worker's in-browser `TestRunner` retries its own failing tests up to `retryCount` times before declaring them failed, exactly as it does in serial mode. + +## Architecture + +### File layout + +``` +src/ +├── index.js # runTests() — thin branch on config.parallel +├── config.js # unchanged except DEFAULT_CONFIG.parallel = false +├── runParallel.js # NEW — parallel orchestration (~200 LoC) +├── mergeMocks.js # NEW — merges per-worker mock maps (~30 LoC) +├── contracts.js # unchanged +├── contractReport.js # unchanged +├── contractMarkdown.js # unchanged +├── buildTestPath.js # unchanged +└── formatMockLabel.js # unchanged + +tests/ +├── runTests.test.js # existing serial tests — unchanged +├── runParallel.test.js # NEW — mirrors runTests.test.js; mocks puppeteer +└── mergeMocks.test.js # NEW — pure-function unit tests +``` + +Keeping the serial path in `index.js` unchanged makes the diff for this feature minimal, the rollback trivial, and the risk to existing users zero. + +### `runTests()` branch + +```js +// src/index.js (sketch) +import { runParallel } from './runParallel.js'; + +export async function runTests() { + const config = loadConfig(); + const workingDir = process.cwd(); + + let contractValidators = []; + if (config.contracts?.length) { + contractValidators = await loadContracts(config.contracts, workingDir); + } + + if (config.parallel) { + return runParallel(config, workingDir, contractValidators); + } + + // existing serial flow — unchanged + ... +} +``` + +### `runParallel(config, workingDir, contractValidators)` flow + +``` +1. Clean config.nycOutputDir so stale serial out.json doesn't bleed into the merge. +2. puppeteer.launch({ + headless: config.headless, + args: mergeArgs(config.puppeteerArgs, ANTI_THROTTLE_FLAGS), + }) +3. In Promise.all for i in 0..1 (WORKERS = 2): + a. browser.createBrowserContext() → ctx[i] + b. ctx[i].newPage() → page[i] + c. If contracts configured: + page[i].exposeFunction('__twdCollectMock', mockCollectorFor(i)) + where mockCollectorFor(i) writes into workerMocks[i] with a + composite dedup key. + d. page[i].goto(config.url) + page[i].waitForSelector('#twd-sidebar-root', { timeout: config.timeout }) + e. page[i].evaluate(runByIdxModN, { workerIndex: i, N: 2, retryCount: config.retryCount }) + → { status, handlers } + (self-filters inside the page: idx % N === workerIndex) + f. coverage = page[i].evaluate(() => window.__coverage__) + if (coverage) fs.writeFileSync(nycOutputDir/out-i.json, JSON.stringify(coverage)) + g. ctx[i].close() +4. Merge results: + - mergedMocks = mergeMocks(workerMocks) + - Per-worker: reportResults(handlers[i], status[i]) with a "Worker i" header + - Sum pass/fail/skip counts across workers +5. If contractValidators.length: + - Enrich each mock with testName via buildTestPath using the mock's + workerIndex to pick the correct handler tree. + - hasContractErrors = printContractReport(validateMocks(mergedMocks, ...)) + - Optionally write markdown report (contractReportPath) — same as serial. +6. browser.close() +7. Return hasFailures (any test failed OR any contract error). +``` + +### Why self-filter by index (not probe + distribute) + +twd-js generates test IDs with `Math.random()` at module load (`twd/src/runner.ts:52`). IDs are not stable across browser contexts. The POC originally tried to probe one context, split IDs in Node, and pass chunks to each worker — and discovered that chunks from the probe context meant nothing in other contexts (`runByIds` silently matched zero tests). + +Self-filtering inside each worker's `page.evaluate` avoids this entirely: each worker enumerates its own `__TWD_STATE__.handlers` and selects `idx % N === workerIndex`. Registration **order** is stable (same source code, same `describe`/`it` sequence), so the partition is deterministic and disjoint. + +### Anti-throttle launch flags + +```js +const ANTI_THROTTLE_FLAGS = [ + '--disable-background-timer-throttling', + '--disable-renderer-backgrounding', + '--disable-backgrounding-occluded-windows', +]; +``` + +These prevent Chromium from de-prioritizing renderers it considers "hidden" — relevant for headless multi-context runs. In POC testing they materially reduced `waitFor` timeouts at N=2-3. The flags are appended to the user-supplied `puppeteerArgs` only if not already present (user's explicit flag wins). + +## Contract Mock Merging + +Each worker has its own page, therefore its own `exposeFunction` registration, therefore its own Node-side collector closure and Map. After both workers finish running their chunks, `mergeMocks` combines them: + +```js +// src/mergeMocks.js (sketch) +export function mergeMocks(workerMaps) { + const merged = new Map(); + workerMaps.forEach((workerMap, workerIndex) => { + for (const [key, mock] of workerMap) { + // Worker-index prefix guarantees no silent collision if two workers + // happen to generate the same random testId. + merged.set(`w${workerIndex}:${key}`, mock); + } + }); + return merged; +} +``` + +Because round-robin distribution puts each test in exactly one worker, the same test's mocks never appear in both workers' maps. The prefix is defense-in-depth against accidental ID collision from the `Math.random()` ID generator. + +Each mock carries its `workerIndex`, so downstream enrichment (`buildTestPath`) uses the correct worker's handler tree when resolving `testId → testName`. + +`validateMocks(merged, contractValidators)` and `printContractReport(...)` run unchanged — they don't care how the Map was built. + +## Reporting + +Per-worker trees plus a summary — simplest correct thing for v1: + +``` +Starting TWD test runner (parallel mode, 2 workers)... +Configuration: { parallel: true, ... } + +Worker 0: 30/60 tests selected +Worker 1: 30/60 tests selected + +────── Worker 0 results ────── +App Component + ✓ should render the main heading + ✓ should handle button clicks + ... +Worker 0: 30 passed, 0 failed, 0 skipped + +────── Worker 1 results ────── +Contract Validation - Users API + ✓ should mock GET /api/users + ... +Worker 1: 30 passed, 0 failed, 0 skipped + +────── Summary ────── +Total: 60 passed, 0 failed, 0 skipped +Parallel test time: 34.8s +``` + +Decisions: +- Reuse `reportResults(handlers, testStatus)` from `twd-js/runner-ci` per worker with that worker's own handler tree and results. No twd-js changes required. +- Contract report (if configured) prints after the summary, using the merged mock set. +- A unified single tree across workers is a deliberate follow-up. It requires canonicalizing tests by suite-path (because IDs differ per context) and is better solved in a twd-js spec that adds deterministic IDs. + +## Error Handling + +- **Worker rejects (Puppeteer / navigation error)**: `Promise.all` rejects → close browser → exit code 1. No partial recovery. +- **Test failures inside a worker**: captured in that worker's `testStatus`, summed into total fail count → exit code 1. The other worker's results still render. +- **Dev server unreachable**: both workers fail `page.goto` — error logged, exit code 1. +- **`waitForSelector` timeout**: reuse `config.timeout` per worker. A timeout in one worker fails the run. +- **Coverage write fails**: log a warning, do not fail the run. Matches existing serial behavior. +- **Contract validator load fails**: already handled in the shared `loadContracts` path — parallel mode inherits that behavior. + +## Testing + +### `tests/runParallel.test.js` + +Mirrors the shape of the existing `tests/runTests.test.js`. Mocks `puppeteer` and `twd-js/runner-ci`. Asserts: + +- When `config.parallel === false`, `runTests` delegates to the existing serial flow (indirectly — `runParallel` is not called). Existing tests for serial behavior in `runTests.test.js` continue to pass unchanged. +- When `config.parallel === true`: + - `puppeteer.launch` called once with anti-throttle flags in `args`. + - `browser.createBrowserContext` called twice. + - Each context creates a page, calls `goto` with `config.url`, and `waitForSelector('#twd-sidebar-root', { timeout: config.timeout })`. + - `page.evaluate` called with a payload containing `workerIndex` and `N: 2`. + - `config.retryCount` is propagated into the evaluate payload. + - Returned statuses are summed into total pass/fail counts. + - Per-worker coverage files `out-0.json` and `out-1.json` are written when the mocked `__coverage__` is non-null. + - Exit value is `true` (has failures) when any worker reports a `fail`. +- Contract collection path: when `config.contracts` is non-empty, each page calls `exposeFunction('__twdCollectMock', ...)`. A synthetic mock pushed through the mock collector appears in the final merged set passed to `validateMocks`. + +### `tests/mergeMocks.test.js` + +Pure-function tests — no puppeteer mocking needed: +- Empty input → empty map. +- Single worker → all entries preserved, keys prefixed. +- Two workers with disjoint keys → union. +- Two workers that happen to use the same inner key (simulated collision) → no overwrite; both preserved under different prefixed keys. +- `workerIndex` attached to each output mock matches the source worker. + +### Manual acceptance + +Against a known-instrumented target app (or `test-example-app` once it receives `vite-plugin-istanbul`): + +- `parallel: false` produces byte-identical output to pre-feature twd-cli. +- `parallel: true` produces the same pass/fail counts as serial for a clean suite. +- `npx nyc report` on the resulting `.nyc_output/` merges `out-0.json` + `out-1.json` into a coherent report comparable to the serial baseline. +- With contracts configured, the contract report shows the same mock-by-mock breakdown as serial. + +## Success Criteria + +- **Correctness**: `parallel: true` produces the same pass/fail/skip counts as `parallel: false` for any suite that passes serially. +- **Coverage parity**: merged coverage at N=2 matches serial baseline within ±1 percentage point on all four nyc metrics (statements/branches/functions/lines). +- **Speedup**: at N=2 on a 4-core dev machine, wallclock is at most 60 % of serial wallclock for suites over 15 seconds. +- **Zero regression**: with `parallel` absent or `false`, output is byte-for-byte identical to the prior release's output for the same config. +- **Contract behavior parity**: with contracts configured, the printed report and markdown artifact match serial exactly. + +## Rollout Notes + +- Feature is opt-in — no user has to do anything to keep existing behavior. +- Document in `README.md`: new config field, expected speedup range, CI caveat (default `retryCount: 2` recommended; bump `timeout` if running on constrained CI runners). +- Document the known issue that `workers > 2` is not currently exposed and that unified reporting is per-worker until a follow-up release. + +## Follow-ups (tracked separately, not this spec) + +1. **Deterministic test IDs in twd-js** — replace `Math.random()` with a hash of the `describe > it` path. Unblocks unified reporting, cross-machine sharding, and reliable retry-by-id. +2. **Unified reporting tree** — once IDs are deterministic, merge per-worker results into one canonical tree. +3. **Configurable `workers: N`** with auto-detection (`os.cpus().length / 2`), plus per-worker retry and timeout budgets. +4. **Cross-CI-job sharding** — `twd-cli run --shard i/N` for splitting across multiple runners. diff --git a/poc/parallel/README.md b/poc/parallel/README.md new file mode 100644 index 0000000..a2f7710 --- /dev/null +++ b/poc/parallel/README.md @@ -0,0 +1,122 @@ +# Parallel Test Execution POC + +Throwaway script that proves Puppeteer browser contexts isolate service workers and that per-worker coverage files merge cleanly. + +See the approved spec at `docs/superpowers/specs/2026-04-21-parallel-test-execution-poc-design.md`. + +## How to run + +1. In one terminal, start the dev server: + ```bash + cd test-example-app + npm run dev + ``` +2. In another terminal: + ```bash + cd test-example-app + rm -rf .nyc_output + node ../poc/parallel/run-parallel.js 2 # N workers; default 2 + npx nyc report --reporter=text + ``` + +## Design note — test distribution + +Test IDs in twd-js are generated via `Math.random()` at module load time +(`twd/src/runner.ts` → `const generateId = () => Math.random()...`), so they are +NOT stable across browser contexts. Each context sees a fresh bundle load and +therefore fresh IDs. + +The original design called for a "probe context" to enumerate all IDs, split +them in Node, and pass chunks to workers — that failed because chunks from +context 0 meant nothing in contexts 1..N-1 (`runByIds` silently matched +nothing). The POC pivoted mid-implementation to **self-filter inside each +worker**: each worker enumerates its own `__TWD_STATE__.handlers`, takes the +slots where `idx % N === workerIndex`, and calls `runByIds` on those. +Registration **order** (the order `describe`/`it` are called in user code) is +stable across contexts, so the partition is deterministic and disjoint. + +Worth addressing in the production feature: twd-js should move to +deterministic IDs (e.g., hash of the `describe > it` path) to make external +tooling possible. + +## Findings + +**Date run:** 2026-04-21 +**Machine:** macOS 15.5 (Darwin 24.5.0) +**Node:** v24.11.0 +**Puppeteer:** 24.42.0 +**Target app on :5173:** `/Users/kevinccbsg/holafly/web-checkout` (instrumented +via `vite-plugin-istanbul`). 60 TWD tests total. + +### Runs + +| N | Tests | Passed | Failed | Skipped | Wallclock | Stmts % | Branches % | Funcs % | Lines % | +|---|-------|--------|--------|---------|-----------|---------|------------|---------|---------| +| 1 | 60 | 60 | 0 | 0 | 61.4 s | 85.91 | 70.93 | 83.90 | 86.93 | +| 2 | 60 | 60 | 0 | 0 | 34.8 s | 85.91 | 70.93 | 83.90 | 86.93 | +| 3 | 60 | 55 | 5 | 0 | ~29 s | 86.03 | 70.93 | 85.05 | 87.07 | +| 4 | 60 | 52 | 8 | 0 | 23.2 s | ~same | ~same | ~same | ~same | + +(N=1 is a single-worker run through the POC, not the serial `twd-cli run` — +this matches same-code-path baselining.) + +### Criteria + +- **(a) Service-worker isolation — PASS.** At N=2 the full 60-test suite passes + cleanly; coverage % is identical to N=1. The suite includes tests that + register concurrent mocks for the same endpoints under the target app's own + TWD mock bridge — if browser contexts shared service worker state, we would + see mock collisions reflected as test failures and/or diverging coverage. + Neither occurred. `browser.createBrowserContext()` isolates SW registrations + as Chromium's documentation promises. +- **(b) Coverage split — PASS.** `.nyc_output/out-0.json` and + `.nyc_output/out-1.json` land at ~534 KB each covering 83 files. Merged with + `npx nyc report --cwd /path/to/app --temp-dir ./.nyc_output` the result is + coherent: 85.91 % statements at N=1 → 85.91 % at N=2 → 86.03 % at N=3. Small + deltas at higher N reflect different execution orderings exercising slightly + different branches, as expected. +- **(c) Tests run to completion — PASS at N=2.** At N ≥ 3 some `waitFor`-based + tests hit their 1-second rule-execution timeout under CPU contention. See + "Anomalies". + +### Anomalies + +- **Concurrency ceiling at N ≥ 3 on this machine/app.** At N=3, 5 tests fail + with errors like `Rule "createCart" was not executed within 1000ms`. At N=4, + 8 tests fail the same way. The same tests pass reliably at N=2 and in + `twd-cli run` serial. This is **not** service-worker contamination — the + failure mode is a deterministic timeout, not a wrong mocked response. It is + consistent with renderer pool contention: with 3-4 Chrome contexts sharing + the machine's CPU, individual request-interception latencies stretch past + the test's 1-second budget. The production feature should either default to + a conservative N (e.g. `os.cpus().length / 2`) or expose per-test timeout + configuration in `twd.config.json`. +- **Random test IDs.** See "Design note" above. Not a parallelization defect, + but a real obstacle for any future feature that needs to refer to a specific + test from outside the browser (sharding across machines, retry of a single + failure, etc.). Worth fixing in twd-js. +- **Speedup at N=2 ≈ 1.74×** (61.4 s → 34.8 s). About 87 % of the theoretical + 2× max, which is very good and suggests little fixed overhead from the + probe/fan-out machinery. + +### Recommendation + +**Green-light the production feature.** The risky assumption (SW isolation +across browser contexts) is confirmed. Per-worker coverage dumps and the nyc +merge path both work without special handling. Build-out should: + +1. Default `workers` to `os.cpus().length / 2`, capped at 4. Let the user + override via `twd.config.json`. +2. Replace the current `NYC_DIR` hardcoding with a config field, and document + the `nyc report --cwd --temp-dir` pattern for cross-project runs. +3. Fix deterministic test IDs in twd-js in parallel (separate spec) so cross- + context operations stop relying on registration-order coincidence. +4. Consider raising the default `waitFor` timeout or making it per-worker + configurable — the 1-second default is fragile at moderate concurrency. +5. Wire contract-mock collection (`__twdCollectMock`) through per-worker + collection buckets and merge before validation. Currently out of scope for + the POC. + +The POC script in this directory is a throwaway reference; production should +be implemented inside `src/` with proper config, retries, and reporting, not +by productionizing `run-parallel.js`. diff --git a/poc/parallel/run-parallel.js b/poc/parallel/run-parallel.js new file mode 100644 index 0000000..efa5dbb --- /dev/null +++ b/poc/parallel/run-parallel.js @@ -0,0 +1,131 @@ +import fs from 'node:fs'; +import path from 'node:path'; +import puppeteer from 'puppeteer'; + +const URL = 'http://localhost:5173'; +const NYC_DIR = path.resolve(process.cwd(), '.nyc_output'); +const PUPPETEER_ARGS = [ + '--no-sandbox', + '--disable-setuid-sandbox', + // Prevent Chromium from de-prioritizing renderers it thinks are hidden — + // relevant when we run multiple contexts concurrently in headless mode. + '--disable-background-timer-throttling', + '--disable-renderer-backgrounding', + '--disable-backgrounding-occluded-windows', +]; +const TIMEOUT = 10000; + +// Test IDs in twd-js are generated by Math.random() at module load +// (twd/src/runner.ts: `const generateId = () => Math.random()...`), so they +// are NOT stable across browser contexts. Each worker therefore enumerates +// its own context's __TWD_STATE__.handlers and self-selects the slots where +// idx % N === workerIndex. Registration ORDER is stable (source-code order +// of describe/it calls), so the partition is deterministic and disjoint. +async function runWorker(workerIndex, N, page) { + const result = await page.evaluate(async (workerIndex, N) => { + const allIds = Array.from(window.__TWD_STATE__.handlers.values()) + .filter((h) => h.type === 'test') + .map((h) => h.id); + const myIds = allIds.filter((_, idx) => idx % N === workerIndex); + + const TestRunner = window.__testRunner; + const status = []; + const runner = new TestRunner({ + onStart: () => {}, + onPass: (t) => status.push({ id: t.id, status: 'pass' }), + onFail: (t, err) => status.push({ + id: t.id, + status: 'fail', + error: `${err.message} (at ${window.location.href})`, + }), + onSkip: (t) => status.push({ id: t.id, status: 'skip' }), + }); + await runner.runByIds(myIds); + return { status, totalInContext: allIds.length, selected: myIds.length }; + }, workerIndex, N); + + console.log( + `Worker ${workerIndex}: ${result.selected}/${result.totalInContext} tests selected` + ); + + const coverage = await page.evaluate(() => window.__coverage__); + if (coverage) { + const outPath = path.join(NYC_DIR, `out-${workerIndex}.json`); + fs.writeFileSync(outPath, JSON.stringify(coverage)); + console.log(`Worker ${workerIndex}: coverage → ${outPath}`); + } else { + console.log(`Worker ${workerIndex}: no __coverage__ on window`); + } + + return result.status; +} + +async function main() { + const N = parseInt(process.argv[2], 10) || 2; + console.log(`Parallel POC — N=${N}, URL=${URL}`); + + fs.rmSync(NYC_DIR, { recursive: true, force: true }); + fs.mkdirSync(NYC_DIR, { recursive: true }); + + const browser = await puppeteer.launch({ + headless: true, + args: PUPPETEER_ARGS, + }); + + try { + console.time('Parallel test time'); + + const workerPromises = []; + for (let i = 0; i < N; i++) { + const workerIndex = i; + workerPromises.push((async () => { + const ctx = await browser.createBrowserContext(); + const page = await ctx.newPage(); + await page.goto(URL); + await page.waitForSelector('#twd-sidebar-root', { timeout: TIMEOUT }); + const result = await runWorker(workerIndex, N, page); + await ctx.close(); + return result; + })()); + } + + const results = await Promise.all(workerPromises); + console.timeEnd('Parallel test time'); + + results.forEach((status, i) => { + const failures = status.filter((s) => s.status === 'fail'); + if (failures.length > 0) { + console.log(`\nWorker ${i} failures:`); + failures.forEach((f) => console.log(` [${f.id}] ${f.error}`)); + } + }); + + let totalPass = 0; + let totalFail = 0; + let totalSkip = 0; + results.forEach((status, i) => { + const pass = status.filter((s) => s.status === 'pass').length; + const fail = status.filter((s) => s.status === 'fail').length; + const skip = status.filter((s) => s.status === 'skip').length; + totalPass += pass; + totalFail += fail; + totalSkip += skip; + console.log(`Worker ${i} done: ${pass} passed, ${fail} failed, ${skip} skipped`); + }); + const totalReported = totalPass + totalFail + totalSkip; + console.log( + `Total: ${totalPass} passed, ${totalFail} failed, ${totalSkip} skipped (${totalReported} reported)` + ); + + if (totalFail > 0) { + process.exitCode = 1; + } + } finally { + await browser.close(); + } +} + +main().catch((err) => { + console.error('POC error:', err); + process.exit(1); +}); diff --git a/src/config.js b/src/config.js index bd6a959..34f5711 100644 --- a/src/config.js +++ b/src/config.js @@ -10,6 +10,7 @@ const DEFAULT_CONFIG = { headless: true, puppeteerArgs: ['--no-sandbox', '--disable-setuid-sandbox'], retryCount: 2, + parallel: false, }; export function loadConfig() { diff --git a/src/index.js b/src/index.js index 6df6fc7..98e486d 100644 --- a/src/index.js +++ b/src/index.js @@ -7,13 +7,23 @@ import { loadContracts, validateMocks } from './contracts.js'; import { printContractReport } from './contractReport.js'; import { generateContractMarkdown } from './contractMarkdown.js'; import { buildTestPath } from './buildTestPath.js'; +import { runParallel } from './runParallel.js'; export async function runTests() { + const config = loadConfig(); + const workingDir = process.cwd(); + + // Parallel mode — delegate early. Serial body below is unchanged. + if (config.parallel) { + let contractValidators = []; + if (config.contracts && config.contracts.length > 0) { + contractValidators = await loadContracts(config.contracts, workingDir); + } + return runParallel(config, workingDir, contractValidators); + } + let browser; try { - const config = loadConfig(); - const workingDir = process.cwd(); - console.log('Starting TWD test runner...'); console.log('Configuration:', JSON.stringify(config, null, 2)); diff --git a/src/mergeMocks.js b/src/mergeMocks.js new file mode 100644 index 0000000..9a16eaf --- /dev/null +++ b/src/mergeMocks.js @@ -0,0 +1,14 @@ +// Merge per-worker mock maps into a single map. +// Key scheme: `w${workerIndex}:${originalKey}` — worker-index prefix is +// defense-in-depth against random-ID collisions across contexts. +// Each output mock gets a workerIndex field so downstream enrichment +// (buildTestPath) can pick the correct worker's handler tree. +export function mergeMocks(workerMaps) { + const merged = new Map(); + workerMaps.forEach((workerMap, workerIndex) => { + for (const [key, mock] of workerMap) { + merged.set(`w${workerIndex}:${key}`, { ...mock, workerIndex }); + } + }); + return merged; +} diff --git a/src/runParallel.js b/src/runParallel.js new file mode 100644 index 0000000..2dcbf8f --- /dev/null +++ b/src/runParallel.js @@ -0,0 +1,208 @@ +import fs from 'fs'; +import path from 'path'; +import puppeteer from 'puppeteer'; +import { reportResults } from 'twd-js/runner-ci'; +import { validateMocks } from './contracts.js'; +import { printContractReport } from './contractReport.js'; +import { generateContractMarkdown } from './contractMarkdown.js'; +import { buildTestPath } from './buildTestPath.js'; +import { mergeMocks } from './mergeMocks.js'; + +const WORKERS = 2; + +const ANTI_THROTTLE_FLAGS = [ + '--disable-background-timer-throttling', + '--disable-renderer-backgrounding', + '--disable-backgrounding-occluded-windows', +]; + +function mergeArgs(userArgs, extras) { + const merged = [...userArgs]; + for (const flag of extras) { + if (!merged.includes(flag)) merged.push(flag); + } + return merged; +} + +function makeMockCollector(workerMocks, workerCounters) { + return (mock) => { + const occKey = `${mock.alias}:${mock.testId}`; + const count = (workerCounters.get(occKey) || 0) + 1; + workerCounters.set(occKey, count); + const dedupKey = `${mock.method}:${mock.url}:${mock.status}:${mock.testId}:${count}`; + workerMocks.set(dedupKey, { ...mock, occurrence: count }); + }; +} + +async function runWorker(browser, workerIndex, config, workingDir, contractsConfigured, workerMocks, workerCounters) { + const ctx = await browser.createBrowserContext(); + const page = await ctx.newPage(); + + if (contractsConfigured) { + await page.exposeFunction( + '__twdCollectMock', + makeMockCollector(workerMocks, workerCounters) + ); + } + + await page.goto(config.url); + await page.waitForSelector('#twd-sidebar-root', { timeout: config.timeout }); + + const { handlers, testStatus } = await page.evaluate( + async (workerIndex, N, retryCount) => { + const allIds = Array.from(window.__TWD_STATE__.handlers.values()) + .filter((h) => h.type === 'test') + .map((h) => h.id); + const myIds = allIds.filter((_, idx) => idx % N === workerIndex); + + const TestRunner = window.__testRunner; + const testStatus = []; + const runner = new TestRunner( + { + onStart: (test) => { test.status = 'running'; }, + onPass: (test, retryAttempt) => { + test.status = 'done'; + const entry = { id: test.id, status: 'pass' }; + if (retryAttempt !== undefined) entry.retryAttempt = retryAttempt; + testStatus.push(entry); + }, + onFail: (test, err) => { + test.status = 'done'; + testStatus.push({ + id: test.id, + status: 'fail', + error: `${err.message} (at ${window.location.href})`, + }); + }, + onSkip: (test) => { + test.status = 'done'; + testStatus.push({ id: test.id, status: 'skip' }); + }, + }, + { retryCount } + ); + const handlers = await runner.runByIds(myIds); + return { handlers: Array.from(handlers.values()), testStatus }; + }, + workerIndex, + WORKERS, + config.retryCount + ); + + if (config.coverage) { + const coverage = await page.evaluate(() => window.__coverage__); + if (coverage) { + const nycDir = path.resolve(workingDir, config.nycOutputDir); + const outPath = path.join(nycDir, `out-${workerIndex}.json`); + fs.writeFileSync(outPath, JSON.stringify(coverage)); + console.log(`Worker ${workerIndex}: coverage → ${outPath}`); + } else { + console.log(`Worker ${workerIndex}: no __coverage__ on window`); + } + } + + await ctx.close(); + return { workerIndex, handlers, testStatus }; +} + +export async function runParallel(config, workingDir, contractValidators) { + let browser; + try { + console.log(`Starting TWD test runner (parallel mode, ${WORKERS} workers)...`); + console.log('Configuration:', JSON.stringify(config, null, 2)); + + const contractsConfigured = config.contracts && config.contracts.length > 0; + const workerMocks = Array.from({ length: WORKERS }, () => new Map()); + const workerCounters = Array.from({ length: WORKERS }, () => new Map()); + + if (config.coverage) { + const nycDir = path.resolve(workingDir, config.nycOutputDir); + if (fs.existsSync(nycDir)) { + fs.rmSync(nycDir, { recursive: true, force: true }); + } + fs.mkdirSync(nycDir, { recursive: true }); + } + + browser = await puppeteer.launch({ + headless: config.headless, + args: mergeArgs(config.puppeteerArgs, ANTI_THROTTLE_FLAGS), + }); + + console.time('Parallel test time'); + const workerResults = await Promise.all( + Array.from({ length: WORKERS }, (_, i) => + runWorker( + browser, + i, + config, + workingDir, + contractsConfigured, + workerMocks[i], + workerCounters[i] + ) + ) + ); + console.timeEnd('Parallel test time'); + + let totalPass = 0; + let totalFail = 0; + let totalSkip = 0; + for (const { workerIndex, handlers, testStatus } of workerResults) { + console.log(`\n────── Worker ${workerIndex} results ──────`); + reportResults(handlers, testStatus); + const pass = testStatus.filter((s) => s.status === 'pass').length; + const fail = testStatus.filter((s) => s.status === 'fail').length; + const skip = testStatus.filter((s) => s.status === 'skip').length; + console.log( + `Worker ${workerIndex}: ${pass} passed, ${fail} failed, ${skip} skipped` + ); + totalPass += pass; + totalFail += fail; + totalSkip += skip; + } + + console.log(`\n────── Summary ──────`); + console.log( + `Total: ${totalPass} passed, ${totalFail} failed, ${totalSkip} skipped` + ); + + let hasFailures = totalFail > 0; + + if (contractsConfigured) { + const merged = mergeMocks(workerMocks); + + for (const [, mock] of merged) { + if (mock.testId) { + const workerHandlers = workerResults[mock.workerIndex].handlers; + mock.testName = buildTestPath(mock.testId, workerHandlers); + } + } + + if (merged.size === 0) { + console.log('\nNo mocks collected — ensure twd-js supports contract collection'); + } + const validationOutput = validateMocks(merged, contractValidators); + const hasContractErrors = printContractReport(validationOutput); + if (hasContractErrors) hasFailures = true; + + if (config.contractReportPath) { + const reportPath = path.resolve(workingDir, config.contractReportPath); + const reportDir = path.dirname(reportPath); + if (!fs.existsSync(reportDir)) { + fs.mkdirSync(reportDir, { recursive: true }); + } + const markdown = generateContractMarkdown(validationOutput); + fs.writeFileSync(reportPath, markdown); + console.log(`Contract report written to ${config.contractReportPath}`); + } + } + + await browser.close(); + console.log('Browser closed.'); + return hasFailures; + } catch (error) { + console.error('Error running tests (parallel):', error); + if (browser) await browser.close(); + throw error; + } +} diff --git a/tests/config.test.js b/tests/config.test.js index a3a1b50..6315997 100644 --- a/tests/config.test.js +++ b/tests/config.test.js @@ -32,6 +32,7 @@ describe('loadConfig', () => { headless: true, puppeteerArgs: ['--no-sandbox', '--disable-setuid-sandbox'], retryCount: 2, + parallel: false, }); expect(fs.existsSync).toHaveBeenCalledWith(path.resolve(mockCwd, 'twd.config.json')); }); @@ -56,6 +57,7 @@ describe('loadConfig', () => { headless: true, puppeteerArgs: ['--no-sandbox', '--disable-setuid-sandbox'], retryCount: 2, + parallel: false, }); expect(fs.readFileSync).toHaveBeenCalledWith( path.resolve(mockCwd, 'twd.config.json'), @@ -73,6 +75,7 @@ describe('loadConfig', () => { headless: false, puppeteerArgs: ['--disable-dev-shm-usage'], retryCount: 3, + parallel: true, }; vi.mocked(fs.existsSync).mockReturnValue(true); @@ -100,6 +103,7 @@ describe('loadConfig', () => { headless: true, puppeteerArgs: ['--no-sandbox', '--disable-setuid-sandbox'], retryCount: 2, + parallel: false, }); expect(consoleWarnSpy).toHaveBeenCalledWith( expect.stringContaining('Warning: Could not parse twd.config.json'), diff --git a/tests/mergeMocks.test.js b/tests/mergeMocks.test.js new file mode 100644 index 0000000..55b9dea --- /dev/null +++ b/tests/mergeMocks.test.js @@ -0,0 +1,47 @@ +import { describe, it, expect } from 'vitest'; +import { mergeMocks } from '../src/mergeMocks.js'; + +describe('mergeMocks', () => { + it('returns an empty map when given an array of empty maps', () => { + const merged = mergeMocks([new Map(), new Map()]); + expect(merged.size).toBe(0); + }); + + it('prefixes entries from a single worker with w0:', () => { + const w0 = new Map([ + ['GET:/api/a:200:t1:1', { alias: 'a', testId: 't1' }], + ]); + const merged = mergeMocks([w0, new Map()]); + expect(merged.size).toBe(1); + expect(merged.has('w0:GET:/api/a:200:t1:1')).toBe(true); + }); + + it('preserves entries from both workers with disjoint keys', () => { + const w0 = new Map([['GET:/a:200:t1:1', { alias: 'a', testId: 't1' }]]); + const w1 = new Map([['GET:/b:200:t2:1', { alias: 'b', testId: 't2' }]]); + const merged = mergeMocks([w0, w1]); + expect(merged.size).toBe(2); + expect(merged.has('w0:GET:/a:200:t1:1')).toBe(true); + expect(merged.has('w1:GET:/b:200:t2:1')).toBe(true); + }); + + it('keeps both entries when two workers happen to use the same inner key', () => { + // Defense-in-depth: twd-js random IDs could collide; the prefix must + // prevent one worker from overwriting the other's mock. + const sharedInnerKey = 'GET:/api/users:200:same-id:1'; + const w0 = new Map([[sharedInnerKey, { alias: 'users', testId: 'same-id', from: 0 }]]); + const w1 = new Map([[sharedInnerKey, { alias: 'users', testId: 'same-id', from: 1 }]]); + const merged = mergeMocks([w0, w1]); + expect(merged.size).toBe(2); + expect(merged.get(`w0:${sharedInnerKey}`).from).toBe(0); + expect(merged.get(`w1:${sharedInnerKey}`).from).toBe(1); + }); + + it('attaches workerIndex to each merged mock', () => { + const w0 = new Map([['k', { alias: 'a' }]]); + const w1 = new Map([['k', { alias: 'a' }]]); + const merged = mergeMocks([w0, w1]); + expect(merged.get('w0:k').workerIndex).toBe(0); + expect(merged.get('w1:k').workerIndex).toBe(1); + }); +}); diff --git a/tests/runParallel.test.js b/tests/runParallel.test.js new file mode 100644 index 0000000..718ba2e --- /dev/null +++ b/tests/runParallel.test.js @@ -0,0 +1,381 @@ +import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest'; +import { runParallel } from '../src/runParallel.js'; + +vi.mock('fs'); +vi.mock('puppeteer'); +vi.mock('twd-js/runner-ci', () => ({ reportResults: vi.fn() })); +vi.mock('../src/contracts.js', () => ({ validateMocks: vi.fn() })); +vi.mock('../src/contractReport.js', () => ({ printContractReport: vi.fn() })); +vi.mock('../src/contractMarkdown.js', () => ({ generateContractMarkdown: vi.fn() })); + +import fs from 'fs'; +import puppeteer from 'puppeteer'; +import { reportResults } from 'twd-js/runner-ci'; + +function createMockPage(evaluateResult, coverage = null) { + const page = { + goto: vi.fn(), + waitForSelector: vi.fn(), + exposeFunction: vi.fn(), + evaluate: vi.fn(), + }; + // page.evaluate is called twice per worker: once for runByIds, once for __coverage__. + page.evaluate + .mockResolvedValueOnce(evaluateResult) + .mockResolvedValueOnce(coverage); + return page; +} + +function createMockContext(page) { + return { + newPage: vi.fn().mockResolvedValue(page), + close: vi.fn(), + }; +} + +function createMockBrowser(contexts) { + let i = 0; + return { + createBrowserContext: vi.fn().mockImplementation(() => contexts[i++]), + close: vi.fn(), + }; +} + +const baseConfig = { + url: 'http://localhost:5173', + timeout: 10000, + coverage: false, + coverageDir: './coverage', + nycOutputDir: './.nyc_output', + headless: true, + puppeteerArgs: [], + retryCount: 2, + parallel: true, +}; + +describe('runParallel', () => { + beforeEach(() => { + vi.clearAllMocks(); + vi.spyOn(console, 'log').mockImplementation(() => {}); + vi.spyOn(console, 'error').mockImplementation(() => {}); + vi.spyOn(console, 'time').mockImplementation(() => {}); + vi.spyOn(console, 'timeEnd').mockImplementation(() => {}); + vi.mocked(fs.existsSync).mockReturnValue(false); + vi.mocked(fs.mkdirSync).mockImplementation(() => {}); + vi.mocked(fs.writeFileSync).mockImplementation(() => {}); + vi.mocked(fs.rmSync).mockImplementation(() => {}); + }); + + afterEach(() => { + vi.restoreAllMocks(); + }); + + it('launches puppeteer once and creates 2 browser contexts', async () => { + const page0 = createMockPage({ handlers: [], testStatus: [] }); + const page1 = createMockPage({ handlers: [], testStatus: [] }); + const browser = createMockBrowser([createMockContext(page0), createMockContext(page1)]); + vi.mocked(puppeteer.launch).mockResolvedValue(browser); + + await runParallel(baseConfig, '/cwd', []); + + expect(puppeteer.launch).toHaveBeenCalledTimes(1); + expect(browser.createBrowserContext).toHaveBeenCalledTimes(2); + }); + + it('appends anti-throttle flags to user-supplied puppeteerArgs', async () => { + const page0 = createMockPage({ handlers: [], testStatus: [] }); + const page1 = createMockPage({ handlers: [], testStatus: [] }); + const browser = createMockBrowser([createMockContext(page0), createMockContext(page1)]); + vi.mocked(puppeteer.launch).mockResolvedValue(browser); + + await runParallel({ ...baseConfig, puppeteerArgs: ['--user-flag'] }, '/cwd', []); + + const launchArgs = vi.mocked(puppeteer.launch).mock.calls[0][0].args; + expect(launchArgs).toContain('--user-flag'); + expect(launchArgs).toContain('--disable-background-timer-throttling'); + expect(launchArgs).toContain('--disable-renderer-backgrounding'); + expect(launchArgs).toContain('--disable-backgrounding-occluded-windows'); + }); + + it('does not duplicate an anti-throttle flag already provided by the user', async () => { + const page0 = createMockPage({ handlers: [], testStatus: [] }); + const page1 = createMockPage({ handlers: [], testStatus: [] }); + const browser = createMockBrowser([createMockContext(page0), createMockContext(page1)]); + vi.mocked(puppeteer.launch).mockResolvedValue(browser); + + await runParallel( + { ...baseConfig, puppeteerArgs: ['--disable-renderer-backgrounding'] }, + '/cwd', + [] + ); + + const launchArgs = vi.mocked(puppeteer.launch).mock.calls[0][0].args; + const count = launchArgs.filter((a) => a === '--disable-renderer-backgrounding').length; + expect(count).toBe(1); + }); + + it('navigates each page to config.url and waits for sidebar', async () => { + const page0 = createMockPage({ handlers: [], testStatus: [] }); + const page1 = createMockPage({ handlers: [], testStatus: [] }); + const browser = createMockBrowser([createMockContext(page0), createMockContext(page1)]); + vi.mocked(puppeteer.launch).mockResolvedValue(browser); + + await runParallel(baseConfig, '/cwd', []); + + expect(page0.goto).toHaveBeenCalledWith('http://localhost:5173'); + expect(page1.goto).toHaveBeenCalledWith('http://localhost:5173'); + expect(page0.waitForSelector).toHaveBeenCalledWith( + '#twd-sidebar-root', + { timeout: 10000 } + ); + expect(page1.waitForSelector).toHaveBeenCalledWith( + '#twd-sidebar-root', + { timeout: 10000 } + ); + }); + + it('passes workerIndex, N=2, and retryCount to each page.evaluate call', async () => { + const page0 = createMockPage({ handlers: [], testStatus: [] }); + const page1 = createMockPage({ handlers: [], testStatus: [] }); + const browser = createMockBrowser([createMockContext(page0), createMockContext(page1)]); + vi.mocked(puppeteer.launch).mockResolvedValue(browser); + + await runParallel({ ...baseConfig, retryCount: 3 }, '/cwd', []); + + // First evaluate call per worker is the runByIds invocation. + expect(page0.evaluate).toHaveBeenNthCalledWith(1, expect.any(Function), 0, 2, 3); + expect(page1.evaluate).toHaveBeenNthCalledWith(1, expect.any(Function), 1, 2, 3); + }); + + it('sums pass/fail/skip counts across workers', async () => { + const page0 = createMockPage({ + handlers: [{ id: 'a', name: 'a', type: 'test' }], + testStatus: [{ id: 'a', status: 'pass' }], + }); + const page1 = createMockPage({ + handlers: [{ id: 'b', name: 'b', type: 'test' }], + testStatus: [{ id: 'b', status: 'fail', error: 'boom' }], + }); + const browser = createMockBrowser([createMockContext(page0), createMockContext(page1)]); + vi.mocked(puppeteer.launch).mockResolvedValue(browser); + + const hasFailures = await runParallel(baseConfig, '/cwd', []); + + expect(hasFailures).toBe(true); + expect(reportResults).toHaveBeenCalledTimes(2); + }); + + it('returns false when all workers pass', async () => { + const page0 = createMockPage({ + handlers: [{ id: 'a', name: 'a', type: 'test' }], + testStatus: [{ id: 'a', status: 'pass' }], + }); + const page1 = createMockPage({ + handlers: [{ id: 'b', name: 'b', type: 'test' }], + testStatus: [{ id: 'b', status: 'pass' }], + }); + const browser = createMockBrowser([createMockContext(page0), createMockContext(page1)]); + vi.mocked(puppeteer.launch).mockResolvedValue(browser); + + const hasFailures = await runParallel(baseConfig, '/cwd', []); + + expect(hasFailures).toBe(false); + }); + + it('writes per-worker coverage files when config.coverage is true and __coverage__ is non-null', async () => { + const page0 = createMockPage( + { handlers: [], testStatus: [] }, + { file0: 'data' } + ); + const page1 = createMockPage( + { handlers: [], testStatus: [] }, + { file1: 'data' } + ); + const browser = createMockBrowser([createMockContext(page0), createMockContext(page1)]); + vi.mocked(puppeteer.launch).mockResolvedValue(browser); + + await runParallel({ ...baseConfig, coverage: true }, '/cwd', []); + + const writes = vi.mocked(fs.writeFileSync).mock.calls.map((c) => c[0]); + expect(writes.some((p) => p.endsWith('out-0.json'))).toBe(true); + expect(writes.some((p) => p.endsWith('out-1.json'))).toBe(true); + }); + + it('does not write coverage files when config.coverage is false', async () => { + const page0 = createMockPage({ handlers: [], testStatus: [] }, { file0: 'data' }); + const page1 = createMockPage({ handlers: [], testStatus: [] }, { file1: 'data' }); + const browser = createMockBrowser([createMockContext(page0), createMockContext(page1)]); + vi.mocked(puppeteer.launch).mockResolvedValue(browser); + + await runParallel({ ...baseConfig, coverage: false }, '/cwd', []); + + expect(fs.writeFileSync).not.toHaveBeenCalled(); + }); + + it('dumps coverage even when a worker has failures', async () => { + const page0 = createMockPage( + { + handlers: [{ id: 'a', name: 'a', type: 'test' }], + testStatus: [{ id: 'a', status: 'fail', error: 'boom' }], + }, + { file0: 'data' } + ); + const page1 = createMockPage( + { handlers: [], testStatus: [] }, + { file1: 'data' } + ); + const browser = createMockBrowser([createMockContext(page0), createMockContext(page1)]); + vi.mocked(puppeteer.launch).mockResolvedValue(browser); + + await runParallel({ ...baseConfig, coverage: true }, '/cwd', []); + + expect(fs.writeFileSync).toHaveBeenCalledTimes(2); + }); + + it('cleans .nyc_output before running when coverage is enabled', async () => { + const page0 = createMockPage({ handlers: [], testStatus: [] }, { a: 1 }); + const page1 = createMockPage({ handlers: [], testStatus: [] }, { a: 1 }); + const browser = createMockBrowser([createMockContext(page0), createMockContext(page1)]); + vi.mocked(puppeteer.launch).mockResolvedValue(browser); + vi.mocked(fs.existsSync).mockReturnValue(true); + + await runParallel({ ...baseConfig, coverage: true }, '/cwd', []); + + expect(fs.rmSync).toHaveBeenCalledWith( + expect.stringContaining('.nyc_output'), + { recursive: true, force: true } + ); + }); + + it('exposes __twdCollectMock on each page when contracts are configured', async () => { + const page0 = createMockPage({ handlers: [], testStatus: [] }); + const page1 = createMockPage({ handlers: [], testStatus: [] }); + const browser = createMockBrowser([createMockContext(page0), createMockContext(page1)]); + vi.mocked(puppeteer.launch).mockResolvedValue(browser); + + const { validateMocks } = await import('../src/contracts.js'); + const { printContractReport } = await import('../src/contractReport.js'); + vi.mocked(validateMocks).mockReturnValue({ results: [], skipped: [] }); + vi.mocked(printContractReport).mockReturnValue(false); + + await runParallel( + { ...baseConfig, contracts: [{ source: './openapi.json' }] }, + '/cwd', + [{ /* sentinel validator */ }] + ); + + expect(page0.exposeFunction).toHaveBeenCalledWith( + '__twdCollectMock', + expect.any(Function) + ); + expect(page1.exposeFunction).toHaveBeenCalledWith( + '__twdCollectMock', + expect.any(Function) + ); + }); + + it('does not expose __twdCollectMock when contracts are not configured', async () => { + const page0 = createMockPage({ handlers: [], testStatus: [] }); + const page1 = createMockPage({ handlers: [], testStatus: [] }); + const browser = createMockBrowser([createMockContext(page0), createMockContext(page1)]); + vi.mocked(puppeteer.launch).mockResolvedValue(browser); + + await runParallel(baseConfig, '/cwd', []); + + expect(page0.exposeFunction).not.toHaveBeenCalled(); + expect(page1.exposeFunction).not.toHaveBeenCalled(); + }); + + it('feeds merged mocks (with workerIndex) into validateMocks', async () => { + function makePage(workerHandlers, workerTestStatus, mockToCollect) { + const page = { + goto: vi.fn(), + waitForSelector: vi.fn(), + exposeFunction: vi.fn(), + evaluate: vi.fn(), + }; + page.evaluate + .mockImplementationOnce(async () => { + const exposed = page.exposeFunction.mock.calls.find( + (c) => c[0] === '__twdCollectMock' + ); + expect(exposed).toBeDefined(); + const collect = exposed[1]; + await collect(mockToCollect); + return { handlers: workerHandlers, testStatus: workerTestStatus }; + }) + .mockResolvedValueOnce(null); // no coverage + return page; + } + + const page0 = makePage( + [{ id: 't-0', name: 'describe0 > test0', type: 'test' }], + [{ id: 't-0', status: 'pass' }], + { + alias: 'getA', + method: 'GET', + url: '/api/a', + status: 200, + response: 'x', + testId: 't-0', + } + ); + const page1 = makePage( + [{ id: 't-1', name: 'describe1 > test1', type: 'test' }], + [{ id: 't-1', status: 'pass' }], + { + alias: 'getB', + method: 'GET', + url: '/api/b', + status: 200, + response: 'y', + testId: 't-1', + } + ); + const browser = createMockBrowser([createMockContext(page0), createMockContext(page1)]); + vi.mocked(puppeteer.launch).mockResolvedValue(browser); + + const { validateMocks } = await import('../src/contracts.js'); + const { printContractReport } = await import('../src/contractReport.js'); + let capturedMocks; + vi.mocked(validateMocks).mockImplementation((mocks) => { + capturedMocks = mocks; + return { results: [], skipped: [] }; + }); + vi.mocked(printContractReport).mockReturnValue(false); + + await runParallel( + { ...baseConfig, contracts: [{ source: './openapi.json' }] }, + '/cwd', + [{ /* sentinel */ }] + ); + + expect(capturedMocks).toBeDefined(); + const entries = Array.from(capturedMocks.values()); + expect(entries).toHaveLength(2); + const aliases = entries.map((e) => e.alias).sort(); + expect(aliases).toEqual(['getA', 'getB']); + const workerIndices = entries.map((e) => e.workerIndex).sort(); + expect(workerIndices).toEqual([0, 1]); + }); + + it('returns true when contract errors are printed', async () => { + const page0 = createMockPage({ handlers: [], testStatus: [] }); + const page1 = createMockPage({ handlers: [], testStatus: [] }); + const browser = createMockBrowser([createMockContext(page0), createMockContext(page1)]); + vi.mocked(puppeteer.launch).mockResolvedValue(browser); + + const { validateMocks } = await import('../src/contracts.js'); + const { printContractReport } = await import('../src/contractReport.js'); + vi.mocked(validateMocks).mockReturnValue({ results: [], skipped: [] }); + vi.mocked(printContractReport).mockReturnValue(true); + + const hasFailures = await runParallel( + { ...baseConfig, contracts: [{ source: './openapi.json' }] }, + '/cwd', + [{ /* sentinel */ }] + ); + + expect(hasFailures).toBe(true); + }); +}); diff --git a/tests/runTests.test.js b/tests/runTests.test.js index 16e38e4..f7cc735 100644 --- a/tests/runTests.test.js +++ b/tests/runTests.test.js @@ -244,4 +244,57 @@ describe("runTests", () => { expect(entries[0].alias).toBe('getPhoto'); expect(entries[0].occurrence).toBe(1); }); + + it("delegates to runParallel and does NOT launch a single-page serial flow when parallel=true", async () => { + // When parallel mode is on, runTests should call runParallel (which + // itself launches puppeteer with createBrowserContext). The serial code + // path uses browser.newPage() on the default context. If parallel + // dispatch works, page.evaluate should never be called via the serial + // newPage() path — evidenced by puppeteer.launch being invoked exactly + // once with args including the anti-throttle flags. + const browser = { + createBrowserContext: vi.fn().mockImplementation(() => ({ + newPage: vi.fn().mockResolvedValue({ + goto: vi.fn(), + waitForSelector: vi.fn(), + exposeFunction: vi.fn(), + evaluate: vi.fn() + .mockResolvedValueOnce({ handlers: [], testStatus: [] }) + .mockResolvedValueOnce(null), + }), + close: vi.fn(), + })), + newPage: vi.fn(), // serial path would call this — we assert it was NOT called + close: vi.fn(), + }; + vi.mocked(puppeteer.launch).mockResolvedValue(browser); + vi.mocked(loadConfig).mockReturnValue({ + ...defaultMockConfig, + parallel: true, + }); + + await runTests(); + + expect(browser.newPage).not.toHaveBeenCalled(); + expect(browser.createBrowserContext).toHaveBeenCalledTimes(2); + + const launchArgs = vi.mocked(puppeteer.launch).mock.calls[0][0].args; + expect(launchArgs).toContain('--disable-renderer-backgrounding'); + }); + + it("runs the serial path when parallel is absent (default false)", async () => { + const testStatus = [{ id: '1', status: 'pass' }]; + const handlers = [{ id: '1', name: 'test1', type: 'test' }]; + const page = createMockPage({ handlers, testStatus }); + const browser = createMockBrowser(page); + vi.mocked(puppeteer.launch).mockResolvedValue(browser); + // defaultMockConfig has no `parallel` field — absent should mean serial. + + await runTests(); + + expect(browser.newPage).toHaveBeenCalled(); + // Anti-throttle flags are a parallel-only behavior — NOT in the serial launch. + const launchArgs = vi.mocked(puppeteer.launch).mock.calls[0][0].args; + expect(launchArgs).not.toContain('--disable-renderer-backgrounding'); + }); });