feat(batch): pre-flight liveness gate (Theme 5 Layer 4)#324
Draft
mitwilli-create wants to merge 1 commit into
Draft
feat(batch): pre-flight liveness gate (Theme 5 Layer 4)#324mitwilli-create wants to merge 1 commit into
mitwilli-create wants to merge 1 commit into
Conversation
…ayer 4)
Closes Theme 5 Layer 4 from the 2026-05-29 task-audit. Adds a pre-flight
filter to batch-runner-batches.mjs::phaseSubmit that reads
data/liveness-cache.json BEFORE the expensive Playwright JD-fetch loop
and skips URLs already marked status='expired'.
Why this matters
The 2026-05-26 batch incident burned ~$0.50-1/URL of Playwright fetch
work on URLs the liveness-cache already knew were dead. The existing
post-fetch dequeue (line ~720) caught them AFTER the spend; this gate
catches them BEFORE.
Implementation
- Reads data/liveness-cache.json once per submit.
- For each item in scopedItems × LIMIT × TIERS slice, looks up the URL.
- If entry.status === 'expired', adds to cacheExpiredFromGate set + filters
the items list.
- Logs each skip to .claude/audit/<DATE>/dead-url-skip-<DATE>.jsonl
(append-only, gitignored, contains per-URL ts + reason + cache_ts).
- Inline dequeue from batch/triage-advance.tsv + archive to
batch/triage-advance-expired/<DATE>-liveness-gate-N.tsv so the gate
is idempotent (next run doesn't re-attempt).
- All errors are non-fatal (the gate fails open — proceeds without
filtering rather than blocking the whole batch).
Bypass
--skip-liveness-gate (CLI flag, emergency only). Logs the bypass.
Safety
- Empty filter result early-exits cleanly ("nothing to batch") rather
than submitting a zero-row payload.
- No external API calls; pure local-file read + filter.
- Gracefully degrades when liveness-cache.json is missing (skips gate,
behavior unchanged from pre-fix).
Test plan
- node --check passes.
- --help still prints status correctly.
- With empty cache: gate is a no-op, batch proceeds normally.
- With cache containing expired URLs that ARE in triage-advance: those
URLs are removed from items + dequeued + logged + the rest submit.
- --skip-liveness-gate: prints BYPASSED line, processes all URLs.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Theme 5 Layer 4 from the 2026-05-29 task-audit. Wires a pre-flight liveness gate into
batch-runner-batches.mjs::phaseSubmitthat filters URLs markedstatus='expired'indata/liveness-cache.jsonBEFORE the expensive Playwright JD-fetch loop. Saves ~$0.50-1/URL on dead postings.Implementation
data/liveness-cache.jsononce per submititemin the LIMIT × TIERS slice: if cache entry isexpired, skip it + dequeue frombatch/triage-advance.tsv+ archive.claude/audit/<DATE>/dead-url-skip-<DATE>.jsonl(gitignored — contains personal pipeline URLs)--skip-liveness-gate(emergency only)What this defends against
The 2026-05-26 batch incident burned ~$0.50-1/URL of Playwright fetch work on URLs the liveness-cache already knew were dead. The existing post-fetch dequeue at line ~720 caught them AFTER the spend; this gate catches them BEFORE.
Combined with the existing post-fetch terminal-failure dequeue (PR #308) and the
expired-status mark indata/liveness-cache.json(populated bycheck-liveness.mjs+ the launchdliveness-sweepplist), this completes the liveness-aware batch flow:Locked decisions (interview 2026-05-29)
Test plan
node --check batch-runner-batches.mjspassesnode batch-runner-batches.mjs --helpprints status with no errors--skip-liveness-gateflag bypasses the gate (prints BYPASSED line).claude/audit/<DATE>/dead-url-skip-<DATE>.jsonlis gitignored (verified viagit check-ignore)Layer 5 follows separately
Layer 5 (cost-trace truthfulness audit) is a standalone script (
scripts/audit-cost-trace.mjs) and ships as its own PR for clean review.🤖 Generated with Claude Code