Skip to content

feat: durable SQLite-backed job store (replaces in-memory _JOBS dict)#199

Closed
seonghobae wants to merge 1 commit into
feat/async-job-modelfrom
feat/sqlite-job-store
Closed

feat: durable SQLite-backed job store (replaces in-memory _JOBS dict)#199
seonghobae wants to merge 1 commit into
feat/async-job-modelfrom
feat/sqlite-job-store

Conversation

@seonghobae

Copy link
Copy Markdown
Contributor

Stacked on #166 (async job API). Retarget to main after #166 lands.

What

Replaces the async job API's per-process in-memory _JOBS dict with job_store.JobStore — a small SQLite (WAL) store — and adds a startup maintenance pass that reconciles the store with reality.

Why (what the dict could not do)

Failure mode with _JOBS dict With durable store
Service restarts → every job vanishes; clients poll a 404 forever Job rows survive; orphaned queued/processing jobs are marked failed: Interrupted by service restart at startup
uvicorn --workers N → submit lands on worker A, status poll on worker B → 404 All workers on the host share one SQLite store
Crash mid-job → temp workspace leaks on disk forever Startup recovery reclaims orphaned workspaces
Results nobody downloads → rows + workspaces accumulate forever Retention sweep purges terminal jobs after 24h (configurable) and removes their workspaces

Design notes

  • Zero new dependencies — stdlib sqlite3, WAL mode + 30s busy timeout, one short-lived connection per operation (thread- and process-safe, no leaked handles).
  • Config: CODEC_CARVER_JOB_DB sets the DB path (point at a persistent volume in production; defaults to a per-host path under the system tempdir so it is zero-config). CODEC_CARVER_JOB_RETENTION_SECONDS tunes the sweep window.
  • Safety: column names are whitelisted (ValueError on anything else), values are always bound parameters; DELETE … RETURNING gives exactly-once workspace cleanup under concurrent result downloads; /jobs/{id}/result keeps the workspace-confinement check from feat: async job API for the web service (submit → status → result) #166 and now degrades to 410 (not a crash) on a corrupted row with no output_path.
  • API contract unchanged: same endpoints, same JSON shapes, same status codes — all feat: async job API for the web service (submit → status → result) #166 tests pass unmodified in behaviour, only their store fixture changed.

Tests

  • New tests/test_job_store.py (15 cases): CRUD roundtrip, reopen-survival (simulated restart / second worker), interrupted-job recovery, stale purge boundaries, field-whitelist rejection (incl. SQL-injection-shaped field names), exactly-once delete, 8-thread concurrent writer stress.
  • tests/test_saas_web.py: job tests ported to an isolated per-test store; new JobDurabilityTests cover status surviving a store reopen and the startup maintenance path end-to-end (verified through the real ASGI lifespan).
  • pytest tests/ → 144 passed; the 5 pre-existing test_media_shrinker xattr failures also fail on the base branch (environment-specific, unrelated). interrogate docstring gate: 100%.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CJRVbDrp1vGYkJgNHMGPpG

Replace the per-process in-memory _JOBS dict with job_store.JobStore,
a SQLite (WAL) store shared by every worker process on the host:

- job state survives service restarts and crashes; polling clients of
  interrupted jobs get an honest 'failed: Interrupted by service
  restart' instead of a permanent 404/hang
- status/result requests work under multi-worker uvicorn/gunicorn,
  since all workers now read the same store
- startup maintenance reconciles the store with reality: orphaned
  queued/processing jobs are failed and their temp workspaces
  reclaimed; terminal jobs past the retention window are purged
  together with their workspaces (no more disk leaks from results
  nobody downloaded)
- CODEC_CARVER_JOB_DB points the store at a persistent volume;
  CODEC_CARVER_JOB_RETENTION_SECONDS tunes the sweep window
- field-whitelisted SQL, one short-lived connection per operation,
  delete-returning semantics so workspace cleanup happens exactly once
  under concurrent deletes

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CJRVbDrp1vGYkJgNHMGPpG
@seonghobae

Copy link
Copy Markdown
Contributor Author

Closing to avoid a duplicate job_store.py and PR sprawl. The standalone, mergeable-now module lives in #197 (feat/durable-job-store → main). This PR's saas_web integration is premature — it's stacked on the still-open #166 and can't merge until that does.

The valuable ideas here (recover_interrupted(), purge_stale(), DELETE … RETURNING for exactly-once cleanup, column whitelist) will be folded into the proper integration PR once #166 merges. Nothing is lost.

@seonghobae seonghobae closed this Jul 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant