Skip to content

feat: add SQLite-backed durable job store#197

Open
seonghobae wants to merge 1 commit into
mainfrom
feat/durable-job-store
Open

feat: add SQLite-backed durable job store#197
seonghobae wants to merge 1 commit into
mainfrom
feat/durable-job-store

Conversation

@seonghobae

Copy link
Copy Markdown
Contributor

Why

The async web job model tracks jobs in an in-memory dict. Every restart wipes all job state, and the dict is invisible to any other process — so the app cannot scale past a single worker and users lose in-flight jobs on redeploy.

What

New stdlib-only module job_store.py: a SQLite-backed (WAL mode) JobStore that any async/worker layer can adopt so jobs survive restarts and are shared across processes.

  • create / get / set_status / list_jobs / delete over a jobs table (id, status, created_at, updated_at, output_path, output_name, error, temp_dir)
  • Status allowlist {queued, processing, done, failed}; invalid status raises ValueError
  • Thread-safe (lock + short-lived connections) and multi-process-safe (SQLite file locking + WAL)
  • Deterministic timestamps: callers inject now: datetime; the store never calls datetime.now()
  • Clear edge semantics: duplicate create raises DuplicateJobError (a ValueError subclass), unknown get returns None, delete is idempotent, set_status preserves previously stored output/error fields when omitted

New files only; no existing files touched.

Tests

tests/test_job_store.py (15 tests, tmp SQLite file, injected timestamps, no network):

  • create → get roundtrip; unknown get returns None; duplicate create raises
  • set_status transitions update status + updated_at; error recording; omitted fields preserved; invalid status ValueError; unknown job KeyError
  • list_jobs with and without status filter; delete + idempotent delete
  • Durability: reopening a new JobStore on the same db_path still sees the job
  • Concurrency: 8 threads racing set_status across 8 jobs, no torn writes

Gates

  • python3.14 -m unittest discover -s tests: 128 tests, no new failures (baseline 5 macOS os.listxattr errors unchanged)
  • python3.14 -m interrogate -c pyproject.toml job_store.py: 100%

🤖 Generated with Claude Code

The async web job model keeps jobs in an in-memory dict, so all job
state is lost on restart and cannot be shared across processes. Add
job_store.py, a stdlib-only (sqlite3, WAL mode) JobStore that any
async/worker layer can adopt:

- create/get/set_status/list_jobs/delete with a queued/processing/
  done/failed status allowlist (invalid status -> ValueError)
- thread-safe via a lock plus short-lived connections; safe across
  processes via SQLite file locking + WAL
- explicit now: datetime injection for deterministic timestamps
- duplicate create raises DuplicateJobError (ValueError subclass);
  unknown get returns None; delete is idempotent

Tests cover roundtrip, transitions, filtering, durability across
store reopen, and concurrent set_status from 8 threads.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CJRVbDrp1vGYkJgNHMGPpG
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant