feat: add SQLite-backed durable job store#197
Open
seonghobae wants to merge 1 commit into
Open
Conversation
The async web job model keeps jobs in an in-memory dict, so all job state is lost on restart and cannot be shared across processes. Add job_store.py, a stdlib-only (sqlite3, WAL mode) JobStore that any async/worker layer can adopt: - create/get/set_status/list_jobs/delete with a queued/processing/ done/failed status allowlist (invalid status -> ValueError) - thread-safe via a lock plus short-lived connections; safe across processes via SQLite file locking + WAL - explicit now: datetime injection for deterministic timestamps - duplicate create raises DuplicateJobError (ValueError subclass); unknown get returns None; delete is idempotent Tests cover roundtrip, transitions, filtering, durability across store reopen, and concurrent set_status from 8 threads. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CJRVbDrp1vGYkJgNHMGPpG
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The async web job model tracks jobs in an in-memory dict. Every restart wipes all job state, and the dict is invisible to any other process — so the app cannot scale past a single worker and users lose in-flight jobs on redeploy.
What
New stdlib-only module
job_store.py: a SQLite-backed (WAL mode)JobStorethat any async/worker layer can adopt so jobs survive restarts and are shared across processes.create/get/set_status/list_jobs/deleteover ajobstable (id, status, created_at, updated_at, output_path, output_name, error, temp_dir){queued, processing, done, failed}; invalid status raisesValueErrornow: datetime; the store never callsdatetime.now()createraisesDuplicateJobError(aValueErrorsubclass), unknowngetreturnsNone,deleteis idempotent,set_statuspreserves previously stored output/error fields when omittedNew files only; no existing files touched.
Tests
tests/test_job_store.py(15 tests, tmp SQLite file, injected timestamps, no network):updated_at; error recording; omitted fields preserved; invalid status ValueError; unknown job KeyErrorJobStoreon the same db_path still sees the jobset_statusacross 8 jobs, no torn writesGates
python3.14 -m unittest discover -s tests: 128 tests, no new failures (baseline 5 macOSos.listxattrerrors unchanged)python3.14 -m interrogate -c pyproject.toml job_store.py: 100%🤖 Generated with Claude Code