Skip to content

Fix ingester DB cache eviction/export safety and startup cleanup#10

Closed
ian-ross wants to merge 9 commits into
mainfrom
db-staging
Closed

Fix ingester DB cache eviction/export safety and startup cleanup#10
ian-ross wants to merge 9 commits into
mainfrom
db-staging

Conversation

@ian-ross
Copy link
Copy Markdown
Member

@ian-ross ian-ross commented May 9, 2026

Summary

  • fix staged-connection eviction so touched (dirty) staged DBs are committed before eviction
  • fail fast when opening staging DBs for VACUUM export if read-only open fails
  • close nursery DB explicitly after promotion in connect-path nursery eviction
  • narrow startup scratch cleanup to only exporter temp files
  • add regression tests for touched-connection eviction commit safety and strict staging-open failure

Motivation

PR #9 had post-review correctness gaps around data-loss risk and fail-open behavior during staging export/import. This PR addresses those before merging the ingester isolation work.

Changes

P1: Prevent uncommitted touched DB evictions

_evict_staged_connections now loops while over cache limit and for touched candidates:

  • tries to commit pending DBs and retries eviction
  • raises if commit does not clear touched state

P2: Fail-fast staging export connection behavior

_open_raw_staging_connection now raises RuntimeError on read-only open failure instead of falling back to read-write.

P3: Resource/cleanup hardening

  • connect-path nursery promotion now explicitly closes promoted nursery DB after promotion
  • startup cleanup in scratch/ingester-export now removes only *.export.*.sqlite files, preserving unrelated scratch artifacts

Tests

  • uv run --python 3.13 pytest tests/feder_ingest/test_db_cache_paths.py -q

ian-ross and others added 9 commits May 5, 2026 22:50
Co-authored-by: GPT-5.5 <openai-codex-gpt-5.5@pi.local>
Pi-Model: openai-codex/gpt-5.5
Co-authored-by: GPT-5.5 <openai-codex-gpt-5.5@pi.local>
Pi-Model: openai-codex/gpt-5.5
Co-authored-by: GPT-5.5 <openai-codex-gpt-5.5@pi.local>
Pi-Model: openai-codex/gpt-5.5
Co-authored-by: GPT-5.5 <openai-codex-gpt-5.5@pi.local>
Pi-Model: openai-codex/gpt-5.5
Co-authored-by: GPT-5.5 <openai-codex-gpt-5.5@pi.local>
Pi-Model: openai-codex/gpt-5.5
Co-authored-by: GPT-5.5 <openai-codex-gpt-5.5@pi.local>
Pi-Model: openai-codex/gpt-5.5
Co-authored-by: GPT-5.5 <openai-codex-gpt-5.5@pi.local>
Pi-Model: openai-codex/gpt-5.5
Co-authored-by: GPT-5.5 <openai-codex-gpt-5.5@pi.local>
Pi-Model: openai-codex/gpt-5.5
Co-authored-by: GPT-5.3 Codex Spark <openai-codex-gpt-5.3-codex-spark@pi.local>
Pi-Model: openai-codex/gpt-5.3-codex-spark
@ian-ross ian-ross closed this May 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant