Skip to content

feat(mem-wal): snapshot-consistent as-of cut for fresh-tier membership#7215

Open
hamersaw wants to merge 1 commit into
mainfrom
bug/wal-snapshot-as-of-cut
Open

feat(mem-wal): snapshot-consistent as-of cut for fresh-tier membership#7215
hamersaw wants to merge 1 commit into
mainfrom
bug/wal-snapshot-as-of-cut

Conversation

@hamersaw

Copy link
Copy Markdown
Contributor

What

Adds AsOfCut { active_generation, active_batch_count } and LsmScanner::contains_pks_as_of (with fresh_tier_block_list threading the cut) so a caller can evaluate fresh-tier PK membership against the exact tier a prior scan observed, instead of the live tier.

Why

Sophon's WAL block-list runs as two independent RPCs (the read arm and a supersession check) that each snapshot the live fresh tier at their own call time. Under concurrent writes the two snapshots disagree, so a base row can be dropped as "superseded" by the check while the arm never delivered a replacement — a transient missing row. The fix pins both phases to the same cut; this PR is the lance half that lets the check reconstruct the arm's snapshot.

How the cut works

The active memtable is the only fresh-tier source that grows between two reads; everything strictly below its generation (frozen memtables, flushed generations) is immutable. So the as-of filter:

  • includes in-memory/flushed sources below active_generation whole (immutable, fully observed),
  • bounds the active generation to its first active_batch_count batches (by append index),
  • excludes in-memory sources above active_generation and flushed generations >= active_generation (produced after the snapshot).

It uses only batch_store.len() and the memtable generation — both always available on the read path — and only ever excludes rows the scan did not observe, so a stale cut under-counts (a tolerable stale read) rather than over-counts (which would drop a row with no replacement).

Note: an earlier approach keyed the cut on per-batch WAL positions (wal_batch_mapping), but that map is only populated by mark_wal_flushed, which is test/bench-only — empty in production. The generation + batch-count cut avoids any write-path dependency.

Tests

fresh_tier_block_list as-of unit tests: active-memtable batch-count bound, newer-gen-excluded / lower-gen-included-whole, flushed-gen at/above active excluded.

🤖 Generated with Claude Code

Add `AsOfCut { active_generation, active_batch_count }` and
`LsmScanner::contains_pks_as_of` (with `fresh_tier_block_list` threading the
cut) so a caller can evaluate fresh-tier PK membership against the exact tier
a prior scan observed, instead of the live tier.

The active memtable is the only fresh-tier source that grows between two
reads; everything strictly below its generation (frozen memtables, flushed
generations) is immutable. The cut includes lower generations whole, bounds
the active generation to its first `active_batch_count` batches (by append
index), and excludes higher generations and flushed generations
>= active_generation. This uses only the batch count and generation -- both
always available -- and only ever excludes rows the scan did not observe, so
a stale cut under-counts (a tolerable stale read) rather than dropping a row.

Sophon's WAL block-list uses this to pin its two-phase supersession check to
the snapshot the read arm scanned, closing a cross-arm transient missing-row.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@github-actions github-actions Bot added the enhancement New feature or request label Jun 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant