Single-attach generation fence (closes split-brain data-loss gap)

[jaredLunde]

Problem

Under a network partition, instd's node-death recovery can let two nodes attach and write the same GlideFS volume at once → split-brain → data loss. A node-lease TTL provably cannot prevent this (a partitioned-but-alive predecessor's lease lapses, a replacement re-forks and re-attaches). Only the storage layer can authoritatively stop two writers. This is the GlideFS half of the recovery fix — the open Inv1 gap in beyond/compute/docs/12-recovery-formal-spec.md §6.1.

Today GlideFS enforces no single-attach: a second node's attach succeeds unconditionally, and the only cross-node guard (the manifest If-Match ETag) fires at sync, after both nodes already uploaded data packs → the loser's packs orphan and its in-memory state diverges. That's data loss, not fencing.

Fix — a monotonic fencing token, checked at the resource

A generation (the orchestrator's per-instance placement generation) checked with my_gen >= stored_gen ⇒ Grant — mirrors instd recovery_kernel::attach_fence verbatim. >= admits an idempotent self re-attach; only a strictly newer generation fences an older holder out. gen 0 = back-compat bypass (callers that send no generation are never fenced), so this is inert until instd plumbs a token.

Manifest-anchored, single source of truth, reusing the existing If-Match manifest CAS (no new primitive):

• block/fence.rs — the pure rule + gen-0 bypass wrapper.
• volume_manifest.rs — generation as a zero-churn trailer (v7 = generation only, v8 = prefetch+generation), emitted only when > 0 so un-fenced manifests stay byte-identical to v5/v6.
• router::enforce_attach_fence + api.rs — attach-time fence: on grant the winner seizes the volume by stamping its generation into the manifest via the If-Match CAS, before serving any I/O (a rejected attach → HTTP 409 with zero packs uploaded — nothing orphans).
• write_cache (flush/inner/write/error) — sync-time fence (the load-bearing half): the generation is stamped on every sync; on a precondition failure the writer re-reads and, if a strictly-newer generation owns the volume, returns a terminal CacheError::Fenced — the flush scheduler stops and the write path latches closed so the guest stops getting ACKs for writes that can never durably commit.

Tests

cargo test -p glidefs --features ublk --lib → 511 passed, 0 failed. New coverage:

• the pure >= rule incl. equal-gen re-attach and the gen-0 bypass;
• v5/v7/v8 manifest round-trips (zero churn proven byte-for-byte);
• a two-router shared-object-store split-brain integration test (stale gen → 409, newer gen wins, superseded owner's re-attach fenced, same-gen re-attach idempotent);
• a write_cache test proving a superseded incumbent is rejected at sync and its write path latches closed.

Deployed and verified on the homelab (built --features ublk; both existing exports re-attach un-fenced via the gen-0 bypass).

Scope notes

• The wired token is the placement-generation (cross-node) half of the formal model's composite token. The same-node half is already covered in GlideFS by the export map + manifest ETag CAS. See spec §6.1.
• The matching instd plumbing (threading InstanceAssignment.generation to the attach) is a separate change on the instd side.

🤖 Generated with Claude Code