wallet: cooperative-yield sync (incremental persist + sentinel)

[saroupille]

Problem

tzel-wallet sync scans [scanned, tree_size) sequentially, persisting wallet.json.scanned only at the end. While running it holds an exclusive lock on wallet.json.lock, so every other wallet op (shield, transfer, unshield, balance, sync, …) blocks for the full duration. On a long-running rollup that's 5–45 min wallclock — unacceptable in daemon-driven flows where users want to act now.

This is the second half of a two-PR pair. PR #23 (merged) added cmd_init --at-tree-size N so a fresh wallet skips the historical scan entirely. This PR makes the scan that DOES need to run preemptable so it doesn't starve the rest of the daemon.

Design — cooperative yield via sentinel file

Mechanism

1. Incremental persist. cmd_rollup_sync scans in batches of --checkpoint-every commits (default DEFAULT_CHECKPOINT_EVERY = 50). Each batch ends in a save_wallet that records w.scanned = batch_end plus newly-discovered notes atomically (temp + fsync + rename + parent-dir fsync — pre-existing invariant, just called more often).
2. Sentinel check at every checkpoint. After persisting a batch, the sync stat()s <wallet>.yield. Present ⇒ exit 0 cleanly. Next sync resumes from the persisted cursor.
3. Stale-PID recovery (audit-driven). The daemon writes its PID into the sentinel; the CLI checks /proc/<pid> and unlinks if dead. Mirrors the existing WalletLock discipline. Without this, a daemon crash mid-slow-op would wedge every subsequent sync forever.
4. Final flush at end-of-loop. Nullifier prune + pending-pool prune run once over the cumulative cm set accumulated across batches.

The companion change in trilitech/tzel-infra#55 wraps slow-lane CLI ops (shield/send/unshield/withdraw) with touch <wallet>.yield before admit + RAII-removed after. So a long initial sync no longer starves slow-lane ops: the slow-lane signals "yield please" via the file, the next checkpoint sees it, sync exits cleanly, slow-lane gets the lock.

CLI surface

tzel-wallet sync                          # K=50 default
tzel-wallet sync --checkpoint-every 200   # less overhead, slower preempt
tzel-wallet sync --checkpoint-every 20    # faster preempt, more fsyncs
TZEL_SYNC_CHECKPOINT_EVERY=100 tzel-wallet sync   # operator override

The default 50 is reasoned, pending empirical measurement on a live octez-smart-rollup-node. Documented inline. Tunable range 10..500 with explicit operator-vs-laptop guidance for slow disks (raise to 100–200) vs fast operator boxes wanting sub-second preemption (drop to 20–30).

Considered alternative — shadow-wallet sync

A strategic-audit pass proposed an alternative design where the scan operates on a <wallet>.scan scratch copy with its own lock, and merges back into canonical at every K-commit boundary. Documented in docs/shadow-wallet-design.md (in tzel-infra) for traceability.

Why we picked cooperative-yield over shadow-wallet (per the comparative audit's verdict):

Failure-class blast radius	Cooperative-yield (this PR)	Shadow-wallet
Worst-case footgun	Stale sentinel file; recovered by kill -0 + unlink, loud (sync exits early forever, immediately diagnosable, rm <wallet>.yield resolves)	Delta-accumulator clear-before-fsync; silent (lost nullifier observations → spent note looks unspent → double-spend rejected at proving time → opaque UX error)
Discovery latency	K commits	K commits (when periodic-merge variant chosen)
Lock contention	Bounded by K	Bounded by K (when periodic-merge)
fsync count on 67k commits	1340	1340
Crash class: stale sentinel	yes (recoverable)	no
Crash class: silent observation drift	no	yes (clear-before-rename ordering bug)

Loud beats silent. Both designs are tactical fixes for the same constraint (file-based wallet + flock). At ~2× LOC for shadow-wallet plus a re-litigation of ADR-002, the migration cost (~2 weeks senior-eng review) buys a "canonical never holds partial scan state" property that no caller currently exercises — while introducing a subtle data-loss class that surfaces only at proving time. Decision: ship cooperative-yield as-is.

Pre-merge audit follow-ups (commits f3a0755, 3edef12)

Independent reviewers caught a real consistency bug missed by the earlier audit rounds: cmd_rollup_sync finalize pinned head_hash for the note slice + nullifiers but called the head-resolving load_pool_balances for pool reads. A slow-lane drain landing between the two reads would silently evict a still-funded pool. Fixed in f3a0755 (load_pool_balances_at_block taking an explicit block_ref) and covered by a regression test that drives two block hashes returning different balances.

A generalising review of the same pattern caught a second instance in cmd_wallet_check (banner pool summary), fixed in 3edef12. The same commit renames the head-resolving wrappers to _at_head so the naming asymmetry (_at_head vs _at_block) makes a future regression loud at code review, and enriches the stale-sentinel recovery warning with the dead daemon's PID for operator triage.

Tests

12 cooperative-yield tests in network_profile_tests (gated on #[serial(sync_checkpoint_hook)] since they share the process-global SYNC_CHECKPOINT_HOOK):

Test	What it asserts
cmd_rollup_sync_exits_at_first_checkpoint_when_sentinel_present	Pre-existing sentinel ⇒ exit at first checkpoint, do NOT remove caller's sentinel
cmd_rollup_sync_resumes_from_checkpointed_cursor	Sentinel triggered mid-scan ⇒ resume from last checkpoint, notes pre-yield in run 1, post-yield in run 2
cmd_rollup_sync_intermediate_wallet_is_always_parseable	Concurrent reader thread sees no half-written wallet.json
cmd_rollup_sync_refuses_zero_checkpoint_every	Validation pre-RPC; error message stable
cmd_rollup_sync_respects_custom_checkpoint_every	K ≠ 50 honoured (locks the runtime parameter contract)
cmd_rollup_sync_http_error_mid_batch_preserves_last_checkpoint	5xx mid-batch ⇒ Err propagates; scanned at last batch boundary; sentinel untouched by CLI
cmd_rollup_sync_panic_mid_batch_preserves_wallet_json	Rust panic mid-batch ⇒ atomic-rename invariant holds; reload works
cmd_rollup_sync_watch_with_zero_checkpoint_every_aborts_first_iter	watch loop aborts on validation, doesn't sleep
cmd_rollup_sync_recovers_from_stale_yield_sentinel	Sentinel with a freshly-reaped child PID ⇒ unlink + continue, scan reaches tree_size (uses freshly_dead_pid() helper instead of a hardcoded 999999, robust against kernel.pid_max lore on container hosts)
cmd_rollup_sync_watch_loop_body_yields_twice_then_finishes	Three-iteration yield-then-resume mirroring cmd_rollup_sync_watch's loop body — sentinel persists across iter 1+2, removed before iter 3. Locks the property that recovered notes from earlier iterations re-feed known_cms on the finalize iteration via w.notes, so a regression that moved seen_cms to a process-local accumulator (or stopped folding w.notes into known_cms) would surface
cmd_rollup_sync_treats_legacy_sentinel_content_as_live	Non-PID content (b"yield") ⇒ yield as before; CLI does NOT unlink (forward-compat with old daemons)

Plus four post-audit regression tests:

• rollup_rpc_load_pool_balances_at_block_uses_pinned_block — two block hashes returning different balances; asserts the pinned form reads from the caller's block, not from a re-resolved head.
• rollup_rpc_load_state_snapshot_at_block_uses_pinned_block — mirror for the state snapshot helper that cmd_wallet_check uses on the same pinned head_hash (Reviewer feat(wallet): wallet-server HTTP API and web dapp #1's ask).
• cmd_rollup_sync_pins_pool_reads_to_finalize_head — e2e test driving a full sync against a stateful HTTP mock whose /head/hash route returns block_old on the first read and block_new thereafter. The fixture publishes one note at index 0 wrapping observed_cm, so the batch's seen_cms includes it; the wallet's pending_deposit.shielded_cm = Some(observed_cm) so finalize evaluates cm_observed = true. Pool funded at block_old, drained at block_new. Correct shape: pool reads at the pinned block_old (funded) ⇒ deposit retained AND head/hash counter resolves exactly once. Regression (load_pool_balances_at_head at the call site): pool reads land on block_new ⇒ drained_on_chain && cm_observed ⇒ deposit evicted ⇒ assertion fails.
• fixture_for_finalize_pin_test_actually_evicts_under_regression_helper — companion that proves the fixture's failure mechanism is real (not tautological): same fixture, but invokes load_pool_balances_at_head directly + apply_scan_feed_finalize, asserts the deposit IS evicted. Locks in that the e2e test above protects a live mechanism.

The first cut of the e2e test (in commit 0392f29) was caught as tautological by independent reviewers A and B: it had tree_size = 0 (loop never ran) and shielded_cm = None (cm_observed always false), so the eviction predicate could not fire and the test passed regardless of the call-site shape. Both reviewers verified mechanically by reverting the call site to _at_head and observing the test still passed. The fixture is now corrected — and the companion test above verifies the failure mechanism via the helper without needing to mutate production code.

Result: cargo +nightly-2025-07-14 test -p tzel-wallet-app --lib → 118 passed, 0 failed, 3 ignored.

Companion change

trilitech/tzel-infra#55 (feat/sync-cooperative-yield) — daemon dispatcher wraps slow-lane task execution with SyncYieldGuard::arm (writes the daemon PID into <wallet>.yield) + RAII Drop (rm), so a long-running sync yields the lock for shield/transfer/unshield/withdraw and resumes after.

The contract is forward-compatible: a daemon shipping the sentinel without this CLI's stale-PID recovery is harmless (legacy CLI just yields normally, no recovery benefit). Once both binaries ship, the cooperative behaviour activates.

Deferred to follow-up PRs

• Daemon-side runner.rs pass-through of TZEL_SYNC_CHECKPOINT_EVERY via the env (unblocks operator override end-to-end).
• friendlyError UI pattern for the validation error once the env var exposes it to non-CLI users.
• Re-tune default K to 250–500 once the design doc's parallel-fetch patch (P2, see PR wallet: concurrent HTTP + parallel decrypt for cmd_rollup_sync #25) lands; at the post-P2 wallclock, K=50 means ~7 fsyncs/s sustained — non-trivial.
• Long-term architectural follow-up: SQLite-backed WalletFile to delegate concurrency to the storage layer (would obsolete this PR's mechanism and the shadow-wallet alternative).