afx: removed sibling architect resurrects after Tower recovery — reconciliation trusts persisted row without liveness check

[amrmelsayed]

Problem

After a machine crash or Tower recovery event, sibling architects that the user deliberately removed via afx workspace remove-architect --name X come back to life — with their old conversation restored via --resume. The user has to remove them again.

Reported by users 2026-07-08.

Root cause (from subagent investigation 2026-07-08)

removeArchitect (packages/codev/src/agent-farm/servers/tower-instances.ts:1115-1193) structurally does the right thing: sets intentionallyStopping, kills the shellper, deletes both DB rows (architects + terminal_sessions), awaits exit. BUT the DB deletes at tower-instances.ts:1174 and tower-instances.ts:1179 are wrapped in silent try { … } catch { /* best-effort cleanup */ }:

try { setArchitectByName(resolvedPath, name, null); } catch { /* best-effort cleanup */ }
try { _deps.deleteTerminalSession(terminalId); } catch { /* best-effort cleanup */ }

The architects row can survive intentional removal via three concerns, combining:

1. Silent DB write failure (SQLITE_BUSY beyond 5s timeout, IO error) — user sees "Removed architect 'X'." even though the row is intact.
2. WAL fsync lag on OS crash: packages/codev/src/agent-farm/db/index.ts:34-40 opens SQLite with synchronous = NORMAL in WAL mode. On power loss / kernel panic, recently committed transactions in WAL that weren't fsync'd can be lost (SQLite's own docs call this the tradeoff of NORMAL vs FULL).
3. launchInstance's sibling reconciliation loop (tower-instances.ts:687-699) reads getArchitects(resolvedPath) and unconditionally calls addArchitect() for each non-main row. addArchitect then reads the stored session_id and resumes via --resume. No liveness check, no cross-check against terminal_sessions.

Net: any residual architects row (from silent write failure, WAL loss, or a partial removal) causes the sibling to come back on the next Tower start — with its old conversation resumed via --resume, which for a stale session id also triggers the RC2 crash-loop.

Fix sketch — two parts

Part 1: remove the silent-fail in removeArchitect

Location: tower-instances.ts:1174-1180

If either DB delete throws, return { success: false, error } and surface it to the user instead of swallowing. Making the user retry when Tower is happy is worse than a false-positive "removed" that doesn't stick.

Part 2: gate sibling reconciliation on a liveness signal

Location: tower-instances.ts:687-699

Before addArchitect() in the reconcile loop, require either:

• A matching terminal_sessions row for the same (workspace, role_id), OR
• A session_id whose jsonl exists on disk

A row that has neither is a "dead" registration and should be pruned rather than respawned.

Optional (defense in depth)

Switch db/index.ts:40 to synchronous = FULL for architect-write transactions. Small blast radius, low write rate, closes the WAL loss window at OS-crash time.

Scope

• packages/codev/src/agent-farm/servers/tower-instances.ts — removeArchitect (surface DB errors) + sibling reconcile loop (liveness gate)
• Optional: packages/codev/src/agent-farm/db/index.ts (synchronous mode for architect writes)
• Tests:
  • simulate DB write failure during removeArchitect, verify user gets error not success: true
  • simulate architects row with no matching terminal_sessions row, verify reconcile prunes rather than respawns
  • simulate architects row with stale session_id (jsonl missing), verify reconcile prunes

Related

• codev adopt / launchInstance: main architect hijacks an unrelated Claude Code session instead of spawning fresh #1145 (RC1 — buildResume has no ownership check)
• The RC2 bugfix filed alongside this one (Claude crash-loops on stale --resume) — surviving architect rows here often carry stale session_ids that trigger RC2's crash-loop as the visible symptom

[amrmelsayed]

On it! Working on this with the PIR protocol (plan + dev-approval gates before PR).