feat: add GitHub events watcher, durable wake-queue, and done-crewmate backstop

[e-jung]

What Changed

• Add fm-github-watch.sh, a GitHub events watcher for PR comments, CI state, and reviews, hardened with concurrent PR polling within the 30s check budget, debounced single-event CI-state transitions, transient-API-error skipping, fork-PR CI roll-up, and atomic seen-state writes
• Make the supervision loop crash-safe with a durable wake-queue and singleton watcher lock (fm-wake-lib.sh, fm-wake-drain.sh, fm-watch.sh), plus fm-guard.sh warnings that surface a stale/missing watcher beacon or pending queued wakes on any fleet-touching script
• Add the done-crewmate check plugin with an fm-plugin.sh lifecycle manager as a recurring backstop for done/failed/blocked crewmates left in live tmux windows, and support batch task dispatch (multiple id=repo pairs) in fm-spawn.sh

Risk Assessment

✅ Low: The round 1 portability fix is correct and complete, and the surrounding plugin/lifecycle code is well-bounded and read-only, with no new material risks introduced.

Testing

Exercised the done-crewmate guard through the watcher's real invocation path (symlinked state/*.check.sh with a fake tmux) and the full fm-plugin.sh lifecycle; every intended behavior holds — recurring wake on done+alive, silence when healthy/progressed, needs-decision exclusion, multi-offender coalescing, no-clobber of per-task checks, and portable FM_ROOT resolution through the symlink. All 16 harness assertions pass and the three existing test suites are green; no failures.

Evidence: done-guard operator transcript

=== done-crewmate guard: operator transcript (fresh-clone fleet root) === --- 1. bootstrap re-arms plugins: fm-plugin.sh sync recreates the watcher glob --- symlink: .../fleet/bin/check-plugins/done-crewmate.check.sh done-crewmate live --- 2. healthy fleet: crewmate still working, window alive -> check is silent --- output: <empty -> watcher keeps sleeping> --- 3. crewmate reports done but firstmate has not torn it down -> RECURRING WAKE --- output: done crewmate ship-1 still alive in tmux - progress or tear down -> watcher wraps as: check: state/done-crewmate.check.sh: ... --- 4. second offender lands; both surface in ONE wake line --- output: done crewmate scout-2 ship-1 still alive in tmux - progress or tear down --- 5. firstmate progresses the work -> check goes silent --- after ship-1 window gone, output: done crewmate scout-2 still alive in tmux ... after scout-2 resumes working, output: <empty -> not idle-done> === portable root resolution (head-commit fix): watcher invokes the SYMLINK === bare-symlink-path invocation (no FM_ROOT_OVERRIDE) output: done crewmate port still alive in tmux - progress or tear down

=== done-crewmate guard: operator transcript (fresh-clone fleet root) ===
fleet root: /tmp/done-guard-demo.DlIZCg/fleet

--- 1. bootstrap re-arms plugins: fm-plugin.sh sync recreates the watcher glob ---
symlink: /tmp/done-guard-demo.DlIZCg/fleet/bin/check-plugins/done-crewmate.check.sh
done-crewmate	live

--- 2. healthy fleet: crewmate still working, window alive -> check is silent ---
watcher runs: bash $STATE/done-crewmate.check.sh
output: <empty -> watcher keeps sleeping>

--- 3. crewmate reports done but firstmate has not torn it down -> RECURRING WAKE ---
watcher runs the same check on its next CHECK_INTERVAL tick:
output: done crewmate ship-1 still alive in tmux - progress or tear down
  -> watcher wraps as: check: state/done-crewmate.check.sh: done crewmate ship-1 still alive in tmux - progress or tear down

--- 4. second offender lands; both surface in ONE wake line (single backstop tick) ---
output: done crewmate scout-2 ship-1 still alive in tmux - progress or tear down

--- 5. firstmate progresses the work (tears down the window) -> check goes silent ---
after ship-1 window gone, output: done crewmate scout-2 still alive in tmux - progress or tear down
after scout-2 resumes working, output: <empty -> not idle-done>

=== portable root resolution (head-commit fix): watcher invokes the SYMLINK ===
BASH_SOURCE would be the state/ symlink path; script must resolve FM_ROOT without readlink -f (GNU-only).
bare-symlink-path invocation (no FM_ROOT_OVERRIDE) output: done crewmate port still alive in tmux - progress or tear down

=== done ===

Evidence: verify-done-guard harness (16/16 pass)

Stage 1 (fm-plugin lifecycle): sync creates symlink -> list reports live -> sync preserves real per-task state file -> rejects reserved 'fm-' name -> rejects invalid name. Stage 2 (behavior): A working->silent, B done+alive->wakes, C done+window-gone->silent, D failed+blocked flagged, D needs-decision excluded, E resumed working->silent, F no-trailing-newline parsed, G meta without window= skipped, H empty state->silent. Stage 3 (portable root): one-level readlink resolves FM_ROOT through symlink; end-to-end watcher-style invocation wakes correctly. Summary: PASS=16 FAIL=0

#!/usr/bin/env bash
# End-to-end verification of the done-crewmate guard + fm-plugin.sh lifecycle.
# Mirrors the real watcher path: it globs $STATE/*.check.sh and runs
# `bash "$c"`, where state/done-crewmate.check.sh is a symlink to the tracked
# canonical copy under bin/check-plugins/. A fake tmux stands in for real tmux.
set -u

# Worktree root: passed in via DONE_GUARD_REPO, else the caller's CWD.
REPO="${DONE_GUARD_REPO:-$PWD}"
[ -f "$REPO/bin/check-plugins/done-crewmate.check.sh" ] || {
  echo "error: worktree root not found (DONE_GUARD_REPO=$REPO)"; exit 2; }

echo "### worktree root: $REPO"
echo

PASS=0; FAIL=0
assert_eq() { # <desc> <expected> <actual>
  if [ "$2" = "$3" ]; then
    echo "ok - $1"; PASS=$((PASS+1))
  else
    echo "not ok - $1"; echo "      expected: [$2]"; echo "      actual:   [$3]"; FAIL=$((FAIL+1))
  fi
}
assert_silent() { # <desc> <actual-output>
  if [ -z "$2" ]; then echo "ok - $1 (silent, as required)"; PASS=$((PASS+1))
  else echo "not ok - $1 (expected silence, got output)"; echo "      got: [$2]"; FAIL=$((FAIL+1)); fi
}

SCRATCH="$(mktemp -d /tmp/done-guard.XXXXXX)"
trap 'rm -rf "$SCRATCH"' EXIT
FLEET="$SCRATCH/fleet"
mkdir -p "$FLEET/bin/check-plugins" "$FLEET/state"

# Reuse the real plugin + lifecycle scripts by symlinking (read-only, no copy edits).
ln -s "$REPO/bin/check-plugins/done-crewmate.check.sh" "$FLEET/bin/check-plugins/done-crewmate.check.sh"
ln -s "$REPO/bin/fm-plugin.sh" "$FLEET/bin/fm-plugin.sh"

# Fake tmux: prints whatever windows we stage in $FAKE_WINDOWS (one per line).
mkdir -p "$SCRATCH/fakebin"
cat > "$SCRATCH/fakebin/tmux" <<'SH'
#!/usr/bin/env bash
set -u
if [ "${1:-}" = "list-windows" ]; then
  printf '%s\n' "$DONE_GUARD_WINDOWS" 2>/dev/null | grep . || true
fi
SH
chmod +x "$SCRATCH/fakebin/tmux"
export PATH="$SCRATCH/fakebin:$PATH"

run_check_via_symlink() {
  # Exactly how the watcher invokes it: `bash "$STATE/<name>.check.sh"` from FM_ROOT.
  ( cd "$FLEET" && FM_ROOT_OVERRIDE="$FLEET" bash "$FLEET/state/done-crewmate.check.sh" 2>/dev/null )
}

echo "### Stage 1: fm-plugin.sh lifecycle (add/list/sync/remove)"
echo
cd "$FLEET"

# sync should create the state symlink for the canonical plugin.
bin/fm-plugin.sh sync
assert_eq "sync creates state/done-crewmate.check.sh symlink pointing at canonical" \
  "$(readlink "$FLEET/state/done-crewmate.check.sh")" \
  "$FLEET/bin/check-plugins/done-crewmate.check.sh"

# list should report it live.
out="$(bin/fm-plugin.sh list)"
case "$out" in
  *done-crewmate*live*) echo "ok - list reports done-crewmate live"; PASS=$((PASS+1)) ;;
  *) echo "not ok - list did not report live: [$out]"; FAIL=$((FAIL+1)) ;;
esac

# sync must NOT clobber a real per-task state file that happens to share a name.
echo "echo real per-task check" > "$FLEET/state/keep-real.check.sh"
# Make a fake canonical plugin so sync would try to link it.
cat > "$FLEET/bin/check-plugins/keep-real.check.sh" <<'SH'
#!/usr/bin/env bash
echo "SHOULD NOT REPLACE"
SH
bin/fm-plugin.sh sync
if [ -L "$FLEET/state/keep-real.check.sh" ]; then
  echo "not ok - sync clobbered a real per-task state file"; FAIL=$((FAIL+1))
else
  echo "ok - sync preserved real per-task state file (not a symlink)"; PASS=$((PASS+1))
fi
# cleanup the synthetic plugin so it doesn't affect the done-crewmate behavior.
rm -f "$FLEET/state/keep-real.check.sh" "$FLEET/bin/check-plugins/keep-real.check.sh"

# name validation: reject names that would collide with task ids (fm-*) and bad chars.
if bin/fm-plugin.sh add 'fm-bad' "$REPO/bin/check-plugins/done-crewmate.check.sh" 2>/dev/null; then
  echo "not ok - add accepted reserved 'fm-' name"; FAIL=$((FAIL+1))
else echo "ok - add rejected reserved 'fm-' name"; PASS=$((PASS+1)); fi
if bin/fm-plugin.sh add 'bad name!' "$REPO/bin/check-plugins/done-crewmate.check.sh" 2>/dev/null; then
  echo "not ok - add accepted invalid name"; FAIL=$((FAIL+1))
else echo "ok - add rejected invalid name"; PASS=$((PASS+1)); fi
echo

echo "### Stage 2: done-crewmate check behavior (the watcher's recurring backstop)"
echo
unset FM_ROOT_OVERRIDE  # the script will resolve root via the symlink path itself

mk_meta() { # <id> <window>
  printf 'window=%s\nproject=demo\nharness=claude\n' "$2" > "$FLEET/state/$1.meta"
}
mk_status() { # <id> <lines...>
  : > "$FLEET/state/$1.status"
  for line in "$@"; do printf '%s\n' "$line" >> "$FLEET/state/$1.status"; done
}

# --- Scenario A: healthy fleet, crewmate still working -> silent.
DONE_GUARD_WINDOWS="" mk_meta taskA "mysess:fm-taskA"
mk_status taskA "working: implementing"
DONE_GUARD_WINDOWS="" assert_silent \
  "A: working crewmate -> silent" "$(DONE_GUARD_WINDOWS="" run_check_via_symlink)"

# --- Scenario B: done + window alive -> wakes with offender id.
mk_meta taskB "mysess:fm-taskB"
mk_status taskB "done: PR https://example.com/pull/1 checks green"
out="$(DONE_GUARD_WINDOWS="mysess:fm-taskB" run_check_via_symlink)"
case "$out" in
  *taskB*alive*progress*or*tear*down*) echo "ok - B: done+alive -> wakes naming offender"; PASS=$((PASS+1)) ;;
  *) echo "not ok - B: unexpected output: [$out]"; FAIL=$((FAIL+1)) ;;
esac

# --- Scenario C: done but window already torn down -> silent (work progressed).
out="$(DONE_GUARD_WINDOWS="" run_check_via_symlink)"
assert_silent "C: done + window gone -> silent" "$out"

# --- Scenario D: failed/blocked are terminal too; needs-decision is NOT.
mk_meta taskD1 "mysess:fm-d1"; mk_status taskD1 "failed: tests red"
mk_meta taskD2 "mysess:fm-d2"; mk_status taskD2 "blocked: waiting on X"
mk_meta taskD3 "mysess:fm-d3"; mk_status taskD3 "needs-decision: pick A or B"
out="$(DONE_GUARD_WINDOWS="mysess:fm-d1
mysess:fm-d2
mysess:fm-d3" run_check_via_symlink)"
case "$out" in
  *taskD1*taskD2*) echo "ok - D: failed+blocked flagged"; PASS=$((PASS+1)) ;;
  *) echo "not ok - D: failed/blocked not flagged: [$out]"; FAIL=$((FAIL+1)) ;;
esac
case "$out" in
  *taskD3*) echo "not ok - D: needs-decision was flagged (should be excluded)"; FAIL=$((FAIL+1)) ;;
  *) echo "ok - D: needs-decision excluded"; PASS=$((PASS+1)) ;;
esac

# --- Scenario E: resumed working AFTER done -> silent (not idle-done).
mk_status taskD1 "done: ..." "working: follow-up"
out="$(DONE_GUARD_WINDOWS="mysess:fm-d1" run_check_via_symlink)"
assert_silent "E: resumed working after done -> silent" "$out"

# --- Scenario F: status file with no trailing newline still parsed (last line wins).
printf 'working: x\ndone: shipped' > "$FLEET/state/taskD2.status"  # no trailing newline
out="$(DONE_GUARD_WINDOWS="mysess:fm-d2" run_check_via_symlink)"
case "$out" in
  *taskD2*) echo "ok - F: missing-trailing-newline status still detected as done"; PASS=$((PASS+1)) ;;
  *) echo "not ok - F: expected taskD2 offender, got: [$out]"; FAIL=$((FAIL+1)) ;;
esac

# --- Scenario G: meta with no window= line is skipped (cannot cross-ref tmux).
printf 'project=demo\n' > "$FLEET/state/nowin.meta"
printf 'done: x\n' > "$FLEET/state/nowin.status"
out="$(DONE_GUARD_WINDOWS="mysess:fm-nowin" run_check_via_symlink)"
case "$out" in
  *nowin*) echo "not ok - G: meta without window= was flagged"; FAIL=$((FAIL+1)) ;;
  *) echo "ok - G: meta without window= skipped"; PASS=$((PASS+1)) ;;
esac

# --- Scenario H: no .meta / no .status at all -> silent.
rm -f "$FLEET/state/"*.meta "$FLEET/state/"*.status
out="$(DONE_GUARD_WINDOWS="mysess:fm-x" run_check_via_symlink)"
assert_silent "H: empty state dir -> silent" "$out"
echo

echo "### Stage 3: portable root resolution (the head-commit fix)"
echo
# The watcher invokes the SYMLINK. The script must resolve through it to find
# FM_ROOT without `readlink -f` (GNU-only). Prove it works without -f and even
# when BASH_SOURCE is the bare symlink path (relative, cwd = FM_ROOT).
cd "$FLEET"
# Sanity: confirm the system readlink here does NOT need -f for our purpose.
resolved_root="$(FM_ROOT_OVERRIDE= bash -c '
  src="state/done-crewmate.check.sh"
  real="$(readlink "$src")" && case "$real" in /*) src="$real" ;; *) src="$(cd -P "$(dirname "$src")" && pwd)/$real";; esac
  d="$(cd -P "$(dirname "$src")" && pwd)"
  for root in "$d/../.." "$d/.."; do [ -d "$root/bin" ] && [ -d "$root/state" ] && { (cd -P "$root" && pwd); exit; }; done
')"
if [ "$resolved_root" = "$FLEET" ]; then
  echo "ok - portable readlink (one-level) resolves FM_ROOT through the symlink"; PASS=$((PASS+1))
else
  echo "not ok - root resolution failed: got [$resolved_root]"; FAIL=$((FAIL+1))
fi

# Verify the actual script output, invoked exactly as the watcher does it (no override).
printf 'window=mysess:fm-portable\nproject=demo\n' > "$FLEET/state/portable.meta"
printf 'done: ship it\n' > "$FLEET/state/portable.status"
out="$(DONE_GUARD_WINDOWS="mysess:fm-portable" bash "$FLEET/state/done-crewmate.check.sh" 2>/dev/null)"
case "$out" in
  *portable*) echo "ok - end-to-end: watcher-style invocation through symlink wakes correctly"; PASS=$((PASS+1)) ;;
  *) echo "not ok - end-to-end invocation produced no wake (root resolution failed in practice): [$out]"; FAIL=$((FAIL+1)) ;;
esac

echo
echo "### Summary: PASS=$PASS FAIL=$FAIL"
[ "$FAIL" -eq 0 ]

Pipeline

Updates from git push no-mistakes

⏭️ **intent** - skipped

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

🔧 **Review** - 1 issue found → auto-fixed ✅

• ⚠️ bin/check-plugins/done-crewmate.check.sh:31 - fm_root() uses readlink -f (line 31), which stock macOS readlink does not support. This is inconsistent with the rest of the codebase, which explicitly handles Darwin for stat in fm-watch.sh, fm-wake-lib.sh, and fm-guard.sh. When readlink -f fails, src is left as the unresolved state/ symlink path, so d resolves to .../state; the first fallback branch (line 33, $d/../..) then climbs TWO levels (it assumes the resolved bin/check-plugins/ location) and returns the PARENT of FM_ROOT instead of FM_ROOT. STATE ends up pointing at a non-existent directory, and line 40 ([ -d "$STATE" ] || exit 0) silently no-ops the entire backstop check. The common case (watcher armed from FM_ROOT) is rescued by the cwd check on line 29, so this only bites on macOS when the watcher's cwd is not FM_ROOT — but there it defeats the whole purpose of the check (forgotten done/failed/blocked crewmates go uncaught). Fix: resolve the symlink portably without readlink -f (e.g. cd -P "$(dirname "$src")" and read the link target iteratively), or drop the two-level branch and rely on the cwd check plus a single-level resolve.

🔧 Fix: Fix done-crewmate check portable root resolution
✅ Re-checked - no issues remain.

✅ **Test** - passed

✅ No issues found.

• bash /tmp/no-mistakes-evidence/01KWBAPZAPNPYV3Z1K1DB1ZGBY/verify-done-guard.sh — 16 assertion harness: fm-plugin.sh add/list/sync/remove, name validation, no-clobber of real per-task state files, and 9 done-crewmate behavior scenarios invoked exactly as the watcher does (bash $STATE/done-crewmate.check.sh via its symlink, with a fake tmux), plus portable root resolution through the symlink without FM_ROOT_OVERRIDE
• Operator transcript via watcher-style invocation: healthy fleet silent; done+alive wakes; failed+blocked flagged, needs-decision excluded; second offender coalesces into one line; tearing down the window / resuming working: silences the backstop; bare-symlink-path invocation resolves FM_ROOT and wakes (head-commit fix)
• tests/fm-github-watch.test.sh
• tests/fm-spawn-batch.test.sh
• tests/fm-wake-queue.test.sh

✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.