feat(adf): Wave 4 -- handoff persistence, budget gates, compound review swarm, self-learning (#706)

* feat(handoff): add handoff_id, from_agent, to_agent, ttl_secs fields

Refs #58

* feat(handoff): in-memory handoff buffer with TTL sweep

- Add HandoffBuffer struct with HashMap storage of HandoffContext + expiry pairs
- Implement methods: new, insert, get, latest_for_agent, sweep_expired, len, is_empty, iter
- Add handoff_buffer field to AgentOrchestrator with default TTL of 86400s (24h)
- Wire buffer.insert into handoff method and sweep_expired into reconcile_tick
- Add latest_handoff_for public query method to AgentOrchestrator
- Add handoff_buffer_ttl_secs config field to OrchestratorConfig with serde default
- Add 13 comprehensive unit tests for HandoffBuffer functionality
- Update existing test configs to include new field

Refs #59

* feat(config): add budget_monthly_cents to AgentDefinition

Refs #62

* feat(orchestrator): add ScopeRegistry and WorktreeManager for compound review

Add new scope.rs module with:
- ScopeRegistry: HashMap-based lock registry with exclusive/non-exclusive modes
- ScopeReservation: tracks agent file pattern reservations with correlation IDs
- WorktreeManager: git worktree create/remove/cleanup operations

Also fix cost_tracker.rs chrono Datelike import.

Refs #66

* feat(guard): add GuardDecision::Sandbox for suspicious command patterns

Three-valued guard: Allow/Block/Sandbox. New suspicious thesaurus with
9 patterns (curl|sh, sudo, ssh, etc). Priority: allowlist > destructive
> suspicious > default-allow. 16 new tests, 52 guard tests total green.

Refs #64

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(orchestrator): remove unused cost_tracker import

The CostTracker wiring is not yet complete; remove premature import.

* feat(agent): add MCP tool index for self-learning system

- Add McpToolEntry type with serialization, tags, and search capabilities
- Add McpToolIndex for indexing and searching MCP tools
- Use terraphim_automata for fast Aho-Corasick pattern matching
- Implement save/load to JSON for persistence
- Add 10+ tests including latency benchmark (< 50ms for 100 tools)

Refs #69

* feat(orchestrator): implement 6-agent compound review swarm

Rewrite compound.rs with parallel agent dispatch, structured findings,
deduplicated output. 6 review groups: Security, Architecture, Performance,
Quality, Domain, DesignQuality. Includes prompt templates and visual file
detection. 135 orchestrator tests green.

Refs #67

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat(types): add CapturedProcedure for self-learning system

New ProcedureStep, ProcedureConfidence, CapturedProcedure types in
terraphim_types. ProcedureStore in terraphim_agent for JSONL persistence
with Aho-Corasick deduplication via terraphim_automata.

Refs #68

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat(types): add PersonaDefinition and SFIA types for agent personas

Introduce PersonaDefinition, CharacteristicDef, SfiaSkillDef types
in a new persona module within terraphim_types. TOML serialisation
and deserialisation with PersonaLoadError. 10 tests covering roundtrip,
file loading, and error cases.

Refs #71

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat(orchestrator): add persona, provider, and resource fields to AgentDefinition

Add 9 new optional fields to AgentDefinition that the production
orchestrator.toml already declares: persona, terraphim_role,
skill_chain, sfia_skills, provider, fallback_provider, fallback_model,
grace_period_secs, max_cpu_seconds. Add persona_data_dir to
OrchestratorConfig. All fields use serde defaults for backward
compatibility. 8 new config tests.

Refs #70

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat(data): add 8 persona TOML files and metaprompt template

Add structured persona definitions for all 8 ADF agents: Ferrox,
Vigil, Carthos, Lux, Conduit, Meridian, Mneme, Echo. Each file
contains identity metadata, SFIA competency profiles with embedded
skill descriptions, and speech style definitions. Include Handlebars
metaprompt template for runtime override. 10 integration tests
validating all persona files parse and render.

Refs #74

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat(orchestrator): add PersonaRegistry and MetapromptRenderer

Implement PersonaRegistry to load persona TOML files from a
configurable directory, and MetapromptRenderer to render them into
metaprompt preambles via Handlebars templates. Includes default
embedded template and graceful degradation when persona dir is not
configured. Wire into AgentOrchestrator.

Refs #72

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat(orchestrator): inject persona metaprompt via stdin at spawn time

Modify spawn_agent() to compose persona preamble + task when a persona
is configured. Add stdin-based prompt delivery to the spawner to avoid
ARG_MAX limits for large enriched prompts. Graceful degradation: if
persona is not found or rendering fails, the bare task is used.

Refs #73

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat(orchestrator): inject persona identity into compound review prompts

Update all 6 compound review prompt templates with persona identity
preambles: Vigil for security, Carthos for architecture and domain,
Ferrox for quality and performance, Lux for design. Add persona field
to ReviewGroupDef. Persona context improves review quality by giving
each agent a distinctive voice and SFIA-calibrated operating scope.

Refs #75

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(orchestrator): normalise cron expressions to 7-field format

parse_cron now correctly handles 5, 6, and 7-field cron expressions by
appending the year wildcard field. This fixes a crash when agents use
day-of-week expressions like "0 2 * * SUN" which the cron crate
rejects in 6-field format.

Refs #75

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(orchestrator): embed compound review prompts at compile time

Replace runtime std::fs::read_to_string() with include_str!() constants
for all 6 compound review prompt templates. The ADF binary runs from
/opt/ai-dark-factory/ but templates live in the source tree, causing
all nightly compound review agents to fail with No such file or directory.

Fixes #78

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(spawner): Claude CLI OAuth auth and model name normalisation

Three fixes for Claude CLI agent failures in systemd:

1. infer_api_keys() no longer requires ANTHROPIC_API_KEY for Claude CLI
   (it uses OAuth, not API keys -- requiring the key caused validation failure)
2. normalise_claude_model() auto-prepends claude- prefix to versioned names
   like opus-4-6 -> claude-opus-4-6 (short aliases like opus pass through)
3. spawn_process() strips ANTHROPIC_API_KEY from Claude CLI subprocess env
   to prevent inherited values from poisoning OAuth flow

Adds 4 new tests for normalisation and CLI name extraction.

Fixes #76

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(orchestrator): resolve flaky persona spawn test race condition

Two fixes for the intermittent test_spawn_agent_persona_not_found_graceful
failure (DEF-001, ~30% failure rate under parallel load):

1. Track persona_found boolean separately from persona.is_some() --
   use_stdin now only triggers when persona was actually resolved.
   Previously, an unfound persona still triggered stdin delivery,
   causing broken pipe when echo exits before the write completes.

2. Switch test_spawn_agent_with_persona_composes_prompt from echo to cat.
   When persona IS found, stdin delivery is correct, but echo ignores
   stdin and exits instantly. cat reads stdin first, avoiding the race.

10/10 consecutive full-suite runs pass with 0 failures.

Fixes #77

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* style: fix formatting and clippy warnings

- Fix tautological assertions in orchestrator tests
- Apply consistent code formatting across modified files

* fix(clippy): resolve all clippy warnings

Clippy fixes:
- Remove duplicate binary target from terraphim-session-analyzer
- Fix needless borrows for generic args
- Fix large enum variants (box AgentDefinition)
- Remove unused BrokenPersona struct
- Fix unnecessary map_or to is_some_and
- Fix expect_fun_call to unwrap_or_else
- Fix unused variable in test_orchestrator_compound_review_manual
- Fix unused import in procedure.rs
- Gate ProcedureStore to test-only since only used in tests

All clippy warnings now resolved with proper implementations.

* fix(orchestrator): address code review findings from issue #708

Critical fixes:
- C-1: Fix failing tests -- use empty groups for test isolation
- C-2: Add validate_agent_name() to prevent path traversal in handoff paths
- C-3: Convert WorktreeManager to async (tokio::process::Command + tokio::fs)
- C-4: Change extract_review_output fallback from pass:true to pass:false

Important fixes:
- I-1: Replace 1s inner timeout with deadline-based timeout_at
- I-3: Remove #[cfg(test)] from ProcedureStore::new()
- I-4: Remove async from ProcedureStore methods that only use std::fs
- I-5: Remove unused scope_registry field and all #[allow(dead_code)]
- I-6: Fix u64-to-i64 TTL overflow -- cap at 100 years
- I-7: Add context field validation in handoff
- I-8: Fix overlaps() false positives with path-separator-aware check

Additional:
- env_remove GIT_INDEX_FILE from spawned git subprocesses so worktree
  and diff operations work correctly during pre-commit hooks
- WorktreeManager.with_base() respects worktree_root config
- mpsc channel buffer clamped to min 1 for empty group configs

Ref: #708

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: add research and design documents for issue #708

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: sync beads

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(orchestrator): correct substitute_env doc comment (#708)

* fix: resolve CI failures in terraphim_tracker and terraphim_agent

- terraphim_tracker/src/linear.rs: remove unused jiff::Zoned import
- terraphim_tracker/tests/linear_integration.rs: replace assert!(true) with comment
- terraphim_agent/procedure.rs: add justification for dead_code on default_path()
- Cargo.lock: update after rebase on main

These are pre-existing issues introduced by recent Linear tracker integration
that were blocking PR #706 from merging.

* fix(ci): stop deleting cargo git checkouts and registry cache in PR workflow

The disk cleanup step was removing ~/.cargo/git/checkouts/* and
~/.cargo/registry/cache/*, which broke compilation of the self_update
git dependency (patched fork for zipsign-api v0.2). Jobs without a
cargo cache restore step (clippy, tests) could not re-fetch the git
checkout, causing "No such file or directory (os error 2)" on the
self_update build script.

Refs #58

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(orchestrator): handle shallow clones in compound review test

CI uses fetch-depth: 1 so HEAD~1 does not exist. Detect this by
running `git rev-parse --verify HEAD~1` first and fall back to
diffing against the empty tree hash when the parent is missing.

Refs #58

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* style: cargo fmt

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Alex <alex@example.com>

Author: 3 people

.beads/issues.jsonl

@@ -34,6 +34,7 @@
 {"id":"terraphim-ai-27u.5","title":"Step 5 session tools and orchestration runway","description":"Implement session tools and sequence spawn/cron runway extending existing issue #560.","notes":"Starting Step 5 implementation after Step 4 commit.","status":"closed","priority":2,"issue_type":"task","owner":"alex@example.com","created_at":"2026-02-27T09:37:45.176541222Z","created_by":"Alex","updated_at":"2026-02-27T11:28:43.805846286Z","closed_at":"2026-02-27T11:28:43.805846286Z","close_reason":"Step 5 implemented: session tools + shared runtime wiring + agent-mode outbound dispatch + spawn baseline + tests","external_ref":"gh-591","dependencies":[{"issue_id":"terraphim-ai-27u.5","depends_on_id":"terraphim-ai-27u","type":"parent-child","created_at":"2026-02-27T09:37:45.178159014Z","created_by":"Alex"}]}
 {"id":"terraphim-ai-2sz","title":"Add embedded device settings fallback to terraphim-cli","description":"Evaluate and implement an embedded DeviceSettings fallback (similar to terraphim-agent) so terraphim-cli doesn't fail on missing settings.","status":"open","priority":2,"issue_type":"task","owner":"alex@metacortex.engineer","created_at":"2026-02-10T08:23:48.689656434Z","created_by":"AlexMikhalev","updated_at":"2026-02-10T08:23:48.689656434Z"}
 {"id":"terraphim-ai-8ld","title":"Rewrite compress() with proxy-first fallback","status":"closed","priority":1,"issue_type":"bug","owner":"alex@example.com","created_at":"2026-02-26T16:28:28.794633105Z","created_by":"Alex","updated_at":"2026-02-26T19:07:04.510972897Z","closed_at":"2026-02-26T19:07:04.510972897Z","close_reason":"Closed"}
+{"id":"terraphim-ai-a79","title":"Fix code review findings for issue #708","status":"closed","priority":1,"issue_type":"task","owner":"alex@example.com","created_at":"2026-03-24T08:59:34.580180696Z","created_by":"Alex","updated_at":"2026-03-24T11:10:43.389957272Z","closed_at":"2026-03-24T11:10:43.389957272Z","close_reason":"All critical and important findings from #708 fixed"}
 {"id":"terraphim-ai-a7x","title":"Implement TinyClaw #594 cron orchestration tool","description":"Add cron tool registration, scheduler dispatch, persistence, and integration tests.","status":"closed","priority":2,"issue_type":"task","owner":"alex@example.com","created_at":"2026-02-27T12:52:39.654990116Z","created_by":"Alex","updated_at":"2026-02-27T12:52:53.968220449Z","closed_at":"2026-02-27T12:52:53.968220449Z","close_reason":"Implemented and verified in this session","external_ref":"gh-594"}
 {"id":"terraphim-ai-aac","title":"Implement TinyClaw #560 terraphim_spawner-backed agent_spawn","description":"Replace baseline subprocess spawning with terraphim_spawner integration and config wiring.","status":"closed","priority":2,"issue_type":"task","owner":"alex@example.com","created_at":"2026-02-27T12:52:39.653484632Z","created_by":"Alex","updated_at":"2026-02-27T12:52:53.990077587Z","closed_at":"2026-02-27T12:52:53.990077587Z","close_reason":"Implemented and verified in this session","external_ref":"gh-560"}
 {"id":"terraphim-ai-cbm","title":"Clarify terraphim-agent TUI offline/server requirement","description":"Determine whether terraphim-agent TUI is expected to work fully offline or requires a running server; document requirement and adjust behavior if needed.","design":"Phase 1/2 docs: docs/plans/terraphim-agent-tui-offline-server-research-2026-02-13.md and docs/plans/terraphim-agent-tui-offline-server-design-2026-02-13.md","acceptance_criteria":"Contract for fullscreen TUI vs REPL/offline is explicit in help/docs; actionable messaging when fullscreen TUI server is unreachable; tests cover mode behavior to prevent regressions.","notes":"Implemented on 2026-02-13: mode-contract wording in CLI/docs, fullscreen TUI server preflight with actionable repl fallback, and regression tests for help/non-TTY/server-failure paths. Validation: cargo fmt --package terraphim_agent; cargo clippy -p terraphim_agent --all-targets -- -D warnings; cargo test -p terraphim_agent --test offline_mode_tests; cargo test -p terraphim_agent --test server_mode_tests test_server_mode_config_show; targeted unit tests in main.rs for URL resolution and error messaging.","status":"closed","priority":2,"issue_type":"task","owner":"alex@metacortex.engineer","created_at":"2026-02-10T08:23:40.310825316Z","created_by":"AlexMikhalev","updated_at":"2026-02-23T10:46:13.066528719Z","closed_at":"2026-02-13T14:41:42.09313609Z"}