fix(orchestrator): session audit remediation — error fidelity, session resume, reconcile visibility

[juliusmarminge]

Stacked on #2829. Remediation work from the full v2 session audit (plan + verified findings in .plans/22-orchestration-v2-audit-remediation.md / .plans/22-orchestration-v2-audit-findings.json — every finding was adversarially re-verified against raw provider logs, the event store, and adapter source).

What's in here

• Persist post-turn provider wakeups (plan Codex web app architecture #8): turn.wakeup buffering in ClaudeAdapterV2, ProviderWakeupService relay/dispatcher minting attach_wakeup runs, cross-run backgrounded-Bash completion. The audit quantified the pre-fix loss: 134 tool calls + all wakeup assistant output across 3 threads, including a GitHub comment posted with no user-visible record. Replay fixture claude_provider_wakeup.
• Honor is_error on Claude SDK results (plan oxc stack #1): 401/529 results with subtype: "success" no longer project as completed runs; the failure keeps the API error text and status code. Fixture claude_result_is_error (from real 401 chunks).
• Resume idle-released sessions (plan Images #15): the create-vs-resume decision now consults the persisted provider thread (firstRunOrdinal < runOrdinal) instead of only an in-memory set that dies with the session runtime — the first message after every 30-min idle gap or restart used to burn as an instant failure (11 occurrences across 6 threads). Fixture claude_idle_resume drives the real idle reaper via a new advance_clock step. App-verified: kill backend mid-thread → first message after restart succeeds with resume: on the wire.
• Preserve real failure causes (plan Add open-in-editor feature with Cmd+O shortcut #2): makeProviderFailure walks the wrapper-error cause chain; failed runs no longer persist only "Claude Agent SDK query failed." with the root cause recorded nowhere. App-verified with a broken Claude binary path.
• Cursor failure correlation data (plan Git integration: branch picker + worktrees #3): failed cursor turns persist run id / requestId / duration (the SDK exposes no error text on results — raised with Cursor alongside the EPIPE report).
• Surface reconcile cancellations (plan feat: Github Integration #5): startup/shutdown reconcile appends a visible "Run interrupted" item with the reason instead of silently eating the user's turn. App-verified via kill-mid-turn.
• OpenCode file_search output/error (plan Add native context menu to delete threads #12): contract + adapter keep tool outcomes; failed reads no longer project pattern-only. App-verified with a real OpenCode agent.
• claude_text_segments regression fixture (plan Standardized messages #11): locks per-uuid assistant text segmentation (text → tool → text ordering).

Verification

Each fix carries replay-fixture coverage built from observed native chunks, plus a real-agent verification round in an isolated dev instance (documented per item in the plan file). Remaining audit items (native-log failure frames, codex collab subagent ingestion, ACP subagent lifecycle, codex shared-log routing, event amplification, steering-latency UX) continue on this branch.

🤖 Generated with Claude Code

---

Note

Medium Risk
Touches orchestration v2 event persistence and replay fixtures; coalescing changes stored event granularity (not final projections) and runs timed flush fibers on the ACP session scope.

Overview
Adds orchestration v2 audit artifacts (.plans/22-orchestration-v2-audit-findings.json and .plans/22-orchestration-v2-audit-remediation.md) that catalog verified ingestion/projection gaps and track remediation status across adapters.

Runtime change in this diff: AcpAdapterV2 no longer persists a full message.updated + turn-item.updated pair on every subagent agent_message_chunk. Chunks accumulate and flush on a ~100ms coalescing window, with immediate flush when the subagent task completes or the parent turn finalizes (so tail text is not dropped after the run fiber ends).

Test/replay: ClaudeAdapterV2.testkit treats SDK stream_event frames as valid replay input (for wakeup transcripts where message_start precedes complete assistant messages).

^{Reviewed by Cursor Bugbot for commit 0e0df04. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Add provider-initiated session wakeup, improve error fidelity, and keep background work visible in the timeline

• Implements provider wakeup support in the Claude adapter: background tasks that complete after a turn ends now emit turn.wakeup events, which the new ProviderWakeupService dispatches as attach_wakeup runs once the thread is idle.
• Adds attachTurn to the session runtime so the orchestrator can adopt a provider-initiated turn without sending a new prompt, gated by a new turnDelivery: "attach" field threaded through effect requests.
• Fixes error fidelity: makeProviderFailure now walks nested cause chains; Claude SDK is_error results map to failed status with API error codes; runner failures emit runner.error protocol log frames.
• Recovery now synthesizes a visible Run interrupted turn-item after reconcile-driven cancellations so interrupted runs are auditable.
• Keeps in-progress background work entries visible in the chat timeline after their originating turn settles, showing a pulsing "Running in the background" indicator until the work reaches a terminal state.
• Labels provider wakeup user messages as resumed (not steer) in the UserMessageIntentBadge.
• Adds four new orchestrator replay fixtures (claude_provider_wakeup, claude_idle_resume, claude_result_is_error, claude_text_segments) covering the new and fixed scenarios.

^{Macroscope summarized 0e0df04.}

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: da91b978-c493-42e7-80a7-595330fe6a66

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch t3code/orchestration-v2-audit-fixes

---

_{Comment @coderabbitai help to get the list of available commands.}

[macroscopeapp]

Approvability

Verdict: Needs human review

Diff is too large for automated approval analysis. A human reviewer should evaluate this PR.

^{You can customize Macroscope's approvability policy. Learn more.}

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 3 potential issues.

There are 4 total unresolved issues (including 1 from previous review).

Autofix Details

Bugbot Autofix prepared fixes for all 3 issues found in the latest run.

• ✅ Fixed: Scheduled flush flag not cleared
  • Added subagent.streamFlushScheduled = false in flushSubagentAssistant so that the schedule flag is properly reset on terminal/synchronous flushes, preventing subsequent chunks from hitting the early return.
• ✅ Fixed: Turn end skips subagent flush
  • Added a loop in finalizeTurn that iterates over context.subagents and calls flushSubagentAssistant for any subagent with streamPendingText === true, ensuring coalesced text is persisted before the turn is marked finalized.
• ✅ Resolved by another fix: Concurrent duplicate subagent snapshots
  • The Bug 1 fix (clearing streamFlushScheduled in flushSubagentAssistant) ensures the forked sleep callback sees streamPendingText === false after a synchronous flush, preventing the duplicate emit race.

Or push these changes by commenting:

@cursor push 59569dd592

Preview (59569dd592)

diff --git a/apps/server/src/orchestration-v2/Adapters/AcpAdapterV2.ts b/apps/server/src/orchestration-v2/Adapters/AcpAdapterV2.ts
--- a/apps/server/src/orchestration-v2/Adapters/AcpAdapterV2.ts
+++ b/apps/server/src/orchestration-v2/Adapters/AcpAdapterV2.ts
@@ -915,6 +915,7 @@
             subagent.assistantText += finalText;
           }
           subagent.streamPendingText = false;
+          subagent.streamFlushScheduled = false;
           yield* emitSubagentAssistantSnapshot(subagent);
         });
 
@@ -2054,6 +2055,11 @@
           if (context.finalized) return;
           context.finalized = true;
           yield* closeTextStreams(context);
+          for (const subagent of context.subagents.values()) {
+            if (subagent.streamPendingText) {
+              yield* flushSubagentAssistant(subagent);
+            }
+          }
           const now = yield* DateTime.now;
           const turn = providerTurnPayload(context, status, now);
           yield* Ref.update(providerTurns, (current) => {

_{You can send follow-ups to the cloud agent here.}

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Autofix Details

Bugbot Autofix prepared a fix for the issue found in the latest run.

• ✅ Fixed: Coalesced wakeup requests silently discarded
  • Added a per-thread pendingWakeups map that stores coalesced requests and re-offers them to the relay queue when the in-flight dispatch completes, ensuring no wakeup is permanently lost.

Or push these changes by commenting:

@cursor push f013d3849b

Preview (f013d3849b)

diff --git a/apps/server/src/orchestration-v2/ProviderWakeupService.ts b/apps/server/src/orchestration-v2/ProviderWakeupService.ts
--- a/apps/server/src/orchestration-v2/ProviderWakeupService.ts
+++ b/apps/server/src/orchestration-v2/ProviderWakeupService.ts
@@ -171,9 +171,11 @@
     // Wakeups dispatch concurrently, one in flight per thread: quiescence
     // waiting for one busy thread must not head-of-line-block every other
     // thread's wakeup (which could be superseded while it waits). A wakeup
-    // arriving while its thread already has one in flight is coalesced away —
-    // the adapter buffers all pending activity behind a single attach.
+    // arriving while its thread already has one in flight is held pending;
+    // when the in-flight dispatch finishes it re-offers the pending request
+    // so a failed dispatch does not permanently lose the wakeup.
     const inFlightThreads = yield* Ref.make(new Set<ThreadId>());
+    const pendingWakeups = yield* Ref.make(new Map<ThreadId, ProviderWakeupRequest>());
     return yield* relay.take.pipe(
       Effect.flatMap((input) =>
         Ref.modify(inFlightThreads, (current) => {
@@ -192,16 +194,37 @@
                       const next = new Set(current);
                       next.delete(input.threadId);
                       return next;
-                    }),
+                    }).pipe(
+                      Effect.andThen(
+                        Ref.modify(pendingWakeups, (m) => {
+                          const pending = m.get(input.threadId);
+                          if (pending === undefined) return [undefined, m] as const;
+                          const next = new Map(m);
+                          next.delete(input.threadId);
+                          return [pending, next] as const;
+                        }),
+                      ),
+                      Effect.flatMap((pending) =>
+                        pending !== undefined ? relay.offer(pending) : Effect.void,
+                      ),
+                    ),
                   ),
                   Effect.forkScoped,
                   Effect.asVoid,
                 )
-              : Effect.logInfo("orchestration-v2.provider-wakeup.coalesced", {
-                  threadId: input.threadId,
-                  providerThreadId: input.providerThreadId,
-                  origin: input.origin,
-                }),
+              : Ref.update(pendingWakeups, (m) => {
+                  const next = new Map(m);
+                  next.set(input.threadId, input);
+                  return next;
+                }).pipe(
+                  Effect.andThen(
+                    Effect.logInfo("orchestration-v2.provider-wakeup.coalesced", {
+                      threadId: input.threadId,
+                      providerThreadId: input.providerThreadId,
+                      origin: input.origin,
+                    }),
+                  ),
+                ),
           ),
         ),
       ),

_{You can send follow-ups to the cloud agent here.}

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Bugbot Autofix prepared a fix for the issue found in the latest run.

• ✅ Fixed: Wakeup drain lacks interrupt cleanup
  • Added Effect.onInterrupt handler to the forked drainThread that deletes the thread's entry from threadStates upon fiber interruption, allowing subsequent wakeups to claim and dispatch normally.

Or push these changes by commenting:

@cursor push 1e87251a89

Preview (1e87251a89)

diff --git a/apps/server/src/orchestration-v2/ProviderWakeupService.ts b/apps/server/src/orchestration-v2/ProviderWakeupService.ts
--- a/apps/server/src/orchestration-v2/ProviderWakeupService.ts
+++ b/apps/server/src/orchestration-v2/ProviderWakeupService.ts
@@ -213,7 +213,17 @@
         }).pipe(
           Effect.flatMap((claimed) =>
             claimed
-              ? drainThread(input).pipe(Effect.forkScoped, Effect.asVoid)
+              ? drainThread(input).pipe(
+                  Effect.onInterrupt(() =>
+                    Ref.update(threadStates, (current) => {
+                      const next = new Map(current);
+                      next.delete(input.threadId);
+                      return next;
+                    }),
+                  ),
+                  Effect.forkScoped,
+                  Effect.asVoid,
+                )
               : Effect.logInfo("orchestration-v2.provider-wakeup.follow-up-parked", {
                   threadId: input.threadId,
                   providerThreadId: input.providerThreadId,

_{You can send follow-ups to the cloud agent here.}

^{Reviewed by Cursor Bugbot for commit 2a8052c. Configure here.}