Investigate SDK premature session.idle — upstream fix or permanent workaround

[PureWeen]

Context

The Copilot SDK/CLI emits premature session.idle events during long tool-executing turns, causing multi-agent workers to be collected with truncated responses. PR #375 added an elaborate workaround (RecoverFromPrematureIdleIfNeededAsync) using:

1. ManualResetEventSlim signal set by re-arm events after premature idle
2. events.jsonl mtime freshness detection (< 15s = CLI still writing)
3. Multi-round recovery loop with bestResponse accumulation
4. 120s recovery timeout

Why this is a hack

• Relies on filesystem side-channel (file mtime) to determine SDK state
• Normal completions can stall ~15s while freshness detection runs
• Multi-round loop adds complexity and edge cases (OCE handling, bestResponse scoping)
• The root cause is the SDK emitting session.idle before the turn is actually complete

Additional finding: backgroundTasks field is inconsistently populated (Mar 2025)

The SessionIdleEvent.Data.BackgroundTasks field (agents/shells arrays) is not reliably populated when sub-agents are running. Within the same worker session, some session.idle events arrive with empty backgroundTasks while others correctly list running agents. This causes:

1. session.idle arrives with empty backgroundTasks → IDLE-DEFER logic does NOT trigger → CompleteResponse fires → IsProcessing=false (premature)
2. Sub-agent finishes → new TurnStartEvent arrives → EVT-REARM re-sets IsProcessing=true
3. Cycle repeats multiple times per worker turn (observed 3x in a single turn)

Evidence from PR Review Squad worker-5 (2026-03-25):

13:26:02 [EVT] worker-5 TurnStart (IsProcessing=True)     ← sub-agents launched
13:29:08 [EVT] worker-5 TurnEnd (IsProcessing=False)      ← premature idle fired, no backgroundTasks
13:29:08 [EVT-REARM] worker-5 re-arming IsProcessing      ← sub-agent finished, new turn
13:31:18 [EVT] worker-5 TurnEnd (IsProcessing=False)      ← premature idle AGAIN
13:31:18 [EVT-REARM] worker-5 re-arming IsProcessing
13:34:23 [EVT] worker-5 TurnEnd (IsProcessing=False)      ← premature idle AGAIN
13:34:23 [EVT-REARM] worker-5 re-arming IsProcessing
13:35:08 [IDLE-DEFER] worker-5 session.idle with active    ← THIS time backgroundTasks IS populated
         background tasks — deferring completion

The EVT-REARM mechanism recovers correctly every time, so no data is lost — but the UI spinner flickers (see #395) and diagnostics are confusing.

Proposed SDK fixes (either would resolve this)

1. Don't emit session.idle while background tasks are active. The SDK clearly tracks them (it populates the field sometimes). Hold the idle event until agents/shells are truly empty.
2. Add a turn ID / correlation token to session.idle. Consumers could match idle events to the prompt that triggered them, distinguishing real completion from stale/premature idle.

Option 1 eliminates the entire class of premature idle bugs. Option 2 is more general and helps with other edge cases too.

Proposed investigation

1. File upstream issue with Copilot SDK team documenting the premature idle behavior
2. Request turn-scoped terminal events or turn IDs so consumers can distinguish real vs premature idle
3. If upstream fix is not forthcoming, evaluate whether the workaround can be simplified (e.g., just use turn ID matching)

Priority

Medium — the workaround works but adds resilience debt in the most critical orchestration path.

[PureWeen]

Update: SDK fix landed + reframing (April 2)

SDK fix

PureWeen/copilot-sdk#2 fixes the sendAndWait premature resolution when backgroundTasks is active. PolyPilot already has its own IDLE-DEFER logic for this.

Reframing

The backgroundTasks inconsistency described in this issue (sometimes populated, sometimes empty) remains an SDK/CLI issue. The EVT-REARM mechanism handles it correctly but causes UI flicker (#395).

New finding: session.idle is ephemeral by design

The SDK marks session.idle as ephemeral: true — it is intentionally NOT written to events.jsonl. This means:

• The "zero-idle" cases in this issue are real bugs (live stream fails to deliver the event)
• But any approach relying on reading session.idle from events.jsonl is fundamentally flawed — it was never supposed to be there

The proposed SDK fixes in this issue (option 1: hold idle until background tasks empty, option 2: turn ID) are still valid and the right path forward.

[PureWeen]

Update: Cannot reproduce backgroundTasks inconsistency (April 2)

Added [IDLE-DIAG] instrumentation to log the raw backgroundTasks payload on every SessionIdleEvent. Results from CLI 1.0.16-1:

Run 1 (1 background agent):

17:35:45 session.idle backgroundTasks={agents=1, shells=0}  <- correctly reports 1 agent
17:36:08 session.idle backgroundTasks=null                   <- correctly reports done

Run 2 (3 background agents):

17:36:52 session.idle backgroundTasks={agents=3, shells=0}  <- all 3 running
17:37:04 session.idle backgroundTasks={agents=1, shells=0}  <- 2 finished, 1 running
17:37:24 session.idle backgroundTasks=null                   <- all done

The CLI correctly and consistently populates backgroundTasks. The count decreases as agents finish. The final idle has null backgroundTasks.

The inconsistency described in this issue body (from March 25) may have been:

1. Fixed in a newer CLI version since then
2. A PolyPilot parsing issue in HasActiveBackgroundTasks() that was subsequently fixed
3. A timing issue where the subagent completed between the idle being generated and PolyPilot checking the field

The core SDK issue (sendAndWait resolving on idle with backgroundTasks) is still valid and addressed by copilot-sdk#2. But the "inconsistent population" claim cannot be verified on current CLI.

[PureWeen]

Investigation update (PR #472 work):

After extensive instrumentation ([IDLE-DIAG] logging added to SessionIdleEvent handler), we confirmed that the CLI correctly populates the backgroundTasks field on session.idle events. Agent counts decrement as agents finish, and the final idle (with empty backgroundTasks) arrives reliably.

The real issue was on the PolyPilot side: IDLE-DEFER was a one-shot mechanism. The first idle with backgroundTasks correctly deferred, but subsequent idles (after agents finished) found IsProcessing=false and became no-ops. Fixed in PR #472 with IDLE-DEFER-REARM.

Cannot reproduce the originally-reported backgroundTasks inconsistency (CLI reporting wrong agent counts). Across 208 real-world IDLE-DEFER events, backgroundTasks was always accurate.

Leaving open for now in case the inconsistency resurfaces, but the practical symptoms are resolved.