fix(mooncake): retry transient transfer failures and make session-kill threshold configurable

[DavidBellamy]

Summary

Two small, additive changes to mooncake disaggregation that together make PD sessions resilient to transient transfer errors.

1. Retry engine.batch_transfer_sync on non-zero return before reporting failure to the polling loop. Mooncake itself retries WC errors per slice, but handshake RPC failures and endpoint resets under load currently bubble straight up. Knobs:

   • SGLANG_DISAGG_TRANSFER_RETRIES (default 3)
   • SGLANG_DISAGG_TRANSFER_RETRY_BACKOFF_MS (default 50)
   • On recovery, logs a warning naming the session and attempt count.
   • On persistent failure, logs session id, slice count, total bytes, and first src/dst/len before returning the original non-zero ret to the caller.

2. Make the session-failure kill threshold configurable via SGLANG_DISAGG_SESSION_FAILURE_THRESHOLD (default 10, was effectively 1). Without this, a single transient hiccup outside the retry window still permakills the session.

Why

Observed under sustained PD disaggregation load on an H200 cluster: transient handshake / endpoint-reset failures take a session out for the rest of the run, even though the underlying connection recovers within milliseconds. The two changes give the session a chance to recover both within a single transfer (loop) and across transfers (threshold).

Behavior

Defaults change behavior, intentionally:

• Retries: 1 → 3. Latency cost on a successful transfer is zero (early return on ret==0).
• Threshold: 1 → 10. Restoring the prior fail-fast mode is one env var: SGLANG_DISAGG_SESSION_FAILURE_THRESHOLD=1.

No new dependencies, no API changes, no changes to call sites outside mooncake/conn.py.

Test plan

• Verified across multiple long-running PD disagg jobs on an H200 cluster (R3-off agentic RL stack); recovered-on-retry warnings observed in the wild without any session ending up in failed_sessions.
• Reviewer to confirm the env-var defaults are acceptable for upstream-style behavior, or request that defaults be reverted to (1, 1) with opt-in via env var.

[DavidBellamy]

Superseded by #16 — same two commits (fix(mooncake): retry transient batch_transfer_sync failures + feat(mooncake): make session-failure kill threshold configurable) on a clean branch off the production pin (0ca02195), without the rebase noise that this PR has accumulated.

Closing as superseded.