Draft
Conversation
Add IVS (Interactive Video Service) as an alternative transport to WebRTC. Users select transport via `transport: "ivs"` in connect options — both paths return the same RealTimeClient interface. New files: transport-manager.ts (shared interface), ivs-connection.ts, ivs-manager.ts. Updated client.ts for transport selection, methods.ts to accept the shared interface, and WebRTCManager to implement it. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add transport dropdown (WebRTC/IVS) to the test page. When IVS is selected, the IVS Web Broadcast SDK is loaded from CDN. The chosen transport is passed to client.realtime.connect(). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The bouncer's IVS handler waits for `ivs_joined` as the first client message after sending `ivs_stage_ready`. The message pump (which handles set_image/prompt) only starts after `ivs_joined` is received. Sending set_image before the stage handshake caused the bouncer to read it instead of `ivs_joined`, rejecting with "Expected ivs_joined message". Fix: reorder IVS connection phases so stage setup completes first, then send initial image/prompt once the bouncer's message pump is running. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The IVS SDK's StageStrategy callbacks take (participant) not (stage, participant). This caused TypeError: Cannot read properties of undefined (reading 'isLocal'). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
IVS SDK v1.14.0 does not pass participant to shouldPublishParticipant or shouldSubscribeToParticipant. Use argument-free callbacks instead: - publish: always true (only called for local publish-eligible participants) - subscribe: always AUDIO_VIDEO (subscribe to all remote streams) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
SDK uses pnpm — package-lock.json was generated by mistake. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add transport-aware subscribe flow so viewers can watch IVS sessions without consuming inference server resources (SFU handles fan-out). - Add optional transport field to subscribe token encoding - Add subscribeIVS path: fetches viewer token from bouncer, creates subscribe-only IVS stage - Export getIVSBroadcastClient and IVSBroadcastModule for reuse Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The realtime client's baseUrl is a WebSocket URL (wss://), but the IVS subscribe endpoint is an HTTP GET. Convert the protocol before calling fetch. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The subscribe stage was subscribing to ALL non-local participants, including the client's own camera feed. Now uses client_publish_participant_id from bouncer's ivs_stage_ready message to skip the client's publish participant and only receive the server's processed output. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Use server_publish_participant_id from the subscribe-ivs response to only subscribe to the server's inference output stream, preventing the viewer from accidentally receiving the client's camera input. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add requestRTCStats to IVS type declarations, store remote/local stage streams in IVSConnection, and expose them via getter methods proxied through IVSManager. Streams are cleared on cleanup. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move delta-tracking state and parse logic into a reusable StatsParser class so it can be shared between WebRTCStatsCollector and the upcoming IVSStatsCollector. No external API change. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Polls IVS stage streams via requestRTCStats() at 1s intervals and feeds merged RTCStatsReport into StatsParser, mirroring the WebRTCStatsCollector pattern. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Wire IVSStatsCollector into client.ts alongside WebRTC stats collection. Both transports now get stats events, video stall detection, and telemetry reporting. Add transport tag to telemetry reports for backend filtering. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Required for IVSManager's public getRemoteStreams/getLocalStreams methods to be type-checkable by downstream consumers. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Two new latency tracking approaches for real-time streams: - Composite RTT: stitches client/server STUN RTTs + pipeline latency - Pixel Marker: embeds seq number in output frame pixels for true E2E measurement Both work across WebRTC and IVS transports, gated by client connect options. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Consolidate CompositeLatencyTracker + PixelLatencyProbe setup/teardown into a single pluggable LatencyDiagnostics class, reducing ~35 lines of inline wiring in client.ts to a simple instantiate/wire/stop pattern. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
IVS publish stage was only sending video tracks and WHIP publish was missing the audio output track, causing audio to be silently dropped — a regression from existing WebRTC behavior. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… start Reject remoteStreamPromise when IVS subscribe stage disconnects during setup, preventing the connect() flow from hanging forever. Store and clear the latency diagnostics delayed-start timer on disconnect to avoid restarting diagnostics after cleanup. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
With high pipeline delays (17-22s, e.g. packet loss + high FPS), probes were TTL-expired before the stamped frame reached the client, causing the measurement to silently drop and report stale low values instead of the actual delay. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Client burns binary seq marker into input frame pixels via canvas processing → server reads and re-stamps → client reads output and computes true round-trip latency through the video path. Changes: - pixel-latency-stamper.ts: canvas-based input frame stamper that wraps camera MediaStreamTrack (draw camera → stamp marker → captureStream) - pixel-latency.ts: E2E mode (stamp input + read output), stats tracking (sent/received/lost/corrupted/outOfOrder), periodic e2e_latency_report - latency-diagnostics.ts: createStamper() wraps localStream, deferred probe creation to start() for stamper availability - client.ts: create stamper before manager.connect() to substitute the published stream when pixelMarker is enabled - types.ts: add E2ELatencyReportMessage, update OutgoingMessage unions Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The canvas stamper was created but not started until 1s after manager.connect(). IVS received a 300x150 canvas track producing zero frames, causing input_video_fps: 0.0 on the server. - Initialize canvas dimensions from track settings in constructor - Start draw loop in createStamper() before returning the stream - Remove redundant stamper.start() from LatencyDiagnostics.start() Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…p bias) Add pixelLatencyEvent and pixelLatencyReport events that surface corrupted, lost, and out-of-order stamps — not just successful ones. - Add PixelLatencyEvent discriminated union (ok, ok_reordered, corrupted, lost) - Add PixelLatencyReport periodic aggregate with pending count - Refactor PixelLatencyProbe constructor to options object (fixes broken onStats wiring) - Emit events for all outcomes in readFrame() and cleanUpOldProbes() - Existing pixelLatency event unchanged for backwards compat Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Read nackCount from inbound-rtp stats and compute per-sample delta. NACK rate > 0 with packet loss = 0 means the proxy cache is recovering losses before they count as lost. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1. Send delta (not cumulative) lost/corrupted/outOfOrder in e2e_latency_report to prevent Datadog double-counting 2. Wrap seq counter at 16 bits to match stamp encoding, preventing false losses after ~36h sessions 3. Start stats collectors when latencyTracking is configured, not only when telemetryEnabled — fixes composite latency RTT=0 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@aws/ivs-web-broadcast doesn't exist on npm. The correct package is amazon-ivs-web-broadcast. Fixes CI frozen-lockfile failure. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…idatePairs Report interval now fires in E2E mode even without sendMessage, so the onReport callback receives periodic updates for local display. Re-add selectedCandidatePairs to WebRTCStats connection type and parser for ICE candidate display in rt-tester. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ing enabled Server (PR #749) gates EstimatedE2ELatencyReporter behind the latency_diagnostics query param. Without this, the server never sends latency_report messages and composite latency stops working. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…rve source FPS The canvas + rAF + captureStream() pipeline ran at requestAnimationFrame rate (~120fps in headless Chrome), overwhelming the server regardless of the source camera's actual frame rate. Replace with MediaStreamTrackProcessor/Generator (Chrome 94+): - 1-in-1-out frame processing preserves source FPS naturally - 99% of frames pass through unchanged (zero copy, no quality loss) - Only stamped frames (~every 2s) go through OffscreenCanvas - Canvas approach kept as fallback for non-Chrome environments Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Allows callers to pass arbitrary query parameters (e.g. enable_recording) to the bouncer/upstream WS URL without SDK changes for each new param. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Zod 4 requires both key and value schemas for z.record(). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Status
🚧 Work in progress — do not merge
🤖 Generated with Claude Code