Add audio/video pass-through protocol and infrastructure

[Copilot]

Enables forwarding of controller's local webcam/microphone to controlled computer as virtual devices.

Changes

Protocol Extensions (libs/hbb_common/protos/message.proto)

• Added ClientAudioFrame, ClientVideoFrame for client→server media streams
• Added PassThroughRequest/PassThroughResponse for device lifecycle management
• Extended Message union with 4 new variants (field numbers 33-36)

Client Infrastructure (src/client/media_passthrough.rs)

pub struct MediaPassThroughManager {
    message_sender: UnboundedSender<Data>,
    audio_active: Arc<Mutex<bool>>,
    video_active: Arc<Mutex<bool>>,
}

• Device lifecycle management (start/stop capture)
• Request/response handling
• Stubbed capture threads (platform-specific implementation needed)

Server Infrastructure (src/server/connection.rs)

• Message handlers: handle_passthrough_request, handle_client_audio_frame, handle_client_video_frame
• Integrated into connection message routing
• Stubbed virtual device routing (platform-specific implementation needed)

Architecture

Controller: Device Capture → Encode → ClientAudioFrame/ClientVideoFrame → Network
Controlled: Network → Decode → Virtual Device → Applications

Remaining Work

Platform-specific implementation required:

• Audio/video capture (WASAPI/PulseAudio/CoreAudio, Media Foundation/V4L2/AVFoundation)
• Virtual device creation (drivers/kernel modules on Windows/Linux, system extensions on macOS)
• UI controls in Flutter client

Documentation:

• README_PASSTHROUGH.md - Architecture and quick start
• PASSTHROUGH_IMPLEMENTATION.md - Platform-specific code examples
• IMPLEMENTATION_SUMMARY.md - Implementation roadmap

Note

Protobuf changes in libs/hbb_common submodule require separate handling (see PROTOCOL_CHANGES_NOTE.md). Code compiles with stubs; platform-specific capture/virtual device code needed for functionality.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• chromium.googlesource.com
  • Triggering command: /usr/lib/git-core/git-remote-https /usr/lib/git-core/git-remote-https REDACTED REDACTED (dns block)
  • Triggering command: /usr/lib/git-core/git-remote-https /usr/lib/git-core/git-remote-https REDACTED REDACTED ep/bin/linux-x64origin (dns block)
  • Triggering command: /usr/lib/git-core/git-remote-https /usr/lib/git-core/git-remote-https REDACTED REDACTED -collect-logs (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

Original prompt

I want to add features to this application. This is a remote desktop application, and I would like to add functionality so that, when a user is remoting into a client, it's able to pass through audio and video.

For example, on computer A I want to connect remotely to computer B. Since I'm not physically there, I could not use a webcam or a microphone. If I wanted to, I want the functionality to be able to turn on my camera or use my camera and my microphone on the computer that I am remoting from, computer A, and pass through that feed into computer B so that it's able to use the video and audio. It gets automatically set to it when it is possible. I'm not sure if setting that setting for the microphone and webcam is possible, but just look into it and try your best.

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.