try webrtc again

.gitignore

@@ -173,3 +173,5 @@ dist
 
 # Finder (MacOS) folder config
 .DS_Store
+
+webrtc-debug.log

docs/webrtc_qr_signaling_plan.md

@@ -0,0 +1,113 @@
+# WebRTC with QR Code Signaling Implementation Plan
+
+This document outlines the plan and current implementation for enabling WebRTC communication between a PWA frontend (hosted on GitHub Pages or locally) and a local Node.js server, using QR codes for the initial signaling handshake.
+
+## I. Project Goals
+
+1.  Enable direct peer-to-peer communication between the PWA and the local server over the LAN.
+2.  Use WebRTC data channels for message exchange.
+3.  Implement a "zero-config" signaling mechanism using QR codes, where the server displays QR codes containing its offer, and the PWA scans them to establish a connection.
+4.  The PWA is designed to be hostable on GitHub Pages (HTTPS, enabling camera access) or run locally.
+5.  The local server runs on the user's machine, using a modular TypeScript codebase.
+
+## II. Core Components & Technologies
+
+*   **Frontend (PWA, `src/`):**
+    *   React (TypeScript) app in `src/App.tsx`.
+    *   HTML, CSS, TypeScript.
+    *   Browser's native `RTCPeerConnection` API.
+    *   QR Code Scanning Library: [`html5-qrcode`](https://github.com/mebjas/html5-qrcode) (used in the React PWA).
+    *   Legacy/alternative implementation: `src/client.ts` (jQuery + Quagga, for barcode scanning, not QR signaling).
+*   **Backend (`server/`):**
+    *   Node.js, TypeScript.
+    *   Modular structure: `server/index.tsx` (entry point), `server/webrtc.ts` (WebRTC and signaling logic), `server/ui.tsx` (Ink UI), `server/helpers.ts`.
+    *   WebRTC Library: `node-datachannel`.
+    *   QR Code Generation Library: `qrcode` (for terminal QR display).
+    *   Terminal UI: [Ink](https://github.com/vadimdemedes/ink) for interactive display and logs.
+
+## III. Implementation Steps
+
+### A. Server-Side (`server/`)
+
+1.  **Dependencies:**
+    *   Uses `node-datachannel` for WebRTC capabilities.
+    *   Uses `qrcode` for generating QR codes in the terminal.
+    *   Uses Ink for terminal UI.
+2.  **WebRTC Initialization:**
+    *   On startup, creates a `PeerConnection` instance using `node-datachannel`.
+    *   Creates a data channel (named "barbarcode-channel").
+3.  **Generate Offer & ICE Candidates:**
+    *   Creates an SDP offer and gathers local ICE candidates.
+    *   The server acts as the "offerer."
+4.  **QR Code Generation & Display:**
+    *   Combines the SDP offer and ICE candidates into a single JSON string.
+    *   **Chunking Data for QR Codes:**
+        *   If the JSON string is too large for one QR code, splits it into manageable chunks.
+        *   Each chunk is a JSON object: `{ part: number, length: number, data: string }`, with `data` base64-encoded.
+    *   Generates QR codes for each chunk and displays them sequentially in the terminal using Ink.
+    *   Loops QR code display until a connection is established.
+5.  **Handle Client's Answer:**
+    *   Once the PWA scans the offer and connects, it sends its SDP answer and ICE candidates back through the established data channel.
+    *   The server listens on the data channel for this answer and sets the remote description.
+6.  **Data Channel Communication:**
+    *   Implements logic to send and receive messages over the data channel once the connection is fully established.
+    *   Handles barcode data sent from the client and executes keystroke patterns as configured.
+
+### B. Client-Side (PWA - `src/App.tsx`)
+
+1.  **QR Code Scanning UI:**
+    *   Provides a button/UI element to initiate QR code scanning.
+    *   Uses `getUserMedia` to access the device camera.
+    *   Integrates `html5-qrcode` to scan QR codes from the camera feed.
+    *   Also supports manual QR chunk input for testing.
+2.  **Data Reassembly:**
+    *   As QR codes are scanned, parses the `{ part, length, data }` structure.
+    *   Collects all parts until all chunks are received.
+    *   Reconstructs the complete JSON string (server's offer and ICE candidates).
+3.  **WebRTC Initialization:**
+    *   Creates an `RTCPeerConnection` instance.
+    *   Uses the server's ICE candidates when creating the connection.
+4.  **Process Offer & Create Answer:**
+    *   Sets the remote description using the server's SDP offer.
+    *   Creates an SDP answer.
+    *   Gathers local ICE candidates for the PWA.
+5.  **Send Answer to Server:**
+    *   Once the `RTCPeerConnection`'s data channel (initiated by the server's offer) becomes available, sends the PWA's SDP answer and its ICE candidates as a JSON string through this data channel.
+6.  **Data Channel Communication:**
+    *   Implements logic to send and receive messages over the data channel.
+
+**Note:** The legacy `src/client.ts` is focused on barcode scanning and WebSocket communication, not QR-based WebRTC signaling.
+
+## IV. Signaling Flow (QR Code Method)
+
+1.  **Server:** Starts, generates WebRTC offer + ICE candidates.
+2.  **Server:** Encodes offer/ICE into chunked QR codes: `{ part: x, length: y, data: chunk_data }`.
+3.  **Server:** Displays QR codes sequentially in the terminal (Ink UI).
+4.  **PWA:** User initiates scanning. PWA uses camera to read QR codes.
+5.  **PWA:** Reassembles the full offer/ICE data from the chunks.
+6.  **PWA:** Initializes `RTCPeerConnection`, sets remote description (server's offer), creates an answer.
+7.  **PWA:** Once the data channel (defined in server's offer) is open, sends its answer + ICE candidates to the server over this channel.
+8.  **Server:** Receives PWA's answer/ICE via data channel, sets remote description.
+9.  **Connection Established:** Secure, bidirectional WebRTC data channel is now active.
+
+## V. Key Considerations & Potential Challenges
+
+*   **QR Code Data Capacity:** Chunking logic is implemented; base64 encoding is used for the `data` field in chunks.
+*   **QR Code Display on Server:** ASCII QR codes are displayed in the terminal using Ink.
+*   **PWA QR Scanning UX:** The React PWA provides clear instructions, error handling, and feedback during sequential scanning.
+*   **ICE Candidate Gathering:** The implementation ensures ICE candidates are gathered before signaling proceeds.
+*   **Error Handling:** Robust error handling is implemented for WebRTC state changes, data channel messages, and QR code processing.
+*   **Security:** WebRTC connections are encrypted. The QR code transfer is an "out-of-band" signaling method.
+*   **Library Versions:** Compatibility between `node-datachannel` and browser WebRTC is maintained.
+
+## VI. Current Status & Next Steps
+
+- The server-side WebRTC logic, QR code generation, and Ink UI are implemented in `server/`.
+- The client-side React PWA with QR scanning and WebRTC logic is implemented in `src/App.tsx`.
+- Legacy barcode scanning (not QR signaling) is available in `src/client.ts`.
+- Testing and UX refinement are ongoing.
+
+**Next Steps:**
+1.  Continue testing the full QR-based signaling flow on local networks and refine UX.
+2.  Expand data channel communication features as needed.
+3.  Update documentation and deployment instructions as the implementation evolves.

package.json

@@ -2,9 +2,9 @@
   "name": "barbarcode",
   "version": "0.1.1",
   "type": "module",
-  "main": "dist/server.cjs",
+  "main": "dist/server/index.cjs",
   "bin": {
-    "barbarcode": "./dist/server.cjs"
+    "barbarcode": "./dist/server/index.cjs"
   },
   "files": [
     "dist/",
@@ -13,9 +13,10 @@
   "scripts": {
     "clean": "rm -rf dist",
     "build:server": "tsc -p tsconfig.app.json",
+    "dev:server": "tsx server/index.tsx",
     "build:landing": "vite build --config landing.vite.config.ts",
     "build": "vite build && pnpm run build:server",
-    "start": "node dist/server.cjs",
+    "start": "node dist/server/index.cjs",
     "dev": "vite",
     "dev:landing": "vite --config landing.vite.config.ts",
     "lint": "biome lint .",
@@ -26,12 +27,21 @@
     "typescript": "^5.0.0"
   },
   "dependencies": {
+    "@types/react": "^18.3.21",
+    "@types/react-dom": "^18.3.7",
+    "@vitejs/plugin-react": "^4.4.1",
     "commander": "^12.1.0",
     "express": "^4.19.2",
+    "html5-qrcode": "^2.3.8",
+    "ink": "^5.2.1",
     "jquery": "^3.7.1",
     "ngrok": "5.0.0-beta.2",
+    "node-datachannel": "^0.27.0",
     "qr": "^0.4.2",
+    "qrcode": "^1.5.4",
     "quagga": "^0.12.1",
+    "react": "^18.3.1",
+    "react-dom": "^18.3.1",
     "robotjs": "^0.6.0",
     "toml": "^3.0.0",
     "ws": "^8.18.0"
@@ -41,6 +51,7 @@
     "@types/express": "^5.0.1",
     "@types/jquery": "^3.5.32",
     "@types/node": "^22.15.17",
+    "@types/qrcode": "^1.5.5",
     "@types/ws": "^8.18.1",
     "autoprefixer": "^10.4.21",
     "biome": "^0.3.3",

patches/node-datachannel@0.27.0.patch

@@ -0,0 +1,148 @@
+diff --git a/dist/cjs/lib/index.cjs b/dist/cjs/lib/index.cjs
+index fbdf1e53243a9a875f05c2185c30ef0f58386eb1..a769e613b4c7ee15f8611228a330037a76b8fbab 100644
+--- a/dist/cjs/lib/index.cjs
++++ b/dist/cjs/lib/index.cjs
+@@ -13,6 +13,9 @@ function preload() {
+ function initLogger(level) {
+   nodeDatachannel.default.initLogger(level);
+ }
++function setLogCallback(cb) {
++  nodeDataChannel.default.setLogCallback(cb);
++}
+ function cleanup() {
+   nodeDatachannel.default.cleanup();
+ }
+@@ -31,6 +34,7 @@ const RtcpReceivingSession = nodeDatachannel.default.RtcpReceivingSession;
+ const DataChannelStream = datachannelStream.default;
+ var n = {
+   initLogger,
++  setLogCallback,
+   cleanup,
+   preload,
+   setSctpSettings,
+diff --git a/dist/esm/lib/index.mjs b/dist/esm/lib/index.mjs
+index b6367360080cae5a204d13600c5912264663d809..09162a86cab63a64588a6313a5fec963ca7a2443 100644
+--- a/dist/esm/lib/index.mjs
++++ b/dist/esm/lib/index.mjs
+@@ -9,6 +9,9 @@ function preload() {
+ function initLogger(level) {
+   nodeDataChannel.initLogger(level);
+ }
++function setLogCallback(cb) {
++  nodeDataChannel.setLogCallback(cb);
++}
+ function cleanup() {
+   nodeDataChannel.cleanup();
+ }
+@@ -27,6 +30,7 @@ const RtcpReceivingSession = nodeDataChannel.RtcpReceivingSession;
+ const DataChannelStream = DataChannelStream$1;
+ var n = {
+   initLogger,
++  setLogCallback,
+   cleanup,
+   preload,
+   setSctpSettings,
+diff --git a/dist/types/lib/index.d.ts b/dist/types/lib/index.d.ts
+index b4e80eb86096d12d4da6d2dffdb7ec09295ee609..c98fccbf3006fc731a53a59f7f573ef6d10007e2 100644
+--- a/dist/types/lib/index.d.ts
++++ b/dist/types/lib/index.d.ts
+@@ -6,6 +6,7 @@ import { WebSocket } from './websocket.js';
+
+ declare function preload(): void;
+ declare function initLogger(level: LogLevel): void;
++declare function setLogCallback(cb: (level: number, message: string) => void): void;
+ declare function cleanup(): void;
+ declare function setSctpSettings(settings: SctpSettings): void;
+ declare function getLibraryVersion(): string;
+@@ -152,6 +153,7 @@ declare const RtcpReceivingSession: {
+ declare const DataChannelStream: typeof DataChannelStream$1;
+ declare const _default: {
+     initLogger: typeof initLogger;
++    setLogCallback: typeof setLogCallback;
+     cleanup: typeof cleanup;
+     preload: typeof preload;
+     setSctpSettings: typeof setSctpSettings;
+diff --git a/src/cpp/rtc-wrapper.cpp b/src/cpp/rtc-wrapper.cpp
+index 341b56e7065067ec64dff685ada44512b5b8806a..e280b6281e7165114a1cc2f0d331c6446885530e 100644
+--- a/src/cpp/rtc-wrapper.cpp
++++ b/src/cpp/rtc-wrapper.cpp
+@@ -21,6 +21,7 @@ Napi::Object RtcWrapper::Init(Napi::Env env, Napi::Object exports)
+     Napi::HandleScope scope(env);
+
+     exports.Set("initLogger", Napi::Function::New(env, &RtcWrapper::initLogger));
++    exports.Set("setLogCallback", Napi::Function::New(env, &RtcWrapper::setLogCallback));
+     exports.Set("cleanup", Napi::Function::New(env, &RtcWrapper::cleanup));
+     exports.Set("preload", Napi::Function::New(env, &RtcWrapper::preload));
+     exports.Set("setSctpSettings", Napi::Function::New(env, &RtcWrapper::setSctpSettings));
+@@ -116,6 +117,37 @@ void RtcWrapper::initLogger(const Napi::CallbackInfo &info)
+     }
+ }
+
++void RtcWrapper::setLogCallback(const Napi::CallbackInfo &info)
++{
++    Napi::Env env = info.Env();
++    if (info.Length() < 1 || !info[0].IsFunction()) {
++        Napi::TypeError::New(env, "Function expected").ThrowAsJavaScriptException();
++        return;
++    }
++    // store new ThreadSafeCallback
++    logCallback = std::make_unique<ThreadSafeCallback>(info[0].As<Napi::Function>());
++
++    // re-register with libdatachannel
++    rtcInitLogger(RTC_LOG_DEBUG,
++      // C-API callback: runs on libdatachannel’s thread
++      [](rtcLogLevel level, const char* cstr) {
++          // Make your own copy so it isn’t freed when this function returns
++          std::string msg(cstr);
++          if (logCallback) {
++              logCallback->call(
++                // JS callback: runs on Node’s main thread
++                [level, msg = std::move(msg)](Napi::Env env, std::vector<napi_value>& args) {
++                    args = {
++                      Napi::Number::New(env, (int)level),
++                      Napi::String::New(env, msg)
++                    };
++                }
++              );
++          }
++      }
++    );
++}
++
+ void RtcWrapper::cleanup(const Napi::CallbackInfo &info)
+ {
+     PLOG_DEBUG << "cleanup() called";
+diff --git a/src/cpp/rtc-wrapper.h b/src/cpp/rtc-wrapper.h
+index 55590796f668bc88816a195fe48dcb4e56f2a59e..01c5beefb8fbeb15536f49d09d056818d9ce18ac 100644
+--- a/src/cpp/rtc-wrapper.h
++++ b/src/cpp/rtc-wrapper.h
+@@ -16,6 +16,7 @@ public:
+     static Napi::Object Init(Napi::Env env, Napi::Object exports);
+     static void preload(const Napi::CallbackInfo &info);
+     static void initLogger(const Napi::CallbackInfo &info);
++    static void setLogCallback(const Napi::CallbackInfo &info);
+     static void cleanup(const Napi::CallbackInfo &info);
+     static void setSctpSettings(const Napi::CallbackInfo &info);
+     static Napi::Value getLibraryVersion(const Napi::CallbackInfo &info);
+diff --git a/src/lib/index.ts b/src/lib/index.ts
+index 67c149eaf75f98af588f5a95674e3539a27615bb..44537f64f1345a3d9d25b06cbf9621ba85414708 100644
+--- a/src/lib/index.ts
++++ b/src/lib/index.ts
+@@ -6,6 +6,9 @@ import { WebSocket } from './websocket';
+
+ export function preload(): void { nodeDataChannel.preload(); }
+ export function initLogger(level: LogLevel): void { nodeDataChannel.initLogger(level); }
++export function setLogCallback(cb: (level: number, message: string) => void): void {
++  nodeDataChannel.setLogCallback(cb);
++}
+ export function cleanup(): void { nodeDataChannel.cleanup(); }
+ export function setSctpSettings(settings: SctpSettings): void { nodeDataChannel.setSctpSettings(settings); }
+ export function getLibraryVersion(): string { return nodeDataChannel.getLibraryVersion(); }
+@@ -159,6 +162,7 @@ export const DataChannelStream = _DataChannelStream;
+
+ export default {
+     initLogger,
++    setLogCallback,
+     cleanup,
+     preload,
+     setSctpSettings,