From 134e2a4697596d4da8973127c4a13b064cd49a04 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 27 Jun 2026 13:35:32 +0000 Subject: [PATCH 1/2] tests: self-building OXT runtime harness for the public API Adds tests/torrent-selftest.livecodescript - the OXT-pass companion to the C++ smoke test. The smoke test proves the shim against libtorrent in CI; this proves the .lcb BINDING actually runs (the foreign-decl signatures, the record walkers, the Data<->Pointer buffer round-trips) - the one layer CI cannot reach, since OXT has no headless way to compile or run .lcb. Paste into a one-card stack's STACK script and close+reopen: it self-builds a UI, starts a session, and runs ~70 checks, then shows a green/red PASS/FAIL list with a summary. It exercises 71 of the 75 public handlers, including round-trips that prove end-to-end wiring: - create + seed a tiny temp torrent -> btFileList returns the file table - btAddWebSeed -> btWebSeeds lists it -> btRemoveWebSeed -> gone - btAddTracker -> btTrackers lists it - btInfoHash -> btFindTorrent recovers the same handle - btSessionPause -> btSessionIsPaused true -> btSessionResume - btDhtKeypair determinism; btDhtPutImmutable -> 40-hex target The four not auto-checked (btMoveStorage, btSetFilePriorities, btAddTorrentWithResume, plain btAddMagnet) are destructive / async / binary-buffer cases, noted in-app for a manual pass. The whole run is wrapped in try/catch, so a mis-declared foreign handler is REPORTED (which call, what error) instead of silently halting the stack - exactly the failure this harness exists to catch, and it doubles as a regression tool for future ABI bumps. check-livecodescript.py now also lints tests/ by default (so CI gates the harness too); verified statically (5 files OK, ASCII-clean, balanced handlers + try/catch, all 70 distinct bt* calls resolve to real public handlers). Needs an OXT run to exercise it for real - that is its whole point. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01371AXB4CUUke7enHHS8okc --- tests/torrent-selftest.livecodescript | 418 ++++++++++++++++++++++++++ tools/check-livecodescript.py | 2 +- 2 files changed, 419 insertions(+), 1 deletion(-) create mode 100644 tests/torrent-selftest.livecodescript diff --git a/tests/torrent-selftest.livecodescript b/tests/torrent-selftest.livecodescript new file mode 100644 index 0000000..ea9b292 --- /dev/null +++ b/tests/torrent-selftest.livecodescript @@ -0,0 +1,418 @@ +script "torrentSelfTest" + +-- torrent-selftest.livecodescript - a self-building RUNTIME harness that drives +-- the TorrentXT public API end to end in OXT and reports PASS / FAIL per handler. +-- This is the OXT-pass companion to the C++ smoke test: the smoke test proves the +-- shim against libtorrent in CI; this proves the .lcb BINDING (the foreign-decl +-- signatures, the record walkers, the buffer round-trips) actually runs - the one +-- layer CI cannot reach, since OXT has no headless way to compile or run .lcb. +-- +-- HOW TO USE: open OXT, make a one-card stack, paste THIS into the STACK script +-- (Object > Stack Script), then close+reopen. It builds its own UI, starts a +-- TorrentXT session, runs ~70 checks, and shows a green/red list. Needs the loaded +-- org.openxtalk.library.torrent (v8+). Run it STANDALONE - only one TorrentXT +-- session may be live per OXT process, so close any client/demo stack first. +-- +-- WHAT IT CHECKS: every non-destructive public handler returns a sane value and +-- the round-trips work (create+seed -> file table; add web seed -> list it -> +-- remove it; add tracker -> list it; info-hash -> find-by-hash; pause -> is-paused; +-- keypair determinism; immutable put -> target hash). It does NOT assert async DHT +-- / tracker results (those return later as events) and it SKIPS the two genuinely +-- destructive handlers (btMoveStorage, btRemoveTorrent-with-delete) - those are +-- noted for a manual pass. A green run means the whole binding surface is wired. + +-- ---- state ------------------------------------------------------------------- +local sSession -- the live session handle (0 = none) +local sTorrent -- a torrent we create + seed, for torrent-scoped checks +local sTorrent2 -- a second torrent from the extended-add check +local sTotal, sPassed, sFailed +local sResultText -- the accumulated PASS/FAIL/section lines +local sTempDir -- parent dir we seed from +local sContentDir -- the content folder inside it + +-- ---- lifecycle --------------------------------------------------------------- +on openStack + stBuild + stRun +end openStack + +on closeStack + stCleanup +end closeStack + +on mouseUp + if the short name of the target is "stRerun" then + stCleanup + stRun + else + pass mouseUp + end if +end mouseUp + +-- ---- self-building UI (idempotent) ------------------------------------------- +command stBuild + if there is a field "stResults" then + exit stBuild + end if + lock screen + set the width of this stack to 660 + set the height of this stack to 620 + set the backgroundColor of this card to "237,239,243" + set the title of this stack to "TorrentXT - API Self-Test" + stMakeLabel "stTitle", " TorrentXT - API Self-Test", "0,0,660,30", 15 + set the opaque of field "stTitle" to true + set the backgroundColor of field "stTitle" to "44,90,160" + set the foregroundColor of field "stTitle" to "255,255,255" + set the textStyle of field "stTitle" to "bold" + stMakeLabel "stSummary", "Running...", "16,38,520,66", 13 + set the textStyle of field "stSummary" to "bold" + stMakeButton "stRerun", "Re-run", "566,38,644,64" + stMakeList "stResults", "16,72,644,604" + set the textFont of field "stResults" to "Courier" + set the textSize of field "stResults" to 11 + unlock screen +end stBuild + +command stMakeLabel pName, pText, pRect, pSize + if there is no field pName then + create field + set the name of the last field to pName + end if + set the rect of field pName to pRect + set the lockText of field pName to true + set the borderWidth of field pName to 0 + set the opaque of field pName to false + set the textSize of field pName to pSize + put pText into field pName +end stMakeLabel + +command stMakeButton pName, pLabel, pRect + if there is no button pName then + create button + set the name of the last button to pName + end if + set the rect of button pName to pRect + set the label of button pName to pLabel +end stMakeButton + +command stMakeList pName, pRect + if there is no field pName then + create field + set the name of the last field to pName + end if + set the rect of field pName to pRect + set the lockText of field pName to true + set the vScrollbar of field pName to true +end stMakeList + +-- ---- assertion plumbing ------------------------------------------------------ +command stAssert pName, pPass + add 1 to sTotal + if pPass then + add 1 to sPassed + put "PASS " & pName & return after sResultText + else + add 1 to sFailed + put "FAIL " & pName & return after sResultText + end if +end stAssert + +command stSection pTitle + put return & "--- " & pTitle & " ---" & return after sResultText +end stSection + +command stNote pMsg + put " " & pMsg & return after sResultText +end stNote + +-- count elements of a bridged list (avoids relying on `the number of elements`) +function stCount pList + local tN, tEl + put 0 into tN + repeat for each element tEl in pList + add 1 to tN + end repeat + return tN +end stCount + +-- is pNeedle one of the URL strings in the list pList? +function stListHas pList, pNeedle + local tEl + repeat for each element tEl in pList + if tEl is pNeedle then + return true + end if + end repeat + return false +end stListHas + +-- ---- the run ----------------------------------------------------------------- +command stRun + put 0 into sTotal + put 0 into sPassed + put 0 into sFailed + put "" into sResultText + put 0 into sTorrent + put 0 into sTorrent2 + + stSection "session + ABI" + -- btStartSession is the one handler that can throw (on ABI skew); a clean + -- start IS the ABI check passing. + local tStartOk + put false into tStartOk + try + put btStartSession() into sSession + put true into tStartOk + catch tErr + put 0 into sSession + end try + stAssert "btStartSession did not throw (ABI matches)", tStartOk + if sSession is 0 or sSession is empty then + stNote "Could not start a session - is another TorrentXT stack open?" + stNote (btLastError()) + stPaint + exit stRun + end if + stAssert "btStartSession returns a positive handle", (sSession > 0) + stAssert "btLastError is a string", (btLastError() is a string) + + -- Wrap the rest: a mis-declared foreign handler (wrong signature) raises a + -- runtime error on its first call. Catch it so the harness REPORTS which call + -- broke and still shows a summary, instead of halting the stack mid-run. + try + stSection "settings" + stAssert "btSetBool enable_dht", (btSetBool(sSession, "enable_dht", true) is 0) + stAssert "btSetBool enable_lsd", (btSetBool(sSession, "enable_lsd", true) is 0) + stAssert "btSetInt download_rate_limit", (btSetInt(sSession, "download_rate_limit", "0") is 0) + stAssert "btSetString user_agent", (btSetString(sSession, "user_agent", "TorrentXT-selftest") is 0) + stAssert "btGetSetting user_agent non-empty", (btGetSetting(sSession, "user_agent") is not "") + stAssert "btSetEncryption", (btSetEncryption(sSession, 1, 1, 3) is 0) + stAssert "btSetInt rejects unknown key", (btSetInt(sSession, "no_such_setting_key", "1") < 0) + + stSection "errors" + btClearError + stAssert "btClearError empties last-error", (btLastError() is "") + + stSection "DHT bootstrap + state" + stAssert "btDhtAddBootstrap", (btDhtAddBootstrap(sSession, "router.bittorrent.com", 6881) is 0) + local tDht + put btDhtState(sSession) into tDht + stAssert "btDhtState returns an array with nodes", (tDht["nodes"] is a number) + local tDhtState + put btDhtSaveState(sSession) into tDhtState + stAssert "btDhtSaveState returns a measurable Data", (the number of bytes in tDhtState is a number) + stAssert "btDhtLoadState round-trips", (btDhtLoadState(sSession, tDhtState) is 0) + + stSection "create + seed (sets up a torrent for the rest)" + stMakeContent + local tTorrentData + put btCreateTorrent(sContentDir, 0, 0, "udp://tracker.opentrackr.org:1337/announce") into tTorrentData + stAssert "btCreateTorrent returns metainfo bytes", (the number of bytes in tTorrentData > 0) + if the number of bytes in tTorrentData > 0 then + put btAddTorrentFile(sSession, tTorrentData, sTempDir) into sTorrent + end if + stAssert "btAddTorrentFile returns a handle", (sTorrent > 0) + stAssert "btTorrentCount >= 1", (btTorrentCount(sSession) >= 1) + stAssert "btTorrentHandleAt(0) is a handle", (btTorrentHandleAt(sSession, 0) > 0) + + if sTorrent > 0 then + stRunTorrentChecks tTorrentData + else + stNote "Skipping torrent-scoped checks: no seed torrent was created." + end if + + stSection "events" + -- the alert drain: after the adds + DHT puts above there should be events + -- queued; we only assert it drains a well-formed list (the dispatcher in + -- torrent-helpers is what fans these out to message-path handlers). + stAssert "btPoll drains a list", (stCount(btPoll(sSession)) >= 0) + + stSection "DHT key-value (BEP44) - offline-verifiable parts" + local tKey, tKey2 + put btDhtKeypair("") into tKey + stAssert "btDhtKeypair publicKey is 64 hex", (the number of chars of tKey["publicKey"] is 64) + stAssert "btDhtKeypair secretKey is 128 hex", (the number of chars of tKey["secretKey"] is 128) + stAssert "btDhtKeypair seed is 64 hex", (the number of chars of tKey["seed"] is 64) + put btDhtKeypair(tKey["seed"]) into tKey2 + stAssert "btDhtKeypair is deterministic from a seed", (tKey2["publicKey"] is tKey["publicKey"]) + local tTarget + put btDhtPutImmutable(sSession, textEncode("TorrentXT self-test value", "UTF-8")) into tTarget + stAssert "btDhtPutImmutable returns a 40-hex target", (the number of chars of tTarget is 40) + stAssert "btDhtGetImmutable accepts the target (async)", (btDhtGetImmutable(sSession, tTarget) is 0) + stAssert "btDhtPutMutable signs + queues", (btDhtPutMutable(sSession, tKey["publicKey"], tKey["secretKey"], "", textEncode("v1", "UTF-8")) is 0) + stAssert "btDhtGetMutable accepts the key (async)", (btDhtGetMutable(sSession, tKey["publicKey"], "") is 0) + + stSection "not auto-checked - confirm by hand" + stNote "btMoveStorage + btRemoveTorrent(deleteFiles=true) are destructive;" + stNote "btSetFilePriorities needs a binary buffer; btAddTorrentWithResume" + stNote "needs async resume bytes; btAddMagnet is covered via btAddMagnetEx." + catch tRunErr + stNote "RUNTIME ERROR - a binding call failed: " & tRunErr + add 1 to sFailed + end try + + stShowSummary + stPaint +end stRun + +-- the bulk of the per-torrent surface +command stRunTorrentChecks pTorrentData + local tStatus, tHash, tFiles, tFile, tWS, tTrackerUrl, tFound + + stSection "status + inspection" + put btTorrentStatus(sTorrent) into tStatus + stAssert "btTorrentStatus has a name", (tStatus["name"] is not "") + stAssert "btTorrentStatus progress is a number", (tStatus["progress"] is a number) + put btInfoHash(sTorrent) into tHash + stAssert "btInfoHash is 40 hex", (the number of chars of tHash is 40) + stAssert "btPieceBitfield returns a measurable Data", (the number of bytes in btPieceBitfield(sTorrent) is a number) + stAssert "btPieceAvailability returns a measurable Data", (the number of bytes in btPieceAvailability(sTorrent) is a number) + put btFileList(sTorrent) into tFiles + stAssert "btFileList returns >= 1 file", (stCount(tFiles) >= 1) + repeat for each element tFile in tFiles + stAssert "btFileList entry has a path", (tFile["path"] is not "") + stAssert "btFileList entry size > 0", (tFile["size"] > 0) + stAssert "btFileList entry priority is a number", (tFile["priority"] is a number) + exit repeat + end repeat + stAssert "btPeerList returns a list", (stCount(btPeerList(sTorrent)) >= 0) + + stSection "control + flags" + stAssert "btPause", (btPause(sTorrent) is 0) + stAssert "btResume", (btResume(sTorrent) is 0) + stAssert "btForceRecheck", (btForceRecheck(sTorrent) is 0) + stAssert "btForceReannounce", (btForceReannounce(sTorrent) is 0) + stAssert "btSetSequentialDownload on", (btSetSequentialDownload(sTorrent, true) is 0) + stAssert "btSetSequentialDownload off", (btSetSequentialDownload(sTorrent, false) is 0) + stAssert "btSetAutoManaged", (btSetAutoManaged(sTorrent, true) is 0) + stAssert "btSetSuperSeeding", (btSetSuperSeeding(sTorrent, false) is 0) + stAssert "btSetShareMode", (btSetShareMode(sTorrent, false) is 0) + stAssert "btSetUploadMode", (btSetUploadMode(sTorrent, false) is 0) + stAssert "btSetTorrentFlags (sequential=512)", (btSetTorrentFlags(sTorrent, "512", "512") is 0) + stAssert "btUnsetTorrentFlags", (btUnsetTorrentFlags(sTorrent, "512") is 0) + stAssert "btSetMaxConnections", (btSetMaxConnections(sTorrent, 50) is 0) + stAssert "btSetMaxUploads", (btSetMaxUploads(sTorrent, 4) is 0) + stAssert "btSetFilePriority", (btSetFilePriority(sTorrent, 0, 4) is 0) + stAssert "btSetPiecePriority", (btSetPiecePriority(sTorrent, 0, 4) is 0) + stAssert "btSetTorrentLimits", (btSetTorrentLimits(sTorrent, "0", "0") is 0) + stAssert "btScrapeTracker (async)", (btScrapeTracker(sTorrent) is 0) + stAssert "btClearTorrentError", (btClearTorrentError(sTorrent) is 0) + + stSection "download queue" + stAssert "btQueuePosition is a number", (btQueuePosition(sTorrent) is a number) + stAssert "btQueueTop", (btQueueTop(sTorrent) is 0) + stAssert "btQueueBottom", (btQueueBottom(sTorrent) is 0) + stAssert "btQueueUp", (btQueueUp(sTorrent) is 0) + stAssert "btQueueDown", (btQueueDown(sTorrent) is 0) + + stSection "trackers + web seeds (round-trips)" + stAssert "btTrackers returns a list", (stCount(btTrackers(sTorrent)) >= 0) + put "udp://tracker.selftest.example:6969/announce" into tTrackerUrl + stAssert "btAddTracker", (btAddTracker(sTorrent, tTrackerUrl, 1) is 0) + put false into tFound + repeat for each element tFile in btTrackers(sTorrent) + if tFile["url"] is tTrackerUrl then put true into tFound + end repeat + stAssert "btTrackers now lists the added tracker", tFound + put "http://seed.selftest.example/data/" into tWS + stAssert "btAddWebSeed", (btAddWebSeed(sTorrent, tWS) is 0) + stAssert "btWebSeeds lists the added seed", stListHas(btWebSeeds(sTorrent), tWS) + stAssert "btRemoveWebSeed", (btRemoveWebSeed(sTorrent, tWS) is 0) + stAssert "btWebSeeds no longer lists it", (not stListHas(btWebSeeds(sTorrent), tWS)) + + stSection "session operations" + stAssert "btListenPort is a number", (btListenPort(sSession) is a number) + stAssert "btSessionIsPaused is false", (btSessionIsPaused(sSession) is false) + put btFindTorrent(sSession, tHash) into tFound + stAssert "btFindTorrent recovers our handle from the info-hash", (tFound is sTorrent) + stAssert "btSessionPause", (btSessionPause(sSession) is 0) + stAssert "btSessionIsPaused is true after pause", (btSessionIsPaused(sSession) is true) + stAssert "btSessionResume", (btSessionResume(sSession) is 0) + stAssert "btDhtAnnounce (async)", (btDhtAnnounce(sSession, tHash, 0) is 0) + + stSection "filtering + streaming" + stAssert "btIpFilterAdd", (btIpFilterAdd(sSession, "10.0.0.0", "10.255.255.255", true) is 0) + stAssert "btIpFilterClear", (btIpFilterClear(sSession) is 0) + stAssert "btSetPieceDeadline", (btSetPieceDeadline(sTorrent, 0, 5000) is 0) + stAssert "btClearPieceDeadlines", (btClearPieceDeadlines(sTorrent) is 0) + + stSection "extended add" + put btAddMagnetEx(sSession, "magnet:?xt=urn:btih:1111111111111111111111111111111111111111", sTempDir, "16", "16") into sTorrent2 + stAssert "btAddMagnetEx (paused) returns a handle", (sTorrent2 > 0) + stAssert "btAddTorrentFileEx returns a handle", (btAddTorrentFileEx(sSession, pTorrentData, sTempDir, "16", "16") > 0) + + stSection "resume data" + stAssert "btSaveResumeData (async)", (btSaveResumeData(sTorrent) is 0) +end stRunTorrentChecks + +-- create a small temp content folder + file to seed (idempotent) +command stMakeContent + local tPayload, tFile + put specialFolderPath("temporary") into sTempDir + if sTempDir is empty then + put specialFolderPath("documents") into sTempDir + end if + if the last char of sTempDir is "/" then + delete the last char of sTempDir + end if + put sTempDir & "/torrentxt-selftest" into sContentDir + if there is no folder sContentDir then + create folder sContentDir + end if + put sContentDir & "/sample.dat" into tFile + put "TorrentXT self-test sample payload block. " into tPayload + repeat 10 times + put tPayload after tPayload + end repeat + put tPayload into url ("binfile:" & tFile) +end stMakeContent + +command stShowSummary + local tSummary + put sPassed && "passed, " & sFailed && "failed (" & sTotal && "checks)" into tSummary + set the text of field "stSummary" to tSummary + if sFailed is 0 then + set the foregroundColor of field "stSummary" to "0,140,70" + else + set the foregroundColor of field "stSummary" to "200,40,40" + end if +end stShowSummary + +-- paint the results list: red FAIL, green PASS, grey sections/notes +command stPaint + local tN, tI, tLine + set the text of field "stResults" to sResultText + put the number of lines of field "stResults" into tN + lock screen + try + repeat with tI = 1 to tN + put line tI of field "stResults" into tLine + if tLine begins with "FAIL" then + set the foregroundColor of line tI of field "stResults" to "200,40,40" + else if tLine begins with "PASS" then + set the foregroundColor of line tI of field "stResults" to "0,140,70" + else + set the foregroundColor of line tI of field "stResults" to "110,110,110" + end if + end repeat + catch tErr + end try + set the vScroll of field "stResults" to 0 + unlock screen +end stPaint + +-- remove any torrents we added and stop the session (idempotent) +command stCleanup + if sSession is not empty and sSession is not 0 then + if sTorrent is not empty and sTorrent > 0 then + btRemoveTorrent sSession, sTorrent, false + end if + if sTorrent2 is not empty and sTorrent2 > 0 then + btRemoveTorrent sSession, sTorrent2, false + end if + btStopSession sSession + end if + put 0 into sSession + put 0 into sTorrent + put 0 into sTorrent2 +end stCleanup diff --git a/tools/check-livecodescript.py b/tools/check-livecodescript.py index 49b64ac..5c4d765 100644 --- a/tools/check-livecodescript.py +++ b/tools/check-livecodescript.py @@ -414,7 +414,7 @@ def gather(paths): def main(argv): - paths = argv[1:] or ["src", "examples"] + paths = argv[1:] or ["src", "examples", "tests"] files = gather(paths) all_problems = [] for f in files: From 265626633cf6ac87d9ba35bc86d275af8b1849ef Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 27 Jun 2026 14:03:26 +0000 Subject: [PATCH 2/2] docs: forward plan for libsodium/ENet/libdatachannel + OXT engine playbook Adds docs/NEXT-EXTENSIONS-PLAN.md - the implementation plan for the next three native wraps (the "secure decentralized real-time" stack on top of TorrentXT): libsodium (trust), ENet (reliable-UDP real-time), libdatachannel/libjuice (WebRTC + NAT traversal). It opens with Part I, a consolidated ENGINE PLAYBOOK that captures every OXT/LiveCode/LCB/FFI gotcha we have uncovered so the next wraps do not repeat them: - the three rules and how each library's threading model changes them (sodium has no threads; ENet is pump-driven; libdatachannel fires callbacks from its own threads -> a mutex-guarded poll queue is mandatory); - LiveCodeScript + LCB gotchas (ASCII-only; itemDelimiter global; no repeat-for-each-line in LCB; constants literal-before-use; unsafe brackets; declarations at handler top; Boolean<->CInt; ZStringUTF8); - the FFI contract (Data<->Pointer does not auto-bridge; the out-buffer bytes-written/-needed convention; no 64-bit foreign int; the getter -1 caveat); - the C-preprocessor macro-comma trap; the exception firewall; the record codec and gen-tagged handle tables; the no-unload-hook lifecycle; the single-thread performance playbook; and the toolchain gotchas (the ABI-version sync gate, committed per-platform binaries, the static gates, "verified statically"). Parts II-IV are the per-library plans (prefix/ABI, phased surface, library gotchas, testing, milestones). Part V covers sequencing, the shared oxtkit/ scaffolding to extract from TorrentXT, a risk register, and a per-library definition of done. Linked from the README documentation list. Plan only - no code/ABI change. ASCII-clean. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01371AXB4CUUke7enHHS8okc --- README.md | 4 + docs/NEXT-EXTENSIONS-PLAN.md | 641 +++++++++++++++++++++++++++++++++++ 2 files changed, 645 insertions(+) create mode 100644 docs/NEXT-EXTENSIONS-PLAN.md diff --git a/README.md b/README.md index 07004e8..5716e83 100644 --- a/README.md +++ b/README.md @@ -217,6 +217,10 @@ Two runnable flagship demos plus the shared poll-dispatcher utility: - **[TorrentXT-IMPLEMENTATION-PLAN.md](docs/TorrentXT-IMPLEMENTATION-PLAN.md)** — the original design brief, kept for the *why* (engine choice, ABI design, risk register). +- **[NEXT-EXTENSIONS-PLAN.md](docs/NEXT-EXTENSIONS-PLAN.md)** — the forward plan + for the next native wraps (libsodium, ENet, libdatachannel) **and** the + consolidated OXT/LiveCode engine playbook: every FFI / LCB / runtime gotcha + we have uncovered, so the next wraps avoid the same mistakes. ## Building from source diff --git a/docs/NEXT-EXTENSIONS-PLAN.md b/docs/NEXT-EXTENSIONS-PLAN.md new file mode 100644 index 0000000..dd25aaa --- /dev/null +++ b/docs/NEXT-EXTENSIONS-PLAN.md @@ -0,0 +1,641 @@ +# Next OXT Native Extensions - Plan & Engine Playbook + +**libsodium - ENet - libdatachannel/libjuice** + +This is the implementation plan for the next three native libraries we intend to +bring to OpenXTalk (OXT) / the xTalk family, and - just as importantly - the +consolidated **engine playbook**: every gotcha and piece of weirdness we have +uncovered wrapping C/C++ libraries for OXT (across Box2Dxt, ShowControl, and +TorrentXT), written down so we never re-learn them. + +> **Read Part I before starting ANY of the three.** It is the reusable "how to +> wrap a native library for OXT without getting bitten" reference. Parts II-IV are +> the per-library plans; Part V is sequencing, shared infrastructure, and risk. +> +> Companion to `docs/TorrentXT-IMPLEMENTATION-PLAN.md` (the original design brief) +> and `/CLAUDE.md` (the as-built record). Where those differ from the code, the +> code wins; where this plan is not yet built, it is marked as plan, not as-built. + +--- + +## Why these three, and in this order + +The coherent theme is **a secure, decentralized, real-time application stack for +xTalk** - capabilities no other xTalk environment has. TorrentXT is already one +leg of it: + +| Layer | Library | Role | Status | +|---|---|---|---| +| Discovery + bulk transfer | **TorrentXT** (libtorrent) | DHT rendezvous + BitTorrent file movement | done (ABI v8) | +| Trust | **libsodium** | identity, signing, encryption, password hashing | planned - **do first** | +| Real-time (reliable UDP) | **ENet** | low-latency peer messaging (games, presence, collab) | planned - **second** | +| Real-time (WebRTC) | **libdatachannel** + **libjuice** | browser-interoperable P2P + real NAT traversal | planned - **third** | + +They compose: libsodium secures the channels; ENet/libdatachannel carry live +messages; and libdatachannel's signaling can ride **TorrentXT's DHT (BEP44)** - +a server-less rendezvous closing the loop. + +**Licenses (all permissive - confirm exact tag when starting):** libsodium ISC; +ENet MIT; libdatachannel and libjuice MPL-2.0. Avoid GPL cores (e.g. most of +FFmpeg) so the static-link-and-ship story stays clean - the same reason +libtorrent (BSD-3) was chosen over alternatives. + +## The house pattern (recap) + +Every wrap so far follows one shape, and these three will too: + +``` + native library (C or C++) owns its own work + |- C++/C shim src/_shim.cpp -> .{so,dll,dylib} (ABI: PFX_*) + |- LCB binding src/.lcb (library org.openxtalk.library.) + |- script helpers examples/-helpers.livecodescript (poll dispatcher) +``` + +- A flat `extern "C"` shim exporting `PFX_*` symbols (a frozen, versioned ABI). +- Generation-tagged **handle tables** (stale handle = harmless no-op). +- The **exception firewall** (every entry point wrapped). +- For anything with inbound events, a **poll-drained event queue** (no callback + ever runs script). +- A thin **LCB binding** of `private foreign handler` decls + public `Xx*` + wrappers that hide handles, pre-size buffers, walk records, and never throw. +- The self-describing **typed KV record codec** for events and snapshots. + +--- + +# PART I - The OXT / LiveCode Engine Playbook (read first) + +Everything below is a mistake we have already made or a constraint we have +already paid for. Internalise it before writing a line of a new wrap. + +## I.1 The three rules (universal - but the nuance differs per library) + +1. **Never call an LCB / script handler from a foreign (library) thread.** + Inbound activity must ride a queue that the script **poll-drains** on a timer; + no callback ever runs script. + - *libsodium*: no threads at all - trivially satisfied (and no poll needed). + - *ENet*: no internal threads either - it is **pump-driven** (you call + `enet_host_service`), so the rule is trivially satisfied, BUT nothing + happens unless you pump, so the poll loop is mandatory. + - *libdatachannel*: **has its own threads and fires callbacks from them.** + This is the rule's worst case and the single most important design + constraint of that binding (see Part IV). + +2. **The exception firewall.** A throw crossing `extern "C"` takes the engine + down. **Every** entry point is `try { ... } catch (...) { set_error(...); + return ; }`. C libraries (sodium, enet) rarely throw, but our own + allocations can `std::bad_alloc`, and libdatachannel is C++ - so the firewall + is mandatory in all three, no exceptions. + +3. **Payload across the FFI.** This rule is **domain-specific**, not universal. + For TorrentXT it meant "gigabytes never cross" (only tiny status records do). + For these three, *payload does cross by design* - it IS the data (a plaintext + to encrypt, a game-state message, a chat line). The rule becomes: **keep what + crosses small** (crypto buffers, control messages, KB-scale), and push bulk + (files, media streams) back to TorrentXT or keep it engine-side. Document a + size budget per binding. + +## I.2 LiveCodeScript (`.livecodescript`) gotchas + +- **ASCII only.** No smart/curly quotes (U+201C/201D/2018/2019) anywhere - not + in code, strings, or comments. They fail OXT compilation. The static checker + enforces zero. Straight `"` and `'` only; write arrows as `->`. +- **`repeat with x = 1 to n`** (NOT `from`). `repeat for each element X in aList` + and `repeat for each line L in aText` are both available here (LiveCodeScript, + unlike LCB - see I.3). +- **`itemDelimiter` / `lineDelimiter` are GLOBAL mutable state.** Set them + *immediately* before each use; never assume their value. A handler that returns + while they are set to something exotic will surprise the next handler. +- **`the round of X`** is the form this codebase uses (match it; do not switch to + `round(X)` mid-file). +- **Commands report via `the result`; functions return a value.** Match the API + shape: a command-style handler is `btAddMagnet s, uri, path` then + `put the result into tH`; a function-style is `put btTorrentStatus(tH) into a`. +- `textEncode(str,"UTF-8")` -> Data; `textDecode(data,"UTF-8")` -> String. +- File IO idioms: `put X into url ("binfile:" & tPath)`; `there is a file tPath` + / `there is no folder tDir`; `create folder tDir`. +- Custom properties for persistence: `set the uXxx of this stack to ...`. +- Self-building demo/test stacks are **idempotent**: guard the build with + `if there is a field "x" then exit ...`, and wrap the build in + `lock screen` / `unlock screen` to avoid flicker. + +## I.3 LCB (LiveCode Builder, `.lcb`) gotchas - stricter than LiveCodeScript + +- **`unsafe ... end unsafe` brackets EVERY foreign call.** +- **Keep ALL `variable`/`local` declarations at the TOP of a handler.** A nested + declaration (a `variable` inside an `if`/`repeat`) has broken whole-script + compilation. Declare everything up front. +- **No `repeat for each line` in LCB.** (Discovered building `btWebSeeds`.) Use a + counted `repeat with i from 1 up to n`. For a bridged list value, + `repeat for each element X in aList` works; for splitting text by line it does + not - so the shim should hand back a **list of records** (the framing below) + rather than newline-joined text the LCB side would have to split. +- **Constants must be LITERAL and declared BEFORE first use.** OXT resolves + constants by lexical position; a forward reference silently evaluates to + nothing. Put `constant k... is ` near the top. +- **Foreign decl shape:** + `private foreign handler _pfx_name(in p as CInt, ...) returns CInt binds to "c:>pfx_name!cdecl"`. + Keep the stable `pfx_` prefix; **never rename an exported symbol** - the + compiled `.lcb` references the string, so a rename breaks the bind at first use. +- **`` handlers resolve by NAME** (`MCMemoryAllocate`, + `MCMemoryDeallocate`, `MCDataGetBytePtr`, `MCDataCreateWithBytes`, + `MCStringDecode`), so they carry **no leading underscore** - renaming them + breaks the bind. +- **Boolean params/returns need explicit conversion.** A foreign handler takes + `CInt`, so convert a Boolean param: `if pVal then put 1 into tV else put 0 into + tV`. To return a Boolean from an int: `return tR is not 0`. +- **`ZStringUTF8`** is the bridge for short, NUL-terminated UTF-8 strings (magnet + URI, hex hash, save path, an error message). +- **Avoid names whose stem shadows an engine token**, even when prefixed; prefer + distinctive multi-word stems. +- **Prefix conventions:** `t` handler-local, `p` parameter, `s` script/module + local, `k` constant. Public API `XxPascalCase`; C ABI `pfx_snake_case`. + +## I.4 The FFI marshalling contract (the part that cost us the most) + +- **A `Data` does NOT auto-bridge to a C `Pointer`.** It marshals as an opaque + `MCDataRef`; passing one where a `Pointer` is declared raises a runtime + `expected type pointer`. This was the hard-won lesson. The two proven shapes + (from the htmltidy/HIDAPI bindings): + - **OUT buffer** (the shim fills it): the binding allocates a raw block via the + builtin `MCMemoryAllocate`, passes it as a real `Pointer` plus its capacity; + the shim writes and returns **bytes-written**, or **`-needed`** (the negative + of the required size) if the block was too small; the binding copies exactly + the written bytes back with `MCDataCreateWithBytes`. **Grow-and-retry once** + on `-needed`, then walk the bytes. **Reuse a persistent buffer** (`sXxxPtr` / + `sXxxCap`) - never reallocate per poll. + - **IN buffer** (the app supplies it): pass `MCDataGetBytePtr(theData)` - the + read-only pointer to the Data's own bytes - plus its length. The shim only + reads it. +- **Scalars:** int -> `CInt`, bool -> `CInt` (0/1), real -> `double`. +- **There is NO 64-bit foreign int.** 64-bit values, offsets, and hashes ride as + **decimal/hex `ZStringUTF8` strings**. +- **Never return a library-owned `const char*`** of unknown lifetime - fill a + caller buffer, or return a defined-lifetime static the engine copies + immediately. Return `""`, never `NULL`, on bad input. +- **The out-buffer convention is a separate family from action codes.** Getters + return bytes-written / `-needed` / `0`-on-bad-handle. Actions return `0` (ok) / + negative (error). Keeping them separate means a small `-needed` can never be + mistaken for an error code. +- **The getter `-1` caveat.** The default is "0 == no-op/empty on a bad handle." + But when `0` is itself a *valid* value for an int getter (e.g. a queue position + of 0), return **`-1`** for "no value / bad handle" and document it - otherwise + the caller cannot tell "position 0" from "no torrent." + +## I.5 C-preprocessor / C++ traps (paid for this session) + +- **THE MACRO-COMMA TRAP.** A top-level comma inside a function-like guard macro + body splits the macro's arguments. `std::array seed;` inside + `BTX_GUARD_*({ ... })` failed with "macro passed 2 arguments" - the + preprocessor protects commas inside **parentheses only**, not `` or `{braces}`. Fixes: hoist a file-scope `using alias = + std::array;` and use the alias; or wrap the declaration in an extra + set of parens; or declare the variable outside the macro body. +- **Exceptions must never cross `extern "C"`** - restated because a C++ library + (libdatachannel) throws, and even C glue can hit `std::bad_alloc` on our + allocations. The firewall macro makes this structural, not a thing you remember. +- **Treat third-party headers as system headers** (`-isystem`) so their warnings + do not pollute our `-Wall -Wextra` (`/W3` on MSVC). Our code stays warning-clean. +- **Pin every dependency version.** Stand up the CI build matrix in Phase 0, not + at the end - the dependency build (Boost for libtorrent; OpenSSL/usrsctp/ICE + for libdatachannel) is the real risk, not the binding. + +## I.6 Handles & the record codec (reuse these verbatim) + +- **Handles** are positive 32-bit ints packing a generation counter above a slot + index, in a validated table. Freeing a slot bumps its generation, so a + stale/removed/never-created handle is a harmless no-op (getters 0/empty, actions + error) - never a crash, never a recycled-slot alias. One table per object kind + (session/torrent; host/peer; peerconnection/channel). Also check the library's + own validity flag where it has one. +- **The record codec** is the wire format for every event and snapshot: + `kvrecord := [count:u16] then [fieldId:u8][type:u8][len:u16][bytes]` repeated; + **all framing integers big-endian**; `type` is 0=int(ASCII) 1=real(ASCII) + 2=utf8 3=raw 4=hex. Higher shapes are count-prefixed lists: + `[count:u16] then [bodyLen:u16][kvrecord]*`. Keep a **single fieldId/alert + registry in a shared header**; the LCB walker mirrors the numbers as `k*` + constants, and a registry checker proves the two never drift. **The drain never + drops a record** - an oversized one is stashed and emitted next call. + +## I.7 Lifecycle & threading model + +- **There is NO deterministic LCB unload hook.** The library cannot tear itself + down. Expose an explicit teardown (`XxStopSession` / `XxClose` / `XxCleanup`) + and document that the app MUST call it (e.g. on `closeStack`). Make it + **idempotent** and a no-op on a stale handle. For global-singleton init + (`sodium_init`, `enet_initialize`, `rtcInit`), init once on load; refuse a + second concurrent *session* only where the library is genuinely single-instance + (libtorrent was; ENet is NOT - it allows many hosts). +- **The poll-drain.** A script timer calls `XxPoll` once per tick -> one FFI + round-trip drains ALL pending events as a record list -> a dispatcher fans them + to message-path handlers. The interval is a **latency/CPU knob**, not a + correctness knob. For real-time (ENet, datachannel) poll faster (16-33 ms); for + crypto (libsodium) there is no poll at all (synchronous functions). + +## I.8 The single-thread performance playbook + +OXT runs script, the FFI, and rendering on ONE interpreted thread. Costs, in +order: (1) interpreter ops, (2) FFI round-trips, (3) property-set redraws. + +- **One FFI round-trip per poll** - a batched drain and one-call snapshots, never + one call per event or per field. +- **Reuse a persistent buffer** in the hot path; rebuilding an N-byte `Data` + every poll is O(N) interpreter work. +- **One clock read per pass** (hoist `the milliseconds` out of loops). +- **UI text <= ~4 Hz and only on change.** An every-frame field relayout+redraw + is the biggest avoidable cost. + +## I.9 Toolchain & process gotchas + +- **ABI-version sync gate.** `PFX_ABI_VERSION` (C header) must equal + `kABIVersion` (LCB). A forgotten bump previously surfaced only at runtime via + `checkABI()`; we added a **static gate** asserting the two match + (`check-record-registry.py`) after hitting exactly that. Bump on ANY change to + the exported surface (new symbol, changed signature, new fieldId/alert). +- **The native binary ships bundled per platform** under + `src/code/-/.{so,dll,dylib}` - bare token, no `lib` + prefix; platform ids `x86_64-linux` / `x86-linux` / `x86_64-win32` / + `x86-win32` / `universal-mac`, **architecture first**, Windows `-win32` for + both bitnesses. A native change is not "done" until the committed binary is + refreshed (`package-extension.py`); CI rebuilds and commits the matrix on merge. +- **Static gates run anywhere (no GUI):** `check-livecodescript.py` (smart + quotes, handler/control/`unsafe` balance, constants-before-use - now scans + `src`, `examples`, AND `tests`), the record-registry cross-check, the record + golden test, and the C++ smoke test under **gcc ASan/UBSan** (clang's ASan + runtime is not installed in our environment). +- **The OXT pass is unavoidable.** There is no headless way to compile or run + `.lcb` / `.livecodescript`. A **self-building runtime self-test stack** (see + `tests/torrent-selftest.livecodescript`) is the companion to the C++ smoke + test - the only way to validate the BINDING. Until a human runs it, everything + binding-side is **"verified statically; needs an OXT pass."** Never claim + runtime behaviour you cannot observe. + +--- + +# PART II - libsodium (the trust layer) - DO FIRST + +## II.1 Why first + +It is the easiest wrap we will ever do, fills the most glaring gap (OXT's crypto +is dated - no ed25519, X25519, AEAD, Argon2, BLAKE2), and is a **force +multiplier**: the real-time layers need it for secure channels, and every serious +app needs signing / encryption / password hashing. Fast win; it also lets us +stand up the shared scaffolding (Part V) on the simplest possible target. + +## II.2 The wrap shape - and how it DIFFERS from TorrentXT + +- **Pure functions. No threads, no event loop, (mostly) no handles.** So in + Phase 1 there is **no poll-drain and no record codec** - just the firewall and + the Data<->Pointer in/out buffer contract, used heavily because the payload IS + the data. +- **Rule 3 inverts:** crypto buffers cross by design. Keep them reasonable + (KB-MB). For huge inputs use the streaming API (Phase 2, with handles) or chunk + in script. +- **Global init:** `sodium_init()` once on load (idempotent; 0 ok / 1 already / -1 + fail). Refuse to operate if it returns -1. +- **Secret-material caveat:** an OXT `Data` is NOT locked/secure memory; + libsodium's `sodium_malloc`/`mlock` protections do not extend to keys held in + script. Document this, and offer `sdMemzero` to best-effort wipe a Data's bytes + after use. + +## II.3 Prefix, ABI, sizes + +- C ABI prefix `sdx_`; LCB public `sd*`; library `org.openxtalk.library.sodium`. +- Reuse the out-buffer convention verbatim. Fixed-size outputs (signature 64, + public key 32, secret key 64, nonce 24, MAC 16, ...) are returned via the + buffer; variable outputs (ciphertext = message + MAC) are sized by the shim. + **Expose the exact sizes as `kSd*` constants** so script can size buffers and + validate inputs. + +## II.4 Phase 1 surface (the essentials) + +| Public handler | libsodium primitive | Notes | +|---|---|---| +| `sdInit` / `sdVersion` | `sodium_init` / `sodium_version_string` | once at load | +| `sdRandomBytes(n)` | `randombytes_buf` | CSPRNG | +| `sdBin2Hex` / `sdHex2Bin` / `sdBin2Base64` / `sdBase642Bin` | `sodium_*` | constant-time; exact size formulas | +| `sdSignKeypair` / `sdSignKeypairFromSeed` | `crypto_sign_keypair` / `_seed_keypair` | ed25519 | +| `sdSign` / `sdVerify` | `crypto_sign_detached` / `_verify_detached` | 64-byte detached sig | +| `sdSecretbox` / `sdSecretboxOpen` | `crypto_secretbox_easy` / `_open_easy` | shared-key auth-enc | +| `sdAeadEncrypt` / `sdAeadDecrypt` | `crypto_aead_xchacha20poly1305_ietf_*` | 24-byte random nonce, +AAD | +| `sdBoxKeypair` / `sdBoxEasy` / `sdBoxOpenEasy` | `crypto_box_*` | X25519 public-key enc | +| `sdBoxSeal` / `sdBoxSealOpen` | `crypto_box_seal*` | anonymous sender | +| `sdPwHashStr` / `sdPwHashStrVerify` | `crypto_pwhash_str*` | Argon2id password storage | +| `sdPwHash` | `crypto_pwhash` | derive a key from a password+salt | +| `sdGenericHash` | `crypto_generichash` | BLAKE2b, optional key, variable len | +| `sdKxKeypair` / `sdKxClientSessionKeys` / `sdKxServerSessionKeys` | `crypto_kx_*` | session-key exchange | +| `sdMemcmp` / `sdMemzero` | `sodium_memcmp` / `sodium_memzero` | constant-time compare; wipe | + +## II.5 Phase 2 (stateful / streaming - introduces handles) + +- `crypto_secretstream_xchacha20poly1305` (chunked file/stream encryption) - state + handles in a gen-tagged table. +- Multipart `crypto_generichash` / `crypto_sign` (init/update/final) - state + handles. +- `crypto_kdf` (derive subkeys from a master key). + +## II.6 libsodium-specific gotchas + +- `sodium_init` MUST run before any other call and is not safe to call + concurrently (we are single-threaded, so call it once at load). +- The "easy" functions have exact buffer-size rules (does the output include the + MAC? the nonce?). Be precise; expose `kSdMacBytes` (16), `kSdNonceBytes` (24), + `kSdSignBytes` (64), etc. so script never mis-sizes a buffer. +- **Argon2 is CPU+memory heavy by design.** `opslimit`/`memlimit` are the latency + knob; ship `interactive`/`moderate`/`sensitive` presets as constants and warn + that `sensitive` can take seconds and **block the single thread** - for a UI, + use `interactive`, or run the hash off a timer/idle pass. +- Hex/base64 helpers are constant-time with exact output sizes + (`sodium_base64_ENCODED_LEN`) - use them rather than hand-rolling. +- It is C and effectively never throws; the firewall is still mandatory for our + own `std::bad_alloc`. +- **Build is the easiest of the three:** tiny, no Boost, trivially static-linked + (autotools or a CMake port). Pin a release (1.0.19+). + +## II.7 Testing + +- A C smoke test linking real libsodium: sign->verify round-trip, encrypt->decrypt + round-trip (secretbox + AEAD + box), `pwhash_str` -> verify, hex/base64 round + trips, keypair-from-seed determinism, bad-input -> clean error, the firewall. + Light dependency -> runs in CI. +- A self-test stack mirroring the above with known test vectors and a green/red + list (the `torrent-selftest` pattern). + +## II.8 Milestones + +0. **Spike:** build + `sdInit`/`sdVersion`/`sdRandomBytes` round-trip across the + FFI (proves the scaffolding on the simplest target). +1. **Essentials:** the Phase 1 table. +2. **Streaming:** secretstream + multipart hashes (handles). +3. **Package + docs + self-test stack;** commit the platform binaries. + +--- + +# PART III - ENet (real-time, step 1) - DO SECOND + +## III.1 What it brings / why second + +Reliable-and-unreliable UDP messaging with channels, sequencing, fragmentation, +and connection management - the low-latency layer OXT most lacks (games, +presence, live collaboration). It **reuses the TorrentXT pattern almost +verbatim**, so it goes fast once libsodium has proven the scaffolding. MIT. + +## III.2 The wrap shape - closest to TorrentXT, but PUMP-DRIVEN + +- **ENet has no internal threads.** You drive it: `enet_host_service(host, + &event, timeoutMs)` pumps the socket and returns one event. So the poll-drain + is: each `enPoll` tick, loop `enet_host_service(host, &e, 0)` (non-blocking) + until it returns 0, draining CONNECT / DISCONNECT / RECEIVE events into the + record list. Rule 1 is trivially satisfied (no foreign thread) - **but nothing + progresses unless you pump**, so the poll loop is the binding's heartbeat and + its cadence sets latency (poll 16-33 ms for real-time, not 250 ms). +- **Handles:** a host table and a peer table (gen-tagged). A peer id maps to an + `ENetPeer*`; validate before use (a disconnected peer is a no-op). +- **Payload crosses** (received packet bytes -> script; script bytes -> a sent + packet) - but these are MESSAGES (game state, control), not bulk. Document that + ENet is not for files (use TorrentXT). +- **Not a single-instance library:** ENet allows MANY hosts per process, so the + "refuse a second session" rule does NOT apply - the handle table holds N hosts. +- **Global init:** `enet_initialize()` once; `enet_deinitialize()` at teardown. + +## III.3 Prefix & reuse + +- C ABI `enx_`; LCB public `en*`; library `org.openxtalk.library.enet`. +- Reuse, essentially unchanged: the record codec + list framing (events as KV + records), the handle tables, the firewall, the out-buffer convention, the + poll-drain dispatcher. + +## III.4 Phase 1 surface + +| Public handler | ENet | Notes | +|---|---|---| +| `enInitialize` / `enDeinitialize` / `enVersion` | `enet_initialize` / `_deinitialize` / `_linked_version` | global | +| `enHostCreateServer(addr,port,maxPeers,channels,inBW,outBW)` | `enet_host_create` (bound) | returns host handle | +| `enHostCreateClient(maxPeers,channels,inBW,outBW)` | `enet_host_create` (unbound) | client host | +| `enHostDestroy(host)` | `enet_host_destroy` | idempotent | +| `enConnect(host, peerHost, port, channels, data)` | `enet_host_connect` | returns peer handle; CONNECT event confirms | +| `enDisconnect` / `enDisconnectNow` / `enResetPeer` | `enet_peer_disconnect*` / `_reset` | | +| `enSend(peer, channel, data, flags)` | `enet_packet_create` + `enet_peer_send` | flags: reliable / unsequenced | +| `enBroadcast(host, channel, data, flags)` | `enet_host_broadcast` | | +| `enFlush(host)` | `enet_host_flush` | send queued now | +| `enSetPeerTimeout` / `enSetPeerPingInterval` / `enSetHostBandwidth` | `enet_peer_*` / `enet_host_bandwidth_limit` | tuning | +| `enPeerStatus(peer)` | from `ENetPeer` fields | record: state, rtt, packetLoss, ... | +| `enPoll(host, out, cap)` | loop `enet_host_service` | the event firehose | + +Events: `enetConnect` (peer), `enetDisconnect` (peer, reason), `enetReceive` +(peer, channel, raw data). + +## III.5 ENet-specific gotchas + +- **Pump or nothing.** `enet_host_service` must be called regularly or no + connect/send/receive makes progress - unlike libtorrent, which self-pumps on + its own threads. The poll loop is mandatory, and its interval is the latency + floor. +- `enet_host_service` returns ONE event per call - loop until it returns 0 each + tick to drain fully (or `enet_host_check_events` after a single service). +- **Packet ownership:** `enet_packet_create` copies the bytes (use the default; + do NOT use `NO_ALLOCATE`). After `enet_peer_send` the host owns the packet - + do not free it. On RECEIVE, **copy the bytes into our record THEN + `enet_packet_destroy`** - never hand script a pointer into ENet-owned memory. +- Channel count is fixed at host create - choose it deliberately. +- Reliable vs unreliable-sequenced vs unsequenced are packet flags - expose as an + int and document. +- `enet_address_set_host` resolves a hostname and **may block briefly on DNS** + (like libtorrent's bootstrap) - accept it, or resolve in script and pass an IP. +- 32-bit packet size cap; keep messages small. + +## III.6 Testing + +- C smoke test: init, host create/destroy, a **loopback** (a server host and a + client host in the same process), connect, send -> receive round-trip via + pumped services, peer status, bogus-handle no-ops, the firewall. +- Self-test stack: same loopback in one stack (ENet's multi-host support makes + this clean), asserting the receive event arrives after a few poll ticks. +- Demo: a tiny LAN chat / echo, or a two-cursor "shared whiteboard" showing + real-time state sync. + +## III.7 Milestones + +0. **Spike:** build + a one-process loopback connect + echo. +1. **Core:** host/peer lifecycle, send/broadcast, the poll drain, events. +2. **Tuning + stats:** bandwidth/timeout/ping, `enPeerStatus`. +3. **Package + docs + self-test + a chat demo.** + +--- + +# PART IV - libdatachannel + libjuice (real-time, step 2) - THE HARD ONE + +## IV.1 What it brings / why last + +WebRTC **data channels**: browser-interoperable peer-to-peer with real NAT +traversal (ICE / STUN / TURN). It is the graduation from ENet - it lets an OXT +app talk to a *browser* and punch through NATs without a relay - but ICE state +machines, the DTLS handshake, SCTP for data channels, and **foreign-thread +callbacks** make it the hardest of the three. MPL-2.0. + +## IV.2 The wrap shape - THE foreign-thread-callback challenge + +- **libdatachannel is C++ and runs its own threads;** its C API (`rtc/rtc.h`) + delivers everything via **callbacks** (`rtcSetStateChangeCallback`, + `rtcSetLocalDescriptionCallback`, `rtcSetLocalCandidateCallback`, + `rtcSetMessageCallback`, `rtcSetOpenCallback`, ...) fired **from those + threads.** This is Rule 1's worst case. The design that makes it safe: + - Each callback runs on libdatachannel's thread and does **only**: lock a + mutex, push a typed event (our handle id + a copy of the payload) onto a + queue, unlock. Nothing else - **no engine call, no script, no allocation that + could throw across the boundary**. + - `dcPoll` (on the script thread) locks the mutex, drains the queue into the + record list, unlocks, returns. Script never runs on a foreign thread. + - **The mutex-guarded inbound queue is the single most important correctness + structure of this binding.** It is the one binding with real concurrency. +- **Signaling is out of band.** WebRTC needs the two peers to exchange an SDP + offer/answer plus ICE candidates over *some* channel you provide. The binding + emits `localDescription` and `localCandidate` events (script ships them to the + peer), and accepts the remote via `dcSetRemoteDescription` / + `dcAddRemoteCandidate`. **The signaling transport can be TorrentXT's DHT + (BEP44)** - a server-less rendezvous - which closes the stack's loop nicely. +- **Payload:** data-channel messages cross to script (chat, game state, + file-chunk control). **Media (audio/video tracks) should NOT cross as payload** + - punt media to a much later, separate phase, or keep it engine-side. **Phase 1 + is DATA CHANNELS ONLY.** +- **Everything is async** - connection setup, ICE gathering, DTLS - all surfaced + as poll-drained events. There is no synchronous "connect"; you create a peer, + exchange signaling, and watch for the `channelOpen` event. + +## IV.3 Prefix & reuse + +- C ABI `dcx_`; LCB public `dc*`; library `org.openxtalk.library.datachannel`. +- Reuse: record codec + list framing, handle tables, firewall, out-buffer. +- **New structure:** the mutex-guarded inbound event queue (Phase 0). + +## IV.4 Phase 1 surface (data channels only) + +| Public handler | libdatachannel | Notes | +|---|---|---| +| `dcInit` / `dcCleanup` | `rtcInitLogger` (optional) / `rtcCleanup` | global; cleanup at teardown | +| `dcCreatePeer(iceServersJSON)` | `rtcCreatePeerConnection` | returns peer handle; wires our callbacks internally | +| `dcCreateDataChannel(peer, label)` | `rtcCreateDataChannel` | returns channel handle; triggers offer | +| `dcSetRemoteDescription(peer, type, sdp)` | `rtcSetRemoteDescription` | from signaling | +| `dcAddRemoteCandidate(peer, cand, mid)` | `rtcAddRemoteCandidate` | from signaling | +| `dcSendMessage(channel, data)` / `dcSendString(channel, text)` | `rtcSendMessage` | binary / text | +| `dcBufferedAmount(channel)` | `rtcGetBufferedAmount` | for backpressure | +| `dcCloseChannel(channel)` / `dcDeletePeer(peer)` | `rtcClose` / `rtcDeletePeerConnection` | idempotent | +| `dcPoll(out, cap)` | drain the mutex queue | the event firehose | + +Events: `dcLocalDescription` (type, sdp), `dcLocalCandidate` (cand, mid), +`dcStateChange` (state), `dcGatheringStateChange`, `dcChannelOpen`, +`dcChannelClosed`, `dcMessage` (channel, data), `dcError` (message). + +## IV.5 libdatachannel-specific gotchas (the big ones) + +- **Foreign-thread callbacks -> mutex queue** (Part IV.2). Never call a + `MCData*`/engine function from a callback. This is non-negotiable. +- **Callbacks can fire AFTER you delete the peer/channel.** Copy *handle ids* + (not pointers) into the queue, and validate the handle (gen-tagged) when the + script drains it - a stale id is a no-op. +- **The C API hands out its own int ids;** they are NOT generation-tagged, so do + not expose them directly (a recycled rtc id would alias). Map them to OUR + gen-tagged handles inside the shim. +- `rtcInit`/`rtcCleanup` are process-global. Init once at load; cleanup at + `dcCleanup` (no unload hook -> document a leak at quit if the app skips it). +- **ICE/TURN config crosses as strings;** TURN credentials are secrets - the same + no-secure-memory caveat as libsodium. +- **Build is the hard one.** libdatachannel pulls OpenSSL (or GnuTLS) for DTLS, + usrsctp for SCTP, libjuice (or libnice) for ICE, and plog. Vendor them via + CMake (`USE_GNUTLS=0` -> OpenSSL; build juice, not nice), static-link + everything, and **reuse the static OpenSSL 3 we already ship for TorrentXT**. + Stand up the platform matrix in Phase 0 - this is the Boost-build-risk lesson + amplified across more dependencies. +- **libjuice alone is a fork in the road.** If you want NAT-traversed UDP without + full WebRTC (no DTLS/SCTP/browser interop), wrapping just libjuice (ICE/STUN/ + TURN, MPL-2.0) is far lighter and could sit between ENet and full + libdatachannel. Decide deliberately. +- **Message-size limits:** SCTP data channels fragment large messages and have + practical caps - keep the control plane small; route bulk to TorrentXT. +- **Backpressure:** surface `bufferedAmount` (and the low-threshold event) so + script can throttle instead of blasting the channel. + +## IV.6 Testing + +- The C smoke test is harder: a **loopback** of two peer connections in one + process, manually shuttling each peer's description/candidates to the other + through the API, pumping until `channelOpen`, then send -> receive. This + exercises the mutex queue under the real threads. +- **Add ThreadSanitizer (TSan) for this binding specifically** - it is the only + one with real concurrency; TSan catches a missing lock that ASan/UBSan will not. +- Self-test stack: provide a one-stack loopback (two peer connections wired to + each other by direct candidate exchange), or document a two-machine manual test + using **DHT BEP44 as the signaling channel** (the headline integration demo). + +## IV.7 Milestones + +0. **The build + the queue:** get the dependency stack compiling on all platforms + (the real risk) and a C loopback spike (two peers, `channelOpen`, echo) + running clean under ASan/UBSan + TSan. +1. **Data channels + signaling events + the mutex queue.** +2. **TURN, backpressure, reconnection.** +3. **(Optional, separate) media tracks** - and only if payload can stay + engine-side. +4. **Package + docs + the flagship demo:** P2P chat/state over **DHT-signalled** + WebRTC, tying datachannel + TorrentXT together. + +--- + +# PART V - Sequencing, shared infrastructure, risk + +## V.1 The order, and why + +1. **libsodium** - lowest risk, highest leverage, no event model. Fastest win; + stand up the shared scaffolding here. +2. **ENet** - reuses the TorrentXT pattern almost verbatim; introduces the + real-time poll cadence; fully loopback-testable in one process. +3. **libdatachannel** - hard; introduces foreign-thread concurrency and the + dependency-build risk; pays off with browser interop + NAT traversal, with + **DHT BEP44 signaling** closing the loop back to TorrentXT. + +## V.2 Shared scaffolding to extract (the highest-leverage investment) + +Factor the proven core out of TorrentXT into a shared `oxtkit/` so all three +(and every future wrap) share one tested implementation: + +- the generation-tagged **handle table**; +- the **record codec** + the registry cross-check tool; +- the **firewall macros** (with the macro-comma lesson baked into a comment); +- the **out-buffer** helpers (allocate / bytes-written / `-needed` / grow-once); +- the **ABI-version sync gate**; +- the **poll-drain queue**, plus a **mutex variant** for libdatachannel; +- the static gates (`check-livecodescript`, record-registry, golden) - already + generic; +- `package-extension.py` + the per-platform committed-binary flow + the CI matrix. + +Doing this once, on libsodium, makes ENet and libdatachannel dramatically faster +and keeps all four bindings behaviourally identical where it matters (handle +safety, the firewall, the drain). + +## V.3 Risk register + +| Risk | Where | Mitigation | +|---|---|---| +| Dependency build (OpenSSL/usrsctp/ICE) | libdatachannel | Phase-0 CI matrix; vendor via CMake; reuse our static OpenSSL 3 | +| Foreign-thread races | libdatachannel | the mutex queue; copy ids not pointers; **add TSan** | +| No unload hook -> leak at quit | all | explicit idempotent teardown; document the closeStack contract | +| Payload-size creep across the FFI | enet, datachannel | document a size budget; route bulk to TorrentXT | +| Secret material in non-secure memory | sodium, datachannel(TURN) | document; offer `sdMemzero`; keep secrets short-lived | +| ABI drift between header and binding | all | the static ABI-sync gate (already built for TorrentXT) | +| License surprise | libjuice/datachannel | confirm MPL-2.0 at the pinned tag before shipping | +| Single-thread blocking (Argon2 `sensitive`) | sodium | presets; warn; run heavy hashing off a timer/idle pass | + +## V.4 Definition of done (per library) + +- C smoke test green under **gcc ASan/UBSan** (+ **TSan** for libdatachannel). +- All static gates green; the **ABI-sync gate** green. +- Four-to-five **platform binaries committed** via the CI matrix. +- A **self-test stack** covering the public surface (the `torrent-selftest` + pattern), runnable in OXT. +- Docs: a getting-started, an api-reference, and an update to this plan. +- Anything binding-side stays **"verified statically; needs an OXT pass"** until a + human runs the self-test stack and reports green. + +--- + +*This document is a plan, not an as-built record. As each library lands, fold its +real surface into its own api-reference and update the status table at the top.*