From 33c8836ba42bfdb1936ecf5c19ba3ba50f19d720 Mon Sep 17 00:00:00 2001 From: n30nex Date: Wed, 17 Jun 2026 19:11:55 -0400 Subject: [PATCH 01/64] Add BLE companion disconnect diagnostics --- README.md | 24 +++-- docs/tdeck-feature-inventory.md | 4 +- docs/tdeck-firmware-roadmap.md | 3 +- docs/tdeck-hardware-dogfood-checklist.md | 1 + scripts/fetch_tdeck_artifact.py | 29 ++++++ scripts/tdeck_smoke.py | 41 +++++--- src/mt_companion.cpp | 122 +++++++++++++++++++---- src/serial_cli.cpp | 6 +- 8 files changed, 185 insertions(+), 45 deletions(-) diff --git a/README.md b/README.md index 722c799..f5b9d9e 100644 --- a/README.md +++ b/README.md @@ -80,9 +80,13 @@ iPhone-style dark look (status bar, battery glyph, grouped settings cards). is still TODO. - **Meshtastic BLE companion** — the firmware exposes the official Meshtastic BLE GATT service (`ToRadio`, `FromRadio`, `FromNum`) through NimBLE, advertises, - and the official app discovers + connects. **Open bug:** the session drops - immediately (connect-then-disconnect) — being fixed. (Wi-Fi and BLE are mutually - exclusive on this RAM-tight ESP32-S3; enabling one frees the other.) + and the official app discovers + connects. Serial `companion` status now reports + connect/disconnect counts, last GAP disconnect reason, negotiated MTU, and + characteristic IO counters for phone-app drop captures; the companion handshake + reports Meshtastic-compatible firmware metadata (`2.7.15.567b8ea`) so current + Android builds do not reject it as too old. **Open bug:** phone pairing/send/ + receive validation is still in progress. (Wi-Fi and BLE are mutually exclusive + on this RAM-tight ESP32-S3; enabling one frees the other.) ### 🛠️ Roadmap — versioned plan - ✅ **0.3** — DM profile shortcuts; **Meshtastic DMs (PKI, both ways)**; **delivery @@ -253,8 +257,14 @@ python scripts/tdeck_smoke.py --port /dev/ttyACM0 --no-stub-upload --skip-build ``` The fetch helper uses the current branch and current commit by default, then -downloads the matching successful `Firmware CI` artifact with `gh`. It refuses -to use an older run unless `--allow-latest-success` is passed. +downloads the matching successful `Firmware CI` artifact with `gh`. On fork +branches, it follows the branch's tracking remote, so a branch tracking +`fork/codex/...` fetches from the fork; pass `--repo owner/name` to override. +It refuses to use an older run unless `--allow-latest-success` is passed. + +The artifact flash path does not require a local T-Deck firmware build. It does +need `gh`, Python `esptool` (`python -m pip install esptool` when PlatformIO's +bundled `esptool.py` is absent), and `pyserial` for the post-flash serial smoke. CI runs the native simulator build, native codec selftest, deterministic simulator scenario, screenshot generation, T-Deck firmware build, and T-Deck size report @@ -337,7 +347,9 @@ for local apps and read-only inspection when present. - **Companion bridge controls** — USB companion mode and BLE companion advertising are separate rows in Meshtastic → Nodes. Only one external app transport owns the bridge at a time: enabling BLE returns USB to the serial console; enabling - USB turns BLE advertising/connection off. + USB turns BLE advertising/connection off. `companion` reports BLE session + counters (`c`/`d`), last disconnect reason (`r`), MTU, ToRadio writes, + FromRadio reads, and FromNum reads/writes for official-app debugging. - **Terminal / Files** — Developer Mode mono console with blinking cursor; read-only Files browser for the mounted SD/local store. diff --git a/docs/tdeck-feature-inventory.md b/docs/tdeck-feature-inventory.md index 416182f..2b66901 100644 --- a/docs/tdeck-feature-inventory.md +++ b/docs/tdeck-feature-inventory.md @@ -50,7 +50,7 @@ Status labels: | Retransmit/resend | Partial | Long-press failed bubble calls `lz_svc_resend`; expired pending sent DMs retry automatically from persisted log metadata up to the retry cap | Needs hardware ACK/retransmit validation. | | Managed flood rebroadcast | Functional, needs validation | `rebroadcast` in `backend_sx1262.cpp` | Needs airtime/backoff validation in busy meshes. | | USB companion bridge | Functional, needs validation | `mt_companion.cpp`, serial commands, UI toggle | Config coverage is minimal; USB remains the hardware-tested app bridge. | -| BLE companion bridge | Partial, needs validation | `mt_companion.cpp` exposes Meshtastic BLE `ToRadio`, `FromRadio`, and `FromNum` over NimBLE; Meshtastic Nodes screen has separate USB/BLE companion rows; `companion ble on\|off\|test` reports status and mailbox proof | Needs official app pairing, reconnect, send, receive, disconnect, and coexistence validation on hardware. | +| BLE companion bridge | Partial, needs validation | `mt_companion.cpp` exposes Meshtastic BLE `ToRadio`, `FromRadio`, and `FromNum` over NimBLE; Meshtastic Nodes screen has separate USB/BLE companion rows; `companion ble on\|off\|test` reports status and mailbox proof; the companion handshake reports Meshtastic-compatible firmware/min-app metadata for current Android builds; `companion` reports connect/disconnect counts, last GAP disconnect reason, MTU, and characteristic IO counters for official-app drop captures | Needs official app pairing, reconnect, send, receive, disconnect, and coexistence validation on hardware. | | Position/telemetry decode | Partial, needs validation | POSITION and TELEMETRY payloads decode into node detail, node DB, serial `nodes`, and codec selftest fixtures | Add stock-device validation plus app-facing map/weather consumers. | | Emergency channel/beacon | Prototype/Planned | Emergency row appears but is disabled; design spec covers SOS | Implement after feedback manager and dual-network send. | @@ -109,7 +109,7 @@ Status labels: | OTA firmware update | Planned | Partition table and design spec | Implement download, hash verify, inactive-slot write, rollback UX. | | Feedback Manager | Planned | Design spec section 8 | Centralize LED, buzzer, keyboard/display feedback and DND. | | Emergency beacon | Planned | Design spec section 12, disabled Emergency UI row | Requires Feedback Manager and dual-network messaging. | -| BLE companion | Partial, needs validation | NimBLE-based Meshtastic GATT service, official UUIDs, raw `ToRadio` writes, queued `FromRadio` reads, `FromNum` notifications, UI toggle, and serial selftest/status | Validate with the official Meshtastic app over BLE before calling V0.5 complete. | +| BLE companion | Partial, needs validation | NimBLE-based Meshtastic GATT service, official UUIDs, raw `ToRadio` writes, queued `FromRadio` reads, `FromNum` notifications, UI toggle, serial selftest/status, and session counters for connect/drop diagnostics | Validate with the official Meshtastic app over BLE before calling V0.5 complete. | | CI and release checks | Partial | `.github/workflows/firmware.yml` runs native simulator build, native protocol selftest, deterministic simulator scenario, screenshot generation, T-Deck build, size reporting, an explicit firmware/static-RAM budget gate, and artifact upload with budget metadata plus screenshots | Add protocol vectors beyond the native selftest and hardware evidence gates. | ## Completion Criteria diff --git a/docs/tdeck-firmware-roadmap.md b/docs/tdeck-firmware-roadmap.md index f987e35..f098e21 100644 --- a/docs/tdeck-firmware-roadmap.md +++ b/docs/tdeck-firmware-roadmap.md @@ -111,9 +111,10 @@ Deliverables: - Add BLE transport for the Meshtastic companion protocol. Implemented in firmware with NimBLE-Arduino and the official Meshtastic BLE GATT service UUIDs: `ToRadio` writes, `FromRadio` reads, and `FromNum` read/notify/write. - Reuse the USB companion handshake/model where possible so node DB, channel, config, and packet forwarding behavior stay consistent. Implemented: USB and BLE both feed the same `ToRadio` handler and `FromRadio` builders. +- Report Meshtastic-compatible app metadata during the companion handshake. Implemented: `MyNodeInfo.min_app_version` uses Meshtastic's current compatibility floor and `DeviceMetadata.firmware_version` reports the current stable Meshtastic firmware line (`2.7.15.567b8ea`) so Android does not hard-block the session as ancient firmware. - Add clear UI state for USB companion, BLE companion, and normal serial console mode. Implemented: Meshtastic -> Nodes now has separate USB and BLE companion rows. - Define what happens when USB and BLE companion clients compete for the radio. Implemented: only one external app bridge is active at a time; enabling BLE disables USB companion mode, and enabling USB turns BLE advertising/connection off. -- Add serial diagnostics and a loopback/selftest equivalent for BLE where practical. Implemented: `companion ble on|off|test`, BLE status reporting, and a BLE mailbox/fromnum selftest. +- Add serial diagnostics and a loopback/selftest equivalent for BLE where practical. Implemented: `companion ble on|off|test`, BLE status reporting, and a BLE mailbox/fromnum selftest. The BLE status line now also captures session-level phone-app drop evidence: connect/disconnect counts (`c`/`d`), last GAP disconnect reason (`r`), negotiated MTU, ToRadio writes, FromRadio reads, and FromNum reads/writes. - Hardware-test pairing, reconnect, send, receive, and disconnect flows with the official app. Exit criteria: diff --git a/docs/tdeck-hardware-dogfood-checklist.md b/docs/tdeck-hardware-dogfood-checklist.md index e152594..0b6bd61 100644 --- a/docs/tdeck-hardware-dogfood-checklist.md +++ b/docs/tdeck-hardware-dogfood-checklist.md @@ -38,6 +38,7 @@ dogfood belong to the later roadmap phases. - Capture the boot banner and every `[ok]` or failure line. - Confirm display, touch, keyboard, trackball, SD, SX1262, Wi-Fi state, battery, and time source are reported. - Run `help` and confirm diagnostics include `dm status`, `rxlog`, `nodes`, `net`, `rf`, `companion`, and `companion ble`. +- For BLE companion phone-app drops, run `companion ble on`, attempt the official app connection, then run `companion` after the drop and capture the whole BLE line. The key fields are `c`/`d` (connect/disconnect counts), `r` (last GAP disconnect reason), `mtu`, `to` (ToRadio writes/last bytes), `fr` (FromRadio reads), and `fn` (FromNum reads/writes). ## Hardware Evidence Log diff --git a/scripts/fetch_tdeck_artifact.py b/scripts/fetch_tdeck_artifact.py index 4387463..5d2f155 100644 --- a/scripts/fetch_tdeck_artifact.py +++ b/scripts/fetch_tdeck_artifact.py @@ -36,7 +36,36 @@ def current_commit(project_dir: Path) -> str: return git(project_dir, "rev-parse", "HEAD") +def repo_from_remote_url(url: str) -> str | None: + clean = url.strip() + if clean.endswith(".git"): + clean = clean[:-4] + marker = "github.com" + if marker not in clean: + return None + tail = clean.split(marker, 1)[1].lstrip("/:").strip("/") + parts = tail.split("/") + if len(parts) >= 2 and parts[0] and parts[1]: + return f"{parts[0]}/{parts[1]}" + return None + + +def tracking_repo(project_dir: Path) -> str | None: + branch = current_branch(project_dir) + if not branch: + return None + try: + remote = git(project_dir, "config", "--get", f"branch.{branch}.remote") + url = git(project_dir, "remote", "get-url", remote) + except SystemExit: + return None + return repo_from_remote_url(url) + + def default_repo(project_dir: Path) -> str: + repo = tracking_repo(project_dir) + if repo: + return repo try: data = json.loads(run_text(["gh", "repo", "view", "--json", "nameWithOwner"], project_dir)) repo = data.get("nameWithOwner") diff --git a/scripts/tdeck_smoke.py b/scripts/tdeck_smoke.py index 63b1431..10933b1 100644 --- a/scripts/tdeck_smoke.py +++ b/scripts/tdeck_smoke.py @@ -37,15 +37,25 @@ def platformio_core_dir() -> Path: return Path.home() / ".platformio" -def find_esptool() -> Path: +def find_esptool_cmd() -> list[str]: root = platformio_core_dir() / "packages" / "tool-esptoolpy" candidates = list(root.rglob("esptool.py")) if root.exists() else [] - if not candidates: - raise FileNotFoundError( - "Could not find PlatformIO's esptool.py. Run `pio run -e tdeck` once " - "or install the espressif32 platform." - ) - return candidates[0] + if candidates: + return [sys.executable, str(candidates[0])] + + probe = subprocess.run( + [sys.executable, "-m", "esptool", "version"], + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + ) + if probe.returncode == 0: + return [sys.executable, "-m", "esptool"] + + raise FileNotFoundError( + "Could not find PlatformIO's esptool.py or the Python esptool module. " + "Run `pio run -e tdeck` once, install the espressif32 platform, or " + "install esptool with `python -m pip install esptool`." + ) def find_boot_app0(artifact_dir: Path | None = None) -> Path: @@ -78,11 +88,10 @@ def require_artifacts(project_dir: Path, env_name: str, artifact_dir: Path | Non def nostub_upload(project_dir: Path, env_name: str, port: str, baud: int, artifact_dir: Path | None) -> None: bootloader, partitions, boot_app0, firmware = require_artifacts(project_dir, env_name, artifact_dir) - esptool = find_esptool() + esptool_cmd = find_esptool_cmd() run( [ - sys.executable, - str(esptool), + *esptool_cmd, "--chip", "esp32s3", "--port", @@ -90,16 +99,16 @@ def nostub_upload(project_dir: Path, env_name: str, port: str, baud: int, artifa "--baud", str(baud), "--before", - "default_reset", + "default-reset", "--after", - "hard_reset", + "hard-reset", "--no-stub", - "write_flash", - "--flash_mode", + "write-flash", + "--flash-mode", "dio", - "--flash_freq", + "--flash-freq", "80m", - "--flash_size", + "--flash-size", "16MB", "0x0", str(bootloader), diff --git a/src/mt_companion.cpp b/src/mt_companion.cpp index ec94ae7..15d014e 100644 --- a/src/mt_companion.cpp +++ b/src/mt_companion.cpp @@ -55,6 +55,12 @@ extern "C" bool lz_mtc_active(void) { return g_companion; } #define MTC_BLE_CFG_PER_POLL 2 #define MTC_BLE_CFG_GAP_MS 20 #define MTC_BLE_NOTIFY_DELAY_MS 100 +/* Android hard-blocks firmware older than 2.3.15 and warns below 2.5.14. + * Advertise the current stable Meshtastic firmware line for app compatibility + * without claiming alpha-only 2.7.18+ capabilities this bridge does not expose. */ +#define MTC_COMPAT_FW_VERSION "2.7.15.567b8ea" +#define MTC_COMPAT_MIN_APP_VERSION 30200U +#define MTC_COMPAT_PIO_ENV "tdeck" static NimBLEServer *g_ble_server; static NimBLECharacteristic *g_ble_fromradio; @@ -65,6 +71,18 @@ static NimBLECharacteristic *g_ble_fromnum; static volatile bool g_ble_ready; static volatile bool g_ble_enabled; static volatile bool g_ble_connected; +static volatile uint32_t g_ble_connect_count; +static volatile uint32_t g_ble_disconnect_count; +static volatile uint32_t g_ble_connected_since_ms; +static volatile uint32_t g_ble_last_disconnect_ms; +static volatile uint32_t g_ble_last_io_ms; +static volatile int g_ble_last_disconnect_reason = -1; +static volatile uint16_t g_ble_last_mtu; +static volatile uint16_t g_ble_last_to_len; +static volatile uint32_t g_ble_to_write_count; +static volatile uint32_t g_ble_from_read_count; +static volatile uint32_t g_ble_fromnum_read_count; +static volatile uint32_t g_ble_fromnum_write_count; typedef struct { uint32_t num; @@ -349,8 +367,11 @@ static void send_fromradio_uint(int field, uint32_t v) /* ---------- FromRadio builders ---------- */ static void send_my_info(void) { - uint8_t m[16]; int n = 0; + uint8_t m[48]; int n = 0; if(!pb_put_uint(m, sizeof m, &n, 1, lz_svc_identity()->num)) return; /* my_node_num */ + if(!pb_put_uint(m, sizeof m, &n, 8, 0)) return; /* reboot_count */ + if(!pb_put_uint(m, sizeof m, &n, 11, MTC_COMPAT_MIN_APP_VERSION)) return; + if(!pb_put_str(m, sizeof m, &n, 13, MTC_COMPAT_PIO_ENV)) return; /* pio_env */ send_fromradio(3, m, n); } @@ -388,8 +409,10 @@ static void send_channel_primary(void) /* DeviceMetadata (FromRadio.metadata=13): firmware, role, hw_model */ static void send_metadata(void) { - uint8_t m[48]; int n = 0; - if(!pb_put_str(m, sizeof m, &n, 1, "0.6.0-limitlezz")) return; /* firmware_version */ + uint8_t m[64]; int n = 0; + if(!pb_put_str(m, sizeof m, &n, 1, MTC_COMPAT_FW_VERSION)) return; /* firmware_version */ + if(!pb_put_uint(m, sizeof m, &n, 4, 1)) return; /* hasWifi */ + if(!pb_put_uint(m, sizeof m, &n, 5, 1)) return; /* hasBluetooth */ if(!pb_put_uint(m, sizeof m, &n, 7, 0)) return; /* role = CLIENT */ if(!pb_put_uint(m, sizeof m, &n, 9, 50)) return; /* hw_model = T_DECK */ send_fromradio(13, m, n); @@ -697,6 +720,7 @@ static void handle_toradio(const uint8_t *b, int len, bool from_ble) class MtcBleServerCallbacks : public NimBLEServerCallbacks { void onConnect(NimBLEServer *server, NimBLEConnInfo &connInfo) override { + uint32_t now = millis(); taskENTER_CRITICAL(&g_ble_mux); g_ble_head = g_ble_count = 0; g_ble_to_head = g_ble_to_count = 0; @@ -706,6 +730,15 @@ class MtcBleServerCallbacks : public NimBLEServerCallbacks { g_ble_cfg_active = false; g_ble_cfg_phase = BLE_CFG_IDLE; g_ble_connected = true; + g_ble_connect_count++; + g_ble_connected_since_ms = now; + g_ble_last_io_ms = 0; + g_ble_last_mtu = 0; + g_ble_last_to_len = 0; + g_ble_to_write_count = 0; + g_ble_from_read_count = 0; + g_ble_fromnum_read_count = 0; + g_ble_fromnum_write_count = 0; taskEXIT_CRITICAL(&g_ble_mux); reset_fromradio_ids(); if(server) { @@ -716,7 +749,8 @@ class MtcBleServerCallbacks : public NimBLEServerCallbacks { } void onDisconnect(NimBLEServer *server, NimBLEConnInfo &connInfo, int reason) override { - (void)server; (void)connInfo; (void)reason; + (void)server; (void)connInfo; + uint32_t now = millis(); taskENTER_CRITICAL(&g_ble_mux); g_ble_connected = false; g_ble_head = g_ble_count = 0; @@ -724,12 +758,19 @@ class MtcBleServerCallbacks : public NimBLEServerCallbacks { g_ble_notify_pending = false; g_ble_cfg_active = false; g_ble_cfg_phase = BLE_CFG_IDLE; + g_ble_disconnect_count++; + g_ble_last_disconnect_reason = reason; + g_ble_last_disconnect_ms = now; + g_ble_connected_since_ms = 0; taskEXIT_CRITICAL(&g_ble_mux); if(g_ble_enabled) NimBLEDevice::startAdvertising(); } void onMTUChange(uint16_t mtu, NimBLEConnInfo &connInfo) override { - (void)mtu; (void)connInfo; + (void)connInfo; + taskENTER_CRITICAL(&g_ble_mux); + g_ble_last_mtu = mtu; + taskEXIT_CRITICAL(&g_ble_mux); } }; @@ -741,16 +782,21 @@ class MtcBleToRadioCallbacks : public NimBLECharacteristicCallbacks { { (void)connInfo; std::string v = chr->getValue(); - if(v.empty() || v.size() > MTC_BLE_MAX_PACKET || !g_ble_to_q) return; + size_t vlen = v.size(); + if(v.empty() || vlen > MTC_BLE_MAX_PACKET || !g_ble_to_q) return; + uint32_t now = millis(); taskENTER_CRITICAL(&g_ble_mux); if(g_ble_to_count >= MTC_BLE_QUEUE_N) { /* drop oldest on overflow */ g_ble_to_head = (g_ble_to_head + 1) % MTC_BLE_QUEUE_N; g_ble_to_count--; } int idx = (g_ble_to_head + g_ble_to_count) % MTC_BLE_QUEUE_N; - g_ble_to_q[idx].len = (uint16_t)v.size(); - memcpy(g_ble_to_q[idx].data, v.data(), v.size()); + g_ble_to_q[idx].len = (uint16_t)vlen; + memcpy(g_ble_to_q[idx].data, v.data(), vlen); g_ble_to_count++; + g_ble_to_write_count++; + g_ble_last_to_len = (uint16_t)vlen; + g_ble_last_io_ms = now; taskEXIT_CRITICAL(&g_ble_mux); } }; @@ -759,6 +805,11 @@ class MtcBleFromRadioCallbacks : public NimBLECharacteristicCallbacks { void onRead(NimBLECharacteristic *chr, NimBLEConnInfo &connInfo) override { (void)chr; (void)connInfo; + uint32_t now = millis(); + taskENTER_CRITICAL(&g_ble_mux); + g_ble_from_read_count++; + g_ble_last_io_ms = now; + taskEXIT_CRITICAL(&g_ble_mux); ble_prepare_fromradio_value(); } }; @@ -767,6 +818,11 @@ class MtcBleFromNumCallbacks : public NimBLECharacteristicCallbacks { void onRead(NimBLECharacteristic *chr, NimBLEConnInfo &connInfo) override { (void)chr; (void)connInfo; + uint32_t now = millis(); + taskENTER_CRITICAL(&g_ble_mux); + g_ble_fromnum_read_count++; + g_ble_last_io_ms = now; + taskEXIT_CRITICAL(&g_ble_mux); ble_set_fromnum(g_ble_next_num ? g_ble_next_num - 1 : 0, false); } void onWrite(NimBLECharacteristic *chr, NimBLEConnInfo &connInfo) override @@ -778,8 +834,11 @@ class MtcBleFromNumCallbacks : public NimBLECharacteristicCallbacks { | ((uint32_t)(uint8_t)v[1] << 8) | ((uint32_t)(uint8_t)v[2] << 16) | ((uint32_t)(uint8_t)v[3] << 24); + uint32_t now = millis(); taskENTER_CRITICAL(&g_ble_mux); g_ble_read_num = want; + g_ble_fromnum_write_count++; + g_ble_last_io_ms = now; taskEXIT_CRITICAL(&g_ble_mux); } } @@ -920,21 +979,48 @@ extern "C" void lz_mtc_ble_poll(void) extern "C" int lz_mtc_ble_status(char *buf, int n) { - const char *state = !g_ble_enabled ? "off" - : g_ble_connected ? "connected" - : "advertising"; uint32_t latest; - int from_q, to_q; - bool syncing; + uint32_t connects, disconnects, connected_since, last_disconnect, last_io; + uint32_t to_writes, from_reads, fromnum_reads, fromnum_writes; + int from_q, to_q, last_reason; + uint16_t mtu, last_to_len; + bool enabled, connected, syncing; taskENTER_CRITICAL(&g_ble_mux); + enabled = g_ble_enabled; + connected = g_ble_connected; latest = g_ble_next_num ? g_ble_next_num - 1 : 0; from_q = g_ble_count; to_q = g_ble_to_count; syncing = g_ble_cfg_active; + connects = g_ble_connect_count; + disconnects = g_ble_disconnect_count; + connected_since = g_ble_connected_since_ms; + last_disconnect = g_ble_last_disconnect_ms; + last_io = g_ble_last_io_ms; + last_reason = g_ble_last_disconnect_reason; + mtu = g_ble_last_mtu; + last_to_len = g_ble_last_to_len; + to_writes = g_ble_to_write_count; + from_reads = g_ble_from_read_count; + fromnum_reads = g_ble_fromnum_read_count; + fromnum_writes = g_ble_fromnum_write_count; taskEXIT_CRITICAL(&g_ble_mux); - return snprintf(buf, n, "BLE companion: %s | from_q=%d to_q=%d sync=%s latest=%lu service=%s", + uint32_t now = millis(); + const char *state = !enabled ? "off" : connected ? "connected" : "advertising"; + const char *age_label = connected ? "up" : "down"; + uint32_t age_ms = connected ? (connected_since ? now - connected_since : 0) + : (last_disconnect ? now - last_disconnect : 0); + uint32_t io_age_ms = last_io ? now - last_io : 0; + return snprintf(buf, n, + "BLE companion: %s | q=%d/%d sync=%s seq=%lu c=%lu d=%lu r=%d mtu=%u " + "to=%lu/%uB fr=%lu fn=%lu/%lu %s=%lums io=%lums svc=%.8s", state, from_q, to_q, syncing ? "config" : "idle", - (unsigned long)latest, MTC_BLE_SERVICE_UUID); + (unsigned long)latest, (unsigned long)connects, + (unsigned long)disconnects, last_reason, (unsigned)mtu, + (unsigned long)to_writes, (unsigned)last_to_len, + (unsigned long)from_reads, (unsigned long)fromnum_reads, + (unsigned long)fromnum_writes, age_label, (unsigned long)age_ms, + (unsigned long)io_age_ms, MTC_BLE_SERVICE_UUID); } extern "C" int lz_mtc_ble_selftest(char *buf, int n) @@ -1057,8 +1143,10 @@ extern "C" int lz_mtc_selftest(char *out, int n) frames++; p += len; } bool ok = !g_cap_overflow && my_info && meta && cfg && chan && complete && nonce == 0x1234abcd; - int written = snprintf(out, n, "%d frames | my_info=%d metadata=%d config=%d channel=%d complete=%d nonce=%08x%s -> %s", - frames, my_info, meta, cfg, chan, complete, (unsigned)nonce, + int written = snprintf(out, n, "%d frames | my_info=%d metadata=%d fw=%s min_app=%u config=%d channel=%d complete=%d nonce=%08x%s -> %s", + frames, my_info, meta, MTC_COMPAT_FW_VERSION, + (unsigned)MTC_COMPAT_MIN_APP_VERSION, cfg, chan, complete, + (unsigned)nonce, g_cap_overflow ? " overflow" : "", ok ? "PASS" : "FAIL"); free(g_cap); g_cap = NULL; g_caplen = 0; g_capcap = 0; diff --git a/src/serial_cli.cpp b/src/serial_cli.cpp index b46d44c..48dd005 100644 --- a/src/serial_cli.cpp +++ b/src/serial_cli.cpp @@ -301,14 +301,14 @@ static void cmd_companion(char *args) char b[120]; lz_mtc_ble_selftest(b, sizeof b); Serial.println(b); } } - if(lz_mtc_ble_status) { char b[180]; lz_mtc_ble_status(b, sizeof b); Serial.println(b); } + if(lz_mtc_ble_status) { char b[240]; lz_mtc_ble_status(b, sizeof b); Serial.println(b); } else Serial.println("[--] BLE companion not present"); return; } if(args && strcmp(args, "test") == 0) { if(lz_mtc_selftest) { char b[160]; lz_mtc_selftest(b, sizeof b); Serial.println(b); } if(lz_mtc_ble_selftest) { char b[120]; lz_mtc_ble_selftest(b, sizeof b); Serial.println(b); } - if(lz_mtc_ble_status) { char b[180]; lz_mtc_ble_status(b, sizeof b); Serial.println(b); } + if(lz_mtc_ble_status) { char b[240]; lz_mtc_ble_status(b, sizeof b); Serial.println(b); } else Serial.println("[--] not present"); return; } @@ -320,7 +320,7 @@ static void cmd_companion(char *args) } if(args && strcmp(args, "off") == 0) { lz_mtc_set_active(false); Serial.println("[ok] companion mode OFF"); return; } Serial.printf("USB companion mode: %s (on|off|test)\n", lz_mtc_active() ? "ON" : "off"); - if(lz_mtc_ble_status) { char b[180]; lz_mtc_ble_status(b, sizeof b); Serial.println(b); } + if(lz_mtc_ble_status) { char b[240]; lz_mtc_ble_status(b, sizeof b); Serial.println(b); } } static void cmd_nodes(void) From 235a50f8f1712a9c34ae0345c5c9b5e6da9e5047 Mon Sep 17 00:00:00 2001 From: n30nex Date: Wed, 17 Jun 2026 19:50:58 -0400 Subject: [PATCH 02/64] Document Android BLE companion validation --- README.md | 23 ++++++++++++++--------- docs/tdeck-feature-inventory.md | 6 +++--- docs/tdeck-firmware-roadmap.md | 10 +++++----- docs/tdeck-hardware-dogfood-checklist.md | 9 +++++++++ 4 files changed, 31 insertions(+), 17 deletions(-) diff --git a/README.md b/README.md index f5b9d9e..43769fe 100644 --- a/README.md +++ b/README.md @@ -80,13 +80,15 @@ iPhone-style dark look (status bar, battery glyph, grouped settings cards). is still TODO. - **Meshtastic BLE companion** — the firmware exposes the official Meshtastic BLE GATT service (`ToRadio`, `FromRadio`, `FromNum`) through NimBLE, advertises, - and the official app discovers + connects. Serial `companion` status now reports - connect/disconnect counts, last GAP disconnect reason, negotiated MTU, and - characteristic IO counters for phone-app drop captures; the companion handshake - reports Meshtastic-compatible firmware metadata (`2.7.15.567b8ea`) so current - Android builds do not reject it as too old. **Open bug:** phone pairing/send/ - receive validation is still in progress. (Wi-Fi and BLE are mutually exclusive - on this RAM-tight ESP32-S3; enabling one frees the other.) + and the official Android app discovers + connects. Serial `companion` status + now reports connect/disconnect counts, last GAP disconnect reason, negotiated + MTU, and characteristic IO counters for phone-app drop captures; the companion + handshake reports Meshtastic-compatible firmware metadata (`2.7.15.567b8ea`) + so current Android builds do not reject it as too old. 2026-06-17 COM8 photo + evidence shows Android connected to `limitlessdeck`, populated nodes, and + LongFast send/receive through the app. Remaining checks: reconnect, + disconnect, and coexistence soak. (Wi-Fi and BLE are mutually exclusive on + this RAM-tight ESP32-S3; enabling one frees the other.) ### 🛠️ Roadmap — versioned plan - ✅ **0.3** — DM profile shortcuts; **Meshtastic DMs (PKI, both ways)**; **delivery @@ -106,12 +108,15 @@ iPhone-style dark look (status bar, battery glyph, grouped settings cards). updates the compose pill in place, Contacts uses virtualized rows, and Settings brightness adjusts without a full screen rebuild; chat rebuilds preserve scroll unless pinned to the newest message. Hardware regression is still open. -- ✅ **0.5** — **BLE companion** for Meshtastic: firmware GATT transport in place. +- ✅ **0.5** — **BLE companion** for Meshtastic: firmware GATT transport in + place, official Android app connection and LongFast send/receive validated on + COM8; reconnect/disconnect soak remains to repeat. - 🚀 **0.6 — this release** — **MeshCore is live**: public-channel chat **and** encrypted DMs (X25519 + AES), in the same unified inbox as Meshtastic and time-shared on the one SX1262 by a **split-airtime scheduler** that never cuts an in-flight RX/TX. **BLE companion** merged and running on hardware - (advertising + GATT mailbox; phone pairing/send/receive validation next). A full + (advertising + GATT mailbox; Android app connection, nodes, and LongFast + send/receive validated; reconnect/disconnect soak next). A full **desktop simulator** (virtual mesh + 50+ self-test assertions) to cut hardware testing. **Wi-Fi and BLE are mutually exclusive** on this RAM-tight ESP32-S3 — enabling one frees the other. diff --git a/docs/tdeck-feature-inventory.md b/docs/tdeck-feature-inventory.md index 2b66901..cbe4ec5 100644 --- a/docs/tdeck-feature-inventory.md +++ b/docs/tdeck-feature-inventory.md @@ -50,7 +50,7 @@ Status labels: | Retransmit/resend | Partial | Long-press failed bubble calls `lz_svc_resend`; expired pending sent DMs retry automatically from persisted log metadata up to the retry cap | Needs hardware ACK/retransmit validation. | | Managed flood rebroadcast | Functional, needs validation | `rebroadcast` in `backend_sx1262.cpp` | Needs airtime/backoff validation in busy meshes. | | USB companion bridge | Functional, needs validation | `mt_companion.cpp`, serial commands, UI toggle | Config coverage is minimal; USB remains the hardware-tested app bridge. | -| BLE companion bridge | Partial, needs validation | `mt_companion.cpp` exposes Meshtastic BLE `ToRadio`, `FromRadio`, and `FromNum` over NimBLE; Meshtastic Nodes screen has separate USB/BLE companion rows; `companion ble on\|off\|test` reports status and mailbox proof; the companion handshake reports Meshtastic-compatible firmware/min-app metadata for current Android builds; `companion` reports connect/disconnect counts, last GAP disconnect reason, MTU, and characteristic IO counters for official-app drop captures | Needs official app pairing, reconnect, send, receive, disconnect, and coexistence validation on hardware. | +| BLE companion bridge | Functional, needs soak | `mt_companion.cpp` exposes Meshtastic BLE `ToRadio`, `FromRadio`, and `FromNum` over NimBLE; Meshtastic Nodes screen has separate USB/BLE companion rows; `companion ble on\|off\|test` reports status and mailbox proof; the companion handshake reports Meshtastic-compatible firmware/min-app metadata for current Android builds; 2026-06-17 COM8 Android photo evidence shows `limitlessdeck` connected as firmware `2.7.15.567b8ea`, nodes populated, and LongFast send/receive through the official app; `companion` reports connect/disconnect counts, last GAP disconnect reason, MTU, and characteristic IO counters for follow-up soak captures | Repeat reconnect, disconnect, and coexistence validation on hardware. | | Position/telemetry decode | Partial, needs validation | POSITION and TELEMETRY payloads decode into node detail, node DB, serial `nodes`, and codec selftest fixtures | Add stock-device validation plus app-facing map/weather consumers. | | Emergency channel/beacon | Prototype/Planned | Emergency row appears but is disabled; design spec covers SOS | Implement after feedback manager and dual-network send. | @@ -76,7 +76,7 @@ Status labels: | Home launcher | Partial | filtered app grid, Developer Mode hides Terminal by default, Messages unread counter badge, scanned local apps flow across paged 4x2 Home screens | V0.95: add full app launch/runtime integration; run hardware visual regression for badge layout. | | Unified inbox | Functional/Partial | Messages tabs, filters, unread highlighting, per-thread badges, mute indicator, channel tab | MeshCore filter is gated; finish hardware responsiveness pass. | | Conversation view | Functional/Partial | compose, in-place draft text refresh, scroll-preserving chat rebuilds, bubbles, status colors, resend long-press, persisted sent-DM delivery metadata | Stock-device ACK/retry interop and hardware chat-log latency still need validation. | -| Meshtastic manager | Functional/Partial | identity card, virtualized node list, channels tab, separate USB and BLE companion toggles | Emergency channel row is disabled; BLE companion needs phone hardware validation. | +| Meshtastic manager | Functional/Partial | identity card, virtualized node list, channels tab, separate USB and BLE companion toggles | Emergency channel row is disabled; BLE companion has Android connect/send/receive proof but still needs reconnect/disconnect soak. | | MeshCore manager | Prototype/Partial | "Coming soon" unless gate is flipped; deeper screen exists behind gate | Do not enable until MeshCore message path works. | | Contacts/detail | Functional/Partial | virtualized contacts list, add contact, messageable role check | Trace action is a no-op; MeshCore contacts locked; hardware long-list scroll needs validation. | | Settings | Functional/Partial | network toggles, Wi-Fi, in-place brightness slider updates, time, system, touch calibration, Developer Mode, `settings.cfg` persistence | Add migration/versioning if the settings schema grows; hardware latency pass still needed. | @@ -109,7 +109,7 @@ Status labels: | OTA firmware update | Planned | Partition table and design spec | Implement download, hash verify, inactive-slot write, rollback UX. | | Feedback Manager | Planned | Design spec section 8 | Centralize LED, buzzer, keyboard/display feedback and DND. | | Emergency beacon | Planned | Design spec section 12, disabled Emergency UI row | Requires Feedback Manager and dual-network messaging. | -| BLE companion | Partial, needs validation | NimBLE-based Meshtastic GATT service, official UUIDs, raw `ToRadio` writes, queued `FromRadio` reads, `FromNum` notifications, UI toggle, serial selftest/status, and session counters for connect/drop diagnostics | Validate with the official Meshtastic app over BLE before calling V0.5 complete. | +| BLE companion | Functional, needs soak | NimBLE-based Meshtastic GATT service, official UUIDs, raw `ToRadio` writes, queued `FromRadio` reads, `FromNum` notifications, UI toggle, serial selftest/status, Meshtastic-compatible firmware metadata, Android app connection/nodes/LongFast send-receive photo proof, and session counters for reconnect/drop diagnostics | Repeat reconnect/disconnect/coexistence soak before closing the V0.5 hardware checklist. | | CI and release checks | Partial | `.github/workflows/firmware.yml` runs native simulator build, native protocol selftest, deterministic simulator scenario, screenshot generation, T-Deck build, size reporting, an explicit firmware/static-RAM budget gate, and artifact upload with budget metadata plus screenshots | Add protocol vectors beyond the native selftest and hardware evidence gates. | ## Completion Criteria diff --git a/docs/tdeck-firmware-roadmap.md b/docs/tdeck-firmware-roadmap.md index f098e21..42ffe26 100644 --- a/docs/tdeck-firmware-roadmap.md +++ b/docs/tdeck-firmware-roadmap.md @@ -32,11 +32,11 @@ The firmware is complete when: These maintainer-provided beta labels are the canonical near-term sequence. The broader phases below preserve that order, then add post-V0.96 completion work for OTA, the full App Store, security, feedback, emergency, and release hardening. -**Current release: Beta 0.6.** MeshCore public chat (V0.6) and encrypted DMs (V0.7) are implemented and hardware-verified against a live mesh. **Two open items:** (1) split airtime (the Meshtastic↔MeshCore TDM scheduler) may not be working reliably and needs re-verification; (2) V0.5 BLE companion advertises/serves GATT and the official app connects, but the session drops immediately (connect-then-disconnect). Also delivered this cycle (outside the milestone list): a desktop SDL2 simulator with a 50+ assertion codec/scenario self-test harness, and Wi-Fi/BLE mutual exclusion (they share scarce internal DMA RAM on the ESP32-S3, so only one is resident at a time). +**Current release: Beta 0.6.** MeshCore public chat (V0.6) and encrypted DMs (V0.7) are implemented and hardware-verified against a live mesh. **Open item:** split airtime (the Meshtastic↔MeshCore TDM scheduler) may not be working reliably and needs re-verification. V0.5 BLE companion now advertises/serves GATT, reports current Meshtastic compatibility metadata, connects to the official Android app, populates nodes, and passes LongFast send/receive photo validation on COM8; reconnect/disconnect/coexistence soak remains to repeat. Also delivered this cycle (outside the milestone list): a desktop SDL2 simulator with a 50+ assertion codec/scenario self-test harness, and Wi-Fi/BLE mutual exclusion (they share scarce internal DMA RAM on the ESP32-S3, so only one is resident at a time). | Version | Milestone | Status | | --- | --- | --- | -| V0.5 | BLE companion for Meshtastic | 🚧 Firmware done — advertises + GATT (ToRadio/FromRadio/FromNum) works on hardware; **connect-then-disconnect** with the official app is open | +| V0.5 | BLE companion for Meshtastic | ✅ Firmware + Android interop validation: advertises + GATT (ToRadio/FromRadio/FromNum), current firmware metadata, nodes populated, LongFast send/receive on COM8; reconnect/disconnect soak remains | | V0.6 | MeshCore public chat and split airtime config | 🚧 Public chat send/receive hardware-verified; **split airtime may not be working — needs re-verification**; config UI still TODO | | V0.7 | MeshCore DMs and private chats | ✅ Encrypted DMs (X25519 ECDH + AES) send/receive hardware-verified against a real MeshCore peer | | V0.8 | MeshCore USB companion and MeshCore BLE companion | ⬜ Not started | @@ -105,7 +105,7 @@ Exit criteria: Goal: let the official Meshtastic app connect wirelessly to the T-Deck radio after the USB companion path is stable. -**Status (Beta 0.6): mostly done — one open bug.** BLE transport, the GATT service, the USB/BLE companion UI rows, USB↔BLE arbitration, and the serial selftest are implemented and on hardware. The official app discovers and connects to the radio, but the session drops immediately (**connect-then-disconnect**) — the one open item, tracked as the current top bug (likely BLE bonding/security or the want_config handshake). +**Status (Beta 0.6): Android interop validated, soak still open.** BLE transport, the GATT service, the USB/BLE companion UI rows, USB↔BLE arbitration, current Meshtastic compatibility metadata, and the serial selftest are implemented and on hardware. On 2026-06-17 the official Android app connected to the COM8 T-Deck as `limitlessdeck`, showed firmware `2.7.15.567b8ea`, populated nodes, and exchanged LongFast traffic through the app. Remaining V0.5 validation is reconnect/disconnect behavior and coexistence soak. Deliverables: @@ -115,11 +115,11 @@ Deliverables: - Add clear UI state for USB companion, BLE companion, and normal serial console mode. Implemented: Meshtastic -> Nodes now has separate USB and BLE companion rows. - Define what happens when USB and BLE companion clients compete for the radio. Implemented: only one external app bridge is active at a time; enabling BLE disables USB companion mode, and enabling USB turns BLE advertising/connection off. - Add serial diagnostics and a loopback/selftest equivalent for BLE where practical. Implemented: `companion ble on|off|test`, BLE status reporting, and a BLE mailbox/fromnum selftest. The BLE status line now also captures session-level phone-app drop evidence: connect/disconnect counts (`c`/`d`), last GAP disconnect reason (`r`), negotiated MTU, ToRadio writes, FromRadio reads, and FromNum reads/writes. -- Hardware-test pairing, reconnect, send, receive, and disconnect flows with the official app. +- Hardware-test pairing, reconnect, send, receive, and disconnect flows with the official app. Pairing/connect plus send/receive are validated by 2026-06-17 Android photo evidence on COM8; reconnect/disconnect/coexistence soak remains. Exit criteria: -- A phone can pair over BLE, see the T-Deck as a Meshtastic companion radio, and send/receive through the T-Deck without USB. Firmware path is implemented; official app hardware validation remains open. +- A phone can pair over BLE, see the T-Deck as a Meshtastic companion radio, and send/receive through the T-Deck without USB. Validated on COM8 with the official Android app on 2026-06-17; repeat reconnect/disconnect/coexistence soak before closing all V0.5 hardware notes. - Normal on-device messaging still works when BLE companion is off. ## Phase 3 - V0.6 MeshCore Public Chat And Split Airtime Config diff --git a/docs/tdeck-hardware-dogfood-checklist.md b/docs/tdeck-hardware-dogfood-checklist.md index 0b6bd61..c7a19a6 100644 --- a/docs/tdeck-hardware-dogfood-checklist.md +++ b/docs/tdeck-hardware-dogfood-checklist.md @@ -57,6 +57,15 @@ dogfood belong to the later roadmap phases. - The ROM saved PC `0x420c67ae` decoded against the flashed ELF to `esp_pm_impl_waiti`, which indicates the previous reset happened while the app was idle rather than at a decoded crash site. - Remaining gap: do not count stock Meshtastic peer dogfood as complete from this serial-only evidence. +### 2026-06-17 COM8 Android BLE Companion Validation + +- Firmware artifact flashed before this validation: fork CI artifact for commit `33c8836` on branch `codex/ble-companion-disconnect-diagnostics`. +- COM8 smoke passed after flashing with `scripts/tdeck_smoke.py --no-stub-upload --skip-build`; companion self-test reported Meshtastic-compatible metadata with `fw=2.7.15.567b8ea` and `min_app=30200`. +- Official Android app connected over Bluetooth to `limitlessdeck` and showed firmware version `2.7.15.567b8ea`, clearing the previous "radio firmware is too old" block. +- Android Nodes tab populated through the BLE companion session, showing 2 nodes of 55 total in the captured view. +- Android LongFast chat sent `Test` and `Ping` through the connected T-Deck and received `Pong! --> limitlessdeck`, proving app-side LongFast send/receive over BLE on this firmware. +- Remaining V0.5 soak gap: repeat reconnect, intentional disconnect, and coexistence testing while capturing the serial `companion` line fields (`c`, `d`, `r`, `mtu`, `to`, `fr`, `fn`). + ## Meshtastic Channel Interop - Receive a LongFast text from Peer A on the T-Deck. From 6532fa374a835aa996f1ab34c46804737b8a7c84 Mon Sep 17 00:00:00 2001 From: n30nex Date: Wed, 17 Jun 2026 19:58:34 -0400 Subject: [PATCH 03/64] Add TDM airtime serial smoke probe --- .github/workflows/firmware.yml | 5 + README.md | 12 ++ docs/tdeck-feature-inventory.md | 2 +- docs/tdeck-firmware-roadmap.md | 1 + docs/tdeck-hardware-dogfood-checklist.md | 7 + scripts/tdm_airtime_smoke.py | 263 +++++++++++++++++++++++ 6 files changed, 289 insertions(+), 1 deletion(-) create mode 100644 scripts/tdm_airtime_smoke.py diff --git a/.github/workflows/firmware.yml b/.github/workflows/firmware.yml index fddb228..94f6aa1 100644 --- a/.github/workflows/firmware.yml +++ b/.github/workflows/firmware.yml @@ -43,6 +43,11 @@ jobs: python -m pip install --upgrade pip python -m pip install platformio + - name: Run Python tooling selftests + run: | + python -m py_compile scripts/serial_harness.py scripts/tdm_airtime_smoke.py + python scripts/tdm_airtime_smoke.py --selftest + - name: Build native simulator run: pio run -e native diff --git a/README.md b/README.md index 722c799..20ee486 100644 --- a/README.md +++ b/README.md @@ -256,6 +256,18 @@ The fetch helper uses the current branch and current commit by default, then downloads the matching successful `Firmware CI` artifact with `gh`. It refuses to use an older run unless `--allow-latest-success` is passed. +For Phase 3 split-airtime checks on a MeshCore-enabled build, run the dedicated +serial TDM probe after flashing: + +```sh +python scripts/tdm_airtime_smoke.py --port COM8 +python scripts/tdm_airtime_smoke.py --port /dev/ttyACM0 +``` + +It drives `net`, `airtime`, and `rf`, asserts the 60/40, 50/50, and 40/60 dwell +splits, checks that the TDM switch counter advances, and fails clearly if the +flashed firmware still has MeshCore gated. + CI runs the native simulator build, native codec selftest, deterministic simulator scenario, screenshot generation, T-Deck firmware build, and T-Deck size report in `.github/workflows/firmware.yml`. It also enforces the current T-Deck budget diff --git a/docs/tdeck-feature-inventory.md b/docs/tdeck-feature-inventory.md index 416182f..5ffe667 100644 --- a/docs/tdeck-feature-inventory.md +++ b/docs/tdeck-feature-inventory.md @@ -59,7 +59,7 @@ Status labels: | Feature | Status | Evidence | Gap / Next Action | | --- | --- | --- | --- | | MeshCore compile-time gate | Partial | `#define LZ_MESHCORE_ENABLED 0` | Gate should stay off until receive/send/unified inbox tests pass. | -| TDM radio scheduler | Partial, needs validation | `lz_backend_set_networks`, profile switcher, settings airtime bar | Needs hardware soak and latency/packet-loss measurements. | +| TDM radio scheduler | Partial, needs validation | `lz_backend_set_networks`, profile switcher, settings airtime bar, serial `rf` diagnostics, and `scripts/tdm_airtime_smoke.py` for Windows/Linux dwell + switch-count smoke | Needs hardware soak and latency/packet-loss measurements after the serial probe passes on a MeshCore-enabled image. | | MeshCore RF profile | Partial, needs validation | 910.525 MHz / 62.5 kHz / SF7 / CR4/5 profile | Confirm target regions and RF compatibility with real MeshCore devices. | | MeshCore ADVERT RX | Partial, needs validation | `mc_parse`, `mc_advert_decode`, `lz_core_on_mc_node` | Only ADVERTs are decoded; encrypted payloads are ignored. | | MeshCore self-advert TX | Partial, needs validation | Ed25519 identity, self-advert builder, serial/UI advert commands | Needs interop proof with real MeshCore nodes. | diff --git a/docs/tdeck-firmware-roadmap.md b/docs/tdeck-firmware-roadmap.md index f987e35..a9d54f3 100644 --- a/docs/tdeck-firmware-roadmap.md +++ b/docs/tdeck-firmware-roadmap.md @@ -134,6 +134,7 @@ Deliverables: - missed-packet rate - Meshtastic delivery impact while MeshCore is enabled - MeshCore delivery impact while Meshtastic is enabled + - repeatable serial smoke for dwell presets and switch-count motion. Implemented as `scripts/tdm_airtime_smoke.py`; it runs on Windows `COM8` or Linux/macOS serial paths, checks the 60/40, 50/50, and 40/60 dwell reports, verifies `switches:` advances between `rf` samples, and fails clearly if MeshCore is still compile-gated. - Confirm target MeshCore RF profiles by region and define how they coexist with the LongFast-only product goal. - Build the split airtime config UI around simple choices, not raw radio parameters. Implemented: Settings now exposes Meshtastic first, Balanced, and MeshCore first presets, persists the choice, and reports the active dwell split through serial diagnostics. - Finish MeshCore packet handling: diff --git a/docs/tdeck-hardware-dogfood-checklist.md b/docs/tdeck-hardware-dogfood-checklist.md index e152594..6eeafb9 100644 --- a/docs/tdeck-hardware-dogfood-checklist.md +++ b/docs/tdeck-hardware-dogfood-checklist.md @@ -38,6 +38,13 @@ dogfood belong to the later roadmap phases. - Capture the boot banner and every `[ok]` or failure line. - Confirm display, touch, keyboard, trackball, SD, SX1262, Wi-Fi state, battery, and time source are reported. - Run `help` and confirm diagnostics include `dm status`, `rxlog`, `nodes`, `net`, `rf`, `companion`, and `companion ble`. +- For MeshCore-enabled Phase 3 builds, run the split-airtime probe after the + basic serial smoke: `python scripts/tdm_airtime_smoke.py --port COM8` on the + Windows rig, or pass the Linux/macOS serial path such as `/dev/ttyACM0`. This + asserts the 60/40, 50/50, and 40/60 dwell presets, checks that `switches:` + advances between `rf` samples, and confirms `net mc off` returns the radio to + `Meshtastic 100%`. If the firmware still has MeshCore gated, the probe exits + with a clear failure instead of recording a false TDM pass. ## Hardware Evidence Log diff --git a/scripts/tdm_airtime_smoke.py b/scripts/tdm_airtime_smoke.py new file mode 100644 index 0000000..1f73dd6 --- /dev/null +++ b/scripts/tdm_airtime_smoke.py @@ -0,0 +1,263 @@ +#!/usr/bin/env python3 +""" +Validate the LimitlezzOS split-airtime TDM scheduler over the serial CLI. + +This is a hardware smoke probe, not a firmware build step. It defaults to COM8 +on Windows and /dev/ttyACM0 elsewhere, but accepts --port for any developer rig. +""" +from __future__ import annotations + +import argparse +import os +import re +import sys +import time +from dataclasses import dataclass + +import serial_harness + + +EXPECTED_SPLITS = { + "mt": (60, 40, 300, 200), + "balanced": (50, 50, 250, 250), + "mc": (40, 60, 200, 300), +} + + +@dataclass(frozen=True) +class Airtime: + label: str + mt_pct: int + mc_pct: int + + +@dataclass(frozen=True) +class RfSnapshot: + mode: str + mt_dwell_ms: int + mc_dwell_ms: int + active: str + switches: int + + +class SmokeFailure(RuntimeError): + pass + + +def default_port() -> str: + env_port = os.environ.get("LZ_SERIAL_PORT") + if env_port: + return env_port + return "COM8" if os.name == "nt" else "/dev/ttyACM0" + + +def parse_airtime(text: str) -> Airtime: + match = re.search(r"airtime:\s*(.*?)\s+MT\s+(\d+)%\s*/\s*MC\s+(\d+)%", text) + if not match: + raise SmokeFailure(f"could not parse airtime output:\n{text}") + return Airtime(match.group(1).strip(), int(match.group(2)), int(match.group(3))) + + +def parse_rf(text: str) -> RfSnapshot: + mode = re.search(r"^mode:\s*(.+)$", text, re.MULTILINE) + dwell = re.search(r"^dwell:\s*Meshtastic\s+(\d+)ms\s*/\s*MeshCore\s+(\d+)ms", text, re.MULTILINE) + active = re.search(r"^active:\s*(Meshtastic|MeshCore)\b", text, re.MULTILINE) + switches = re.search(r"^switches:\s*(\d+)", text, re.MULTILINE) + missing = [ + name + for name, found in ( + ("mode", mode), + ("dwell", dwell), + ("active", active), + ("switches", switches), + ) + if not found + ] + if missing: + raise SmokeFailure(f"missing rf field(s) {', '.join(missing)} in:\n{text}") + return RfSnapshot( + mode=mode.group(1).strip(), + mt_dwell_ms=int(dwell.group(1)), + mc_dwell_ms=int(dwell.group(2)), + active=active.group(1), + switches=int(switches.group(1)), + ) + + +def assert_split(snapshot: RfSnapshot, preset: str) -> None: + mt_pct, mc_pct, mt_ms, mc_ms = EXPECTED_SPLITS[preset] + expected_mode = f"SPLIT MT {mt_pct}% / MC {mc_pct}%" + if not snapshot.mode.startswith(expected_mode): + raise SmokeFailure(f"expected mode prefix '{expected_mode}', got '{snapshot.mode}'") + if (snapshot.mt_dwell_ms, snapshot.mc_dwell_ms) != (mt_ms, mc_ms): + raise SmokeFailure( + f"expected dwell {mt_ms}/{mc_ms}ms for {preset}, " + f"got {snapshot.mt_dwell_ms}/{snapshot.mc_dwell_ms}ms" + ) + + +def assert_airtime(airtime: Airtime, preset: str) -> None: + mt_pct, mc_pct, _mt_ms, _mc_ms = EXPECTED_SPLITS[preset] + if (airtime.mt_pct, airtime.mc_pct) != (mt_pct, mc_pct): + raise SmokeFailure( + f"expected airtime split {mt_pct}/{mc_pct}% for {preset}, " + f"got {airtime.mt_pct}/{airtime.mc_pct}%" + ) + + +def run_cmd(port, command: str, timeout: float) -> str: + print(f"[tdm] > {command}") + output = serial_harness.run_command(port, command, timeout) + print(output.rstrip()) + if "[err] MeshCore is gated" in output: + raise SmokeFailure( + "MeshCore is gated in this firmware. Build or flash a MeshCore-enabled " + "TDM image before running split-airtime smoke." + ) + return output + + +def restore_state(port, timeout: float, mt_on: bool, mc_on: bool, preset: str | None) -> None: + try: + if preset: + serial_harness.run_command(port, f"airtime {preset}", timeout) + serial_harness.run_command(port, f"net mt {'on' if mt_on else 'off'}", timeout) + serial_harness.run_command(port, f"net mc {'on' if mc_on else 'off'}", timeout) + except Exception as exc: # pragma: no cover - best-effort hardware cleanup + print(f"[tdm] warning: state restore failed: {exc}", file=sys.stderr) + + +def parse_net_state(text: str) -> tuple[bool, bool]: + match = re.search(r"networks:\s*Meshtastic\s+(on|off),\s*MeshCore\s+(on|off)", text) + if not match: + raise SmokeFailure(f"could not parse network state:\n{text}") + return match.group(1) == "on", match.group(2) == "on" + + +def run_smoke(args: argparse.Namespace) -> None: + print(f"[tdm] opening {args.port} @ {args.baud}") + with serial_harness.open_port_retry( + args.port, args.baud, args.timeout, args.open_timeout, args.dtr, args.rts + ) as port: + if args.reset: + serial_harness.pulse_reset(port, args.reset_settle) + serial_harness.sync_prompt(port, args.boot_timeout) + + initial_net = run_cmd(port, "net", args.timeout) + initial_mt, initial_mc = parse_net_state(initial_net) + initial_airtime = parse_airtime(run_cmd(port, "airtime", args.timeout)) + restore_preset = ( + "balanced" + if (initial_airtime.mt_pct, initial_airtime.mc_pct) == (50, 50) + else "mc" + if (initial_airtime.mt_pct, initial_airtime.mc_pct) == (40, 60) + else "mt" + ) + + try: + run_cmd(port, "net mt on", args.timeout) + run_cmd(port, "net mc on", args.timeout) + + for preset in ("mt", "balanced", "mc"): + airtime = parse_airtime(run_cmd(port, f"airtime {preset}", args.timeout)) + assert_airtime(airtime, preset) + snapshot = parse_rf(run_cmd(port, "rf", args.timeout)) + assert_split(snapshot, preset) + + parse_airtime(run_cmd(port, "airtime balanced", args.timeout)) + first = parse_rf(run_cmd(port, "rf", args.timeout)) + time.sleep(args.settle) + second = parse_rf(run_cmd(port, "rf", args.timeout)) + assert_split(second, "balanced") + if second.switches <= first.switches: + raise SmokeFailure( + f"expected TDM switches to increase after {args.settle:.1f}s; " + f"before={first.switches} after={second.switches}" + ) + + run_cmd(port, "net mc off", args.timeout) + single = parse_rf(run_cmd(port, "rf", args.timeout)) + if single.mode != "Meshtastic 100%": + raise SmokeFailure(f"expected Meshtastic 100% after net mc off, got '{single.mode}'") + + print("[tdm] split-airtime smoke PASS") + finally: + if not args.no_restore: + restore_state(port, args.timeout, initial_mt, initial_mc, restore_preset) + + +def selftest() -> None: + airtime = parse_airtime("airtime: Balanced MT 50% / MC 50%\n") + assert_airtime(airtime, "balanced") + rf1 = parse_rf( + "mode: SPLIT MT 50% / MC 50% (Balanced)\n" + "dwell: Meshtastic 250ms / MeshCore 250ms\n" + "active: Meshtastic 906.875 MHz BW 250.0 SF11 CR4/5 (slot 111ms left)\n" + "Meshtastic: 906.875 MHz BW250 SF11 rx 3\n" + "MeshCore: 910.525 MHz BW62.5 SF7 rx 2\n" + "switches: 41\n" + ) + rf2 = parse_rf( + "mode: SPLIT MT 50% / MC 50% (Balanced)\n" + "dwell: Meshtastic 250ms / MeshCore 250ms\n" + "active: MeshCore 910.525 MHz BW 62.5 SF7 CR4/5 (slot 88ms left)\n" + "Meshtastic: 906.875 MHz BW250 SF11 rx 3\n" + "MeshCore: 910.525 MHz BW62.5 SF7 rx 2\n" + "switches: 44\n" + ) + assert_split(rf1, "balanced") + assert_split(rf2, "balanced") + if rf2.switches <= rf1.switches: + raise SmokeFailure("selftest switch counter did not increase") + single = parse_rf( + "mode: Meshtastic 100%\n" + "dwell: Meshtastic 250ms / MeshCore 250ms\n" + "active: Meshtastic 906.875 MHz BW 250.0 SF11 CR4/5 (slot 0ms left)\n" + "Meshtastic: 906.875 MHz BW250 SF11 rx 3\n" + "MeshCore: 910.525 MHz BW62.5 SF7 rx 2\n" + "switches: 44\n" + ) + if single.mode != "Meshtastic 100%": + raise SmokeFailure("selftest failed to parse single-network mode") + print("[tdm] parser selftest PASS") + + +def main() -> int: + parser = argparse.ArgumentParser(description="Smoke-test split-airtime TDM over the LimitlezzOS serial CLI.") + parser.add_argument("--port", default=default_port()) + parser.add_argument("--baud", type=int, default=115200) + parser.add_argument("--timeout", type=float, default=30.0) + parser.add_argument("--open-timeout", type=float, default=60.0) + parser.add_argument("--boot-timeout", type=float, default=45.0) + parser.add_argument("--settle", type=float, default=1.4, help="Seconds to wait between rf samples.") + parser.add_argument("--dtr", choices=["default", "on", "off"], default="off") + parser.add_argument("--rts", choices=["default", "on", "off"], default="off") + parser.add_argument("--reset", action="store_true", help="Pulse reset before waiting for the prompt.") + parser.add_argument("--reset-settle", type=float, default=1.5) + parser.add_argument("--no-restore", action="store_true", help="Leave the tested network/airtime state active.") + parser.add_argument("--selftest", action="store_true", help="Run parser checks without opening serial.") + args = parser.parse_args() + + try: + if args.selftest: + selftest() + else: + run_smoke(args) + return 0 + except SmokeFailure as exc: + print(f"[tdm] FAIL: {exc}", file=sys.stderr) + return 2 + except serial_harness.RomDownloadMode as exc: + print("[tdm] device is in ESP-ROM download mode, not LimitlezzOS", file=sys.stderr) + print(str(exc).strip(), file=sys.stderr) + return 3 + except TimeoutError as exc: + partial = str(exc).strip() + print("[tdm] timed out waiting for prompt/output", file=sys.stderr) + if partial: + print(partial, file=sys.stderr) + return 1 + + +if __name__ == "__main__": + raise SystemExit(main()) From b3c9602632627d15b958e1cfffaa3a577550a5f2 Mon Sep 17 00:00:00 2001 From: n30nex Date: Wed, 17 Jun 2026 20:08:17 -0400 Subject: [PATCH 04/64] Add MeshCore TDM validation artifact --- .github/workflows/firmware.yml | 69 +++++++++++++++++++++++- README.md | 8 ++- docs/tdeck-feature-inventory.md | 2 +- docs/tdeck-firmware-roadmap.md | 2 +- docs/tdeck-hardware-dogfood-checklist.md | 17 +++--- platformio.ini | 9 ++++ scripts/fetch_tdeck_artifact.py | 53 +++++++++++++++--- scripts/tdeck_smoke.py | 42 +++++++++------ 8 files changed, 167 insertions(+), 35 deletions(-) diff --git a/.github/workflows/firmware.yml b/.github/workflows/firmware.yml index 94f6aa1..8a7185e 100644 --- a/.github/workflows/firmware.yml +++ b/.github/workflows/firmware.yml @@ -45,7 +45,7 @@ jobs: - name: Run Python tooling selftests run: | - python -m py_compile scripts/serial_harness.py scripts/tdm_airtime_smoke.py + python -m py_compile scripts/check_tdeck_budget.py scripts/fetch_tdeck_artifact.py scripts/serial_harness.py scripts/tdeck_smoke.py scripts/tdm_airtime_smoke.py python scripts/tdm_airtime_smoke.py --selftest - name: Build native simulator @@ -125,6 +125,73 @@ jobs: .pio/build/tdeck/tdeck-build.txt .pio/build/tdeck/tdeck-size.txt + - name: Build MeshCore-enabled TDM firmware + run: | + set -o pipefail + pio run -e tdeck-meshcore 2>&1 | tee tdeck-meshcore-build.txt + + - name: Capture MeshCore-enabled size report and budget + run: | + pio run -e tdeck-meshcore -t size | tee tdeck-meshcore-size.txt + cp tdeck-meshcore-build.txt .pio/build/tdeck-meshcore/tdeck-build.txt + cp tdeck-meshcore-size.txt .pio/build/tdeck-meshcore/tdeck-size.txt + python scripts/check_tdeck_budget.py \ + --firmware .pio/build/tdeck-meshcore/firmware.bin \ + --size-report tdeck-meshcore-build.txt \ + --json-out .pio/build/tdeck-meshcore/size-budget.json \ + --markdown-out .pio/build/tdeck-meshcore/SIZE_BUDGET.md \ + --manifest-out .pio/build/tdeck-meshcore/SIZE_BUDGET.txt + { + echo "## MeshCore-enabled T-Deck firmware budget" + echo + cat .pio/build/tdeck-meshcore/SIZE_BUDGET.md + echo + echo "## MeshCore-enabled T-Deck firmware size" + echo + echo "\`\`\`" + cat tdeck-meshcore-size.txt + echo "\`\`\`" + echo + echo "- firmware.bin: $(stat -c%s .pio/build/tdeck-meshcore/firmware.bin) bytes" + echo "- firmware.elf: $(stat -c%s .pio/build/tdeck-meshcore/firmware.elf) bytes" + } >> "$GITHUB_STEP_SUMMARY" + + - name: Prepare MeshCore-enabled flash bundle + env: + ARTIFACT_SHA: ${{ github.event.pull_request.head.sha || github.sha }} + run: | + cp ~/.platformio/packages/framework-arduinoespressif32/tools/partitions/boot_app0.bin .pio/build/tdeck-meshcore/boot_app0.bin + { + echo "repo=${GITHUB_REPOSITORY}" + echo "sha=${ARTIFACT_SHA}" + echo "github_sha=${GITHUB_SHA}" + echo "workflow=${GITHUB_WORKFLOW}" + echo "run_id=${GITHUB_RUN_ID}" + echo "env=tdeck-meshcore" + echo "meshcore_enabled=1" + echo "flash_offsets=0x0 bootloader.bin 0x8000 partitions.bin 0xe000 boot_app0.bin 0x10000 firmware.bin" + cat .pio/build/tdeck-meshcore/SIZE_BUDGET.txt + } > .pio/build/tdeck-meshcore/FLASH_MANIFEST.txt + + - name: Upload MeshCore-enabled TDM artifacts + uses: actions/upload-artifact@v7 + with: + name: tdeck-meshcore-firmware-${{ github.event.pull_request.head.sha || github.sha }} + if-no-files-found: error + path: | + .pio/build/tdeck-meshcore/bootloader.bin + .pio/build/tdeck-meshcore/boot_app0.bin + .pio/build/tdeck-meshcore/firmware.bin + .pio/build/tdeck-meshcore/firmware.elf + .pio/build/tdeck-meshcore/firmware.map + .pio/build/tdeck-meshcore/partitions.bin + .pio/build/tdeck-meshcore/FLASH_MANIFEST.txt + .pio/build/tdeck-meshcore/SIZE_BUDGET.md + .pio/build/tdeck-meshcore/SIZE_BUDGET.txt + .pio/build/tdeck-meshcore/size-budget.json + .pio/build/tdeck-meshcore/tdeck-build.txt + .pio/build/tdeck-meshcore/tdeck-size.txt + - name: Upload native screenshots uses: actions/upload-artifact@v7 with: diff --git a/README.md b/README.md index 20ee486..b089d7d 100644 --- a/README.md +++ b/README.md @@ -257,10 +257,16 @@ downloads the matching successful `Firmware CI` artifact with `gh`. It refuses to use an older run unless `--allow-latest-success` is passed. For Phase 3 split-airtime checks on a MeshCore-enabled build, run the dedicated -serial TDM probe after flashing: +serial TDM probe after flashing. The default `tdeck` firmware stays conservative; +CI also uploads an opt-in `tdeck-meshcore` flash bundle for this validation path: ```sh +python scripts/fetch_tdeck_artifact.py --env tdeck-meshcore +python scripts/tdeck_smoke.py --port COM8 --env tdeck-meshcore --no-stub-upload --skip-build --artifact-dir .pio/ci-artifacts/tdeck-meshcore python scripts/tdm_airtime_smoke.py --port COM8 + +python scripts/fetch_tdeck_artifact.py --env tdeck-meshcore +python scripts/tdeck_smoke.py --port /dev/ttyACM0 --env tdeck-meshcore --no-stub-upload --skip-build --artifact-dir .pio/ci-artifacts/tdeck-meshcore python scripts/tdm_airtime_smoke.py --port /dev/ttyACM0 ``` diff --git a/docs/tdeck-feature-inventory.md b/docs/tdeck-feature-inventory.md index 5ffe667..5ac89e1 100644 --- a/docs/tdeck-feature-inventory.md +++ b/docs/tdeck-feature-inventory.md @@ -59,7 +59,7 @@ Status labels: | Feature | Status | Evidence | Gap / Next Action | | --- | --- | --- | --- | | MeshCore compile-time gate | Partial | `#define LZ_MESHCORE_ENABLED 0` | Gate should stay off until receive/send/unified inbox tests pass. | -| TDM radio scheduler | Partial, needs validation | `lz_backend_set_networks`, profile switcher, settings airtime bar, serial `rf` diagnostics, and `scripts/tdm_airtime_smoke.py` for Windows/Linux dwell + switch-count smoke | Needs hardware soak and latency/packet-loss measurements after the serial probe passes on a MeshCore-enabled image. | +| TDM radio scheduler | Partial, needs validation | `lz_backend_set_networks`, profile switcher, settings airtime bar, serial `rf` diagnostics, opt-in `tdeck-meshcore` CI artifact, and `scripts/tdm_airtime_smoke.py` for Windows/Linux dwell + switch-count smoke | Needs hardware soak and latency/packet-loss measurements after the serial probe passes on a MeshCore-enabled image. | | MeshCore RF profile | Partial, needs validation | 910.525 MHz / 62.5 kHz / SF7 / CR4/5 profile | Confirm target regions and RF compatibility with real MeshCore devices. | | MeshCore ADVERT RX | Partial, needs validation | `mc_parse`, `mc_advert_decode`, `lz_core_on_mc_node` | Only ADVERTs are decoded; encrypted payloads are ignored. | | MeshCore self-advert TX | Partial, needs validation | Ed25519 identity, self-advert builder, serial/UI advert commands | Needs interop proof with real MeshCore nodes. | diff --git a/docs/tdeck-firmware-roadmap.md b/docs/tdeck-firmware-roadmap.md index a9d54f3..6876778 100644 --- a/docs/tdeck-firmware-roadmap.md +++ b/docs/tdeck-firmware-roadmap.md @@ -134,7 +134,7 @@ Deliverables: - missed-packet rate - Meshtastic delivery impact while MeshCore is enabled - MeshCore delivery impact while Meshtastic is enabled - - repeatable serial smoke for dwell presets and switch-count motion. Implemented as `scripts/tdm_airtime_smoke.py`; it runs on Windows `COM8` or Linux/macOS serial paths, checks the 60/40, 50/50, and 40/60 dwell reports, verifies `switches:` advances between `rf` samples, and fails clearly if MeshCore is still compile-gated. + - repeatable serial smoke for dwell presets and switch-count motion. Implemented as `scripts/tdm_airtime_smoke.py` plus an opt-in `tdeck-meshcore` CI artifact; it runs on Windows `COM8` or Linux/macOS serial paths, checks the 60/40, 50/50, and 40/60 dwell reports, verifies `switches:` advances between `rf` samples, and fails clearly if MeshCore is still compile-gated. - Confirm target MeshCore RF profiles by region and define how they coexist with the LongFast-only product goal. - Build the split airtime config UI around simple choices, not raw radio parameters. Implemented: Settings now exposes Meshtastic first, Balanced, and MeshCore first presets, persists the choice, and reports the active dwell split through serial diagnostics. - Finish MeshCore packet handling: diff --git a/docs/tdeck-hardware-dogfood-checklist.md b/docs/tdeck-hardware-dogfood-checklist.md index 6eeafb9..2ddc4c8 100644 --- a/docs/tdeck-hardware-dogfood-checklist.md +++ b/docs/tdeck-hardware-dogfood-checklist.md @@ -38,13 +38,16 @@ dogfood belong to the later roadmap phases. - Capture the boot banner and every `[ok]` or failure line. - Confirm display, touch, keyboard, trackball, SD, SX1262, Wi-Fi state, battery, and time source are reported. - Run `help` and confirm diagnostics include `dm status`, `rxlog`, `nodes`, `net`, `rf`, `companion`, and `companion ble`. -- For MeshCore-enabled Phase 3 builds, run the split-airtime probe after the - basic serial smoke: `python scripts/tdm_airtime_smoke.py --port COM8` on the - Windows rig, or pass the Linux/macOS serial path such as `/dev/ttyACM0`. This - asserts the 60/40, 50/50, and 40/60 dwell presets, checks that `switches:` - advances between `rf` samples, and confirms `net mc off` returns the radio to - `Meshtastic 100%`. If the firmware still has MeshCore gated, the probe exits - with a clear failure instead of recording a false TDM pass. +- For MeshCore-enabled Phase 3 builds, fetch and flash the opt-in CI artifact + with `python scripts/fetch_tdeck_artifact.py --env tdeck-meshcore`, then + `python scripts/tdeck_smoke.py --port COM8 --env tdeck-meshcore --no-stub-upload --skip-build --artifact-dir .pio/ci-artifacts/tdeck-meshcore`. + Run the split-airtime probe after the basic serial smoke: + `python scripts/tdm_airtime_smoke.py --port COM8` on the Windows rig, or pass + the Linux/macOS serial path such as `/dev/ttyACM0`. This asserts the 60/40, + 50/50, and 40/60 dwell presets, checks that `switches:` advances between `rf` + samples, and confirms `net mc off` returns the radio to `Meshtastic 100%`. If + the firmware still has MeshCore gated, the probe exits with a clear failure + instead of recording a false TDM pass. ## Hardware Evidence Log diff --git a/platformio.ini b/platformio.ini index c1014d5..8acf435 100644 --- a/platformio.ini +++ b/platformio.ini @@ -40,6 +40,15 @@ build_flags = ; Display: LovyanGFX (configured in code for the T-Deck ST7789V) — the ; known-good driver Meshtastic uses; manages the shared SPI bus correctly. +; Opt-in Phase 3 validation image. This keeps the default `tdeck` firmware +; conservative while letting CI produce an artifact that can run the real +; Meshtastic/MeshCore TDM scheduler on COM8 or /dev/ttyACM0. +[env:tdeck-meshcore] +extends = env:tdeck +build_flags = + ${env:tdeck.build_flags} + -D LZ_MESHCORE_ENABLED=1 + [env:native] platform = native build_src_filter = +<*> - +<../sim/> diff --git a/scripts/fetch_tdeck_artifact.py b/scripts/fetch_tdeck_artifact.py index 4387463..8e99166 100644 --- a/scripts/fetch_tdeck_artifact.py +++ b/scripts/fetch_tdeck_artifact.py @@ -36,7 +36,36 @@ def current_commit(project_dir: Path) -> str: return git(project_dir, "rev-parse", "HEAD") +def repo_from_remote_url(url: str) -> str | None: + raw = url.strip() + if raw.startswith("git@github.com:"): + raw = raw.removeprefix("git@github.com:") + elif raw.startswith("https://github.com/"): + raw = raw.removeprefix("https://github.com/") + elif raw.startswith("http://github.com/"): + raw = raw.removeprefix("http://github.com/") + else: + return None + raw = raw.removesuffix(".git").strip("/") + parts = raw.split("/") + if len(parts) >= 2 and parts[0] and parts[1]: + return f"{parts[0]}/{parts[1]}" + return None + + +def tracking_repo(project_dir: Path) -> str | None: + try: + upstream = git(project_dir, "rev-parse", "--abbrev-ref", "--symbolic-full-name", "@{u}") + remote = upstream.split("/", 1)[0] + return repo_from_remote_url(git(project_dir, "remote", "get-url", remote)) + except SystemExit: + return None + + def default_repo(project_dir: Path) -> str: + repo = tracking_repo(project_dir) + if repo: + return repo try: data = json.loads(run_text(["gh", "repo", "view", "--json", "nameWithOwner"], project_dir)) repo = data.get("nameWithOwner") @@ -114,16 +143,22 @@ def bundle_dir(out_dir: Path) -> Path: raise SystemExit(f"Multiple possible artifact bundle directories found:\n{choices}") +def artifact_prefix(env_name: str) -> str: + return "tdeck-meshcore-firmware" if env_name == "tdeck-meshcore" else "tdeck-firmware" + + def main() -> int: parser = argparse.ArgumentParser(description="Download the current branch T-Deck firmware artifact.") parser.add_argument("--project-dir", default=Path(__file__).resolve().parents[1]) parser.add_argument("--repo", help="GitHub repo to read Actions artifacts from, e.g. ItsLimitlezz/LimitlezzOS.") parser.add_argument("--workflow", default="Firmware CI") + parser.add_argument("--env", choices=["tdeck", "tdeck-meshcore"], default="tdeck", + help="Firmware artifact flavor to download.") parser.add_argument("--branch", help="Branch name. Defaults to the current git branch.") parser.add_argument("--commit", help="Commit SHA. Defaults to HEAD.") parser.add_argument("--run-id", type=int, help="Download a specific run instead of searching.") parser.add_argument("--artifact-name", help="Artifact name. Defaults to tdeck-firmware-.") - parser.add_argument("--out", default=Path(".pio") / "ci-artifacts" / "tdeck") + parser.add_argument("--out", help="Output directory. Defaults to .pio/ci-artifacts/.") parser.add_argument("--allow-latest-success", action="store_true", help="Allow newest successful run if HEAD has none.") args = parser.parse_args() @@ -131,7 +166,8 @@ def main() -> int: repo = args.repo or default_repo(project_dir) branch = args.branch or current_branch(project_dir) commit = args.commit or current_commit(project_dir) - out_dir = (project_dir / args.out).resolve() if not Path(args.out).is_absolute() else Path(args.out) + out_arg = Path(args.out) if args.out else Path(".pio") / "ci-artifacts" / args.env + out_dir = (project_dir / out_arg).resolve() if not out_arg.is_absolute() else out_arg if args.run_id is not None: run_id = args.run_id @@ -144,18 +180,19 @@ def main() -> int: artifact_sha = chosen["headSha"] run_url = chosen["url"] - artifact_name = args.artifact_name or f"tdeck-firmware-{artifact_sha}" + prefix = artifact_prefix(args.env) + artifact_name = args.artifact_name or f"{prefix}-{artifact_sha}" if args.artifact_name is None: artifacts = load_artifacts(project_dir, repo, run_id) names = [a.get("name", "") for a in artifacts] if artifact_name not in names: - tdeck_names = [name for name in names if name.startswith("tdeck-firmware-")] - if len(tdeck_names) == 1: + matching_names = [name for name in names if name.startswith(f"{prefix}-")] + if len(matching_names) == 1: print( - f"[artifact] expected {artifact_name}, using run artifact {tdeck_names[0]}", + f"[artifact] expected {artifact_name}, using run artifact {matching_names[0]}", file=sys.stderr, ) - artifact_name = tdeck_names[0] + artifact_name = matching_names[0] else: available = "\n".join(f" {name}" for name in names) raise SystemExit( @@ -185,7 +222,7 @@ def main() -> int: print(f"[artifact] name: {artifact_name}") print(f"[artifact] dir: {bundle}") print("[artifact] flash:") - print(f" python scripts/tdeck_smoke.py --port COM8 --no-stub-upload --skip-build --artifact-dir {bundle}") + print(f" python scripts/tdeck_smoke.py --port COM8 --env {args.env} --no-stub-upload --skip-build --artifact-dir {bundle}") return 0 diff --git a/scripts/tdeck_smoke.py b/scripts/tdeck_smoke.py index 63b1431..6813bb6 100644 --- a/scripts/tdeck_smoke.py +++ b/scripts/tdeck_smoke.py @@ -37,15 +37,26 @@ def platformio_core_dir() -> Path: return Path.home() / ".platformio" -def find_esptool() -> Path: +def find_esptool_cmd() -> list[str]: root = platformio_core_dir() / "packages" / "tool-esptoolpy" candidates = list(root.rglob("esptool.py")) if root.exists() else [] - if not candidates: - raise FileNotFoundError( - "Could not find PlatformIO's esptool.py. Run `pio run -e tdeck` once " - "or install the espressif32 platform." - ) - return candidates[0] + if candidates: + return [sys.executable, str(candidates[0])] + + probe = subprocess.run( + [sys.executable, "-m", "esptool", "version"], + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + check=False, + ) + if probe.returncode == 0: + return [sys.executable, "-m", "esptool"] + + raise FileNotFoundError( + "Could not find PlatformIO's esptool.py or `python -m esptool`. Run " + "`pio run -e tdeck` once, install the espressif32 platform, or install " + "esptool with `python -m pip install esptool`." + ) def find_boot_app0(artifact_dir: Path | None = None) -> Path: @@ -78,11 +89,10 @@ def require_artifacts(project_dir: Path, env_name: str, artifact_dir: Path | Non def nostub_upload(project_dir: Path, env_name: str, port: str, baud: int, artifact_dir: Path | None) -> None: bootloader, partitions, boot_app0, firmware = require_artifacts(project_dir, env_name, artifact_dir) - esptool = find_esptool() + esptool_cmd = find_esptool_cmd() run( [ - sys.executable, - str(esptool), + *esptool_cmd, "--chip", "esp32s3", "--port", @@ -90,16 +100,16 @@ def nostub_upload(project_dir: Path, env_name: str, port: str, baud: int, artifa "--baud", str(baud), "--before", - "default_reset", + "default-reset", "--after", - "hard_reset", + "hard-reset", "--no-stub", - "write_flash", - "--flash_mode", + "write-flash", + "--flash-mode", "dio", - "--flash_freq", + "--flash-freq", "80m", - "--flash_size", + "--flash-size", "16MB", "0x0", str(bootloader), From 1552fa8560768a7c0d1be97dc5e8d73bae83cf4a Mon Sep 17 00:00:00 2001 From: n30nex Date: Wed, 17 Jun 2026 20:16:17 -0400 Subject: [PATCH 05/64] Polish MeshCore artifact workflow --- README.md | 12 +++++---- platformio.ini | 7 ++--- scripts/fetch_tdeck_artifact.py | 2 +- scripts/tdeck_smoke.py | 46 ++++++++++++++++++++++++++++----- 4 files changed, 52 insertions(+), 15 deletions(-) diff --git a/README.md b/README.md index b089d7d..c939a46 100644 --- a/README.md +++ b/README.md @@ -275,11 +275,13 @@ splits, checks that the TDM switch counter advances, and fails clearly if the flashed firmware still has MeshCore gated. CI runs the native simulator build, native codec selftest, deterministic simulator -scenario, screenshot generation, T-Deck firmware build, and T-Deck size report -in `.github/workflows/firmware.yml`. It also enforces the current T-Deck budget -gate (2,200,000 bytes for `firmware.bin`, 307,200 bytes static RAM), writes the -result into `FLASH_MANIFEST.txt`, then uploads the firmware artifacts from -`.pio/build/tdeck` plus the generated simulator screenshots. +scenario, screenshot generation, the default T-Deck firmware build, the opt-in +MeshCore-enabled TDM validation build, and size reports for both firmware +artifacts in `.github/workflows/firmware.yml`. It also enforces the current +T-Deck budget gate (2,200,000 bytes for `firmware.bin`, 307,200 bytes static +RAM), writes each result into its own `FLASH_MANIFEST.txt`, then uploads the +`tdeck-firmware-` and `tdeck-meshcore-firmware-` bundles plus the +generated simulator screenshots. Current footprint: ~1.48 MB flash (28.2% of the 5 MB OTA slot), 271 KB static RAM (82.7%) — the rest of RAM is PSRAM-backed double framebuffers. Message history, diff --git a/platformio.ini b/platformio.ini index 8acf435..8d9e8ed 100644 --- a/platformio.ini +++ b/platformio.ini @@ -1,8 +1,9 @@ ; LimitlezzOS — mesh-native handheld OS for LilyGO T-Deck ; -; Two environments: -; tdeck — ESP32-S3 firmware (Arduino core, TFT_eSPI, LVGL 8.3) -; native — desktop simulator (SDL2) sharing the exact same UI code +; Main environments: +; tdeck — ESP32-S3 firmware (Arduino core, TFT_eSPI, LVGL 8.3) +; tdeck-meshcore — opt-in Phase 3 validation image with MeshCore enabled +; native — desktop simulator (SDL2) sharing the exact same UI code ; ; LVGL is pinned to 8.3.11 per the master spec ("Lock LVGL version immediately"). diff --git a/scripts/fetch_tdeck_artifact.py b/scripts/fetch_tdeck_artifact.py index 8e99166..5e118f3 100644 --- a/scripts/fetch_tdeck_artifact.py +++ b/scripts/fetch_tdeck_artifact.py @@ -157,7 +157,7 @@ def main() -> int: parser.add_argument("--branch", help="Branch name. Defaults to the current git branch.") parser.add_argument("--commit", help="Commit SHA. Defaults to HEAD.") parser.add_argument("--run-id", type=int, help="Download a specific run instead of searching.") - parser.add_argument("--artifact-name", help="Artifact name. Defaults to tdeck-firmware-.") + parser.add_argument("--artifact-name", help="Artifact name. Defaults to the selected env prefix plus .") parser.add_argument("--out", help="Output directory. Defaults to .pio/ci-artifacts/.") parser.add_argument("--allow-latest-success", action="store_true", help="Allow newest successful run if HEAD has none.") args = parser.parse_args() diff --git a/scripts/tdeck_smoke.py b/scripts/tdeck_smoke.py index 6813bb6..3e329e7 100644 --- a/scripts/tdeck_smoke.py +++ b/scripts/tdeck_smoke.py @@ -10,6 +10,7 @@ import argparse import os +import re import subprocess import sys from pathlib import Path @@ -59,6 +60,38 @@ def find_esptool_cmd() -> list[str]: ) +def esptool_major(cmd: list[str]) -> int | None: + probe = subprocess.run( + [*cmd, "version"], + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, + text=True, + check=False, + ) + match = re.search(r"v?(\d+)\.\d+", probe.stdout) + return int(match.group(1)) if match else None + + +def esptool_names(cmd: list[str]) -> dict[str, str]: + if (esptool_major(cmd) or 5) >= 5: + return { + "default_reset": "default-reset", + "hard_reset": "hard-reset", + "write_flash": "write-flash", + "flash_mode": "--flash-mode", + "flash_freq": "--flash-freq", + "flash_size": "--flash-size", + } + return { + "default_reset": "default_reset", + "hard_reset": "hard_reset", + "write_flash": "write_flash", + "flash_mode": "--flash_mode", + "flash_freq": "--flash_freq", + "flash_size": "--flash_size", + } + + def find_boot_app0(artifact_dir: Path | None = None) -> Path: if artifact_dir is not None: bundled = artifact_dir / "boot_app0.bin" @@ -90,6 +123,7 @@ def require_artifacts(project_dir: Path, env_name: str, artifact_dir: Path | Non def nostub_upload(project_dir: Path, env_name: str, port: str, baud: int, artifact_dir: Path | None) -> None: bootloader, partitions, boot_app0, firmware = require_artifacts(project_dir, env_name, artifact_dir) esptool_cmd = find_esptool_cmd() + names = esptool_names(esptool_cmd) run( [ *esptool_cmd, @@ -100,16 +134,16 @@ def nostub_upload(project_dir: Path, env_name: str, port: str, baud: int, artifa "--baud", str(baud), "--before", - "default-reset", + names["default_reset"], "--after", - "hard-reset", + names["hard_reset"], "--no-stub", - "write-flash", - "--flash-mode", + names["write_flash"], + names["flash_mode"], "dio", - "--flash-freq", + names["flash_freq"], "80m", - "--flash-size", + names["flash_size"], "16MB", "0x0", str(bootloader), From 88e97315d56b1397cd9eab9ba5a1245ca5348f7d Mon Sep 17 00:00:00 2001 From: n30nex Date: Wed, 17 Jun 2026 20:33:15 -0400 Subject: [PATCH 06/64] Guard blocked Windows serial ports --- scripts/serial_harness.py | 63 ++++++++++++++++++++++++++++++++++++++- scripts/tdeck_smoke.py | 11 +++++-- 2 files changed, 71 insertions(+), 3 deletions(-) diff --git a/scripts/serial_harness.py b/scripts/serial_harness.py index 70203ec..91ee735 100644 --- a/scripts/serial_harness.py +++ b/scripts/serial_harness.py @@ -14,6 +14,7 @@ try: import serial + from serial.tools import list_ports except ImportError as exc: # pragma: no cover - host setup guard raise SystemExit("pyserial is required. PlatformIO installs it, or run: pip install pyserial") from exc @@ -30,12 +31,70 @@ "companion test": "PASS", } ROM_DOWNLOAD_MARKERS = ("waiting for download", "DOWNLOAD(USB/UART0)", "DOWNLOAD") +AUTO_PORT_NAMES = {"auto", "esp32", "tdeck"} +ESPRESSIF_USB_IDS = {(0x303A, 0x1001)} +FORBIDDEN_PORT_NAMES = {"COM11", "COM29"} class RomDownloadMode(RuntimeError): pass +def is_auto_port(name: str) -> bool: + return name.strip().lower() in AUTO_PORT_NAMES + + +def is_forbidden_port(name: str) -> bool: + return name.strip().upper() in FORBIDDEN_PORT_NAMES + + +def describe_port(info) -> str: + vid_pid = "" + if info.vid is not None and info.pid is not None: + vid_pid = f" VID:PID={info.vid:04X}:{info.pid:04X}" + serial_no = f" SER={info.serial_number}" if info.serial_number else "" + desc = f" {info.description}" if info.description else "" + return f"{info.device}{vid_pid}{serial_no}{desc}".strip() + + +def available_ports() -> str: + ports = [p for p in list_ports.comports() if not is_forbidden_port(p.device)] + if not ports: + return "no allowed serial ports visible" + return "\n".join(f" {describe_port(p)}" for p in ports) + + +def esp32_port_candidates(): + return [ + p + for p in list_ports.comports() + if p.vid is not None + and p.pid is not None + and (int(p.vid), int(p.pid)) in ESPRESSIF_USB_IDS + and not is_forbidden_port(p.device) + ] + + +def resolve_port_name(name: str) -> str: + if is_forbidden_port(name): + raise serial.SerialException(f"{name} is blocked by the local serial safety policy") + if not is_auto_port(name): + return name + candidates = esp32_port_candidates() + if len(candidates) == 1: + return candidates[0].device + if not candidates: + raise serial.SerialException( + "no Espressif ESP32-S3 USB serial/JTAG port found for --port auto; " + f"visible ports:\n{available_ports()}" + ) + choices = "\n".join(f" {describe_port(p)}" for p in candidates) + raise serial.SerialException( + "multiple Espressif ESP32-S3 USB serial/JTAG ports found for --port auto; " + f"specify one with --port:\n{choices}" + ) + + def decode(data: bytes) -> str: return data.decode("utf-8", errors="replace").replace("\r\n", "\n").replace("\r", "\n") @@ -83,7 +142,7 @@ def open_port_retry(name: str, baud: int, timeout: float, open_timeout: float, d last_error: Exception | None = None while True: try: - return open_port(name, baud, timeout, dtr, rts) + return open_port(resolve_port_name(name), baud, timeout, dtr, rts) except serial.SerialException as exc: last_error = exc if time.monotonic() >= end: @@ -152,6 +211,8 @@ def main() -> int: with open_port_retry(args.port, args.baud, args.timeout, min(args.open_timeout, remaining_boot), args.dtr, args.rts) as port: + if is_auto_port(args.port): + print(f"[serial] auto-selected {port.port}") if args.open_only: print("[serial] open ok") return 0 diff --git a/scripts/tdeck_smoke.py b/scripts/tdeck_smoke.py index 3e329e7..e7353a6 100644 --- a/scripts/tdeck_smoke.py +++ b/scripts/tdeck_smoke.py @@ -15,6 +15,8 @@ import sys from pathlib import Path +import serial_harness + DEFAULT_COMMANDS = ["id", "sys", "net", "rf", "stats", "wifi", "companion test"] @@ -180,15 +182,20 @@ def main() -> int: project_dir = Path(args.project_dir).resolve() artifact_dir = Path(args.artifact_dir).resolve() if args.artifact_dir else None + try: + upload_port = serial_harness.resolve_port_name(args.port) + except Exception as exc: + raise SystemExit(f"invalid or unavailable serial port {args.port!r}: {exc}") from exc + if not args.skip_upload: if args.no_stub_upload: if not args.skip_build: run(["pio", "run", "-e", args.env], cwd=project_dir) - nostub_upload(project_dir, args.env, args.port, args.upload_baud, artifact_dir) + nostub_upload(project_dir, args.env, upload_port, args.upload_baud, artifact_dir) else: if args.skip_build or artifact_dir is not None: raise SystemExit("--skip-build/--artifact-dir require --no-stub-upload") - run(["pio", "run", "-e", args.env, "-t", "upload", "--upload-port", args.port], cwd=project_dir) + run(["pio", "run", "-e", args.env, "-t", "upload", "--upload-port", upload_port], cwd=project_dir) harness = project_dir / "scripts" / "serial_harness.py" cmd = [ From 4e1593445eabaee402910c621e1f381ce9659fa3 Mon Sep 17 00:00:00 2001 From: n30nex Date: Wed, 17 Jun 2026 20:42:33 -0400 Subject: [PATCH 07/64] Document COM8 TDM smoke evidence --- docs/tdeck-feature-inventory.md | 2 +- docs/tdeck-firmware-roadmap.md | 8 ++++---- docs/tdeck-hardware-dogfood-checklist.md | 14 ++++++++++++++ 3 files changed, 19 insertions(+), 5 deletions(-) diff --git a/docs/tdeck-feature-inventory.md b/docs/tdeck-feature-inventory.md index 5ac89e1..7e26aa6 100644 --- a/docs/tdeck-feature-inventory.md +++ b/docs/tdeck-feature-inventory.md @@ -59,7 +59,7 @@ Status labels: | Feature | Status | Evidence | Gap / Next Action | | --- | --- | --- | --- | | MeshCore compile-time gate | Partial | `#define LZ_MESHCORE_ENABLED 0` | Gate should stay off until receive/send/unified inbox tests pass. | -| TDM radio scheduler | Partial, needs validation | `lz_backend_set_networks`, profile switcher, settings airtime bar, serial `rf` diagnostics, opt-in `tdeck-meshcore` CI artifact, and `scripts/tdm_airtime_smoke.py` for Windows/Linux dwell + switch-count smoke | Needs hardware soak and latency/packet-loss measurements after the serial probe passes on a MeshCore-enabled image. | +| TDM radio scheduler | Partial, serial smoke passed | `lz_backend_set_networks`, profile switcher, settings airtime bar, serial `rf` diagnostics, opt-in `tdeck-meshcore` CI artifact, and `scripts/tdm_airtime_smoke.py` for Windows/Linux dwell + switch-count smoke. The 2026-06-18 COM8 run passed 60/40, 50/50, 40/60 dwell checks and switch-count motion on a MeshCore-enabled image. | Needs hardware soak and latency/packet-loss measurements with real simultaneous Meshtastic/MeshCore traffic. | | MeshCore RF profile | Partial, needs validation | 910.525 MHz / 62.5 kHz / SF7 / CR4/5 profile | Confirm target regions and RF compatibility with real MeshCore devices. | | MeshCore ADVERT RX | Partial, needs validation | `mc_parse`, `mc_advert_decode`, `lz_core_on_mc_node` | Only ADVERTs are decoded; encrypted payloads are ignored. | | MeshCore self-advert TX | Partial, needs validation | Ed25519 identity, self-advert builder, serial/UI advert commands | Needs interop proof with real MeshCore nodes. | diff --git a/docs/tdeck-firmware-roadmap.md b/docs/tdeck-firmware-roadmap.md index 6876778..8a91563 100644 --- a/docs/tdeck-firmware-roadmap.md +++ b/docs/tdeck-firmware-roadmap.md @@ -32,12 +32,12 @@ The firmware is complete when: These maintainer-provided beta labels are the canonical near-term sequence. The broader phases below preserve that order, then add post-V0.96 completion work for OTA, the full App Store, security, feedback, emergency, and release hardening. -**Current release: Beta 0.6.** MeshCore public chat (V0.6) and encrypted DMs (V0.7) are implemented and hardware-verified against a live mesh. **Two open items:** (1) split airtime (the Meshtastic↔MeshCore TDM scheduler) may not be working reliably and needs re-verification; (2) V0.5 BLE companion advertises/serves GATT and the official app connects, but the session drops immediately (connect-then-disconnect). Also delivered this cycle (outside the milestone list): a desktop SDL2 simulator with a 50+ assertion codec/scenario self-test harness, and Wi-Fi/BLE mutual exclusion (they share scarce internal DMA RAM on the ESP32-S3, so only one is resident at a time). +**Current release: Beta 0.6.** MeshCore public chat (V0.6) and encrypted DMs (V0.7) are implemented and hardware-verified against a live mesh. **Two open items:** (1) split airtime (the Meshtastic/MeshCore TDM scheduler) has a 2026-06-18 COM8 serial dwell/switch smoke pass on the opt-in MeshCore image, but still needs packet-loss, latency, and real dual-network traffic soak; (2) V0.5 BLE companion advertises/serves GATT and the official app connects, but the session drops immediately (connect-then-disconnect). Also delivered this cycle (outside the milestone list): a desktop SDL2 simulator with a 50+ assertion codec/scenario self-test harness, and Wi-Fi/BLE mutual exclusion (they share scarce internal DMA RAM on the ESP32-S3, so only one is resident at a time). | Version | Milestone | Status | | --- | --- | --- | | V0.5 | BLE companion for Meshtastic | 🚧 Firmware done — advertises + GATT (ToRadio/FromRadio/FromNum) works on hardware; **connect-then-disconnect** with the official app is open | -| V0.6 | MeshCore public chat and split airtime config | 🚧 Public chat send/receive hardware-verified; **split airtime may not be working — needs re-verification**; config UI still TODO | +| V0.6 | MeshCore public chat and split airtime config | In progress - public chat send/receive hardware-verified; split-airtime serial dwell/switch smoke passed on COM8; packet-loss, latency, and real dual-network traffic soak still open | | V0.7 | MeshCore DMs and private chats | ✅ Encrypted DMs (X25519 ECDH + AES) send/receive hardware-verified against a real MeshCore peer | | V0.8 | MeshCore USB companion and MeshCore BLE companion | ⬜ Not started | | V0.9 | Code review, optimization, and emoji polish | ⬜ Not started | @@ -125,7 +125,7 @@ Exit criteria: Goal: make MeshCore visible in the real product through public chat first, while giving users a simple way to understand and control split airtime. -**Status (Beta 0.6): mostly done — split airtime suspect.** MeshCore ADVERT interop, public/default channel receive, group/room text, the send path through `lz_svc_send_text`, unified-inbox wiring, the public-chat network toggle, and dual-network unread badges all work on a live mesh. **Open:** the split-airtime TDM scheduler (time-sharing the one SX1262 between Meshtastic and MeshCore) **may not be working reliably and needs re-verification** — earlier hardware checks looked OK but the behavior is now in question. Also remaining: a user-facing split-airtime *config UI* (currently a fixed 60/40 split toward Meshtastic). +**Status (Beta 0.6): mostly done - split airtime serial smoke passed, soak still open.** MeshCore ADVERT interop, public/default channel receive, group/room text, the send path through `lz_svc_send_text`, unified-inbox wiring, the public-chat network toggle, and dual-network unread badges all work on a live mesh. The 2026-06-18 COM8 run on the opt-in `tdeck-meshcore` artifact proved the 60/40, 50/50, and 40/60 dwell reports, switch-count motion, and restore to `Meshtastic 100%`. **Open:** packet-loss, latency, and real simultaneous Meshtastic/MeshCore traffic impact are not yet soaked. Deliverables: @@ -134,7 +134,7 @@ Deliverables: - missed-packet rate - Meshtastic delivery impact while MeshCore is enabled - MeshCore delivery impact while Meshtastic is enabled - - repeatable serial smoke for dwell presets and switch-count motion. Implemented as `scripts/tdm_airtime_smoke.py` plus an opt-in `tdeck-meshcore` CI artifact; it runs on Windows `COM8` or Linux/macOS serial paths, checks the 60/40, 50/50, and 40/60 dwell reports, verifies `switches:` advances between `rf` samples, and fails clearly if MeshCore is still compile-gated. + - repeatable serial smoke for dwell presets and switch-count motion. Implemented as `scripts/tdm_airtime_smoke.py` plus an opt-in `tdeck-meshcore` CI artifact; it runs on Windows `COM8` or Linux/macOS serial paths, checks the 60/40, 50/50, and 40/60 dwell reports, verifies `switches:` advances between `rf` samples, and fails clearly if MeshCore is still compile-gated. Passed on COM8 on 2026-06-18 against the opt-in MeshCore artifact; soak metrics above remain open. - Confirm target MeshCore RF profiles by region and define how they coexist with the LongFast-only product goal. - Build the split airtime config UI around simple choices, not raw radio parameters. Implemented: Settings now exposes Meshtastic first, Balanced, and MeshCore first presets, persists the choice, and reports the active dwell split through serial diagnostics. - Finish MeshCore packet handling: diff --git a/docs/tdeck-hardware-dogfood-checklist.md b/docs/tdeck-hardware-dogfood-checklist.md index 2ddc4c8..df7c1a1 100644 --- a/docs/tdeck-hardware-dogfood-checklist.md +++ b/docs/tdeck-hardware-dogfood-checklist.md @@ -51,6 +51,20 @@ dogfood belong to the later roadmap phases. ## Hardware Evidence Log +### 2026-06-18 COM8 MeshCore TDM Smoke + +- Firmware flashed: opt-in `tdeck-meshcore` GitHub Actions artifact from `n30nex/LimitlezzOS` run `27728176574`, commit `1552fa8`. +- Artifact manifest recorded `meshcore_enabled=1`, `budget_status=pass`, `firmware_bytes=1536832`, and `static_ram_bytes=274676`. +- Port boundary: only `COM8` was opened/flashed/probed during this validation. +- Direct ROM flashing with `python scripts/tdeck_smoke.py --port COM8 --env tdeck-meshcore --no-stub-upload --skip-build --artifact-dir .pio/ci-artifacts/tdeck-meshcore --upload-baud 460800` succeeded; bootloader, partitions, `boot_app0.bin`, and firmware hashes all verified. +- Split-airtime smoke passed with `python scripts/tdm_airtime_smoke.py --port COM8 --open-timeout 60 --boot-timeout 60 --timeout 30`. +- TDM evidence: `airtime mt` reported 60/40 with 300/200 ms dwell and `switches: 2`; `airtime balanced` reported 50/50 with 250/250 ms dwell and `switches: 3`; `airtime mc` reported 40/60 with 200/300 ms dwell and `switches: 4`. +- Switch motion evidence: after returning to balanced mode, `rf` advanced from `switches: 5` to `switches: 10` during the settle window and the active side flipped from MeshCore to Meshtastic. +- Restore evidence: `net mc off` returned `rf` to `mode: Meshtastic 100%` with no additional switch-count growth. +- Follow-up serial smoke passed with `python scripts/tdeck_smoke.py --skip-upload --port COM8 --env tdeck-meshcore --open-timeout 60 --boot-timeout 60 --timeout 30`. +- Serial smoke evidence: identity `!a20d1428` / `limitlessdeck`, battery 100% on USB, `net` reported Meshtastic on and MeshCore off after restore, `rf` reported `mode: Meshtastic 100%`, `stats` reported radio TX/RX counters, Wi-Fi reported `cred=nvs`, and `companion test` reported `63 frames ... -> PASS`. +- Remaining gap: this proves the opt-in MeshCore TDM image reports correct dwell presets and live switch motion on COM8; it does not yet prove packet loss, latency, or real simultaneous Meshtastic/MeshCore traffic impact. + ### 2026-06-14 COM8 Smoke Attempt - Branch/commit flashed: `codex/tdeck-firmware-audit-roadmap` at `d5f69e0`. From 43b415de085074d83d614b800124ac867017c54c Mon Sep 17 00:00:00 2001 From: n30nex Date: Wed, 17 Jun 2026 20:57:53 -0400 Subject: [PATCH 08/64] Add local app sample pack --- .github/workflows/firmware.yml | 5 + README.md | 6 + docs/tdeck-feature-inventory.md | 3 +- docs/tdeck-firmware-roadmap.md | 7 +- docs/tdeck-local-app-manifest.md | 33 +++ examples/local-apps/README.md | 26 ++ examples/local-apps/aprs-bridge/main.lua | 5 + examples/local-apps/aprs-bridge/manifest.json | 12 + examples/local-apps/calculator/main.lua | 5 + examples/local-apps/calculator/manifest.json | 12 + examples/local-apps/field-notes/main.lua | 6 + examples/local-apps/field-notes/manifest.json | 12 + examples/local-apps/lora-chess/main.lua | 6 + examples/local-apps/lora-chess/manifest.json | 12 + examples/local-apps/mesh-bbs/main.lua | 5 + examples/local-apps/mesh-bbs/manifest.json | 12 + examples/local-apps/offline-maps/main.lua | 5 + .../local-apps/offline-maps/manifest.json | 12 + examples/local-apps/signal-scope/main.lua | 5 + .../local-apps/signal-scope/manifest.json | 12 + examples/local-apps/weather-mesh/main.lua | 5 + .../local-apps/weather-mesh/manifest.json | 12 + scripts/validate_local_app_samples.py | 268 ++++++++++++++++++ sim/main_sim.c | 6 + src/services/mesh.h | 1 + src/services/mesh_core.c | 6 + src/services/store.c | 5 + src/ui/screens/scr_apps.c | 10 +- src/ui/ui.c | 1 + src/ui/ui.h | 1 + 30 files changed, 513 insertions(+), 3 deletions(-) create mode 100644 examples/local-apps/README.md create mode 100644 examples/local-apps/aprs-bridge/main.lua create mode 100644 examples/local-apps/aprs-bridge/manifest.json create mode 100644 examples/local-apps/calculator/main.lua create mode 100644 examples/local-apps/calculator/manifest.json create mode 100644 examples/local-apps/field-notes/main.lua create mode 100644 examples/local-apps/field-notes/manifest.json create mode 100644 examples/local-apps/lora-chess/main.lua create mode 100644 examples/local-apps/lora-chess/manifest.json create mode 100644 examples/local-apps/mesh-bbs/main.lua create mode 100644 examples/local-apps/mesh-bbs/manifest.json create mode 100644 examples/local-apps/offline-maps/main.lua create mode 100644 examples/local-apps/offline-maps/manifest.json create mode 100644 examples/local-apps/signal-scope/main.lua create mode 100644 examples/local-apps/signal-scope/manifest.json create mode 100644 examples/local-apps/weather-mesh/main.lua create mode 100644 examples/local-apps/weather-mesh/manifest.json create mode 100644 scripts/validate_local_app_samples.py diff --git a/.github/workflows/firmware.yml b/.github/workflows/firmware.yml index fddb228..5ef9ddf 100644 --- a/.github/workflows/firmware.yml +++ b/.github/workflows/firmware.yml @@ -43,6 +43,11 @@ jobs: python -m pip install --upgrade pip python -m pip install platformio + - name: Validate local app samples + run: | + python -m py_compile scripts/validate_local_app_samples.py + python scripts/validate_local_app_samples.py + - name: Build native simulator run: pio run -e native diff --git a/README.md b/README.md index 722c799..77e1227 100644 --- a/README.md +++ b/README.md @@ -300,8 +300,14 @@ for local apps and read-only inspection when present. clears scoped app data on request, opens a manifest detail shell, and launches local apps into the SDK 0.1 foreground shell with bounded app-provided actions and scoped storage counters plus read-only `{time}` / `{battery}` tokens; + Close/Esc terminates the foreground session instead of leaving it resident; unsupported action effects launch-block instead of being ignored; the static catalog remains a prototype (GET -> "..." -> OPEN). +- **Local app sample pack** - `examples/local-apps/` contains copyable SDK 0.1 + packages for Calculator, Field Notes, Offline Maps, Weather Mesh, Mesh BBS, + Signal Scope, LoRa Chess, and APRS Bridge; CI validates that each package + stays inside the firmware's bounded manifest, permission, token, action, and + scoped-storage rules. - **Contacts / detail** — unified directory with network dots; detail page with Message (jumps into the bound conversation) and spec table. - **Settings** — airtime scheduler bar that rebalances live when the diff --git a/docs/tdeck-feature-inventory.md b/docs/tdeck-feature-inventory.md index 416182f..7eb8094 100644 --- a/docs/tdeck-feature-inventory.md +++ b/docs/tdeck-feature-inventory.md @@ -82,7 +82,7 @@ Status labels: | Settings | Functional/Partial | network toggles, Wi-Fi, in-place brightness slider updates, time, system, touch calibration, Developer Mode, `settings.cfg` persistence | Add migration/versioning if the settings schema grows; hardware latency pass still needed. | | Wi-Fi setup | Functional, needs validation | async scan/connect, saved SSID/password, auto-connect | Credentials are plaintext on SD; only one saved network. | | System/battery page | Functional/Partial | live stats and battery arc | Hardware values need calibration/validation. | -| App Store | Prototype/Partial | `LZ_STORE` static catalog remains; local manifests from SD/appfs are scanned, listed as installed local apps, open a manifest detail shell, show storage quota usage, clear scoped app data on request, launch into the SDK 0.1 foreground shell, expose bounded foreground actions with scoped storage counters plus read-only `{time}`/`{battery}` tokens, reject unsupported action effects, and show rejected package diagnostics in Developer Mode | Network catalog, download, install/update, script execution/richer API injection, and runtime crash capture are still missing. | +| App Store | Prototype/Partial | `LZ_STORE` static catalog remains; local manifests from SD/appfs are scanned, listed as installed local apps, open a manifest detail shell, show storage quota usage, clear scoped app data on request, launch into the SDK 0.1 foreground shell, explicitly terminate foreground sessions on Close/Esc, expose bounded foreground actions with scoped storage counters plus read-only `{time}`/`{battery}` tokens, reject unsupported action effects, and show rejected package diagnostics in Developer Mode | Network catalog, download, install/update, script execution/richer API injection, and runtime crash capture are still missing. | | Terminal | Functional/Partial | interactive UI terminal behind Developer Mode; serial CLI always available over USB | Expand diagnostics once Developer Mode grows into a full power-user surface. | | Files | Functional/Partial | read-only bounded filesystem browser rooted at mounted SD/local store or mounted FAT appfs; when both are present it starts at a Storage root picker | Add gated file actions later. | @@ -94,6 +94,7 @@ Status labels: | App manifest | Partial | `docs/tdeck-local-app-manifest.md`; bounded manifest parser requires `id`, `name`, and relative `entry`, with optional version/author/summary/icon/hue plus SDK `api_version` and permission metadata | Extend once the runtime lifecycle and package actions are chosen. | | App permissions | Partial | Local manifests can declare allowlisted SDK namespaces (`display`, `input`, `storage`, mesh, time, battery, notifications, Wi-Fi); unknown permission names reject the package before Home/App Store; `storage` prepares a scoped package `data/` directory with a 64 KB launch-time quota guard, SDK action counters require both `input` and `storage`, and `{time}`/`{battery}` tokens require matching `system_time`/`battery` permission before launch | Implement least-privilege API injection when the runtime is selected. | | Local app scanner | Partial | `lz_store_scan_apps` scans `/sd/limitlezz/apps`, `/sd/apps`, `/appfs/apps`, simulator `/apps`, and simulator `/appfs/apps`; accepted apps appear in the paged Home launcher and App Store; rejected packages are exposed through Developer Mode diagnostics; simulator selftest covers appfs-only discovery, valid metadata, storage sandbox prep, quota usage, clear-data behavior, foreground launch metadata/actions, storage counter persistence, read-only time/battery token gating, unsupported action-effect blocking, oversized entry blocking, and rejected unsafe packages | Add script execution, richer app lifecycle hooks, and broader user-facing data actions once memory profiling picks a runtime. | +| Local app sample pack | Functional, CI-covered | `examples/local-apps/` provides copyable SDK 0.1 packages for Calculator, Field Notes, Offline Maps, Weather Mesh, Mesh BBS, Signal Scope, LoRa Chess, and APRS Bridge; `scripts/validate_local_app_samples.py` checks manifest limits, supported permissions, token permissions, foreground actions, and scoped counter effects in CI | Keep samples aligned with the eventual richer runtime/API injection and add richer example behavior once the interpreter is selected. | | Network app catalog | Planned | Wi-Fi service notes; design spec | Fetch `index.json`, verify TLS/metadata, cache results. | | App download/install/update | Planned | App Store prototype only | SHA256 verify, extract, version updates, rollback failed installs. | | Optional map app | Planned | Store data includes maps; maintainer notes prefer maps as optional | Keep maps out of the base firmware. | diff --git a/docs/tdeck-firmware-roadmap.md b/docs/tdeck-firmware-roadmap.md index f987e35..cdc6162 100644 --- a/docs/tdeck-firmware-roadmap.md +++ b/docs/tdeck-firmware-roadmap.md @@ -247,6 +247,8 @@ Deliverables: `storage`; unknown action effects and malformed counter effects are launch-blocked instead of ignored. App sessions terminate on exit; App Store opens the manifest detail shell with a trackball-accessible `OPEN` action. + Implemented termination now explicitly clears the foreground session on + Close/Esc while preserving the selected app manifest for the detail view. Script execution and richer injected runtime APIs remain below, with initial read-only `{time}` and `{battery}` token injection now routed through declared `system_time` and `battery` permissions. @@ -283,8 +285,11 @@ Deliverables: data, and display-only apps that declare actions are blocked for missing input permission, while runtime crash capture remains below. - Convert prototype catalog examples into installable sample apps where practical: + Implemented as copyable SDK 0.1 packages in `examples/local-apps/`, with a + CI-validated sample-pack checker that mirrors the firmware manifest, token, + foreground-action, and scoped-storage rules. - Calculator - - Notes + - Field Notes - Offline Maps shell - Weather Mesh - Mesh BBS diff --git a/docs/tdeck-local-app-manifest.md b/docs/tdeck-local-app-manifest.md index fb8198e..f7baa53 100644 --- a/docs/tdeck-local-app-manifest.md +++ b/docs/tdeck-local-app-manifest.md @@ -30,6 +30,8 @@ than fit on the first screen. The App Store also lists accepted local apps as installed local packages and opens the same manifest detail shell. Home launches the app into the SDK 0.1 foreground shell; the detail shell also has an `OPEN` action. +Closing the app from the foreground shell, or navigating back with Esc, clears +the active session while preserving the manifest selection for the detail view. When Developer Mode is enabled, the App Store also shows rejected local package folders with a short reason such as `missing manifest`, `unsafe id`, `bad @@ -163,3 +165,34 @@ are launch-blocked before any future runtime code can run. The App Store detail screen also provides `Clear local data` for storage-enabled apps; it removes only files and folders inside that app's scoped `data/` directory and then recreates the directory for later use. + +## Sample App Pack + +The repository includes copyable SDK 0.1 packages in `examples/local-apps/`: + +- Calculator +- Field Notes +- Offline Maps +- Weather Mesh +- Mesh BBS +- Signal Scope +- LoRa Chess +- APRS Bridge + +Validate the pack before copying it to a simulator or card-style root: + +```sh +python scripts/validate_local_app_samples.py +``` + +For local simulator/data-root testing, install the samples under an app root: + +```sh +python scripts/validate_local_app_samples.py --install-root .pio/local-app-samples --clean +``` + +Then copy `.pio/local-app-samples/apps/` folders to `/sd/limitlezz/apps/`, +`/sd/apps/`, or `/appfs/apps/` on hardware. The validator mirrors the firmware's +SDK 0.1 limits for manifest size, string fields, safe entry paths, supported +permissions, `{time}` / `{battery}` token permissions, foreground action count, +and storage-scoped counter effects. diff --git a/examples/local-apps/README.md b/examples/local-apps/README.md new file mode 100644 index 0000000..1fc873b --- /dev/null +++ b/examples/local-apps/README.md @@ -0,0 +1,26 @@ +# Local App Samples + +These packages are SDK 0.1 foreground-shell samples that can be copied into a +T-Deck SD/appfs app root. They intentionally use only manifest metadata, +bounded display text, foreground actions, read-only tokens, and scoped counter +storage. They do not execute arbitrary script code. + +Install into a simulator or card-style root: + +```sh +python scripts/validate_local_app_samples.py --install-root .pio/local-app-samples --clean +``` + +The resulting layout is: + +```text +.pio/local-app-samples/apps//manifest.json +.pio/local-app-samples/apps//main.lua +``` + +For hardware, copy the package folders under one of the firmware scan roots: + +- `/sd/limitlezz/apps/` +- `/sd/apps/` +- `/appfs/apps/` + diff --git a/examples/local-apps/aprs-bridge/main.lua b/examples/local-apps/aprs-bridge/main.lua new file mode 100644 index 0000000..9e42fff --- /dev/null +++ b/examples/local-apps/aprs-bridge/main.lua @@ -0,0 +1,5 @@ +-- title: APRS Bridge +-- status: Bridge disabled in SDK 0.1 +-- body: This shell keeps APRS and Wi-Fi bridge work out of the base OS image. +-- action: Queue | Packet #{count} queued | Queued {count} placeholder APRS packets locally. | counter:packets +return true diff --git a/examples/local-apps/aprs-bridge/manifest.json b/examples/local-apps/aprs-bridge/manifest.json new file mode 100644 index 0000000..dd581d2 --- /dev/null +++ b/examples/local-apps/aprs-bridge/manifest.json @@ -0,0 +1,12 @@ +{ + "id": "aprs.bridge", + "name": "APRS Bridge", + "version": "0.1.0", + "author": "Limitless", + "summary": "APRS bridge shell", + "entry": "main.lua", + "api_version": "0.1", + "permissions": ["display", "input", "storage", "mesh_read", "mesh_send", "network_wifi"], + "icon": "lan", + "hue": 205 +} diff --git a/examples/local-apps/calculator/main.lua b/examples/local-apps/calculator/main.lua new file mode 100644 index 0000000..a4bc422 --- /dev/null +++ b/examples/local-apps/calculator/main.lua @@ -0,0 +1,5 @@ +-- title: Calculator +-- status: SDK 0.1 foreground sandbox +-- body: A tiny keypad shell for field math. Runtime arithmetic lands after the VM choice. +-- action: Clear | Calculator cleared | The display state was reset inside the foreground shell. +return true diff --git a/examples/local-apps/calculator/manifest.json b/examples/local-apps/calculator/manifest.json new file mode 100644 index 0000000..982957e --- /dev/null +++ b/examples/local-apps/calculator/manifest.json @@ -0,0 +1,12 @@ +{ + "id": "calc.local", + "name": "Calculator", + "version": "0.1.0", + "author": "Limitless", + "summary": "Offline field calculator", + "entry": "main.lua", + "api_version": "0.1", + "permissions": ["display", "input"], + "icon": "calculate", + "hue": 18 +} diff --git a/examples/local-apps/field-notes/main.lua b/examples/local-apps/field-notes/main.lua new file mode 100644 index 0000000..0f44f3f --- /dev/null +++ b/examples/local-apps/field-notes/main.lua @@ -0,0 +1,6 @@ +-- title: Field Notes +-- status: Local notes sandbox +-- body: Notes will stay in scoped app storage once text editing APIs land. +-- action: New note | Note #{count} staged | Created {count} placeholder notes in scoped app data. | counter:notes +-- action: Pin | Pin #{count} staged | Pinned {count} note markers for this app only. | counter:pins +return true diff --git a/examples/local-apps/field-notes/manifest.json b/examples/local-apps/field-notes/manifest.json new file mode 100644 index 0000000..ebd7a4f --- /dev/null +++ b/examples/local-apps/field-notes/manifest.json @@ -0,0 +1,12 @@ +{ + "id": "notes.local", + "name": "Field Notes", + "version": "0.1.0", + "author": "Limitless", + "summary": "Scratchpad for field work", + "entry": "main.lua", + "api_version": "0.1", + "permissions": ["display", "input", "storage"], + "icon": "note", + "hue": 175 +} diff --git a/examples/local-apps/lora-chess/main.lua b/examples/local-apps/lora-chess/main.lua new file mode 100644 index 0000000..b5fabf1 --- /dev/null +++ b/examples/local-apps/lora-chess/main.lua @@ -0,0 +1,6 @@ +-- title: LoRa Chess +-- status: Waiting for a local board +-- body: Turn state stays scoped to this app before mesh move routing is added. +-- action: New game | Game #{count} staged | Created {count} local chess boards. | counter:games +-- action: Move | Move #{count} staged | Recorded {count} local moves in scoped storage. | counter:moves +return true diff --git a/examples/local-apps/lora-chess/manifest.json b/examples/local-apps/lora-chess/manifest.json new file mode 100644 index 0000000..558cf6a --- /dev/null +++ b/examples/local-apps/lora-chess/manifest.json @@ -0,0 +1,12 @@ +{ + "id": "chess.mesh", + "name": "LoRa Chess", + "version": "0.1.0", + "author": "Limitless", + "summary": "Turn-based mesh game shell", + "entry": "main.lua", + "api_version": "0.1", + "permissions": ["display", "input", "storage", "mesh_send", "notifications"], + "icon": "game", + "hue": 35 +} diff --git a/examples/local-apps/mesh-bbs/main.lua b/examples/local-apps/mesh-bbs/main.lua new file mode 100644 index 0000000..e41fa2f --- /dev/null +++ b/examples/local-apps/mesh-bbs/main.lua @@ -0,0 +1,5 @@ +-- title: Mesh BBS +-- status: Local bulletin board shell +-- body: Public posts stay inside the app sandbox until mesh APIs are injected. +-- action: Sync | BBS sync #{count} | Checked the bulletin board shell {count} times. | counter:syncs +return true diff --git a/examples/local-apps/mesh-bbs/manifest.json b/examples/local-apps/mesh-bbs/manifest.json new file mode 100644 index 0000000..470fc18 --- /dev/null +++ b/examples/local-apps/mesh-bbs/manifest.json @@ -0,0 +1,12 @@ +{ + "id": "bbs.mesh", + "name": "Mesh BBS", + "version": "0.1.0", + "author": "Limitless", + "summary": "Bulletin board shell", + "entry": "main.lua", + "api_version": "0.1", + "permissions": ["display", "input", "storage", "mesh_read", "mesh_send"], + "icon": "description", + "hue": 280 +} diff --git a/examples/local-apps/offline-maps/main.lua b/examples/local-apps/offline-maps/main.lua new file mode 100644 index 0000000..2415a7f --- /dev/null +++ b/examples/local-apps/offline-maps/main.lua @@ -0,0 +1,5 @@ +-- title: Offline Maps +-- status: Map tiles unavailable in SDK 0.1 +-- body: This shell proves maps can live outside the base firmware image. +-- action: Center | Map centered #{count} | Centered the local map shell {count} times. | counter:center +return true diff --git a/examples/local-apps/offline-maps/manifest.json b/examples/local-apps/offline-maps/manifest.json new file mode 100644 index 0000000..0d5b160 --- /dev/null +++ b/examples/local-apps/offline-maps/manifest.json @@ -0,0 +1,12 @@ +{ + "id": "maps.local", + "name": "Offline Maps", + "version": "0.1.0", + "author": "Limitless", + "summary": "Local map shell", + "entry": "main.lua", + "api_version": "0.1", + "permissions": ["display", "input", "storage"], + "icon": "map", + "hue": 110 +} diff --git a/examples/local-apps/signal-scope/main.lua b/examples/local-apps/signal-scope/main.lua new file mode 100644 index 0000000..979a3ec --- /dev/null +++ b/examples/local-apps/signal-scope/main.lua @@ -0,0 +1,5 @@ +-- title: Signal Scope +-- status: Snapshot at {time} +-- body: Battery {battery}. Packet counters and RSSI traces will arrive through mesh_read. +-- action: Snapshot | Scope refreshed at {time} | Refreshed the bounded foreground display. +return true diff --git a/examples/local-apps/signal-scope/manifest.json b/examples/local-apps/signal-scope/manifest.json new file mode 100644 index 0000000..9d6767e --- /dev/null +++ b/examples/local-apps/signal-scope/manifest.json @@ -0,0 +1,12 @@ +{ + "id": "scope.local", + "name": "Signal Scope", + "version": "0.1.0", + "author": "Limitless", + "summary": "Packet and RSSI viewer", + "entry": "main.lua", + "api_version": "0.1", + "permissions": ["display", "input", "mesh_read", "system_time", "battery"], + "icon": "terminal", + "hue": 210 +} diff --git a/examples/local-apps/weather-mesh/main.lua b/examples/local-apps/weather-mesh/main.lua new file mode 100644 index 0000000..0b57593 --- /dev/null +++ b/examples/local-apps/weather-mesh/main.lua @@ -0,0 +1,5 @@ +-- title: Weather Mesh +-- status: Forecast snapshot at {time} +-- body: Battery {battery}. Mesh weather reports will appear here when read APIs land. +-- action: Refresh | Weather refreshed #{count} | Weather Mesh refreshed {count} times. | counter:refreshes +return true diff --git a/examples/local-apps/weather-mesh/manifest.json b/examples/local-apps/weather-mesh/manifest.json new file mode 100644 index 0000000..200a184 --- /dev/null +++ b/examples/local-apps/weather-mesh/manifest.json @@ -0,0 +1,12 @@ +{ + "id": "weather.mesh", + "name": "Weather Mesh", + "version": "0.1.0", + "author": "Limitless", + "summary": "Local weather dashboard", + "entry": "main.lua", + "api_version": "0.1", + "permissions": ["display", "input", "storage", "mesh_read", "system_time", "battery"], + "icon": "weather", + "hue": 48 +} diff --git a/scripts/validate_local_app_samples.py b/scripts/validate_local_app_samples.py new file mode 100644 index 0000000..90b5da6 --- /dev/null +++ b/scripts/validate_local_app_samples.py @@ -0,0 +1,268 @@ +#!/usr/bin/env python3 +"""Validate and optionally install the SDK 0.1 local app sample pack.""" + +from __future__ import annotations + +import argparse +import json +import re +import shutil +import sys +from pathlib import Path + + +REPO_ROOT = Path(__file__).resolve().parents[1] +DEFAULT_SAMPLES = REPO_ROOT / "examples" / "local-apps" + +MANIFEST_MAX_BYTES = 1535 +ENTRY_MAX_BYTES = 1024 +ACTION_MAX = 2 + +FIELD_LIMITS = { + "id": 23, + "name": 31, + "version": 15, + "author": 27, + "summary": 71, + "entry": 47, + "api_version": 11, + "icon": 19, +} + +SUPPORTED_PERMISSIONS = { + "display", + "input", + "storage", + "mesh_read", + "mesh_send", + "system_time", + "battery", + "notifications", + "network_wifi", +} + +SUPPORTED_SDKS = {"0.1", "0.1.0"} +SAFE_ID = re.compile(r"^[A-Za-z0-9_.-]+$") +SAFE_COUNTER = re.compile(r"^[A-Za-z0-9_-]{1,19}$") + + +class ValidationError(RuntimeError): + pass + + +def byte_len(value: str) -> int: + return len(value.encode("utf-8")) + + +def check_string(manifest: dict[str, object], key: str, required: bool = False) -> str: + value = manifest.get(key) + if value is None: + if required: + raise ValidationError(f"missing {key}") + return "" + if not isinstance(value, str) or not value: + raise ValidationError(f"{key} must be a non-empty string") + limit = FIELD_LIMITS[key] + if byte_len(value) > limit: + raise ValidationError(f"{key} exceeds firmware limit {limit} bytes") + return value + + +def check_entry_name(entry: str) -> None: + if entry.startswith(("/", "\\")): + raise ValidationError("entry must be relative") + if ".." in entry: + raise ValidationError("entry must not contain '..'") + if "\\" in entry or ":" in entry: + raise ValidationError("entry must use safe POSIX-style separators") + if any(ord(ch) < 32 for ch in entry): + raise ValidationError("entry contains control characters") + + +def parse_metadata_line(line: str) -> tuple[str, str] | None: + stripped = line.strip() + if stripped.startswith("--"): + stripped = stripped[2:].strip() + elif stripped.startswith("#"): + stripped = stripped[1:].strip() + for sep in (":", "="): + if sep in stripped: + key, value = stripped.split(sep, 1) + key = key.strip() + if key in {"title", "status", "body", "text", "action"}: + return key, value.strip() + return None + + +def action_parts(spec: str) -> list[str]: + return [part.strip() for part in spec.split("|")] + + +def validate_action(spec: str, permissions: set[str], package: Path) -> None: + parts = action_parts(spec) + label = parts[0] if parts else "" + if not label: + raise ValidationError("action label is required") + if byte_len(label) > 23: + raise ValidationError("action label exceeds firmware limit 23 bytes") + if len(parts) > 1 and byte_len(parts[1]) > 47: + raise ValidationError("action status exceeds firmware limit 47 bytes") + if len(parts) > 2 and byte_len(parts[2]) > 191: + raise ValidationError("action body exceeds firmware limit 191 bytes") + effect = parts[3] if len(parts) > 3 else "" + if effect: + if byte_len(effect) > 31: + raise ValidationError("action effect exceeds firmware limit 31 bytes") + if effect.startswith("counter:") or effect.startswith("count:"): + key = effect.split(":", 1)[1] + if not SAFE_COUNTER.fullmatch(key): + raise ValidationError("counter action uses an unsafe key") + if "storage" not in permissions: + raise ValidationError("counter action requires storage permission") + else: + raise ValidationError(f"unsupported action effect {effect!r}") + if "input" not in permissions: + raise ValidationError(f"{package.name} declares actions without input permission") + + +def validate_entry(entry_path: Path, permissions: set[str], package: Path) -> None: + size = entry_path.stat().st_size + if size == 0: + raise ValidationError("entry is empty") + if size > ENTRY_MAX_BYTES: + raise ValidationError(f"entry exceeds firmware limit {ENTRY_MAX_BYTES} bytes") + text = entry_path.read_text(encoding="utf-8") + if "{time}" in text and "system_time" not in permissions: + raise ValidationError("{time} token requires system_time permission") + if "{battery}" in text and "battery" not in permissions: + raise ValidationError("{battery} token requires battery permission") + actions = 0 + for line in text.splitlines(): + parsed = parse_metadata_line(line) + if not parsed: + continue + key, value = parsed + if key == "action": + actions += 1 + if actions > ACTION_MAX: + raise ValidationError(f"more than {ACTION_MAX} actions") + validate_action(value, permissions, package) + elif key == "title" and byte_len(value) > 31: + raise ValidationError("entry title exceeds firmware limit 31 bytes") + elif key == "status" and byte_len(value) > 63: + raise ValidationError("entry status exceeds firmware limit 63 bytes") + elif key in {"body", "text"} and byte_len(value) > 95: + raise ValidationError("entry body line exceeds safe sample line limit 95 bytes") + + +def validate_package(package: Path) -> dict[str, object]: + manifest_path = package / "manifest.json" + if not manifest_path.is_file(): + raise ValidationError("missing manifest.json") + if manifest_path.stat().st_size == 0: + raise ValidationError("manifest is empty") + if manifest_path.stat().st_size > MANIFEST_MAX_BYTES: + raise ValidationError(f"manifest exceeds firmware limit {MANIFEST_MAX_BYTES} bytes") + try: + manifest = json.loads(manifest_path.read_text(encoding="utf-8")) + except json.JSONDecodeError as exc: + raise ValidationError(f"manifest JSON error: {exc}") from exc + if not isinstance(manifest, dict): + raise ValidationError("manifest must be an object") + + app_id = check_string(manifest, "id", required=True) + if not SAFE_ID.fullmatch(app_id): + raise ValidationError("id contains unsafe characters") + check_string(manifest, "name", required=True) + check_string(manifest, "version") + check_string(manifest, "author") + summary = manifest.get("summary", manifest.get("description", "")) + if summary: + if not isinstance(summary, str): + raise ValidationError("summary/description must be a string") + if byte_len(summary) > FIELD_LIMITS["summary"]: + raise ValidationError("summary exceeds firmware limit 71 bytes") + entry = check_string(manifest, "entry", required=True) + check_entry_name(entry) + api_version = check_string(manifest, "api_version") or "0.1" + if api_version not in SUPPORTED_SDKS: + raise ValidationError("unsupported SDK") + check_string(manifest, "icon") + hue = manifest.get("hue", -1) + if not isinstance(hue, int) or hue < -1 or hue > 359: + raise ValidationError("hue must be -1..359") + permissions_value = manifest.get("permissions") + if not isinstance(permissions_value, list) or not permissions_value: + raise ValidationError("permissions must be a non-empty array") + permissions = set() + for item in permissions_value: + if not isinstance(item, str) or item not in SUPPORTED_PERMISSIONS: + raise ValidationError(f"unsupported permission {item!r}") + permissions.add(item) + if "display" not in permissions: + raise ValidationError("display permission is required for SDK 0.1 samples") + + entry_path = package / entry + if not entry_path.is_file(): + raise ValidationError("missing entry file") + validate_entry(entry_path, permissions, package) + return manifest + + +def iter_packages(samples_dir: Path) -> list[Path]: + packages = [p for p in samples_dir.iterdir() if p.is_dir()] + return sorted(packages, key=lambda p: p.name) + + +def install_packages(packages: list[Path], install_root: Path, clean: bool) -> None: + apps_dir = install_root / "apps" + apps_dir.mkdir(parents=True, exist_ok=True) + for package in packages: + dest = apps_dir / package.name + if dest.exists(): + if not clean: + raise ValidationError(f"{dest} already exists; pass --clean to replace samples") + shutil.rmtree(dest) + shutil.copytree(package, dest, ignore=shutil.ignore_patterns("data")) + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--samples-dir", type=Path, default=DEFAULT_SAMPLES) + parser.add_argument("--install-root", type=Path, help="copy samples under /apps after validation") + parser.add_argument("--clean", action="store_true", help="replace existing sample folders at --install-root") + args = parser.parse_args() + + samples_dir = args.samples_dir.resolve() + if not samples_dir.is_dir(): + print(f"[apps] missing sample directory: {samples_dir}", file=sys.stderr) + return 2 + + packages = iter_packages(samples_dir) + if not packages: + print("[apps] no sample packages found", file=sys.stderr) + return 2 + + seen_ids: set[str] = set() + try: + for package in packages: + manifest = validate_package(package) + app_id = str(manifest["id"]) + if app_id in seen_ids: + raise ValidationError(f"duplicate app id {app_id}") + seen_ids.add(app_id) + print(f"[apps] ok {package.name}: {app_id}") + if args.install_root: + install_root = args.install_root.resolve() + install_packages(packages, install_root, args.clean) + print(f"[apps] installed {len(packages)} samples under {install_root / 'apps'}") + except ValidationError as exc: + print(f"[apps] FAIL: {exc}", file=sys.stderr) + return 1 + + print(f"[apps] sample pack PASS ({len(packages)} packages)") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/sim/main_sim.c b/sim/main_sim.c index 9c54b9b..7e74463 100644 --- a/sim/main_sim.c +++ b/sim/main_sim.c @@ -733,6 +733,7 @@ static int codec_selftest(void) extern bool lz_store_clear_app_data(const lz_local_app_t *app, char *err, int err_cap); extern bool lz_store_start_local_app(const lz_local_app_t *app, lz_local_app_session_t *out); extern bool lz_store_local_app_action(lz_local_app_session_t *session, int idx); + extern void lz_store_stop_local_app(lz_local_app_session_t *session); sim_reset_dir("lzdata_appscan"); sim_mkdirs("lzdata_appscan/apps/weather"); sim_mkdirs("lzdata_appscan/apps/bad"); @@ -814,6 +815,11 @@ static int codec_selftest(void) "local app foreground storage counter persists"); CHECK(action2_ok && run.data_used_bytes > 1536, "local app foreground storage counter stays in app data quota"); + lz_store_stop_local_app(&run); + CHECK(!run.entry_loaded && run.action_count == 0 && run.data_path[0] == 0, + "local app foreground session terminates cleanly"); + CHECK(!lz_store_local_app_action(&run, 0), + "local app foreground actions stop after termination"); bool clear_ok = an == 1 && lz_store_clear_app_data(&apps[0], data_err, sizeof data_err); CHECK(clear_ok, "local app clear data succeeds inside scoped storage"); used = 123; diff --git a/src/services/mesh.h b/src/services/mesh.h index aacfc8d..5f31612 100644 --- a/src/services/mesh.h +++ b/src/services/mesh.h @@ -234,6 +234,7 @@ bool lz_svc_app_data_usage(const lz_local_app_t *app, uint32_t *used, uint32_t * bool lz_svc_clear_app_data(const lz_local_app_t *app, char *err, int err_cap); bool lz_svc_start_local_app(const lz_local_app_t *app, lz_local_app_session_t *out); bool lz_svc_local_app_action(lz_local_app_session_t *session, int idx); +void lz_svc_stop_local_app(lz_local_app_session_t *session); /* ---- nodes ---- */ int lz_svc_nodes(const lz_node_rt **out); /* all heard nodes */ diff --git a/src/services/mesh_core.c b/src/services/mesh_core.c index 665f367..47077db 100644 --- a/src/services/mesh_core.c +++ b/src/services/mesh_core.c @@ -25,6 +25,7 @@ bool lz_store_app_data_usage(const lz_local_app_t *app, uint32_t *used, uint32_t bool lz_store_clear_app_data(const lz_local_app_t *app, char *err, int err_cap); bool lz_store_start_local_app(const lz_local_app_t *app, lz_local_app_session_t *out); bool lz_store_local_app_action(lz_local_app_session_t *session, int idx); +void lz_store_stop_local_app(lz_local_app_session_t *session); void lz_store_append(const char *addr, const lz_msg_rt *m); int lz_store_load_tail(const char *addr, lz_msg_rt *ring, int cap); bool lz_store_find_delivery(const char *addr, uint32_t pkt_id, lz_msg_rt *out); @@ -402,6 +403,11 @@ bool lz_svc_local_app_action(lz_local_app_session_t *session, int idx) return local_app_expand_session(session); } +void lz_svc_stop_local_app(lz_local_app_session_t *session) +{ + lz_store_stop_local_app(session); +} + const char *lz_fmt_ago(uint32_t ts, char *buf, size_t n) { if(ts == 0) { snprintf(buf, n, "-"); return buf; } diff --git a/src/services/store.c b/src/services/store.c index 0d73515..ce621bd 100644 --- a/src/services/store.c +++ b/src/services/store.c @@ -919,6 +919,11 @@ bool lz_store_local_app_action(lz_local_app_session_t *session, int idx) return true; } +void lz_store_stop_local_app(lz_local_app_session_t *session) +{ + if(session) memset(session, 0, sizeof *session); +} + /* addr strings can contain '!' etc; keep alnum only in filenames */ static void log_name(char *out, size_t n, const char *addr) { diff --git a/src/ui/screens/scr_apps.c b/src/ui/screens/scr_apps.c index a044024..f597c66 100644 --- a/src/ui/screens/scr_apps.c +++ b/src/ui/screens/scr_apps.c @@ -336,6 +336,11 @@ void lz_start_local_app(void) lz_go(LZ_V_LOCALAPP_RUN); } +void lz_stop_local_app(void) +{ + lz_svc_stop_local_app(&S.local_app_run); +} + void lz_open_local_app(const lz_local_app_t *app) { if(!app) return; @@ -500,7 +505,10 @@ static void local_app_run_activate(int idx) if(lz_svc_local_app_action(r, idx)) lz_rebuild(); return; } - if(idx == actions) lz_back(); + if(idx == actions) { + lz_stop_local_app(); + lz_back(); + } } void lz_scr_local_app_run(lv_obj_t *root) diff --git a/src/ui/ui.c b/src/ui/ui.c index 745856f..a93e7a4 100644 --- a/src/ui/ui.c +++ b/src/ui/ui.c @@ -327,6 +327,7 @@ void lz_go(lz_view_t v) void lz_back(void) { lz_settings_flush(); + if(S.view == LZ_V_LOCALAPP_RUN) lz_stop_local_app(); lz_view_t v = LZ_V_HOME; if(S.nav_depth > 0) v = S.nav_stack[--S.nav_depth]; if(v == LZ_V_LOCK) v = LZ_V_HOME; diff --git a/src/ui/ui.h b/src/ui/ui.h index 3fd5520..d735445 100644 --- a/src/ui/ui.h +++ b/src/ui/ui.h @@ -130,6 +130,7 @@ void lz_scr_local_app(lv_obj_t *root); void lz_scr_local_app_run(lv_obj_t *root); void lz_open_local_app(const lz_local_app_t *app); void lz_start_local_app(void); +void lz_stop_local_app(void); void lz_scr_contacts(lv_obj_t *root); void lz_scr_contact(lv_obj_t *root); void lz_scr_settings(lv_obj_t *root); From 7aecfc7aa47565a0228143befb63406abe223cf4 Mon Sep 17 00:00:00 2001 From: n30nex Date: Wed, 17 Jun 2026 21:07:52 -0400 Subject: [PATCH 09/64] Instrument TDM hold timing --- README.md | 4 +- docs/tdeck-feature-inventory.md | 2 +- docs/tdeck-firmware-roadmap.md | 4 +- docs/tdeck-hardware-dogfood-checklist.md | 4 ++ src/backend_sx1262.cpp | 71 +++++++++++++++++++++++- 5 files changed, 80 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 722c799..b255aae 100644 --- a/README.md +++ b/README.md @@ -333,7 +333,9 @@ for local apps and read-only inspection when present. state are live; identity, settings, node table, and message history persist across reboots; nothing on screen is hard-coded demo data on hardware. - **Serial console** — a USB-CDC command shell (`help`, `time`, `tz`, `net`, - `rf`, `dm status`, `nodes`, `send`, `stats`, `wifi`, `sys`, …) for control + diagnostics. + `rf`, `dm status`, `nodes`, `send`, `stats`, `wifi`, `sys`, …) for control + + diagnostics; `rf` also reports TDM delayed-switch timing and RX/ACK hold + counters for split-airtime soak runs. - **Companion bridge controls** — USB companion mode and BLE companion advertising are separate rows in Meshtastic → Nodes. Only one external app transport owns the bridge at a time: enabling BLE returns USB to the serial console; enabling diff --git a/docs/tdeck-feature-inventory.md b/docs/tdeck-feature-inventory.md index 416182f..a930088 100644 --- a/docs/tdeck-feature-inventory.md +++ b/docs/tdeck-feature-inventory.md @@ -59,7 +59,7 @@ Status labels: | Feature | Status | Evidence | Gap / Next Action | | --- | --- | --- | --- | | MeshCore compile-time gate | Partial | `#define LZ_MESHCORE_ENABLED 0` | Gate should stay off until receive/send/unified inbox tests pass. | -| TDM radio scheduler | Partial, needs validation | `lz_backend_set_networks`, profile switcher, settings airtime bar | Needs hardware soak and latency/packet-loss measurements. | +| TDM radio scheduler | Partial, needs validation | `lz_backend_set_networks`, profile switcher, settings airtime bar, serial `rf` diagnostics with delayed-switch average/max and RX/ACK hold counters | Needs hardware soak and packet-loss measurements with real simultaneous Meshtastic/MeshCore traffic. | | MeshCore RF profile | Partial, needs validation | 910.525 MHz / 62.5 kHz / SF7 / CR4/5 profile | Confirm target regions and RF compatibility with real MeshCore devices. | | MeshCore ADVERT RX | Partial, needs validation | `mc_parse`, `mc_advert_decode`, `lz_core_on_mc_node` | Only ADVERTs are decoded; encrypted payloads are ignored. | | MeshCore self-advert TX | Partial, needs validation | Ed25519 identity, self-advert builder, serial/UI advert commands | Needs interop proof with real MeshCore nodes. | diff --git a/docs/tdeck-firmware-roadmap.md b/docs/tdeck-firmware-roadmap.md index f987e35..a0743c3 100644 --- a/docs/tdeck-firmware-roadmap.md +++ b/docs/tdeck-firmware-roadmap.md @@ -130,7 +130,9 @@ Goal: make MeshCore visible in the real product through public chat first, while Deliverables: - Validate TDM on real hardware: - - slot switching latency + - slot switching latency. Instrumented in `rf`: the scheduler now reports + delayed-switch count, average/max lateness, and whether expired slots were + held by in-flight RX or MeshCore ACK dwell. - missed-packet rate - Meshtastic delivery impact while MeshCore is enabled - MeshCore delivery impact while Meshtastic is enabled diff --git a/docs/tdeck-hardware-dogfood-checklist.md b/docs/tdeck-hardware-dogfood-checklist.md index e152594..0bfa82b 100644 --- a/docs/tdeck-hardware-dogfood-checklist.md +++ b/docs/tdeck-hardware-dogfood-checklist.md @@ -38,6 +38,10 @@ dogfood belong to the later roadmap phases. - Capture the boot banner and every `[ok]` or failure line. - Confirm display, touch, keyboard, trackball, SD, SX1262, Wi-Fi state, battery, and time source are reported. - Run `help` and confirm diagnostics include `dm status`, `rxlog`, `nodes`, `net`, `rf`, `companion`, and `companion ble`. +- For split-airtime runs, capture `rf` before and after traffic. The `timing:` + line reports delayed-switch count, average/max lateness, and whether expired + slots were held by in-flight RX or MeshCore ACK dwell; this proves scheduler + hold behavior but not packet-loss rate by itself. ## Hardware Evidence Log diff --git a/src/backend_sx1262.cpp b/src/backend_sx1262.cpp index d21bde2..7abb8fe 100644 --- a/src/backend_sx1262.cpp +++ b/src/backend_sx1262.cpp @@ -85,6 +85,59 @@ static const uint32_t SLOT_TOTAL_MS = 500; static const uint32_t ACK_DWELL_MC = 700; /* linger on MC after our want-ack DM (peer TXT_ACK_DELAY 200 + ack airtime + ~1 hop) */ static uint32_t g_rx_mt, g_rx_mc; /* per-network packet counts */ static uint32_t g_switches; /* TDM profile switches */ +static bool g_switch_waiting; /* slot expired and is waiting to retune */ +static bool g_switch_wait_rx; /* current late switch saw RX hold */ +static bool g_switch_wait_ack; /* current late switch saw ACK-dwell hold */ +static uint32_t g_switch_due_ms; /* original slot deadline for the waiting switch */ +static uint32_t g_switch_late_count; /* switches delayed by RX/ACK holds */ +static uint64_t g_switch_late_total_ms; +static uint32_t g_switch_late_max_ms; +static uint32_t g_rx_hold_count; /* expired-slot episodes held for in-flight RX */ +static uint32_t g_ack_hold_count; /* expired-slot episodes held for MC ACK dwell */ + +static void tdm_wait_reset(void) +{ + g_switch_waiting = false; + g_switch_wait_rx = false; + g_switch_wait_ack = false; + g_switch_due_ms = 0; +} + +static void tdm_wait_due(uint32_t due) +{ + if(g_switch_waiting) return; + g_switch_waiting = true; + g_switch_due_ms = due; + g_switch_wait_rx = false; + g_switch_wait_ack = false; +} + +static void tdm_note_rx_hold(void) +{ + if(!g_switch_wait_rx) { + g_switch_wait_rx = true; + g_rx_hold_count++; + } +} + +static void tdm_note_ack_hold(void) +{ + if(!g_switch_wait_ack) { + g_switch_wait_ack = true; + g_ack_hold_count++; + } +} + +static void tdm_note_switch(uint32_t now) +{ + if(g_switch_waiting && (g_switch_wait_rx || g_switch_wait_ack)) { + uint32_t late = now - g_switch_due_ms; + g_switch_late_count++; + g_switch_late_total_ms += late; + if(late > g_switch_late_max_ms) g_switch_late_max_ms = late; + } + tdm_wait_reset(); +} static uint32_t slot_ms_for(int which) { @@ -115,6 +168,7 @@ static void apply_profile(int which) * airtime of a frame on that profile (SF11/BW250 is far slower than SF7/BW62.5). */ static void slot_begin(int which, uint32_t now) { + tdm_wait_reset(); apply_profile(which); g_slot_until = now + slot_ms_for(which); g_rx_guard_until = g_slot_until + (which == PROF_MT ? 2000u : 600u); @@ -856,14 +910,18 @@ void lz_backend_loop(void) if(g_net_mt && g_net_mc && g_slot_until && (int32_t)(now - g_slot_until) >= 0) { drain_rx(); /* same-tick: a frame may have completed since the top */ now = millis(); + tdm_wait_due(g_slot_until); uint16_t irq = radio.getIrqStatus(); bool rx_in_flight = (irq & RADIOLIB_SX126X_IRQ_HEADER_VALID) && !(irq & RADIOLIB_SX126X_IRQ_RX_DONE); if(rx_in_flight && (int32_t)(now - g_rx_guard_until) < 0) { + tdm_note_rx_hold(); /* hold: finish the in-flight frame */ } else if((int32_t)(now - g_ack_dwell_until) < 0) { + tdm_note_ack_hold(); /* hold: lingering on MC to catch our DM's ACK (g_ack_dwell_until=0 => no hold) */ } else { + tdm_note_switch(now); slot_begin(g_active == PROF_MT ? PROF_MC : PROF_MT, now); /* clears g_ack_dwell_until */ g_switches++; } @@ -996,6 +1054,7 @@ extern "C" void lz_backend_set_networks(bool mt, bool mc) if(!g_ok) return; drain_rx(); /* don't drop a frame just received on the old profile */ g_ack_dwell_until = 0; + tdm_wait_reset(); if(mt && mc) { /* share the radio, start on Meshtastic */ slot_begin(PROF_MT, millis()); } else if(mt) { @@ -1014,6 +1073,7 @@ extern "C" void lz_backend_set_airtime(int mode) g_airtime_mode = next; if(!g_ok || !(g_net_mt && g_net_mc)) return; drain_rx(); + tdm_wait_reset(); slot_begin(g_active, millis()); } @@ -1100,19 +1160,26 @@ extern "C" int lz_backend_tdm_info(char *buf, int n) : g_net_mc ? "MeshCore 100%" : "idle"); const rf_prof_t *a = &g_prof[g_active]; uint32_t rem = (g_slot_until && millis() < g_slot_until) ? g_slot_until - millis() : 0; + uint32_t late_avg = g_switch_late_count + ? (uint32_t)(g_switch_late_total_ms / g_switch_late_count) + : 0; return snprintf(buf, n, "mode: %s\n" "dwell: Meshtastic %lums / MeshCore %lums\n" "active: %s %.3f MHz BW %.1f SF%d CR4/%d (slot %lums left)\n" "Meshtastic: %.3f MHz BW%.0f SF%d rx %lu\n" "MeshCore: %.3f MHz BW%.1f SF%d rx %lu\n" - "switches: %lu", + "switches: %lu\n" + "timing: late %lu avg %lums max %lums | holds rx %lu ack %lu", mode, (unsigned long)slot_ms_for(PROF_MT), (unsigned long)slot_ms_for(PROF_MC), g_active == PROF_MT ? "Meshtastic" : "MeshCore", (double)a->freq, (double)a->bw, a->sf, a->cr, (unsigned long)rem, (double)g_prof[PROF_MT].freq, (double)g_prof[PROF_MT].bw, g_prof[PROF_MT].sf, (unsigned long)g_rx_mt, (double)g_prof[PROF_MC].freq, (double)g_prof[PROF_MC].bw, g_prof[PROF_MC].sf, (unsigned long)g_rx_mc, - (unsigned long)g_switches); + (unsigned long)g_switches, + (unsigned long)g_switch_late_count, (unsigned long)late_avg, + (unsigned long)g_switch_late_max_ms, + (unsigned long)g_rx_hold_count, (unsigned long)g_ack_hold_count); } #endif /* LZ_TARGET_TDECK */ From 51b74d25e4843652a67ad74149e2a4e50a24fd09 Mon Sep 17 00:00:00 2001 From: n30nex Date: Wed, 17 Jun 2026 21:14:55 -0400 Subject: [PATCH 10/64] Document COM8 TDM timing smoke --- docs/tdeck-hardware-dogfood-checklist.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/docs/tdeck-hardware-dogfood-checklist.md b/docs/tdeck-hardware-dogfood-checklist.md index 0bfa82b..fafceef 100644 --- a/docs/tdeck-hardware-dogfood-checklist.md +++ b/docs/tdeck-hardware-dogfood-checklist.md @@ -45,6 +45,16 @@ dogfood belong to the later roadmap phases. ## Hardware Evidence Log +### 2026-06-18 COM8 TDM Timing Diagnostic Smoke + +- Firmware flashed: PR #12 `tdeck-firmware-7aecfc7aa47565a0228143befb63406abe223cf4` artifact from upstream Firmware CI run `27730027088`. +- Artifact manifest recorded `budget_status=pass`, `firmware_bytes=1535408`, and `static_ram_bytes=274708`. +- Port boundary: only `COM8` was opened/flashed/probed during this validation. +- Direct ROM flashing with `python -m esptool --chip esp32s3 --port COM8 --baud 460800 --no-stub ... write-flash ...` succeeded; bootloader, partitions, `boot_app0.bin`, and firmware hashes all verified. +- COM8 serial smoke passed with `python scripts/tdeck_smoke.py --skip-upload --port COM8 --open-timeout 60 --boot-timeout 60 --timeout 30 --commands id rf stats`. +- `rf` reported the new timing diagnostic line on hardware: `timing: late 0 avg 0ms max 0ms | holds rx 0 ack 0`. +- This run used the default Meshtastic-only image, so it proves the diagnostic is present and stable in serial output; split-airtime RX/ACK hold counts still need a MeshCore-enabled soak run after the TDM artifact branch lands. + ### 2026-06-14 COM8 Smoke Attempt - Branch/commit flashed: `codex/tdeck-firmware-audit-roadmap` at `d5f69e0`. From 40e900a82981779fc9cf1911d9e8d5995da2db43 Mon Sep 17 00:00:00 2001 From: n30nex Date: Thu, 18 Jun 2026 17:24:35 -0400 Subject: [PATCH 11/64] Add local app runtime budget guard --- README.md | 20 ++++++------ docs/tdeck-feature-inventory.md | 4 +-- docs/tdeck-firmware-roadmap.md | 4 +++ docs/tdeck-local-app-manifest.md | 14 ++++++--- sim/main_sim.c | 54 ++++++++++++++++++++++++++++++++ src/services/mesh.h | 7 +++++ src/services/mesh_core.c | 8 +++-- src/services/store.c | 51 ++++++++++++++++++++++++++++++ src/ui/screens/scr_apps.c | 15 ++++++--- 9 files changed, 155 insertions(+), 22 deletions(-) diff --git a/README.md b/README.md index 722c799..3a73269 100644 --- a/README.md +++ b/README.md @@ -123,12 +123,13 @@ iPhone-style dark look (status bar, battery glyph, grouped settings cards). terminates on exit. Storage-enabled actions can increment a safe counter in the app's scoped `data/` directory, unsupported action effects fail closed, and apps with matching permissions can use read-only `{time}` / `{battery}` - tokens in foreground text. SDK `api_version` and permission metadata are - parsed fail-closed, with rejected package diagnostics visible in Developer - Mode. Apps that request `storage` get a scoped package `data/` directory - prepared with a 64 KB launch-time quota guard, and the App Store detail screen - can clear only that app's scoped data. Script execution, richer API injection, - downloads, and updates are still TODO. + tokens in foreground text. Loaded entry source plus app-controlled foreground + metadata are charged against a 704-byte resident runtime budget. SDK + `api_version` and permission metadata are parsed fail-closed, with rejected + package diagnostics visible in Developer Mode. Apps that request `storage` + get a scoped package `data/` directory prepared with a 64 KB launch-time quota + guard, and the App Store detail screen can clear only that app's scoped data. + Script execution, richer API injection, downloads, and updates are still TODO. - **App flash (`appfs`)** - T-Deck builds mount the FAT `appfs` partition at `/appfs` without formatting, expose it beside SD/local storage in Files, and scan `/appfs/apps` even when the SD card is absent. @@ -299,9 +300,10 @@ for local apps and read-only inspection when present. `data/` directories for storage-enabled local apps, reports quota usage, clears scoped app data on request, opens a manifest detail shell, and launches local apps into the SDK 0.1 foreground shell with bounded app-provided actions - and scoped storage counters plus read-only `{time}` / `{battery}` tokens; - unsupported action effects launch-block instead of being ignored; the static - catalog remains a prototype (GET -> "..." -> OPEN). + and scoped storage counters plus read-only `{time}` / `{battery}` tokens. The + foreground shell reports and enforces the 704-byte resident runtime metadata + budget; unsupported action effects launch-block instead of being ignored; the + static catalog remains a prototype (GET -> "..." -> OPEN). - **Contacts / detail** — unified directory with network dots; detail page with Message (jumps into the bound conversation) and spec table. - **Settings** — airtime scheduler bar that rebalances live when the diff --git a/docs/tdeck-feature-inventory.md b/docs/tdeck-feature-inventory.md index 416182f..e451160 100644 --- a/docs/tdeck-feature-inventory.md +++ b/docs/tdeck-feature-inventory.md @@ -82,7 +82,7 @@ Status labels: | Settings | Functional/Partial | network toggles, Wi-Fi, in-place brightness slider updates, time, system, touch calibration, Developer Mode, `settings.cfg` persistence | Add migration/versioning if the settings schema grows; hardware latency pass still needed. | | Wi-Fi setup | Functional, needs validation | async scan/connect, saved SSID/password, auto-connect | Credentials are plaintext on SD; only one saved network. | | System/battery page | Functional/Partial | live stats and battery arc | Hardware values need calibration/validation. | -| App Store | Prototype/Partial | `LZ_STORE` static catalog remains; local manifests from SD/appfs are scanned, listed as installed local apps, open a manifest detail shell, show storage quota usage, clear scoped app data on request, launch into the SDK 0.1 foreground shell, expose bounded foreground actions with scoped storage counters plus read-only `{time}`/`{battery}` tokens, reject unsupported action effects, and show rejected package diagnostics in Developer Mode | Network catalog, download, install/update, script execution/richer API injection, and runtime crash capture are still missing. | +| App Store | Prototype/Partial | `LZ_STORE` static catalog remains; local manifests from SD/appfs are scanned, listed as installed local apps, open a manifest detail shell, show storage quota usage, clear scoped app data on request, launch into the SDK 0.1 foreground shell, expose bounded foreground actions with scoped storage counters plus read-only `{time}`/`{battery}` tokens, enforce a 704-byte resident source/metadata runtime budget, reject unsupported action effects, and show rejected package diagnostics in Developer Mode | Network catalog, download, install/update, script execution/richer API injection, and runtime crash capture are still missing. | | Terminal | Functional/Partial | interactive UI terminal behind Developer Mode; serial CLI always available over USB | Expand diagnostics once Developer Mode grows into a full power-user surface. | | Files | Functional/Partial | read-only bounded filesystem browser rooted at mounted SD/local store or mounted FAT appfs; when both are present it starts at a Storage root picker | Add gated file actions later. | @@ -93,7 +93,7 @@ Status labels: | Lua sandbox | Planned | Design spec section 9 | Choose Lua/eLua/minimal interpreter after memory profiling. | | App manifest | Partial | `docs/tdeck-local-app-manifest.md`; bounded manifest parser requires `id`, `name`, and relative `entry`, with optional version/author/summary/icon/hue plus SDK `api_version` and permission metadata | Extend once the runtime lifecycle and package actions are chosen. | | App permissions | Partial | Local manifests can declare allowlisted SDK namespaces (`display`, `input`, `storage`, mesh, time, battery, notifications, Wi-Fi); unknown permission names reject the package before Home/App Store; `storage` prepares a scoped package `data/` directory with a 64 KB launch-time quota guard, SDK action counters require both `input` and `storage`, and `{time}`/`{battery}` tokens require matching `system_time`/`battery` permission before launch | Implement least-privilege API injection when the runtime is selected. | -| Local app scanner | Partial | `lz_store_scan_apps` scans `/sd/limitlezz/apps`, `/sd/apps`, `/appfs/apps`, simulator `/apps`, and simulator `/appfs/apps`; accepted apps appear in the paged Home launcher and App Store; rejected packages are exposed through Developer Mode diagnostics; simulator selftest covers appfs-only discovery, valid metadata, storage sandbox prep, quota usage, clear-data behavior, foreground launch metadata/actions, storage counter persistence, read-only time/battery token gating, unsupported action-effect blocking, oversized entry blocking, and rejected unsafe packages | Add script execution, richer app lifecycle hooks, and broader user-facing data actions once memory profiling picks a runtime. | +| Local app scanner | Partial | `lz_store_scan_apps` scans `/sd/limitlezz/apps`, `/sd/apps`, `/appfs/apps`, simulator `/apps`, and simulator `/appfs/apps`; accepted apps appear in the paged Home launcher and App Store; rejected packages are exposed through Developer Mode diagnostics; simulator selftest covers appfs-only discovery, valid metadata, storage sandbox prep, quota usage, clear-data behavior, foreground launch metadata/actions, runtime memory-budget enforcement, storage counter persistence, read-only time/battery token gating, unsupported action-effect blocking, oversized entry blocking, and rejected unsafe packages | Add script execution, richer app lifecycle hooks, and broader user-facing data actions once memory profiling picks a runtime. | | Network app catalog | Planned | Wi-Fi service notes; design spec | Fetch `index.json`, verify TLS/metadata, cache results. | | App download/install/update | Planned | App Store prototype only | SHA256 verify, extract, version updates, rollback failed installs. | | Optional map app | Planned | Store data includes maps; maintainer notes prefer maps as optional | Keep maps out of the base firmware. | diff --git a/docs/tdeck-firmware-roadmap.md b/docs/tdeck-firmware-roadmap.md index f987e35..eb6da59 100644 --- a/docs/tdeck-firmware-roadmap.md +++ b/docs/tdeck-firmware-roadmap.md @@ -259,6 +259,10 @@ Deliverables: current session body/status plus scoped counter state, and background execution is not exposed. - Enforce memory cap through the runtime allocator or equivalent guard. + Implemented for the SDK 0.1 foreground shell: loaded entry source plus + app-controlled title, status, body, action, effect, and storage-path metadata + are charged against a 704-byte resident runtime budget, and over-budget apps + are launch-blocked before future script runtime code can run. - Implement a small initial SDK: - UI primitives compatible with the T-Deck screen - mesh send/receive API through the service, not radio hardware diff --git a/docs/tdeck-local-app-manifest.md b/docs/tdeck-local-app-manifest.md index fb8198e..e4cd3b3 100644 --- a/docs/tdeck-local-app-manifest.md +++ b/docs/tdeck-local-app-manifest.md @@ -48,12 +48,15 @@ Each package directory must contain: exist The current SDK 0.1 foreground shell reads bounded display metadata and up to -two bounded foreground actions from the entry file. The entry metadata budget is +two bounded foreground actions from the entry file. The entry source budget is 1 KB; larger entry files are shown as launch-blocked instead of being -truncated. It accepts optional `title:`, `status:`, `body:`, `text:`, and -`action:` lines, including Lua-comment style lines such as `-- body: Local -dashboard`. Script execution and richer API injection are still later runtime -work. +truncated. The loaded entry source plus parsed foreground metadata are also +charged against a 704-byte runtime budget covering the resident title, status, +body, action labels, action text, effects, and storage path. Apps that exceed +either budget are launch-blocked before future runtime code can run. It accepts +optional `title:`, `status:`, `body:`, `text:`, and `action:` lines, including +Lua-comment style lines such as `-- body: Local dashboard`. Script execution +and richer API injection are still later runtime work. Action lines use pipe-separated fields: @@ -147,6 +150,7 @@ The scanner rejects packages when: missing on disk - `api_version` names an unsupported SDK version - `permissions` is not an array of supported namespace strings +- the foreground entry exceeds the source budget or parsed runtime memory budget The current firmware scans local app manifests and can open them in a safe foreground shell with bounded foreground actions, including a storage-scoped diff --git a/sim/main_sim.c b/sim/main_sim.c index 9c54b9b..36d4122 100644 --- a/sim/main_sim.c +++ b/sim/main_sim.c @@ -79,6 +79,12 @@ static void sim_write_bytes(const char *path, int bytes) fclose(f); } +static void sim_fwrite_repeat(FILE *f, char c, int bytes) +{ + if(!f) return; + for(int i = 0; i < bytes; i++) fputc(c, f); +} + static void sim_write_local_app(const char *datadir, const char *slug, const char *id, const char *name, const char *entry_name, const char *icon, @@ -803,6 +809,10 @@ static int codec_selftest(void) "local app foreground session keeps scoped storage"); CHECK(run_ok && run.action_count == 1 && strcmp(run.actions[0].label, "Refresh") == 0, "local app foreground session exposes bounded action"); + CHECK(run_ok && run.runtime_used_bytes > 0 && + run.runtime_used_bytes <= run.runtime_budget_bytes && + run.runtime_budget_bytes == LZ_LOCAL_APP_RUNTIME_BUDGET_BYTES, + "local app foreground session reports runtime memory budget"); bool action_ok = run_ok && lz_store_local_app_action(&run, 0); CHECK(action_ok && run.action_last == 1 && strcmp(run.status, "Forecast refreshed #1") == 0 && @@ -913,6 +923,50 @@ static int codec_selftest(void) CHECK(!badeffect_ok && badeffect && strcmp(badeffect_run.error, "unsupported action effect") == 0, "local app foreground rejects unsupported action effects"); + sim_mkdirs("lzdata_appscan/apps/fatmeta"); + FILE *fmm = fopen("lzdata_appscan/apps/fatmeta/manifest.json", "wb"); + if(fmm) { + fputs("{\"id\":\"fatmeta.local\",\"name\":\"Fat Metadata\",\"entry\":\"main.lua\"," + "\"permissions\":[\"display\",\"input\",\"storage\"]}", fmm); + fclose(fmm); + } + FILE *fme = fopen("lzdata_appscan/apps/fatmeta/main.lua", "wb"); + if(fme) { + fputs("-- title: Fat Metadata\n-- status: ", fme); + sim_fwrite_repeat(fme, 's', 63); + fputs("\n-- body: ", fme); + sim_fwrite_repeat(fme, 'b', 95); + fputs("\n-- body: ", fme); + sim_fwrite_repeat(fme, 'c', 95); + fputs("\n-- body: ", fme); + sim_fwrite_repeat(fme, 'd', 95); + fputs("\n-- action: ", fme); + sim_fwrite_repeat(fme, 'l', 23); + fputs(" | ", fme); + sim_fwrite_repeat(fme, 'a', 47); + fputs(" | ", fme); + sim_fwrite_repeat(fme, 'c', 10); + fputs(" | counter:abcdefghijklmnopqrs\n-- action: ", fme); + sim_fwrite_repeat(fme, 'm', 23); + fputs(" | ", fme); + sim_fwrite_repeat(fme, 'd', 47); + fputs(" | ", fme); + sim_fwrite_repeat(fme, 'e', 10); + fputs(" | counter:tsrqponmlkjihgfedcb\n", fme); + fclose(fme); + } + lz_local_app_t budget_apps[LZ_MAX_LOCAL_APPS]; + int budgetn = lz_store_scan_apps(budget_apps, LZ_MAX_LOCAL_APPS); + lz_local_app_t *fatmeta = NULL; + for(int i = 0; i < budgetn; i++) + if(strcmp(budget_apps[i].id, "fatmeta.local") == 0) fatmeta = &budget_apps[i]; + lz_local_app_session_t fatmeta_run; + bool fatmeta_ok = fatmeta && lz_store_start_local_app(fatmeta, &fatmeta_run); + CHECK(!fatmeta_ok && fatmeta && + strcmp(fatmeta_run.error, "runtime memory cap exceeded") == 0, + "local app foreground session blocks runtime metadata over budget"); + CHECK(fatmeta && fatmeta_run.runtime_used_bytes > fatmeta_run.runtime_budget_bytes, + "local app runtime budget records overage"); sim_mkdirs("lzdata_appscan/apps/huge"); FILE *hm = fopen("lzdata_appscan/apps/huge/manifest.json", "wb"); if(hm) { diff --git a/src/services/mesh.h b/src/services/mesh.h index aacfc8d..0dd35e0 100644 --- a/src/services/mesh.h +++ b/src/services/mesh.h @@ -30,6 +30,7 @@ extern "C" { #define LZ_MAX_LOCAL_APP_ISSUES 8 #define LZ_LOCAL_APP_BODY_MAX 360 #define LZ_LOCAL_APP_ENTRY_MAX 1024u +#define LZ_LOCAL_APP_RUNTIME_BUDGET_BYTES 704u #define LZ_LOCAL_APP_DATA_QUOTA_BYTES (64u * 1024u) #define LZ_LOCAL_APP_ACTION_MAX 2 #define LZ_LOCAL_APP_ACTION_EFFECT_MAX 32 @@ -210,6 +211,9 @@ typedef struct { char error[48]; /* launch blocked reason, if any */ uint32_t data_used_bytes; uint32_t data_quota_bytes; + uint32_t entry_source_bytes; + uint32_t runtime_used_bytes; /* app-controlled resident text/action state */ + uint32_t runtime_budget_bytes; uint16_t permissions; /* manifest permissions captured at launch */ bool entry_loaded; bool storage_ready; @@ -234,6 +238,9 @@ bool lz_svc_app_data_usage(const lz_local_app_t *app, uint32_t *used, uint32_t * bool lz_svc_clear_app_data(const lz_local_app_t *app, char *err, int err_cap); bool lz_svc_start_local_app(const lz_local_app_t *app, lz_local_app_session_t *out); bool lz_svc_local_app_action(lz_local_app_session_t *session, int idx); +uint32_t lz_local_app_runtime_used(const lz_local_app_session_t *session); +void lz_local_app_runtime_refresh(lz_local_app_session_t *session); +bool lz_local_app_runtime_within_budget(lz_local_app_session_t *session); /* ---- nodes ---- */ int lz_svc_nodes(const lz_node_rt **out); /* all heard nodes */ diff --git a/src/services/mesh_core.c b/src/services/mesh_core.c index 665f367..01b3b89 100644 --- a/src/services/mesh_core.c +++ b/src/services/mesh_core.c @@ -156,7 +156,11 @@ static void local_app_battery_token(char *out, size_t cap) static bool local_app_expand_session(lz_local_app_session_t *s) { - if(!s || s->error[0]) return false; + if(!s) return false; + if(s->error[0]) { + lz_local_app_runtime_refresh(s); + return true; + } bool need_time = local_app_session_has_token(s, "{time}"); bool need_battery = local_app_session_has_token(s, "{battery}"); if(need_time && (s->permissions & LZ_APP_PERM_SYSTEM_TIME) == 0) @@ -169,7 +173,7 @@ static bool local_app_expand_session(lz_local_app_session_t *s) local_app_battery_token(battery_s, sizeof battery_s); local_app_expand_text(s->status, sizeof s->status, time_s, battery_s); local_app_expand_text(s->body, sizeof s->body, time_s, battery_s); - return true; + return lz_local_app_runtime_within_budget(s); } static uint32_t next_packet_id(void) diff --git a/src/services/store.c b/src/services/store.c index 0d73515..325f125 100644 --- a/src/services/store.c +++ b/src/services/store.c @@ -564,13 +564,60 @@ static void refresh_session_data_usage(lz_local_app_session_t *s) s->data_used_bytes = used; } +static void runtime_count_text(uint32_t *used, const char *text) +{ + if(!used || !text || !text[0]) return; + size_t len = strlen(text) + 1u; + if(len > UINT32_MAX - *used) *used = UINT32_MAX; + else *used += (uint32_t)len; +} + +uint32_t lz_local_app_runtime_used(const lz_local_app_session_t *s) +{ + if(!s) return 0; + uint32_t used = s->entry_source_bytes; + runtime_count_text(&used, s->title); + runtime_count_text(&used, s->status); + runtime_count_text(&used, s->body); + if(s->storage_ready) runtime_count_text(&used, s->data_path); + for(int i = 0; i < s->action_count && i < LZ_LOCAL_APP_ACTION_MAX; i++) { + runtime_count_text(&used, s->actions[i].label); + runtime_count_text(&used, s->actions[i].status); + runtime_count_text(&used, s->actions[i].body); + runtime_count_text(&used, s->actions[i].effect); + } + return used; +} + +void lz_local_app_runtime_refresh(lz_local_app_session_t *s) +{ + if(!s) return; + if(!s->runtime_budget_bytes) + s->runtime_budget_bytes = LZ_LOCAL_APP_RUNTIME_BUDGET_BYTES; + s->runtime_used_bytes = lz_local_app_runtime_used(s); +} + +bool lz_local_app_runtime_within_budget(lz_local_app_session_t *s) +{ + if(!s) return false; + lz_local_app_runtime_refresh(s); + if(s->runtime_used_bytes <= s->runtime_budget_bytes) return true; + snprintf(s->status, sizeof s->status, "Launch blocked"); + snprintf(s->body, sizeof s->body, "App runtime metadata exceeds the SDK memory cap."); + snprintf(s->error, sizeof s->error, "runtime memory cap exceeded"); + return false; +} + static bool app_session_fail(lz_local_app_session_t *out, const lz_local_app_t *app, const char *msg) { if(out) { + if(!out->runtime_budget_bytes) + out->runtime_budget_bytes = LZ_LOCAL_APP_RUNTIME_BUDGET_BYTES; if(app && app->name[0]) snprintf(out->title, sizeof out->title, "%s", app->name); if(!out->status[0]) snprintf(out->status, sizeof out->status, "Launch blocked"); snprintf(out->error, sizeof out->error, "%s", msg ? msg : "unknown error"); + lz_local_app_runtime_refresh(out); } return false; } @@ -802,6 +849,7 @@ bool lz_store_start_local_app(const lz_local_app_t *app, lz_local_app_session_t { if(!out) return false; memset(out, 0, sizeof *out); + out->runtime_budget_bytes = LZ_LOCAL_APP_RUNTIME_BUDGET_BYTES; if(!app || !app->id[0]) return app_session_fail(out, app, "missing app"); out->permissions = app->permissions; snprintf(out->title, sizeof out->title, "%s", app->name[0] ? app->name : app->id); @@ -844,6 +892,7 @@ bool lz_store_start_local_app(const lz_local_app_t *app, lz_local_app_session_t fclose(f); raw[n] = 0; if(n == 0) return app_session_fail(out, app, "empty entry"); + out->entry_source_bytes = (uint32_t)n; out->entry_loaded = true; bool have_body = false; @@ -880,6 +929,7 @@ bool lz_store_start_local_app(const lz_local_app_t *app, lz_local_app_session_t if(app->summary[0]) snprintf(out->body, sizeof out->body, "%s", app->summary); else snprintf(out->body, sizeof out->body, "This local app opened in the safe SDK shell."); } + if(!lz_local_app_runtime_within_budget(out)) return false; if(out->action_count > 0 && (app->permissions & LZ_APP_PERM_INPUT) == 0) return app_session_fail(out, app, "input permission missing"); char effect_err[48]; @@ -916,6 +966,7 @@ bool lz_store_local_app_action(lz_local_app_session_t *session, int idx) if(a->body[0]) snprintf(session->body, sizeof session->body, "%s", a->body); } session->action_last = (uint8_t)(idx + 1); + (void)lz_local_app_runtime_within_budget(session); return true; } diff --git a/src/ui/screens/scr_apps.c b/src/ui/screens/scr_apps.c index a044024..80401a7 100644 --- a/src/ui/screens/scr_apps.c +++ b/src/ui/screens/scr_apps.c @@ -593,6 +593,7 @@ void lz_scr_local_app_run(lv_obj_t *root) char perms[104]; char storage[80]; char runtime[48]; + char memory[48]; app_perm_list(a->permissions, perms, sizeof perms); if(a->permissions & LZ_APP_PERM_STORAGE) { if(r->storage_ready) @@ -609,19 +610,25 @@ void lz_scr_local_app_run(lv_obj_t *root) (unsigned)r->action_count, r->action_count == 1 ? "" : "s"); else snprintf(runtime, sizeof runtime, "%s", r->entry_loaded ? "foreground only" : "not loaded"); - const char *ks[4] = { "Permissions", "Storage", "Entry", "Runtime" }; - const char *vs[4] = { perms, storage, a->entry, runtime }; + if(r->runtime_budget_bytes) + snprintf(memory, sizeof memory, "%lu / %lu B", + (unsigned long)r->runtime_used_bytes, + (unsigned long)r->runtime_budget_bytes); + else + snprintf(memory, sizeof memory, "not measured"); + const char *ks[5] = { "Permissions", "Storage", "Entry", "Runtime", "Memory" }; + const char *vs[5] = { perms, storage, a->entry, runtime, memory }; lv_obj_t *meta = lz_card(body); lv_obj_set_height(meta, LV_SIZE_CONTENT); lv_obj_set_flex_flow(meta, LV_FLEX_FLOW_COLUMN); - for(int i = 0; i < 4; i++) { + for(int i = 0; i < 5; i++) { lv_obj_t *row = lz_box(meta); lv_obj_set_width(row, lv_pct(100)); lv_obj_set_height(row, LV_SIZE_CONTENT); lv_obj_set_flex_flow(row, LV_FLEX_FLOW_COLUMN); lv_obj_set_style_pad_hor(row, 11, 0); lv_obj_set_style_pad_ver(row, 7, 0); - if(i < 3) { + if(i < 4) { lv_obj_set_style_border_side(row, LV_BORDER_SIDE_BOTTOM, 0); lv_obj_set_style_border_width(row, 1, 0); lv_obj_set_style_border_color(row, lv_color_hex(0x21262D), 0); From 0560b3504cd20aa95b53c8ec9bec549747bced3c Mon Sep 17 00:00:00 2001 From: n30nex Date: Thu, 18 Jun 2026 18:04:18 -0400 Subject: [PATCH 12/64] Add MeshCore companion v0 serial surface --- README.md | 7 +- docs/tdeck-feature-inventory.md | 2 +- docs/tdeck-firmware-roadmap.md | 13 +- docs/tdeck-meshcore-companion-protocol.md | 387 ++++++++++++++++++++++ scripts/mc_companion_usb_smoke.py | 326 ++++++++++++++++++ sim/main_sim.c | 30 +- src/serial_cli.cpp | 51 +++ src/services/mesh.h | 9 + src/services/mesh_core.c | 99 ++++++ 9 files changed, 919 insertions(+), 5 deletions(-) create mode 100644 docs/tdeck-meshcore-companion-protocol.md create mode 100644 scripts/mc_companion_usb_smoke.py diff --git a/README.md b/README.md index 722c799..810e7e8 100644 --- a/README.md +++ b/README.md @@ -113,7 +113,11 @@ iPhone-style dark look (status bar, battery glyph, grouped settings cards). enabling one frees the other. ### 🔭 Later -- **MeshCore companion bridge** — let the companion app speak MeshCore too (Public + DMs are already on-device). +- **MeshCore companion bridge** — V0 protocol foundation is drafted, and an + initial `companion mc ...` USB serial-console smoke surface can report + snapshots and exercise send boundaries; the formal USB/BLE bridge remains + planned, and it is not official MeshCore app compatible unless the real + MeshCore app protocol is confirmed. - **Roll the iPhone look everywhere** — grouped cards / dividers across Messages, Nodes, Contacts. - **Local app platform** - scan local app manifests from `/sd/limitlezz/apps`, `/sd/apps`, `/appfs/apps`, and simulator data dirs, then show accepted apps @@ -143,6 +147,7 @@ iPhone-style dark look (status bar, battery glyph, grouped settings cards). - [`docs/tdeck-firmware-audit.md`](docs/tdeck-firmware-audit.md) - current firmware audit and risk list. - [`docs/tdeck-feature-inventory.md`](docs/tdeck-feature-inventory.md) - feature-by-feature implementation inventory. - [`docs/tdeck-firmware-roadmap.md`](docs/tdeck-firmware-roadmap.md) - roadmap to a complete T-Deck firmware. +- [`docs/tdeck-meshcore-companion-protocol.md`](docs/tdeck-meshcore-companion-protocol.md) - draft Phase 5/V0.8 MeshCore companion line protocol. - [`docs/tdeck-hardware-dogfood-checklist.md`](docs/tdeck-hardware-dogfood-checklist.md) - stock-device hardware proof checklist. ![screens](docs/screens.png) diff --git a/docs/tdeck-feature-inventory.md b/docs/tdeck-feature-inventory.md index 416182f..542acbf 100644 --- a/docs/tdeck-feature-inventory.md +++ b/docs/tdeck-feature-inventory.md @@ -65,7 +65,7 @@ Status labels: | MeshCore self-advert TX | Partial, needs validation | Ed25519 identity, self-advert builder, serial/UI advert commands | Needs interop proof with real MeshCore nodes. | | MeshCore public channel / rooms | Planned | README says receive/default Public channel still ahead | V0.6: implement group text decode/send, room model, and split airtime config. | | MeshCore DMs | Planned | MeshCore contacts are non-messageable while gated | V0.7: implement key/session model, send path, ACKs, and UI routing. | -| MeshCore companion bridge | Planned | README lists as later | V0.8: build MeshCore USB and BLE companions after MeshCore messaging is stable. | +| MeshCore companion bridge | Planned/In progress | `docs/tdeck-meshcore-companion-protocol.md` drafts the V0 USB serial line protocol; `companion mc ...` provides an initial firmware smoke surface for snapshots, sends, and self-test | V0.8: formalize USB first, mirror to BLE later, and do not claim external MeshCore app compatibility until the real app protocol is confirmed. | ## User Interface diff --git a/docs/tdeck-firmware-roadmap.md b/docs/tdeck-firmware-roadmap.md index f987e35..145214e 100644 --- a/docs/tdeck-firmware-roadmap.md +++ b/docs/tdeck-firmware-roadmap.md @@ -39,7 +39,7 @@ These maintainer-provided beta labels are the canonical near-term sequence. The | V0.5 | BLE companion for Meshtastic | 🚧 Firmware done — advertises + GATT (ToRadio/FromRadio/FromNum) works on hardware; **connect-then-disconnect** with the official app is open | | V0.6 | MeshCore public chat and split airtime config | 🚧 Public chat send/receive hardware-verified; **split airtime may not be working — needs re-verification**; config UI still TODO | | V0.7 | MeshCore DMs and private chats | ✅ Encrypted DMs (X25519 ECDH + AES) send/receive hardware-verified against a real MeshCore peer | -| V0.8 | MeshCore USB companion and MeshCore BLE companion | ⬜ Not started | +| V0.8 | MeshCore USB companion and MeshCore BLE companion | 🚧 Protocol foundation drafted; USB/BLE implementation still planned and not external-app compatible yet | | V0.9 | Code review, optimization, and emoji polish | ⬜ Not started | | V0.95 | Basic app SDK and infrastructure; Home UI supports adding apps and multiple home screens | 🚧 Local manifest scanner, Home paging, and detail shell started; runtime/catalog still TODO | | V0.96 | Upgraded Wi-Fi password storage | ⬜ Not started | @@ -175,18 +175,27 @@ Exit criteria: Goal: expose MeshCore companion functionality only after native MeshCore messaging is stable. +**Status:** protocol foundation in progress with an initial USB serial-console +smoke surface. `companion mc hello|status|nodes|threads|send|dm|test` now +exercises the firmware-owned MeshCore snapshots and send boundaries for COM8 +validation, while the formal `MC0` bridge, BLE transport, events, and real +external MeshCore app compatibility are still planned work. + Deliverables: -- Define the MeshCore companion protocol surface for node DB, public chat, private chats, and send/receive forwarding. +- Define the MeshCore companion protocol surface for node DB, public chat, private chats, and send/receive forwarding. Drafted as `docs/tdeck-meshcore-companion-protocol.md`. +- Add a USB serial-console smoke surface that reports MeshCore companion status, nodes, threads, Public send, DM send, and self-test. - Implement MeshCore USB companion mode. - Implement MeshCore BLE companion mode. - Add UI and serial commands that distinguish Meshtastic companion from MeshCore companion. - Decide whether one companion session or one network can own the external-app bridge at a time. +- Confirm the real MeshCore app protocol before claiming compatibility with existing external MeshCore apps. - Hardware-test pairing, reconnect, send, receive, and disconnect flows. Exit criteria: - MeshCore companion works over USB and BLE without breaking on-device messaging or Meshtastic companion behavior. +- Existing MeshCore apps are only called compatible after their real app protocol is confirmed and mapped. ## Phase 6 - V0.9 Code Review, Optimization, And Emoji Polish diff --git a/docs/tdeck-meshcore-companion-protocol.md b/docs/tdeck-meshcore-companion-protocol.md new file mode 100644 index 0000000..4c4df4d --- /dev/null +++ b/docs/tdeck-meshcore-companion-protocol.md @@ -0,0 +1,387 @@ +# T-Deck MeshCore Companion V0 Protocol + +Status: Phase 5 / V0.8 protocol foundation plus an initial firmware serial +smoke surface. The current firmware exposes `companion mc hello`, `status`, +`nodes`, `threads`, `send`, `dm`, and `test` for USB console validation; the +formal `MC0` request/response bridge, live events, BLE transport, and external +app compatibility are still planned. + +## Goal + +MeshCore companion V0 should expose the MeshCore features that are already +owned by the T-Deck firmware without pretending to be an official MeshCore app +transport. The first target is a small USB serial protocol that lets a purpose +built test tool or LimitlezzOS companion app: + +- identify the T-Deck MeshCore identity and current bridge capabilities +- read a snapshot of known MeshCore nodes +- send Public channel text +- send DMs to a known MeshCore name or address +- receive message, node, send-status, and snapshot-change events +- recover cleanly from validation, routing, and radio errors + +BLE can reuse the same logical lines later, but USB is the V0 transport. + +## Compatibility Boundary + +This V0 protocol is not external-app compatible yet. Do not claim that existing +MeshCore phone or desktop apps can connect to LimitlezzOS through this bridge +unless a real MeshCore app protocol is confirmed and mapped. + +For V0, supported clients are test tools and purpose-built LimitlezzOS +companion experiments. The T-Deck remains the source of truth for MeshCore +keys, node state, address matching, radio scheduling, and delivery state. + +## Transport And Framing + +The USB bridge uses line-oriented text after the user or firmware has entered +MeshCore companion mode. The activation UI/serial command is intentionally left +to the firmware implementation, but it must be distinct from Meshtastic +companion mode. + +The current implementation step keeps this in the existing serial console as +`companion mc ...` commands so CI and COM8 hardware smokes can validate the +service boundary before the formal host protocol takes ownership of the port. + +- Encoding: UTF-8 text lines. +- Line ending: `\n`; firmware should accept `\r\n`. +- Request prefix: `MC0`. +- Maximum inbound line length: advertised by `HELLO` as `max_line`. +- Request ids: host-generated ASCII tokens, 1-12 characters, echoed by the + device. +- Field format: `key=value` pairs separated by one space. +- String values: percent-encoded UTF-8 bytes. Encode at least space, `%`, `=`, + `\r`, and `\n`. +- Numbers: decimal unless a field explicitly says hex. +- Booleans: `0` or `1`. +- Addresses and keys: lowercase hex, treated as opaque by the host. + +Single-line response: + +```text +MC0 OK key=value ... +MC0 ERR code= retry=<0|1> message= +``` + +Snapshot response: + +```text +MC0 BEGIN type= rev= count= more=<0|1> cursor= +MC0 key=value ... +MC0 END type= rev= count= more=<0|1> cursor= +``` + +Device event: + +```text +MC0 EVT key=value ... +``` + +Events are asynchronous and may appear between request responses unless the +firmware documents a stricter ordering. A host that loses event sequence +continuity must call `STATUS` and refresh affected snapshots. + +## Request Surface + +### `HELLO` + +The host must send `HELLO` before any other request. + +Request: + +```text +MC0 1 HELLO proto=0 app=limitlezz-test host=windows want=events +``` + +Response: + +```text +MC0 1 OK proto=0 fw=0.8-draft device=tdeck session=84 caps=identity,nodes,status,send_public,send_dm,events max_line=512 max_text=180 event_seq=120 nodes_rev=42 messages_rev=77 +``` + +Required fields: + +- `proto`: selected protocol version. V0 is `0`. +- `caps`: comma-separated capability tokens. +- `max_line`: maximum inbound request line the firmware accepts. +- `max_text`: maximum decoded message body bytes accepted for V0 sends. +- `event_seq`: last event sequence number seen by the device. +- `nodes_rev`: current node snapshot revision. +- `messages_rev`: current message/delivery snapshot revision. + +If the host requests an unsupported protocol version, the device returns +`ERR code=unsupported_version`. + +### `IDENTITY` + +Returns the local MeshCore identity as the firmware currently understands it. + +Request: + +```text +MC0 2 IDENTITY +``` + +Response: + +```text +MC0 2 OK enabled=1 name=Jess addr=4f8e21a0 role=chat pubkey=6b1d... addr_format=meshcore-hex advert_ready=1 +``` + +Required fields: + +- `enabled`: whether MeshCore is currently enabled. +- `name`: percent-encoded local display name. +- `addr`: local MeshCore address, lowercase hex, opaque to the host. +- `role`: current local MeshCore role, such as `chat`, `router`, or + `unknown`. +- `addr_format`: the address encoding advertised for this session. +- `advert_ready`: whether the firmware has enough identity state to advertise. + +Private keys must never be exposed through this protocol. + +### `STATUS` + +Returns bridge, radio, queue, and snapshot status. This is the host's resync +anchor after reconnect, dropped events, or errors. + +Request: + +```text +MC0 3 STATUS +``` + +Response: + +```text +MC0 3 OK mc=on bridge=usb mc_companion=idle mt_companion=off tdm=active airtime=balanced queue=0 event_seq=120 nodes_rev=42 messages_rev=77 +``` + +Suggested fields: + +- `mc`: `on`, `off`, or `disabled`. +- `bridge`: active transport, usually `usb` for V0. +- `mc_companion`: `idle`, `attached`, or `streaming`. +- `mt_companion`: Meshtastic companion state so clients can detect conflicts. +- `tdm`: `active`, `mc_only`, `mt_only`, or `idle`. +- `airtime`: current split-airtime preset. +- `queue`: queued outbound MeshCore sends owned by the firmware. +- `event_seq`, `nodes_rev`, `messages_rev`: resync counters. + +### `NODES` + +Returns a bounded snapshot of the firmware's known MeshCore nodes. + +Request: + +```text +MC0 4 NODES since=0 limit=50 +``` + +Response: + +```text +MC0 4 BEGIN type=nodes rev=42 count=2 more=0 cursor=end +MC0 4 NODE addr=4f8e21a0 name=Limitlezz role=chat seen_ms=12000 snr=-9 rssi=-112 public_key=present dm=ready +MC0 4 NODE addr=12ab9001 name=Hilltop role=router seen_ms=180000 snr=-14 rssi=-118 public_key=missing dm=not_messageable +MC0 4 END type=nodes rev=42 count=2 more=0 cursor=end +``` + +Request fields: + +- `since`: last `nodes_rev` known by the host, or `0` for a full snapshot. +- `limit`: maximum rows requested. Firmware may cap this below the requested + value. +- `cursor`: optional opaque cursor from a previous `NODES` response. + +Node fields: + +- `addr`: MeshCore address, lowercase hex, opaque to the host. +- `name`: percent-encoded display name, if known. +- `role`: `chat`, `router`, `repeater`, `sensor`, or `unknown`. +- `seen_ms`: milliseconds since last heard, or `-1` if unknown. +- `snr`, `rssi`: last RF quality values, or omitted when unknown. +- `public_key`: `present`, `missing`, or `unknown`. +- `dm`: `ready`, `no_key`, `not_messageable`, or `unknown`. + +V0 sends DMs only to nodes already known by `addr` or an unambiguous `name`. +It does not include contact import, key exchange control, or remote node +mutation. + +### `SEND_PUBLIC` + +Queues text for the default MeshCore Public channel or a known room token. + +Request: + +```text +MC0 5 SEND_PUBLIC room=public text=Hello%20mesh client_mid=pc-0001 +``` + +Immediate response: + +```text +MC0 5 OK accepted=1 msg_id=mc-804 queue=1 status=queued +``` + +Later events: + +```text +MC0 EVT 121 tx_status client_mid=pc-0001 msg_id=mc-804 kind=public status=sent +MC0 EVT 122 tx_status client_mid=pc-0001 msg_id=mc-804 kind=public status=delivered +``` + +Request fields: + +- `room`: `public` for V0, or a firmware-known room token later. +- `text`: percent-encoded UTF-8 body. +- `client_mid`: optional host-generated id for de-duplicating retries. + +If the command is accepted, later delivery failure is reported as +`tx_status status=failed`; it is not a synchronous `ERR`. + +### `SEND_DM` + +Queues a MeshCore private message to a known address or an unambiguous known +name. Address is preferred because display names can collide. + +By address: + +```text +MC0 6 SEND_DM to_addr=4f8e21a0 text=Meet%20at%20camp client_mid=pc-0002 +``` + +By known name: + +```text +MC0 7 SEND_DM to_name=Limitlezz text=Copy%20that client_mid=pc-0003 +``` + +Immediate response: + +```text +MC0 6 OK accepted=1 msg_id=mc-805 to_addr=4f8e21a0 status=queued +``` + +Later event: + +```text +MC0 EVT 123 tx_status client_mid=pc-0002 msg_id=mc-805 kind=dm to_addr=4f8e21a0 status=delivered +``` + +V0 name matching rules: + +- Match against the firmware's known MeshCore display name and short name. +- Case-insensitive exact match only. +- If zero nodes match, return `ERR code=not_found`. +- If more than one node matches, return `ERR code=ambiguous_name`. +- If the matching node lacks a usable session/key, return `ERR code=no_key`. +- If the node role is not messageable, return `ERR code=not_messageable`. + +The host does not manage MeshCore private keys or sessions in V0. + +### `EVENTS` + +Controls live event streaming. A V0 implementation may start events after +`HELLO want=events`, but it should still accept this explicit command. + +Request: + +```text +MC0 8 EVENTS mode=on types=nodes,messages,tx,status +``` + +Response: + +```text +MC0 8 OK events=on types=nodes,messages,tx,status event_seq=123 +``` + +Supported event types: + +```text +MC0 EVT 124 node_upsert addr=4f8e21a0 nodes_rev=43 +MC0 EVT 125 snapshot_dirty type=nodes rev=43 reason=node_upsert +MC0 EVT 126 rx_public msg_id=mc-806 from_addr=4f8e21a0 from_name=Limitlezz room=public text=Copy%20CH0. +MC0 EVT 127 rx_dm msg_id=mc-807 from_addr=4f8e21a0 from_name=Limitlezz text=Direct%20copy. +MC0 EVT 128 tx_status client_mid=pc-0002 msg_id=mc-805 kind=dm status=failed reason=ack_timeout retry=1 +MC0 EVT 129 status mc=on tdm=active airtime=balanced queue=0 +``` + +Snapshot events are hints. The host should use `STATUS`, `NODES`, or later +message snapshot commands for authoritative state after reconnect. + +## Error Semantics + +Synchronous `ERR` means the request was not accepted. Accepted sends fail later +through `tx_status`. + +Common error codes: + +| Code | Meaning | Retry | +| --- | --- | --- | +| `bad_request` | Missing field, malformed id, bad encoding, or invalid value | No | +| `unsupported_version` | Host requested an unsupported `proto` | No | +| `unknown_command` | Verb is not implemented | No | +| `not_ready` | MeshCore state is still starting or identity is unavailable | Yes | +| `meshcore_disabled` | MeshCore is disabled by settings or build gate | No | +| `busy` | Radio/bridge cannot accept more work right now | Yes | +| `not_found` | Address/name is unknown | No | +| `ambiguous_name` | Name matched more than one known node | No | +| `not_messageable` | Node role/session cannot receive DMs | No | +| `no_key` | No usable private-chat key/session for that node | No | +| `text_too_long` | Decoded text exceeds `max_text` or current packet budget | No | +| `rate_limited` | Host is sending too fast | Yes | +| `send_failed` | Firmware rejected the send before queueing | Maybe | +| `internal` | Unexpected firmware failure | Maybe | + +Error example: + +```text +MC0 10 ERR code=ambiguous_name retry=0 message=Multiple%20nodes%20named%20Limitlezz +``` + +Rules: + +- Every `ERR` includes `code`, `retry`, and `message`. +- `message` is for humans and must not be parsed for behavior. +- `retry=1` means the same request may succeed later; it is not a guarantee. +- If a request is accepted with `OK`, later delivery failures must use + `tx_status status=failed`. +- Unknown fields are ignored unless they change the meaning of a command. +- Unknown required behavior should be negotiated with `caps`, not guessed. + +## V0 Roadmap + +1. Document the V0 line protocol and keep it separate from Meshtastic + companion protocol claims. +2. Add an initial USB serial-console smoke surface for `hello`, `status`, + `nodes`, `threads`, Public send, DM send, and self-test. +3. Add a USB-only MeshCore companion mode with `HELLO`, `IDENTITY`, `STATUS`, + and `NODES`. +4. Add `SEND_PUBLIC` and `SEND_DM` using the existing firmware-owned MeshCore + send paths. +5. Add event streaming for receive, send-status, node-change, and status + changes. +6. Add snapshot revision counters and reconnect/resync behavior. +7. Mirror the same logical protocol over BLE only after USB behavior is stable. +8. Revisit external-app compatibility only after the real MeshCore app protocol + is confirmed. + +## Validation Checklist + +- USB serial host can enter MeshCore companion mode without enabling + Meshtastic companion mode. +- `HELLO`, `IDENTITY`, `STATUS`, and `NODES` work after boot and after + reconnect. +- Public send queues, emits `tx_status`, and appears in the on-device inbox. +- DM send works by address and by an unambiguous known name. +- DM name collision returns `ambiguous_name` without sending. +- Missing key/session returns `no_key` without sending. +- Dropped or skipped event sequence is recoverable through `STATUS` and + snapshots. +- On-device MeshCore messaging still works while no companion client is + attached. +- Meshtastic USB/BLE companion behavior does not regress. +- README/roadmap do not call this official MeshCore app compatibility until + the real MeshCore app protocol is confirmed. diff --git a/scripts/mc_companion_usb_smoke.py b/scripts/mc_companion_usb_smoke.py new file mode 100644 index 0000000..49b96d4 --- /dev/null +++ b/scripts/mc_companion_usb_smoke.py @@ -0,0 +1,326 @@ +#!/usr/bin/env python3 +""" +Smoke-test MeshCore USB companion v0 serial-console commands. + +The helper is intentionally conservative: it defaults to the implemented +`companion mc ...` command names, but every command and marker can be +overridden while the firmware command surface is still settling. +""" +from __future__ import annotations + +import argparse +import os +import sys +import time +from dataclasses import dataclass, field +from string import Formatter + +from serial_harness import ( + RomDownloadMode, + open_port_retry, + pulse_reset, + run_command, + sync_prompt, +) +from serial_harness import serial # pyserial module imported by the harness + + +DEFAULT_STATUS_COMMAND = "companion mc status" +DEFAULT_TEST_COMMAND = "companion mc test" +DEFAULT_PUBLIC_TEMPLATE = "companion mc send {text}" +DEFAULT_STATUS_MARKERS = ["mccomp: status", "MeshCore", "MC companion"] +DEFAULT_TEST_MARKERS = ["PASS"] +DEFAULT_PUBLIC_MARKERS = ["[ok]", "sent", "queued"] +ERROR_MARKERS = ("[err]", "unknown command", "Unknown command", "usage:") +FALLBACK_HINTS = ( + "USB companion mode:", + "BLE companion:", + "not present", + "not implemented", +) + + +@dataclass +class CommandSpec: + label: str + command: str + markers: list[str] = field(default_factory=list) + + +class SmokeFailure(RuntimeError): + def __init__(self, message: str, exit_code: int = 2) -> None: + super().__init__(message) + self.exit_code = exit_code + + +def split_markers(value: str) -> list[str]: + markers = [part.strip() for part in value.split("|")] + return [marker for marker in markers if marker] + + +def parse_expectations(values: list[str] | None) -> dict[str, list[str]]: + expectations: dict[str, list[str]] = {} + for value in values or []: + if "=" not in value: + raise SystemExit( + f"invalid --expect value {value!r}; expected COMMAND=MARKER " + "or COMMAND=MARKER1|MARKER2" + ) + command, marker_text = value.split("=", 1) + command = command.strip() + markers = split_markers(marker_text) + if not command or not markers: + raise SystemExit(f"invalid --expect value {value!r}; command and marker are required") + expectations.setdefault(command, []).extend(markers) + return expectations + + +def validate_public_template(template: str) -> None: + field_names = { + name + for _, name, _, _ in Formatter().parse(template) + if name is not None and name != "" + } + unsupported = sorted(field_names - {"text"}) + if unsupported: + names = ", ".join(unsupported) + raise SystemExit(f"--public-command-template only supports {{text}}; unsupported: {names}") + + +def add_markers( + specs: list[CommandSpec], + command: str, + markers: list[str], + no_default_expect: bool, +) -> None: + if no_default_expect: + return + for spec in specs: + if spec.command == command: + spec.markers.extend(marker for marker in markers if marker not in spec.markers) + return + + +def build_command_specs(args: argparse.Namespace) -> list[CommandSpec]: + if args.commands is not None: + specs = [CommandSpec(f"command {idx}", command) for idx, command in enumerate(args.commands, 1)] + else: + specs = [ + CommandSpec("status", args.status_command), + CommandSpec("test", args.test_command), + ] + + if args.public_text is not None: + validate_public_template(args.public_command_template) + specs.append( + CommandSpec( + "public", + args.public_command_template.format(text=args.public_text), + ) + ) + + if not specs: + raise SystemExit("[mc-smoke] no commands requested; refusing to report a false PASS") + + add_markers(specs, args.status_command, args.status_marker, args.no_default_expect) + add_markers(specs, args.test_command, args.test_marker, args.no_default_expect) + if args.public_text is not None: + add_markers(specs, specs[-1].command, args.public_marker, args.no_default_expect) + + explicit_expectations = parse_expectations(args.expect) + for spec in specs: + spec.markers.extend(explicit_expectations.get(spec.command, [])) + + unused = sorted(set(explicit_expectations) - {spec.command for spec in specs}) + if unused: + names = "\n ".join(unused) + raise SystemExit(f"--expect was provided for command(s) not in this smoke run:\n {names}") + + return specs + + +def open_console(args: argparse.Namespace) -> tuple[serial.Serial, str]: + boot_timeout = args.boot_timeout if args.boot_timeout is not None else args.open_timeout + boot_deadline = time.monotonic() + boot_timeout + last_rom = "" + rom_notice_printed = False + prompt_notice_printed = False + disconnect_notice_printed = False + + print(f"[mc-smoke] opening {args.port} @ {args.baud}") + while True: + remaining_boot = max(0.1, boot_deadline - time.monotonic()) + port = open_port_retry( + args.port, + args.baud, + args.timeout, + min(args.open_timeout, remaining_boot), + args.dtr, + args.rts, + ) + try: + if not prompt_notice_printed: + print("[mc-smoke] waiting for LimitlezzOS prompt") + prompt_notice_printed = True + if args.reset: + pulse_reset(port, args.reset_settle) + initial = sync_prompt(port, remaining_boot) + return port, initial + except RomDownloadMode as exc: + last_rom = str(exc).strip() + port.close() + if time.monotonic() >= boot_deadline: + raise + if not rom_notice_printed: + print( + "[mc-smoke] ESP-ROM download mode detected; press RESET or power-cycle " + "the T-Deck, still waiting..." + ) + rom_notice_printed = True + time.sleep(1.0) + except serial.SerialException as exc: + port.close() + if time.monotonic() >= boot_deadline: + raise SmokeFailure(f"[mc-smoke] serial error on {args.port}: {exc}", 1) from exc + if not disconnect_notice_printed: + print( + f"[mc-smoke] {args.port} disconnected during USB boot handoff; " + "waiting for it to return..." + ) + disconnect_notice_printed = True + time.sleep(1.0) + except TimeoutError as exc: + port.close() + if last_rom: + message = "\n".join( + [ + "[mc-smoke] device is in ESP-ROM download mode, not LimitlezzOS", + "[mc-smoke] ROM output:", + last_rom, + "[mc-smoke] press RESET or power-cycle the T-Deck, then rerun the smoke.", + ] + ) + raise SmokeFailure(message, 3) from exc + partial = str(exc).strip() + lines = ["[mc-smoke] timed out waiting for the LimitlezzOS prompt"] + if partial: + lines.extend(["[mc-smoke] partial output:", partial]) + lines.extend( + [ + f"[mc-smoke] requested port: {args.port}", + "[mc-smoke] hint: make sure the device is running text-console mode.", + ] + ) + raise SmokeFailure("\n".join(lines), 1) from exc + + +def assert_output(spec: CommandSpec, output: str, allow_error_output: bool) -> None: + if not allow_error_output: + for marker in ERROR_MARKERS: + if marker in output: + raise SmokeFailure( + f"[mc-smoke] command {spec.command!r} reported error marker {marker!r}" + ) + + if not spec.markers: + return + + if any(marker in output for marker in spec.markers): + return + + marker_list = ", ".join(repr(marker) for marker in spec.markers) + lines = [ + f"[mc-smoke] missing expected marker for {spec.command!r}: one of {marker_list}", + ] + if any(hint in output for hint in FALLBACK_HINTS): + lines.append( + "[mc-smoke] the command may not be implemented yet or may have fallen " + "back to the existing companion status path" + ) + lines.append( + "[mc-smoke] use --status-command/--test-command/--commands or " + "--expect to match the firmware surface, or --no-default-expect for " + "prompt-only probing" + ) + raise SmokeFailure("\n".join(lines)) + + +def run_specs(port: serial.Serial, specs: list[CommandSpec], args: argparse.Namespace) -> None: + for spec in specs: + marker_note = ", ".join(repr(marker) for marker in spec.markers) or "none" + print(f"[mc-smoke] > {spec.command} [{spec.label}; expect: {marker_note}]") + output = run_command(port, spec.command, args.timeout) + if output.strip(): + print(output.rstrip()) + assert_output(spec, output, args.allow_error_output) + + +def main() -> int: + parser = argparse.ArgumentParser( + description="Smoke-test MeshCore companion USB v0 commands over the LimitlezzOS serial console." + ) + parser.add_argument("--port", default=os.environ.get("LZ_SERIAL_PORT", "COM8")) + parser.add_argument("--baud", type=int, default=115200) + parser.add_argument("--timeout", type=float, default=30.0, help="Seconds to wait for each command response.") + parser.add_argument("--open-timeout", type=float, default=60.0, help="Seconds to wait for a COM port to appear.") + parser.add_argument("--boot-timeout", type=float, help="Seconds to wait for the first LimitlezzOS prompt.") + parser.add_argument("--dtr", choices=["default", "on", "off"], default="off") + parser.add_argument("--rts", choices=["default", "on", "off"], default="off") + parser.add_argument("--reset", action="store_true", help="Pulse reset before waiting for the prompt.") + parser.add_argument("--reset-settle", type=float, default=1.5, help="Seconds to wait after reset pulse.") + parser.add_argument("--status-command", default=DEFAULT_STATUS_COMMAND) + parser.add_argument("--test-command", default=DEFAULT_TEST_COMMAND) + parser.add_argument( + "--commands", + nargs="*", + help="Replace the default status/test command list. Pair with --expect for custom markers.", + ) + parser.add_argument("--public-text", help="Append a public MeshCore smoke send command with this text.") + parser.add_argument("--public-command-template", default=DEFAULT_PUBLIC_TEMPLATE) + parser.add_argument("--status-marker", action="append", default=list(DEFAULT_STATUS_MARKERS)) + parser.add_argument("--test-marker", action="append", default=list(DEFAULT_TEST_MARKERS)) + parser.add_argument("--public-marker", action="append", default=list(DEFAULT_PUBLIC_MARKERS)) + parser.add_argument( + "--expect", + action="append", + help="Add a marker assertion as COMMAND=MARKER or COMMAND=MARKER1|MARKER2.", + ) + parser.add_argument("--no-default-expect", action="store_true", help="Do not assert built-in planned markers.") + parser.add_argument("--allow-error-output", action="store_true", help="Do not fail on [err]/usage/unknown markers.") + args = parser.parse_args() + + specs = build_command_specs(args) + try: + port, initial = open_console(args) + with port: + if initial.strip(): + print(initial.rstrip()) + run_specs(port, specs, args) + print("[mc-smoke] smoke PASS") + return 0 + except serial.SerialException as exc: + print(f"[mc-smoke] serial error on {args.port}: {exc}", file=sys.stderr) + return 1 + except RomDownloadMode as exc: + partial = str(exc).strip() + print("[mc-smoke] device is in ESP-ROM download mode, not LimitlezzOS", file=sys.stderr) + if partial: + print("[mc-smoke] ROM output:", file=sys.stderr) + print(partial, file=sys.stderr) + print("[mc-smoke] press RESET or power-cycle the T-Deck, then rerun the smoke.", file=sys.stderr) + return 3 + except TimeoutError as exc: + partial = str(exc).strip() + print("[mc-smoke] timed out waiting for prompt/output", file=sys.stderr) + if partial: + print("[mc-smoke] partial output:", file=sys.stderr) + print(partial, file=sys.stderr) + print(f"[mc-smoke] requested port: {args.port}", file=sys.stderr) + return 1 + except SmokeFailure as exc: + print(str(exc), file=sys.stderr) + return exc.exit_code + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/sim/main_sim.c b/sim/main_sim.c index 9c54b9b..dfe6ef8 100644 --- a/sim/main_sim.c +++ b/sim/main_sim.c @@ -1048,7 +1048,35 @@ static int codec_selftest(void) sim_reset_dir("lzdata_appfsroot"); } - /* 11. MeshCore Public-channel GRP_TXT: decode a known reference vector, + /* 12. MeshCore companion v0 line surface: snapshot helpers emit stable + * markers and send through the service boundary. */ + { + lz_svc_init(NULL, false); + uint8_t pub[32] = {0}; + pub[0] = 0x42; + lz_core_on_mc_node(pub, "CompanionPeer", 1, -7.5f); + lz_core_on_mc_channel_text("CompanionPeer", "public hello", -7.5f); + lz_core_on_mc_dm(pub, "CompanionPeer", "dm hello", -7.5f); + char hello[180], status[220], nodes[420], threads[520]; + lz_svc_mc_companion_hello(hello, sizeof hello); + lz_svc_mc_companion_status(status, sizeof status); + lz_svc_mc_companion_nodes(nodes, sizeof nodes); + lz_svc_mc_companion_threads(threads, sizeof threads); + CHECK(strstr(hello, "mccomp: hello v0") != NULL, + "MeshCore companion v0 hello reports protocol"); + CHECK(strstr(status, "nodes=1") != NULL && strstr(status, "threads=2") != NULL, + "MeshCore companion v0 status counts snapshots"); + CHECK(strstr(nodes, "CompanionPeer") != NULL && strstr(nodes, "dm=yes") != NULL, + "MeshCore companion v0 node snapshot lists messageable peer"); + CHECK(strstr(threads, "public hello") != NULL && strstr(threads, "dm hello") != NULL, + "MeshCore companion v0 thread snapshot lists public and DM threads"); + CHECK(lz_svc_mc_companion_send_public("public from companion"), + "MeshCore companion v0 public send uses service boundary"); + CHECK(lz_svc_mc_companion_send_dm("CompanionPeer", "dm from companion"), + "MeshCore companion v0 DM send uses service boundary"); + } + + /* 13. MeshCore Public-channel GRP_TXT: decode a known reference vector, * reject a wrong key (MAC), and round-trip an encode. Vector generated * against the documented scheme (AES-128-ECB + HMAC-SHA256 trunc-2). */ { diff --git a/src/serial_cli.cpp b/src/serial_cli.cpp index b46d44c..f4f171e 100644 --- a/src/serial_cli.cpp +++ b/src/serial_cli.cpp @@ -62,6 +62,7 @@ static void cmd_help(void) " mc test build+verify our advert (proves nodes will accept it)\n" " companion on|off USB acts as a Meshtastic-app companion radio\n" " companion ble on|off|test BLE Meshtastic-app companion advertising\n" + " companion mc hello|status|nodes|threads|send|dm|test MeshCore companion v0\n" " companion test loopback-verify the companion protocol\n" " touch [cal|debug|S X Y] touch: 'cal' runs on-screen calibration, 'debug' logs taps, 'S X Y' sets transform\n" " dm status show pending sent-DM delivery state\n" @@ -292,6 +293,56 @@ static void cmd_touch(char *args) static void cmd_companion(char *args) { + if(args && strncmp(args, "mc", 2) == 0 && (args[2] == 0 || args[2] == ' ')) { + char *sub = args + 2; + while(*sub == ' ') sub++; + if(!sub[0] || strcmp(sub, "status") == 0) { + char b[220]; lz_svc_mc_companion_status(b, sizeof b); Serial.print(b); + return; + } + if(strcmp(sub, "hello") == 0) { + char b[220]; lz_svc_mc_companion_hello(b, sizeof b); Serial.print(b); + return; + } + if(strcmp(sub, "nodes") == 0) { + char b[760]; lz_svc_mc_companion_nodes(b, sizeof b); Serial.print(b); + return; + } + if(strcmp(sub, "threads") == 0) { + char b[760]; lz_svc_mc_companion_threads(b, sizeof b); Serial.print(b); + return; + } + if(strcmp(sub, "test") == 0) { + char h[220], st[220], nodes[220]; + lz_svc_mc_companion_hello(h, sizeof h); + lz_svc_mc_companion_status(st, sizeof st); + lz_svc_mc_companion_nodes(nodes, sizeof nodes); + bool ok = strstr(h, "mccomp: hello") && strstr(st, "mccomp: status") && + strstr(nodes, "mccomp-node:"); + Serial.printf("MeshCore companion v0 selftest: %s\n", ok ? "PASS" : "FAIL"); + return; + } + if(strncmp(sub, "send ", 5) == 0) { + const char *text = sub + 5; + if(lz_svc_mc_companion_send_public(text)) + Serial.println("[ok] mc companion public send queued"); + else + Serial.println("[err] mc companion public send failed"); + return; + } + if(strncmp(sub, "dm ", 3) == 0) { + char *p = sub + 3; char *sp = strchr(p, ' '); + if(!sp) { Serial.println("usage: companion mc dm "); return; } + *sp = 0; const char *text = sp + 1; + if(lz_svc_mc_companion_send_dm(p, text)) + Serial.printf("[ok] mc companion DM queued to %s\n", p); + else + Serial.println("[err] mc companion DM failed"); + return; + } + Serial.println("usage: companion mc hello|status|nodes|threads|send |dm |test"); + return; + } if(args && strncmp(args, "ble", 3) == 0) { char state[8] = {0}; if(sscanf(args, "ble %7s", state) == 1) { diff --git a/src/services/mesh.h b/src/services/mesh.h index aacfc8d..a6377a7 100644 --- a/src/services/mesh.h +++ b/src/services/mesh.h @@ -263,6 +263,15 @@ bool lz_svc_resend(int tail_idx); /* retry a failed sent DM (long-press) */ const char *lz_svc_delivery_fail_label(uint8_t reason); int lz_svc_delivery_diag(char *buf, int n); /* serial: pending DM ACK state */ +/* MeshCore companion v0: line-oriented snapshot/send surface for USB smoke and + * future external bridge work. This is not an external app compatibility claim. */ +int lz_svc_mc_companion_hello(char *buf, int n); +int lz_svc_mc_companion_status(char *buf, int n); +int lz_svc_mc_companion_nodes(char *buf, int n); +int lz_svc_mc_companion_threads(char *buf, int n); +bool lz_svc_mc_companion_send_public(const char *text); +bool lz_svc_mc_companion_send_dm(const char *name, const char *text); + /* ---- radio stats (airtime accounting) ---- */ typedef struct { uint32_t tx_count, rx_count; float util_pct; } lz_radio_stats_t; void lz_svc_radio_stats(lz_radio_stats_t *out); diff --git a/src/services/mesh_core.c b/src/services/mesh_core.c index 665f367..f69b1f3 100644 --- a/src/services/mesh_core.c +++ b/src/services/mesh_core.c @@ -1043,6 +1043,105 @@ int lz_svc_delivery_diag(char *buf, int n) return pos; } +int lz_svc_mc_companion_hello(char *buf, int n) +{ + if(!buf || n <= 0) return 0; + char addr[24]; + lz_backend_mc_addr(addr, sizeof addr); + const char *build = +#if LZ_MESHCORE_ENABLED + "enabled"; +#else + "gated"; +#endif + return snprintf(buf, (size_t)n, + "mccomp: hello v0 build=%s meshcore=%s addr=%s protocol=line\n", + build, build, addr); +} + +int lz_svc_mc_companion_status(char *buf, int n) +{ + if(!buf || n <= 0) return 0; + int mc_nodes = 0, mc_threads = 0, unread = 0; + for(int i = 0; i < g_node_count; i++) + if(g_nodes[i].net == LZ_NET_MC) mc_nodes++; + for(int i = 0; i < g_thread_count; i++) { + if(g_threads[i].net != LZ_NET_MC) continue; + mc_threads++; + unread += g_threads[i].unread; + } + char addr[24]; + lz_backend_mc_addr(addr, sizeof addr); + return snprintf(buf, (size_t)n, + "mccomp: status v0 addr=%s nodes=%d threads=%d unread=%d public=%s dm=%s\n", + addr, mc_nodes, mc_threads, unread, + lz_backend_mc_send_public ? "sendable" : "unavailable", + lz_backend_mc_dm ? "sendable" : "unavailable"); +} + +static bool mc_companion_dm_target(const lz_node_rt *n) +{ + if(!n || n->net != LZ_NET_MC) return false; + return strcmp(n->role, "Chat") == 0; +} + +int lz_svc_mc_companion_nodes(char *buf, int n) +{ + int pos = 0, listed = 0; + if(!buf || n <= 0) return 0; + for(int i = 0; i < g_node_count; i++) { + const lz_node_rt *nd = &g_nodes[i]; + if(nd->net != LZ_NET_MC) continue; + char ago[12]; + lz_fmt_ago(nd->last_heard, ago, sizeof ago); + pos = buf_appendf(buf, n, pos, + "mccomp-node: name=\"%s\" id=%s role=%s snr=%.1f last=%s dm=%s\n", + nd->name, nd->id, nd->role, (double)nd->snr, ago, + mc_companion_dm_target(nd) ? "yes" : "no"); + listed++; + } + if(!listed) + pos = buf_appendf(buf, n, pos, "mccomp-node: none\n"); + return pos; +} + +int lz_svc_mc_companion_threads(char *buf, int n) +{ + int pos = 0, listed = 0; + if(!buf || n <= 0) return 0; + for(int oi = 0; oi < g_thread_count; oi++) { + int idx = (oi < LZ_MAX_THREADS) ? g_order[oi] : -1; + if(idx < 0 || idx >= g_thread_count) continue; + const lz_thread_rt *t = &g_threads[idx]; + if(t->net != LZ_NET_MC) continue; + char ago[12]; + lz_fmt_ago(t->last_ts, ago, sizeof ago); + pos = buf_appendf(buf, n, pos, + "mccomp-thread: name=\"%s\" addr=%s kind=%s unread=%d last=%s text=\"%s\"\n", + t->name, t->addr, t->is_channel ? "public" : "dm", + t->unread, ago, t->last_text); + listed++; + } + if(!listed) + pos = buf_appendf(buf, n, pos, "mccomp-thread: none\n"); + return pos; +} + +bool lz_svc_mc_companion_send_public(const char *text) +{ + if(!text || !text[0]) return false; + lz_thread_rt *t = lz_svc_mc_channel_thread(); + return t && lz_svc_send_text(t, text); +} + +bool lz_svc_mc_companion_send_dm(const char *name, const char *text) +{ + if(!name || !name[0] || !text || !text[0]) return false; + lz_node_rt *n = lz_svc_node_by_name(name); + if(!mc_companion_dm_target(n)) return false; + return lz_backend_mc_dm && lz_backend_mc_dm(name, text); +} + /* ---------- inbound events from backends ---------- */ void lz_core_on_heard(uint32_t from, float snr) From ed8cfef954b41334863fe5d4b26786d164b797aa Mon Sep 17 00:00:00 2001 From: n30nex Date: Thu, 18 Jun 2026 18:32:23 -0400 Subject: [PATCH 13/64] Add MeshCore MC0 USB companion mode --- docs/tdeck-meshcore-companion-protocol.md | 27 ++ scripts/mc_companion_usb_smoke.py | 344 +++++++++++++++++++++- sim/main_sim.c | 30 ++ src/main_tdeck.cpp | 7 +- src/mc_companion.cpp | 77 +++++ src/mt_companion.cpp | 2 + src/serial_cli.cpp | 36 ++- src/services/mesh.h | 9 + src/services/mesh_core.c | 342 +++++++++++++++++++++ 9 files changed, 857 insertions(+), 17 deletions(-) create mode 100644 src/mc_companion.cpp diff --git a/docs/tdeck-meshcore-companion-protocol.md b/docs/tdeck-meshcore-companion-protocol.md index 4c4df4d..e5cea30 100644 --- a/docs/tdeck-meshcore-companion-protocol.md +++ b/docs/tdeck-meshcore-companion-protocol.md @@ -43,6 +43,21 @@ The current implementation step keeps this in the existing serial console as `companion mc ...` commands so CI and COM8 hardware smokes can validate the service boundary before the formal host protocol takes ownership of the port. +Formal USB MC0 mode is an explicit mode switch rather than an implicit reuse of +the normal console prompt. The firmware enters MC0 mode with: + +```text +companion mc usb on +MC0 1 HELLO proto=0 app=limitlezz-smoke host=windows want=none +MC0 2 STATUS +MC0 3 NODES since=0 limit=5 +MC0 99 EXIT +``` + +Host smoke keeps the mode-entry command and exit line configurable for older +artifacts and later protocol revisions, but it must not treat Meshtastic +companion mode as a fallback. + - Encoding: UTF-8 text lines. - Line ending: `\n`; firmware should accept `\r\n`. - Request prefix: `MC0`. @@ -370,6 +385,18 @@ Rules: ## Validation Checklist +- Default smoke remains the serial-console boundary check: + `python scripts/mc_companion_usb_smoke.py` runs `companion mc status` and + `companion mc test`. +- Formal USB MC0 smoke is opt-in: + `python scripts/mc_companion_usb_smoke.py --mc0-usb` enters the configured + USB mode, sends `HELLO`, `STATUS`, and `NODES`, asserts `MC0 ... OK`, + `BEGIN`, and `END` response markers, then exits through the configured + `MC0 EXIT` line. +- If firmware lands different command names, use the smoke helper's + `--mc0-enter-command`, `--mc0-*-template`, `--mc0-*-marker`, and + `--mc0-exit-template` flags instead of editing firmware or weakening the + assertions. - USB serial host can enter MeshCore companion mode without enabling Meshtastic companion mode. - `HELLO`, `IDENTITY`, `STATUS`, and `NODES` work after boot and after diff --git a/scripts/mc_companion_usb_smoke.py b/scripts/mc_companion_usb_smoke.py index 49b96d4..1d7f61f 100644 --- a/scripts/mc_companion_usb_smoke.py +++ b/scripts/mc_companion_usb_smoke.py @@ -4,7 +4,8 @@ The helper is intentionally conservative: it defaults to the implemented `companion mc ...` command names, but every command and marker can be -overridden while the firmware command surface is still settling. +overridden while the firmware command surface is still settling. Use +`--mc0-usb` to probe the formal raw MC0 request/response mode. """ from __future__ import annotations @@ -17,10 +18,12 @@ from serial_harness import ( RomDownloadMode, + decode, open_port_retry, pulse_reset, run_command, sync_prompt, + write_line, ) from serial_harness import serial # pyserial module imported by the harness @@ -28,16 +31,27 @@ DEFAULT_STATUS_COMMAND = "companion mc status" DEFAULT_TEST_COMMAND = "companion mc test" DEFAULT_PUBLIC_TEMPLATE = "companion mc send {text}" +DEFAULT_MC0_ENTER_COMMAND = "companion mc usb on" +DEFAULT_MC0_HELLO_ID = "1" +DEFAULT_MC0_STATUS_ID = "2" +DEFAULT_MC0_NODES_ID = "3" +DEFAULT_MC0_EXIT_ID = "99" +DEFAULT_MC0_HELLO_TEMPLATE = "MC0 {id} HELLO proto=0 app=limitlezz-smoke host=windows want=none" +DEFAULT_MC0_STATUS_TEMPLATE = "MC0 {id} STATUS" +DEFAULT_MC0_NODES_TEMPLATE = "MC0 {id} NODES since=0 limit=5" +DEFAULT_MC0_EXIT_TEMPLATE = "MC0 {id} EXIT" DEFAULT_STATUS_MARKERS = ["mccomp: status", "MeshCore", "MC companion"] DEFAULT_TEST_MARKERS = ["PASS"] DEFAULT_PUBLIC_MARKERS = ["[ok]", "sent", "queued"] ERROR_MARKERS = ("[err]", "unknown command", "Unknown command", "usage:") +MC0_ERROR_MARKERS = (" ERR code=", " ERR ") FALLBACK_HINTS = ( "USB companion mode:", "BLE companion:", "not present", "not implemented", ) +PROMPT_TEXT = "lz> " @dataclass @@ -58,6 +72,13 @@ def split_markers(value: str) -> list[str]: return [marker for marker in markers if marker] +def expand_markers(values: list[str] | None) -> list[str]: + markers: list[str] = [] + for value in values or []: + markers.extend(split_markers(value)) + return markers + + def parse_expectations(values: list[str] | None) -> dict[str, list[str]]: expectations: dict[str, list[str]] = {} for value in values or []: @@ -75,16 +96,26 @@ def parse_expectations(values: list[str] | None) -> dict[str, list[str]]: return expectations -def validate_public_template(template: str) -> None: +def validate_template_fields(option: str, template: str, allowed: set[str]) -> None: field_names = { name for _, name, _, _ in Formatter().parse(template) if name is not None and name != "" } - unsupported = sorted(field_names - {"text"}) + unsupported = sorted(field_names - allowed) if unsupported: names = ", ".join(unsupported) - raise SystemExit(f"--public-command-template only supports {{text}}; unsupported: {names}") + allowed_names = ", ".join(f"{{{name}}}" for name in sorted(allowed)) or "no fields" + raise SystemExit(f"{option} only supports {allowed_names}; unsupported: {names}") + + +def validate_public_template(template: str) -> None: + validate_template_fields("--public-command-template", template, {"text"}) + + +def format_mc0_template(option: str, template: str, request_id: str) -> str: + validate_template_fields(option, template, {"id"}) + return template.format(id=request_id) def add_markers( @@ -139,6 +170,50 @@ def build_command_specs(args: argparse.Namespace) -> list[CommandSpec]: return specs +def default_marker_override(custom: list[str] | None, defaults: list[str]) -> list[str]: + if custom is None: + return list(defaults) + return expand_markers(custom) + + +def build_mc0_specs(args: argparse.Namespace) -> list[CommandSpec]: + hello_command = format_mc0_template( + "--mc0-hello-template", args.mc0_hello_template, args.mc0_hello_id + ) + status_command = format_mc0_template( + "--mc0-status-template", args.mc0_status_template, args.mc0_status_id + ) + nodes_command = format_mc0_template( + "--mc0-nodes-template", args.mc0_nodes_template, args.mc0_nodes_id + ) + return [ + CommandSpec( + "mc0 hello", + hello_command, + default_marker_override( + args.mc0_hello_marker, + [f"MC0 {args.mc0_hello_id} OK"], + ), + ), + CommandSpec( + "mc0 status", + status_command, + default_marker_override( + args.mc0_status_marker, + [f"MC0 {args.mc0_status_id} OK"], + ), + ), + CommandSpec( + "mc0 nodes", + nodes_command, + default_marker_override( + args.mc0_nodes_marker, + [f"MC0 {args.mc0_nodes_id} BEGIN", f"MC0 {args.mc0_nodes_id} END"], + ), + ), + ] + + def open_console(args: argparse.Namespace) -> tuple[serial.Serial, str]: boot_timeout = args.boot_timeout if args.boot_timeout is not None else args.open_timeout boot_deadline = time.monotonic() + boot_timeout @@ -214,13 +289,31 @@ def open_console(args: argparse.Namespace) -> tuple[serial.Serial, str]: raise SmokeFailure("\n".join(lines), 1) from exc +def has_all_markers(output: str, markers: list[str]) -> bool: + return all(marker in output for marker in markers) + + +def output_has_response_boundary(output: str, markers: list[str]) -> bool: + if not markers: + return True + if PROMPT_TEXT in markers and PROMPT_TEXT in output: + return True + return output.endswith("\n") + + +def fail_on_error_markers(label: str, output: str, allow_error_output: bool) -> None: + if allow_error_output: + return + for marker in ERROR_MARKERS: + if marker in output: + raise SmokeFailure(f"[mc-smoke] {label} reported error marker {marker!r}") + for marker in MC0_ERROR_MARKERS: + if marker in output: + raise SmokeFailure(f"[mc-smoke] {label} reported MC0 error marker {marker!r}") + + def assert_output(spec: CommandSpec, output: str, allow_error_output: bool) -> None: - if not allow_error_output: - for marker in ERROR_MARKERS: - if marker in output: - raise SmokeFailure( - f"[mc-smoke] command {spec.command!r} reported error marker {marker!r}" - ) + fail_on_error_markers(f"command {spec.command!r}", output, allow_error_output) if not spec.markers: return @@ -245,6 +338,49 @@ def assert_output(spec: CommandSpec, output: str, allow_error_output: bool) -> N raise SmokeFailure("\n".join(lines)) +def assert_mc0_output(spec: CommandSpec, output: str, allow_error_output: bool) -> None: + fail_on_error_markers(spec.label, output, allow_error_output) + if not has_all_markers(output, spec.markers): + missing = ", ".join(repr(marker) for marker in spec.markers if marker not in output) + raise SmokeFailure(f"[mc-smoke] missing MC0 marker(s) for {spec.command!r}: {missing}") + + +def read_until_markers_or_idle( + port: serial.Serial, + markers: list[str], + timeout: float, + idle_timeout: float, + label: str, + allow_error_output: bool, +) -> str: + end = time.monotonic() + timeout + idle_deadline = time.monotonic() + idle_timeout + buf = bytearray() + while time.monotonic() < end: + chunk = port.read(1) + if chunk: + buf.extend(chunk) + idle_deadline = time.monotonic() + idle_timeout + output = decode(bytes(buf)) + fail_on_error_markers(label, output, allow_error_output) + if has_all_markers(output, markers) and output_has_response_boundary(output, markers): + return output + else: + if not markers and time.monotonic() >= idle_deadline: + return decode(bytes(buf)) + time.sleep(0.03) + + output = decode(bytes(buf)) + if markers: + missing = ", ".join(repr(marker) for marker in markers if marker not in output) + detail = f"[mc-smoke] timed out waiting for {label}; missing marker(s): {missing}" + else: + detail = f"[mc-smoke] timed out waiting for {label}" + if output.strip(): + detail = "\n".join([detail, "[mc-smoke] partial output:", output.rstrip()]) + raise SmokeFailure(detail) + + def run_specs(port: serial.Serial, specs: list[CommandSpec], args: argparse.Namespace) -> None: for spec in specs: marker_note = ", ".join(repr(marker) for marker in spec.markers) or "none" @@ -255,6 +391,87 @@ def run_specs(port: serial.Serial, specs: list[CommandSpec], args: argparse.Name assert_output(spec, output, args.allow_error_output) +def run_mc0_specs(port: serial.Serial, specs: list[CommandSpec], args: argparse.Namespace) -> None: + # These command names are intentionally configurable so older artifacts or + # later protocol revisions can still be probed without weakening checks. + entered = False + if args.mc0_enter_command: + enter_markers = default_marker_override(args.mc0_enter_marker, []) + enter_spec = CommandSpec("mc0 enter", args.mc0_enter_command, enter_markers) + marker_note = ", ".join(repr(marker) for marker in enter_spec.markers) or "none" + print(f"[mc-smoke] > {enter_spec.command} [{enter_spec.label}; expect: {marker_note}]") + port.reset_input_buffer() + write_line(port, enter_spec.command) + output = read_until_markers_or_idle( + port, + enter_spec.markers, + args.mc0_enter_timeout, + args.mc0_idle_timeout, + enter_spec.label, + args.allow_error_output, + ) + if output.strip(): + print(output.rstrip()) + assert_mc0_output(enter_spec, output, args.allow_error_output) + if args.mc0_enter_settle > 0: + time.sleep(args.mc0_enter_settle) + entered = True + + failure: SmokeFailure | None = None + try: + for spec in specs: + marker_note = ", ".join(repr(marker) for marker in spec.markers) or "none" + print(f"[mc-smoke] > {spec.command} [{spec.label}; expect: {marker_note}]") + write_line(port, spec.command) + output = read_until_markers_or_idle( + port, + spec.markers, + args.timeout, + args.mc0_idle_timeout, + spec.label, + args.allow_error_output, + ) + if output.strip(): + print(output.rstrip()) + assert_mc0_output(spec, output, args.allow_error_output) + except SmokeFailure as exc: + failure = exc + + exit_failure: SmokeFailure | None = None + if entered and args.mc0_exit_template: + exit_command = format_mc0_template( + "--mc0-exit-template", args.mc0_exit_template, args.mc0_exit_id + ) + exit_markers = default_marker_override( + args.mc0_exit_marker, + [f"MC0 {args.mc0_exit_id} OK", PROMPT_TEXT], + ) + exit_spec = CommandSpec("mc0 exit", exit_command, exit_markers) + marker_note = ", ".join(repr(marker) for marker in exit_spec.markers) or "none" + print(f"[mc-smoke] > {exit_spec.command} [{exit_spec.label}; expect: {marker_note}]") + try: + write_line(port, exit_spec.command) + output = read_until_markers_or_idle( + port, + exit_spec.markers, + args.mc0_exit_timeout, + args.mc0_idle_timeout, + exit_spec.label, + args.allow_error_output, + ) + if output.strip(): + print(output.rstrip()) + assert_mc0_output(exit_spec, output, args.allow_error_output) + except SmokeFailure as exc: + exit_failure = exc + print(str(exc), file=sys.stderr) + + if failure is not None: + raise failure + if exit_failure is not None: + raise exit_failure + + def main() -> int: parser = argparse.ArgumentParser( description="Smoke-test MeshCore companion USB v0 commands over the LimitlezzOS serial console." @@ -287,15 +504,118 @@ def main() -> int: ) parser.add_argument("--no-default-expect", action="store_true", help="Do not assert built-in planned markers.") parser.add_argument("--allow-error-output", action="store_true", help="Do not fail on [err]/usage/unknown markers.") + mc0 = parser.add_argument_group("formal MC0 USB mode") + mc0.add_argument( + "--mc0-usb", + action="store_true", + help=( + "Opt into the formal MC0 USB protocol smoke instead of the " + "default serial-console status/test smoke." + ), + ) + mc0.add_argument( + "--mc0-enter-command", + default=DEFAULT_MC0_ENTER_COMMAND, + help=( + "Console command used to enter formal MeshCore USB mode. " + "Override this when testing older artifacts or protocol revisions." + ), + ) + mc0.add_argument( + "--mc0-enter-marker", + action="append", + help=( + "Optional marker to require after the enter command. Repeat or use " + "MARKER1|MARKER2. No marker is required by default because the " + "command may take ownership of the port without printing a prompt." + ), + ) + mc0.add_argument( + "--mc0-enter-timeout", + type=float, + default=5.0, + help="Seconds to wait for optional output from the MC0 enter command.", + ) + mc0.add_argument( + "--mc0-enter-settle", + type=float, + default=0.2, + help="Seconds to wait after the enter command before sending MC0 requests.", + ) + mc0.add_argument( + "--mc0-idle-timeout", + type=float, + default=0.5, + help="Seconds of quiet serial input that ends an optional MC0 read.", + ) + mc0.add_argument("--mc0-hello-id", default=DEFAULT_MC0_HELLO_ID) + mc0.add_argument("--mc0-status-id", default=DEFAULT_MC0_STATUS_ID) + mc0.add_argument("--mc0-nodes-id", default=DEFAULT_MC0_NODES_ID) + mc0.add_argument("--mc0-exit-id", default=DEFAULT_MC0_EXIT_ID) + mc0.add_argument( + "--mc0-hello-template", + default=DEFAULT_MC0_HELLO_TEMPLATE, + help="MC0 HELLO line template; supports {id}.", + ) + mc0.add_argument( + "--mc0-status-template", + default=DEFAULT_MC0_STATUS_TEMPLATE, + help="MC0 STATUS line template; supports {id}.", + ) + mc0.add_argument( + "--mc0-nodes-template", + default=DEFAULT_MC0_NODES_TEMPLATE, + help="MC0 NODES line template; supports {id}.", + ) + mc0.add_argument( + "--mc0-hello-marker", + action="append", + help="Override HELLO expected marker(s). Defaults to 'MC0 OK'.", + ) + mc0.add_argument( + "--mc0-status-marker", + action="append", + help="Override STATUS expected marker(s). Defaults to 'MC0 OK'.", + ) + mc0.add_argument( + "--mc0-nodes-marker", + action="append", + help=( + "Override NODES expected marker(s). Defaults to both " + "'MC0 BEGIN' and 'MC0 END'." + ), + ) + mc0.add_argument( + "--mc0-exit-template", + default=DEFAULT_MC0_EXIT_TEMPLATE, + help=( + "MC0 line used to leave formal USB mode and return to the console; " + "supports {id}. Set to an empty string to skip the exit attempt." + ), + ) + mc0.add_argument( + "--mc0-exit-marker", + action="append", + help="Override exit expected marker(s). Defaults to MC0 OK plus the LimitlezzOS prompt.", + ) + mc0.add_argument( + "--mc0-exit-timeout", + type=float, + default=10.0, + help="Seconds to wait for the MC0 exit response or console prompt.", + ) args = parser.parse_args() - specs = build_command_specs(args) + specs = build_mc0_specs(args) if args.mc0_usb else build_command_specs(args) try: port, initial = open_console(args) with port: if initial.strip(): print(initial.rstrip()) - run_specs(port, specs, args) + if args.mc0_usb: + run_mc0_specs(port, specs, args) + else: + run_specs(port, specs, args) print("[mc-smoke] smoke PASS") return 0 except serial.SerialException as exc: diff --git a/sim/main_sim.c b/sim/main_sim.c index dfe6ef8..4481b9b 100644 --- a/sim/main_sim.c +++ b/sim/main_sim.c @@ -1074,6 +1074,36 @@ static int codec_selftest(void) "MeshCore companion v0 public send uses service boundary"); CHECK(lz_svc_mc_companion_send_dm("CompanionPeer", "dm from companion"), "MeshCore companion v0 DM send uses service boundary"); + char mc0[900], proto[120]; + bool mc0_exit = false; + lz_svc_mc_companion_handle_line("MC0 1 HELLO proto=0 app=selftest", mc0, sizeof mc0, &mc0_exit); + CHECK(strstr(mc0, "MC0 1 OK proto=0") != NULL && strstr(mc0, "caps=") != NULL, + "MeshCore MC0 HELLO reports protocol capabilities"); + lz_svc_mc_companion_handle_line("MC0 2 STATUS", mc0, sizeof mc0, &mc0_exit); + CHECK(strstr(mc0, "MC0 2 OK") != NULL && strstr(mc0, "nodes=1") != NULL && + strstr(mc0, "threads=2") != NULL, + "MeshCore MC0 STATUS counts snapshots"); + lz_svc_mc_companion_handle_line("MC0 3 NODES since=0 limit=5", mc0, sizeof mc0, &mc0_exit); + CHECK(strstr(mc0, "MC0 3 BEGIN type=nodes") != NULL && + strstr(mc0, "name=CompanionPeer") != NULL && + strstr(mc0, "MC0 3 END type=nodes") != NULL, + "MeshCore MC0 NODES snapshot lists peer"); + lz_svc_mc_companion_handle_line("MC0 4 THREADS", mc0, sizeof mc0, &mc0_exit); + CHECK(strstr(mc0, "MC0 4 BEGIN type=threads") != NULL && + strstr(mc0, "text=CompanionPeer%3A%20public%20hello") != NULL && + strstr(mc0, "text=dm%20hello") != NULL, + "MeshCore MC0 THREADS snapshot lists encoded thread text"); + lz_svc_mc_companion_handle_line("MC0 5 SEND_PUBLIC text=mc0%20public", mc0, sizeof mc0, &mc0_exit); + CHECK(strstr(mc0, "MC0 5 OK accepted=1") != NULL, + "MeshCore MC0 SEND_PUBLIC uses service boundary"); + lz_svc_mc_companion_handle_line("MC0 6 SEND_DM to_name=companionpeer text=mc0%20dm", mc0, sizeof mc0, &mc0_exit); + CHECK(strstr(mc0, "MC0 6 OK accepted=1") != NULL, + "MeshCore MC0 SEND_DM uses service boundary"); + lz_svc_mc_companion_handle_line("MC0 7 EXIT", mc0, sizeof mc0, &mc0_exit); + CHECK(mc0_exit && strstr(mc0, "state=detached") != NULL, + "MeshCore MC0 EXIT returns to console"); + lz_svc_mc_companion_selftest(proto, sizeof proto); + CHECK(strstr(proto, "PASS") != NULL, "MeshCore MC0 protocol selftest passes"); } /* 13. MeshCore Public-channel GRP_TXT: decode a known reference vector, diff --git a/src/main_tdeck.cpp b/src/main_tdeck.cpp index c620373..4d520a7 100644 --- a/src/main_tdeck.cpp +++ b/src/main_tdeck.cpp @@ -168,6 +168,8 @@ void lz_cli_poll(void); extern "C" void lz_mtc_poll(void); /* Meshtastic companion bridge (mt_companion.cpp) */ extern "C" void lz_mtc_ble_begin(void); extern "C" void lz_mtc_ble_poll(void); +extern "C" bool lz_mcc_usb_active(void); /* MeshCore companion bridge (mc_companion.cpp) */ +extern "C" void lz_mcc_usb_poll(void); static void flush_cb(lv_disp_drv_t *drv, const lv_area_t *area, lv_color_t *px) { @@ -548,8 +550,9 @@ void loop() kb_backlight_update(); lz_mtc_ble_poll(); /* BLE advert/connection maintenance */ - if(lz_mtc_active()) lz_mtc_poll(); /* companion mode: USB speaks the Meshtastic app protocol */ - else lz_cli_poll(); /* otherwise: the text command console */ + if(lz_mcc_usb_active()) lz_mcc_usb_poll(); /* MeshCore companion mode: USB speaks MC0 text */ + else if(lz_mtc_active()) lz_mtc_poll(); /* Meshtastic companion mode: USB speaks Stream API */ + else lz_cli_poll(); /* otherwise: the text command console */ lz_svc_loop(); static int last_wifi = -1; lz_wifi_loop(); diff --git a/src/mc_companion.cpp b/src/mc_companion.cpp new file mode 100644 index 0000000..226cd7d --- /dev/null +++ b/src/mc_companion.cpp @@ -0,0 +1,77 @@ +#ifdef LZ_TARGET_TDECK + +#include +#include +#include "services/mesh.h" +#include "ui/ui.h" + +static bool g_mc_usb; +static char g_mc_line[512]; +static uint16_t g_mc_len; + +extern "C" bool lz_mcc_usb_active(void) +{ + return g_mc_usb; +} + +extern "C" void lz_mcc_usb_set_active(bool on) +{ + if(on) { + if(lz_mtc_active()) lz_mtc_set_active(false); + if(lz_mtc_ble_enabled()) lz_mtc_ble_set_enabled(false); + } + g_mc_usb = on; + g_mc_len = 0; +} + +extern "C" int lz_mcc_usb_status(char *buf, int n) +{ + return snprintf(buf, (size_t)n, "MeshCore USB companion: %s%s", + g_mc_usb ? "MC0 attached" : "off", + g_mc_usb ? " (send MC0 EXIT for console)" : ""); +} + +extern "C" int lz_mcc_usb_selftest(char *buf, int n) +{ + char svc[80]; + int written = lz_svc_mc_companion_selftest(svc, sizeof svc); + (void)written; + bool ok = strstr(svc, "PASS") != NULL; + return snprintf(buf, (size_t)n, "MeshCore USB MC0 selftest: %s | %s", + ok ? "PASS" : "FAIL", svc); +} + +static void mc_usb_handle_line(void) +{ + char out[1200]; + bool exit_mode = false; + g_mc_line[g_mc_len] = 0; + lz_svc_mc_companion_handle_line(g_mc_line, out, sizeof out, &exit_mode); + if(out[0]) Serial.print(out); + g_mc_len = 0; + if(exit_mode) { + g_mc_usb = false; + Serial.print("\nlz> "); + } +} + +extern "C" void lz_mcc_usb_poll(void) +{ + while(Serial.available()) { + lz_note_activity(); + char c = (char)Serial.read(); + if(c == '\r') continue; + if(c == '\n') { + mc_usb_handle_line(); + } else if(g_mc_len < sizeof(g_mc_line) - 1) { + g_mc_line[g_mc_len++] = c; + } else { + static const char msg[] = + "MC0 0 ERR code=bad_request retry=0 message=line%20too%20long\n"; + Serial.print(msg); + g_mc_len = 0; + } + } +} + +#endif /* LZ_TARGET_TDECK */ diff --git a/src/mt_companion.cpp b/src/mt_companion.cpp index ec94ae7..8b46000 100644 --- a/src/mt_companion.cpp +++ b/src/mt_companion.cpp @@ -846,6 +846,7 @@ extern "C" void lz_mtc_ble_set_enabled(bool on) * both be resident — free WiFi first so the controller can init. */ if(lz_wifi_enabled()) lz_wifi_set_enabled(false); if(g_companion) g_companion = false; /* one external app bridge at a time */ + if(lz_mcc_usb_active()) lz_mcc_usb_set_active(false); if(!g_ble_ready) lz_mtc_ble_begin(); if(!g_ble_ready) { /* controller init failed (out of RAM): */ lz_wifi_set_enabled(true); /* don't strand the user — restore WiFi */ @@ -1001,6 +1002,7 @@ extern "C" void lz_mtc_poll(void) /* ---------- mode control + self-test ---------- */ extern "C" void lz_mtc_set_active(bool on) { + if(on && lz_mcc_usb_active()) lz_mcc_usb_set_active(false); if(on && g_ble_enabled) lz_mtc_ble_set_enabled(false); g_companion = on; if(on) { diff --git a/src/serial_cli.cpp b/src/serial_cli.cpp index f4f171e..b048935 100644 --- a/src/serial_cli.cpp +++ b/src/serial_cli.cpp @@ -63,6 +63,7 @@ static void cmd_help(void) " companion on|off USB acts as a Meshtastic-app companion radio\n" " companion ble on|off|test BLE Meshtastic-app companion advertising\n" " companion mc hello|status|nodes|threads|send|dm|test MeshCore companion v0\n" + " companion mc usb on|off|status|test USB speaks MeshCore MC0\n" " companion test loopback-verify the companion protocol\n" " touch [cal|debug|S X Y] touch: 'cal' runs on-screen calibration, 'debug' logs taps, 'S X Y' sets transform\n" " dm status show pending sent-DM delivery state\n" @@ -296,6 +297,31 @@ static void cmd_companion(char *args) if(args && strncmp(args, "mc", 2) == 0 && (args[2] == 0 || args[2] == ' ')) { char *sub = args + 2; while(*sub == ' ') sub++; + if(strncmp(sub, "usb", 3) == 0 && (sub[3] == 0 || sub[3] == ' ')) { + char *state = sub + 3; + while(*state == ' ') state++; + if(!state[0] || strcmp(state, "status") == 0) { + char b[140]; lz_mcc_usb_status(b, sizeof b); Serial.println(b); + return; + } + if(strcmp(state, "test") == 0) { + char b[180]; lz_mcc_usb_selftest(b, sizeof b); Serial.println(b); + return; + } + if(strcmp(state, "on") == 0) { + lz_mcc_usb_set_active(true); + Serial.println("[ok] MeshCore MC0 USB companion mode ON"); + Serial.println(" send 'MC0 1 EXIT' to return to the text console"); + return; + } + if(strcmp(state, "off") == 0) { + lz_mcc_usb_set_active(false); + Serial.println("[ok] MeshCore MC0 USB companion mode OFF"); + return; + } + Serial.println("usage: companion mc usb on|off|status|test"); + return; + } if(!sub[0] || strcmp(sub, "status") == 0) { char b[220]; lz_svc_mc_companion_status(b, sizeof b); Serial.print(b); return; @@ -314,12 +340,15 @@ static void cmd_companion(char *args) } if(strcmp(sub, "test") == 0) { char h[220], st[220], nodes[220]; + char proto[120]; lz_svc_mc_companion_hello(h, sizeof h); lz_svc_mc_companion_status(st, sizeof st); lz_svc_mc_companion_nodes(nodes, sizeof nodes); + lz_svc_mc_companion_selftest(proto, sizeof proto); bool ok = strstr(h, "mccomp: hello") && strstr(st, "mccomp: status") && - strstr(nodes, "mccomp-node:"); + strstr(nodes, "mccomp-node:") && strstr(proto, "PASS"); Serial.printf("MeshCore companion v0 selftest: %s\n", ok ? "PASS" : "FAIL"); + Serial.println(proto); return; } if(strncmp(sub, "send ", 5) == 0) { @@ -340,13 +369,13 @@ static void cmd_companion(char *args) Serial.println("[err] mc companion DM failed"); return; } - Serial.println("usage: companion mc hello|status|nodes|threads|send |dm |test"); + Serial.println("usage: companion mc hello|status|nodes|threads|send |dm |test|usb on|off|status|test"); return; } if(args && strncmp(args, "ble", 3) == 0) { char state[8] = {0}; if(sscanf(args, "ble %7s", state) == 1) { - if(strcmp(state, "on") == 0) lz_mtc_ble_set_enabled(true); + if(strcmp(state, "on") == 0) { lz_mcc_usb_set_active(false); lz_mtc_ble_set_enabled(true); } if(strcmp(state, "off") == 0) lz_mtc_ble_set_enabled(false); if(strcmp(state, "test") == 0 && lz_mtc_ble_selftest) { char b[120]; lz_mtc_ble_selftest(b, sizeof b); Serial.println(b); @@ -364,6 +393,7 @@ static void cmd_companion(char *args) return; } if(args && strcmp(args, "on") == 0) { + lz_mcc_usb_set_active(false); Serial.println("[ok] companion mode ON - USB now speaks the Meshtastic app protocol"); Serial.println(" (text console returns after 'companion off' or reboot)"); lz_mtc_set_active(true); diff --git a/src/services/mesh.h b/src/services/mesh.h index a6377a7..9994459 100644 --- a/src/services/mesh.h +++ b/src/services/mesh.h @@ -271,6 +271,8 @@ int lz_svc_mc_companion_nodes(char *buf, int n); int lz_svc_mc_companion_threads(char *buf, int n); bool lz_svc_mc_companion_send_public(const char *text); bool lz_svc_mc_companion_send_dm(const char *name, const char *text); +int lz_svc_mc_companion_handle_line(const char *line, char *buf, int n, bool *exit_mode); +int lz_svc_mc_companion_selftest(char *buf, int n); /* ---- radio stats (airtime accounting) ---- */ typedef struct { uint32_t tx_count, rx_count; float util_pct; } lz_radio_stats_t; @@ -373,6 +375,13 @@ void lz_mtc_ble_set_enabled(bool on); int lz_mtc_ble_status(char *buf, int n); int lz_mtc_ble_selftest(char *buf, int n); +/* MeshCore companion bridge: USB serial speaks the MC0 line protocol when active. */ +bool lz_mcc_usb_active(void); +void lz_mcc_usb_set_active(bool on); +void lz_mcc_usb_poll(void); +int lz_mcc_usb_status(char *buf, int n); +int lz_mcc_usb_selftest(char *buf, int n); + /* called by backends on radio events */ void lz_core_on_text(uint32_t from, uint32_t to, const char *text, int hops_used, float snr); void lz_core_on_nodeinfo(uint32_t from, const char *id, const char *long_name, diff --git a/src/services/mesh_core.c b/src/services/mesh_core.c index f69b1f3..efc40fa 100644 --- a/src/services/mesh_core.c +++ b/src/services/mesh_core.c @@ -1142,6 +1142,348 @@ bool lz_svc_mc_companion_send_dm(const char *name, const char *text) return lz_backend_mc_dm && lz_backend_mc_dm(name, text); } +static const char *mc0_skip_ws(const char *p) +{ + while(p && (*p == ' ' || *p == '\t')) p++; + return p; +} + +static bool mc0_next_token(const char **p, char *out, int cap) +{ + const char *s = mc0_skip_ws(*p); + int i = 0; + if(!s || !*s || cap <= 0) return false; + while(*s && *s != ' ' && *s != '\t' && *s != '\r' && *s != '\n') { + if(i + 1 < cap) out[i++] = *s; + s++; + } + out[i] = 0; + *p = s; + return i > 0; +} + +static bool mc0_hexval(char c, int *v) +{ + if(c >= '0' && c <= '9') { *v = c - '0'; return true; } + if(c >= 'a' && c <= 'f') { *v = c - 'a' + 10; return true; } + if(c >= 'A' && c <= 'F') { *v = c - 'A' + 10; return true; } + return false; +} + +static void mc0_decode_value(const char *src, char *dst, int cap) +{ + int out = 0; + if(cap <= 0) return; + while(src && *src && out + 1 < cap) { + if(*src == '%' && src[1] && src[2]) { + int hi, lo; + if(mc0_hexval(src[1], &hi) && mc0_hexval(src[2], &lo)) { + dst[out++] = (char)((hi << 4) | lo); + src += 3; + continue; + } + } + dst[out++] = *src++; + } + dst[out] = 0; +} + +static int mc0_append_pct(char *buf, int n, int pos, const char *s) +{ + static const char hex[] = "0123456789ABCDEF"; + if(!s) s = ""; + while(*s) { + unsigned char c = (unsigned char)*s++; + bool plain = (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') || + (c >= '0' && c <= '9') || c == '-' || c == '_' || + c == '.' || c == '~' || c == '!'; + if(plain) { + if(pos + 1 < n) buf[pos] = (char)c; + pos++; + } else { + if(pos + 3 < n) { + buf[pos] = '%'; + buf[pos + 1] = hex[(c >> 4) & 0x0F]; + buf[pos + 2] = hex[c & 0x0F]; + } + pos += 3; + } + } + if(n > 0) buf[pos < n ? pos : n - 1] = 0; + return pos; +} + +static bool mc0_get_arg(const char *p, const char *key, char *out, int cap) +{ + char tok[220]; + size_t klen = strlen(key); + while(mc0_next_token(&p, tok, sizeof tok)) { + if(strncmp(tok, key, klen) == 0 && tok[klen] == '=') { + mc0_decode_value(tok + klen + 1, out, cap); + return out && cap > 0 && out[0] != 0; + } + } + if(out && cap > 0) out[0] = 0; + return false; +} + +static int mc0_ok_prefix(char *buf, int n, const char *id) +{ + return snprintf(buf, (size_t)n, "MC0 %s OK ", id && id[0] ? id : "0"); +} + +static int mc0_err(char *buf, int n, const char *id, + const char *code, bool retry, const char *msg) +{ + int pos = snprintf(buf, (size_t)n, "MC0 %s ERR code=%s retry=%d message=", + id && id[0] ? id : "0", code ? code : "internal", retry ? 1 : 0); + pos = mc0_append_pct(buf, n, pos, msg ? msg : ""); + pos = buf_appendf(buf, n, pos, "\n"); + return pos; +} + +static int mc0_count_nodes(void) +{ + int c = 0; + for(int i = 0; i < g_node_count; i++) + if(g_nodes[i].net == LZ_NET_MC) c++; + return c; +} + +static int mc0_count_threads(int *unread_out) +{ + int c = 0, unread = 0; + for(int i = 0; i < g_thread_count; i++) { + if(g_threads[i].net != LZ_NET_MC) continue; + c++; + unread += g_threads[i].unread; + } + if(unread_out) *unread_out = unread; + return c; +} + +static int mc0_hello(char *buf, int n, const char *id) +{ + int pos = mc0_ok_prefix(buf, n, id); + return buf_appendf(buf, n, pos, + "proto=0 fw=0.8-draft device=tdeck caps=identity,nodes,status,threads,send_public,send_dm,events,exit max_line=512 max_text=%d event_seq=0 nodes_rev=0 messages_rev=0\n", + LZ_TEXT_MAX); +} + +static int mc0_identity(char *buf, int n, const char *id) +{ + char addr[24]; + lz_backend_mc_addr(addr, sizeof addr); + int pos = mc0_ok_prefix(buf, n, id); + pos = buf_appendf(buf, n, pos, "enabled=%d name=", LZ_MESHCORE_ENABLED ? 1 : 0); + pos = mc0_append_pct(buf, n, pos, g_id.long_name); + return buf_appendf(buf, n, pos, + " addr=%s role=chat addr_format=meshcore-id advert_ready=1\n", + addr); +} + +static int mc0_status(char *buf, int n, const char *id) +{ + char addr[24]; + int unread = 0; + int nodes = mc0_count_nodes(); + int threads = mc0_count_threads(&unread); + lz_backend_mc_addr(addr, sizeof addr); + int pos = mc0_ok_prefix(buf, n, id); + return buf_appendf(buf, n, pos, + "proto=0 mc=%s bridge=usb mc_companion=attached mt_companion=%s addr=%s nodes=%d threads=%d unread=%d public=%d dm=%d event_seq=0 nodes_rev=0 messages_rev=0\n", + LZ_MESHCORE_ENABLED ? "on" : "disabled", + lz_mtc_active() ? "on" : "off", + addr, nodes, threads, unread, + lz_backend_mc_send_public ? 1 : 0, + lz_backend_mc_dm ? 1 : 0); +} + +static int mc0_nodes(char *buf, int n, const char *id) +{ + int count = mc0_count_nodes(); + int pos = buf_appendf(buf, n, 0, + "MC0 %s BEGIN type=nodes rev=0 count=%d more=0 cursor=end\n", + id, count); + for(int i = 0; i < g_node_count; i++) { + const lz_node_rt *nd = &g_nodes[i]; + if(nd->net != LZ_NET_MC) continue; + uint32_t seen_ms = 0; + uint32_t now_s = now_epoch(); + if(nd->last_heard && now_s >= nd->last_heard) + seen_ms = (now_s - nd->last_heard) * 1000u; + pos = buf_appendf(buf, n, pos, "MC0 %s NODE addr=%s name=", id, nd->id); + pos = mc0_append_pct(buf, n, pos, nd->name); + pos = buf_appendf(buf, n, pos, " role="); + pos = mc0_append_pct(buf, n, pos, nd->role); + pos = buf_appendf(buf, n, pos, " seen_ms=%lu snr=%.1f dm=%s\n", + (unsigned long)seen_ms, (double)nd->snr, + mc_companion_dm_target(nd) ? "ready" : "not_messageable"); + } + return buf_appendf(buf, n, pos, + "MC0 %s END type=nodes rev=0 count=%d more=0 cursor=end\n", + id, count); +} + +static int mc0_threads(char *buf, int n, const char *id) +{ + int count = mc0_count_threads(NULL); + int pos = buf_appendf(buf, n, 0, + "MC0 %s BEGIN type=threads rev=0 count=%d more=0 cursor=end\n", + id, count); + for(int oi = 0; oi < g_thread_count; oi++) { + int idx = (oi < LZ_MAX_THREADS) ? g_order[oi] : -1; + if(idx < 0 || idx >= g_thread_count) continue; + const lz_thread_rt *t = &g_threads[idx]; + if(t->net != LZ_NET_MC) continue; + pos = buf_appendf(buf, n, pos, "MC0 %s THREAD addr=%s name=", id, t->addr); + pos = mc0_append_pct(buf, n, pos, t->name); + pos = buf_appendf(buf, n, pos, " kind=%s unread=%d last=%lu text=", + t->is_channel ? "public" : "dm", t->unread, + (unsigned long)t->last_ts); + pos = mc0_append_pct(buf, n, pos, t->last_text); + pos = buf_appendf(buf, n, pos, "\n"); + } + return buf_appendf(buf, n, pos, + "MC0 %s END type=threads rev=0 count=%d more=0 cursor=end\n", + id, count); +} + +static lz_node_rt *mc0_node_by_addr(const char *addr) +{ + if(!addr || !addr[0]) return NULL; + for(int i = 0; i < g_node_count; i++) + if(g_nodes[i].net == LZ_NET_MC && strcmp(g_nodes[i].id, addr) == 0) + return &g_nodes[i]; + return NULL; +} + +static char mc0_fold_char(char c) +{ + return (c >= 'A' && c <= 'Z') ? (char)(c - 'A' + 'a') : c; +} + +static bool mc0_name_eq(const char *a, const char *b) +{ + if(!a || !b) return false; + while(*a && *b) { + if(mc0_fold_char(*a++) != mc0_fold_char(*b++)) return false; + } + return *a == 0 && *b == 0; +} + +static lz_node_rt *mc0_node_by_name_unique(const char *name, bool *ambiguous) +{ + lz_node_rt *match = NULL; + if(ambiguous) *ambiguous = false; + if(!name || !name[0]) return NULL; + for(int i = 0; i < g_node_count; i++) { + if(g_nodes[i].net != LZ_NET_MC || !mc0_name_eq(g_nodes[i].name, name)) continue; + if(match) { + if(ambiguous) *ambiguous = true; + return match; + } + match = &g_nodes[i]; + } + return match; +} + +static int mc0_send_public(char *buf, int n, const char *id, const char *args) +{ + char text[LZ_TEXT_MAX + 1]; + if(!mc0_get_arg(args, "text", text, sizeof text)) + return mc0_err(buf, n, id, "bad_request", false, "missing text"); + if((int)strlen(text) > LZ_TEXT_MAX) + return mc0_err(buf, n, id, "text_too_long", false, "text too long"); + if(!lz_svc_mc_companion_send_public(text)) + return mc0_err(buf, n, id, "send_failed", true, "public send failed"); + int pos = mc0_ok_prefix(buf, n, id); + return buf_appendf(buf, n, pos, "accepted=1 kind=public status=queued\n"); +} + +static int mc0_send_dm(char *buf, int n, const char *id, const char *args) +{ + char text[LZ_TEXT_MAX + 1], name[64], addr[32]; + if(!mc0_get_arg(args, "text", text, sizeof text)) + return mc0_err(buf, n, id, "bad_request", false, "missing text"); + if((int)strlen(text) > LZ_TEXT_MAX) + return mc0_err(buf, n, id, "text_too_long", false, "text too long"); + + bool have_name = mc0_get_arg(args, "to_name", name, sizeof name); + bool have_addr = mc0_get_arg(args, "to_addr", addr, sizeof addr); + lz_node_rt *target = NULL; + bool ambiguous = false; + if(have_addr) { + target = mc0_node_by_addr(addr); + if(!target) return mc0_err(buf, n, id, "not_found", false, "node not found"); + } else if(have_name) { + target = mc0_node_by_name_unique(name, &ambiguous); + if(ambiguous) return mc0_err(buf, n, id, "ambiguous_name", false, "name is ambiguous"); + } else { + return mc0_err(buf, n, id, "bad_request", false, "missing to_name or to_addr"); + } + if(!target) return mc0_err(buf, n, id, "not_found", false, "node not found"); + if(!mc_companion_dm_target(target)) + return mc0_err(buf, n, id, "not_messageable", false, "node cannot receive DMs"); + if(!lz_svc_mc_companion_send_dm(target->name, text)) + return mc0_err(buf, n, id, "send_failed", true, "DM send failed"); + + int pos = mc0_ok_prefix(buf, n, id); + pos = buf_appendf(buf, n, pos, "accepted=1 kind=dm to_name="); + pos = mc0_append_pct(buf, n, pos, target->name); + return buf_appendf(buf, n, pos, " status=queued\n"); +} + +int lz_svc_mc_companion_handle_line(const char *line, char *buf, int n, bool *exit_mode) +{ + char prefix[8], id[16], verb[24]; + const char *p = line; + if(exit_mode) *exit_mode = false; + if(!buf || n <= 0) return 0; + buf[0] = 0; + if(!mc0_next_token(&p, prefix, sizeof prefix) || strcmp(prefix, "MC0") != 0) + return mc0_err(buf, n, "0", "bad_request", false, "expected MC0 prefix"); + if(!mc0_next_token(&p, id, sizeof id)) + return mc0_err(buf, n, "0", "bad_request", false, "missing request id"); + if(!mc0_next_token(&p, verb, sizeof verb)) + return mc0_err(buf, n, id, "bad_request", false, "missing command"); + + if(strcmp(verb, "HELLO") == 0) return mc0_hello(buf, n, id); + if(strcmp(verb, "IDENTITY") == 0) return mc0_identity(buf, n, id); + if(strcmp(verb, "STATUS") == 0) return mc0_status(buf, n, id); + if(strcmp(verb, "NODES") == 0) return mc0_nodes(buf, n, id); + if(strcmp(verb, "THREADS") == 0) return mc0_threads(buf, n, id); + if(strcmp(verb, "SEND_PUBLIC") == 0) return mc0_send_public(buf, n, id, p); + if(strcmp(verb, "SEND_DM") == 0) return mc0_send_dm(buf, n, id, p); + if(strcmp(verb, "EVENTS") == 0) { + int pos = mc0_ok_prefix(buf, n, id); + return buf_appendf(buf, n, pos, "events=off types=none event_seq=0\n"); + } + if(strcmp(verb, "EXIT") == 0) { + if(exit_mode) *exit_mode = true; + int pos = mc0_ok_prefix(buf, n, id); + return buf_appendf(buf, n, pos, "mode=usb state=detached\n"); + } + return mc0_err(buf, n, id, "unknown_command", false, "unknown MC0 command"); +} + +int lz_svc_mc_companion_selftest(char *buf, int n) +{ + char out[900]; + bool exit_mode = false; + bool ok = true; + lz_svc_mc_companion_handle_line("MC0 1 HELLO proto=0 app=selftest", out, sizeof out, &exit_mode); + ok = ok && strstr(out, "MC0 1 OK proto=0") != NULL; + lz_svc_mc_companion_handle_line("MC0 2 STATUS", out, sizeof out, &exit_mode); + ok = ok && strstr(out, "MC0 2 OK") != NULL && strstr(out, "bridge=usb") != NULL; + lz_svc_mc_companion_handle_line("MC0 3 NODES", out, sizeof out, &exit_mode); + ok = ok && strstr(out, "MC0 3 BEGIN type=nodes") != NULL && + strstr(out, "MC0 3 END type=nodes") != NULL; + lz_svc_mc_companion_handle_line("MC0 4 EXIT", out, sizeof out, &exit_mode); + ok = ok && exit_mode && strstr(out, "state=detached") != NULL; + return snprintf(buf, (size_t)n, "MeshCore MC0 protocol selftest: %s", ok ? "PASS" : "FAIL"); +} + /* ---------- inbound events from backends ---------- */ void lz_core_on_heard(uint32_t from, float snr) From 1c0ea12991abc4ae6d43ffcc8a5f1cd54d4b5598 Mon Sep 17 00:00:00 2001 From: n30nex Date: Thu, 18 Jun 2026 18:41:06 -0400 Subject: [PATCH 14/64] Reduce MC0 USB static DRAM use --- src/mc_companion.cpp | 19 +++++++++++++++++-- src/serial_cli.cpp | 8 ++++++-- 2 files changed, 23 insertions(+), 4 deletions(-) diff --git a/src/mc_companion.cpp b/src/mc_companion.cpp index 226cd7d..14719b5 100644 --- a/src/mc_companion.cpp +++ b/src/mc_companion.cpp @@ -2,11 +2,14 @@ #include #include +#include #include "services/mesh.h" #include "ui/ui.h" +#define MC_USB_LINE_MAX 512 + static bool g_mc_usb; -static char g_mc_line[512]; +static char *g_mc_line; static uint16_t g_mc_len; extern "C" bool lz_mcc_usb_active(void) @@ -19,6 +22,12 @@ extern "C" void lz_mcc_usb_set_active(bool on) if(on) { if(lz_mtc_active()) lz_mtc_set_active(false); if(lz_mtc_ble_enabled()) lz_mtc_ble_set_enabled(false); + if(!g_mc_line) g_mc_line = (char *)malloc(MC_USB_LINE_MAX); + if(!g_mc_line) { + g_mc_usb = false; + g_mc_len = 0; + return; + } } g_mc_usb = on; g_mc_len = 0; @@ -45,6 +54,7 @@ static void mc_usb_handle_line(void) { char out[1200]; bool exit_mode = false; + if(!g_mc_line) return; g_mc_line[g_mc_len] = 0; lz_svc_mc_companion_handle_line(g_mc_line, out, sizeof out, &exit_mode); if(out[0]) Serial.print(out); @@ -57,13 +67,18 @@ static void mc_usb_handle_line(void) extern "C" void lz_mcc_usb_poll(void) { + if(!g_mc_line) { + g_mc_usb = false; + g_mc_len = 0; + return; + } while(Serial.available()) { lz_note_activity(); char c = (char)Serial.read(); if(c == '\r') continue; if(c == '\n') { mc_usb_handle_line(); - } else if(g_mc_len < sizeof(g_mc_line) - 1) { + } else if(g_mc_len < MC_USB_LINE_MAX - 1) { g_mc_line[g_mc_len++] = c; } else { static const char msg[] = diff --git a/src/serial_cli.cpp b/src/serial_cli.cpp index b048935..ccb1a9e 100644 --- a/src/serial_cli.cpp +++ b/src/serial_cli.cpp @@ -310,8 +310,12 @@ static void cmd_companion(char *args) } if(strcmp(state, "on") == 0) { lz_mcc_usb_set_active(true); - Serial.println("[ok] MeshCore MC0 USB companion mode ON"); - Serial.println(" send 'MC0 1 EXIT' to return to the text console"); + if(lz_mcc_usb_active()) { + Serial.println("[ok] MeshCore MC0 USB companion mode ON"); + Serial.println(" send 'MC0 1 EXIT' to return to the text console"); + } else { + Serial.println("[err] MeshCore MC0 USB companion could not allocate line buffer"); + } return; } if(strcmp(state, "off") == 0) { From 1d337128bbc6026595acf95f12b35800c854d199 Mon Sep 17 00:00:00 2001 From: n30nex Date: Thu, 18 Jun 2026 19:12:22 -0400 Subject: [PATCH 15/64] Add app notification feedback diagnostics --- README.md | 15 ++++--- docs/tdeck-feature-inventory.md | 6 +-- docs/tdeck-firmware-roadmap.md | 6 ++- docs/tdeck-local-app-manifest.md | 34 +++++++++------ sim/main_sim.c | 55 ++++++++++++++++++++++++ src/serial_cli.cpp | 32 ++++++++++++++ src/services/feedback.c | 73 ++++++++++++++++++++++++++++++++ src/services/mesh.h | 19 +++++++++ src/services/mesh_core.c | 26 +++++++++++- src/services/store.c | 25 +++++++++++ 10 files changed, 265 insertions(+), 26 deletions(-) create mode 100644 src/services/feedback.c diff --git a/README.md b/README.md index 722c799..c36c5d0 100644 --- a/README.md +++ b/README.md @@ -122,13 +122,14 @@ iPhone-style dark look (status bar, battery glyph, grouped settings cards). metadata plus up to two bounded foreground actions from the entry file and terminates on exit. Storage-enabled actions can increment a safe counter in the app's scoped `data/` directory, unsupported action effects fail closed, - and apps with matching permissions can use read-only `{time}` / `{battery}` - tokens in foreground text. SDK `api_version` and permission metadata are - parsed fail-closed, with rejected package diagnostics visible in Developer - Mode. Apps that request `storage` get a scoped package `data/` directory - prepared with a 64 KB launch-time quota guard, and the App Store detail screen - can clear only that app's scoped data. Script execution, richer API injection, - downloads, and updates are still TODO. + apps with matching permissions can use read-only `{time}` / `{battery}` + tokens in foreground text, and apps with `notifications` can request a + feedback-service notification through a bounded `notify:` action effect. SDK + `api_version` and permission metadata are parsed fail-closed, with rejected + package diagnostics visible in Developer Mode. Apps that request `storage` + get a scoped package `data/` directory prepared with a 64 KB launch-time quota + guard, and the App Store detail screen can clear only that app's scoped data. + Script execution, richer API injection, downloads, and updates are still TODO. - **App flash (`appfs`)** - T-Deck builds mount the FAT `appfs` partition at `/appfs` without formatting, expose it beside SD/local storage in Files, and scan `/appfs/apps` even when the SD card is absent. diff --git a/docs/tdeck-feature-inventory.md b/docs/tdeck-feature-inventory.md index 416182f..ab13c0c 100644 --- a/docs/tdeck-feature-inventory.md +++ b/docs/tdeck-feature-inventory.md @@ -82,7 +82,7 @@ Status labels: | Settings | Functional/Partial | network toggles, Wi-Fi, in-place brightness slider updates, time, system, touch calibration, Developer Mode, `settings.cfg` persistence | Add migration/versioning if the settings schema grows; hardware latency pass still needed. | | Wi-Fi setup | Functional, needs validation | async scan/connect, saved SSID/password, auto-connect | Credentials are plaintext on SD; only one saved network. | | System/battery page | Functional/Partial | live stats and battery arc | Hardware values need calibration/validation. | -| App Store | Prototype/Partial | `LZ_STORE` static catalog remains; local manifests from SD/appfs are scanned, listed as installed local apps, open a manifest detail shell, show storage quota usage, clear scoped app data on request, launch into the SDK 0.1 foreground shell, expose bounded foreground actions with scoped storage counters plus read-only `{time}`/`{battery}` tokens, reject unsupported action effects, and show rejected package diagnostics in Developer Mode | Network catalog, download, install/update, script execution/richer API injection, and runtime crash capture are still missing. | +| App Store | Prototype/Partial | `LZ_STORE` static catalog remains; local manifests from SD/appfs are scanned, listed as installed local apps, open a manifest detail shell, show storage quota usage, clear scoped app data on request, launch into the SDK 0.1 foreground shell, expose bounded foreground actions with scoped storage counters plus read-only `{time}`/`{battery}` tokens and permission-gated `notify:` feedback requests, reject unsupported action effects, and show rejected package diagnostics in Developer Mode | Network catalog, download, install/update, script execution/richer API injection, and runtime crash capture are still missing. | | Terminal | Functional/Partial | interactive UI terminal behind Developer Mode; serial CLI always available over USB | Expand diagnostics once Developer Mode grows into a full power-user surface. | | Files | Functional/Partial | read-only bounded filesystem browser rooted at mounted SD/local store or mounted FAT appfs; when both are present it starts at a Storage root picker | Add gated file actions later. | @@ -92,7 +92,7 @@ Status labels: | --- | --- | --- | --- | | Lua sandbox | Planned | Design spec section 9 | Choose Lua/eLua/minimal interpreter after memory profiling. | | App manifest | Partial | `docs/tdeck-local-app-manifest.md`; bounded manifest parser requires `id`, `name`, and relative `entry`, with optional version/author/summary/icon/hue plus SDK `api_version` and permission metadata | Extend once the runtime lifecycle and package actions are chosen. | -| App permissions | Partial | Local manifests can declare allowlisted SDK namespaces (`display`, `input`, `storage`, mesh, time, battery, notifications, Wi-Fi); unknown permission names reject the package before Home/App Store; `storage` prepares a scoped package `data/` directory with a 64 KB launch-time quota guard, SDK action counters require both `input` and `storage`, and `{time}`/`{battery}` tokens require matching `system_time`/`battery` permission before launch | Implement least-privilege API injection when the runtime is selected. | +| App permissions | Partial | Local manifests can declare allowlisted SDK namespaces (`display`, `input`, `storage`, mesh, time, battery, notifications, Wi-Fi); unknown permission names reject the package before Home/App Store; `storage` prepares a scoped package `data/` directory with a 64 KB launch-time quota guard, SDK action counters require both `input` and `storage`, `notify:` actions require `notifications`, and `{time}`/`{battery}` tokens require matching `system_time`/`battery` permission before launch | Implement least-privilege API injection when the runtime is selected. | | Local app scanner | Partial | `lz_store_scan_apps` scans `/sd/limitlezz/apps`, `/sd/apps`, `/appfs/apps`, simulator `/apps`, and simulator `/appfs/apps`; accepted apps appear in the paged Home launcher and App Store; rejected packages are exposed through Developer Mode diagnostics; simulator selftest covers appfs-only discovery, valid metadata, storage sandbox prep, quota usage, clear-data behavior, foreground launch metadata/actions, storage counter persistence, read-only time/battery token gating, unsupported action-effect blocking, oversized entry blocking, and rejected unsafe packages | Add script execution, richer app lifecycle hooks, and broader user-facing data actions once memory profiling picks a runtime. | | Network app catalog | Planned | Wi-Fi service notes; design spec | Fetch `index.json`, verify TLS/metadata, cache results. | | App download/install/update | Planned | App Store prototype only | SHA256 verify, extract, version updates, rollback failed installs. | @@ -107,7 +107,7 @@ Status labels: | Encrypted local store | Planned | README hardening section | Encrypt messages, keys, identity, and app data when password is set. | | Wi-Fi credential hardening | Functional for T-Deck, sim file-backed | T-Deck `lz_store_save_wifi/load_wifi` use ESP32 NVS with legacy `wifi.cfg` migration/removal; serial `wifi` reports `cred=nvs` without printing the password | Native simulator intentionally keeps file-backed credentials for repeatable desktop tests; broaden later if encrypted whole-store support lands. | | OTA firmware update | Planned | Partition table and design spec | Implement download, hash verify, inactive-slot write, rollback UX. | -| Feedback Manager | Planned | Design spec section 8 | Centralize LED, buzzer, keyboard/display feedback and DND. | +| Feedback Manager | Partial | A minimal service boundary records app notification requests and exposes serial `feedback status|test` plus `app notify test` diagnostics | Centralize LED, buzzer, keyboard/display feedback, DND, priority queues, and emergency behavior. | | Emergency beacon | Planned | Design spec section 12, disabled Emergency UI row | Requires Feedback Manager and dual-network messaging. | | BLE companion | Partial, needs validation | NimBLE-based Meshtastic GATT service, official UUIDs, raw `ToRadio` writes, queued `FromRadio` reads, `FromNum` notifications, UI toggle, and serial selftest/status | Validate with the official Meshtastic app over BLE before calling V0.5 complete. | | CI and release checks | Partial | `.github/workflows/firmware.yml` runs native simulator build, native protocol selftest, deterministic simulator scenario, screenshot generation, T-Deck build, size reporting, an explicit firmware/static-RAM budget gate, and artifact upload with budget metadata plus screenshots | Add protocol vectors beyond the native selftest and hardware evidence gates. | diff --git a/docs/tdeck-firmware-roadmap.md b/docs/tdeck-firmware-roadmap.md index f987e35..16fb300 100644 --- a/docs/tdeck-firmware-roadmap.md +++ b/docs/tdeck-firmware-roadmap.md @@ -274,7 +274,11 @@ Deliverables: text can use `{time}` and `{battery}` only when the manifest declares the matching `system_time` or `battery` permission; missing permissions block launch before the app shell opens. - - notification request API routed through Feedback Manager + - notification request API routed through Feedback Manager. Initial + implementation: SDK 0.1 foreground actions can use a permission-gated + `notify:` effect that records the request through a tiny feedback service + with serial `feedback status|test` and `app notify test` diagnostics. Full + LED/buzzer/DND/emergency policy remains Phase 11 work. - no direct hardware access - Add Developer Mode app diagnostics and crash/error display. Partially implemented: rejected local package folders appear in App Store under diff --git a/docs/tdeck-local-app-manifest.md b/docs/tdeck-local-app-manifest.md index fb8198e..acf2575 100644 --- a/docs/tdeck-local-app-manifest.md +++ b/docs/tdeck-local-app-manifest.md @@ -63,17 +63,23 @@ Action lines use pipe-separated fields: The first field is the button label, the second replaces the foreground status after activation, and the third replaces the body text. The optional fourth -field is a tiny SDK effect. The only supported effect is `counter:`, -which increments `/data/.count` and expands `{count}` in the -status/body. Counter keys may contain up to 19 letters, numbers, `_`, and `-` -characters only. Unknown effects and malformed counter keys are launch-blocked -instead of being ignored. +field is a tiny SDK effect. Supported effects are: + +- `counter:` increments `/data/.count` and expands + `{count}` in the status/body. Counter keys may contain up to 19 letters, + numbers, `_`, and `-` characters only. +- `notify:` requests a user-visible app notification through the + firmware feedback service. This is a lightweight service boundary only; full + LED/buzzer/DND/emergency policy remains later Feedback Manager work. + +Unknown effects and malformed effect payloads are launch-blocked instead of +being ignored. Actions require the `input` permission; display-only apps that declare actions are launch-blocked. Counter actions also require `storage` permission and stay -inside the scoped app `data/` directory and the 64 KB quota. Actions do not -execute arbitrary script and do not grant raw filesystem, radio, or hardware -access. +inside the scoped app `data/` directory and the 64 KB quota. Notification +actions require `notifications`. Actions do not execute arbitrary script and do +not grant raw filesystem, radio, or hardware access. The SDK 0.1 foreground shell also supports tiny read-only value tokens in `status:`, `body:`, `text:`, and action status/body fields: @@ -150,12 +156,12 @@ The scanner rejects packages when: The current firmware scans local app manifests and can open them in a safe foreground shell with bounded foreground actions, including a storage-scoped -counter effect and read-only `{time}` / `{battery}` value injection. Script -execution, richer sandbox API injection, richer data APIs, and network catalog -installs remain later app-platform work. Permission metadata is parsed and -displayed now so packages can fail closed before richer runtime APIs are added, -and apps that declare `storage` get a scoped `data/` directory prepared under -their own package. +counter effect, a permission-gated `notify:` feedback request, and read-only +`{time}` / `{battery}` value injection. Script execution, richer sandbox API +injection, richer data APIs, and network catalog installs remain later +app-platform work. Permission metadata is parsed and displayed now so packages +can fail closed before richer runtime APIs are added, and apps that declare +`storage` get a scoped `data/` directory prepared under their own package. Storage-enabled local apps have a 64 KB `data/` quota in this early shell. The App Store detail and foreground shell show current usage, and over-quota apps diff --git a/sim/main_sim.c b/sim/main_sim.c index 9c54b9b..1e48e7a 100644 --- a/sim/main_sim.c +++ b/sim/main_sim.c @@ -985,16 +985,47 @@ static int codec_selftest(void) } FILE *nbe = fopen("lzdata_apptokens/apps/nobattery/main.lua", "wb"); if(nbe) { fputs("-- body: Needs {battery}\n", nbe); fclose(nbe); } + sim_mkdirs("lzdata_apptokens/apps/notifier"); + FILE *nfm = fopen("lzdata_apptokens/apps/notifier/manifest.json", "wb"); + if(nfm) { + fputs("{\"id\":\"notify.local\",\"name\":\"Notifier\",\"entry\":\"main.lua\"," + "\"permissions\":[\"display\",\"input\",\"notifications\"]}", nfm); + fclose(nfm); + } + FILE *nfe = fopen("lzdata_apptokens/apps/notifier/main.lua", "wb"); + if(nfe) { + fputs("-- body: Permissioned app notification\n" + "-- action: Alert | Notification sent | Feedback service recorded it | notify:Field alert ready\n", + nfe); + fclose(nfe); + } + sim_mkdirs("lzdata_apptokens/apps/nonotify"); + FILE *nnm = fopen("lzdata_apptokens/apps/nonotify/manifest.json", "wb"); + if(nnm) { + fputs("{\"id\":\"nonotify.local\",\"name\":\"No Notify\",\"entry\":\"main.lua\"," + "\"permissions\":[\"display\",\"input\"]}", nnm); + fclose(nnm); + } + FILE *nne = fopen("lzdata_apptokens/apps/nonotify/main.lua", "wb"); + if(nne) { + fputs("-- body: Missing notification permission\n" + "-- action: Alert | Notification sent | Should not route | notify:Denied alert\n", + nne); + fclose(nne); + } lz_svc_init("lzdata_apptokens", false); lz_svc_set_time(1781274180); lz_local_app_t apps[LZ_MAX_LOCAL_APPS]; int an = lz_svc_scan_apps(apps, LZ_MAX_LOCAL_APPS); lz_local_app_t *status = NULL, *notime = NULL, *nobattery = NULL; + lz_local_app_t *notify = NULL, *nonotify = NULL; for(int i = 0; i < an; i++) { if(strcmp(apps[i].id, "status.local") == 0) status = &apps[i]; if(strcmp(apps[i].id, "notime.local") == 0) notime = &apps[i]; if(strcmp(apps[i].id, "nobattery.local") == 0) nobattery = &apps[i]; + if(strcmp(apps[i].id, "notify.local") == 0) notify = &apps[i]; + if(strcmp(apps[i].id, "nonotify.local") == 0) nonotify = &apps[i]; } lz_local_app_session_t run; bool run_ok = status && lz_svc_start_local_app(status, &run); @@ -1014,6 +1045,30 @@ static int codec_selftest(void) CHECK(!denied_battery_ok && nobattery && strcmp(denied_battery.error, "battery permission missing") == 0, "local app SDK battery token requires battery permission"); + lz_feedback_status_t fb_before, fb_after; + lz_svc_feedback_status(&fb_before); + lz_local_app_session_t notify_run; + bool notify_ok = notify && lz_svc_start_local_app(notify, ¬ify_run); + bool notify_action_ok = notify_ok && lz_svc_local_app_action(¬ify_run, 0); + lz_svc_feedback_status(&fb_after); + CHECK(notify_action_ok && + fb_after.request_count == fb_before.request_count + 1 && + strcmp(fb_after.last_source, "Notifier") == 0 && + strcmp(fb_after.last_title, "Alert") == 0 && + strcmp(fb_after.last_body, "Field alert ready") == 0, + "local app notification action routes through feedback service"); + lz_local_app_session_t nonotify_run; + bool nonotify_ok = nonotify && lz_svc_start_local_app(nonotify, &nonotify_run); + CHECK(!nonotify_ok && nonotify && + strcmp(nonotify_run.error, "notifications permission missing") == 0, + "local app notification action requires notifications permission"); + char fb_diag[180], fb_test[120]; + int fb_diag_n = lz_svc_feedback_diag(fb_diag, sizeof fb_diag); + int fb_test_ok = lz_svc_feedback_selftest(fb_test, sizeof fb_test); + CHECK(fb_diag_n > 0 && strstr(fb_diag, "feedback: ready") != NULL, + "feedback diagnostics report service status"); + CHECK(fb_test_ok == 1 && strstr(fb_test, "PASS") != NULL, + "feedback selftest records a notification request"); lz_store_init(NULL); sim_reset_dir("lzdata_apptokens"); } diff --git a/src/serial_cli.cpp b/src/serial_cli.cpp index b46d44c..809745c 100644 --- a/src/serial_cli.cpp +++ b/src/serial_cli.cpp @@ -63,6 +63,8 @@ static void cmd_help(void) " companion on|off USB acts as a Meshtastic-app companion radio\n" " companion ble on|off|test BLE Meshtastic-app companion advertising\n" " companion test loopback-verify the companion protocol\n" + " feedback status|test feedback/app-notification diagnostics\n" + " app notify test request a test app notification\n" " touch [cal|debug|S X Y] touch: 'cal' runs on-screen calibration, 'debug' logs taps, 'S X Y' sets transform\n" " dm status show pending sent-DM delivery state\n" " dm test|req |send PKI DM: self-test / request a node's key / send a DM\n" @@ -395,6 +397,34 @@ static void cmd_wifi(char *args) Serial.println(nn); } +static void cmd_feedback(char *args) +{ + if(args && strcmp(args, "test") == 0) { + char b[120]; + lz_svc_feedback_selftest(b, sizeof b); + Serial.println(b); + return; + } + if(args && args[0] && strcmp(args, "status") != 0) { + Serial.println("usage: feedback status|test"); + return; + } + char b[220]; + lz_svc_feedback_diag(b, sizeof b); + Serial.print(b); +} + +static void cmd_app(char *args) +{ + if(args && strcmp(args, "notify test") == 0) { + lz_svc_feedback_notify("serial-app", "App notification", "SDK notify smoke"); + Serial.println("[ok] app notification requested"); + cmd_feedback((char *)"status"); + return; + } + Serial.println("usage: app notify test"); +} + static void cmd_sys(void) { lz_sysinfo_t si; @@ -429,6 +459,8 @@ static void dispatch(char *line) else if(!strcmp(line, "rf")) cmd_rf(args); else if(!strcmp(line, "mc")) cmd_mc(args); else if(!strcmp(line, "companion")) cmd_companion(args); + else if(!strcmp(line, "feedback")) cmd_feedback(args); + else if(!strcmp(line, "app")) cmd_app(args); else if(!strcmp(line, "touch")) cmd_touch(args); else if(!strcmp(line, "dm")) cmd_dm(args); else if(!strcmp(line, "rxlog")) { diff --git a/src/services/feedback.c b/src/services/feedback.c new file mode 100644 index 0000000..a6889a0 --- /dev/null +++ b/src/services/feedback.c @@ -0,0 +1,73 @@ +#include "mesh.h" +#include +#include + +extern uint32_t lz_tick_ms(void); + +static lz_feedback_status_t g_feedback; + +static void bounded_copy(char *out, size_t cap, const char *src) +{ + if(!out || cap == 0) return; + if(!src) src = ""; + size_t j = 0; + while(*src && j + 1 < cap) { + char c = *src++; + if(c == '\r' || c == '\n' || c < 32) c = ' '; + out[j++] = c; + } + while(j > 0 && out[j - 1] == ' ') j--; + out[j] = 0; +} + +bool lz_svc_feedback_notify(const char *source, const char *title, const char *body) +{ + if(!title || !title[0]) title = "Notification"; + if(!body || !body[0]) body = "App requested attention"; + if(!source || !source[0]) source = "system"; + + g_feedback.request_count++; + g_feedback.last_ms = lz_tick_ms(); + bounded_copy(g_feedback.last_source, sizeof g_feedback.last_source, source); + bounded_copy(g_feedback.last_title, sizeof g_feedback.last_title, title); + bounded_copy(g_feedback.last_body, sizeof g_feedback.last_body, body); + return true; +} + +void lz_svc_feedback_status(lz_feedback_status_t *out) +{ + if(!out) return; + *out = g_feedback; +} + +int lz_svc_feedback_diag(char *buf, int n) +{ + if(!buf || n <= 0) return 0; + if(g_feedback.request_count == 0) { + snprintf(buf, (size_t)n, "feedback: ready requests=0\n"); + } else { + snprintf(buf, (size_t)n, + "feedback: ready requests=%lu last_ms=%lu source=%s title=\"%s\" body=\"%s\"\n", + (unsigned long)g_feedback.request_count, + (unsigned long)g_feedback.last_ms, + g_feedback.last_source[0] ? g_feedback.last_source : "-", + g_feedback.last_title, + g_feedback.last_body); + } + return (int)strlen(buf); +} + +int lz_svc_feedback_selftest(char *buf, int n) +{ + uint32_t before = g_feedback.request_count; + bool ok = lz_svc_feedback_notify("selftest", "Feedback test", "notification route ok"); + bool advanced = g_feedback.request_count == before + 1; + bool kept = strcmp(g_feedback.last_source, "selftest") == 0 && + strcmp(g_feedback.last_title, "Feedback test") == 0; + if(buf && n > 0) { + snprintf(buf, (size_t)n, "Feedback selftest: %s requests=%lu", + (ok && advanced && kept) ? "PASS" : "FAIL", + (unsigned long)g_feedback.request_count); + } + return ok && advanced && kept ? 1 : 0; +} diff --git a/src/services/mesh.h b/src/services/mesh.h index aacfc8d..b130673 100644 --- a/src/services/mesh.h +++ b/src/services/mesh.h @@ -34,6 +34,9 @@ extern "C" { #define LZ_LOCAL_APP_ACTION_MAX 2 #define LZ_LOCAL_APP_ACTION_EFFECT_MAX 32 #define LZ_LOCAL_APP_ACTION_BODY_MAX 192 +#define LZ_FEEDBACK_SOURCE_MAX 24 +#define LZ_FEEDBACK_TITLE_MAX 32 +#define LZ_FEEDBACK_BODY_MAX 96 #define LZ_APP_PERM_DISPLAY 0x0001u #define LZ_APP_PERM_INPUT 0x0002u @@ -235,6 +238,22 @@ bool lz_svc_clear_app_data(const lz_local_app_t *app, char *err, int err_cap); bool lz_svc_start_local_app(const lz_local_app_t *app, lz_local_app_session_t *out); bool lz_svc_local_app_action(lz_local_app_session_t *session, int idx); +/* ---- feedback / app notifications ---- + * Minimal V0.95 service boundary: local apps can request user-visible feedback + * without owning LED, buzzer, keyboard backlight, or future DND policy. */ +typedef struct { + uint32_t request_count; + uint32_t last_ms; + char last_source[LZ_FEEDBACK_SOURCE_MAX]; + char last_title[LZ_FEEDBACK_TITLE_MAX]; + char last_body[LZ_FEEDBACK_BODY_MAX]; +} lz_feedback_status_t; + +bool lz_svc_feedback_notify(const char *source, const char *title, const char *body); +void lz_svc_feedback_status(lz_feedback_status_t *out); +int lz_svc_feedback_diag(char *buf, int n); +int lz_svc_feedback_selftest(char *buf, int n); + /* ---- nodes ---- */ int lz_svc_nodes(const lz_node_rt **out); /* all heard nodes */ lz_node_rt *lz_svc_node_by_name(const char *name); diff --git a/src/services/mesh_core.c b/src/services/mesh_core.c index 665f367..1f86819 100644 --- a/src/services/mesh_core.c +++ b/src/services/mesh_core.c @@ -396,10 +396,34 @@ bool lz_svc_start_local_app(const lz_local_app_t *app, lz_local_app_session_t *o return local_app_expand_session(out); } +static bool local_app_notify_effect(const char *effect, char *body, size_t cap) +{ + if(!effect || !body || cap == 0) return false; + if(strncmp(effect, "notify:", 7) != 0 || !effect[7]) return false; + const char *src = effect + 7; + while(*src == ' ' || *src == '\t') src++; + size_t j = 0; + while(*src && j + 1 < cap) { + char c = *src++; + if(c == '\r' || c == '\n' || c < 32) c = ' '; + body[j++] = c; + } + while(j > 0 && body[j - 1] == ' ') j--; + body[j] = 0; + return body[0] != 0; +} + bool lz_svc_local_app_action(lz_local_app_session_t *session, int idx) { if(!lz_store_local_app_action(session, idx)) return false; - return local_app_expand_session(session); + bool ok = local_app_expand_session(session); + if(ok && idx >= 0 && idx < session->action_count) { + char note[LZ_FEEDBACK_BODY_MAX]; + if(local_app_notify_effect(session->actions[idx].effect, note, sizeof note)) { + lz_svc_feedback_notify(session->title, session->actions[idx].label, note); + } + } + return ok; } const char *lz_fmt_ago(uint32_t ts, char *buf, size_t n) diff --git a/src/services/store.c b/src/services/store.c index 0d73515..8cb1a0f 100644 --- a/src/services/store.c +++ b/src/services/store.c @@ -478,6 +478,14 @@ static bool effect_counter_key(const char *effect, char *key, size_t cap) return j > 0; } +static bool effect_notify_text(const char *effect, char *text, size_t cap) +{ + if(!effect || !text || cap == 0) return false; + if(strncmp(effect, "notify:", 7) != 0 || !effect[7]) return false; + clean_line_copy(text, cap, effect + 7); + return text[0] != 0; +} + static bool action_needs_storage(const lz_local_app_session_t *s) { if(!s) return false; @@ -487,10 +495,20 @@ static bool action_needs_storage(const lz_local_app_session_t *s) return false; } +static bool action_needs_notifications(const lz_local_app_session_t *s) +{ + if(!s) return false; + char text[LZ_FEEDBACK_BODY_MAX]; + for(int i = 0; i < s->action_count; i++) + if(effect_notify_text(s->actions[i].effect, text, sizeof text)) return true; + return false; +} + static bool action_effect_error(const lz_local_app_session_t *s, char *err, size_t cap) { if(!s) return false; char key[20]; + char text[LZ_FEEDBACK_BODY_MAX]; for(int i = 0; i < s->action_count; i++) { const char *effect = s->actions[i].effect; if(!effect[0]) continue; @@ -499,6 +517,11 @@ static bool action_effect_error(const lz_local_app_session_t *s, char *err, size if(err && cap > 0) snprintf(err, cap, "bad action effect"); return true; } + if(strncmp(effect, "notify:", 7) == 0) { + if(effect_notify_text(effect, text, sizeof text)) continue; + if(err && cap > 0) snprintf(err, cap, "bad action effect"); + return true; + } if(err && cap > 0) snprintf(err, cap, "unsupported action effect"); return true; } @@ -887,6 +910,8 @@ bool lz_store_start_local_app(const lz_local_app_t *app, lz_local_app_session_t return app_session_fail(out, app, effect_err); if(action_needs_storage(out) && !out->storage_ready) return app_session_fail(out, app, "storage permission missing"); + if(action_needs_notifications(out) && (out->permissions & LZ_APP_PERM_NOTIFICATIONS) == 0) + return app_session_fail(out, app, "notifications permission missing"); return true; } From d9f880d85411958c3ecfd1019f103da86c52b491 Mon Sep 17 00:00:00 2001 From: n30nex Date: Thu, 18 Jun 2026 21:35:26 -0400 Subject: [PATCH 16/64] Add app catalog schema diagnostics --- README.md | 12 +- docs/tdeck-app-catalog-schema.md | 65 ++++++ docs/tdeck-feature-inventory.md | 2 +- docs/tdeck-firmware-roadmap.md | 2 +- sim/main_sim.c | 64 +++++- src/serial_cli.cpp | 20 ++ src/services/mesh.h | 13 ++ src/services/mesh_core.c | 18 ++ src/services/store.c | 367 +++++++++++++++++++++++++++++++ 9 files changed, 553 insertions(+), 10 deletions(-) create mode 100644 docs/tdeck-app-catalog-schema.md diff --git a/README.md b/README.md index 722c799..95da254 100644 --- a/README.md +++ b/README.md @@ -127,8 +127,10 @@ iPhone-style dark look (status bar, battery glyph, grouped settings cards). parsed fail-closed, with rejected package diagnostics visible in Developer Mode. Apps that request `storage` get a scoped package `data/` directory prepared with a 64 KB launch-time quota guard, and the App Store detail screen - can clear only that app's scoped data. Script execution, richer API injection, - downloads, and updates are still TODO. + can clear only that app's scoped data. The future network catalog now has a + bounded `index.json` schema validator and serial `app catalog status|test` + diagnostics. Script execution, richer API injection, catalog fetch, downloads, + and updates are still TODO. - **App flash (`appfs`)** - T-Deck builds mount the FAT `appfs` partition at `/appfs` without formatting, expose it beside SD/local storage in Files, and scan `/appfs/apps` even when the SD card is absent. @@ -144,6 +146,7 @@ iPhone-style dark look (status bar, battery glyph, grouped settings cards). - [`docs/tdeck-feature-inventory.md`](docs/tdeck-feature-inventory.md) - feature-by-feature implementation inventory. - [`docs/tdeck-firmware-roadmap.md`](docs/tdeck-firmware-roadmap.md) - roadmap to a complete T-Deck firmware. - [`docs/tdeck-hardware-dogfood-checklist.md`](docs/tdeck-hardware-dogfood-checklist.md) - stock-device hardware proof checklist. +- [`docs/tdeck-app-catalog-schema.md`](docs/tdeck-app-catalog-schema.md) - first Network App Store catalog schema. ![screens](docs/screens.png) @@ -300,8 +303,9 @@ for local apps and read-only inspection when present. clears scoped app data on request, opens a manifest detail shell, and launches local apps into the SDK 0.1 foreground shell with bounded app-provided actions and scoped storage counters plus read-only `{time}` / `{battery}` tokens; - unsupported action effects launch-block instead of being ignored; the static - catalog remains a prototype (GET -> "..." -> OPEN). + unsupported action effects launch-block instead of being ignored; network + catalog schema validation exists, while fetch/download/install remains ahead; + the static catalog remains a prototype (GET -> "..." -> OPEN). - **Contacts / detail** — unified directory with network dots; detail page with Message (jumps into the bound conversation) and spec table. - **Settings** — airtime scheduler bar that rebalances live when the diff --git a/docs/tdeck-app-catalog-schema.md b/docs/tdeck-app-catalog-schema.md new file mode 100644 index 0000000..af15dd8 --- /dev/null +++ b/docs/tdeck-app-catalog-schema.md @@ -0,0 +1,65 @@ +# T-Deck App Catalog Schema + +This is the first Network App Store increment. It defines and validates the +future `index.json` shape, but it does not fetch, download, install, update, or +uninstall packages yet. + +Catalog indexes are expected at: + +- `/sd/limitlezz/catalog/index.json` +- `/appfs/catalog/index.json` + +The index must fit in `4096` bytes and use this top-level shape: + +```json +{ + "schema": "limitlezz.app_catalog.v1", + "updated": "2026-06-18T00:00:00Z", + "apps": [] +} +``` + +Each app entry is bounded and fail-closed: + +```json +{ + "id": "weather.mesh", + "name": "Weather Mesh", + "version": "0.1.0", + "author": "Limitless", + "description": "Local weather reports", + "icon": "weather", + "hue": 48, + "api_version": "0.1", + "compatibility": "tdeck", + "permissions": ["display", "network_wifi"], + "download_url": "https://apps.example.invalid/weather.mesh.zip", + "sha256": "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef", + "size": 32768, + "screenshots": ["https://apps.example.invalid/weather.bmp"] +} +``` + +Validation rules: + +- `id` uses the same safe token rules as local app manifests. +- `api_version` must be supported by the local SDK compatibility gate. +- `permissions` must use the existing allowlist. +- `download_url` and optional `screenshots` must be `http://` or `https://` + URLs without whitespace or control characters. +- `sha256` must be exactly 64 hex characters. +- `size` must be nonzero and no more than `2 MB` for this first package path. +- `hue`, if present, must be `-1` or `0..359`. +- The catalog can list up to `24` apps. + +Serial diagnostics: + +```text +app catalog status +app catalog test +``` + +`app catalog status` validates a cached index if one exists and otherwise +reports that no cached catalog is present. `app catalog test` runs a built-in +valid/invalid schema selftest so hardware smoke can prove the parser without +requiring Wi-Fi or SD setup. diff --git a/docs/tdeck-feature-inventory.md b/docs/tdeck-feature-inventory.md index 416182f..06bf5fa 100644 --- a/docs/tdeck-feature-inventory.md +++ b/docs/tdeck-feature-inventory.md @@ -94,7 +94,7 @@ Status labels: | App manifest | Partial | `docs/tdeck-local-app-manifest.md`; bounded manifest parser requires `id`, `name`, and relative `entry`, with optional version/author/summary/icon/hue plus SDK `api_version` and permission metadata | Extend once the runtime lifecycle and package actions are chosen. | | App permissions | Partial | Local manifests can declare allowlisted SDK namespaces (`display`, `input`, `storage`, mesh, time, battery, notifications, Wi-Fi); unknown permission names reject the package before Home/App Store; `storage` prepares a scoped package `data/` directory with a 64 KB launch-time quota guard, SDK action counters require both `input` and `storage`, and `{time}`/`{battery}` tokens require matching `system_time`/`battery` permission before launch | Implement least-privilege API injection when the runtime is selected. | | Local app scanner | Partial | `lz_store_scan_apps` scans `/sd/limitlezz/apps`, `/sd/apps`, `/appfs/apps`, simulator `/apps`, and simulator `/appfs/apps`; accepted apps appear in the paged Home launcher and App Store; rejected packages are exposed through Developer Mode diagnostics; simulator selftest covers appfs-only discovery, valid metadata, storage sandbox prep, quota usage, clear-data behavior, foreground launch metadata/actions, storage counter persistence, read-only time/battery token gating, unsupported action-effect blocking, oversized entry blocking, and rejected unsafe packages | Add script execution, richer app lifecycle hooks, and broader user-facing data actions once memory profiling picks a runtime. | -| Network app catalog | Planned | Wi-Fi service notes; design spec | Fetch `index.json`, verify TLS/metadata, cache results. | +| Network app catalog | Partial | `docs/tdeck-app-catalog-schema.md`; bounded `limitlezz.app_catalog.v1` validator plus serial `app catalog status\|test` diagnostics | Fetch `index.json` over Wi-Fi, verify TLS/source metadata, cache results, and feed validated entries into App Store state. | | App download/install/update | Planned | App Store prototype only | SHA256 verify, extract, version updates, rollback failed installs. | | Optional map app | Planned | Store data includes maps; maintainer notes prefer maps as optional | Keep maps out of the base firmware. | | APRS/weather/BBS/scope/game apps | Planned/Prototype catalog entries | Static `LZ_STORE` rows | Implement as sandboxed apps once runtime exists. | diff --git a/docs/tdeck-firmware-roadmap.md b/docs/tdeck-firmware-roadmap.md index f987e35..f9d5f52 100644 --- a/docs/tdeck-firmware-roadmap.md +++ b/docs/tdeck-firmware-roadmap.md @@ -322,7 +322,7 @@ Goal: let users install and update apps from a repository. Deliverables: -- Define catalog `index.json` schema: app id, name, version, author, description, icon id/color, permissions, download URL, SHA256, size, compatibility, screenshots if desired. +- Define catalog `index.json` schema: app id, name, version, author, description, icon id/color, permissions, download URL, SHA256, size, compatibility, screenshots if desired. Implemented: a bounded `limitlezz.app_catalog.v1` validator rejects unsafe IDs, unsupported permissions/SDK versions, non-HTTP package URLs, bad SHA256 values, oversize packages, and malformed optional screenshots; serial `app catalog status|test` exposes the result without requiring Wi-Fi. - Fetch catalog over Wi-Fi. - Cache catalog for offline browsing. - Download app zip/package. diff --git a/sim/main_sim.c b/sim/main_sim.c index 9c54b9b..bf38584 100644 --- a/sim/main_sim.c +++ b/sim/main_sim.c @@ -946,7 +946,63 @@ static int codec_selftest(void) sim_reset_dir("lzdata_appscan"); } - /* 10. service-level SDK token injection: dynamic read-only values are + /* 11. network app catalog schema: validate bounded future install indexes + * before fetch/download/install code trusts them. */ + { + static const char valid_catalog[] = + "{\"schema\":\"limitlezz.app_catalog.v1\",\"updated\":\"2026-06-18T00:00:00Z\"," + "\"apps\":[" + "{\"id\":\"weather.mesh\",\"name\":\"Weather Mesh\",\"version\":\"0.1.0\"," + "\"author\":\"Limitless\",\"description\":\"Local weather reports\"," + "\"icon\":\"weather\",\"hue\":48,\"api_version\":\"0.1\"," + "\"compatibility\":\"tdeck\",\"permissions\":[\"display\",\"network_wifi\"]," + "\"download_url\":\"https://apps.example.invalid/weather.mesh.zip\"," + "\"sha256\":\"0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef\"," + "\"size\":32768,\"screenshots\":[\"https://apps.example.invalid/weather.bmp\"]}," + "{\"id\":\"notes.local\",\"name\":\"Field Notes\",\"version\":\"0.1.0\"," + "\"author\":\"Limitless\",\"description\":\"Simple local notes\"," + "\"icon\":\"notes\",\"api_version\":\"0.1\",\"compatibility\":\"tdeck\"," + "\"permissions\":[\"display\",\"input\",\"storage\"]," + "\"download_url\":\"https://apps.example.invalid/notes.local.zip\"," + "\"sha256\":\"abcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcd\"," + "\"size\":49152}]}"; + static const char bad_catalog[] = + "{\"schema\":\"limitlezz.app_catalog.v1\",\"apps\":[" + "{\"id\":\"weather.mesh\",\"name\":\"Weather Mesh\",\"version\":\"0.1.0\"," + "\"author\":\"Limitless\",\"description\":\"Bad catalog\",\"icon\":\"weather\"," + "\"api_version\":\"0.1\",\"compatibility\":\"tdeck\"," + "\"permissions\":[\"display\",\"raw_radio\"]," + "\"download_url\":\"file:///sd/apps/weather.zip\"," + "\"sha256\":\"bad\",\"size\":0}]}"; + + lz_app_catalog_report_t report; + CHECK(lz_svc_validate_app_catalog_json(valid_catalog, &report) && + report.ok && report.app_count == 2 && report.rejected_count == 0, + "app catalog schema accepts valid bounded index"); + CHECK(!lz_svc_validate_app_catalog_json(bad_catalog, &report) && + !report.ok && report.rejected_count == 1 && + strcmp(report.first_error, "bad download_url") == 0, + "app catalog schema rejects unsafe download URL"); + char self[160]; + lz_svc_app_catalog_selftest(self, sizeof self); + CHECK(strstr(self, "PASS") != NULL, + "app catalog selftest covers valid and invalid indexes"); + + extern void lz_store_init(const char *datadir); + sim_reset_dir("lzdata_catalog"); + sim_mkdirs("lzdata_catalog/catalog"); + FILE *cf = fopen("lzdata_catalog/catalog/index.json", "wb"); + if(cf) { fputs(valid_catalog, cf); fclose(cf); } + lz_store_init("lzdata_catalog"); + char diag[160]; + lz_svc_app_catalog_diag(diag, sizeof diag); + CHECK(strstr(diag, "ready apps=2") != NULL, + "app catalog diagnostics report cached index"); + lz_store_init(NULL); + sim_reset_dir("lzdata_catalog"); + } + + /* 12. service-level SDK token injection: dynamic read-only values are * expanded only when the matching permissions are declared. */ { extern void lz_store_init(const char *datadir); @@ -1018,7 +1074,7 @@ static int codec_selftest(void) sim_reset_dir("lzdata_apptokens"); } - /* 11. appfs root support: apps can be discovered from appfs even when + /* 13. appfs root support: apps can be discovered from appfs even when * SD-backed persistence is absent, and Files can expose both roots. */ { extern void lz_store_init(const char *datadir); @@ -1048,7 +1104,7 @@ static int codec_selftest(void) sim_reset_dir("lzdata_appfsroot"); } - /* 11. MeshCore Public-channel GRP_TXT: decode a known reference vector, + /* 14. MeshCore Public-channel GRP_TXT: decode a known reference vector, * reject a wrong key (MAC), and round-trip an encode. Vector generated * against the documented scheme (AES-128-ECB + HMAC-SHA256 trunc-2). */ { @@ -1082,7 +1138,7 @@ static int codec_selftest(void) strcmp(rm.text, "hi there") == 0, "MeshCore GRP_TXT round-trip fields"); } - /* 13. MeshCore DM (TXT_MSG): ECDH derive (vs orlp/standard reference) then a + /* 15. MeshCore DM (TXT_MSG): ECDH derive (vs orlp/standard reference) then a * full encode->parse->decode round-trip + ACK match + MAC tamper check. */ { uint8_t seedA[32], pubA[32], pubB[32], ref[32]; diff --git a/src/serial_cli.cpp b/src/serial_cli.cpp index b46d44c..d9fc788 100644 --- a/src/serial_cli.cpp +++ b/src/serial_cli.cpp @@ -63,6 +63,7 @@ static void cmd_help(void) " companion on|off USB acts as a Meshtastic-app companion radio\n" " companion ble on|off|test BLE Meshtastic-app companion advertising\n" " companion test loopback-verify the companion protocol\n" + " app catalog status|test app catalog schema diagnostics\n" " touch [cal|debug|S X Y] touch: 'cal' runs on-screen calibration, 'debug' logs taps, 'S X Y' sets transform\n" " dm status show pending sent-DM delivery state\n" " dm test|req |send PKI DM: self-test / request a node's key / send a DM\n" @@ -323,6 +324,24 @@ static void cmd_companion(char *args) if(lz_mtc_ble_status) { char b[180]; lz_mtc_ble_status(b, sizeof b); Serial.println(b); } } +static void cmd_app(char *args) +{ + if(args && strcmp(args, "catalog test") == 0) { + char b[180]; + lz_svc_app_catalog_selftest(b, sizeof b); + Serial.print(b); + return; + } + if(!args || !args[0] || strcmp(args, "catalog") == 0 || + strcmp(args, "catalog status") == 0) { + char b[180]; + lz_svc_app_catalog_diag(b, sizeof b); + Serial.print(b); + return; + } + Serial.println("usage: app catalog status | app catalog test"); +} + static void cmd_nodes(void) { const lz_node_rt *nodes; @@ -429,6 +448,7 @@ static void dispatch(char *line) else if(!strcmp(line, "rf")) cmd_rf(args); else if(!strcmp(line, "mc")) cmd_mc(args); else if(!strcmp(line, "companion")) cmd_companion(args); + else if(!strcmp(line, "app")) cmd_app(args); else if(!strcmp(line, "touch")) cmd_touch(args); else if(!strcmp(line, "dm")) cmd_dm(args); else if(!strcmp(line, "rxlog")) { diff --git a/src/services/mesh.h b/src/services/mesh.h index aacfc8d..a6f66c8 100644 --- a/src/services/mesh.h +++ b/src/services/mesh.h @@ -34,6 +34,8 @@ extern "C" { #define LZ_LOCAL_APP_ACTION_MAX 2 #define LZ_LOCAL_APP_ACTION_EFFECT_MAX 32 #define LZ_LOCAL_APP_ACTION_BODY_MAX 192 +#define LZ_APP_CATALOG_JSON_MAX 4096u +#define LZ_APP_CATALOG_MAX_APPS 24 #define LZ_APP_PERM_DISPLAY 0x0001u #define LZ_APP_PERM_INPUT 0x0002u @@ -195,6 +197,14 @@ typedef struct { char path[112]; /* package directory */ } lz_local_app_issue_t; +typedef struct { + bool ok; + int app_count; + int rejected_count; + char first_id[24]; + char first_error[64]; +} lz_app_catalog_report_t; + typedef struct { char label[24]; /* app-provided foreground control label */ char status[48]; /* bounded status shown after activation */ @@ -234,6 +244,9 @@ bool lz_svc_app_data_usage(const lz_local_app_t *app, uint32_t *used, uint32_t * bool lz_svc_clear_app_data(const lz_local_app_t *app, char *err, int err_cap); bool lz_svc_start_local_app(const lz_local_app_t *app, lz_local_app_session_t *out); bool lz_svc_local_app_action(lz_local_app_session_t *session, int idx); +bool lz_svc_validate_app_catalog_json(const char *json, lz_app_catalog_report_t *out); +int lz_svc_app_catalog_diag(char *buf, int n); +int lz_svc_app_catalog_selftest(char *buf, int n); /* ---- nodes ---- */ int lz_svc_nodes(const lz_node_rt **out); /* all heard nodes */ diff --git a/src/services/mesh_core.c b/src/services/mesh_core.c index 665f367..1517e5b 100644 --- a/src/services/mesh_core.c +++ b/src/services/mesh_core.c @@ -25,6 +25,9 @@ bool lz_store_app_data_usage(const lz_local_app_t *app, uint32_t *used, uint32_t bool lz_store_clear_app_data(const lz_local_app_t *app, char *err, int err_cap); bool lz_store_start_local_app(const lz_local_app_t *app, lz_local_app_session_t *out); bool lz_store_local_app_action(lz_local_app_session_t *session, int idx); +bool lz_store_validate_app_catalog_json(const char *json, lz_app_catalog_report_t *out); +int lz_store_app_catalog_diag(char *buf, int n); +int lz_store_app_catalog_selftest(char *buf, int n); void lz_store_append(const char *addr, const lz_msg_rt *m); int lz_store_load_tail(const char *addr, lz_msg_rt *ring, int cap); bool lz_store_find_delivery(const char *addr, uint32_t pkt_id, lz_msg_rt *out); @@ -402,6 +405,21 @@ bool lz_svc_local_app_action(lz_local_app_session_t *session, int idx) return local_app_expand_session(session); } +bool lz_svc_validate_app_catalog_json(const char *json, lz_app_catalog_report_t *out) +{ + return lz_store_validate_app_catalog_json(json, out); +} + +int lz_svc_app_catalog_diag(char *buf, int n) +{ + return lz_store_app_catalog_diag(buf, n); +} + +int lz_svc_app_catalog_selftest(char *buf, int n) +{ + return lz_store_app_catalog_selftest(buf, n); +} + const char *lz_fmt_ago(uint32_t ts, char *buf, size_t n) { if(ts == 0) { snprintf(buf, n, "-"); return buf; } diff --git a/src/services/store.c b/src/services/store.c index 0d73515..6f483ba 100644 --- a/src/services/store.c +++ b/src/services/store.c @@ -292,6 +292,32 @@ static bool json_get_string(const char *json, const char *key, char *out, size_t return j > 0; } +static bool json_get_string_bounded(const char *json, const char *key, char *out, size_t n) +{ + if(!out || n == 0) return false; + const char *p = json_value_for(json, key); + if(!p || *p != '"') return false; + p++; + size_t j = 0; + bool too_long = false; + while(*p && *p != '"') { + char c = *p++; + if(c == '\\' && *p) { + char e = *p++; + if(e == 'n') c = '\n'; + else if(e == 'r') c = '\r'; + else if(e == 't') c = '\t'; + else c = e; + } + if(c < 32) continue; + if(j + 1 < n) out[j++] = c; + else too_long = true; + } + if(*p != '"' || too_long) return false; + out[j] = 0; + return j > 0; +} + static bool json_get_int(const char *json, const char *key, int *out) { const char *p = json_value_for(json, key); @@ -303,6 +329,18 @@ static bool json_get_int(const char *json, const char *key, int *out) return true; } +static bool json_get_u32(const char *json, const char *key, uint32_t *out) +{ + const char *p = json_value_for(json, key); + if(!p || !out) return false; + if(*p == '-') return false; + char *end = NULL; + unsigned long v = strtoul(p, &end, 10); + if(end == p || v > UINT32_MAX) return false; + *out = (uint32_t)v; + return true; +} + static uint16_t app_permission_bit(const char *name) { if(strcmp(name, "display") == 0) return LZ_APP_PERM_DISPLAY; @@ -375,6 +413,335 @@ static bool safe_entry(const char *s) return true; } +#define LZ_APP_CATALOG_PACKAGE_MAX_BYTES (2u * 1024u * 1024u) +#define LZ_APP_CATALOG_SCREENSHOT_MAX 4 + +static bool catalog_fail(lz_app_catalog_report_t *r, const char *id, const char *msg) +{ + if(r) { + r->ok = false; + r->rejected_count++; + if(!r->first_error[0]) { + snprintf(r->first_error, sizeof r->first_error, "%s", msg ? msg : "invalid catalog"); + if(id && id[0]) snprintf(r->first_id, sizeof r->first_id, "%s", id); + } + } + return false; +} + +static bool catalog_url_ok(const char *url) +{ + if(!url || !url[0]) return false; + if(strncmp(url, "https://", 8) != 0 && strncmp(url, "http://", 7) != 0) return false; + for(int i = 0; url[i]; i++) { + char c = url[i]; + if(c <= 32 || c == '"' || c == '<' || c == '>') return false; + } + return true; +} + +static bool catalog_sha256_ok(const char *s) +{ + if(!s) return false; + for(int i = 0; i < 64; i++) { + char c = s[i]; + bool hex = (c >= '0' && c <= '9') || (c >= 'a' && c <= 'f') || + (c >= 'A' && c <= 'F'); + if(!hex) return false; + } + return s[64] == 0; +} + +static const char *json_array_for(const char *json, const char *key) +{ + const char *p = json_value_for(json, key); + if(!p || *p != '[') return NULL; + return p; +} + +static bool catalog_string_array_ok(const char *p, bool urls) +{ + if(!p || *p != '[') return false; + p = skip_ws(p + 1); + int count = 0; + if(*p == ']') return true; + for(;;) { + if(++count > LZ_APP_CATALOG_SCREENSHOT_MAX) return false; + if(*p != '"') return false; + p++; + char item[128]; + size_t j = 0; + bool too_long = false; + while(*p && *p != '"') { + char c = *p++; + if(c == '\\' && *p) c = *p++; + if(c < 32) continue; + if(j + 1 < sizeof item) item[j++] = c; + else too_long = true; + } + if(*p != '"' || too_long) return false; + item[j] = 0; + if(urls && !catalog_url_ok(item)) return false; + p = skip_ws(p + 1); + if(*p == ',') { p = skip_ws(p + 1); continue; } + if(*p == ']') return true; + return false; + } +} + +static const char *catalog_next_object(const char *p, char *out, size_t cap, + bool *done, bool *too_big) +{ + if(done) *done = false; + if(too_big) *too_big = false; + if(!p || !out || cap == 0) return NULL; + p = skip_ws(p); + if(*p == ',') p = skip_ws(p + 1); + if(*p == ']') { + if(done) *done = true; + return p + 1; + } + if(*p != '{') return NULL; + + int depth = 0; + bool in_str = false; + bool esc = false; + size_t j = 0; + for(; *p; p++) { + char c = *p; + if(j + 1 < cap) out[j++] = c; + else if(too_big) *too_big = true; + + if(in_str) { + if(esc) esc = false; + else if(c == '\\') esc = true; + else if(c == '"') in_str = false; + } else { + if(c == '"') in_str = true; + else if(c == '{') depth++; + else if(c == '}') { + depth--; + if(depth == 0) { + out[j] = 0; + return p + 1; + } + } + } + } + return NULL; +} + +static bool catalog_validate_app(const char *obj, lz_app_catalog_report_t *r) +{ + char id[24], name[32], version[16], author[28], desc[96], icon[20]; + char api[12], compat[32], url[128], sha[65]; + uint32_t size = 0; + int hue = -1; + + if(!json_get_string_bounded(obj, "id", id, sizeof id)) + return catalog_fail(r, NULL, "missing id"); + if(!safe_id(id)) return catalog_fail(r, id, "unsafe id"); + if(!json_get_string_bounded(obj, "name", name, sizeof name)) + return catalog_fail(r, id, "missing name"); + if(!json_get_string_bounded(obj, "version", version, sizeof version)) + return catalog_fail(r, id, "missing version"); + if(!json_get_string_bounded(obj, "author", author, sizeof author)) + return catalog_fail(r, id, "missing author"); + if(!json_get_string_bounded(obj, "description", desc, sizeof desc)) + return catalog_fail(r, id, "missing description"); + if(!json_get_string_bounded(obj, "icon", icon, sizeof icon)) + return catalog_fail(r, id, "missing icon"); + if(!json_get_string_bounded(obj, "api_version", api, sizeof api)) + return catalog_fail(r, id, "missing api_version"); + if(!api_version_supported(api)) return catalog_fail(r, id, "unsupported SDK"); + if(!json_get_string_bounded(obj, "compatibility", compat, sizeof compat)) + return catalog_fail(r, id, "missing compatibility"); + if(!json_get_string_bounded(obj, "download_url", url, sizeof url)) + return catalog_fail(r, id, "missing download_url"); + if(!catalog_url_ok(url)) return catalog_fail(r, id, "bad download_url"); + if(!json_get_string_bounded(obj, "sha256", sha, sizeof sha)) + return catalog_fail(r, id, "missing sha256"); + if(!catalog_sha256_ok(sha)) return catalog_fail(r, id, "bad sha256"); + if(!json_get_u32(obj, "size", &size)) + return catalog_fail(r, id, "missing size"); + if(size == 0 || size > LZ_APP_CATALOG_PACKAGE_MAX_BYTES) + return catalog_fail(r, id, "bad size"); + if(json_get_int(obj, "hue", &hue) && (hue < -1 || hue > 359)) + return catalog_fail(r, id, "bad hue"); + + uint16_t perms = 0; + const char *p = json_value_for(obj, "permissions"); + if(!p || !json_parse_permissions_value(p, &perms)) + return catalog_fail(r, id, "bad permissions"); + + const char *shots = json_value_for(obj, "screenshots"); + if(shots && !catalog_string_array_ok(shots, true)) + return catalog_fail(r, id, "bad screenshots"); + + if(r) r->app_count++; + return true; +} + +bool lz_store_validate_app_catalog_json(const char *json, lz_app_catalog_report_t *out) +{ + lz_app_catalog_report_t r; + memset(&r, 0, sizeof r); + r.ok = false; + if(!json || !json[0]) { + catalog_fail(&r, NULL, "empty catalog"); + if(out) *out = r; + return false; + } + if(strlen(json) > LZ_APP_CATALOG_JSON_MAX) { + catalog_fail(&r, NULL, "catalog too large"); + if(out) *out = r; + return false; + } + + char schema[32]; + if(!json_get_string_bounded(json, "schema", schema, sizeof schema) || + strcmp(schema, "limitlezz.app_catalog.v1") != 0) { + catalog_fail(&r, NULL, "bad schema"); + if(out) *out = r; + return false; + } + + const char *apps = json_array_for(json, "apps"); + if(!apps) { + catalog_fail(&r, NULL, "missing apps"); + if(out) *out = r; + return false; + } + + const char *p = skip_ws(apps + 1); + if(*p == ']') { + catalog_fail(&r, NULL, "empty apps"); + if(out) *out = r; + return false; + } + + char obj[1536]; + for(;;) { + bool done = false, too_big = false; + p = catalog_next_object(p, obj, sizeof obj, &done, &too_big); + if(done) break; + if(!p) { + catalog_fail(&r, NULL, "bad apps array"); + if(out) *out = r; + return false; + } + if(too_big) { + catalog_fail(&r, NULL, "app entry too large"); + if(out) *out = r; + return false; + } + if(r.app_count + r.rejected_count >= LZ_APP_CATALOG_MAX_APPS) { + catalog_fail(&r, NULL, "too many apps"); + if(out) *out = r; + return false; + } + if(!catalog_validate_app(obj, &r)) { + if(out) *out = r; + return false; + } + } + + r.ok = r.app_count > 0 && r.rejected_count == 0; + if(!r.ok && !r.first_error[0]) snprintf(r.first_error, sizeof r.first_error, "invalid catalog"); + if(out) *out = r; + return r.ok; +} + +static int catalog_report_line(char *buf, int n, const char *prefix, + const lz_app_catalog_report_t *r) +{ + if(!buf || n <= 0 || !r) return 0; + if(r->ok) + return snprintf(buf, (size_t)n, "%s: ready apps=%d rejected=0\n", + prefix, r->app_count); + if(r->first_id[0]) + return snprintf(buf, (size_t)n, "%s: invalid apps=%d rejected=%d first=%s error=\"%s\"\n", + prefix, r->app_count, r->rejected_count, r->first_id, r->first_error); + return snprintf(buf, (size_t)n, "%s: invalid apps=%d rejected=%d error=\"%s\"\n", + prefix, r->app_count, r->rejected_count, r->first_error); +} + +static bool catalog_read_file(const char *path, lz_app_catalog_report_t *r) +{ + FILE *f = fopen(path, "rb"); + if(!f) return false; + char json[LZ_APP_CATALOG_JSON_MAX + 2]; + size_t n = fread(json, 1, sizeof json - 1, f); + fclose(f); + json[n] = 0; + if(n >= sizeof json - 1) { + lz_app_catalog_report_t tmp; + memset(&tmp, 0, sizeof tmp); + tmp.ok = false; + catalog_fail(&tmp, NULL, "catalog too large"); + if(r) *r = tmp; + return true; + } + lz_store_validate_app_catalog_json(json, r); + return true; +} + +int lz_store_app_catalog_diag(char *buf, int n) +{ + if(!buf || n <= 0) return 0; + char path[160]; + if(g_persist) { + path_join(path, sizeof path, g_dir, "catalog/index.json"); + lz_app_catalog_report_t r; + if(catalog_read_file(path, &r)) + return catalog_report_line(buf, n, "app catalog", &r); + } + if(g_appfs_dir[0]) { + path_join(path, sizeof path, g_appfs_dir, "catalog/index.json"); + lz_app_catalog_report_t r; + if(catalog_read_file(path, &r)) + return catalog_report_line(buf, n, "app catalog", &r); + } + return snprintf(buf, (size_t)n, "app catalog: no cached index\n"); +} + +int lz_store_app_catalog_selftest(char *buf, int n) +{ + static const char valid[] = + "{\"schema\":\"limitlezz.app_catalog.v1\",\"updated\":\"2026-06-18T00:00:00Z\"," + "\"apps\":[" + "{\"id\":\"weather.mesh\",\"name\":\"Weather Mesh\",\"version\":\"0.1.0\"," + "\"author\":\"Limitless\",\"description\":\"Local weather reports\"," + "\"icon\":\"weather\",\"hue\":48,\"api_version\":\"0.1\"," + "\"compatibility\":\"tdeck\",\"permissions\":[\"display\",\"network_wifi\"]," + "\"download_url\":\"https://apps.example.invalid/weather.mesh.zip\"," + "\"sha256\":\"0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef\"," + "\"size\":32768,\"screenshots\":[\"https://apps.example.invalid/weather.bmp\"]}," + "{\"id\":\"notes.local\",\"name\":\"Field Notes\",\"version\":\"0.1.0\"," + "\"author\":\"Limitless\",\"description\":\"Simple local notes\"," + "\"icon\":\"notes\",\"api_version\":\"0.1\",\"compatibility\":\"tdeck\"," + "\"permissions\":[\"display\",\"input\",\"storage\"]," + "\"download_url\":\"https://apps.example.invalid/notes.local.zip\"," + "\"sha256\":\"abcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcdefabcd\"," + "\"size\":49152}]}"; + static const char invalid[] = + "{\"schema\":\"limitlezz.app_catalog.v1\",\"apps\":[" + "{\"id\":\"bad.local\",\"name\":\"Bad\",\"version\":\"0.1.0\"," + "\"author\":\"Limitless\",\"description\":\"Bad checksum\",\"icon\":\"bug\"," + "\"api_version\":\"0.1\",\"compatibility\":\"tdeck\"," + "\"permissions\":[\"display\"],\"download_url\":\"https://apps.example.invalid/bad.zip\"," + "\"sha256\":\"not-a-sha\",\"size\":1024}]}"; + + lz_app_catalog_report_t ok, bad; + bool valid_ok = lz_store_validate_app_catalog_json(valid, &ok); + bool invalid_ok = lz_store_validate_app_catalog_json(invalid, &bad); + const char *result = (valid_ok && ok.app_count == 2 && !invalid_ok && + strcmp(bad.first_error, "bad sha256") == 0) ? "PASS" : "FAIL"; + return snprintf(buf, (size_t)n, + "App catalog selftest: %s valid=%d invalid_error=\"%s\"\n", + result, ok.app_count, bad.first_error); +} + static bool local_app_seen(const lz_local_app_t *out, int n, const char *id) { for(int i = 0; i < n; i++) From e306161572d54794b11e54027629cc9b68909656 Mon Sep 17 00:00:00 2001 From: n30nex Date: Fri, 19 Jun 2026 05:03:26 -0400 Subject: [PATCH 17/64] Add OTA manifest diagnostics --- README.md | 5 + docs/tdeck-feature-inventory.md | 2 +- docs/tdeck-firmware-roadmap.md | 9 +- docs/tdeck-ota-manifest.md | 98 ++++++++++++++++ sim/main_sim.c | 54 ++++++++- src/serial_cli.cpp | 30 +++++ src/services/mesh.h | 20 ++++ src/services/mesh_core.c | 12 ++ src/services/store.c | 192 ++++++++++++++++++++++++++++++++ 9 files changed, 415 insertions(+), 7 deletions(-) create mode 100644 docs/tdeck-ota-manifest.md diff --git a/README.md b/README.md index 722c799..338069c 100644 --- a/README.md +++ b/README.md @@ -132,6 +132,10 @@ iPhone-style dark look (status bar, battery glyph, grouped settings cards). - **App flash (`appfs`)** - T-Deck builds mount the FAT `appfs` partition at `/appfs` without formatting, expose it beside SD/local storage in Files, and scan `/appfs/apps` even when the SD card is absent. +- **OTA manifest diagnostics** - validate cached `limitlezz.ota_manifest.v1` + metadata from `/sd/limitlezz/ota`, `/sd/ota`, or `/appfs/ota`; serial + `ota status` and `ota test` prove the parser before Wi-Fi download, + inactive-slot write, and rollback are implemented. - **Security**: optional device **password/PIN**, and **encrypt the data files** (messages, identity, keys) when a password is set. - **Hardening**: Wi-Fi passwords are stored in plaintext on the SD card @@ -144,6 +148,7 @@ iPhone-style dark look (status bar, battery glyph, grouped settings cards). - [`docs/tdeck-feature-inventory.md`](docs/tdeck-feature-inventory.md) - feature-by-feature implementation inventory. - [`docs/tdeck-firmware-roadmap.md`](docs/tdeck-firmware-roadmap.md) - roadmap to a complete T-Deck firmware. - [`docs/tdeck-hardware-dogfood-checklist.md`](docs/tdeck-hardware-dogfood-checklist.md) - stock-device hardware proof checklist. +- [`docs/tdeck-ota-manifest.md`](docs/tdeck-ota-manifest.md) - cached OTA firmware manifest schema and serial diagnostics. ![screens](docs/screens.png) diff --git a/docs/tdeck-feature-inventory.md b/docs/tdeck-feature-inventory.md index 416182f..c83d535 100644 --- a/docs/tdeck-feature-inventory.md +++ b/docs/tdeck-feature-inventory.md @@ -106,7 +106,7 @@ Status labels: | Device PIN/password | Planned | README later/security section | Needed before encrypted local data UX. | | Encrypted local store | Planned | README hardening section | Encrypt messages, keys, identity, and app data when password is set. | | Wi-Fi credential hardening | Functional for T-Deck, sim file-backed | T-Deck `lz_store_save_wifi/load_wifi` use ESP32 NVS with legacy `wifi.cfg` migration/removal; serial `wifi` reports `cred=nvs` without printing the password | Native simulator intentionally keeps file-backed credentials for repeatable desktop tests; broaden later if encrypted whole-store support lands. | -| OTA firmware update | Planned | Partition table and design spec | Implement download, hash verify, inactive-slot write, rollback UX. | +| OTA firmware update | Partial | `docs/tdeck-ota-manifest.md`; `partitions.csv`; serial `ota status` and `ota test`; bounded cached-manifest validator rejects bad schema, board, URL, SHA-256, and oversized binaries before any updater trusts them | Implement Wi-Fi fetch/download, binary hash verify, inactive-slot write, boot-partition switch, rollback UX, update UI, and feedback routing. | | Feedback Manager | Planned | Design spec section 8 | Centralize LED, buzzer, keyboard/display feedback and DND. | | Emergency beacon | Planned | Design spec section 12, disabled Emergency UI row | Requires Feedback Manager and dual-network messaging. | | BLE companion | Partial, needs validation | NimBLE-based Meshtastic GATT service, official UUIDs, raw `ToRadio` writes, queued `FromRadio` reads, `FromNum` notifications, UI toggle, and serial selftest/status | Validate with the official Meshtastic app over BLE before calling V0.5 complete. | diff --git a/docs/tdeck-firmware-roadmap.md b/docs/tdeck-firmware-roadmap.md index f987e35..8b395d8 100644 --- a/docs/tdeck-firmware-roadmap.md +++ b/docs/tdeck-firmware-roadmap.md @@ -343,9 +343,16 @@ Exit criteria: Goal: update the OS without USB flashing. +**Status:** partial. Implemented: a bounded `limitlezz.ota_manifest.v1` +validator, cached manifest discovery from SD/appfs, serial `ota status`, serial +`ota test`, and native selftest coverage. Fetch/download, binary hash verify, +inactive-slot write, boot-partition switch, rollback, UI, and feedback routing +remain TODO. + Deliverables: -- Implement firmware update manifest alongside the app catalog. +- Implement firmware update manifest alongside the app catalog. Implemented: + `docs/tdeck-ota-manifest.md` plus cached manifest diagnostics. - Download firmware binary over Wi-Fi. - Verify SHA256 before writing. - Write to inactive OTA partition. diff --git a/docs/tdeck-ota-manifest.md b/docs/tdeck-ota-manifest.md new file mode 100644 index 0000000..1c95c1c --- /dev/null +++ b/docs/tdeck-ota-manifest.md @@ -0,0 +1,98 @@ +# T-Deck OTA Firmware Manifest + +This is the first Phase 10 OTA increment: a bounded manifest contract and +diagnostics path. It validates update metadata before any Wi-Fi downloader, +inactive-slot writer, boot-partition switch, or rollback flow trusts it. + +Implemented: + +- `ota status` over the USB serial console. +- `ota test` over the USB serial console. +- Native simulator selftest coverage for valid and invalid manifests. +- Cached manifest discovery from SD/local storage and the `appfs` partition. + +Still TODO: + +- fetch the manifest over Wi-Fi +- download the firmware binary +- verify the binary SHA-256 after download +- write to the inactive OTA slot +- set the OTA boot partition and mark the new firmware healthy +- rollback UX and failure recovery +- user-facing update screen and Feedback Manager progress routing + +## Cache Paths + +The firmware looks for one cached JSON manifest at the first matching path: + +1. `/sd/limitlezz/ota/manifest.json` +2. `/sd/ota/manifest.json` +3. `/appfs/ota/manifest.json` + +The native simulator uses the same layout under its data directory, for example +`lzdata/ota/manifest.json` and `lzdata/appfs/ota/manifest.json`. + +## Schema + +The manifest is a tiny top-level JSON object. The parser is intentionally +bounded: no allocation, no recursion, a maximum file size of 2 KB, and +fail-closed validation. + +Required fields: + +```json +{ + "schema": "limitlezz.ota_manifest.v1", + "version": "0.97.0", + "channel": "beta", + "board": "tdeck", + "firmware_url": "https://updates.example/tdeck/0.97.0/firmware.bin", + "sha256": "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef", + "size": 1539920 +} +``` + +Optional fields: + +```json +{ + "min_version": "0.96.0", + "notes_url": "https://updates.example/tdeck/0.97.0/notes" +} +``` + +Validation rules: + +- `schema` must be `limitlezz.ota_manifest.v1`. +- `board` must be `tdeck`. +- `version`, `channel`, and `min_version` use only letters, numbers, `_`, `-`, + and `.`. +- `firmware_url` and `notes_url` must be `http://` or `https://` URLs without + whitespace, quotes, or backslashes. +- `sha256` must be exactly 64 hex characters. +- `size` must be greater than 0 and no larger than the T-Deck OTA slot size + from `partitions.csv` (`0x500000`, 5,242,880 bytes). + +## Serial Diagnostics + +Fresh hardware with no cached manifest: + +```text +lz> ota status +ota manifest: no cached manifest +``` + +Valid cached manifest: + +```text +lz> ota status +ota manifest: valid version=0.97.0 channel=beta board=tdeck size=1539920 source=/sd/limitlezz/ota/manifest.json +firmware: https://updates.example/tdeck/0.97.0/firmware.bin +``` + +Built-in parser proof: + +```text +lz> ota test +OTA manifest selftest: PASS valid=1 invalid_error="bad sha256" +``` diff --git a/sim/main_sim.c b/sim/main_sim.c index 9c54b9b..fef5f22 100644 --- a/sim/main_sim.c +++ b/sim/main_sim.c @@ -718,7 +718,51 @@ static int codec_selftest(void) lz_store_init(NULL); } - /* 10. local app scanner: valid manifests become local apps; broken packages + /* 10. OTA manifest diagnostics: cached metadata is bounded and fail-closed + * before any downloader or inactive-slot writer trusts it. */ + { + extern void lz_store_init(const char *datadir); + extern bool lz_store_ota_manifest_status(lz_ota_manifest_t *out); + extern int lz_store_ota_manifest_selftest(char *buf, int n); + sim_reset_dir("lzdata_ota"); + sim_mkdirs("lzdata_ota/ota"); + FILE *mf = fopen("lzdata_ota/ota/manifest.json", "wb"); + if(mf) { + fputs("{\"schema\":\"limitlezz.ota_manifest.v1\",\"version\":\"0.97.0\"," + "\"channel\":\"beta\",\"board\":\"tdeck\"," + "\"firmware_url\":\"https://updates.limitlezz.example/tdeck/0.97.0/firmware.bin\"," + "\"sha256\":\"0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef\"," + "\"size\":1539920}", mf); + fclose(mf); + } + lz_store_init("lzdata_ota"); + lz_ota_manifest_t ota; + CHECK(lz_store_ota_manifest_status(&ota) && ota.valid, + "OTA manifest status accepts cached valid manifest"); + CHECK(strcmp(ota.version, "0.97.0") == 0 && ota.size_bytes == 1539920u, + "OTA manifest status keeps version and size"); + + mf = fopen("lzdata_ota/ota/manifest.json", "wb"); + if(mf) { + fputs("{\"schema\":\"limitlezz.ota_manifest.v1\",\"version\":\"0.97.0\"," + "\"channel\":\"beta\",\"board\":\"tdeck\"," + "\"firmware_url\":\"ftp://updates.example/fw.bin\"," + "\"sha256\":\"0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef\"," + "\"size\":1539920}", mf); + fclose(mf); + } + CHECK(!lz_store_ota_manifest_status(&ota) && ota.found && !ota.valid && + strcmp(ota.error, "bad firmware_url") == 0, + "OTA manifest status rejects unsafe firmware URL"); + char diag[160]; + lz_store_ota_manifest_selftest(diag, sizeof diag); + CHECK(strstr(diag, "PASS") != NULL && strstr(diag, "bad sha256") != NULL, + "OTA manifest selftest reports parser pass"); + lz_store_init(NULL); + sim_reset_dir("lzdata_ota"); + } + + /* 11. local app scanner: valid manifests become local apps; broken packages * are ignored before they can reach Home/App Store. */ { extern void lz_store_init(const char *datadir); @@ -946,7 +990,7 @@ static int codec_selftest(void) sim_reset_dir("lzdata_appscan"); } - /* 10. service-level SDK token injection: dynamic read-only values are + /* 12. service-level SDK token injection: dynamic read-only values are * expanded only when the matching permissions are declared. */ { extern void lz_store_init(const char *datadir); @@ -1018,7 +1062,7 @@ static int codec_selftest(void) sim_reset_dir("lzdata_apptokens"); } - /* 11. appfs root support: apps can be discovered from appfs even when + /* 13. appfs root support: apps can be discovered from appfs even when * SD-backed persistence is absent, and Files can expose both roots. */ { extern void lz_store_init(const char *datadir); @@ -1048,7 +1092,7 @@ static int codec_selftest(void) sim_reset_dir("lzdata_appfsroot"); } - /* 11. MeshCore Public-channel GRP_TXT: decode a known reference vector, + /* 14. MeshCore Public-channel GRP_TXT: decode a known reference vector, * reject a wrong key (MAC), and round-trip an encode. Vector generated * against the documented scheme (AES-128-ECB + HMAC-SHA256 trunc-2). */ { @@ -1082,7 +1126,7 @@ static int codec_selftest(void) strcmp(rm.text, "hi there") == 0, "MeshCore GRP_TXT round-trip fields"); } - /* 13. MeshCore DM (TXT_MSG): ECDH derive (vs orlp/standard reference) then a + /* 15. MeshCore DM (TXT_MSG): ECDH derive (vs orlp/standard reference) then a * full encode->parse->decode round-trip + ACK match + MAC tamper check. */ { uint8_t seedA[32], pubA[32], pubB[32], ref[32]; diff --git a/src/serial_cli.cpp b/src/serial_cli.cpp index b46d44c..eeeba8f 100644 --- a/src/serial_cli.cpp +++ b/src/serial_cli.cpp @@ -69,6 +69,7 @@ static void cmd_help(void) " nodes list heard nodes\n" " send broadcast text on the channel\n" " stats radio TX/RX + airtime utilization\n" + " ota [status|test] cached OTA firmware manifest diagnostics\n" " wifi [scan|on|off] wifi status / control\n" " sys battery, uptime, memory\n" " id this node's identity\n" @@ -366,6 +367,34 @@ static void cmd_stats(void) (unsigned)st.tx_count, (unsigned)st.rx_count, st.util_pct); } +static void cmd_ota(char *args) +{ + if(args && strcmp(args, "test") == 0) { + char b[160]; + lz_svc_ota_manifest_selftest(b, sizeof b); + Serial.println(b); + return; + } + if(args && args[0] && strcmp(args, "status") != 0) { + Serial.println("usage: ota [status|test]"); + return; + } + + lz_ota_manifest_t m; + lz_svc_ota_manifest_status(&m); + if(!m.found) { + Serial.println("ota manifest: no cached manifest"); + } else if(!m.valid) { + Serial.printf("ota manifest: invalid source=%s error=\"%s\"\n", + m.source, m.error); + } else { + Serial.printf("ota manifest: valid version=%s channel=%s board=%s size=%lu source=%s\n", + m.version, m.channel, m.board, + (unsigned long)m.size_bytes, m.source); + Serial.printf("firmware: %s\n", m.firmware_url); + } +} + static void print_wifi_text(const char *text) { if(!text || !text[0]) { Serial.print("(none)"); return; } @@ -439,6 +468,7 @@ static void dispatch(char *line) else if(!strcmp(line, "nodes")) cmd_nodes(); else if(!strcmp(line, "send")) cmd_send(args); else if(!strcmp(line, "stats")) cmd_stats(); + else if(!strcmp(line, "ota")) cmd_ota(args); else if(!strcmp(line, "wifi")) cmd_wifi(args); else if(!strcmp(line, "sys")) cmd_sys(); else if(!strcmp(line, "id")) cmd_id(); diff --git a/src/services/mesh.h b/src/services/mesh.h index aacfc8d..2ca4f2e 100644 --- a/src/services/mesh.h +++ b/src/services/mesh.h @@ -34,6 +34,9 @@ extern "C" { #define LZ_LOCAL_APP_ACTION_MAX 2 #define LZ_LOCAL_APP_ACTION_EFFECT_MAX 32 #define LZ_LOCAL_APP_ACTION_BODY_MAX 192 +#define LZ_OTA_MANIFEST_SCHEMA "limitlezz.ota_manifest.v1" +#define LZ_OTA_BOARD_TDECK "tdeck" +#define LZ_OTA_SLOT_MAX_BYTES 0x500000u #define LZ_APP_PERM_DISPLAY 0x0001u #define LZ_APP_PERM_INPUT 0x0002u @@ -218,6 +221,21 @@ typedef struct { lz_local_app_action_t actions[LZ_LOCAL_APP_ACTION_MAX]; } lz_local_app_session_t; +typedef struct { + bool found; + bool valid; + char source[112]; /* cached manifest file path */ + char error[48]; /* plain-language rejection reason */ + char version[24]; + char channel[16]; + char board[16]; + char firmware_url[128]; + char sha256[65]; + uint32_t size_bytes; + char min_version[24]; + char notes_url[96]; +} lz_ota_manifest_t; + /* ---- lifecycle ---- */ void lz_svc_init(const char *datadir, bool seed_demo); /* datadir NULL = RAM only */ void lz_svc_loop(void); /* pump backend + timers */ @@ -234,6 +252,8 @@ bool lz_svc_app_data_usage(const lz_local_app_t *app, uint32_t *used, uint32_t * bool lz_svc_clear_app_data(const lz_local_app_t *app, char *err, int err_cap); bool lz_svc_start_local_app(const lz_local_app_t *app, lz_local_app_session_t *out); bool lz_svc_local_app_action(lz_local_app_session_t *session, int idx); +bool lz_svc_ota_manifest_status(lz_ota_manifest_t *out); +int lz_svc_ota_manifest_selftest(char *buf, int n); /* ---- nodes ---- */ int lz_svc_nodes(const lz_node_rt **out); /* all heard nodes */ diff --git a/src/services/mesh_core.c b/src/services/mesh_core.c index 665f367..a6e3714 100644 --- a/src/services/mesh_core.c +++ b/src/services/mesh_core.c @@ -25,6 +25,8 @@ bool lz_store_app_data_usage(const lz_local_app_t *app, uint32_t *used, uint32_t bool lz_store_clear_app_data(const lz_local_app_t *app, char *err, int err_cap); bool lz_store_start_local_app(const lz_local_app_t *app, lz_local_app_session_t *out); bool lz_store_local_app_action(lz_local_app_session_t *session, int idx); +bool lz_store_ota_manifest_status(lz_ota_manifest_t *out); +int lz_store_ota_manifest_selftest(char *buf, int n); void lz_store_append(const char *addr, const lz_msg_rt *m); int lz_store_load_tail(const char *addr, lz_msg_rt *ring, int cap); bool lz_store_find_delivery(const char *addr, uint32_t pkt_id, lz_msg_rt *out); @@ -402,6 +404,16 @@ bool lz_svc_local_app_action(lz_local_app_session_t *session, int idx) return local_app_expand_session(session); } +bool lz_svc_ota_manifest_status(lz_ota_manifest_t *out) +{ + return lz_store_ota_manifest_status(out); +} + +int lz_svc_ota_manifest_selftest(char *buf, int n) +{ + return lz_store_ota_manifest_selftest(buf, n); +} + const char *lz_fmt_ago(uint32_t ts, char *buf, size_t n) { if(ts == 0) { snprintf(buf, n, "-"); return buf; } diff --git a/src/services/store.c b/src/services/store.c index 0d73515..0ceff4f 100644 --- a/src/services/store.c +++ b/src/services/store.c @@ -303,6 +303,20 @@ static bool json_get_int(const char *json, const char *key, int *out) return true; } +static bool json_get_u32(const char *json, const char *key, uint32_t *out) +{ + const char *p = json_value_for(json, key); + if(!p) return false; + char *end = NULL; + unsigned long v = strtoul(p, &end, 10); + if(end == p) return false; + while(*end == ' ' || *end == '\t' || *end == '\r' || *end == '\n') end++; + if(*end && *end != ',' && *end != '}') return false; + if(v > UINT32_MAX) return false; + *out = (uint32_t)v; + return true; +} + static uint16_t app_permission_bit(const char *name) { if(strcmp(name, "display") == 0) return LZ_APP_PERM_DISPLAY; @@ -728,6 +742,184 @@ int lz_store_scan_app_issues(lz_local_app_issue_t *out, int cap) return count; } +/* ---- OTA firmware manifest diagnostics ---- */ + +static bool ota_fail(lz_ota_manifest_t *out, const char *msg) +{ + if(out) { + out->valid = false; + snprintf(out->error, sizeof out->error, "%s", msg ? msg : "invalid manifest"); + } + return false; +} + +static bool safe_ota_token(const char *s) +{ + if(!s || !s[0]) return false; + for(int i = 0; s[i]; i++) { + char c = s[i]; + bool ok = (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') || + (c >= '0' && c <= '9') || c == '_' || c == '-' || c == '.'; + if(!ok) return false; + } + return true; +} + +static bool safe_http_url(const char *s) +{ + if(!s || !s[0]) return false; + bool ok_scheme = strncmp(s, "https://", 8) == 0 || strncmp(s, "http://", 7) == 0; + if(!ok_scheme) return false; + for(int i = 0; s[i]; i++) + if((unsigned char)s[i] <= 32 || s[i] == '"' || s[i] == '\\') return false; + return true; +} + +static bool hex64(const char *s) +{ + if(!s) return false; + for(int i = 0; i < 64; i++) { + char c = s[i]; + bool ok = (c >= '0' && c <= '9') || (c >= 'a' && c <= 'f') || + (c >= 'A' && c <= 'F'); + if(!ok) return false; + } + return s[64] == 0; +} + +static bool parse_ota_manifest_json(const char *json, lz_ota_manifest_t *out) +{ + char schema[36]; + if(!json || !out) return false; + out->found = true; + out->valid = false; + out->error[0] = 0; + + if(!json_get_string(json, "schema", schema, sizeof schema)) + return ota_fail(out, "missing schema"); + if(strcmp(schema, LZ_OTA_MANIFEST_SCHEMA) != 0) + return ota_fail(out, "unsupported schema"); + if(!json_get_string(json, "version", out->version, sizeof out->version)) + return ota_fail(out, "missing version"); + if(!json_get_string(json, "channel", out->channel, sizeof out->channel)) + return ota_fail(out, "missing channel"); + if(!json_get_string(json, "board", out->board, sizeof out->board)) + return ota_fail(out, "missing board"); + if(!json_get_string(json, "firmware_url", out->firmware_url, sizeof out->firmware_url)) + return ota_fail(out, "missing firmware_url"); + if(!json_get_string(json, "sha256", out->sha256, sizeof out->sha256)) + return ota_fail(out, "missing sha256"); + if(!json_get_u32(json, "size", &out->size_bytes)) + return ota_fail(out, "missing size"); + + json_get_string(json, "min_version", out->min_version, sizeof out->min_version); + json_get_string(json, "notes_url", out->notes_url, sizeof out->notes_url); + + if(!safe_ota_token(out->version)) return ota_fail(out, "bad version"); + if(!safe_ota_token(out->channel)) return ota_fail(out, "bad channel"); + if(strcmp(out->board, LZ_OTA_BOARD_TDECK) != 0) return ota_fail(out, "unsupported board"); + if(!safe_http_url(out->firmware_url)) return ota_fail(out, "bad firmware_url"); + if(!hex64(out->sha256)) return ota_fail(out, "bad sha256"); + if(out->size_bytes == 0 || out->size_bytes > LZ_OTA_SLOT_MAX_BYTES) + return ota_fail(out, "bad size"); + if(out->min_version[0] && !safe_ota_token(out->min_version)) + return ota_fail(out, "bad min_version"); + if(out->notes_url[0] && !safe_http_url(out->notes_url)) + return ota_fail(out, "bad notes_url"); + + out->valid = true; + return true; +} + +static bool load_ota_manifest_path(const char *path, lz_ota_manifest_t *out) +{ + if(!path || !out || !path_is_file(path)) return false; + memset(out, 0, sizeof *out); + out->found = true; + snprintf(out->source, sizeof out->source, "%s", path); + + FILE *f = fopen(path, "rb"); + if(!f) return ota_fail(out, "manifest unreadable"); + char json[2049]; + size_t n = fread(json, 1, sizeof json - 1, f); + fclose(f); + json[n] = 0; + if(n == 0) return ota_fail(out, "empty manifest"); + if(n >= sizeof json - 1) return ota_fail(out, "manifest too large"); + + parse_ota_manifest_json(json, out); + return true; +} + +bool lz_store_ota_manifest_status(lz_ota_manifest_t *out) +{ + if(!out) return false; + memset(out, 0, sizeof *out); + + char path[160]; + if(g_persist) { + path_join(path, sizeof path, g_dir, "ota/manifest.json"); + if(load_ota_manifest_path(path, out)) return out->valid; + + const char *root = lz_store_file_root(); + if(root && strcmp(root, g_dir) != 0) { + path_join(path, sizeof path, root, "ota/manifest.json"); + if(load_ota_manifest_path(path, out)) return out->valid; + } + } + + if(g_appfs_dir[0]) { + path_join(path, sizeof path, g_appfs_dir, "ota/manifest.json"); + if(load_ota_manifest_path(path, out)) return out->valid; + } + + snprintf(out->error, sizeof out->error, "no cached manifest"); + return false; +} + +int lz_store_ota_manifest_selftest(char *buf, int n) +{ + if(!buf || n <= 0) return 0; + static const char *good = + "{\"schema\":\"limitlezz.ota_manifest.v1\",\"version\":\"0.97.0\"," + "\"channel\":\"beta\",\"board\":\"tdeck\"," + "\"firmware_url\":\"https://updates.limitlezz.example/tdeck/0.97.0/firmware.bin\"," + "\"sha256\":\"0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef\"," + "\"size\":1539920,\"min_version\":\"0.96.0\"," + "\"notes_url\":\"https://updates.limitlezz.example/tdeck/0.97.0/notes\"}"; + static const char *bad_sha = + "{\"schema\":\"limitlezz.ota_manifest.v1\",\"version\":\"0.97.0\"," + "\"channel\":\"beta\",\"board\":\"tdeck\"," + "\"firmware_url\":\"https://updates.limitlezz.example/tdeck/0.97.0/firmware.bin\"," + "\"sha256\":\"bad\",\"size\":1539920}"; + static const char *bad_size = + "{\"schema\":\"limitlezz.ota_manifest.v1\",\"version\":\"0.97.0\"," + "\"channel\":\"beta\",\"board\":\"tdeck\"," + "\"firmware_url\":\"https://updates.limitlezz.example/tdeck/0.97.0/firmware.bin\"," + "\"sha256\":\"0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef\"," + "\"size\":5242881}"; + + lz_ota_manifest_t m; + memset(&m, 0, sizeof m); + bool ok_valid = parse_ota_manifest_json(good, &m) && m.valid && + m.size_bytes == 1539920u && strcmp(m.board, LZ_OTA_BOARD_TDECK) == 0; + + memset(&m, 0, sizeof m); + bool ok_sha = !parse_ota_manifest_json(bad_sha, &m) && + strcmp(m.error, "bad sha256") == 0; + char sha_error[48]; + snprintf(sha_error, sizeof sha_error, "%s", m.error); + + memset(&m, 0, sizeof m); + bool ok_size = !parse_ota_manifest_json(bad_size, &m) && + strcmp(m.error, "bad size") == 0; + + bool pass = ok_valid && ok_sha && ok_size; + return snprintf(buf, (size_t)n, "OTA manifest selftest: %s valid=%d invalid_error=\"%s\"", + pass ? "PASS" : "FAIL", ok_valid ? 1 : 0, + sha_error[0] ? sha_error : "-"); +} + bool lz_store_prepare_app_data(const lz_local_app_t *app, char *path_out, int path_cap, char *err, int err_cap) { From fa44512d4d11357cd65a74262611f3dda3c7ebbe Mon Sep 17 00:00:00 2001 From: n30nex Date: Fri, 19 Jun 2026 05:21:57 -0400 Subject: [PATCH 18/64] Add device PIN verifier diagnostics --- README.md | 12 +- docs/tdeck-device-security.md | 70 +++++++++ docs/tdeck-feature-inventory.md | 4 +- docs/tdeck-firmware-roadmap.md | 9 +- sim/main_sim.c | 50 +++++- src/serial_cli.cpp | 48 ++++++ src/services/mesh.h | 16 ++ src/services/mesh_core.c | 30 ++++ src/services/store.c | 262 ++++++++++++++++++++++++++++++++ 9 files changed, 488 insertions(+), 13 deletions(-) create mode 100644 docs/tdeck-device-security.md diff --git a/README.md b/README.md index 722c799..4aad92d 100644 --- a/README.md +++ b/README.md @@ -132,11 +132,12 @@ iPhone-style dark look (status bar, battery glyph, grouped settings cards). - **App flash (`appfs`)** - T-Deck builds mount the FAT `appfs` partition at `/appfs` without formatting, expose it beside SD/local storage in Files, and scan `/appfs/apps` even when the SD card is absent. -- **Security**: optional device **password/PIN**, and **encrypt the data files** - (messages, identity, keys) when a password is set. -- **Hardening**: Wi-Fi passwords are stored in plaintext on the SD card - (`/limitlezz/wifi.cfg`) — move to NVS or encrypt (covered by the above when a - password is set). +- **Device security** - optional device PIN verifier with serial + `security status`, `security set`, `security check`, `security clear`, and + `security test`; this is setup groundwork and does not encrypt local data yet. +- **Security still ahead**: use the PIN/password secret to encrypt message + history, identity, keys, and app data, with honest recovery wording when a PIN + is forgotten. ## Audit and completion plan @@ -144,6 +145,7 @@ iPhone-style dark look (status bar, battery glyph, grouped settings cards). - [`docs/tdeck-feature-inventory.md`](docs/tdeck-feature-inventory.md) - feature-by-feature implementation inventory. - [`docs/tdeck-firmware-roadmap.md`](docs/tdeck-firmware-roadmap.md) - roadmap to a complete T-Deck firmware. - [`docs/tdeck-hardware-dogfood-checklist.md`](docs/tdeck-hardware-dogfood-checklist.md) - stock-device hardware proof checklist. +- [`docs/tdeck-device-security.md`](docs/tdeck-device-security.md) - device PIN verifier contract and remaining encrypted-store work. ![screens](docs/screens.png) diff --git a/docs/tdeck-device-security.md b/docs/tdeck-device-security.md new file mode 100644 index 0000000..9f433a9 --- /dev/null +++ b/docs/tdeck-device-security.md @@ -0,0 +1,70 @@ +# T-Deck Device Security + +This is the first Phase 12 security increment: an optional device PIN verifier +and diagnostics path. It proves the setup/check/clear flow without claiming that +local data is encrypted yet. + +Implemented: + +- `security status` over the USB serial console. +- `security set ` for a 4-12 digit PIN. +- `security check ` to verify the saved PIN. +- `security clear ` to remove the verifier after a correct PIN. +- `security test` for the bounded verifier selftest. +- Native simulator selftest coverage for set/check/reject/clear behavior. + +Still TODO: + +- lock-screen or Settings UI for setting and unlocking with the PIN +- data encryption using the PIN-derived secret +- migration of existing plaintext message history, identities, keys, and app data +- recovery wording for forgotten PINs +- secure boot or flash-encryption guidance for advanced builds + +## Storage Contract + +The firmware stores one verifier file at: + +```text +/sd/limitlezz/security.cfg +``` + +The simulator uses the same filename under its data directory. The format is a +single line: + +```text +1|pin-sha256|2048|<16 hex salt>|<64 hex verifier> +``` + +The PIN itself is never written to disk. The verifier is a salted, iterated +SHA-256 value with the current work factor from `LZ_SECURITY_KDF_ROUNDS`. + +This is only an authentication gate for later encrypted storage work. Until the +encrypted store lands, a lost SD card can still reveal plaintext message logs, +identity files, MeshCore keys, Meshtastic PKI keys, and app data. + +## Serial Diagnostics + +Fresh hardware with no configured PIN: + +```text +lz> security status +security: no device PIN set (not configured); encrypted-store=not-enabled +``` + +Set and verify a PIN: + +```text +lz> security set 123456 +[ok] device PIN verifier set (data encryption not enabled yet) + +lz> security check 123456 +[ok] PIN accepted +``` + +Built-in verifier proof: + +```text +lz> security test +PIN verifier selftest: PASS min=4 max=12 rounds=2048 +``` diff --git a/docs/tdeck-feature-inventory.md b/docs/tdeck-feature-inventory.md index 416182f..273db30 100644 --- a/docs/tdeck-feature-inventory.md +++ b/docs/tdeck-feature-inventory.md @@ -103,8 +103,8 @@ Status labels: | Feature | Status | Evidence | Gap / Next Action | | --- | --- | --- | --- | -| Device PIN/password | Planned | README later/security section | Needed before encrypted local data UX. | -| Encrypted local store | Planned | README hardening section | Encrypt messages, keys, identity, and app data when password is set. | +| Device PIN/password | Partial | `docs/tdeck-device-security.md`; serial `security status`, `security set`, `security check`, `security clear`, and `security test`; `security.cfg` stores a salted, iterated SHA-256 verifier instead of plaintext PINs | Add Settings/lock-screen UI, unlock state, forgotten-PIN recovery language, and encrypted-store integration. | +| Encrypted local store | Planned | README Device security note; Phase 12 roadmap | Encrypt messages, keys, identity, and app data when password is set; migrate existing plaintext stores. | | Wi-Fi credential hardening | Functional for T-Deck, sim file-backed | T-Deck `lz_store_save_wifi/load_wifi` use ESP32 NVS with legacy `wifi.cfg` migration/removal; serial `wifi` reports `cred=nvs` without printing the password | Native simulator intentionally keeps file-backed credentials for repeatable desktop tests; broaden later if encrypted whole-store support lands. | | OTA firmware update | Planned | Partition table and design spec | Implement download, hash verify, inactive-slot write, rollback UX. | | Feedback Manager | Planned | Design spec section 8 | Centralize LED, buzzer, keyboard/display feedback and DND. | diff --git a/docs/tdeck-firmware-roadmap.md b/docs/tdeck-firmware-roadmap.md index f987e35..c55321a 100644 --- a/docs/tdeck-firmware-roadmap.md +++ b/docs/tdeck-firmware-roadmap.md @@ -389,9 +389,16 @@ Exit criteria: Goal: protect the user's local data without making setup hard. +**Status:** partial. Implemented: a bounded optional device PIN verifier stored +as salted, iterated SHA-256; serial `security status`, `security set`, +`security check`, `security clear`, and `security test`; native selftest +coverage. Local data encryption, migration, UI, and recovery wording remain +TODO. + Deliverables: -- Add optional device PIN/password. +- Add optional device PIN/password. Implemented as the first PIN verifier + groundwork in `docs/tdeck-device-security.md`; UI and unlock flow remain TODO. - Use the secret to encrypt: - message history - identity diff --git a/sim/main_sim.c b/sim/main_sim.c index 9c54b9b..b14fbf8 100644 --- a/sim/main_sim.c +++ b/sim/main_sim.c @@ -718,7 +718,47 @@ static int codec_selftest(void) lz_store_init(NULL); } - /* 10. local app scanner: valid manifests become local apps; broken packages + /* 10. Device PIN verifier: stores only a bounded salted verifier for the + * first Phase 12 setup gate; encryption lands in later slices. */ + { + extern void lz_store_init(const char *datadir); + extern bool lz_store_security_status(lz_security_status_t *out); + extern bool lz_store_security_set_pin(const char *pin, char *err, int err_cap); + extern bool lz_store_security_check_pin(const char *pin); + extern bool lz_store_security_clear_pin(const char *pin, char *err, int err_cap); + extern int lz_store_security_selftest(char *buf, int n); + sim_reset_dir("lzdata_security"); + lz_store_init("lzdata_security"); + lz_security_status_t sec; + CHECK(!lz_store_security_status(&sec) && !sec.configured && sec.valid, + "security status starts with no PIN"); + char err[64] = {0}; + CHECK(!lz_store_security_set_pin("12ab", err, sizeof err) && + strcmp(err, "PIN must use digits only") == 0, + "security PIN rejects non-digits"); + CHECK(lz_store_security_set_pin("123456", err, sizeof err), + "security PIN verifier stores"); + CHECK(lz_store_security_status(&sec) && sec.configured && sec.valid && + sec.rounds == LZ_SECURITY_KDF_ROUNDS, + "security status reports configured verifier"); + CHECK(lz_store_security_check_pin("123456"), "security PIN accepts correct value"); + CHECK(!lz_store_security_check_pin("000000"), "security PIN rejects wrong value"); + CHECK(!lz_store_security_clear_pin("000000", err, sizeof err) && + strcmp(err, "PIN rejected") == 0, + "security clear requires correct PIN"); + CHECK(lz_store_security_clear_pin("123456", err, sizeof err), + "security PIN verifier clears"); + CHECK(!lz_store_security_status(&sec) && !sec.configured, + "security status returns to no PIN"); + char diag[120]; + lz_store_security_selftest(diag, sizeof diag); + CHECK(strstr(diag, "PASS") != NULL && strstr(diag, "rounds=") != NULL, + "security PIN selftest reports parser/KDF pass"); + lz_store_init(NULL); + sim_reset_dir("lzdata_security"); + } + + /* 11. local app scanner: valid manifests become local apps; broken packages * are ignored before they can reach Home/App Store. */ { extern void lz_store_init(const char *datadir); @@ -946,7 +986,7 @@ static int codec_selftest(void) sim_reset_dir("lzdata_appscan"); } - /* 10. service-level SDK token injection: dynamic read-only values are + /* 12. service-level SDK token injection: dynamic read-only values are * expanded only when the matching permissions are declared. */ { extern void lz_store_init(const char *datadir); @@ -1018,7 +1058,7 @@ static int codec_selftest(void) sim_reset_dir("lzdata_apptokens"); } - /* 11. appfs root support: apps can be discovered from appfs even when + /* 13. appfs root support: apps can be discovered from appfs even when * SD-backed persistence is absent, and Files can expose both roots. */ { extern void lz_store_init(const char *datadir); @@ -1048,7 +1088,7 @@ static int codec_selftest(void) sim_reset_dir("lzdata_appfsroot"); } - /* 11. MeshCore Public-channel GRP_TXT: decode a known reference vector, + /* 14. MeshCore Public-channel GRP_TXT: decode a known reference vector, * reject a wrong key (MAC), and round-trip an encode. Vector generated * against the documented scheme (AES-128-ECB + HMAC-SHA256 trunc-2). */ { @@ -1082,7 +1122,7 @@ static int codec_selftest(void) strcmp(rm.text, "hi there") == 0, "MeshCore GRP_TXT round-trip fields"); } - /* 13. MeshCore DM (TXT_MSG): ECDH derive (vs orlp/standard reference) then a + /* 15. MeshCore DM (TXT_MSG): ECDH derive (vs orlp/standard reference) then a * full encode->parse->decode round-trip + ACK match + MAC tamper check. */ { uint8_t seedA[32], pubA[32], pubB[32], ref[32]; diff --git a/src/serial_cli.cpp b/src/serial_cli.cpp index b46d44c..d2119d0 100644 --- a/src/serial_cli.cpp +++ b/src/serial_cli.cpp @@ -69,6 +69,7 @@ static void cmd_help(void) " nodes list heard nodes\n" " send broadcast text on the channel\n" " stats radio TX/RX + airtime utilization\n" + " security [status|test|set |check |clear ]\n" " wifi [scan|on|off] wifi status / control\n" " sys battery, uptime, memory\n" " id this node's identity\n" @@ -366,6 +367,52 @@ static void cmd_stats(void) (unsigned)st.tx_count, (unsigned)st.rx_count, st.util_pct); } +static void cmd_security(char *args) +{ + if(args && strcmp(args, "test") == 0) { + char b[120]; + lz_svc_security_selftest(b, sizeof b); + Serial.println(b); + return; + } + if(args && strncmp(args, "set ", 4) == 0) { + char err[64] = {0}; + if(lz_svc_security_set_pin(args + 4, err, sizeof err)) + Serial.println("[ok] device PIN verifier set (data encryption not enabled yet)"); + else + Serial.printf("[err] %s\n", err[0] ? err : "PIN not set"); + return; + } + if(args && strncmp(args, "check ", 6) == 0) { + Serial.println(lz_svc_security_check_pin(args + 6) ? "[ok] PIN accepted" : "[err] PIN rejected"); + return; + } + if(args && strncmp(args, "clear ", 6) == 0) { + char err[64] = {0}; + if(lz_svc_security_clear_pin(args + 6, err, sizeof err)) + Serial.println("[ok] device PIN verifier cleared"); + else + Serial.printf("[err] %s\n", err[0] ? err : "PIN not cleared"); + return; + } + if(args && args[0] && strcmp(args, "status") != 0) { + Serial.println("usage: security [status|test|set |check |clear ]"); + return; + } + + lz_security_status_t st; + lz_svc_security_status(&st); + if(st.configured && st.valid) { + Serial.printf("security: PIN set rounds=%lu salt=%s encrypted-store=not-enabled\n", + (unsigned long)st.rounds, st.salt); + } else if(st.configured && !st.valid) { + Serial.printf("security: invalid verifier error=\"%s\"\n", st.error); + } else { + Serial.printf("security: no device PIN set (%s); encrypted-store=not-enabled\n", + st.error[0] ? st.error : "not configured"); + } +} + static void print_wifi_text(const char *text) { if(!text || !text[0]) { Serial.print("(none)"); return; } @@ -439,6 +486,7 @@ static void dispatch(char *line) else if(!strcmp(line, "nodes")) cmd_nodes(); else if(!strcmp(line, "send")) cmd_send(args); else if(!strcmp(line, "stats")) cmd_stats(); + else if(!strcmp(line, "security")) cmd_security(args); else if(!strcmp(line, "wifi")) cmd_wifi(args); else if(!strcmp(line, "sys")) cmd_sys(); else if(!strcmp(line, "id")) cmd_id(); diff --git a/src/services/mesh.h b/src/services/mesh.h index aacfc8d..b3a7945 100644 --- a/src/services/mesh.h +++ b/src/services/mesh.h @@ -34,6 +34,9 @@ extern "C" { #define LZ_LOCAL_APP_ACTION_MAX 2 #define LZ_LOCAL_APP_ACTION_EFFECT_MAX 32 #define LZ_LOCAL_APP_ACTION_BODY_MAX 192 +#define LZ_SECURITY_PIN_MIN 4 +#define LZ_SECURITY_PIN_MAX 12 +#define LZ_SECURITY_KDF_ROUNDS 2048u #define LZ_APP_PERM_DISPLAY 0x0001u #define LZ_APP_PERM_INPUT 0x0002u @@ -218,6 +221,14 @@ typedef struct { lz_local_app_action_t actions[LZ_LOCAL_APP_ACTION_MAX]; } lz_local_app_session_t; +typedef struct { + bool configured; /* a device PIN verifier exists */ + bool valid; /* false = security.cfg is corrupt/unsupported */ + uint32_t rounds; /* verifier KDF work factor */ + char salt[17]; /* hex salt, diagnostics only */ + char error[48]; /* unset / corrupt reason */ +} lz_security_status_t; + /* ---- lifecycle ---- */ void lz_svc_init(const char *datadir, bool seed_demo); /* datadir NULL = RAM only */ void lz_svc_loop(void); /* pump backend + timers */ @@ -234,6 +245,11 @@ bool lz_svc_app_data_usage(const lz_local_app_t *app, uint32_t *used, uint32_t * bool lz_svc_clear_app_data(const lz_local_app_t *app, char *err, int err_cap); bool lz_svc_start_local_app(const lz_local_app_t *app, lz_local_app_session_t *out); bool lz_svc_local_app_action(lz_local_app_session_t *session, int idx); +bool lz_svc_security_status(lz_security_status_t *out); +bool lz_svc_security_set_pin(const char *pin, char *err, int err_cap); +bool lz_svc_security_check_pin(const char *pin); +bool lz_svc_security_clear_pin(const char *pin, char *err, int err_cap); +int lz_svc_security_selftest(char *buf, int n); /* ---- nodes ---- */ int lz_svc_nodes(const lz_node_rt **out); /* all heard nodes */ diff --git a/src/services/mesh_core.c b/src/services/mesh_core.c index 665f367..5d944d8 100644 --- a/src/services/mesh_core.c +++ b/src/services/mesh_core.c @@ -25,6 +25,11 @@ bool lz_store_app_data_usage(const lz_local_app_t *app, uint32_t *used, uint32_t bool lz_store_clear_app_data(const lz_local_app_t *app, char *err, int err_cap); bool lz_store_start_local_app(const lz_local_app_t *app, lz_local_app_session_t *out); bool lz_store_local_app_action(lz_local_app_session_t *session, int idx); +bool lz_store_security_status(lz_security_status_t *out); +bool lz_store_security_set_pin(const char *pin, char *err, int err_cap); +bool lz_store_security_check_pin(const char *pin); +bool lz_store_security_clear_pin(const char *pin, char *err, int err_cap); +int lz_store_security_selftest(char *buf, int n); void lz_store_append(const char *addr, const lz_msg_rt *m); int lz_store_load_tail(const char *addr, lz_msg_rt *ring, int cap); bool lz_store_find_delivery(const char *addr, uint32_t pkt_id, lz_msg_rt *out); @@ -402,6 +407,31 @@ bool lz_svc_local_app_action(lz_local_app_session_t *session, int idx) return local_app_expand_session(session); } +bool lz_svc_security_status(lz_security_status_t *out) +{ + return lz_store_security_status(out); +} + +bool lz_svc_security_set_pin(const char *pin, char *err, int err_cap) +{ + return lz_store_security_set_pin(pin, err, err_cap); +} + +bool lz_svc_security_check_pin(const char *pin) +{ + return lz_store_security_check_pin(pin); +} + +bool lz_svc_security_clear_pin(const char *pin, char *err, int err_cap) +{ + return lz_store_security_clear_pin(pin, err, err_cap); +} + +int lz_svc_security_selftest(char *buf, int n) +{ + return lz_store_security_selftest(buf, n); +} + const char *lz_fmt_ago(uint32_t ts, char *buf, size_t n) { if(ts == 0) { snprintf(buf, n, "-"); return buf; } diff --git a/src/services/store.c b/src/services/store.c index 0d73515..ad5dc5a 100644 --- a/src/services/store.c +++ b/src/services/store.c @@ -17,11 +17,16 @@ * everything RAM-only. */ #include "mesh.h" +#include "mc_crypto.h" #include #include #include #include +#include #include +#ifdef LZ_TARGET_TDECK +#include "esp_system.h" +#endif #ifdef _WIN32 #include #else @@ -1190,6 +1195,263 @@ static void remove_legacy_wifi_file(void) remove(path); } +/* ---- device PIN verifier ---- + * + * First Phase 12 security increment. This stores only a salted, iterated + * verifier for an optional device PIN. It does not encrypt user data yet. + */ + +static bool is_hex_n(const char *s, int n) +{ + if(!s) return false; + for(int i = 0; i < n; i++) { + char c = s[i]; + bool ok = (c >= '0' && c <= '9') || (c >= 'a' && c <= 'f') || + (c >= 'A' && c <= 'F'); + if(!ok) return false; + } + return s[n] == 0; +} + +static void hex32(char out[65], const uint8_t in[32]) +{ + static const char *h = "0123456789abcdef"; + for(int i = 0; i < 32; i++) { + out[i * 2] = h[in[i] >> 4]; + out[i * 2 + 1] = h[in[i] & 15]; + } + out[64] = 0; +} + +static void set_security_err(char *err, int cap, const char *msg) +{ + if(err && cap > 0) snprintf(err, (size_t)cap, "%s", msg ? msg : "security error"); +} + +static bool valid_pin(const char *pin, char *err, int cap) +{ + if(!pin) { + set_security_err(err, cap, "missing PIN"); + return false; + } + int n = (int)strlen(pin); + if(n < LZ_SECURITY_PIN_MIN || n > LZ_SECURITY_PIN_MAX) { + set_security_err(err, cap, "PIN length must be 4-12 digits"); + return false; + } + for(int i = 0; i < n; i++) { + if(pin[i] < '0' || pin[i] > '9') { + set_security_err(err, cap, "PIN must use digits only"); + return false; + } + } + set_security_err(err, cap, ""); + return true; +} + +static uint32_t security_random32(void) +{ +#ifdef LZ_TARGET_TDECK + return esp_random(); +#else + static uint32_t ctr = 0x6c7a5049u; + uintptr_t mix = (uintptr_t)&ctr ^ (uintptr_t)g_dir; + uint32_t t = (uint32_t)time(NULL); + ctr = ctr * 1664525u + 1013904223u + t + (uint32_t)mix; + return ctr ^ (uint32_t)(mix >> 16); +#endif +} + +static void security_make_salt(char out[17]) +{ + uint8_t raw[8]; + uint32_t a = security_random32(); + uint32_t b = security_random32(); + memcpy(raw, &a, 4); + memcpy(raw + 4, &b, 4); + static const char *h = "0123456789abcdef"; + for(int i = 0; i < 8; i++) { + out[i * 2] = h[raw[i] >> 4]; + out[i * 2 + 1] = h[raw[i] & 15]; + } + out[16] = 0; +} + +static void security_pin_kdf(const char *pin, const char *salt, uint32_t rounds, char out_hex[65]) +{ + uint8_t digest[32]; + lz_sha256_ctx c; + lz_sha256_init(&c); + lz_sha256_update(&c, (const uint8_t *)"limitlezz.pin.v1", 16); + lz_sha256_update(&c, (const uint8_t *)salt, strlen(salt)); + lz_sha256_update(&c, (const uint8_t *)pin, strlen(pin)); + lz_sha256_final(&c, digest); + + if(rounds < 1) rounds = 1; + for(uint32_t i = 1; i < rounds; i++) { + uint8_t ctr[4]; + ctr[0] = (uint8_t)(i & 0xFF); + ctr[1] = (uint8_t)((i >> 8) & 0xFF); + ctr[2] = (uint8_t)((i >> 16) & 0xFF); + ctr[3] = (uint8_t)((i >> 24) & 0xFF); + lz_sha256_init(&c); + lz_sha256_update(&c, digest, sizeof digest); + lz_sha256_update(&c, (const uint8_t *)salt, strlen(salt)); + lz_sha256_update(&c, ctr, sizeof ctr); + lz_sha256_final(&c, digest); + } + hex32(out_hex, digest); +} + +static bool security_load(char salt[17], char verifier[65], uint32_t *rounds, + lz_security_status_t *status) +{ + if(status) { + memset(status, 0, sizeof *status); + status->valid = true; + snprintf(status->error, sizeof status->error, "not configured"); + } + if(!g_persist) { + if(status) snprintf(status->error, sizeof status->error, "storage unavailable"); + return false; + } + + char path[128]; + path_for(path, sizeof path, "security.cfg"); + FILE *f = fopen(path, "r"); + if(!f) return false; + char line[180]; + bool have_line = fgets(line, sizeof line, f) != NULL; + fclose(f); + if(!have_line) { + if(status) { + status->configured = true; + status->valid = false; + snprintf(status->error, sizeof status->error, "empty verifier"); + } + return false; + } + line[strcspn(line, "\r\n")] = 0; + char *cur = line; + char *ver = field(&cur), *kind = field(&cur), *roundf = field(&cur), + *saltf = field(&cur), *hashf = cur; + char *rend = NULL; + unsigned long r = roundf ? strtoul(roundf, &rend, 10) : 0; + bool ok = ver && strcmp(ver, "1") == 0 && + kind && strcmp(kind, "pin-sha256") == 0 && + roundf && saltf && hashf && + is_hex_n(saltf, 16) && is_hex_n(hashf, 64) && + rend && *rend == 0 && + r >= 1 && r <= 65536u; + if(!ok) { + if(status) { + status->configured = true; + status->valid = false; + snprintf(status->error, sizeof status->error, "corrupt verifier"); + } + return false; + } + if(salt) snprintf(salt, 17, "%s", saltf); + if(verifier) snprintf(verifier, 65, "%s", hashf); + if(rounds) *rounds = (uint32_t)r; + if(status) { + status->configured = true; + status->valid = true; + status->rounds = (uint32_t)r; + snprintf(status->salt, sizeof status->salt, "%s", saltf); + snprintf(status->error, sizeof status->error, "ok"); + } + return true; +} + +bool lz_store_security_status(lz_security_status_t *out) +{ + return security_load(NULL, NULL, NULL, out); +} + +bool lz_store_security_set_pin(const char *pin, char *err, int err_cap) +{ + if(!valid_pin(pin, err, err_cap)) return false; + if(!g_persist) { + set_security_err(err, err_cap, "storage unavailable"); + return false; + } + char salt[17], verifier[65]; + security_make_salt(salt); + security_pin_kdf(pin, salt, LZ_SECURITY_KDF_ROUNDS, verifier); + + char path[128], tmp[132]; + path_for(path, sizeof path, "security.cfg"); + snprintf(tmp, sizeof tmp, "%s.tmp", path); + FILE *f = fopen(tmp, "w"); + if(!f) { + set_security_err(err, err_cap, "verifier write failed"); + return false; + } + fprintf(f, "1|pin-sha256|%u|%s|%s\n", + (unsigned)LZ_SECURITY_KDF_ROUNDS, salt, verifier); + fclose(f); + remove(path); + if(rename(tmp, path) != 0) { + remove(tmp); + set_security_err(err, err_cap, "verifier save failed"); + return false; + } + set_security_err(err, err_cap, "ok"); + return true; +} + +bool lz_store_security_check_pin(const char *pin) +{ + if(!valid_pin(pin, NULL, 0)) return false; + char salt[17], verifier[65], probe[65]; + uint32_t rounds = 0; + if(!security_load(salt, verifier, &rounds, NULL)) return false; + security_pin_kdf(pin, salt, rounds, probe); + unsigned diff = 0; + for(int i = 0; i < 64; i++) diff |= (unsigned char)(probe[i] ^ verifier[i]); + return diff == 0; +} + +bool lz_store_security_clear_pin(const char *pin, char *err, int err_cap) +{ + lz_security_status_t st; + bool configured = security_load(NULL, NULL, NULL, &st); + if(st.configured && !st.valid) { + set_security_err(err, err_cap, st.error); + return false; + } + if(!configured) { + set_security_err(err, err_cap, "not configured"); + return true; + } + if(!lz_store_security_check_pin(pin)) { + set_security_err(err, err_cap, "PIN rejected"); + return false; + } + char path[128]; + path_for(path, sizeof path, "security.cfg"); + remove(path); + set_security_err(err, err_cap, "ok"); + return true; +} + +int lz_store_security_selftest(char *buf, int n) +{ + if(!buf || n <= 0) return 0; + char err[64] = {0}; + char a[65], b[65], c[65]; + security_pin_kdf("123456", "0011223344556677", 32, a); + security_pin_kdf("123456", "0011223344556677", 32, b); + security_pin_kdf("123457", "0011223344556677", 32, c); + bool pass = valid_pin("1234", err, sizeof err) && + !valid_pin("12ab", err, sizeof err) && + strcmp(a, b) == 0 && strcmp(a, c) != 0 && is_hex_n(a, 64); + return snprintf(buf, (size_t)n, "PIN verifier selftest: %s min=%d max=%d rounds=%u", + pass ? "PASS" : "FAIL", LZ_SECURITY_PIN_MIN, + LZ_SECURITY_PIN_MAX, (unsigned)LZ_SECURITY_KDF_ROUNDS); +} + int lz_store_load_threads(lz_thread_rt *out, int cap) { if(!g_persist) return 0; From e50cd5df14613165e7069eb71e2699081690d1ce Mon Sep 17 00:00:00 2001 From: n30nex Date: Fri, 19 Jun 2026 05:34:51 -0400 Subject: [PATCH 19/64] Add Meshtastic protobuf guard vectors --- README.md | 10 +++-- docs/tdeck-feature-inventory.md | 2 +- docs/tdeck-firmware-roadmap.md | 7 ++- sim/main_sim.c | 76 ++++++++++++++++++++++++++++----- src/services/mtproto.c | 72 ++++++++++++++++++------------- 5 files changed, 121 insertions(+), 46 deletions(-) diff --git a/README.md b/README.md index 722c799..1ac6b47 100644 --- a/README.md +++ b/README.md @@ -258,10 +258,12 @@ to use an older run unless `--allow-latest-success` is passed. CI runs the native simulator build, native codec selftest, deterministic simulator scenario, screenshot generation, T-Deck firmware build, and T-Deck size report -in `.github/workflows/firmware.yml`. It also enforces the current T-Deck budget -gate (2,200,000 bytes for `firmware.bin`, 307,200 bytes static RAM), writes the -result into `FLASH_MANIFEST.txt`, then uploads the firmware artifacts from -`.pio/build/tdeck` plus the generated simulator screenshots. +in `.github/workflows/firmware.yml`. The codec selftest includes Meshtastic +channel/frame/protobuf guard vectors plus MeshCore crypto references. CI also +enforces the current T-Deck budget gate (2,200,000 bytes for `firmware.bin`, +307,200 bytes static RAM), writes the result into `FLASH_MANIFEST.txt`, then +uploads the firmware artifacts from `.pio/build/tdeck` plus the generated +simulator screenshots. Current footprint: ~1.48 MB flash (28.2% of the 5 MB OTA slot), 271 KB static RAM (82.7%) — the rest of RAM is PSRAM-backed double framebuffers. Message history, diff --git a/docs/tdeck-feature-inventory.md b/docs/tdeck-feature-inventory.md index 416182f..9d1f926 100644 --- a/docs/tdeck-feature-inventory.md +++ b/docs/tdeck-feature-inventory.md @@ -110,7 +110,7 @@ Status labels: | Feedback Manager | Planned | Design spec section 8 | Centralize LED, buzzer, keyboard/display feedback and DND. | | Emergency beacon | Planned | Design spec section 12, disabled Emergency UI row | Requires Feedback Manager and dual-network messaging. | | BLE companion | Partial, needs validation | NimBLE-based Meshtastic GATT service, official UUIDs, raw `ToRadio` writes, queued `FromRadio` reads, `FromNum` notifications, UI toggle, and serial selftest/status | Validate with the official Meshtastic app over BLE before calling V0.5 complete. | -| CI and release checks | Partial | `.github/workflows/firmware.yml` runs native simulator build, native protocol selftest, deterministic simulator scenario, screenshot generation, T-Deck build, size reporting, an explicit firmware/static-RAM budget gate, and artifact upload with budget metadata plus screenshots | Add protocol vectors beyond the native selftest and hardware evidence gates. | +| CI and release checks | Partial | `.github/workflows/firmware.yml` runs native simulator build, native protocol selftest, deterministic simulator scenario, screenshot generation, T-Deck build, size reporting, an explicit firmware/static-RAM budget gate, and artifact upload with budget metadata plus screenshots. The codec selftest now includes Meshtastic channel/hash, frame-boundary, and malformed protobuf guard vectors alongside existing MeshCore crypto references. | Add broader stock-device packet captures and hardware evidence gates. | ## Completion Criteria diff --git a/docs/tdeck-firmware-roadmap.md b/docs/tdeck-firmware-roadmap.md index f987e35..bb78308 100644 --- a/docs/tdeck-firmware-roadmap.md +++ b/docs/tdeck-firmware-roadmap.md @@ -203,6 +203,10 @@ Deliverables: - Add screenshot and deterministic scenario coverage to CI. Implemented: Firmware CI now runs `--simtest`, generates native simulator screenshots, and uploads the screenshot artifact for release review. +- Add protocol unit vectors to CI. First slice implemented in the native codec + selftest: Meshtastic custom 32-byte PSK/channel hash, text-frame buffer + bounds, truncated headers, and malformed protobuf tag/value/length rejection + for Data, POSITION, and TELEMETRY decoders. - Clean up dead demo data and stale comments that no longer match product state. - Add basic emoji rendering/input support appropriate for the T-Deck screen and memory budget. - Re-run hardware dogfood tests on Meshtastic-only, MeshCore-only, and split-airtime modes. @@ -425,7 +429,8 @@ Deliverables: - simulator selftest - screenshot generation where host supports SDL2 - size budget - - protocol unit tests/test vectors + - protocol unit tests/test vectors. First Meshtastic parser guard vectors are + implemented in `--selftest`; broader stock-device captures remain. - Hardware test matrix: - T-Deck and T-Deck Plus - SD present/absent diff --git a/sim/main_sim.c b/sim/main_sim.c index 9c54b9b..9b1e7bb 100644 --- a/sim/main_sim.c +++ b/sim/main_sim.c @@ -597,7 +597,63 @@ static int codec_selftest(void) CHECK(pl > 0 && mt_data_decode(pb, pl, &back), "routing Data encodes/decodes"); CHECK(back.request_id == 0xdeadbeef, "request_id (fixed32) round-trips"); - /* 5. POSITION decode: lat/lon fixed32, altitude varint, precision_bits */ + /* 5. Meshtastic guard vectors: channel variants, tight buffers, and + * malformed protobuf inputs must fail closed instead of decoding junk. */ + { + uint8_t psk32[32]; + for(int i = 0; i < 32; i++) psk32[i] = (uint8_t)(i + 1); + mt_set_channel("FieldOps", psk32, (int)sizeof psk32); + CHECK(mt_channel_hash() == 0x2e, "channel hash(FieldOps,32B PSK) == 0x2e"); + + uint8_t cframe[256]; + int cflen = mt_build_text(cframe, sizeof cframe, from, to, 0xabcdef01u, 5, false, "custom PSK"); + mt_frame_t cf; + CHECK(cflen > MT_HEADER_LEN && mt_header_read(cframe, cflen, &cf) && + cf.channel_hash == 0x2e && cf.hop_limit == 5 && !cf.want_ack, + "custom channel text frame carries expected hash/flags"); + + uint8_t bad_psk[3] = { 1, 2, 3 }; + mt_set_channel("LongFast", bad_psk, (int)sizeof bad_psk); + CHECK(mt_channel_hash() == 0x08, "invalid PSK length falls back to LongFast default"); + mt_set_channel("LongFast", NULL, 0); + + uint8_t tiny[20]; + CHECK(mt_build_text(tiny, sizeof tiny, from, to, 0x10203040u, 3, false, + "this message cannot fit") < 0, + "mt_build_text rejects undersized output buffer"); + + mt_frame_t short_header; + CHECK(!mt_header_read(frame, MT_HEADER_LEN - 1, &short_header), + "header truncation rejected"); + + mt_data_t bd; + uint8_t bad_tag[] = { 0x80 }; + CHECK(!mt_data_decode(bad_tag, sizeof bad_tag, &bd), + "Data rejects unterminated tag varint"); + uint8_t bad_value[] = { 0x08, 0x80 }; + CHECK(!mt_data_decode(bad_value, sizeof bad_value, &bd), + "Data rejects unterminated value varint"); + uint8_t bad_len[] = { 0x12, 0x80 }; + CHECK(!mt_data_decode(bad_len, sizeof bad_len, &bd), + "Data rejects unterminated length varint"); + uint8_t bad_unknown[] = { 0x48, 0x80 }; + CHECK(!mt_data_decode(bad_unknown, sizeof bad_unknown, &bd), + "Data rejects unterminated unknown varint field"); + uint8_t bad_wire[] = { 0x0F }; + CHECK(!mt_data_decode(bad_wire, sizeof bad_wire, &bd), + "Data rejects unsupported wire type"); + + mt_position_t bp; + uint8_t bad_pos[] = { 0x18, 0x80 }; + CHECK(!mt_position_decode(bad_pos, sizeof bad_pos, &bp), + "POSITION rejects unterminated altitude varint"); + mt_telemetry_t bt; + uint8_t bad_tel[] = { 0x12, 0x80 }; + CHECK(!mt_telemetry_decode(bad_tel, sizeof bad_tel, &bt), + "TELEMETRY rejects unterminated submessage length"); + } + + /* 6. POSITION decode: lat/lon fixed32, altitude varint, precision_bits */ { uint8_t pos[] = { 0x0D, 0x44,0x33,0x22,0x11, /* field 1 latitude_i = 0x11223344 */ @@ -623,7 +679,7 @@ static int codec_selftest(void) CHECK(!mt_position_decode(ovf, sizeof ovf, &po), "POSITION oversized length rejected"); } - /* 6. TELEMETRY device metrics: battery varint, voltage float, uptime varint */ + /* 7. TELEMETRY device metrics: battery varint, voltage float, uptime varint */ { float voltage = 4.10f; uint8_t dm[16]; int dn = 0; @@ -640,7 +696,7 @@ static int codec_selftest(void) CHECK(t.has_uptime && t.uptime_s == 3600, "TELEMETRY uptime"); } - /* 7. TELEMETRY env metrics: a hostile NaN float must decode without OOB and + /* 8. TELEMETRY env metrics: a hostile NaN float must decode without OOB and * be preserved as NaN so the clamp layer (not the decoder) rejects it */ { uint32_t nanbits = 0x7FC00000u; float humidity = 55.0f, pressure = 1013.0f; @@ -663,7 +719,7 @@ static int codec_selftest(void) CHECK(!mt_telemetry_decode(bad, sizeof bad, &tb), "TELEMETRY oversized submsg rejected"); } - /* 8. store delivery-metadata round-trip: updating a DM that is NOT the first + /* 9. store delivery-metadata round-trip: updating a DM that is NOT the first * self-record must not desync the scan over the preceding v3 meta record. */ { extern void lz_store_init(const char *datadir); @@ -697,7 +753,7 @@ static int codec_selftest(void) lz_store_init(NULL); /* back to RAM-only */ } - /* 9. Wi-Fi credential store round-trip. T-Deck uses an NVS backend under the + /* 10. Wi-Fi credential store round-trip. T-Deck uses an NVS backend under the * same API; native keeps the file path for simulator repeatability. */ { extern void lz_store_init(const char *datadir); @@ -718,7 +774,7 @@ static int codec_selftest(void) lz_store_init(NULL); } - /* 10. local app scanner: valid manifests become local apps; broken packages + /* 11. local app scanner: valid manifests become local apps; broken packages * are ignored before they can reach Home/App Store. */ { extern void lz_store_init(const char *datadir); @@ -946,7 +1002,7 @@ static int codec_selftest(void) sim_reset_dir("lzdata_appscan"); } - /* 10. service-level SDK token injection: dynamic read-only values are + /* 12. service-level SDK token injection: dynamic read-only values are * expanded only when the matching permissions are declared. */ { extern void lz_store_init(const char *datadir); @@ -1018,7 +1074,7 @@ static int codec_selftest(void) sim_reset_dir("lzdata_apptokens"); } - /* 11. appfs root support: apps can be discovered from appfs even when + /* 13. appfs root support: apps can be discovered from appfs even when * SD-backed persistence is absent, and Files can expose both roots. */ { extern void lz_store_init(const char *datadir); @@ -1048,7 +1104,7 @@ static int codec_selftest(void) sim_reset_dir("lzdata_appfsroot"); } - /* 11. MeshCore Public-channel GRP_TXT: decode a known reference vector, + /* 14. MeshCore Public-channel GRP_TXT: decode a known reference vector, * reject a wrong key (MAC), and round-trip an encode. Vector generated * against the documented scheme (AES-128-ECB + HMAC-SHA256 trunc-2). */ { @@ -1082,7 +1138,7 @@ static int codec_selftest(void) strcmp(rm.text, "hi there") == 0, "MeshCore GRP_TXT round-trip fields"); } - /* 13. MeshCore DM (TXT_MSG): ECDH derive (vs orlp/standard reference) then a + /* 15. MeshCore DM (TXT_MSG): ECDH derive (vs orlp/standard reference) then a * full encode->parse->decode round-trip + ACK match + MAC tamper check. */ { uint8_t seedA[32], pubA[32], pubB[32], ref[32]; diff --git a/src/services/mtproto.c b/src/services/mtproto.c index f5be3b8..6015ebe 100644 --- a/src/services/mtproto.c +++ b/src/services/mtproto.c @@ -167,25 +167,29 @@ static int pb_varint(uint8_t *b, uint64_t v) return i; } -static uint64_t pb_read_varint(const uint8_t *b, int len, int *pos) +static bool pb_read_varint(const uint8_t *b, int len, int *pos, uint64_t *out) { uint64_t v = 0; int shift = 0; while(*pos < len && shift < 64) { uint8_t byte = b[(*pos)++]; v |= (uint64_t)(byte & 0x7F) << shift; - if(!(byte & 0x80)) break; + if(!(byte & 0x80)) { + if(out) *out = v; + return true; + } shift += 7; } - return v; + return false; } static bool pb_skip(const uint8_t *b, int len, int *pos, int wire) { if(wire == 0) { - pb_read_varint(b, len, pos); - return *pos <= len; + uint64_t ignored; + return pb_read_varint(b, len, pos, &ignored); } else if(wire == 2) { - uint64_t l = pb_read_varint(b, len, pos); + uint64_t l; + if(!pb_read_varint(b, len, pos, &l)) return false; if((uint64_t)*pos + l > (uint64_t)len) return false; *pos += (int)l; return true; @@ -252,13 +256,17 @@ bool mt_data_decode(const uint8_t *buf, int len, mt_data_t *d) memset(d, 0, sizeof *d); int pos = 0; while(pos < len) { - uint64_t tag = pb_read_varint(buf, len, &pos); + uint64_t tag; + if(!pb_read_varint(buf, len, &pos, &tag)) return false; int field = (int)(tag >> 3); int wire = (int)(tag & 0x07); if(field == 1 && wire == 0) { - d->portnum = (uint8_t)pb_read_varint(buf, len, &pos); + uint64_t v; + if(!pb_read_varint(buf, len, &pos, &v)) return false; + d->portnum = (uint8_t)v; } else if(field == 2 && wire == 2) { - uint64_t l = pb_read_varint(buf, len, &pos); + uint64_t l; + if(!pb_read_varint(buf, len, &pos, &l)) return false; /* compare as uint64: a huge attacker length must not cast to a * negative int and slip past the bound (OOB read from a packet) */ if((uint64_t)pos + l > (uint64_t)len) return false; @@ -267,23 +275,16 @@ bool mt_data_decode(const uint8_t *buf, int len, mt_data_t *d) d->plen = (uint8_t)copy; pos += (int)l; /* l <= len-pos here, so the cast is safe */ } else if(field == 3 && wire == 0) { - d->want_response = pb_read_varint(buf, len, &pos) != 0; + uint64_t v; + if(!pb_read_varint(buf, len, &pos, &v)) return false; + d->want_response = v != 0; } else if(field == 6 && wire == 5) { /* request_id: fixed32 LE */ if(pos + 4 > len) return false; d->request_id = (uint32_t)buf[pos] | ((uint32_t)buf[pos+1] << 8) | ((uint32_t)buf[pos+2] << 16) | ((uint32_t)buf[pos+3] << 24); pos += 4; - } else { - /* skip unknown field by wire type (bounded against malformed input) */ - if(wire == 0) pb_read_varint(buf, len, &pos); - else if(wire == 2) { - uint64_t l = pb_read_varint(buf, len, &pos); - if((uint64_t)pos + l > (uint64_t)len) return false; - pos += (int)l; - } - else if(wire == 5) pos += 4; - else if(wire == 1) pos += 8; - else return false; + } else if(!pb_skip(buf, len, &pos, wire)) { + return false; } } return true; @@ -296,7 +297,8 @@ bool mt_position_decode(const uint8_t *buf, int len, mt_position_t *p) int pos = 0; bool ok = true; while(pos < len) { - uint64_t tag = pb_read_varint(buf, len, &pos); + uint64_t tag; + if(!pb_read_varint(buf, len, &pos, &tag)) return false; int field = (int)(tag >> 3); int wire = (int)(tag & 0x07); if(field == 1 && wire == 5) { @@ -306,12 +308,15 @@ bool mt_position_decode(const uint8_t *buf, int len, mt_position_t *p) p->longitude_i = (int32_t)pb_read_fixed32(buf, len, &pos, &ok); p->has_lon = ok; } else if(field == 3 && wire == 0) { - p->altitude_m = (int32_t)pb_read_varint(buf, len, &pos); + uint64_t v; + if(!pb_read_varint(buf, len, &pos, &v)) return false; + p->altitude_m = (int32_t)v; p->has_alt = true; } else if((field == 4 || field == 7) && wire == 5) { p->time = pb_read_fixed32(buf, len, &pos, &ok); } else if(field == 23 && wire == 0) { - uint64_t v = pb_read_varint(buf, len, &pos); + uint64_t v; + if(!pb_read_varint(buf, len, &pos, &v)) return false; p->precision_bits = v > 255 ? 255 : (uint8_t)v; } else if(!pb_skip(buf, len, &pos, wire)) { return false; @@ -326,18 +331,22 @@ static bool mt_device_metrics_decode(const uint8_t *buf, int len, mt_telemetry_t int pos = 0; bool ok = true; while(pos < len) { - uint64_t tag = pb_read_varint(buf, len, &pos); + uint64_t tag; + if(!pb_read_varint(buf, len, &pos, &tag)) return false; int field = (int)(tag >> 3); int wire = (int)(tag & 0x07); if(field == 1 && wire == 0) { - uint64_t v = pb_read_varint(buf, len, &pos); + uint64_t v; + if(!pb_read_varint(buf, len, &pos, &v)) return false; t->battery_level = v > 255 ? 255 : (uint8_t)v; t->has_battery = true; } else if(field == 2 && wire == 5) { t->voltage = pb_read_float(buf, len, &pos, &ok); t->has_voltage = ok; } else if(field == 5 && wire == 0) { - t->uptime_s = (uint32_t)pb_read_varint(buf, len, &pos); + uint64_t v; + if(!pb_read_varint(buf, len, &pos, &v)) return false; + t->uptime_s = (uint32_t)v; t->has_uptime = true; } else if(!pb_skip(buf, len, &pos, wire)) { return false; @@ -352,7 +361,8 @@ static bool mt_environment_metrics_decode(const uint8_t *buf, int len, mt_teleme int pos = 0; bool ok = true; while(pos < len) { - uint64_t tag = pb_read_varint(buf, len, &pos); + uint64_t tag; + if(!pb_read_varint(buf, len, &pos, &tag)) return false; int field = (int)(tag >> 3); int wire = (int)(tag & 0x07); if(field == 1 && wire == 5) { @@ -378,11 +388,13 @@ bool mt_telemetry_decode(const uint8_t *buf, int len, mt_telemetry_t *t) memset(t, 0, sizeof *t); int pos = 0; while(pos < len) { - uint64_t tag = pb_read_varint(buf, len, &pos); + uint64_t tag; + if(!pb_read_varint(buf, len, &pos, &tag)) return false; int field = (int)(tag >> 3); int wire = (int)(tag & 0x07); if((field == 2 || field == 3) && wire == 2) { - uint64_t l = pb_read_varint(buf, len, &pos); + uint64_t l; + if(!pb_read_varint(buf, len, &pos, &l)) return false; if((uint64_t)pos + l > (uint64_t)len) return false; bool ok = field == 2 ? mt_device_metrics_decode(buf + pos, (int)l, t) From c60825d4b4885cf91f80ab3739c7d00d5fd9304b Mon Sep 17 00:00:00 2001 From: n30nex Date: Fri, 19 Jun 2026 05:43:45 -0400 Subject: [PATCH 20/64] Add T-Deck smoke reattach retry --- README.md | 3 + docs/tdeck-firmware-roadmap.md | 2 +- docs/tdeck-hardware-dogfood-checklist.md | 4 ++ scripts/tdeck_smoke.py | 88 ++++++++++++++++++------ 4 files changed, 74 insertions(+), 23 deletions(-) diff --git a/README.md b/README.md index 722c799..9e255c8 100644 --- a/README.md +++ b/README.md @@ -242,6 +242,9 @@ On the Windows COM8 T-Deck, the ROM stub upload path can be flaky. Use through PlatformIO's packaged `esptool.py --no-stub`, and run the serial CLI smoke in one pass. PowerShell users can run the same flow with `powershell -ExecutionPolicy Bypass -File scripts\tdeck_smoke.ps1 -Port COM8 -NoStubUpload`. +For the standard read-only smoke commands, `tdeck_smoke.py` now automatically +tries one no-reset serial reattach if the first post-flash console attach times +out; custom command lists do not retry unless `--reattach-retries` is provided. **GitHub Actions artifact → local T-Deck** (fast remote build, local hardware proof): diff --git a/docs/tdeck-firmware-roadmap.md b/docs/tdeck-firmware-roadmap.md index f987e35..2bcc7cb 100644 --- a/docs/tdeck-firmware-roadmap.md +++ b/docs/tdeck-firmware-roadmap.md @@ -52,7 +52,7 @@ Deliverables: - Confirm and encode the correct LilyGO T-Deck board profile: 16 MB flash, PSRAM, upload flash size, partition compatibility, and memory type. - Add CI for `pio run -e tdeck` with firmware size reporting. Implemented in `.github/workflows/firmware.yml`; CI builds the T-Deck target, captures `pio run -t size`, and uploads firmware artifacts. -- Make CI artifacts usable for local hardware smoke on slow hosts. Implemented: Firmware CI now uploads a flash bundle with bootloader, partitions, boot app, firmware, ELF/map, and manifest; `scripts/fetch_tdeck_artifact.py` downloads the exact-commit artifact for local `scripts/tdeck_smoke.py --skip-build` flashing. +- Make CI artifacts usable for local hardware smoke on slow hosts. Implemented: Firmware CI now uploads a flash bundle with bootloader, partitions, boot app, firmware, ELF/map, and manifest; `scripts/fetch_tdeck_artifact.py` downloads the exact-commit artifact for local `scripts/tdeck_smoke.py --skip-build` flashing; the standard read-only smoke command set automatically retries one no-reset serial reattach after transient post-flash console timeouts. - Add a simulator CI path or clear host-specific setup docs for SDL2. Implemented in `.github/workflows/firmware.yml`; Ubuntu CI installs SDL2, Windows can install the local SDL2 bundle with `scripts/ensure_sdl2_windows.ps1`, and `pio run -e native` runs through `scripts/pio_native_sdl2.py`. - Make `pio run -e native` fail clearly when SDL2 is absent. Implemented by `scripts/pio_native_sdl2.py`; Linux/macOS use `sdl2-config` or `pkg-config`, while Windows uses `SDL2_DIR` or the local `.deps` bundle. - Add a release checklist covering build, flash, boot log, display, touch, keyboard, trackball, SD, radio, Wi-Fi, companion, and sleep. diff --git a/docs/tdeck-hardware-dogfood-checklist.md b/docs/tdeck-hardware-dogfood-checklist.md index e152594..45070ea 100644 --- a/docs/tdeck-hardware-dogfood-checklist.md +++ b/docs/tdeck-hardware-dogfood-checklist.md @@ -34,6 +34,10 @@ dogfood belong to the later roadmap phases. or the same command with the Linux serial device path. - For serial CLI smoke without flashing, run `python scripts/tdeck_smoke.py --skip-upload --port COM8` on the Windows rig or pass the Linux/macOS device path such as `/dev/ttyACM0`. +- The standard read-only smoke command set retries one no-reset serial reattach + automatically after a timeout, which covers the COM8 post-flash USB handoff + case without reflashing. For custom command lists, pass `--reattach-retries N` + only when repeating the command sequence is safe. - Open the USB console at 115200 baud. - Capture the boot banner and every `[ok]` or failure line. - Confirm display, touch, keyboard, trackball, SD, SX1262, Wi-Fi state, battery, and time source are reported. diff --git a/scripts/tdeck_smoke.py b/scripts/tdeck_smoke.py index 63b1431..1d93a3c 100644 --- a/scripts/tdeck_smoke.py +++ b/scripts/tdeck_smoke.py @@ -30,6 +30,11 @@ def run(cmd: list[str], cwd: Path) -> None: subprocess.run(cmd, cwd=cwd, check=True) +def run_status(cmd: list[str], cwd: Path) -> int: + print("[smoke] " + " ".join(cmd), flush=True) + return subprocess.run(cmd, cwd=cwd, check=False).returncode + + def platformio_core_dir() -> Path: override = os.environ.get("PLATFORMIO_CORE_DIR") if override: @@ -130,6 +135,20 @@ def main() -> int: parser.add_argument("--open-timeout", type=float, default=60.0) parser.add_argument("--boot-timeout", type=float, default=45.0) parser.add_argument("--timeout", type=float, default=30.0) + parser.add_argument( + "--reattach-retries", + type=int, + help=( + "Retry the serial smoke by reopening the port without reflashing or resetting. " + "Defaults to 1 for the standard read-only smoke commands, 0 for custom commands." + ), + ) + parser.add_argument( + "--reattach-timeout", + type=float, + default=90.0, + help="Command/boot timeout used by reattach retries.", + ) parser.add_argument("--no-expect", action="store_true") parser.add_argument("--commands", nargs="*", default=DEFAULT_COMMANDS) args = parser.parse_args() @@ -146,29 +165,54 @@ def main() -> int: raise SystemExit("--skip-build/--artifact-dir require --no-stub-upload") run(["pio", "run", "-e", args.env, "-t", "upload", "--upload-port", args.port], cwd=project_dir) + reattach_retries = args.reattach_retries + if reattach_retries is None: + reattach_retries = 1 if args.commands == DEFAULT_COMMANDS else 0 + harness = project_dir / "scripts" / "serial_harness.py" - cmd = [ - sys.executable, - str(harness), - "--port", - args.port, - "--baud", - str(args.baud), - "--open-timeout", - str(args.open_timeout), - "--boot-timeout", - str(args.boot_timeout), - "--timeout", - str(args.timeout), - ] - if args.no_expect: - cmd.append("--no-expect") - if args.reset or (not args.skip_upload and not args.no_reset_after_upload): - cmd.append("--reset") - cmd.append("--commands") - cmd.extend(args.commands) - run(cmd, cwd=project_dir) - return 0 + + def harness_cmd(timeout: float, boot_timeout: float, reset: bool) -> list[str]: + cmd = [ + sys.executable, + str(harness), + "--port", + args.port, + "--baud", + str(args.baud), + "--open-timeout", + str(args.open_timeout), + "--boot-timeout", + str(boot_timeout), + "--timeout", + str(timeout), + ] + if args.no_expect: + cmd.append("--no-expect") + if reset: + cmd.append("--reset") + cmd.append("--commands") + cmd.extend(args.commands) + return cmd + + reset_first = args.reset or (not args.skip_upload and not args.no_reset_after_upload) + first = harness_cmd(args.timeout, args.boot_timeout, reset_first) + rc = run_status(first, cwd=project_dir) + if rc == 0: + return 0 + + for attempt in range(1, max(0, reattach_retries) + 1): + print( + f"[smoke] serial smoke failed with exit {rc}; " + f"reattach retry {attempt}/{reattach_retries} without reflashing/reset", + flush=True, + ) + retry_timeout = max(args.timeout, args.reattach_timeout) + retry_boot = max(args.boot_timeout, args.reattach_timeout) + rc = run_status(harness_cmd(retry_timeout, retry_boot, False), cwd=project_dir) + if rc == 0: + return 0 + + return rc if __name__ == "__main__": From b7f21cde667322dd68e7a608e509a97504af5a89 Mon Sep 17 00:00:00 2001 From: n30nex Date: Fri, 19 Jun 2026 05:54:18 -0400 Subject: [PATCH 21/64] Add T-Deck release evidence checklist --- README.md | 4 + docs/tdeck-feature-inventory.md | 6 +- docs/tdeck-firmware-roadmap.md | 4 +- docs/tdeck-release-checklist.md | 142 +++++++++++++++++++++ scripts/release_evidence.py | 217 ++++++++++++++++++++++++++++++++ 5 files changed, 368 insertions(+), 5 deletions(-) create mode 100644 docs/tdeck-release-checklist.md create mode 100644 scripts/release_evidence.py diff --git a/README.md b/README.md index 722c799..3285faf 100644 --- a/README.md +++ b/README.md @@ -144,6 +144,7 @@ iPhone-style dark look (status bar, battery glyph, grouped settings cards). - [`docs/tdeck-feature-inventory.md`](docs/tdeck-feature-inventory.md) - feature-by-feature implementation inventory. - [`docs/tdeck-firmware-roadmap.md`](docs/tdeck-firmware-roadmap.md) - roadmap to a complete T-Deck firmware. - [`docs/tdeck-hardware-dogfood-checklist.md`](docs/tdeck-hardware-dogfood-checklist.md) - stock-device hardware proof checklist. +- [`docs/tdeck-release-checklist.md`](docs/tdeck-release-checklist.md) - slow-host Actions artifact and COM8 release evidence checklist. ![screens](docs/screens.png) @@ -256,6 +257,9 @@ The fetch helper uses the current branch and current commit by default, then downloads the matching successful `Firmware CI` artifact with `gh`. It refuses to use an older run unless `--allow-latest-success` is passed. +For release PRs, `python scripts/release_evidence.py --artifact-dir .pio/ci-artifacts/tdeck --port COM8` +prints the required local, Actions, artifact, and COM8 evidence checklist. + CI runs the native simulator build, native codec selftest, deterministic simulator scenario, screenshot generation, T-Deck firmware build, and T-Deck size report in `.github/workflows/firmware.yml`. It also enforces the current T-Deck budget diff --git a/docs/tdeck-feature-inventory.md b/docs/tdeck-feature-inventory.md index 416182f..bdb39a7 100644 --- a/docs/tdeck-feature-inventory.md +++ b/docs/tdeck-feature-inventory.md @@ -28,8 +28,8 @@ Status labels: | --- | --- | --- | --- | | T-Deck PlatformIO firmware target | Functional, CI-covered, needs hardware validation | `platformio.ini`, `src/main_tdeck.cpp`; `.github/workflows/firmware.yml` builds `pio run -e tdeck`, captures size output, and uploads firmware artifacts | Keep validating board profile against actual T-Deck flash/PSRAM during hardware smoke runs. | | OTA-ready partition layout | Partial | `partitions.csv` has `ota_0`, `ota_1`, `otadata`, `config`, `appfs` | OTA service and update UI are not implemented. | -| Display and LVGL shell | Functional, needs validation | LovyanGFX ST7789 setup, LVGL buffers, UI screens | Hardware flash/smoke checklist needed for every release. | -| Trackball and keyboard input | Functional, needs validation | GPIO interrupts and I2C keyboard polling in `main_tdeck.cpp` | Add hardware test checklist and input regression tests in simulator. | +| Display and LVGL shell | Functional, needs validation | LovyanGFX ST7789 setup, LVGL buffers, UI screens | Use `docs/tdeck-release-checklist.md` for every release hardware pass. | +| Trackball and keyboard input | Functional, needs validation | GPIO interrupts and I2C keyboard polling in `main_tdeck.cpp` | Use the release checklist for hardware input proof; add input regression tests in simulator. | | Touch and calibration | Functional, needs validation | GT911 driver and `touch.cfg` persistence | Add explicit touch calibration validation after flash. | | SD-backed data directory | Functional, needs validation | SD mount and `/sd/limitlezz` store | Add corruption/power-loss tests and storage quota handling. | | Battery/system telemetry | Partial | ADC battery reading, system screen stats | Charge state is inferred/limited; no dedicated charge-detect line. | @@ -110,7 +110,7 @@ Status labels: | Feedback Manager | Planned | Design spec section 8 | Centralize LED, buzzer, keyboard/display feedback and DND. | | Emergency beacon | Planned | Design spec section 12, disabled Emergency UI row | Requires Feedback Manager and dual-network messaging. | | BLE companion | Partial, needs validation | NimBLE-based Meshtastic GATT service, official UUIDs, raw `ToRadio` writes, queued `FromRadio` reads, `FromNum` notifications, UI toggle, and serial selftest/status | Validate with the official Meshtastic app over BLE before calling V0.5 complete. | -| CI and release checks | Partial | `.github/workflows/firmware.yml` runs native simulator build, native protocol selftest, deterministic simulator scenario, screenshot generation, T-Deck build, size reporting, an explicit firmware/static-RAM budget gate, and artifact upload with budget metadata plus screenshots | Add protocol vectors beyond the native selftest and hardware evidence gates. | +| CI and release checks | Partial | `.github/workflows/firmware.yml` runs native simulator build, native protocol selftest, deterministic simulator scenario, screenshot generation, T-Deck build, size reporting, an explicit firmware/static-RAM budget gate, and artifact upload with budget metadata plus screenshots; `docs/tdeck-release-checklist.md` and `scripts/release_evidence.py` define the exact-artifact COM8 evidence path | Add protocol vectors beyond the native selftest and keep expanding hardware evidence gates. | ## Completion Criteria diff --git a/docs/tdeck-firmware-roadmap.md b/docs/tdeck-firmware-roadmap.md index f987e35..1bc7ba4 100644 --- a/docs/tdeck-firmware-roadmap.md +++ b/docs/tdeck-firmware-roadmap.md @@ -55,7 +55,7 @@ Deliverables: - Make CI artifacts usable for local hardware smoke on slow hosts. Implemented: Firmware CI now uploads a flash bundle with bootloader, partitions, boot app, firmware, ELF/map, and manifest; `scripts/fetch_tdeck_artifact.py` downloads the exact-commit artifact for local `scripts/tdeck_smoke.py --skip-build` flashing. - Add a simulator CI path or clear host-specific setup docs for SDL2. Implemented in `.github/workflows/firmware.yml`; Ubuntu CI installs SDL2, Windows can install the local SDL2 bundle with `scripts/ensure_sdl2_windows.ps1`, and `pio run -e native` runs through `scripts/pio_native_sdl2.py`. - Make `pio run -e native` fail clearly when SDL2 is absent. Implemented by `scripts/pio_native_sdl2.py`; Linux/macOS use `sdl2-config` or `pkg-config`, while Windows uses `SDL2_DIR` or the local `.deps` bundle. -- Add a release checklist covering build, flash, boot log, display, touch, keyboard, trackball, SD, radio, Wi-Fi, companion, and sleep. +- Add a release checklist covering build, flash, boot log, display, touch, keyboard, trackball, SD, radio, Wi-Fi, companion, and sleep. Implemented in `docs/tdeck-release-checklist.md`, with `scripts/release_evidence.py` generating a PR-ready slow-host Actions artifact plus COM8 evidence skeleton. - Update README status wording so "working", "partial", "prototype", and "planned" are distinct. - Persist user settings beyond identity/Wi-Fi/touch/keys: brightness, timeout, clock format, time zone, keyboard light, TX power, network toggles, power saving. Implemented through `settings.cfg`; Wi-Fi credential hardening remains V0.96. - Hide Terminal behind a temporary Developer Mode setting or remove it from the default launcher until Developer Mode exists. Implemented: Terminal is hidden from Home until Developer Mode is enabled in Settings. @@ -418,7 +418,7 @@ Deliverables: - user guide - app developer guide - hardware flashing/recovery guide - - release checklist + - release checklist. Initial T-Deck release evidence checklist is implemented in `docs/tdeck-release-checklist.md`; broader user/recovery docs remain. - troubleshooting - Automated checks: - T-Deck compile diff --git a/docs/tdeck-release-checklist.md b/docs/tdeck-release-checklist.md new file mode 100644 index 0000000..459bc6a --- /dev/null +++ b/docs/tdeck-release-checklist.md @@ -0,0 +1,142 @@ +# T-Deck Release Checklist + +Use this checklist before calling a LimitlezzOS T-Deck firmware PR or release +ready. It is intentionally biased toward slow maintainer hosts: run quick local +native checks, build the T-Deck binary in GitHub Actions, then flash the exact +Actions artifact on the COM8 T-Deck. + +## Required Evidence + +- Branch name, PR number, commit SHA, and whether the PR is draft or ready. +- GitHub Actions `Firmware CI` run URL for the exact PR head commit. +- Downloaded artifact directory and `FLASH_MANIFEST.txt` contents. +- Local native/simulator check results. +- COM8 flash log, including chip, MAC, flash byte counts, and hash verification. +- Serial smoke output for `id`, `sys`, `net`, `rf`, `stats`, `wifi`, and + `companion test`. +- Manual hardware notes for display, touch, keyboard, trackball, SD/appfs, + radio, Wi-Fi, companion mode, and sleep/wake. + +Generate a PR-ready skeleton with: + +```sh +python scripts/release_evidence.py --artifact-dir .pio/ci-artifacts/tdeck --port COM8 +``` + +## Local Sanity Checks + +Run local checks that do not require a T-Deck firmware build: + +```sh +python -m py_compile scripts/tdeck_smoke.py scripts/fetch_tdeck_artifact.py scripts/release_evidence.py +pio run -e native +.pio/build/native/program --selftest +.pio/build/native/program --simtest +``` + +If SDL2 is missing on Windows, install the local bundle once: + +```powershell +powershell -ExecutionPolicy Bypass -File scripts\ensure_sdl2_windows.ps1 +``` + +## GitHub Actions Build + +Push the branch, wait for `Firmware CI`, then download the exact commit +artifact: + +```sh +git push fork HEAD +gh run list --repo ItsLimitlezz/LimitlezzOS --workflow "Firmware CI" --branch --limit 5 +gh run watch --repo ItsLimitlezz/LimitlezzOS --exit-status --interval 10 +python scripts/fetch_tdeck_artifact.py --repo ItsLimitlezz/LimitlezzOS --branch --commit --out .pio/ci-artifacts/tdeck +``` + +Confirm `FLASH_MANIFEST.txt` contains the expected `repo=`, `sha=`, +`workflow=`, `run_id=`, `flash_offsets=`, and size-budget lines. + +Do not flash an artifact from a different commit unless the PR body explicitly +calls that out as a non-release diagnostic. + +## COM8 Flash And Serial Smoke + +Flash the downloaded Actions artifact: + +```sh +python scripts/tdeck_smoke.py --port COM8 --no-stub-upload --skip-build --artifact-dir .pio/ci-artifacts/tdeck --open-timeout 60 --boot-timeout 60 --timeout 30 +``` + +Capture: + +- ESP32-S3 revision and MAC. +- Flash byte counts and hash verification for bootloader, partitions, + boot app, and firmware. +- Device identity from `id`. +- Battery, uptime, storage, and heap/PSRAM from `sys`. +- Network toggles from `net`. +- Radio profile/status from `rf`. +- Packet/service counters from `stats`. +- Wi-Fi saved-state and credential backend from `wifi`; never record a + password. +- `companion test` result. + +If the post-flash serial console misses the first boot window, retry a +read-only smoke without reflashing: + +```sh +python scripts/tdeck_smoke.py --port COM8 --skip-upload --open-timeout 60 --boot-timeout 90 --timeout 90 --no-expect --commands id sys net rf stats wifi "companion test" +``` + +## Manual Hardware Pass + +After serial smoke, verify the physical device: + +- Display renders the lock screen and Home without obvious tearing or inverted + colors. +- Trackball moves focus in all directions and click selects. +- Keyboard types in a conversation composer; backspace and Enter work. +- Touch can tap a list row and a top-level navigation target. +- Files can enter the mounted SD/local store or appfs root when present. +- Wi-Fi screen shows saved state correctly and can scan/connect when testing a + release candidate. +- Meshtastic USB companion mode can be enabled and disabled without losing the + serial console after exit. +- Sleep/dim timeout wakes with the intended two-step wake/unlock behavior. +- For release candidates, include at least one stock Meshtastic peer and any + MeshCore peer or split-airtime scenario claimed by the release notes. + +## Release Gates + +- `Firmware CI` is green for the exact head commit. +- COM8 exact-artifact flash and serial smoke pass. +- Firmware and static-RAM size budgets pass in the artifact manifest. +- No known P0/P1 issue is being hidden by release wording. +- README, roadmap, and inventory status match the actual tested state. +- User-facing release notes distinguish working, partial, prototype, planned, + and needs-validation features. +- Attached binaries or release assets come from the same Actions run recorded + in the evidence. + +## PR Evidence Template + +```md +## Validation + +- Local: `python -m py_compile ...` +- Local: `pio run -e native` +- Local: `.pio/build/native/program --selftest` +- Local: `.pio/build/native/program --simtest` +- Actions: Firmware CI for `` +- Artifact: `.pio/ci-artifacts/tdeck`, manifest `sha=` +- Hardware: COM8 exact-artifact flash/smoke passed + +## Hardware Notes + +- Flash: ESP32-S3 rev , MAC , hashes verified +- Identity: `` +- System: `` +- Radio/network: `` +- Wi-Fi: `` +- Companion: `` +- Manual: display/touch/keyboard/trackball/SD/Wi-Fi/sleep checked +``` diff --git a/scripts/release_evidence.py b/scripts/release_evidence.py new file mode 100644 index 0000000..141b9b6 --- /dev/null +++ b/scripts/release_evidence.py @@ -0,0 +1,217 @@ +#!/usr/bin/env python3 +""" +Print a release evidence checklist for a T-Deck firmware PR. + +This helper keeps slow local hosts on the intended path: use local checks for +native/simulator confidence, use GitHub Actions for the T-Deck build, then +flash the exact Actions artifact on the hardware port. +""" +from __future__ import annotations + +import argparse +import json +import os +import subprocess +from pathlib import Path + + +REQUIRED_FLASH_FILES = ("bootloader.bin", "boot_app0.bin", "partitions.bin", "firmware.bin") + + +def run_text(cmd: list[str], cwd: Path, *, required: bool = False) -> str | None: + try: + return subprocess.check_output(cmd, cwd=cwd, text=True, stderr=subprocess.STDOUT).strip() + except FileNotFoundError as exc: + if required: + raise SystemExit(f"missing command: {cmd[0]}") from exc + except subprocess.CalledProcessError as exc: + if required: + raise SystemExit(exc.output.strip() or f"command failed: {' '.join(cmd)}") from exc + return None + + +def git(project_dir: Path, *args: str) -> str: + return run_text(["git", *args], project_dir, required=True) or "" + + +def default_repo(project_dir: Path) -> str: + data = run_text(["gh", "repo", "view", "--json", "nameWithOwner"], project_dir) + if data: + try: + repo = json.loads(data).get("nameWithOwner") + if repo: + return repo + except json.JSONDecodeError: + pass + + remote = run_text(["git", "remote", "get-url", "origin"], project_dir) + if remote and "github.com" in remote: + cleaned = remote.rstrip("/").removesuffix(".git") + for marker in ("github.com:", "github.com/"): + if marker in cleaned: + return cleaned.split(marker, 1)[1] + return "ItsLimitlezz/LimitlezzOS" + + +def default_port() -> str: + return os.environ.get("LZ_SERIAL_PORT") or ("COM8" if os.name == "nt" else "/dev/ttyACM0") + + +def load_runs(project_dir: Path, repo: str, workflow: str, branch: str) -> list[dict] | None: + data = run_text( + [ + "gh", + "run", + "list", + "--repo", + repo, + "--workflow", + workflow, + "--branch", + branch, + "--limit", + "20", + "--json", + "databaseId,headSha,status,conclusion,createdAt,url", + ], + project_dir, + ) + if not data: + return None + try: + return json.loads(data) + except json.JSONDecodeError: + return None + + +def choose_run(runs: list[dict] | None, commit: str) -> dict | None: + if not runs: + return None + for run in runs: + if run.get("headSha", "").lower() == commit.lower(): + return run + return None + + +def find_manifest(artifact_dir: Path) -> Path | None: + direct = artifact_dir / "FLASH_MANIFEST.txt" + if direct.exists(): + return direct + candidates = sorted(artifact_dir.rglob("FLASH_MANIFEST.txt")) if artifact_dir.exists() else [] + if not candidates: + return None + return candidates[0] + + +def parse_manifest(path: Path | None) -> dict[str, str]: + if path is None: + return {} + values: dict[str, str] = {} + for raw in path.read_text(encoding="utf-8", errors="replace").splitlines(): + if "=" not in raw: + continue + key, value = raw.split("=", 1) + key = key.strip() + if key and all(c.isalnum() or c == "_" for c in key): + values[key] = value.strip() + return values + + +def present_flash_files(artifact_dir: Path) -> list[str]: + return [name for name in REQUIRED_FLASH_FILES if (artifact_dir / name).exists()] + + +def md_code(value: object) -> str: + return f"`{value}`" + + +def run_line(run: dict | None, commit: str) -> str: + if run is None: + return f"not found yet for {md_code(commit)}" + status = run.get("status") or "unknown" + conclusion = run.get("conclusion") or "pending" + run_id = run.get("databaseId") or "unknown" + url = run.get("url") or "" + suffix = f" ([run {run_id}]({url}))" if url else f" (run {run_id})" + return f"{md_code(status + '/' + conclusion)}{suffix}" + + +def main() -> int: + parser = argparse.ArgumentParser(description="Print a T-Deck release evidence checklist.") + parser.add_argument("--project-dir", default=Path(__file__).resolve().parents[1]) + parser.add_argument("--repo", help="GitHub repo, e.g. ItsLimitlezz/LimitlezzOS.") + parser.add_argument("--workflow", default="Firmware CI") + parser.add_argument("--branch", help="Branch name. Defaults to the current git branch.") + parser.add_argument("--commit", help="Commit SHA. Defaults to HEAD.") + parser.add_argument("--artifact-dir", default=Path(".pio") / "ci-artifacts" / "tdeck") + parser.add_argument("--port", default=default_port()) + args = parser.parse_args() + + project_dir = Path(args.project_dir).resolve() + branch = args.branch or git(project_dir, "branch", "--show-current") + commit = args.commit or git(project_dir, "rev-parse", "HEAD") + short_commit = commit[:12] + repo = args.repo or default_repo(project_dir) + artifact_dir = Path(args.artifact_dir) + if not artifact_dir.is_absolute(): + artifact_dir = (project_dir / artifact_dir).resolve() + + run = choose_run(load_runs(project_dir, repo, args.workflow, branch), commit) + manifest_path = find_manifest(artifact_dir) + manifest = parse_manifest(manifest_path) + files = present_flash_files(artifact_dir) + actions_status = "passed" if run and run.get("status") == "completed" and run.get("conclusion") == "success" else "pending" + artifact_status = "downloaded" if manifest_path and len(files) == len(REQUIRED_FLASH_FILES) else "pending" + + print("## T-Deck Release Evidence") + print() + print(f"- Repository: {md_code(repo)}") + print(f"- Branch: {md_code(branch)}") + print(f"- Commit: {md_code(commit)}") + print(f"- Workflow: {md_code(args.workflow)}") + print(f"- Actions run: {run_line(run, commit)}") + print(f"- Artifact dir: {md_code(artifact_dir)}") + print(f"- Artifact manifest: {md_code(manifest_path) if manifest_path else 'not downloaded yet'}") + print(f"- Manifest SHA: {md_code(manifest.get('sha', 'not recorded'))}") + print(f"- Flash bundle files: {md_code(', '.join(files) if files else 'not downloaded yet')}") + print(f"- Hardware port: {md_code(args.port)}") + print() + print("### Local Sanity") + print() + print("- [ ] `python -m py_compile scripts/tdeck_smoke.py scripts/fetch_tdeck_artifact.py scripts/release_evidence.py`") + print("- [ ] `pio run -e native`") + print("- [ ] `.pio/build/native/program --selftest`") + print("- [ ] `.pio/build/native/program --simtest`") + print() + print("### GitHub Actions Build") + print() + print(f"- [ ] Push branch: `git push fork HEAD:{branch}`") + print(f"- [ ] Wait for exact-commit Actions success: `gh run watch --repo {repo} --exit-status --interval 10`") + print( + "- [ ] Fetch exact artifact: " + f"`python scripts/fetch_tdeck_artifact.py --repo {repo} --branch {branch} --commit {commit} --out .pio/ci-artifacts/tdeck`" + ) + print("- [ ] Confirm `FLASH_MANIFEST.txt` `sha=` matches the PR head commit.") + print() + print("### COM8 Hardware Smoke") + print() + print( + "- [ ] Flash exact artifact: " + f"`python scripts/tdeck_smoke.py --port {args.port} --no-stub-upload --skip-build " + "--artifact-dir .pio/ci-artifacts/tdeck --open-timeout 60 --boot-timeout 60 --timeout 30`" + ) + print("- [ ] Record flash chip, MAC, byte counts, and hash verification.") + print("- [ ] Record serial `id`, `sys`, `net`, `rf`, `stats`, `wifi`, and `companion test` output.") + print("- [ ] Manually verify display, touch, keyboard, trackball, SD/appfs browsing, Wi-Fi state, and sleep/wake.") + print() + print("### PR Evidence Snippet") + print() + print(f"- Local native checks: pending for {md_code(short_commit)}.") + print(f"- GitHub Actions `{args.workflow}`: {actions_status} for {md_code(short_commit)}.") + print(f"- Artifact: {artifact_status} at {md_code(artifact_dir)}.") + print("- Hardware: pending COM8 exact-artifact flash/smoke.") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) From a2b65f063a0fcc31b9746e55bfacb596a17a33d4 Mon Sep 17 00:00:00 2001 From: n30nex Date: Fri, 19 Jun 2026 06:01:51 -0400 Subject: [PATCH 22/64] Add T-Deck flashing recovery guide --- README.md | 3 + docs/tdeck-firmware-roadmap.md | 2 +- docs/tdeck-flashing-recovery.md | 166 ++++++++++++++++++++++++++++++++ 3 files changed, 170 insertions(+), 1 deletion(-) create mode 100644 docs/tdeck-flashing-recovery.md diff --git a/README.md b/README.md index 722c799..a2c1426 100644 --- a/README.md +++ b/README.md @@ -363,6 +363,9 @@ sandbox, App Store networking, OTA, and the Feedback Manager (LED/buzzer/backlig ## Flashing & first hardware test +For the full Actions-artifact flash and recovery workflow, see +[`docs/tdeck-flashing-recovery.md`](docs/tdeck-flashing-recovery.md). + ```sh pio run -e tdeck -t upload # build + flash over USB-C pio device monitor -b 115200 # watch the boot diagnostics diff --git a/docs/tdeck-firmware-roadmap.md b/docs/tdeck-firmware-roadmap.md index f987e35..d0d6ee1 100644 --- a/docs/tdeck-firmware-roadmap.md +++ b/docs/tdeck-firmware-roadmap.md @@ -417,7 +417,7 @@ Deliverables: - Full docs: - user guide - app developer guide - - hardware flashing/recovery guide + - hardware flashing/recovery guide. Initial T-Deck Actions-artifact flashing and recovery guide is implemented in `docs/tdeck-flashing-recovery.md`. - release checklist - troubleshooting - Automated checks: diff --git a/docs/tdeck-flashing-recovery.md b/docs/tdeck-flashing-recovery.md new file mode 100644 index 0000000..d53a06a --- /dev/null +++ b/docs/tdeck-flashing-recovery.md @@ -0,0 +1,166 @@ +# T-Deck Flashing And Recovery Guide + +This guide covers the supported LimitlezzOS flash path for LilyGO T-Deck and +T-Deck Plus hardware. T-Deck Pro uses a different board layout and is not +covered by this firmware target. + +The maintainer workflow favors GitHub Actions for the T-Deck build, then local +COM8 validation with the exact artifact. That keeps slow Windows hosts from +spending time on full ESP32 builds while still proving the binary on hardware. + +## Known Hardware Target + +- Board: LilyGO T-Deck or T-Deck Plus with ESP32-S3 and SX1262. +- Flash: 16 MB. +- Default maintainer serial port: `COM8`. +- Linux/macOS examples usually use `/dev/ttyACM0` or `/dev/ttyUSB0`. +- Firmware offsets: `0x0 bootloader.bin`, `0x8000 partitions.bin`, + `0xe000 boot_app0.bin`, `0x10000 firmware.bin`. +- App/user data on the SD card is separate from ESP32 flash, but appfs and OTA + partitions are on ESP32 flash. + +## Preflight + +- Close `pio device monitor`, serial terminals, Meshtastic clients, and other + tools that may own the serial port. +- Use a USB data cable, not a charge-only cable. +- Keep the device powered from USB during the whole flash. +- On Windows, confirm the expected COM port in Device Manager after plug-in. +- Confirm GitHub CLI auth if downloading Actions artifacts: + +```sh +gh auth status +``` + +## Preferred Slow-Host Flash Path + +Push the branch and let GitHub Actions build the firmware: + +```sh +git push fork HEAD +gh run list --repo ItsLimitlezz/LimitlezzOS --workflow "Firmware CI" --branch --limit 5 +gh run watch --repo ItsLimitlezz/LimitlezzOS --exit-status --interval 10 +``` + +Download the artifact for the exact commit: + +```sh +python scripts/fetch_tdeck_artifact.py --repo ItsLimitlezz/LimitlezzOS --branch --commit --out .pio/ci-artifacts/tdeck +``` + +Check `FLASH_MANIFEST.txt` before flashing: + +```powershell +Get-Content .pio\ci-artifacts\tdeck\FLASH_MANIFEST.txt +``` + +The manifest `sha=` must match the PR head commit. The `budget_status=` line +must be `pass` for a release candidate. + +Flash and run the standard serial smoke: + +```sh +python scripts/tdeck_smoke.py --port COM8 --no-stub-upload --skip-build --artifact-dir .pio/ci-artifacts/tdeck --open-timeout 60 --boot-timeout 60 --timeout 30 +``` + +Use `/dev/ttyACM0` or `/dev/ttyUSB0` instead of `COM8` on non-Windows hosts. + +## Local Build Flash Path + +Use local T-Deck builds only when you intentionally need a host-built binary: + +```sh +pio run -e tdeck +python scripts/tdeck_smoke.py --port COM8 --no-stub-upload +``` + +The no-stub path is preferred on the maintainer T-Deck because the ROM stub +upload path can be flaky on this host. + +## Serial Smoke Proof + +A passing smoke should include: + +- ESP32-S3 chip revision and MAC during flash. +- Hash verification for every flashed image. +- `=== boot complete ===` during boot. +- `id` reporting the expected node identity. +- `sys` reporting battery, CPU, RAM, and uptime. +- `net` reporting network toggles. +- `rf` reporting the active radio profile. +- `stats` reporting radio counters. +- `wifi` reporting saved state and credential backend without printing a + password. +- `companion test` ending in `PASS`. +- `[serial] smoke PASS`. + +## Read-Only Retry After Flash + +USB CDC can briefly disappear during boot. If flash succeeded but the first +serial smoke times out, do not reflash. Reattach and run a read-only smoke: + +```sh +python scripts/tdeck_smoke.py --port COM8 --skip-upload --open-timeout 60 --boot-timeout 90 --timeout 90 --no-expect --commands id sys net rf stats wifi "companion test" +``` + +If this passes, record both facts: the exact-artifact flash completed, and the +read-only retry completed the serial proof. + +## Recovery Playbook + +### Port Missing + +- Unplug and replug USB, then recheck the port name. +- Try another USB data cable and port. +- Close any monitor or phone companion tool that may hold the port. +- If the port changes after boot, rerun the command with the new port. + +### Upload Will Not Connect + +- Use `--no-stub-upload`. +- Lower upload speed if a local command was using a faster baud. +- Power-cycle the device and rerun the command. +- If the board exposes physical boot/reset controls, enter the ROM bootloader + with the board controls, then rerun the same flash command. + +### Flash Succeeds But No Prompt Appears + +- Run the read-only retry command above. +- Increase `--boot-timeout` to `120` on especially slow boot attempts. +- Confirm the device is not left in USB companion mode; the text console smoke + needs the LimitlezzOS serial prompt. + +### Boot Diagnostics Show Hardware Failures + +- If SD and radio fail while display works, check SPI chip-select behavior and + the SD card state. +- If `SX1262 radio (RadioLib begin=...)` reports a non-zero code, treat it as a + radio bring-up problem rather than a UI problem. +- If display colors look inverted, verify the panel revision and inversion + setting. +- If touch fails, rerun touch calibration from Settings after a successful boot. + +### Last-Resort Flash Recovery + +Only erase flash when ordinary reflashing cannot recover the device. This wipes +ESP32 flash partitions, including OTA/appfs/config data, but does not erase the +removable SD card: + +```sh +pio run -e tdeck -t erase --upload-port COM8 +python scripts/tdeck_smoke.py --port COM8 --no-stub-upload +``` + +Record that an erase was used, because it changes the upgrade/recovery evidence +for the run. + +## Release Evidence + +For PRs and releases, record: + +- Branch, commit, PR number, and Actions run URL. +- Artifact path and manifest contents. +- Flash chip/MAC/hash verification. +- Serial smoke output. +- Any read-only retry or recovery step used. +- Any manual hardware checks performed after boot. From 3f689cabc8fd3a779f44a04e94c944527963ad1c Mon Sep 17 00:00:00 2001 From: n30nex Date: Fri, 19 Jun 2026 06:10:21 -0400 Subject: [PATCH 23/64] Add T-Deck troubleshooting guide --- README.md | 1 + docs/tdeck-firmware-roadmap.md | 2 +- docs/tdeck-troubleshooting.md | 275 +++++++++++++++++++++++++++++++++ 3 files changed, 277 insertions(+), 1 deletion(-) create mode 100644 docs/tdeck-troubleshooting.md diff --git a/README.md b/README.md index 722c799..b920d02 100644 --- a/README.md +++ b/README.md @@ -144,6 +144,7 @@ iPhone-style dark look (status bar, battery glyph, grouped settings cards). - [`docs/tdeck-feature-inventory.md`](docs/tdeck-feature-inventory.md) - feature-by-feature implementation inventory. - [`docs/tdeck-firmware-roadmap.md`](docs/tdeck-firmware-roadmap.md) - roadmap to a complete T-Deck firmware. - [`docs/tdeck-hardware-dogfood-checklist.md`](docs/tdeck-hardware-dogfood-checklist.md) - stock-device hardware proof checklist. +- [`docs/tdeck-troubleshooting.md`](docs/tdeck-troubleshooting.md) - build, flash, boot, radio, storage, Wi-Fi, and companion troubleshooting. ![screens](docs/screens.png) diff --git a/docs/tdeck-firmware-roadmap.md b/docs/tdeck-firmware-roadmap.md index f987e35..2abff61 100644 --- a/docs/tdeck-firmware-roadmap.md +++ b/docs/tdeck-firmware-roadmap.md @@ -419,7 +419,7 @@ Deliverables: - app developer guide - hardware flashing/recovery guide - release checklist - - troubleshooting + - troubleshooting. Initial T-Deck troubleshooting guide is implemented in `docs/tdeck-troubleshooting.md`. - Automated checks: - T-Deck compile - simulator selftest diff --git a/docs/tdeck-troubleshooting.md b/docs/tdeck-troubleshooting.md new file mode 100644 index 0000000..b3eabac --- /dev/null +++ b/docs/tdeck-troubleshooting.md @@ -0,0 +1,275 @@ +# T-Deck Troubleshooting Guide + +Use this guide when a LimitlezzOS T-Deck build, flash, boot, radio session, or +hardware smoke does not behave as expected. Start by capturing evidence before +changing state; most failures can be separated into host/USB, boot hardware, +storage, radio, Wi-Fi, companion, or app-platform issues. + +## First Evidence To Capture + +Record these before reflashing or erasing anything: + +- Branch, commit, and dirty/clean state: + +```sh +git status -sb +git rev-parse HEAD +``` + +- Exact firmware artifact source: + +```powershell +Get-Content .pio\ci-artifacts\tdeck\FLASH_MANIFEST.txt +``` + +- Serial smoke or read-only smoke: + +```sh +python scripts/tdeck_smoke.py --port COM8 --skip-upload --open-timeout 60 --boot-timeout 90 --timeout 90 --no-expect --commands id sys net rf stats wifi "companion test" +``` + +- Native sanity on the host: + +```sh +pio run -e native +.pio/build/native/program --selftest +.pio/build/native/program --simtest +``` + +If the issue is tied to a PR or release, paste the command output and the +artifact manifest into the PR evidence instead of summarizing from memory. + +## Host Or USB Problems + +### COM8 Is Missing Or Busy + +Checks: + +- Close PlatformIO monitor, Meshtastic clients, terminal windows, and any other + serial tool. +- Unplug and replug the device; confirm the Windows COM port did not change. +- Try a known data-capable USB cable and a different USB port. + +Useful commands: + +```sh +python scripts/serial_harness.py --port COM8 --baud 115200 --open-only --open-timeout 5 +``` + +If the open-only check fails, fix port ownership or cabling before reflashing. + +### Upload Connects Then Fails During Stub Handoff + +The maintainer COM8 path can be flaky with the ESP ROM stub. Prefer the +no-stub helper: + +```sh +python scripts/tdeck_smoke.py --port COM8 --no-stub-upload +``` + +For slow hosts, use the GitHub Actions artifact instead of a local T-Deck build: + +```sh +python scripts/fetch_tdeck_artifact.py --repo ItsLimitlezz/LimitlezzOS --branch --commit --out .pio/ci-artifacts/tdeck +python scripts/tdeck_smoke.py --port COM8 --no-stub-upload --skip-build --artifact-dir .pio/ci-artifacts/tdeck +``` + +## Flash Succeeds But Smoke Times Out + +USB CDC may disconnect briefly during boot. If flashing completed and all hashes +verified, do not erase or reflash first. Run a read-only reattach: + +```sh +python scripts/tdeck_smoke.py --port COM8 --skip-upload --open-timeout 60 --boot-timeout 90 --timeout 90 --no-expect --commands id sys net rf stats wifi "companion test" +``` + +If the retry passes, record both facts: + +- exact-artifact flash succeeded +- first post-flash smoke timed out during serial reattach +- read-only retry completed `[serial] smoke PASS` + +If the retry still fails, capture the partial boot output and check whether the +device is in text console mode or an external companion mode. + +## Boot Hardware Diagnostics + +The boot log isolates most hardware bring-up failures: + +```text +=== LimitlezzOS boot === +[ok] peripheral power (GPIO10) HIGH +[ok] shared SPI bus up (SCK40/MISO38/MOSI41) +[ok] ST7789 display init + backlight on +[ok] keyboard @0x55 +[ok] GT911 touch @0x5D +[ok] trackball + keyboard input +[ok] microSD /sd/limitlezz +[ok] SX1262 radio (RadioLib begin=0) +[ok] node id !... +=== boot complete === +``` + +Common interpretations: + +- Display works, but SD and radio fail: inspect shared SPI chip-select behavior + and SD card seating. +- `SX1262 radio (RadioLib begin=...)` is non-zero: treat this as a radio + bring-up problem, not a UI problem. +- Keyboard or touch fail: inspect I2C bring-up first; rerun touch calibration + only after GT911 is detected. +- Display colors look inverted: verify panel revision and the inversion setting. +- `appfs not mounted`: expected when the appfs partition is empty or absent; not + the same as an SD mount failure. + +## Serial Console Diagnostics + +Run these from the `lz>` prompt: + +```text +help +id +sys +net +rf +stats +wifi +nodes +dm status +companion test +``` + +What to look for: + +- `id`: node identity should be stable across reboot when storage is present. +- `sys`: battery, CPU, RAM, and uptime should update. +- `net`: confirms whether Meshtastic and MeshCore are enabled. +- `rf`: confirms active profile and dwell split. +- `stats`: confirms radio TX/RX counters and airtime utilization. +- `wifi`: must report credential backend without printing a password. +- `dm status`: use when delivery state, retries, or ACKs look wrong. +- `companion test`: should end in `PASS` for the USB companion framing path. + +## SD, Appfs, And Persistence + +Symptoms: + +- identity changes after reboot +- message history disappears +- local apps do not appear +- Files only shows one storage root + +Checks: + +- Boot log includes `microSD /sd/limitlezz` when SD is mounted. +- `sys` reports sane RAM and uptime after boot. +- Files screen exposes SD/local storage and appfs only when those roots mount. +- Local app packages use safe IDs, relative entry paths, and supported SDK + permission names. + +Recovery: + +- Reseat or replace the SD card if SD mount is missing. +- Test RAM-only behavior deliberately by booting without SD, but do not count + persistence tests as passed in that state. +- Use Developer Mode app diagnostics for rejected local packages. + +## Wi-Fi Problems + +Checks: + +```text +wifi +net +sys +``` + +Interpretation: + +- `wifi: off saved=(none)` means no remembered network is currently configured. +- `cred=nvs` on hardware means saved credentials are not stored as plaintext on + SD. +- Wi-Fi and BLE cannot both stay resident on this RAM-tight ESP32-S3 path; test + one transport at a time. + +Recovery: + +- Forget and rejoin the network from Settings if the saved SSID is stale. +- Keep password evidence out of logs and PR bodies. +- If Wi-Fi testing is not part of the release claim, record the saved/off state + instead of forcing a network join. + +## Radio, Mesh, And Delivery Issues + +Checks: + +```text +rf +stats +nodes +dm status +rxlog on +``` + +Interpretation: + +- No RX count increase: verify region/preset/channel and antenna. +- LongFast public messages missing: confirm the stock peer can talk to another + stock device on the same channel. +- DMs stuck pending: inspect `dm status`, peer reachability, and ACK timeout + state. +- Duplicate messages: capture `rxlog` evidence and dedup behavior. +- MeshCore behavior looks wrong: confirm whether MeshCore is enabled and whether + the branch under test has the relevant MeshCore feature gate opened. +- TDM concerns require a timed hardware run; a single `rf` output proves current + profile state, not packet-loss behavior under load. + +## Companion Problems + +USB companion: + +- Run `companion test` first; it should report frame counts and `PASS`. +- If the text console disappears after companion use, disable companion mode and + reattach serial. +- Only one external app transport should own the bridge at a time. + +BLE companion: + +- `companion ble on` enables BLE diagnostics. +- The official app connect-then-disconnect issue is still tracked as a known + validation gap; do not mark BLE phone validation complete from the mailbox + selftest alone. + +## Native Simulator And CI Issues + +If native builds fail on Windows, install the local SDL2 bundle: + +```powershell +powershell -ExecutionPolicy Bypass -File scripts\ensure_sdl2_windows.ps1 +``` + +If GitHub Actions is green but local T-Deck builds are slow or flaky, use the +uploaded firmware artifact and keep local validation to native/simulator checks. + +If Actions fails: + +- Open the failing `Firmware CI` job. +- Check whether native build, selftest, simtest, screenshot generation, T-Deck + build, size budget, or artifact upload failed. +- Fix the earliest failing step first. +- Do not flash a PR artifact until the exact head commit has a successful + Actions run. + +## When To Erase Flash + +Erase flash only after ordinary reflashing and read-only serial recovery fail. +An erase wipes ESP32 flash partitions, including OTA/appfs/config data, but does +not erase a removable SD card: + +```sh +pio run -e tdeck -t erase --upload-port COM8 +python scripts/tdeck_smoke.py --port COM8 --no-stub-upload +``` + +Record the erase in PR or release evidence because it changes the recovery path +being validated. From 0ae0ab5fcb3f2253185333239ba8b0248ff74188 Mon Sep 17 00:00:00 2001 From: n30nex Date: Fri, 19 Jun 2026 06:17:56 -0400 Subject: [PATCH 24/64] Add T-Deck user guide --- README.md | 1 + docs/tdeck-firmware-roadmap.md | 2 +- docs/tdeck-user-guide.md | 208 +++++++++++++++++++++++++++++++++ 3 files changed, 210 insertions(+), 1 deletion(-) create mode 100644 docs/tdeck-user-guide.md diff --git a/README.md b/README.md index 722c799..66f6716 100644 --- a/README.md +++ b/README.md @@ -144,6 +144,7 @@ iPhone-style dark look (status bar, battery glyph, grouped settings cards). - [`docs/tdeck-feature-inventory.md`](docs/tdeck-feature-inventory.md) - feature-by-feature implementation inventory. - [`docs/tdeck-firmware-roadmap.md`](docs/tdeck-firmware-roadmap.md) - roadmap to a complete T-Deck firmware. - [`docs/tdeck-hardware-dogfood-checklist.md`](docs/tdeck-hardware-dogfood-checklist.md) - stock-device hardware proof checklist. +- [`docs/tdeck-user-guide.md`](docs/tdeck-user-guide.md) - practical on-device user guide. ![screens](docs/screens.png) diff --git a/docs/tdeck-firmware-roadmap.md b/docs/tdeck-firmware-roadmap.md index f987e35..813638c 100644 --- a/docs/tdeck-firmware-roadmap.md +++ b/docs/tdeck-firmware-roadmap.md @@ -415,7 +415,7 @@ Deliverables: - Close remaining gaps from the audit and feature inventory. - Full docs: - - user guide + - user guide. Initial T-Deck user guide is implemented in `docs/tdeck-user-guide.md`. - app developer guide - hardware flashing/recovery guide - release checklist diff --git a/docs/tdeck-user-guide.md b/docs/tdeck-user-guide.md new file mode 100644 index 0000000..87a1590 --- /dev/null +++ b/docs/tdeck-user-guide.md @@ -0,0 +1,208 @@ +# T-Deck User Guide + +This guide is for people using LimitlezzOS on a LilyGO T-Deck or T-Deck Plus. +It focuses on the normal on-device experience and avoids Developer Mode unless +you need diagnostics. + +## What LimitlezzOS Is For + +LimitlezzOS is a handheld mesh OS for the T-Deck. The main experience is: + +- send and receive Meshtastic LongFast channel messages +- send and receive encrypted Meshtastic direct messages +- keep a unified inbox with clear network labels +- use the trackball, keyboard, and touch screen without a phone +- optionally bridge to companion apps over USB or BLE when those paths are + enabled + +Some roadmap features are still partial or planned. The README and firmware +roadmap are the authority for current release status. + +## Controls + +- Trackball roll: move focus or scroll lists. +- Trackball click: select the focused item. +- Touch: tap rows, tabs, buttons, and toggles. +- Keyboard: type in message composers and text fields. +- Enter: send or confirm when the composer/action expects it. +- Backspace or Escape in the simulator: go back. +- `sym + L`: lock the device from most screens. + +The interface is designed so every common workflow can be completed with the +trackball and keyboard. + +## First Boot + +On first boot, LimitlezzOS asks for: + +1. Long name: the name other mesh users should see. +2. Short tag: a compact Meshtastic short name, usually four characters. +3. Network choice: Meshtastic is the beginner-safe default path. + +After onboarding, the device opens to the lock screen. Future boots skip +onboarding unless local identity storage is reset. + +## Lock Screen And Home + +The lock screen shows time, battery, active network indicators, and unread +message notifications. + +- Wake once with touch, key, or trackball input. +- Unlock with a second input. +- Tap a notification to open the relevant conversation. +- Multiple unread threads show a combined notification count. + +Home uses a paged launcher grid. Built-in apps and accepted local apps appear as +tiles. Terminal is hidden until Developer Mode is enabled. + +## Messages + +Messages is the main inbox. + +- Direct threads and channel threads are separated by tabs. +- Network filters let you view all messages, Meshtastic, or MeshCore when + enabled. +- Unread conversations are highlighted. +- The Home Messages tile shows a badge for unread conversations. +- Long-press a conversation to mute or unmute it. + +Muted conversations keep history but do not contribute to the lock-screen or +Home unread badges. + +## Sending A Channel Message + +1. Open Messages. +2. Choose the channel thread, such as LongFast. +3. Type with the keyboard. +4. Press Enter or the focused send action. + +If sending fails, the message state and diagnostics should make the failure +visible instead of silently losing the message. + +## Sending A Direct Message + +1. Open Messages or Contacts. +2. Choose a contact that is messageable on the active network. +3. Type your message. +4. Send from the composer. + +Sent direct messages can show sending, delivered, failed, retry, and failure +reason states. Long-press failed messages to retry when that action is +available. + +## Contacts And Nodes + +The Meshtastic manager shows heard nodes, signal hints, and node details. +Contacts are people you deliberately add, not every node ever heard. + +Use Contacts when you want a stable person-centric list. Use Meshtastic Nodes +when you are inspecting live mesh presence and diagnostics. + +## Wi-Fi + +Open Settings, then Wi-Fi, to scan and join a network. + +- The device remembers one network. +- Auto-connect can rejoin the saved network. +- Forget removes the saved network so you can enter a new password. +- On T-Deck hardware, saved Wi-Fi credentials use the hardware credential + backend reported by serial diagnostics as `cred=nvs`. + +Wi-Fi and BLE share scarce ESP32-S3 resources. If a BLE companion workflow is +being tested, keep Wi-Fi expectations modest and test one transport at a time. + +## Time, Display, And Sleep + +Settings includes common device preferences: + +- brightness +- sleep timeout +- keyboard light mode +- time zone +- 12-hour or 24-hour clock +- TX power +- network toggles +- power saving + +Sleep is two-step: the first input wakes the screen while staying locked, and a +second input unlocks. This helps prevent accidental pocket interaction. + +## Files And Storage + +Files is a read-only browser for mounted storage roots. + +- SD/local storage holds user data such as identity, messages, settings, and + local apps when an SD card is present. +- Appfs is a separate flash-backed app partition when mounted. +- If both are present, Files starts at a storage root picker. +- Without SD, the OS can still run, but persistence tests should not be counted + as passing. + +## Local Apps + +Local apps can be copied to supported app directories such as SD or appfs app +roots. Accepted app manifests appear in Home and App Store views. + +Current local app support is intentionally limited: + +- app metadata and permissions are validated before launch +- apps run in a foreground shell +- storage is scoped to the app +- unsupported actions fail closed +- rejected packages can be inspected in Developer Mode diagnostics + +Network catalog install/update, richer script execution, and the full runtime +API are still roadmap work. + +## Companion Modes + +Meshtastic companion mode lets an external app use the T-Deck radio through the +firmware bridge. + +- USB companion mode is the hardware-tested companion path. +- BLE companion firmware support exists, but phone app validation is still a + tracked gap in the roadmap. +- Only one external companion transport should own the bridge at a time. +- Leaving companion mode should return the serial console for diagnostics. + +## Developer Mode + +Developer Mode reveals Terminal and extra diagnostics. Normal users should not +need it for everyday messaging. + +Use Developer Mode when you need to inspect: + +- serial-style diagnostics on device +- rejected local app packages +- advanced state while testing a PR or release candidate + +Turn Developer Mode off again to return Home to the simpler consumer layout. + +## Basic Recovery + +If something looks wrong: + +1. Reboot the device. +2. Confirm the battery and USB power are stable. +3. Check whether the SD card is present if identity, settings, messages, or + apps disappeared. +4. Check Settings for network toggles and Wi-Fi state. +5. Use Developer Mode or USB serial diagnostics only when the normal UI does not + explain the issue. + +For build, flash, boot, radio, companion, or storage diagnostics, use the +troubleshooting guide once it is present in your branch or release docs. + +## What To Report + +When reporting a problem, include: + +- device model: T-Deck or T-Deck Plus +- firmware branch and commit if known +- whether an SD card was installed +- what screen you were on +- what you expected to happen +- what actually happened +- whether the issue repeats after reboot +- any serial `id`, `sys`, `net`, `rf`, `stats`, `wifi`, or `companion test` + output available from the maintainer/debug workflow From 90cbde4f92dc933b4aa43e86aba5d92abb3e4fdd Mon Sep 17 00:00:00 2001 From: n30nex Date: Fri, 19 Jun 2026 06:26:03 -0400 Subject: [PATCH 25/64] Add T-Deck app developer guide --- README.md | 1 + docs/tdeck-app-developer-guide.md | 293 ++++++++++++++++++++++++++++++ docs/tdeck-firmware-roadmap.md | 2 +- 3 files changed, 295 insertions(+), 1 deletion(-) create mode 100644 docs/tdeck-app-developer-guide.md diff --git a/README.md b/README.md index 722c799..1e0278a 100644 --- a/README.md +++ b/README.md @@ -144,6 +144,7 @@ iPhone-style dark look (status bar, battery glyph, grouped settings cards). - [`docs/tdeck-feature-inventory.md`](docs/tdeck-feature-inventory.md) - feature-by-feature implementation inventory. - [`docs/tdeck-firmware-roadmap.md`](docs/tdeck-firmware-roadmap.md) - roadmap to a complete T-Deck firmware. - [`docs/tdeck-hardware-dogfood-checklist.md`](docs/tdeck-hardware-dogfood-checklist.md) - stock-device hardware proof checklist. +- [`docs/tdeck-app-developer-guide.md`](docs/tdeck-app-developer-guide.md) - SDK 0.1 local app developer guide. ![screens](docs/screens.png) diff --git a/docs/tdeck-app-developer-guide.md b/docs/tdeck-app-developer-guide.md new file mode 100644 index 0000000..c2ccc58 --- /dev/null +++ b/docs/tdeck-app-developer-guide.md @@ -0,0 +1,293 @@ +# T-Deck App Developer Guide + +This guide explains how to build local LimitlezzOS apps for the current SDK +0.1 foreground shell. It documents what works today. Network catalog installs, +arbitrary script execution, richer Lua APIs, background tasks, and direct mesh +send/receive APIs are still roadmap work. + +For the lower-level manifest reference, see +[`docs/tdeck-local-app-manifest.md`](tdeck-local-app-manifest.md). + +## Current App Model + +SDK 0.1 apps are local packages discovered from SD or appfs. Accepted packages +appear in Home and App Store, then launch into a safe foreground shell. + +Current guarantees: + +- apps are foreground-only +- manifests and entry files are bounded before launch +- permissions are parsed fail-closed +- storage is scoped to the app package +- app data has an early 64 KB quota +- entry text can display bounded status/body content +- up to two foreground actions can update the session +- the only write effect is a scoped `counter:` file +- `{time}` and `{battery}` are read-only tokens gated by permissions + +Not implemented yet: + +- arbitrary Lua/script execution +- background tasks +- network catalog download/update +- app package signatures or SHA verification +- mesh send/receive APIs +- notifications API behavior +- raw hardware, radio, filesystem, or kernel access + +## Package Locations + +On T-Deck hardware, place packages in one of: + +```text +/sd/limitlezz/apps// +/sd/apps// +/appfs/apps// +``` + +In the native simulator, packages are discovered from: + +```text +/apps// +/appfs/apps// +``` + +Each package must contain: + +```text +manifest.json + +data/ optional; prepared automatically for storage apps +assets/ optional; reserved for later richer runtimes +``` + +## Minimal App + +Create a package directory: + +```text +weather.mesh/ + manifest.json + main.lua +``` + +`manifest.json`: + +```json +{ + "id": "weather.mesh", + "name": "Weather Mesh", + "version": "0.1.0", + "author": "Limitless", + "summary": "Local weather dashboard", + "entry": "main.lua", + "api_version": "0.1", + "permissions": ["display", "input", "storage", "system_time", "battery"], + "icon": "weather", + "hue": 48 +} +``` + +`main.lua`: + +```lua +-- title: Weather Mesh +-- status: Updated at {time} +-- body: Battery {battery}. Forecast refreshed {count} times. +-- action: Refresh | Refreshed #{count} | Local forecast refreshed {count} times at {time}. | counter:refreshes +``` + +The entry file is read as metadata today. The Lua-looking comment style is only +a convenient format; the firmware does not execute Lua code in SDK 0.1. + +## Manifest Fields + +Required fields: + +- `id`: safe package id containing letters, numbers, `_`, `-`, or `.` +- `name`: display name +- `entry`: relative path to the package entry file + +Optional fields: + +- `version`: shown in app detail, defaults to `0.0.0` +- `author`: shown in app detail, defaults to `local` +- `summary` or `description`: short display text +- `api_version`: compatibility gate, defaults to `0.1` +- `permissions`: supported permission names, defaults to `display` and `input` +- `icon`: token such as `calculate`, `note`, `weather`, `map`, `game`, + `terminal`, `folder`, or `description` +- `hue`: tile color hint; unsupported values fall back to neutral styling + +Unsupported SDK versions and unknown permissions reject the package before it +appears on Home. + +## Permissions + +Supported permission names: + +- `display`: required to show foreground content +- `input`: required for app-provided actions +- `storage`: prepares scoped `data/` and enables counter actions +- `mesh_read`: reserved for future mesh read APIs +- `mesh_send`: reserved for future mesh send APIs +- `system_time`: enables `{time}` token expansion +- `battery`: enables `{battery}` token expansion +- `notifications`: reserved for future notification APIs +- `network_wifi`: reserved for future Wi-Fi/network APIs + +Permission rules: + +- An app that declares actions without `input` is launch-blocked. +- A counter action without `storage` is launch-blocked. +- `{time}` without `system_time` is launch-blocked. +- `{battery}` without `battery` is launch-blocked. +- Unknown permissions are rejected at scan time. + +Declare only the namespaces the app needs. Future runtimes will use the same +principle: undeclared APIs should be absent from the app environment. + +## Entry Metadata + +The SDK 0.1 foreground shell reads these entry lines: + +```text +title: +status: +body:
+text: +action: