From 64738eeec115157ae5c4502c866bad42466aa2d8 Mon Sep 17 00:00:00 2001 From: Ho1yShif Date: Fri, 10 Jul 2026 00:14:19 -0700 Subject: [PATCH 1/2] fix(webui): persist chat history and WebUI transcripts across deploys Chat history did not fully survive redeploys: the sidebar listed past conversations but opening one showed an empty thread. Root cause was nanobot's data_dir resolving to /app (baked into the image, wiped every deploy), so the WebUI display transcripts under data_dir/webui were lost while session files on the persistent disk survived. - entrypoint.sh: copy the selected config template onto the mounted disk and run with --config on the disk, so data_dir resolves under the mount (webui/, cron, media, logs now persist). Overwrite on every boot; secrets stay as ${VAR} placeholders resolved in memory. - transcript.py: build_webui_thread_response reconstructs the thread from the durable session file when the display transcript is missing, instead of 404ing. Recovers orphaned chats and self-heals future transcript loss. - render.yaml: correct the misleading disk comment; stays Starter / 1 GB. - README: HeyGen offload recipe for the hyperframes CLI App, and a note that history persistence needs the disk, not a bigger plan. Co-Authored-By: Claude Opus 4.8 (1M context) --- README.md | 25 ++++++++++- entrypoint.sh | 19 ++++++-- nanobot/webui/transcript.py | 30 ++++++++++--- render.yaml | 16 ++++--- tests/utils/test_webui_transcript.py | 66 ++++++++++++++++++++++++++++ 5 files changed, 140 insertions(+), 16 deletions(-) diff --git a/README.md b/README.md index 86d07fc6..f0e5832a 100644 --- a/README.md +++ b/README.md @@ -14,7 +14,7 @@ One Render web service running nanobot's `gateway` and its bundled WebUI. You ch This is not a free deploy. Expect: -- **Render Starter plan** — ~$7/mo for the web service. Runs the core experience well; resource-intensive tasks (npm-based CLI Apps, image rendering) may need a larger plan — see the note under [Deploy](#deploy). +- **Render Starter plan** — ~$7/mo for the web service. Runs the core experience well; resource-intensive tasks (npm-based CLI Apps, image rendering) may need a larger plan — see [Resource sizing](#resource-sizing) under Deploy. - **1 GB persistent disk** — ~$0.25/mo for sessions and memory. - **Anthropic API usage** — billed separately by Anthropic based on your agent's LLM calls. @@ -35,7 +35,28 @@ This is not a free deploy. Expect: 5. The WebUI shows an access prompt — paste the same `NANOBOT_WEB_TOKEN` to sign in. It's stored only in your browser. 6. Send a message. You're talking to your agent. -**Note:** The Starter plan comfortably runs the core experience — chat, web search/fetch, and persistent memory. Resource-intensive actions, such as installing npm-based CLI Apps or generating/rendering images, can briefly overwhelm the small instance: it may trigger a ~15s restart before recovering, or fail with `No space left on device`. To run these reliably, upgrade to a larger plan (e.g. **Standard**, 2 GB) under **Settings → Instance Type**, and/or increase your persistent disk to 5 GB or larger on the **Disk** page, in your Render dashboard. +### Resource sizing + +The Starter plan comfortably runs the core experience — chat, web search/fetch, and persistent memory. Resource-intensive actions, such as installing npm-based CLI Apps or generating/rendering images, can briefly overwhelm the small instance: it may trigger a ~15s restart before recovering, or fail with `No space left on device`. To run these reliably, upgrade to a larger plan (e.g. **Standard**, 2 GB) under **Settings → Instance Type**, and/or increase your persistent disk to 5 GB or larger on the **Disk** page, in your Render dashboard. + +**Chat history persistence needs the disk, not a bigger plan.** Sessions, memory, and the chat transcripts the WebUI renders all live on the persistent disk and survive deploys on the default Starter plan. Upgrading the instance type is only about RAM/CPU for heavy in-process work — it doesn't affect what's retained across deploys. + +### Heavy CLI Apps (e.g. video rendering) + +Some CLI Apps do work the chat service isn't sized for. Two limits to know: + +- **System dependencies aren't bundled.** Installing a CLI App only runs its package-manager install (`npm install -g`, `pip`, etc.) — it does *not* install system binaries like `ffmpeg` or a headless browser (Chrome/Chromium). Apps that need those (for example, browser-based video renderers) won't run on the stock image, because the container runs as a non-root user and can't `apt install` at runtime. Getting them working means baking those packages into the [`Dockerfile`](./Dockerfile) and rebuilding. +- **In-process rendering is memory-hungry.** A single headless-browser + `ffmpeg` render can need 1–2 GB of RAM and multiple cores — more than Starter (512 MB) and often more than Standard (2 GB). Baking the dependencies in doesn't change this; the render *process* is what consumes the memory, so it will still hit the instance's cap. + +Because this is a chat app, the recommended approach is **not** to grow the web service around occasional heavy renders. Instead, **offload the render**: use the app's own hosted/cloud render if it has one (for example, hyperframes' HeyGen cloud render — no local browser or `ffmpeg` needed), which keeps this service small and responsive. Only bake in the system dependencies and size up the plan if you specifically want rendering to happen in-process on this instance. + +**Example — set up HeyGen for the hyperframes CLI App.** Some installable apps benefit from wiring up a hosted render backend. To render video with hyperframes without local Chrome/`ffmpeg` or a plan upgrade: + +1. Get a HeyGen API key (HeyGen renders are HeyGen-hosted and pay-per-credit). +2. In the Render dashboard, add a `HEYGEN_API_KEY` environment variable on the service (**Settings → Environment**), then let the service redeploy. +3. Ask the agent to render via `hyperframes cloud render …`; it picks up the key from the environment. + +Don't try `hyperframes auth login` — it's an interactive browser flow that can't complete inside a headless container. Use the `HEYGEN_API_KEY` env var instead. It isn't part of the base Blueprint, so add it only if you install hyperframes; the default deploy keeps just `ANTHROPIC_API_KEY` and `NANOBOT_WEB_TOKEN`. ## Environment variables diff --git a/entrypoint.sh b/entrypoint.sh index d206340b..3155232a 100755 --- a/entrypoint.sh +++ b/entrypoint.sh @@ -23,13 +23,24 @@ esac dir="$HOME/.nanobot" mkdir -p "$dir" || echo "[entrypoint] warning: mkdir $dir failed" +# Copy the selected config onto the mounted disk so nanobot's data_dir +# (config_path.parent) resolves under the mount. Otherwise data_dir is /app — +# baked into the image and wiped every deploy — which loses the WebUI display +# transcripts (webui/), cron, media and logs even though session files persist. +# The committed template is the source of truth, so overwrite on every boot; +# secrets stay as ${VAR} placeholders in the file and are interpolated in memory +# at load time, so nothing secret is written to disk. +RUNTIME_CONFIG="$dir/config.json" +cp "$CONFIG" "$RUNTIME_CONFIG" || echo "[entrypoint] warning: cp $CONFIG -> $RUNTIME_CONFIG failed" + if [ "$(id -u)" != "0" ]; then # Already non-root (platform forced a user). Can't chown; just run. echo "[entrypoint] not root — running app as $(id -un)" - exec nanobot "$@" --config "$CONFIG" + exec nanobot "$@" --config "$RUNTIME_CONFIG" fi -# Running as root: make the mounted disk writable by the app user. +# Running as root: make the mounted disk (incl. the copied config) writable and +# readable by the app user. chown -R nanobot:nanobot "$dir" || echo "[entrypoint] warning: chown $dir failed" # Drop to the non-root user if the runtime grants the capability to do so @@ -37,8 +48,8 @@ chown -R nanobot:nanobot "$dir" || echo "[entrypoint] warning: chown $dir failed # still write the root-owned disk. if setpriv --reuid=nanobot --regid=nanobot --init-groups true 2>/dev/null; then echo "[entrypoint] dropping privileges to nanobot via setpriv" - exec setpriv --reuid=nanobot --regid=nanobot --init-groups nanobot "$@" --config "$CONFIG" + exec setpriv --reuid=nanobot --regid=nanobot --init-groups nanobot "$@" --config "$RUNTIME_CONFIG" fi echo "[entrypoint] setpriv privilege-drop not permitted — running app as root" -exec nanobot "$@" --config "$CONFIG" +exec nanobot "$@" --config "$RUNTIME_CONFIG" diff --git a/nanobot/webui/transcript.py b/nanobot/webui/transcript.py index a1b4dec1..93b1bc2d 100644 --- a/nanobot/webui/transcript.py +++ b/nanobot/webui/transcript.py @@ -775,12 +775,17 @@ def append_fork_marker(session_key: str) -> None: ) -def write_session_messages_as_transcript( - target_key: str, +def session_messages_to_transcript_lines( + session_key: str, messages: list[dict[str, Any]], -) -> None: - """Write a minimal WebUI transcript from already-truncated session messages.""" - target_chat_id = _chat_id_from_session_key(target_key) +) -> list[dict[str, Any]]: + """Build minimal WebUI transcript rows from persisted session messages. + + The durable session file is the source of truth when the ephemeral display + transcript is missing; these rows replay through + ``replay_transcript_to_ui_messages`` just like real transcript lines. + """ + target_chat_id = _chat_id_from_session_key(session_key) rows: list[dict[str, Any]] = [] for msg in messages: if is_hidden_history_message(msg): @@ -805,6 +810,15 @@ def write_session_messages_as_transcript( else: continue rows.append(row) + return rows + + +def write_session_messages_as_transcript( + target_key: str, + messages: list[dict[str, Any]], +) -> None: + """Write a minimal WebUI transcript from already-truncated session messages.""" + rows = session_messages_to_transcript_lines(target_key, messages) _write_transcript_lines(target_key, rows) @@ -1930,6 +1944,12 @@ def build_webui_thread_response( lines, page = _select_transcript_page(session_key, limit=limit, before=before) else: lines = read_transcript_lines(session_key) + if not lines and session_messages: + # The display transcript lives on the (historically ephemeral) data dir and + # can be wiped by a redeploy even though the durable session file survives. + # Reconstruct the thread from the session messages so orphaned chats render + # instead of 404ing, and so any future transcript loss self-heals. + lines = session_messages_to_transcript_lines(session_key, session_messages) if not lines: return None lines = inject_missing_user_events_from_session(session_key, lines, session_messages) diff --git a/render.yaml b/render.yaml index 97288239..74ec51d5 100644 --- a/render.yaml +++ b/render.yaml @@ -11,9 +11,11 @@ services: # Render's Docker Command REPLACES the Dockerfile ENTRYPOINT (it is not # appended to it), so invoke the entrypoint explicitly. The entrypoint # prepares the mounted disk, drops to the non-root user when permitted, and - # picks the config: `--config /app/render-config.json` normally, or - # `--config /app/render-demo-config.json` when DEMO is true. It appends the - # --config flag itself, so pass only `gateway` here. + # copies the selected config template onto the disk + # (render-config.json normally, or render-demo-config.json when DEMO is + # true) at $HOME/.nanobot/config.json — so the runtime data_dir lands on the + # persistent disk. It appends the --config flag itself, so pass only + # `gateway` here. dockerCommand: /usr/local/bin/entrypoint.sh gateway plan: starter healthCheckPath: / @@ -44,8 +46,12 @@ services: # Total messages per browser session (each browser gets its own session). - key: DEMO_MAX_MESSAGES_PER_SESSION value: 30 - # Persist sessions / memory across deploys. Config lives in /app (code dir), - # so this mount does not shadow it. + # Persist sessions, memory, and the WebUI display transcripts across deploys. + # The entrypoint copies the config onto this mount, so nanobot's runtime + # data_dir (config_path.parent) resolves here too — keeping webui/ (chat + # history the UI renders), cron, media, and logs durable, not just session + # files. Starter / 1 GB is the lean default; history persistence needs this + # disk, not a bigger plan. disk: name: nanobot-data mountPath: /home/nanobot/.nanobot diff --git a/tests/utils/test_webui_transcript.py b/tests/utils/test_webui_transcript.py index 7242b792..08754d85 100644 --- a/tests/utils/test_webui_transcript.py +++ b/tests/utils/test_webui_transcript.py @@ -607,6 +607,72 @@ def test_replay_uses_stream_end_final_text() -> None: assert msgs[1]["content"] == "![Diagram](/api/media/sig/payload)" +def test_build_response_reconstructs_thread_from_session_when_transcript_missing( + tmp_path, + monkeypatch, +) -> None: + """A wiped display transcript should self-heal from the durable session file.""" + monkeypatch.setattr("nanobot.config.paths.get_data_dir", lambda: tmp_path) + key = "websocket:orphaned" + # No transcript lines were ever written (the ephemeral webui dir was wiped on + # redeploy), but the persisted session file still holds the conversation. + assert read_transcript_lines(key) == [] + + out = build_webui_thread_response( + key, + session_messages=[ + {"role": "user", "content": "hello", "timestamp": "2026-06-02T10:00:00"}, + {"role": "assistant", "content": "hi there"}, + {"role": "user", "content": "how are you?", "timestamp": "2026-06-02T10:01:00"}, + {"role": "assistant", "content": "doing well"}, + ], + ) + + assert out is not None + assert [(m["role"], m["content"]) for m in out["messages"]] == [ + ("user", "hello"), + ("assistant", "hi there"), + ("user", "how are you?"), + ("assistant", "doing well"), + ] + + +def test_build_response_reconstructs_paginated_thread_from_session( + tmp_path, + monkeypatch, +) -> None: + """The live endpoint paginates (limit/direction); the fallback must serve it too.""" + monkeypatch.setattr("nanobot.config.paths.get_data_dir", lambda: tmp_path) + key = "websocket:orphaned-page" + + out = build_webui_thread_response( + key, + session_messages=[ + {"role": "user", "content": "hello"}, + {"role": "assistant", "content": "hi there"}, + ], + limit=160, + direction="latest", + ) + + assert out is not None + assert [(m["role"], m["content"]) for m in out["messages"]] == [ + ("user", "hello"), + ("assistant", "hi there"), + ] + assert out["page"]["loaded_message_count"] == 2 + assert out["page"]["has_more_before"] is False + + +def test_build_response_still_none_when_transcript_and_session_both_empty( + tmp_path, + monkeypatch, +) -> None: + monkeypatch.setattr("nanobot.config.paths.get_data_dir", lambda: tmp_path) + assert build_webui_thread_response("websocket:nothing", session_messages=[]) is None + assert build_webui_thread_response("websocket:nothing", session_messages=None) is None + + def test_build_response_backfills_legacy_sse_only_transcripts(tmp_path, monkeypatch) -> None: monkeypatch.setattr("nanobot.config.paths.get_data_dir", lambda: tmp_path) key = "websocket:t-legacy" From 485c2808e73b47ec5ca9b8322ba3f4a0e6ae2615 Mon Sep 17 00:00:00 2001 From: Ho1yShif Date: Fri, 10 Jul 2026 00:42:38 -0700 Subject: [PATCH 2/2] fix(matrix): allow mxc:/matrix: schemes through markdown so nh3 can keep them mistune's HTMLRenderer treats any non-http(s) scheme as harmful and rewrites it to "#harmful-link" before the nh3 cleaner runs. That stripped mxc:// image sources and matrix: links even though MATRIX_HTML_CLEANER is explicitly built to preserve them (via _filter_matrix_html_attribute and MATRIX_ALLOWED_URL_SCHEMES), breaking test_send_keeps_only_mxc_image_sources. Pass an HTMLRenderer with allow_harmful_protocols=("mxc:", "matrix:") so mistune emits the URLs unchanged and nh3 remains the single security gate. Non-mxc image srcs, javascript:, and data: URLs are still stripped by nh3 as before. Co-Authored-By: Claude Opus 4.8 (1M context) --- nanobot/channels/matrix.py | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/nanobot/channels/matrix.py b/nanobot/channels/matrix.py index 481b2b86..f9553ad8 100644 --- a/nanobot/channels/matrix.py +++ b/nanobot/channels/matrix.py @@ -18,7 +18,7 @@ try: import aiohttp import nh3 - from mistune import create_markdown + from mistune import HTMLRenderer, create_markdown from nio import ( AsyncClient, AsyncClientConfig, @@ -76,8 +76,11 @@ class _MediaTooLargeError(Exception): """Raised when an inbound Matrix media download exceeds the configured cap.""" +# mistune treats any non-http(s) scheme as harmful and rewrites it to +# "#harmful-link" before nh3 ever runs. Allow the Matrix schemes through here so +# nh3 (MATRIX_HTML_CLEANER below) remains the single security gate for URLs. MATRIX_MARKDOWN = create_markdown( - escape=True, + renderer=HTMLRenderer(escape=True, allow_harmful_protocols=("mxc:", "matrix:")), plugins=["table", "strikethrough", "url", "superscript", "subscript"], )