fix(webui): persist chat history and WebUI transcripts across deploys

README.md

@@ -14,7 +14,7 @@ One Render web service running nanobot's `gateway` and its bundled WebUI. You ch
 
 This is not a free deploy. Expect:
 
-- **Render Starter plan** — ~$7/mo for the web service. Runs the core experience well; resource-intensive tasks (npm-based CLI Apps, image rendering) may need a larger plan — see the note under [Deploy](#deploy).
+- **Render Starter plan** — ~$7/mo for the web service. Runs the core experience well; resource-intensive tasks (npm-based CLI Apps, image rendering) may need a larger plan — see [Resource sizing](#resource-sizing) under Deploy.
 - **1 GB persistent disk** — ~$0.25/mo for sessions and memory.
 - **Anthropic API usage** — billed separately by Anthropic based on your agent's LLM calls.
 
@@ -35,7 +35,28 @@ This is not a free deploy. Expect:
 5. The WebUI shows an access prompt — paste the same `NANOBOT_WEB_TOKEN` to sign in. It's stored only in your browser.
 6. Send a message. You're talking to your agent.
 
-**Note:** The Starter plan comfortably runs the core experience — chat, web search/fetch, and persistent memory. Resource-intensive actions, such as installing npm-based CLI Apps or generating/rendering images, can briefly overwhelm the small instance: it may trigger a ~15s restart before recovering, or fail with `No space left on device`. To run these reliably, upgrade to a larger plan (e.g. **Standard**, 2 GB) under **Settings → Instance Type**, and/or increase your persistent disk to 5 GB or larger on the **Disk** page, in your Render dashboard.
+### Resource sizing
+
+The Starter plan comfortably runs the core experience — chat, web search/fetch, and persistent memory. Resource-intensive actions, such as installing npm-based CLI Apps or generating/rendering images, can briefly overwhelm the small instance: it may trigger a ~15s restart before recovering, or fail with `No space left on device`. To run these reliably, upgrade to a larger plan (e.g. **Standard**, 2 GB) under **Settings → Instance Type**, and/or increase your persistent disk to 5 GB or larger on the **Disk** page, in your Render dashboard.
+
+**Chat history persistence needs the disk, not a bigger plan.** Sessions, memory, and the chat transcripts the WebUI renders all live on the persistent disk and survive deploys on the default Starter plan. Upgrading the instance type is only about RAM/CPU for heavy in-process work — it doesn't affect what's retained across deploys.
+
+### Heavy CLI Apps (e.g. video rendering)
+
+Some CLI Apps do work the chat service isn't sized for. Two limits to know:
+
+- **System dependencies aren't bundled.** Installing a CLI App only runs its package-manager install (`npm install -g`, `pip`, etc.) — it does *not* install system binaries like `ffmpeg` or a headless browser (Chrome/Chromium). Apps that need those (for example, browser-based video renderers) won't run on the stock image, because the container runs as a non-root user and can't `apt install` at runtime. Getting them working means baking those packages into the [`Dockerfile`](./Dockerfile) and rebuilding.
+- **In-process rendering is memory-hungry.** A single headless-browser + `ffmpeg` render can need 1–2 GB of RAM and multiple cores — more than Starter (512 MB) and often more than Standard (2 GB). Baking the dependencies in doesn't change this; the render *process* is what consumes the memory, so it will still hit the instance's cap.
+
+Because this is a chat app, the recommended approach is **not** to grow the web service around occasional heavy renders. Instead, **offload the render**: use the app's own hosted/cloud render if it has one (for example, hyperframes' HeyGen cloud render — no local browser or `ffmpeg` needed), which keeps this service small and responsive. Only bake in the system dependencies and size up the plan if you specifically want rendering to happen in-process on this instance.
+
+**Example — set up HeyGen for the hyperframes CLI App.** Some installable apps benefit from wiring up a hosted render backend. To render video with hyperframes without local Chrome/`ffmpeg` or a plan upgrade:
+
+1. Get a HeyGen API key (HeyGen renders are HeyGen-hosted and pay-per-credit).
+2. In the Render dashboard, add a `HEYGEN_API_KEY` environment variable on the service (**Settings → Environment**), then let the service redeploy.
+3. Ask the agent to render via `hyperframes cloud render …`; it picks up the key from the environment.
+
+Don't try `hyperframes auth login` — it's an interactive browser flow that can't complete inside a headless container. Use the `HEYGEN_API_KEY` env var instead. It isn't part of the base Blueprint, so add it only if you install hyperframes; the default deploy keeps just `ANTHROPIC_API_KEY` and `NANOBOT_WEB_TOKEN`.
 
 ## Environment variables
 

entrypoint.sh

@@ -23,22 +23,33 @@ esac
 dir="$HOME/.nanobot"
 mkdir -p "$dir" || echo "[entrypoint] warning: mkdir $dir failed"
 
+# Copy the selected config onto the mounted disk so nanobot's data_dir
+# (config_path.parent) resolves under the mount. Otherwise data_dir is /app —
+# baked into the image and wiped every deploy — which loses the WebUI display
+# transcripts (webui/), cron, media and logs even though session files persist.
+# The committed template is the source of truth, so overwrite on every boot;
+# secrets stay as ${VAR} placeholders in the file and are interpolated in memory
+# at load time, so nothing secret is written to disk.
+RUNTIME_CONFIG="$dir/config.json"
+cp "$CONFIG" "$RUNTIME_CONFIG" || echo "[entrypoint] warning: cp $CONFIG -> $RUNTIME_CONFIG failed"
+
 if [ "$(id -u)" != "0" ]; then
     # Already non-root (platform forced a user). Can't chown; just run.
     echo "[entrypoint] not root — running app as $(id -un)"
-    exec nanobot "$@" --config "$CONFIG"
+    exec nanobot "$@" --config "$RUNTIME_CONFIG"
 fi
 
-# Running as root: make the mounted disk writable by the app user.
+# Running as root: make the mounted disk (incl. the copied config) writable and
+# readable by the app user.
 chown -R nanobot:nanobot "$dir" || echo "[entrypoint] warning: chown $dir failed"
 
 # Drop to the non-root user if the runtime grants the capability to do so
 # (setpriv needs CAP_SETUID/CAP_SETGID). Otherwise stay root so the app can
 # still write the root-owned disk.
 if setpriv --reuid=nanobot --regid=nanobot --init-groups true 2>/dev/null; then
     echo "[entrypoint] dropping privileges to nanobot via setpriv"
-    exec setpriv --reuid=nanobot --regid=nanobot --init-groups nanobot "$@" --config "$CONFIG"
+    exec setpriv --reuid=nanobot --regid=nanobot --init-groups nanobot "$@" --config "$RUNTIME_CONFIG"
 fi
 
 echo "[entrypoint] setpriv privilege-drop not permitted — running app as root"
-exec nanobot "$@" --config "$CONFIG"
+exec nanobot "$@" --config "$RUNTIME_CONFIG"

nanobot/channels/matrix.py

@@ -18,7 +18,7 @@
 try:
     import aiohttp
     import nh3
-    from mistune import create_markdown
+    from mistune import HTMLRenderer, create_markdown
     from nio import (
         AsyncClient,
         AsyncClientConfig,
@@ -76,8 +76,11 @@
 class _MediaTooLargeError(Exception):
     """Raised when an inbound Matrix media download exceeds the configured cap."""
 
+# mistune treats any non-http(s) scheme as harmful and rewrites it to
+# "#harmful-link" before nh3 ever runs. Allow the Matrix schemes through here so
+# nh3 (MATRIX_HTML_CLEANER below) remains the single security gate for URLs.
 MATRIX_MARKDOWN = create_markdown(
-    escape=True,
+    renderer=HTMLRenderer(escape=True, allow_harmful_protocols=("mxc:", "matrix:")),
     plugins=["table", "strikethrough", "url", "superscript", "subscript"],
 )
 

nanobot/webui/transcript.py

@@ -775,12 +775,17 @@ def append_fork_marker(session_key: str) -> None:
     )
 
 
-def write_session_messages_as_transcript(
-    target_key: str,
+def session_messages_to_transcript_lines(
+    session_key: str,
     messages: list[dict[str, Any]],
-) -> None:
-    """Write a minimal WebUI transcript from already-truncated session messages."""
-    target_chat_id = _chat_id_from_session_key(target_key)
+) -> list[dict[str, Any]]:
+    """Build minimal WebUI transcript rows from persisted session messages.
+
+    The durable session file is the source of truth when the ephemeral display
+    transcript is missing; these rows replay through
+    ``replay_transcript_to_ui_messages`` just like real transcript lines.
+    """
+    target_chat_id = _chat_id_from_session_key(session_key)
     rows: list[dict[str, Any]] = []
     for msg in messages:
         if is_hidden_history_message(msg):
@@ -805,6 +810,15 @@ def write_session_messages_as_transcript(
         else:
             continue
         rows.append(row)
+    return rows
+
+
+def write_session_messages_as_transcript(
+    target_key: str,
+    messages: list[dict[str, Any]],
+) -> None:
+    """Write a minimal WebUI transcript from already-truncated session messages."""
+    rows = session_messages_to_transcript_lines(target_key, messages)
     _write_transcript_lines(target_key, rows)
 
 
@@ -1930,6 +1944,12 @@ def build_webui_thread_response(
         lines, page = _select_transcript_page(session_key, limit=limit, before=before)
     else:
         lines = read_transcript_lines(session_key)
+    if not lines and session_messages:
+        # The display transcript lives on the (historically ephemeral) data dir and
+        # can be wiped by a redeploy even though the durable session file survives.
+        # Reconstruct the thread from the session messages so orphaned chats render
+        # instead of 404ing, and so any future transcript loss self-heals.
+        lines = session_messages_to_transcript_lines(session_key, session_messages)
     if not lines:
         return None
     lines = inject_missing_user_events_from_session(session_key, lines, session_messages)

render.yaml

@@ -11,9 +11,11 @@ services:
     # Render's Docker Command REPLACES the Dockerfile ENTRYPOINT (it is not
     # appended to it), so invoke the entrypoint explicitly. The entrypoint
     # prepares the mounted disk, drops to the non-root user when permitted, and
-    # picks the config: `--config /app/render-config.json` normally, or
-    # `--config /app/render-demo-config.json` when DEMO is true. It appends the
-    # --config flag itself, so pass only `gateway` here.
+    # copies the selected config template onto the disk
+    # (render-config.json normally, or render-demo-config.json when DEMO is
+    # true) at $HOME/.nanobot/config.json — so the runtime data_dir lands on the
+    # persistent disk. It appends the --config flag itself, so pass only
+    # `gateway` here.
     dockerCommand: /usr/local/bin/entrypoint.sh gateway
     plan: starter
     healthCheckPath: /
@@ -44,8 +46,12 @@ services:
       # Total messages per browser session (each browser gets its own session).
       - key: DEMO_MAX_MESSAGES_PER_SESSION
         value: 30
-    # Persist sessions / memory across deploys. Config lives in /app (code dir),
-    # so this mount does not shadow it.
+    # Persist sessions, memory, and the WebUI display transcripts across deploys.
+    # The entrypoint copies the config onto this mount, so nanobot's runtime
+    # data_dir (config_path.parent) resolves here too — keeping webui/ (chat
+    # history the UI renders), cron, media, and logs durable, not just session
+    # files. Starter / 1 GB is the lean default; history persistence needs this
+    # disk, not a bigger plan.
     disk:
       name: nanobot-data
       mountPath: /home/nanobot/.nanobot

tests/utils/test_webui_transcript.py

@@ -607,6 +607,72 @@ def test_replay_uses_stream_end_final_text() -> None:
     assert msgs[1]["content"] == "![Diagram](/api/media/sig/payload)"
 
 
+def test_build_response_reconstructs_thread_from_session_when_transcript_missing(
+    tmp_path,
+    monkeypatch,
+) -> None:
+    """A wiped display transcript should self-heal from the durable session file."""
+    monkeypatch.setattr("nanobot.config.paths.get_data_dir", lambda: tmp_path)
+    key = "websocket:orphaned"
+    # No transcript lines were ever written (the ephemeral webui dir was wiped on
+    # redeploy), but the persisted session file still holds the conversation.
+    assert read_transcript_lines(key) == []
+
+    out = build_webui_thread_response(
+        key,
+        session_messages=[
+            {"role": "user", "content": "hello", "timestamp": "2026-06-02T10:00:00"},
+            {"role": "assistant", "content": "hi there"},
+            {"role": "user", "content": "how are you?", "timestamp": "2026-06-02T10:01:00"},
+            {"role": "assistant", "content": "doing well"},
+        ],
+    )
+
+    assert out is not None
+    assert [(m["role"], m["content"]) for m in out["messages"]] == [
+        ("user", "hello"),
+        ("assistant", "hi there"),
+        ("user", "how are you?"),
+        ("assistant", "doing well"),
+    ]
+
+
+def test_build_response_reconstructs_paginated_thread_from_session(
+    tmp_path,
+    monkeypatch,
+) -> None:
+    """The live endpoint paginates (limit/direction); the fallback must serve it too."""
+    monkeypatch.setattr("nanobot.config.paths.get_data_dir", lambda: tmp_path)
+    key = "websocket:orphaned-page"
+
+    out = build_webui_thread_response(
+        key,
+        session_messages=[
+            {"role": "user", "content": "hello"},
+            {"role": "assistant", "content": "hi there"},
+        ],
+        limit=160,
+        direction="latest",
+    )
+
+    assert out is not None
+    assert [(m["role"], m["content"]) for m in out["messages"]] == [
+        ("user", "hello"),
+        ("assistant", "hi there"),
+    ]
+    assert out["page"]["loaded_message_count"] == 2
+    assert out["page"]["has_more_before"] is False
+
+
+def test_build_response_still_none_when_transcript_and_session_both_empty(
+    tmp_path,
+    monkeypatch,
+) -> None:
+    monkeypatch.setattr("nanobot.config.paths.get_data_dir", lambda: tmp_path)
+    assert build_webui_thread_response("websocket:nothing", session_messages=[]) is None
+    assert build_webui_thread_response("websocket:nothing", session_messages=None) is None
+
+
 def test_build_response_backfills_legacy_sse_only_transcripts(tmp_path, monkeypatch) -> None:
     monkeypatch.setattr("nanobot.config.paths.get_data_dir", lambda: tmp_path)
     key = "websocket:t-legacy"