feat: add term-llm Hub and delegation

[sam-saffron-jarvis]

What

Adds term-llm Hub as a single dashboard/proxy over multiple term-llm web nodes, plus opt-in cross-node delegation.

Hub v1 includes:

• term-llm serve hub
• Hub bearer auth (--auth bearer, --token, TERM_LLM_HUB_TOKEN) with --auth none restricted to loopback
• static/config, local-store, and contain node discovery
• /node/<id>/... node UI proxy with server-side bearer-token injection
• Back to Hub UI context injection
• node health/status API with dashboard diagnostics
• direct and reverse/private node connection modes
• cross-node delegation APIs
• hub_delegate and hub_check_delegation tools
• jobs-v2 delegated run creation/trigger/status/cancel
• delegation reconnect/recovery hardening for partial job/trigger failures
• delegation ledger/audit trail
• target-node sidebar sessions titled like Delegation from jarvis-home
• Hub Delegations dashboard panel with returned artifact previews

Safety model

Delegation is default-off and opt-in per node:

delegation:
  enabled: true

Without delegation.enabled: true, a node can neither originate nor accept delegated work.

Accepting delegated work still requires a target workdir:

delegation:
  enabled: true
  accept_from: [jarvis-home]
  workdir: /work

Existing narrowing knobs remain available:

• to
• accept_from
• allowed_agents
• allowed_models
• max_in_flight

But the user-facing safety story is the simple switch: delegation participation is off until the operator enables it.

Hub auth

Hub now has first-party bearer auth:

TERM_LLM_HUB_TOKEN=... term-llm serve hub --auth bearer

• --auth bearer is the default.
• --auth none is allowed only on loopback.
• Public/non-loopback binds require Hub auth.
• /api/connect remains node-authenticated so reverse nodes can connect with their node token.
• Node-originated delegation calls keep using node auth (X-Term-LLM-Node-ID + node bearer token), not the operator Hub token.

This keeps Hub auth simple: one operator token now; no users/RBAC/OAuth in this PR.

Reverse/private nodes

Adds reverse connection mode for private nodes where the Hub has a public address but cannot reach the node directly.

Hub config:

nodes:
  - id: artist
    name: Artist
    connection: reverse
    base_path: /chat
    token: <artist-token>
    delegation:
      enabled: true
      accept_from: [jarvis-home]
      workdir: /work

Private node:

term-llm serve web jobs \
  --base-path /chat \
  --token "$ARTIST_TOKEN" \
  --hub-url https://hub.example.com \
  --hub-node-id artist \
  --hub-connect reverse

The node dials GET /api/connect as a websocket using its node id + bearer token. Hub requests then use the same node record and policy path as direct nodes; only the transport changes. Both ends send websocket pings every 20s, require pong/read activity within 60s, and use write deadlines so silent half-open sockets are detected and the node reconnects instead of sitting stale.

Reverse transport now streams responses in bounded chunks over the websocket, with cancellation propagation. Direct-node streams still use the normal reverse proxy path.

This keeps the architecture simple:

• direct nodes: Hub → node HTTP
• reverse nodes: node → Hub websocket, Hub sends request frames over it
• same /node/<id>/... proxy
• same delegation APIs
• same token injection/redaction model
• no offline queue or second delegation API

Dashboard diagnostics

Node cards now surface token-safe diagnostics for common bad setups:

• reverse node disconnected
• missing node bearer token
• delegation enabled without a workdir
• delegation accept policy present but no jobs capability detected
• obvious origin/target policy mismatches

The point is to show the rake before someone steps on it.

Delegation recovery hardening

Partial target failures are now more recoverable:

• delegation records are persisted before target job creation/trigger where possible
• known jobs without known runs stay refreshable instead of becoming dead terminal errors
• trigger-response-loss can recover by polling target runs
• cancel refreshes/reuses recovered run IDs
• stale transport errors clear when a later refresh finds the real run state

Still no offline queue. Offline nodes fail fast; reconnect lets subsequent refresh/cancel paths recover known work.

Node-anywhere docs

Docs cover both modes:

• direct nodes can run on the same machine, Docker/contain, VM, cloud runner, remote server, private network/tunnel
• reverse nodes can run behind NAT/firewalls with no inbound port, as long as they can dial the Hub

Demo

Standalone isolated demo video, using two clean node homes/session stores rather than Sam's real Jarvis setup:

• Jarvis Home asks Artist to draw a picture
• Hub dashboard shows delegated work
• Artist sidebar shows Delegation from jarvis-home
• back to Hub after success
• original Jarvis Home session shows the returned image

Video artifact:

/chat/files/term-llm-hub-standalone-demo.mp4

Tests

go test ./cmd ./internal/hub
go test ./...
go build ./...

Live smoke:

Hub bearer auth rejects unauthenticated /api/nodes
Hub configured with connection: reverse
Private node started with --hub-connect reverse
GET /api/nodes shows connection=reverse and connected status
GET /node/private/healthz succeeds through the reverse websocket