Skip to content

feat(gateway): send a model this node does not serve to a peer - #297

Merged
T-Crypt merged 2 commits into
nextfrom
T-Crypt/gateway-peers
Nov 3, 2026
Merged

T-Crypt merged 2 commits into
nextfrom
T-Crypt/gateway-peers

Conversation

@T-Crypt

@T-Crypt T-Crypt commented Nov 3, 2026

Copy link
Copy Markdown
Owner

This is the first step toward one URL for every model on the tailnet (stage D in docs/design.md).

What it does

A [[peer]] entry in berth.toml names another node's gateway and an owner-private file that holds that node's key:

[[peer]]
name     = "galaxy"
url      = "https://galaxy.tail2a8bb.ts.net:9799"
key_file = "/home/you/.config/berth/peers/galaxy.key"
  • A plain /v1 chat, completion or embedding request for a model that no engine on this node serves goes to the first peer that lists the model. The request uses the peer's key. The client's key belongs to this node and is not sent on.
  • A plain GET /v1/models adds the peers' models, with owned_by set to the peer's name.
  • The peer's own berthd leases its own card. A lease never crosses a machine, so design rule 1 (the driver decides) still holds on each node.
  • /cards/CARD/v1 stays local. This node answers first.
  • A peer answers a list request (X-Berth-Peer: 1) with its own models only. A node forwards only to a node that serves the model itself, so two nodes that name each other cannot send a request back and forth.
  • berth config check refuses a peer URL that is not a *.ts.net name, a tailnet address (100.64.0.0/10, fd7a:115c:a1e0::/48) or loopback, because the key goes with every request.
  • The gateway reads each peer's key when it starts. If a key file does not read, the gateway exits with the reason.

When no [[peer]] is set, nothing changes.

Proof

  • TestAModelOnlyAPeerServesGoesToThePeer: two real gateways, each with its own berthd and key. The model is listed under the peer's name, and the peer's engine answers the stream. This node takes no lease.
  • TestPeersDoNotPassARequestBackAndForth: two nodes name each other, and a model neither serves gets a 404 at once. Without the list-side header, the test fails after 5 s of ping-pong (mutation run).
  • TestAPeerThatIsDownListsNothing: the local list and the local 404 are unchanged.
  • TestPeerURLStaysOnTheTailnet, TestPeerNeedsANameAKeyFileAndNoTwin: the config rules.
  • Live on aphotic: two throwaway gateways on loopback (A on :19800, B on :19801), each with its own key, and a CPU stub engine behind B. A listed peer-echo as owned by nodeb, and a chat to A was answered by B's engine. The engine received no Authorization header. A's key sent straight to B got 401. The production lease table and the GPU did not change.

The full proof passes: build, gofmt, vet, tests, the Windows build and vet, and golangci-lint.

Limits

  • A Windows gateway cannot read key_file yet, because owner-private files are checked only on Unix. This matches api_key_file.
  • The first peer that lists a model wins, even when another peer has the model warm. Ranking nodes by reach is the next slice on the roadmap, followed by peer cards on the harbor's SWITCH tab.

A [[peer]] entry names another node's gateway on the tailnet and a file
with that node's key. A plain /v1 request for a model no engine here
serves goes to the first peer that lists it, under the peer's key, and
GET /v1/models lists the peers' models with owned_by set to the peer.
The peer's berthd leases its own card; no lease crosses a machine.

A peer answers a list request with its own models only (X-Berth-Peer),
so two nodes that name each other cannot pass a request back and forth.
The peer URL must be a *.ts.net name, a tailnet address or loopback,
because the key goes with every request.
@T-Crypt
T-Crypt merged commit f67570b into next Nov 3, 2026
4 checks passed
@T-Crypt
T-Crypt deleted the T-Crypt/gateway-peers branch November 3, 2026 17:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant