Repository navigation
feat(gateway): send a model this node does not serve to a peer - #297
Merged
Merged
Conversation
A [[peer]] entry names another node's gateway on the tailnet and a file with that node's key. A plain /v1 request for a model no engine here serves goes to the first peer that lists it, under the peer's key, and GET /v1/models lists the peers' models with owned_by set to the peer. The peer's berthd leases its own card; no lease crosses a machine. A peer answers a list request with its own models only (X-Berth-Peer), so two nodes that name each other cannot pass a request back and forth. The peer URL must be a *.ts.net name, a tailnet address or loopback, because the key goes with every request.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is the first step toward one URL for every model on the tailnet (stage D in
docs/design.md).What it does
A
[[peer]]entry inberth.tomlnames another node's gateway and an owner-private file that holds that node's key:/v1chat, completion or embedding request for a model that no engine on this node serves goes to the first peer that lists the model. The request uses the peer's key. The client's key belongs to this node and is not sent on.GET /v1/modelsadds the peers' models, withowned_byset to the peer's name./cards/CARD/v1stays local. This node answers first.X-Berth-Peer: 1) with its own models only. A node forwards only to a node that serves the model itself, so two nodes that name each other cannot send a request back and forth.berth config checkrefuses a peer URL that is not a*.ts.netname, a tailnet address (100.64.0.0/10,fd7a:115c:a1e0::/48) or loopback, because the key goes with every request.When no
[[peer]]is set, nothing changes.Proof
TestAModelOnlyAPeerServesGoesToThePeer: two real gateways, each with its own berthd and key. The model is listed under the peer's name, and the peer's engine answers the stream. This node takes no lease.TestPeersDoNotPassARequestBackAndForth: two nodes name each other, and a model neither serves gets a 404 at once. Without the list-side header, the test fails after 5 s of ping-pong (mutation run).TestAPeerThatIsDownListsNothing: the local list and the local 404 are unchanged.TestPeerURLStaysOnTheTailnet,TestPeerNeedsANameAKeyFileAndNoTwin: the config rules.peer-echoas owned bynodeb, and a chat to A was answered by B's engine. The engine received no Authorization header. A's key sent straight to B got 401. The production lease table and the GPU did not change.The full proof passes: build, gofmt, vet, tests, the Windows build and vet, and golangci-lint.
Limits
key_fileyet, because owner-private files are checked only on Unix. This matchesapi_key_file.