feat(cluster): RDMA smoke test + encrypted pipeline inference over Thunderbolt#192
Open
anupsv wants to merge 10 commits into
Open
feat(cluster): RDMA smoke test + encrypted pipeline inference over Thunderbolt#192anupsv wants to merge 10 commits into
anupsv wants to merge 10 commits into
Conversation
…in-backed Reflects commit 4a0dae5 (PersistentEnclaveKey.swift). Key changes: - TB-003 how_it_works: document Security framework persistent key, access group SLDQ2GJ6TL.io.darkbloom.provider, kSecAttrIsPermanent, and the errSecMissingEntitlement fallback behaviour on patched binaries - TB-003 current_limitations: add two new limitations — team-scoped cross-binary keychain access, and silent ephemeral fallback that defers rejection to the coordinator rather than failing at the process boundary - TB-009 how_it_works: rewrite SE key lifecycle section to reflect persistent identity across restarts; rotation now requires explicit keychain deletion - T-013 (binary tampering) mitigations: add keychain access group enforcement as a fourth, implemented mitigation; update detection_hint - T-033 (attestation replay) affected_files: add PersistentEnclaveKey.swift and AttestationSigner.swift; update mitigation wording - T-035 (repudiation after rotation) description, mitigations, detection_hint: reframe rotation as an explicit operator action rather than automatic per launch; note kSecAttrIsPermanent as a positive mitigation; add open finding that coordinator cannot detect opportunistic keychain delete + re-registration Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Minimal Swift package that tests jaccl/RDMA connectivity between two Apple Silicon Macs over Thunderbolt without any model stack. rdma-ping initializes a DistributedGroup via jaccl (MLX distributed backend), runs N rounds of all_sum across both ranks, verifies correctness, and reports per-round latency. Tested live: two M5 Max (128 GB) over Thunderbolt at ~40 µs avg latency, 20/20 rounds correct. Usage: # Mac A (rank 0): ./rdma-ping --rank 0 --coordinator 169.254.106.209:9999 --rdma-device rdma_en2 # Mac B (rank 1) simultaneously: ./rdma-ping --rank 1 --coordinator 169.254.106.209:9999 --rdma-device rdma_en1 Requires macOS 26.2+, RDMA enabled in Recovery, Thunderbolt cable, and mlx.metallib symlinked next to the binary. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Benchmark ResultsRunner: 1-provider-streaming1 providers, 1 users, 30 requests, concurrency=5, streaming=true
Latency Decomposition
Assertion Report: FAIL
1-provider-non-streaming1 providers, 1 users, 20 requests, concurrency=5, streaming=false
Latency Decomposition
Assertion Report: FAIL
7-provider-multi-model7 providers, 5 users, 50 requests, concurrency=10, streaming=true
Latency Decomposition
Assertion Report: FAIL
3-provider-high-concurrency3 providers, 10 users, 60 requests, concurrency=20, streaming=true
Latency Decomposition
Assertion Report: FAIL
1-provider-queue-saturation1 providers, 10 users, 40 requests, concurrency=15, streaming=true
Latency Decomposition
Assertion Report: FAIL
3-provider-20-users3 providers, 20 users, 60 requests, concurrency=10, streaming=true
Latency Decomposition
Assertion Report: FAIL
1-provider-scaling1 providers, 5 users, 30 requests, concurrency=10, streaming=true
Latency Decomposition
Assertion Report: FAIL
3-provider-scaling3 providers, 5 users, 30 requests, concurrency=10, streaming=true
Latency Decomposition
Assertion Report: FAIL
5-provider-scaling5 providers, 5 users, 30 requests, concurrency=10, streaming=true
Latency Decomposition
Assertion Report: FAIL
3-provider-heavy-100conc-10kb3 providers, 20 users, 100 requests, concurrency=100, streaming=true
Latency Decomposition
Assertion Report: FAIL
|
Benchmark ResultsRunner: 1-provider-streaming1 providers, 1 users, 30 requests, concurrency=5, streaming=true
Latency Decomposition
Assertion Report: FAIL
1-provider-non-streaming1 providers, 1 users, 20 requests, concurrency=5, streaming=false
Latency Decomposition
Assertion Report: FAIL
7-provider-multi-model7 providers, 5 users, 50 requests, concurrency=10, streaming=true
Latency Decomposition
Assertion Report: FAIL
3-provider-high-concurrency3 providers, 10 users, 60 requests, concurrency=20, streaming=true
Latency Decomposition
Assertion Report: FAIL
1-provider-queue-saturation1 providers, 10 users, 40 requests, concurrency=15, streaming=true
Latency Decomposition
Assertion Report: FAIL
3-provider-20-users3 providers, 20 users, 60 requests, concurrency=10, streaming=true
Latency Decomposition
Assertion Report: FAIL
1-provider-scaling1 providers, 5 users, 30 requests, concurrency=10, streaming=true
Latency Decomposition
Assertion Report: FAIL
3-provider-scaling3 providers, 5 users, 30 requests, concurrency=10, streaming=true
Latency Decomposition
Assertion Report: FAIL
5-provider-scaling5 providers, 5 users, 30 requests, concurrency=10, streaming=true
Latency Decomposition
Assertion Report: FAIL
3-provider-heavy-100conc-10kb3 providers, 20 users, 100 requests, concurrency=100, streaming=true
Latency Decomposition
Assertion Report: FAIL
|
Implements secure two-Mac pipeline-parallel inference using SE-backed
mutual authentication and AES-256-GCM for all tensor data in transit.
RDMA is used only for jaccl-based collective ops (allreduce); the
pipeline activation/token transfer path runs over ThunderboltLink TCP
with cryptographic protection, making it immune to librdma shim injection
and process memory attacks.
New P2P infrastructure:
- ThunderboltLink: TCP framing layer over wiredEthernet (bridge0), port
7777, 8-byte LE length-prefixed frames
- MLXDistributed: Swift wrappers for jaccl C API (DistributedGroup,
allSum, allGather, distributedSend/recv)
- ClusterControlMessage: wire types (HandshakeHello/Ack, PongPayload),
ClusterFrame encoding, ClusterHealth, ClusterError
- ClusterHandshake: SE mutual auth + ephemeral X25519 ECDH session key
(HKDF-SHA256, nonce0||nonce1 salt, "darkbloom-cluster-session-v1"
info). TOFU pinning: peer SE key stored at
/etc/darkbloom/cluster-peer-{IP}.sekey on first connection and
verified on all subsequent connections
- TensorCrypto: AES-256-GCM seal/open for MLXArray activations and
token IDs. Wire format: nonce(12) + ciphertext + tag(16). Shape
derived as [1, seqLen, hiddenDim] — seqLen embedded in plaintext
- ClusterSession: ClusterSession actor (rank 0) with connect loop,
5 s ping loop, 3-missed-ping teardown, and auto-reconnect with fresh
handshake each time. ClusterPeer class (rank 1) dispatches ping/pong
and inference frames
- EncryptedPipelineInference: EncryptedPipelineEngine actor (rank 0)
and EncryptedPipelineServer class (rank 1). Per-step retry: 3
attempts × 3 s delay, then ClusterError.serviceUnavailable (HTTP 429)
- PipelineInference: jaccl-based pipeline engine (kept for reference;
encrypted path is preferred for production)
CLI additions (darkbloom cluster):
- cluster setup: writes jaccl devices JSON + env file to /etc/darkbloom
- cluster devices: lists RDMA interfaces via ibv_devices
- cluster ping: jaccl all_sum smoke test with latency reporting
- cluster run: establishes ClusterSession (rank 0) or ClusterPeer
(rank 1) and prints health status; EncryptedPipelineEngine wired once
model loading is integrated
- link: ThunderboltLink bandwidth test (send/receive over bridge0)
Package.swift: add MLX as direct dependency of darkbloom executable
(ClusterPing uses MLXArray + eval inline)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
anupsv
changed the title
Benchmark ResultsRunner: 1-provider-streaming1 providers, 1 users, 30 requests, concurrency=5, streaming=true
Latency Decomposition
Assertion Report: FAIL
1-provider-non-streaming1 providers, 1 users, 20 requests, concurrency=5, streaming=false
Latency Decomposition
Assertion Report: FAIL
7-provider-multi-model7 providers, 5 users, 50 requests, concurrency=10, streaming=true
Latency Decomposition
Assertion Report: FAIL
3-provider-high-concurrency3 providers, 10 users, 60 requests, concurrency=20, streaming=true
Latency Decomposition
Assertion Report: FAIL
1-provider-queue-saturation1 providers, 10 users, 40 requests, concurrency=15, streaming=true
Latency Decomposition
Assertion Report: FAIL
3-provider-20-users3 providers, 20 users, 60 requests, concurrency=10, streaming=true
Latency Decomposition
Assertion Report: FAIL
1-provider-scaling1 providers, 5 users, 30 requests, concurrency=10, streaming=true
Latency Decomposition
Assertion Report: FAIL
3-provider-scaling3 providers, 5 users, 30 requests, concurrency=10, streaming=true
Latency Decomposition
Assertion Report: FAIL
5-provider-scaling5 providers, 5 users, 30 requests, concurrency=10, streaming=true
Latency Decomposition
Assertion Report: FAIL
3-provider-heavy-100conc-10kb3 providers, 20 users, 100 requests, concurrency=100, streaming=true
Latency Decomposition
Assertion Report: FAIL
|
Replaces the TOFU (Trust On First Use) disk-file approach for cluster
handshake SE key pinning with coordinator-verified distribution via the
macOS Keychain.
The previous approach stored the peer's SE public key in
/etc/darkbloom/cluster-peer-{IP}.sekey on first connection. Root on
either Mac could overwrite the file to accept a rogue peer. The new
approach:
1. `darkbloom cluster setup --peer-serial <serial> --peer-ip <ip>`
fetches the peer's SE public key from the coordinator, which
verified it through Apple MDM attestation (MDA cert chain signed
by Apple Enterprise Attestation Root CA).
2. The fetched key is stored in the macOS Keychain with
kSecAttrAccessibleWhenUnlockedThisDeviceOnly — modifying it
requires SE access, not just root.
3. ClusterHandshake.verifyAgainstKeychain() compares the SE key
received during every handshake against the Keychain entry.
An unknown key or a mismatch throws ClusterError.peerSEKeyMismatch
and the connection is rejected immediately.
Coordinator (Go):
- GET /v1/cluster/peer-key?serial=<serial> (requirePrivyAuth)
Queries the persistent store — works even when the peer is offline.
Returns { serial, se_public_key, trust_level, mda_verified }.
Swift — new files:
- ClusterPeerKeychain: store/load/delete Keychain entries keyed by
peer Thunderbolt IP (service "io.darkbloom.cluster.peer-sekey")
- ClusterCoordinatorClient: HTTP client for /v1/cluster/peer-key;
converts wss:// coordinator WebSocket URL to https:// automatically
Swift — updated files:
- ClusterHandshake: pinOrVerify (TOFU) → verifyAgainstKeychain
Removed pinnedKeyPath and all disk-file I/O
- ClusterControlMessage: add ClusterError.peerSEKeyNotPinned
- ClusterCommand (ClusterSetup): add --peer-serial and --peer-ip flags;
fetch + pin peer SE key as the first step of `cluster setup`
Usage (run on both Macs before `cluster run`):
Mac A: darkbloom cluster setup --rank 0 --peer-serial <B serial> \
--peer-ip 169.254.58.74 --coordinator :9999
Mac B: darkbloom cluster setup --rank 1 --peer-serial <A serial> \
--peer-ip 169.254.106.209 --coordinator 169.254.106.209:9999
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Providers started with `darkbloom serve --rdma-enabled` register themselves with the coordinator as RDMA-capable. Thunderbolt-connected peers automatically discover each other without manual `cluster setup`: Coordinator: - Add `rdma_enabled` field to RegisterMessage (protocol) - Add `RDMAEnabled` field to registry.Provider; set from registration - Add ListRDMAEnabledPeers() to registry — returns connected providers that have RDMAEnabled=true and completed SE attestation - Add GET /v1/cluster/rdma-peers endpoint (Privy-authenticated) - Add GET /v1/cluster/rdma-peers route to server mux Swift provider: - Add rdmaEnabled to ProviderMessage.Register, CoordinatorClientConfig, ProviderLoopConfig; wire through codec → registration message - Add ClusterCoordinatorClient.fetchRDMAPeers() for the new endpoint - New ClusterDiscovery actor: NWPathMonitor watches wiredEthernet, reads own Thunderbolt IP via getifaddrs, finds peer IP via arp -a, fetches RDMA peer list from coordinator, pins SE key in Keychain, elects rank (lower IPv4 = rank 0), starts ClusterSession or ClusterPeer - Add --rdma-enabled flag to `darkbloom serve` (Start command) - Hot-plug: NWPathMonitor re-fires on cable connect/disconnect Providers without --rdma-enabled are not in the RDMA list, preventing unauthorized cluster connections (DoS mitigation as designed). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Closes a class of holes in the RDMA auto-discovery surface where the
rdma_enabled flag was operator-controlled and the peer-list endpoint
was readable by any authenticated user.
Provider (Swift):
- New RDMACapability: chip family M5+ AND /usr/bin/rdma_ctl=enabled.
Operator-passed --rdma-enabled is downgraded to false locally when
the hardware can't honor it, with a stderr warning.
- AttestationBlob now signs rdmaEnabled (alphabetical, between
rdmaDisabled and secureBootEnabled). AttestationBuilder takes the
effective value as a parameter.
- StartCommand wires effectiveRDMA through ProviderLoopConfig so the
ClusterDiscovery actor and the SE-signed blob carry the same value.
Coordinator (Go):
- AttestationBlob + VerificationResult gain RDMAEnabled (always-emit,
matches Swift's non-optional Bool encoding). marshalSortedJSON
updated for canonical-JSON parity.
- After attestation.Verify succeeds, msg.RDMAEnabled vs
result.RDMAEnabled is reconciled — registry pinned to the attested
value via SetRDMAEnabled, mismatches logged.
- GET /v1/cluster/rdma-peers now requires the caller to own a
currently-connected RDMA-enabled provider (was: any Privy JWT).
Non-providers get 403; caller's own providers are excluded from
the response.
- ListRDMAEnabledPeersForCaller filter also requires Attested=true
AND Status not in {Untrusted, Offline}. Closes a window where
attestation.Verify populates SerialNumber/PublicKey *before*
signature verification — without the Attested gate, tampered
providers with valid identifiers could appear in the peer list.
Tests:
- Swift: chip-family gate, rdma_ctl-missing fail-closed, signed
blob round-trip, strict-decode requirement.
- Go: VerificationResult.RDMAEnabled round-trip, SE-signed bit
tamper detection, SetRDMAEnabled overwrite + no-op, 9 peer-list
symmetric-gate scenarios (non-provider/empty/self-tampered/
non-RDMA/unattested/tampered/untrusted/offline cases).
Threat model:
- TB-010 (provider_to_provider_cluster) added, T-041..T-045 cover
ARP positional matching, ARP retry cancellability, silent re-pin,
JWT in memory, peer-list disclosure. T-045 mitigations note both
the symmetric-access gate and the Attested+Status filter as
implemented.
Plus a small gofmt fix in cluster_handlers.go (pre-existing trailing
newline) that was blocking CI.
Picks up Layr-Labs/mlx-swift#2 which exposes the Cmlx library product and enables the jaccl distributed backend that provider-swift's ClusterSession / EncryptedPipelineInference depend on. Without this bump, CI's fresh `swift build -c debug` fails with: product 'Cmlx' required by package 'provider-swift' target 'ProviderCore' not found in package 'mlx-swift'. Tracking issue for the upstream deviation: #193.
Picks up Layr-Labs/mlx-swift-lm#24 which adds the callPartial methods on LlamaModelInner and LlamaModel that EncryptedPipelineInference and PipelineInference need for two-rank pipeline-parallel inference. Without this bump, CI's `swift build` fails with: error: value of type 'LlamaModel' has no member 'callPartial' Related: #193 (upstream mlx-swift distributed deviation tracker).
anupsv
force-pushed
the
rdma-connection-test
branch
from
7c75cc7 to
16ea9f5
Compare
Force-pushes on Layr-Labs/mlx-swift#2 and Layr-Labs/mlx-swift-lm#24 landed new SHAs (fa6a4e8, c2fbbdc) — bump the submodule pointers to match.
Benchmark ResultsRunner: 1-provider-streaming1 providers, 1 users, 30 requests, concurrency=5, streaming=true
Latency Decomposition
Assertion Report: FAIL
1-provider-non-streaming1 providers, 1 users, 20 requests, concurrency=5, streaming=false
Latency Decomposition
Assertion Report: FAIL
7-provider-multi-model7 providers, 5 users, 50 requests, concurrency=10, streaming=true
Latency Decomposition
Assertion Report: FAIL
3-provider-high-concurrency3 providers, 10 users, 60 requests, concurrency=20, streaming=true
Latency Decomposition
Assertion Report: FAIL
1-provider-queue-saturation1 providers, 10 users, 40 requests, concurrency=15, streaming=true
Latency Decomposition
Assertion Report: FAIL
3-provider-20-users3 providers, 20 users, 60 requests, concurrency=10, streaming=true
Latency Decomposition
Assertion Report: FAIL
1-provider-scaling1 providers, 5 users, 30 requests, concurrency=10, streaming=true
Latency Decomposition
Assertion Report: FAIL
3-provider-scaling3 providers, 5 users, 30 requests, concurrency=10, streaming=true
Latency Decomposition
Assertion Report: FAIL
5-provider-scaling5 providers, 5 users, 30 requests, concurrency=10, streaming=true
Latency Decomposition
Assertion Report: FAIL
3-provider-heavy-100conc-10kb3 providers, 20 users, 100 requests, concurrency=100, streaming=true
Latency Decomposition
Assertion Report: FAIL
|
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds the complete RDMA pipeline inference stack with SE-authenticated encrypted tensor transfer over Thunderbolt, plus fully automated peer discovery.
rdma-connection-test/) — standalone Swift package that verifies jaccl/RDMA works between two Apple Silicon Macs over ThunderboltProviderCore/P2P/darkbloom serve --rdma-enabledautomatically finds the Thunderbolt peer without manual serial pairingWhy not plain RDMA for tensor transfer?
RDMA (
mlx_distributed_send/recv) is a black-box C API — we can't wrap it with encryption. An attacker with root could inject alibrdma.dylibshim and intercept activation tensors in transit. Instead:allreduce) where confidentiality is less criticalNew files
ProviderCore/P2P/ThunderboltLink.swiftwiredEthernet(bridge0), port 7777, 8-byte LE length prefixMLXDistributed.swiftClusterControlMessage.swiftClusterFrameencoding,ClusterHealth,ClusterErrorClusterHandshake.swiftClusterPeerKeychain.swiftkSecAttrAccessibleWhenUnlockedThisDeviceOnly)ClusterCoordinatorClient.swiftfetchPeerSEKey+fetchRDMAPeersTensorCrypto.swiftMLXArrayClusterSession.swiftClusterPeerEncryptedPipelineInference.swiftEncryptedPipelineEngine(rank 0) +EncryptedPipelineServer(rank 1)ClusterDiscovery.swiftRDMACapability.swiftrdma_ctl statuscapability gate (downgrades--rdma-enabledwhen hardware can't honor it)Security/AttestationBuilder.swiftrdmaEnabledfield to the signedAttestationBlob; builder takes effective value as a parameterCoordinator
protocol/messages.gordma_enabledtoRegisterMessageattestation/attestation.goRDMAEnabledtoAttestationBlob+VerificationResult; include inmarshalSortedJSONregistry/registry.goRDMAEnabledtoProvider; addSetRDMAEnabled()(coordinator pins to attested value on mismatch);ListRDMAEnabledPeersForCaller()enforces symmetric-access + Attested + Status filtersapi/cluster_handlers.goGET /v1/cluster/peer-key+GET /v1/cluster/rdma-peers(rdma-peers returns 403 unless caller owns an Attested RDMA provider)api/provider.goattestation.Verify, reconcilemsg.RDMAEnabledvsresult.RDMAEnabled— attested value wins, mismatch loggedapi/server.goProtocol flows
Handshake (once per connection)
SE key pinning is coordinator-verified: keys come from MDM attestation records,
stored in macOS Keychain (
kSecAttrAccessibleWhenUnlockedThisDeviceOnly).Inference (per token)
RDMA auto-discovery flow
End-to-end, including the capability gate (binary refuses to honor
--rdma-enabledon non-M5 hardware), the SE-signedrdmaEnabledfield in the attestation blob, and the coordinator's symmetric-access gate on/v1/cluster/rdma-peers.sequenceDiagram participant MacA as Mac A (M5, --rdma-enabled) participant SE_A as SE (Mac A) participant Coord as Coordinator participant MacB as Mac B (M5, --rdma-enabled) rect rgb(230, 240, 255) Note over MacA,MacB: Registration — capability gate runs once, locally Note over MacA: RDMACapability.isAvailable()<br/>chip = M5+? rdma_ctl = enabled?<br/>(else effectiveRDMA = false) MacA->>SE_A: sign AttestationBlob { ..., rdmaEnabled: true } SE_A-->>MacA: signature MacA->>Coord: register { rdma_enabled: true, attestation } Coord->>Coord: Verify SE signature<br/>Check msg.rdma_enabled == blob.rdmaEnabled<br/>(mismatch → registry pinned to attested value) Coord->>Coord: SetAttested(true) — provider now peer-list eligible MacB->>Coord: register { rdma_enabled: true, attestation } end rect rgb(235, 250, 235) Note over MacA: Thunderbolt cable plugged in Note over MacA: NWPathMonitor fires (wiredEthernet up) MacA->>MacA: getifaddrs → own IP (169.254.x.x) MacA->>Coord: GET /v1/cluster/rdma-peers (Bearer JWT) Note over Coord: Symmetric gate:<br/>caller account must own an<br/>Attested + Online RDMA provider<br/>(else 403)<br/>Filter: rdma && serial && seKey<br/>&& Attested && Status ∉ {Untrusted, Offline}<br/>Excludes caller's own providers Coord-->>MacA: [{ serial: MacB, se_public_key, trust_level }] MacA->>MacA: arp -a -i bridge100 → peer IP MacA->>MacA: Keychain.store(peerSEKey, peerIP) end rect rgb(255, 245, 230) Note over MacA,MacB: Rank election (deterministic, no coordination) alt own IP < peer IP (rank 0) MacA->>MacA: ClusterSession.start() → connect to MacB Note over MacA,MacB: ClusterHandshake + ECDH session key MacA-->>MacB: ping every 5 s else own IP > peer IP (rank 1) MacA->>MacA: ClusterPeer.serve() → listen for MacB end endCapability gate + tamper rejection
How the system handles operators who misuse
--rdma-enabledand OS-tampered providers that lie in the unsigned registration field.sequenceDiagram participant Op as Operator participant Bin as Provider binary participant SE as Secure Enclave participant Coord as Coordinator rect rgb(255, 240, 240) Note over Op,Bin: Path A — operator on non-M5 (M3/M4) passes --rdma-enabled Op->>Bin: darkbloom serve --rdma-enabled Bin->>Bin: RDMACapability.unavailableReason()<br/>"this Mac is not M5 or later" Bin-->>Op: stderr: "--rdma-enabled ignored: ..." Note over Bin: effectiveRDMA = false Bin->>SE: sign blob { rdmaEnabled: false } Bin->>Coord: register { rdma_enabled: false, attestation } Note over Coord: Provider excluded from peer list<br/>(rdma == false in registry) end rect rgb(255, 240, 240) Note over Bin,Coord: Path B — OS-tampered: flip msg.rdma_enabled but SE only signs effective value Bin->>SE: sign blob { rdmaEnabled: false }<br/>(capability gate ran honestly) SE-->>Bin: signature Note over Bin: Tampered shim flips unsigned field Bin->>Coord: register { rdma_enabled: true, attestation } Coord->>Coord: Verify SE signature ✓<br/>msg.rdma_enabled=true ≠ blob.rdmaEnabled=false Coord->>Coord: WARN log + SetRDMAEnabled(false)<br/>(attested value wins) Note over Coord: Registry RDMAEnabled = false<br/>provider not in peer list end rect rgb(255, 240, 240) Note over Bin,Coord: Path C — Bad-signature attestation (forged blob) Bin->>Coord: register { rdma_enabled: true, tampered attestation } Coord->>Coord: Verify SE signature ✗<br/>SetAttestationResult(Valid=false)<br/>MarkUntrusted (if policy) Note over Coord: AttestationResult.SerialNumber & PublicKey<br/>are populated (pre-sigcheck) but Attested=false<br/>and Status=Untrusted →<br/>peer-list filter excludes endFull handshake + inference flow
sequenceDiagram participant R0 as Rank 0 participant SE0 as SE (Rank 0) participant SE1 as SE (Rank 1) participant R1 as Rank 1 rect rgb(230, 255, 230) Note over R0,R1: Handshake R0->>SE0: sign SHA256(x25519_pub₀ ∥ nonce₀) SE0-->>R0: sig₀ R0->>R1: HandshakeHello {se_pub₀, x25519_pub₀, nonce₀, sig₀} Note right of R1: Keychain verify se_pub₀ R1->>SE1: verify sig₀ R1->>SE1: sign SHA256(nonce₀ ∥ nonce₁ ∥ x25519_pub₁) SE1-->>R1: sig₁ R1->>R0: HandshakeAck {se_pub₁, x25519_pub₁, nonce₁, sig₁} Note left of R0: Keychain verify se_pub₁, verify sig₁ R0->>R0: sessionKey = HKDF(X25519(sk₀,pk₁), nonce₀∥nonce₁) R1->>R1: sessionKey = HKDF(X25519(sk₁,pk₀), nonce₀∥nonce₁) end rect rgb(255, 250, 220) Note over R0,R1: Ping loop (every 5 s) loop Health monitor R0->>R1: ping R1-->>R0: pong {modelLoaded, memoryPressure} end end rect rgb(255, 235, 230) Note over R0,R1: Inference loop Each decode step R0->>R0: forward layers [0, splitLayer) R0->>R1: inferenceStep {AES-GCM(seqLen ∥ activation)} R1->>R1: forward layers [splitLayer, N) + argmax R1-->>R0: inferenceToken {AES-GCM(tokenID)} end R0->>R1: inferenceStep {AES-GCM(seqLen=-1)} endReconnect flow
Usage
Auto-discovery (new)
Manual setup (still supported)
DoS protection / tamper resistance
Defense in depth — every layer below has to be defeated to enroll a malicious peer:
--rdma-enabledsendrdma_enabled: false. The coordinator never includes them inGET /v1/cluster/rdma-peers. A Thunderbolt connection from a non-RDMA peer is rejected during handshake (SE key not pinned →ClusterError.peerSEKeyNotPinned).RDMACapability.isAvailable()checks chip family is M5+ AND/usr/bin/rdma_ctl status == "enabled". If either fails, the binary downgradeseffectiveRDMAtofalseregardless of what the operator typed, prints a warning, and signsrdmaEnabled: falseinto the attestation blob.rdmaEnabled. The flag lives inside the SE-signedAttestationBlob(alphabetical position betweenrdmaDisabledandsecureBootEnabled). A compromised OS that tries to flip the unsignedRegisterMessage.rdma_enabledpost-hoc creates a mismatch; the coordinator detects it (provider.go: SetRDMAEnabled(attested value)) and pins the registry to the attested value, logging the discrepancy.GET /v1/cluster/rdma-peersrequires the calling account to own at least one currently-connected RDMA-enabled provider. Consumers and accounts without a matching provider receive403. The caller's own providers are excluded from the response.Attested != trueORStatus ∈ {Untrusted, Offline}. This closes a subtle window whereattestation.VerifypopulatesSerialNumber/PublicKeyon the result before checking the signature — a tampered provider with valid-looking identifiers would otherwise satisfy the presence-of-identifiers check.peerSEKeyNotPinnedor signature mismatch).