Skip to content

Latest commit

 

History

History
188 lines (139 loc) · 8.11 KB

README.md

File metadata and controls

188 lines (139 loc) · 8.11 KB

TEEBridge: Cross-KMS Secret Sharing for dstack CVMs

TEEBridge enables CVMs running on different dstack KMS ecosystems to verify each other's attestations and share secrets. A smart contract on Base accepts DstackProofs from any whitelisted KMS root, creating a unified membership registry with ECIES-encrypted onboarding.

Architecture

┌─────────────────┐         ┌──────────────────┐
│  CVM-A          │         │  CVM-B           │
│  Phala Cloud    │         │  hosted.dstack   │
│  Base KMS       │         │  Sepolia KMS     │
│                 │         │                  │
│  Flask :8080    │         │  Flask :8080     │
│  GET /proof     │         │  GET /proof      │
│  GET /info      │         │  GET /onboarding │
└────────┬────────┘         └────────┬─────────┘
         │                           │
         │  register.py              │  register.py
         │  (fetches /proof,         │  (fetches /proof,
         │   submits tx)             │   submits tx)
         │                           │
         └──────────┐   ┌────────────┘
                    ▼   ▼
            ┌───────────────────┐
            │  TEEBridge.sol    │
            │  Base Mainnet     │
            │                   │
            │  allowedKmsRoots  │
            │  allowedCode      │
            │  members          │
            │  onboarding msgs  │
            └───────────────────┘
                    ▲
                    │
              onboard.py
              (ECIES encrypt,
               submit tx)

Key Insight: HTTP Proof Pattern

The CVM never needs private keys or RPC access for registration. Instead:

  1. CVM runs an HTTP server exposing /proof (DstackProof JSON derived from KMS)
  2. External script (register.py) fetches the proof and submits the registerDstack tx
  3. External script (onboard.py) encrypts secrets to a member's pubkey and posts them on-chain
  4. CVM polls the contract for onboarding messages and decrypts with its KMS-derived key

This eliminates the need for encrypted env delivery (which is broken on some dstack deployments).

Files

File Purpose
bridge_agent.py CVM agent: Flask server with /proof, /info, /onboarding endpoints + background onboarding poller
register.py External: fetch proof from CVM (or paste from serial logs), register on TEEBridge. Auto-adds KMS root and code to allowlist.
onboard.py External: ECIES-encrypt a secret to a member's pubkey, post onboard tx
contracts/TEEBridge.sol Solidity contract: multi-KMS membership registry with onboarding
docker-compose-bridge.yaml Compose for Phala Cloud deployment
docker-compose-bridge-hosted.yaml Compose for hosted.dstack.info (serial logging, onboarding polling)
hosted.sh Wrapper for vmm-cli.py with hosted.dstack.info credentials

Usage

Deploy CVM

Phala Cloud:

phala deploy --name my-bridge \
  --compose docker-compose-bridge.yaml \
  --image dstack-0.5.4 --node-id 26 \
  --kms base --private-key $KEY --rpc-url https://mainnet.base.org

hosted.dstack.info:

# Generate compose manifest
./hosted.sh compose --name my-bridge \
  --docker-compose docker-compose-bridge-hosted.yaml \
  --kms --public-logs --output /tmp/compose.json

# Whitelist compose hash on Sepolia DstackApp
cast send $DSTACK_APP "addComposeHash(bytes32)" $HASH \
  --rpc-url https://sepolia.drpc.org --private-key $KEY

# Deploy
./hosted.sh deploy --name my-bridge --image dstack-0.5.7 \
  --compose /tmp/compose.json \
  --kms-url https://kms.hosted.dstack.info:9100 \
  --app-id $DSTACK_APP \
  --port tcp:0.0.0.0:18080:8080

Register CVM

# From HTTP endpoint (if gateway works):
python3 register.py --cvm-url https://APP_ID-8080.gateway.domain \
  --bridge 0x254057d9d92FC7F75E3D49F0c6B0be9eE2A334D5 \
  --private-key $KEY

# From serial logs (if gateway is broken):
# 1. Find PROOF_JSON=... line in serial/container logs
# 2. Pass it directly:
python3 register.py --proof-json '{"code_id":"0x...","dstack_proof":{...}}' \
  --bridge 0x254057d9d92FC7F75E3D49F0c6B0be9eE2A334D5 \
  --private-key $KEY

Send Secret

python3 onboard.py \
  --from-member 0x5682abc2... \
  --to-member 0xc8af48f2... \
  --secret "my secret data" \
  --bridge 0x254057d9d92FC7F75E3D49F0c6B0be9eE2A334D5 \
  --private-key $KEY

Receive Secret

If BRIDGE_CONTRACT env is set, CVM-B polls every 60s and prints decrypted messages:

ONBOARDING from=5682abc2... payload=my secret data

Or hit the HTTP endpoint:

GET /onboarding?bridge=0x254057d9d92FC7F75E3D49F0c6B0be9eE2A334D5

Deployed Instances

CVM-A CVM-B
Platform Phala Cloud hosted.dstack.info
KMS Base (0x52d3CF51...) Sepolia (0x9d456Bb7...)
Image dstack-0.5.4 dstack-0.5.7
TEEBridge 0x254057d9d92FC7F75E3D49F0c6B0be9eE2A334D5 (Base) same

dstack Pitfalls & Workarounds

Encrypted Env Delivery is Fragile

Problem: vmm-cli.py deploy --env-file requires the CLI to fetch an encrypt pubkey from KMS. On hosted.dstack.info, KMS port 9100 is firewalled from the internet, and the VMM's own kms_url points to localhost (wrong — KMS is in a separate CVM). Both paths to get the pubkey fail.

Workaround: The HTTP Proof Pattern eliminates the need for env vars entirely. The CVM derives all keys from the KMS socket at runtime. Registration and onboarding happen externally.

Gateway Broken on Self-Hosted dstack

Problem: VMM config has gateway_urls = ["http://127.0.0.1:8082"], but the gateway runs in a separate CVM. CVMs use QEMU user-mode networking where 127.0.0.1 is the CVM's own localhost. Gateway registration fails silently.

Workaround: Deploy without --gateway. Use --port tcp:0.0.0.0:HOST_PORT:CVM_PORT for direct port mapping (only reachable from the host, firewalled from internet). Print critical data to serial logs via tee /dev/ttyS0.

Container Logs Not Visible Without Gateway

Problem: vmm-cli.py logs only shows the serial console. Docker container stdout is invisible. The guest agent's log endpoint requires gateway access.

Workaround: Use CMD ["sh", "-c", "python -u agent.py 2>&1 | tee /dev/ttyS0"] with privileged: true to pipe container output to the serial console. /dev/ttyS0 is the serial port — not /dev/console, not /dev/kmsg.

Compose Hash Churn on Sepolia KMS

Problem: hosted.dstack.info uses Sepolia DstackApp contracts for KMS auth. Every compose change produces a new hash that must be whitelisted via addComposeHash(bytes32) — a Sepolia tx per iteration.

Impact: 10+ Sepolia txs during a debugging sprint, each taking 15-30s.

Mitigation: Get the compose right before deploying. The HTTP Proof Pattern reduces iteration cycles since the CVM code is stable and external scripts handle the changing logic.

Docker COPY Doesn't Work in CVMs

Problem: Neither Phala CLI nor vmm-cli upload Docker build context. COPY . . fails.

Workaround: Inline all code via RUN cat > file.py <<'PYEOF' ... PYEOF in dockerfile_inline.

Image/Node Version Mismatch (Phala Cloud)

Problem: Using the wrong image version for a node causes silent failures. dstack-0.5.4 nodes need --image dstack-0.5.4 --node-id 26.

pip Package Name: eciespy

The PyPI package is eciespy (not ecies), but the Python import is from ecies import ....

Phala Cloud Gateway Unreliable

Problem: Gateway access (the APP_ID-PORT.gateway.domain URL pattern) sometimes returns connection errors (TLS EOF, exit code 56) even when the container is running and healthy.

Workaround: Use phala ssh (requires --dev-os) or fall back to serial log output with PROOF_JSON= pattern. The --proof-json flag on register.py accepts pasted proof from logs.