From 9ce70362fd293f63342a2cb6740bb6ff5e6a3492 Mon Sep 17 00:00:00 2001 From: Dylan Kilkenny Date: Tue, 31 Mar 2026 17:10:18 +0100 Subject: [PATCH 1/2] Add Symbiotic Templates documentation Add per-provider docs (LayerZero DVN, Chainlink CCV) and shared docs (architecture, setup, CLI, deployment, security, troubleshooting) for the Symbiotic cross-chain verification templates. Includes navigation entry under Open Source Tools in ethereum-evm sidebar. Signed-off-by: Dylan Kilkenny --- content/symbiotic/architecture.mdx | 144 ++++++++++++++ content/symbiotic/chainlink-ccv.mdx | 131 +++++++++++++ content/symbiotic/cli.mdx | 268 ++++++++++++++++++++++++++ content/symbiotic/deployment.mdx | 168 ++++++++++++++++ content/symbiotic/index.mdx | 26 +++ content/symbiotic/layerzero.mdx | 128 ++++++++++++ content/symbiotic/security.mdx | 113 +++++++++++ content/symbiotic/setup.mdx | 183 ++++++++++++++++++ content/symbiotic/troubleshooting.mdx | 187 ++++++++++++++++++ src/navigation/ethereum-evm.json | 51 +++++ 10 files changed, 1399 insertions(+) create mode 100644 content/symbiotic/architecture.mdx create mode 100644 content/symbiotic/chainlink-ccv.mdx create mode 100644 content/symbiotic/cli.mdx create mode 100644 content/symbiotic/deployment.mdx create mode 100644 content/symbiotic/index.mdx create mode 100644 content/symbiotic/layerzero.mdx create mode 100644 content/symbiotic/security.mdx create mode 100644 content/symbiotic/setup.mdx create mode 100644 content/symbiotic/troubleshooting.mdx diff --git a/content/symbiotic/architecture.mdx b/content/symbiotic/architecture.mdx new file mode 100644 index 00000000..fa5993ec --- /dev/null +++ b/content/symbiotic/architecture.mdx @@ -0,0 +1,144 @@ +--- +title: Architecture +--- + +System overview for the Symbiotic multi-provider template. + +## Core Model + +1. One active provider per running stack (`config/environments/.json`). +2. Shared off-chain runtime: + - OZ Monitor for ingress + - 3 operator processes + - 3 Symbiotic relay sidecars for BLS signatures + - OZ Relayer for destination tx submission + - Redis queue +3. Provider-specific on-chain contracts and calldata format. + +## Provider Matrix + +| Provider | Source ingress event | Destination submit call | Local | Testnet | Mainnet | +| --- | --- | --- | --- | --- | --- | +| [`layerzero`](/symbiotic/layerzero) | `JobAssigned` | `SymbioticLayerZeroDVN.submitProof(...)` | Supported | Supported | Not yet | +| [`chainlink_ccv`](/symbiotic/chainlink-ccv) | `CCIPMessageSent` | `OffRamp.execute(...)` | Supported | Not yet | Not yet | + +See per-provider pages for detailed message flows and code pointers. + +## Shared Off-Chain Runtime + +```mermaid +flowchart LR + subgraph source["Source Chain"] + Event["Provider Ingress Event"] + end + + Event --> Monitor["OZ Monitor"] + Monitor --> Operators["Operators (x3)"] + Operators <--> Relays["Symbiotic Relays\n(BLS sidecars)"] + Operators --> Relayer["OZ Relayer"] + + subgraph dest["Destination Chain"] + Submit["Provider Destination Call"] + end + + Relayer --> Submit +``` + +All providers share the same off-chain pipeline. The provider abstraction determines: +- Which event the monitor watches for +- How operators encode the payload +- What calldata the relayer submits + +## Merkle Tree Batching + +Messages are batched into Merkle trees for gas efficiency: + +1. Multiple messages are collected into a batch +2. Each message becomes a leaf in the Merkle tree +3. The Merkle root is signed by operators +4. Proofs allow verifying individual messages against the signed root + +This means: +- One signature covers many messages +- On-chain verification cost is amortized +- Individual messages can be verified independently + +## Symbiotic Integration + +Symbiotic provides the shared security layer: + +- **Operator Registration**: Operators stake and register their BLS public keys +- **Settlement Contract**: Verifies BLS signatures and checks quorum +- **Slashing**: Misbehaving operators can be penalized (production) + +The Settlement contract: +1. Maintains the list of registered operators and their public keys +2. Defines the quorum threshold +3. Verifies aggregated signatures +4. Reports verification results to the provider contract + +## BLS Signing Pipeline + +1. Operators sign provider-defined payloads through Symbiotic relay sidecars. +2. Aggregation/quorum logic comes from settlement-backed Symbiotic attestation rules. +3. Provider-specific contracts decode and enforce those attestations on the destination execution path. + +## Message Status Lifecycle + +```mermaid +flowchart LR + Pending --> Processing --> Signed --> Submitted --> Confirmed +``` + +| Status | Description | +|--------|-------------| +| Pending | Received via webhook, awaiting batching | +| Processing | Batched into Merkle tree, awaiting BLS signatures | +| Signed | Quorum signatures collected, ready for submission | +| Submitted | Sent to OZ Relayer | +| Confirmed | On-chain TX confirmed | + +## Operator Internals + +| Module | Location | Purpose | +|--------|----------|---------| +| API Server | `operator/src/api/` | Axum HTTP server, webhook endpoint, debug routes | +| Provider | `operator/src/provider/` | Provider trait, event decoding, message storage | +| SignerJob | `operator/src/signer/` | Batches messages into Merkle trees, requests BLS signatures | +| RelaySubmitterJob | `operator/src/relay_submitter/` | Submits signed proofs via OZ Relayer | +| Storage | `operator/src/storage/` | redb key-value store (messages, Merkle trees, submissions) | +| Crypto | `operator/src/crypto/` | Merkle tree construction, leaf hashing, signing message encoding | + +## Adding a New Provider + +1. Create `operator/src/provider/yourprovider.rs` implementing the `Provider` trait: + +```rust +#[async_trait] +pub trait Provider: Send + Sync + 'static { + fn name(&self) -> &'static str; + async fn handle_webhook_event(&self, event: &WebhookEvent) -> Result<(), ProviderError>; + + // Optional overrides: + fn register_api_routes(&self, router: Router) -> Router { router } + async fn acceptance_hook(&self, _msg: &MessageData) -> Result<(), ProviderError> { Ok(()) } +} +``` + +2. Add configuration to `operator/src/config/mod.rs`. +3. Register in `create_provider()` in `operator/src/provider/mod.rs`. +4. Create provider-specific monitor template in `config/templates/oz-monitor/monitors/`. +5. Create `docs/.mdx` following the structure of existing provider docs (e.g., [LayerZero](/symbiotic/layerzero)). +6. Update this file's provider matrix, the [docs index](/symbiotic), and the project README. + +## Environment Comparison + +| Aspect | Local (Anvil) | Testnet | Production | +|--------|---------------|---------|------------| +| Source chain | Anvil 31337 | Base Sepolia 84532 | Mainnet | +| Dest/Settlement chain | Anvil 31338 | Sepolia 11155111 | Mainnet | +| Operators | 3 (local containers) | 3 (local containers) | 1+ (distributed) | +| Symbiotic Core | Deployed locally | Pre-deployed on Sepolia | Pre-deployed | +| BLS Keys | Deterministic | Deterministic | Hardware security | +| Quorum | 2-of-3 | 2-of-3 | Configurable | +| OZ Services | Local | Local | Hosted by OZ | diff --git a/content/symbiotic/chainlink-ccv.mdx b/content/symbiotic/chainlink-ccv.mdx new file mode 100644 index 00000000..1ccf9aad --- /dev/null +++ b/content/symbiotic/chainlink-ccv.mdx @@ -0,0 +1,131 @@ +--- +title: Chainlink CCV +--- + +Symbiotic-secured Cross-Chain Verifier (CCV) for Chainlink CCIP-compatible message verification. + +## Overview + +The CCV provider implements a Symbiotic-backed verifier compatible with Chainlink's CCIP CCV interface. When a message is sent through an OnRamp-compatible contract, a `CCIPMessageSent` event is emitted. Operators build a CCV payload, collect BLS attestations via Symbiotic relay sidecars, and the relayer submits the proof to `OffRamp.execute(...)` on the destination chain. The OffRamp calls `SymbioticCCV.verifyMessage(...)` for each message, and success is confirmed when `MessageExecuted(messageId)` is emitted. + + + +This template supports the **Symbiotic CCV variant** only. The Chainlink auxiliary devenv stack (aggregator, indexer, verifier, executor) is not required. + + + +## Message Flow + +```mermaid +sequenceDiagram + participant App as Source App + participant OnRamp as OnRamp (Source) + participant Monitor as OZ Monitor + participant Operators as Operators (x3) + participant Relay as Symbiotic Relay (BLS) + participant Relayer as OZ Relayer + participant OffRamp as OffRamp (Dest) + participant CCV as SymbioticCCV (Dest) + + App->>OnRamp: sendMessage() + OnRamp-->>Monitor: CCIPMessageSent event + Monitor->>Operators: HMAC webhook + Operators->>Operators: build CCV payload + Merkle tree + Operators->>Relay: sign Merkle root (BLS) + Relay-->>Operators: aggregated signature + Operators->>Relayer: OffRamp.execute calldata + Relayer->>OffRamp: execute(message, ccvs, verifierResults) + OffRamp->>CCV: verifyMessage(message, messageId, verifierResults) + CCV-->>OffRamp: verified + OffRamp-->>OffRamp: emit MessageExecuted(messageId) +``` + +## Code Pointers + +### Contracts + +- `contracts/src/ccv/SymbioticCCV.sol` -- CCV verifier implementation (`verifyMessage`, `forwardToVerifier`) +- `contracts/src/ccv/interfaces/` -- CCV interface definitions (`ICrossChainVerifierV1`, etc.) +- `contracts/src/ccv/libraries/` -- CCV encoding and helper libraries +- `contracts/src/symbiotic/Settlement.sol` -- BLS signature verification and quorum enforcement +- `contracts/src/symbiotic/KeyRegistry.sol` -- Operator BLS public key registry +- `contracts/src/symbiotic/Driver.sol` -- Epoch and genesis management + +### Operator (Rust) + +- `operator/src/provider/chainlink_ccv.rs` -- Decodes `CCIPMessageSent` events, builds CCV payloads +- `operator/src/provider/mod.rs` -- `Provider` trait and registration + +### Config Templates + +- `config/templates/oz-monitor/monitors/ccip_message_sent.json` -- Monitor job for `CCIPMessageSent` events + +## Configuration + +Select CCV as the active provider: + +```json +// config/environments/.json +{ + "activeProvider": "chainlink_ccv" +} +``` + +Chain config is shared across providers — chain IDs from `chains.source.chainId` and `chains.destination.chainId` are used as CCIP chain selectors at runtime. + +Address resolution for CCV scripts: + +1. `CCV_*` environment variables (highest priority) +2. `deployments/.json` + +CCV settlement addresses in deployment state: + +- Destination: `destination.chainlinkCcv.settlement` in `deployments/.json` + +Available `CCV_*` override variables: + +| Variable | Description | +|----------|-------------| +| `CCV_SOURCE_ADDRESS` | SymbioticCCV on source chain | +| `CCV_DEST_ADDRESS` | SymbioticCCV on destination chain | +| `CCV_SOURCE_ONRAMP_ADDRESS` | Source OnRamp-compatible contract | +| `CCV_DEST_OFFRAMP_ADDRESS` | Destination OffRamp submit target | + +## Usage + +```bash +# Select chainlink_ccv provider in config/environments/local.json +# "activeProvider": "chainlink_ccv" + +# Start the stack +make start + +# Send a test message +make send MSG="hello" + +# Watch until MessageExecuted on destination +make watch + +# Or run both +make e2e +``` + +`make send` sends through the source mock `OnRamp.sendMessage(...)`, emitting `CCIPMessageSent`. + +`make watch` succeeds only when `MessageExecuted(messageId)` is found on the destination chain (not just relayer submission). + +See [CLI Reference](/symbiotic/cli) for full command options. + +## Deployment Status + +| Environment | Status | +|-------------|--------| +| Local | Supported (Symbiotic-only mock path) | +| Testnet | Not yet | +| Mainnet | Not yet | + +## Common Issues + +- **EpochTooStale revert (0xf5ab0d81)** -- Settlement epoch data is stale. Refresh genesis or tune epoch timing. See [Troubleshooting](/symbiotic/troubleshooting#epochtoostale-revert-0xf5ab0d81). +- **Watch does not reach success** -- CCV requires destination `MessageExecuted(messageId)`, not just relayer submission. See [Troubleshooting](/symbiotic/troubleshooting#watch-does-not-reach-success). +- **Submission fails at estimate-gas** -- Common causes: stale epoch, incorrect CCV addresses, settlement not initialized. See [Troubleshooting](/symbiotic/troubleshooting#submission-fails-at-estimate-gas). diff --git a/content/symbiotic/cli.mdx b/content/symbiotic/cli.mdx new file mode 100644 index 00000000..dc1607f2 --- /dev/null +++ b/content/symbiotic/cli.mdx @@ -0,0 +1,268 @@ +--- +title: CLI & API Reference +--- + +Command-line interface and HTTP API for the Symbiotic multi-provider template. + +## Make Commands + +### `make send` + +Send one test message. Provider-aware based on `activeProvider`. + +```bash +make send +make send MSG="test message" +make send ENV=testnet MSG="hello" +``` + +### `make watch` + +Watch a previously sent message until it lands on the destination chain. + +```bash +make watch +make watch ENV=testnet TIMEOUT=300 +make watch GUID=0x... +make watch TX=0x... +``` + +| Variable | Description | +|----------|-------------| +| `ENV` | Environment (default: local) | +| `TIMEOUT` | Max wait in seconds | +| `GUID` | Watch specific message by GUID | +| `TX` | Watch message by source TX hash | + +### `make e2e` + +Send a message, then watch it to completion. + +```bash +make e2e +make e2e MSG="custom message" +make e2e ENV=testnet MSG="hello" TIMEOUT=180 +``` + +Example output: + +``` +[18:53:21] Operators: waiting to batch +[18:53:28] Operators: collecting BLS signatures +[18:53:30] Operators: signed (quorum reached) +[18:53:32] Relayer: submitted +[18:53:34] Relayer: confirmed (tx: 0x4617...) +[18:53:34] Destination target: verified on-chain (tx: 0x4617...) + +Message verified on destination chain! +``` + +## Direct xtask Commands + +Use the Rust CLI directly: + +```bash +cargo xtask --env local msg send "hello" +cargo xtask --env local msg watch --timeout 120 +cargo xtask --env local msg e2e "hello" --timeout 120 +``` + +### `cargo xtask msg send` + +```bash +cargo xtask --env local msg send "hello" +cargo xtask --env local msg send "hello" --gas 250000 +cargo xtask --env local msg send "hello" --json +``` + +### `cargo xtask msg watch` + +```bash +cargo xtask --env local msg watch +cargo xtask --env local msg watch --id 0x... +cargo xtask --env local msg watch --tx 0x... +cargo xtask --env local msg watch --timeout 300 +``` + +### `cargo xtask msg e2e` + +```bash +cargo xtask --env local msg e2e "hello" +cargo xtask --env local msg e2e "hello" --timeout 300 +cargo xtask --env local msg e2e "hello" --json +``` + +## Message Cache + +After `send`, xtask saves message details to: + +``` +generated//msg-cache.json +``` + +`watch` uses this cache when no explicit `--id` or `--tx` is provided. + +## HTTP API + +Each operator exposes HTTP endpoints on ports 3001-3003. + +### Webhook Endpoints + +#### POST /webhook/events + +Receives provider ingress events from OZ Monitor. + +**Authentication:** HMAC-SHA256 via two headers: +- `X-Signature`: Hex-encoded HMAC-SHA256 of `body + timestamp` +- `X-Timestamp`: Unix timestamp in milliseconds + +The webhook secret must match between operator (`WEBHOOK_SECRET` env var) and OZ Monitor trigger config (`config.secret.value`). + +#### POST /api/v1/webhooks/oz-relayer + +Receives transaction status updates from OZ Relayer. + +**Authentication:** Base64-encoded HMAC-SHA256 of raw JSON body in `X-Signature` header, using `OZ_RELAYER_WEBHOOK_SECRET`. + +### Debug Endpoints + +#### GET /debug/v1/messages + +List messages with processing and submission status. + +```bash +curl -s http://localhost:3001/debug/v1/messages +curl "http://localhost:3001/debug/v1/messages?status=pending&limit=10" +``` + +| Parameter | Default | Description | +|-----------|---------|-------------| +| `status` | (all) | Filter: `pending`, `processing`, `signed` | +| `limit` | 50 | Max messages returned | +| `offset` | 0 | Pagination offset | + +#### GET /debug/v1/messages/:message_id + +Get a specific message by ID. + +#### GET /debug/v1/pending + +List Merkle roots awaiting BLS signatures. + +### Proof Endpoints + +#### POST /api/v1/layerzero/proof + +Retrieve Merkle proofs for processed messages. + +```bash +curl -X POST http://localhost:3001/api/v1/layerzero/proof \ + -H "Content-Type: application/json" \ + -d '{"message_ids": ["0xabc123..."]}' +``` + +Response fields: `root_hash`, `root_proof` (BLS signature), `index`, `leaf`, `siblings`, `original_list`. + +#### POST /api/v1/layerzero/verify + +Verify a Merkle proof is valid (testing only). + +### GET /healthz + +Returns `200 OK` if healthy. + +## Webhook Configuration + +OZ Monitor sends events via HMAC-SHA256 authenticated webhooks. + +### Trigger Template + +Webhook triggers are defined in `config/templates/oz-monitor/triggers/` and copied to `generated//oz-monitor/triggers/` at startup. + +Key settings: + +| Setting | Description | +|---------|-------------| +| `url.value` | Operator endpoint (use Docker service name in compose) | +| `secret.value` | Must match `WEBHOOK_SECRET` in operator `.env` | +| `payload_mode` | Must be `"raw"` | + +### Monitor Jobs + +Provider-specific monitor templates: +- `config/templates/oz-monitor/monitors/layerzero_job_assigned.json` +- `config/templates/oz-monitor/monitors/ccip_message_sent.json` + +### Webhook Payload Format + +```json +{ + "EVM": { + "logs": [{ "address": "0x...", "topics": ["..."], "data": "0x..." }], + "matched_on_args": { + "events": [{ + "signature": "JobAssigned(address,bytes,uint256,address)", + "args": [{ "name": "dvn", "kind": "address", "value": "0x..." }] + }] + }, + "monitor": { "name": "LayerZero JobAssigned" }, + "network_slug": "anvil-source", + "transaction": { "hash": "0x...", "blockNumber": 123 } + } +} +``` + +For CCV, the event signature is `CCIPMessageSent(...)` with different args. + +## Retry Configuration + +### Symbiotic Relay (Linear Backoff) + +For gRPC calls to BLS signing sidecars. + +``` +backoff = retry_backoff x (attempt + 1) +``` + +### OZ Relayer (Exponential Backoff with Jitter) + +For HTTP calls to the transaction relayer. + +``` +base = retry_backoff x 2^attempt +jitter = random(0, base x 0.25) +backoff = min(base + jitter, 60s) +``` + +### Retry Settings + +| Setting | Description | Default | +|---------|-------------|---------| +| `max_retries` | Maximum retry attempts (0 = no retries) | 3 | +| `retry_backoff` | Base backoff duration | 1s | +| `timeout` | Request timeout (OZ Relayer only) | 30s | + +### Retryable vs Non-Retryable Errors + +**Retried:** HTTP 429, HTTP 500-504, network errors (connection refused, timeout, DNS failure). + +**Not retried:** HTTP 4xx (except 429), domain errors (chain not configured, transaction not found). + +### Tuning Examples + +```json +// Low-latency (devnet/testnet) +{ "oz_relayer": { "max_retries": 5, "retry_backoff": "100ms" } } + +// Production +{ "oz_relayer": { "max_retries": 3, "retry_backoff": "1s" } } + +// High-volume +{ "oz_relayer": { "max_retries": 5, "retry_backoff": "2s" } } +``` + +| Config | Symbiotic Relay total | OZ Relayer total | +|--------|----------------------|------------------| +| 1s / 3 retries | ~6s | ~9s | +| 1s / 5 retries | ~15s | ~39s | +| 2s / 3 retries | ~12s | ~18s | diff --git a/content/symbiotic/deployment.mdx b/content/symbiotic/deployment.mdx new file mode 100644 index 00000000..f5958481 --- /dev/null +++ b/content/symbiotic/deployment.mdx @@ -0,0 +1,168 @@ +--- +title: Deployment +--- + +Deploying and operating the stack beyond local development. + + + +Testnet deployment currently supports the `layerzero` provider only. CCV is local-only for now. + + + +## Testnet (LayerZero) + +- Source: Base Sepolia (`84532`) +- Destination: Sepolia (`11155111`) +- Provider: `layerzero` + +### Runtime Model + +```bash +make validate ENV=testnet +make deploy ENV=testnet +make refresh-genesis ENV=testnet # when validation says genesis is stale +make run-operators ENV=testnet +make e2e ENV=testnet +``` + +| Command | What it does | +|---------|-------------| +| `validate` | Read-only checks: config, chain reachability, deployment state, operator state, relayer signer safety | +| `deploy` | Deploy managed contracts, update `deployments/testnet.json` and `generated/testnet/` | +| `refresh-genesis` | Refresh settlement genesis without redeploying contracts | +| `run-operators` | Start non-local operator-side services | + +### Environment Inputs + +Config lives in `config/environments/testnet.json`: +- Chain IDs and EIDs +- Default RPC URLs +- LayerZero predeploys +- Symbiotic Core predeploys +- Relay timing + +xtask resolves RPC URLs from the environment JSON first. `SOURCE_RPC_URL` / `DEST_RPC_URL` in `.env` are fallback overrides only. + +Required `.env` values: + +```bash +PRIVATE_KEY=0x +KEYSTORE_PASSPHRASE= +OPERATOR_1_PRIVATE_KEY=0x +OPERATOR_2_PRIVATE_KEY=0x +OPERATOR_3_PRIVATE_KEY=0x +``` + +Optional relayer bootstrap inputs: + +```bash +RELAYER_1_PRIVATE_KEY=0x +RELAYER_2_PRIVATE_KEY=0x +RELAYER_3_PRIVATE_KEY=0x +``` + +Relayer private keys are setup-time inputs only. The steady-state runtime source is the OZ relayer keystore files under `config/oz-relayer/keys/`. + +### Workflow + +#### 1. Generate keys + +```bash +make setup +``` + + + +For public testnets: do not use known local/dev keys. Ensure deployer, operators, and relayer signers all have testnet ETH. + + + +#### 2. Validate first + +```bash +make validate ENV=testnet +``` + +Catches: missing keys, underfunded accounts, stale genesis, relayer signer issues. + +#### 3. Deploy managed contracts + +```bash +make deploy ENV=testnet +``` + +Updates `deployments/testnet.json` and `generated/testnet/`. + +#### 4. Refresh genesis when needed + +```bash +make refresh-genesis ENV=testnet +``` + +Use this instead of redeploying when contracts are already in place and only the settlement header is stale. + +#### 5. Start operator services + +```bash +make run-operators ENV=testnet +``` + +Starts from `docker-compose.yml` (no `docker-compose.local.yml` overlay): +- Operators, OZ Monitor, OZ Relayer, Symbiotic relay sidecars + +#### 6. Send and verify + +```bash +make e2e ENV=testnet MSG="hello" +``` + +### How Testnet Differs From Local + +| Area | Local | Testnet | +|------|-------|---------| +| Entrypoint | `make start` | `make deploy` + `make run-operators` | +| Chains | Local Anvil | Base Sepolia + Sepolia | +| LayerZero endpoints | Local mocks | Predeployed | +| Symbiotic Core | Deployed fresh | Predeployed on destination | +| Genesis refresh | Folded into startup | Explicit `make refresh-genesis` when stale | +| Compose files | `docker-compose.yml` + `docker-compose.local.yml` | `docker-compose.yml` only | +| Operator registration | Auto-impersonate | Separate step (real chains) | + +### Testnet Architecture + +```text +Base Sepolia (84532) Sepolia (11155111) +-------------------- ------------------- +LZ V2 Endpoint (pre-deployed) LZ V2 Endpoint (pre-deployed) +DVN.assignJob() DVN.submitProof() -> Settlement +TestOApp.send() TestOApp.lzReceive() + Driver, KeyRegistry, VotingPowers + + OZ Monitor -> Operators -> Symbiotic Relays -> OZ Relayer + (local Docker containers) +``` + +### Testnet Troubleshooting + +**`validation failed: genesis stale`** -- Run `make refresh-genesis ENV=testnet`. + +**`insufficient funds for gas`** -- Fund the deployer address from `PRIVATE_KEY`. + +**Sidecars fail with RPC rate limits** -- Three sidecars syncing at once can overwhelm weak testnet RPC plans. Use higher-throughput RPCs or temporarily reduce sidecar count. + +**Keys look drained immediately** -- Do not use known local/dev keys on public testnets. Regenerate with `make setup`. + +**Fresh relay deploys on shared testnet** -- Most fragile path. Prefer reusing existing environment and `make refresh-genesis` for stale-genesis repair. + +## Mainnet + +Not yet supported. Key differences from testnet will include: +- Hardware security for BLS keys +- Distributed operators (not co-located containers) +- OZ-hosted services +- Configurable quorum thresholds + +## Address Management + +All deployment addresses are canonical in `deployments/.json`. This file is updated by `make deploy` and read by all runtime components. diff --git a/content/symbiotic/index.mdx b/content/symbiotic/index.mdx new file mode 100644 index 00000000..706845bc --- /dev/null +++ b/content/symbiotic/index.mdx @@ -0,0 +1,26 @@ +--- +title: Symbiotic Templates +--- + +## For Operators + +Running, configuring, and monitoring the stack. + +1. [Setup](/symbiotic/setup) -- Config structure, environment setup, running locally +2. [Deployment](/symbiotic/deployment) -- Testnet and mainnet deployment +3. Choose your provider: + - [LayerZero](/symbiotic/layerzero) -- DVN for LayerZero V2 + - [Chainlink CCV](/symbiotic/chainlink-ccv) -- Cross-Chain Verifier for CCIP +4. [CLI & API Reference](/symbiotic/cli) -- Commands, HTTP endpoints, webhook config +5. [Troubleshooting](/symbiotic/troubleshooting) -- Common issues and debugging + +## For Integrators + +Understanding the system and adding new providers. + +1. [Architecture](/symbiotic/architecture) -- Provider model, shared infra, Merkle batching, BLS signing +2. Choose your provider: + - [LayerZero](/symbiotic/layerzero) -- Message flow, contracts, code pointers + - [Chainlink CCV](/symbiotic/chainlink-ccv) -- Message flow, contracts, code pointers +3. [Architecture: Adding a New Provider](/symbiotic/architecture#adding-a-new-provider) -- Provider trait, registration, templates +4. [Security](/symbiotic/security) -- Trust model, access control, invariants diff --git a/content/symbiotic/layerzero.mdx b/content/symbiotic/layerzero.mdx new file mode 100644 index 00000000..91e4d569 --- /dev/null +++ b/content/symbiotic/layerzero.mdx @@ -0,0 +1,128 @@ +--- +title: LayerZero +--- + +Symbiotic-secured DVN (Decentralized Verifier Network) for LayerZero V2 cross-chain messaging. + +## Overview + +The LayerZero provider implements a DVN that uses Symbiotic shared security to verify cross-chain messages. When a message is sent through LayerZero's `SendUln302`, the DVN contract emits a `JobAssigned` event. Operators batch these jobs into Merkle trees, collect BLS signatures through Symbiotic relay sidecars, and submit the signed proof to the destination DVN contract. The destination DVN verifies the BLS quorum via the Settlement contract and forwards verification to LayerZero's `ReceiveUln302`. + +## Message Flow + +```mermaid +sequenceDiagram + participant App as User App + participant SendUln as SendUln302 (Source) + participant DVN_S as DVN.assignJob (Source) + participant Monitor as OZ Monitor + participant Operators as Operators (x3) + participant Relay as Symbiotic Relay (BLS) + participant Relayer as OZ Relayer + participant DVN_D as DVN.submitProof (Dest) + participant Settlement as Settlement (BLS verify) + participant RecvUln as ReceiveUln302 (Dest) + + App->>SendUln: send message + SendUln->>DVN_S: assignJob() + DVN_S-->>Monitor: JobAssigned event + Monitor->>Operators: HMAC webhook + Operators->>Operators: batch into Merkle tree + Operators->>Relay: sign Merkle root (BLS) + Relay-->>Operators: aggregated signature + Operators->>Relayer: submitProof calldata + Relayer->>DVN_D: submitProof(root, proof, signatures) + DVN_D->>Settlement: verify BLS quorum + Settlement-->>DVN_D: quorum valid + DVN_D->>RecvUln: verify() +``` + +## Code Pointers + +### Contracts + +- `contracts/src/SymbioticLayerZeroDVN.sol` -- DVN contract handling `assignJob` (source) and `submitProof` (destination) +- `contracts/src/symbiotic/Settlement.sol` -- BLS signature verification and quorum enforcement +- `contracts/src/symbiotic/KeyRegistry.sol` -- Operator BLS public key registry +- `contracts/src/symbiotic/VotingPowers.sol` -- Operator voting power tracking +- `contracts/src/symbiotic/Driver.sol` -- Epoch and genesis management +- `contracts/src/examples/TestOApp.sol` -- Test application for sending/receiving messages + +### Operator (Rust) + +- `operator/src/provider/layerzero.rs` -- Decodes `JobAssigned` events, stores messages +- `operator/src/provider/mod.rs` -- `Provider` trait and registration +- `operator/src/crypto/mod.rs` -- Merkle tree construction, DVN leaf hashing +- `operator/src/signer/mod.rs` -- Batches messages, requests BLS signatures +- `operator/src/relay_submitter/mod.rs` -- Submits signed proofs via OZ Relayer + +### Config Templates + +- `config/templates/oz-monitor/monitors/layerzero_job_assigned.json` -- Monitor job for `JobAssigned` events +- `config/templates/oz-monitor/triggers/webhook_layerzero.json` -- Webhook trigger template + +## Configuration + +Select LayerZero as the active provider: + +```json +// config/environments/.json +{ + "activeProvider": "layerzero" +} +``` + +Chain config is shared across providers and lives at the top level: + +| Field | Description | +|-------|-------------| +| `chains.source.chainId` | Source chain ID | +| `chains.destination.chainId` | Destination chain ID | +| `chains.source.eid` | LayerZero endpoint ID for source | +| `chains.destination.eid` | LayerZero endpoint ID for destination | + +LayerZero predeploys (testnet/mainnet) go in `chains..predeploys.layerzero`: + +```json +{ + "predeploys": { + "layerzero": { + "endpoint": "0x6EDCE65403992e310A62460808c4b910D972f10f", + "sendUln302": "0xC1868e054425D378095A003EcbA3823a5D0135C9" + } + } +} +``` + +`make deploy` and `make start` use these values to generate runtime configs (`destination_chains`, `chain_relayers`, `eid_to_chain_id`) under `generated//`. Validation fails if chain IDs/EIDs drift from the generated deployment state. + +## Usage + +```bash +# Select layerzero provider in config/environments/local.json +# "activeProvider": "layerzero" + +# Start the stack +make start + +# Send a test message +make send MSG="hello" + +# Watch until destination verification +make watch + +# Or run both in one shot +make e2e +``` + +`make send` sends through `TestOApp.send(...)` which calls `SendUln302`, triggering `DVN.assignJob()`. + +`make watch` succeeds when destination target verification is observed on-chain. + +See [CLI Reference](/symbiotic/cli) for full command options. + +## Common Issues + +- **Message stuck at "Processing"** -- BLS signatures not aggregating. Check sidecar health and operator key registration. See [Troubleshooting](/symbiotic/troubleshooting#bls-signatures-not-aggregating). +- **Quorum not reached** -- All 3 operators must be running and receiving the same events. See [Troubleshooting](/symbiotic/troubleshooting#quorum-not-reached). +- **submitProof reverts** -- Check that the OZ Relayer address is authorized as a submitter on the DVN contract, and that Settlement has correct operator keys. See [Troubleshooting](/symbiotic/troubleshooting#layerzero-issues). diff --git a/content/symbiotic/security.mdx b/content/symbiotic/security.mdx new file mode 100644 index 00000000..7df1803e --- /dev/null +++ b/content/symbiotic/security.mdx @@ -0,0 +1,113 @@ +--- +title: Security +--- + +Security architecture and trust assumptions for the Symbiotic multi-provider template. + +## Shared Trust Model + +| Entity | Trust Level | Notes | +|--------|-------------|-------| +| **Settlement** | Trusted | Symbiotic contract for BLS signature verification | +| **Authorized Submitters** | Semi-trusted | Whitelisted addresses that submit proofs; cannot forge signatures but can grief (spam invalid proofs) | +| **Owner** | Trusted | Admin with pause/unpause, submitter management | +| **External users** | Untrusted | Cannot call privileged functions directly | + +### Symbiotic Security Layer + +- Operators stake and register BLS public keys via `KeyRegistry` +- Settlement contract verifies BLS quorum before accepting proofs +- Slashing handled by Symbiotic core contracts (production) + +### Webhook Authentication + +Webhooks between OZ Monitor and operators use HMAC-SHA256: +1. Monitor computes `HMAC-SHA256(secret, body + timestamp)` +2. Signature sent in `X-Signature` header +3. Timestamp (ms since epoch) sent in `X-Timestamp` header +4. Operator rejects invalid/missing signatures or expired timestamps (HTTP 401) + +Secrets must be at least 32 characters. See [CLI Reference](/symbiotic/cli#webhook-configuration) for config details. + +## LayerZero DVN Security + +Trust assumptions specific to the `SymbioticLayerZeroDVN` contract. + +| Entity | Trust Level | Notes | +|--------|-------------|-------| +| **SendUln302** | Trusted | LayerZero's send library; only caller for `assignJob` | + +### Access Control + +#### Source Chain + +| Function | Caller | Purpose | +|----------|--------|---------| +| `assignJob` | SendUln302 only | Register verification job, emit event | +| `getFee` | Anyone | Query verification fee (view) | + +#### Destination Chain + +| Function | Caller | Purpose | +|----------|--------|---------| +| `submitProof` | Authorized submitters | Submit signed Merkle proof for verification | + +#### Admin + +| Function | Caller | Purpose | +|----------|--------|---------| +| `addSubmitter` / `removeSubmitter` | Owner | Manage submitter whitelist | +| `setBaseFee` | Owner | Update verification fee | +| `pause` / `unpause` | Owner | Emergency controls | +| `withdraw` | Owner | Recover ETH (force-sent or accidental) | +| `transferOwnership` | Owner | Transfer admin rights | + +### Invariants + +1. **Leaf monotonicity**: `verifiedLeaves[leaf]` transitions `false -> true` only, never back +2. **Root monotonicity**: `verifiedRoots[root]` transitions `false -> true` only, never back +3. **Signature requirement**: Uncached roots require valid BLS quorum from Settlement +4. **Packet header integrity**: Verified packets have exactly 81 bytes and correct `dstEid` +5. **No ETH custody**: Contract does not collect fees; `assignJob` rejects `msg.value > 0` + +### Deployment Modes + +| Mode | sendUln | receiveUln | settlement | Use case | +|------|---------|------------|------------|----------| +| Source only | Set | Zero | Zero | Emit `JobAssigned` events | +| Destination only | Zero | Set | Set | Verify proofs, call ReceiveUln | +| Bidirectional | Set | Set | Set | Both functions on same chain | + +### What the DVN Does NOT Do + +- **Fee custody**: Fees handled by LayerZero's fee accounting +- **Signature generation**: BLS signing happens off-chain via Symbiotic Relay +- **Slashing**: Handled by Symbiotic core contracts + +## Chainlink CCV Security + +Trust assumptions specific to the `SymbioticCCV` contract. + +### Access Control + +| Function | Caller | Purpose | +|----------|--------|---------| +| `forwardToVerifier` | OnRamp | Source-chain hook for CCV registration | +| `verifyMessage` | OffRamp | Destination verification hook | +| `getFee` | Anyone | Quote verification fee (view) | + +### Invariants + +CCV verification requires: +1. Valid BLS quorum signature from Settlement +2. Correct message ID derivation +3. Epoch freshness (reverts with `EpochTooStale` if settlement data is stale) + +## External Dependencies + +| Dependency | Version | Purpose | +|------------|---------|---------| +| `@openzeppelin/contracts` | 5.x | MerkleProof verification | +| `@symbioticfi/relay-contracts` | - | Settlement base contracts | +| LayerZero V2 | - | ILayerZeroDVN interface | +| Chainlink CCIP | - | CCV interfaces (ICrossChainVerifierV1) | diff --git a/content/symbiotic/setup.mdx b/content/symbiotic/setup.mdx new file mode 100644 index 00000000..bbbba4ae --- /dev/null +++ b/content/symbiotic/setup.mdx @@ -0,0 +1,183 @@ +--- +title: Setup +--- + +Getting the stack running locally. + +## Prerequisites + +- Docker and Docker Compose v2+ +- [Foundry](https://book.getfoundry.sh/getting-started/installation) (`forge`, `cast`, `anvil`) +- [Rust/Cargo](https://rustup.rs/) (for `make dev-operator`) +- `jq` + +## Config Structure + +``` +config/ +├── environments/ # Per-network config (local.json, testnet.json) +├── templates/ # Service config templates (oz-monitor, oz-relayer) +├── oz-monitor/ # Static monitor config +└── oz-relayer/ # Static relayer config and keystores + +deployments/ +└── .json # Canonical deployment addresses + +generated/ +└── / # Generated runtime config and message cache +``` + +**How it works:** +1. `make deploy` deploys contracts and updates `deployments/.json` +2. `make deploy` and `make start` generate provider-specific runtime config under `generated//` +3. Docker containers mount from `generated//` + +**To customize configs:** Edit templates in `config/templates/`, then rerun `make deploy` or `make start`. + +## Environment Setup + +Bootstrap local `.env` and operator keys: + +```bash +make setup +``` + +This generates `.env` from `.env.example` and creates BLS keys and relayer keystores. + + + +`make start` also auto-bootstraps `.env` and keystores if missing, so `make setup` is only needed for explicit regeneration. + + + +### Environment Variables + +| Variable | Description | +|----------|-------------| +| `PRIVATE_KEY` | Deployer key (default: Anvil account 0) | +| `LOG_LEVEL` | Logging level (debug, info, warn, error) | +| `WEBHOOK_SECRET` | HMAC secret for webhook auth (min 32 chars) | +| `OZ_RELAYER_WEBHOOK_SECRET` | Secret for OZ Relayer webhook auth (min 32 chars) | +| `OZ_RELAYER_API_KEY` | **Required.** Relayer API authentication | +| `SIDECAR_*_SECRET_KEYS` | BLS keys per operator (generated) | + +Generate secrets with: + +```bash +openssl rand -hex 32 +``` + + + +The operator will fail to start if `WEBHOOK_SECRET`, `OZ_RELAYER_WEBHOOK_SECRET`, or `OZ_RELAYER_API_KEY` are missing. Secrets must be at least 32 characters. + + + +## Provider Selection + +Provider is set in `config/environments/.json`: + +```json +{ + "activeProvider": "layerzero" +} +``` + +All `make` commands (`start`, `send`, `watch`, `e2e`) are provider-aware based on this field. + +See provider-specific config in [LayerZero](/symbiotic/layerzero#configuration) or [Chainlink CCV](/symbiotic/chainlink-ccv#configuration). + +## Running Locally + +```bash +# Start the full stack (auto-bootstrap + deploy + start services) +make start + +# Check service health +make status + +# Send a test message and watch it complete +make e2e +``` + +### Common Commands + +``` +make start Start the full local stack +make deploy Deploy contracts and generate service config +make stop Stop all containers (preserve state) +make clean Full reset (stop + remove volumes + markers) + +make restart-operators Rebuild and restart all 3 operators +make restart-monitor Restart oz-monitor (config reload) +make restart-relayer Restart oz-relayer +make restart-relays Restart symbiotic-relay-1/2/3 + +make dev-operator Run operator-1 locally (cargo run) +make rebuild-operators Docker rebuild + restart all operators +make shell Interactive shell with addresses loaded + +make test Run unit tests (forge + cargo) +make test-contracts Run contract tests only + +make logs-operators Follow all 3 operator logs +make logs-operator-N Follow operator-N logs (N=1,2,3) +make logs-monitor Follow oz-monitor logs +make logs-relayer Follow oz-relayer logs +make logs-relays Follow symbiotic-relay-1/2/3 logs + +make status Show running containers and health +make help Show all available commands +``` + +### Local Operator Development + +Run operator-1 outside Docker for fast iteration: + +```bash +# Start the full stack first +make start + +# Run operator-1 locally (replaces the Docker container) +make dev-operator +``` + +This runs `cargo run` with the generated config and `RUST_LOG=debug`. The local operator connects to the same Docker services and receives the same webhooks. + +## Operator Configuration + +Runtime configs are generated at `generated//operator-{n}/config.json`. + +Key settings: + +| Setting | Default | Description | +|---------|---------|-------------| +| `signer.event_poll_interval` | 15s | How often to check for new pending messages | +| `signer.sign_job_interval` | 1s | How often to retry pending Merkle roots | +| `signer.sign_worker_count` | 5 | Concurrent signing workers | +| `signer.min_batch_size` | 1 | Minimum messages before creating a tree | +| `oz_relayer.poll_interval` | 5s | How often to check for signed trees to submit | +| `oz_relayer.status_poll_interval` | 30s | How often to poll OZ Relayer for tx status | +| `symbiotic_relay.key_tag` | 15 | BLS key identifier in the sidecar | + +## Contract Addresses + +After deployment, canonical addresses are in `deployments/.json`. + +For manual testing, `make shell` opens an interactive shell with `.env` sourced and `ENV_CONFIG` / `DEPLOYMENTS_FILE` exported: + +```bash +make shell +# Then use jq to extract addresses: +jq '.source.layerzero.dvn' $DEPLOYMENTS_FILE +``` + +## Service Ports + +| Service | Port | Purpose | +|---------|------|---------| +| anvil (source) | 8545 | Source chain RPC | +| anvil (dest) | 8546 | Destination chain RPC | +| operator-1/2/3 | 3001-3003 | Operator debug APIs | +| symbiotic-relay-1/2/3 | 8081-8083 | BLS sidecars | +| oz-relayer | 8080 | Transaction relayer | diff --git a/content/symbiotic/troubleshooting.mdx b/content/symbiotic/troubleshooting.mdx new file mode 100644 index 00000000..f3e8f131 --- /dev/null +++ b/content/symbiotic/troubleshooting.mdx @@ -0,0 +1,187 @@ +--- +title: Troubleshooting +--- + +Common issues and solutions for the Symbiotic multi-provider template. + +## Checking Message Status + +```bash +# List all messages +curl http://localhost:3001/debug/v1/messages + +# Filter by status +curl "http://localhost:3001/debug/v1/messages?status=pending" + +# Get specific message +curl http://localhost:3001/debug/v1/messages/0xabc123... + +# Check pending Merkle roots +curl http://localhost:3001/debug/v1/pending +``` + +## Shared Issues + +### Webhook Not Received + +1. Check operator logs: + ```bash + make logs-operators | grep webhook + ``` +2. Verify connectivity (operators must be reachable from OZ Monitor): + ```bash + docker compose exec oz-monitor curl -s http://operator-1:3000/healthz + ``` +3. Confirm trigger is linked in the active monitor config under `generated//oz-monitor/monitors/` and rerun `make deploy` or `make start` if needed. + +### Authentication Failures (401) + +1. Verify secrets match between OZ Monitor config and operator `.env` +2. Ensure secret is at least 32 characters +3. Check for trailing whitespace in config files + +### Payload Parsing Errors + +1. Ensure `payload_mode: "raw"` is set in trigger config +2. Check operator logs for deserialization errors + +### BLS Signatures Not Aggregating + +1. Check sidecar status: + ```bash + docker compose ps symbiotic-relay-1 + docker compose logs symbiotic-relay-1 + ``` +2. Verify BLS keys in `.env` +3. Verify operator keys are registered in the Settlement contract + +### Quorum Not Reached + +1. All required operators must be running +2. Check operators are receiving the same events +3. Verify keys are registered in Settlement + +### Relayer Request Failed After All Retries + +1. Check OZ Relayer: + ```bash + curl -H "Authorization: Bearer $OZ_RELAYER_API_KEY" http://localhost:8080/api/v1/health + ``` +2. Check relayer logs: + ```bash + make logs-relayer + ``` +3. For 429 errors: increase `retry_backoff` or reduce submission rate + +### Service Won't Start + +1. Check missing env vars: + ```bash + docker compose config + ``` +2. Verify secrets are at least 32 characters +3. Check port conflicts: + ```bash + lsof -i :3001 # operator + lsof -i :8080 # relayer + ``` + +### Anvil Not Responding + +```bash +docker compose restart anvil anvil-settlement +docker compose logs anvil +``` + +### Contracts Not Found + +```bash +cat deployments/local.json +# Re-deploy if needed: +make clean && make start +``` + +### First-Run Genesis Waits + +On fresh devnet, `make start` may wait while voting power is captured. + +Symptoms: logs show `genesis not ready: totalVotingPower 0 < quorumThreshold 1`. + +This is expected on clean boot. If stuck, reset: `make clean && make start`. + +## LayerZero Issues + +### Message Stuck at "Processing" + +BLS signing not completing. Check sidecar health and logs: + +```bash +docker compose ps symbiotic-relay-1 +docker compose logs symbiotic-relay-1 +``` + +### submitProof Reverts + +Check Settlement contract state (operator keys, quorum threshold). Inspect: + +```bash +make logs-relayer | grep -i "revert\|error" +``` + +## Chainlink CCV Issues + +### EpochTooStale Revert (0xf5ab0d81) + +Settlement epoch/timestamp data is stale. Relayer fails at gas estimation. + +```bash +make logs-relayer | grep -E "estimate_gas|custom error|0xf5ab0d81" +``` + +Fix: refresh genesis or tune epoch timing parameters. + +### Watch Does Not Reach Success + +CCV requires destination `MessageExecuted(messageId)`, not just relayer submission. + +```bash +# Verify provider selection +jq -r '.activeProvider' config/environments/local.json + +# Check message lifecycle +make watch + +# If Failed, inspect relayer +make logs-relayer +``` + +### Submission Fails at estimate-gas + +```bash +make logs-relayer | grep -E "estimate_gas|custom error" +``` + +Common causes: stale epoch, incorrect CCV addresses, settlement not initialized. + +## Log Analysis + +### Useful Commands + +```bash +make logs-operators # All operator logs +make logs-operators 2>&1 | grep -i error # Errors only +make logs-operators | grep -i "retrying" # Retry activity +make logs-operators | grep "retries exhausted" +make logs-monitor # OZ Monitor +make logs-relayer # OZ Relayer +``` + +### Common Log Patterns + +| Pattern | Meaning | +|---------|---------| +| `webhook received` | Event received from OZ Monitor | +| `message batched` | Message added to Merkle tree | +| `signatures aggregated` | BLS quorum reached | +| `proof submitted` | Sent to OZ Relayer | +| `tx confirmed` | On-chain confirmation | diff --git a/src/navigation/ethereum-evm.json b/src/navigation/ethereum-evm.json index 8ce1463b..5644007d 100644 --- a/src/navigation/ethereum-evm.json +++ b/src/navigation/ethereum-evm.json @@ -797,6 +797,57 @@ "type": "separator", "name": "Open Source Tools" }, + { + "type": "folder", + "name": "Symbiotic Templates", + "index": { + "type": "page", + "name": "Overview", + "url": "/symbiotic" + }, + "children": [ + { + "type": "page", + "name": "Setup", + "url": "/symbiotic/setup" + }, + { + "type": "page", + "name": "Architecture", + "url": "/symbiotic/architecture" + }, + { + "type": "page", + "name": "LayerZero", + "url": "/symbiotic/layerzero" + }, + { + "type": "page", + "name": "Chainlink CCV", + "url": "/symbiotic/chainlink-ccv" + }, + { + "type": "page", + "name": "CLI & API Reference", + "url": "/symbiotic/cli" + }, + { + "type": "page", + "name": "Deployment", + "url": "/symbiotic/deployment" + }, + { + "type": "page", + "name": "Security", + "url": "/symbiotic/security" + }, + { + "type": "page", + "name": "Troubleshooting", + "url": "/symbiotic/troubleshooting" + } + ] + }, { "type": "folder", "name": "Relayer", From 95fdab3a86e043b98871d13bc874d91cdac69bde Mon Sep 17 00:00:00 2001 From: Dylan Kilkenny Date: Tue, 2 Jun 2026 14:40:27 +0100 Subject: [PATCH 2/2] docs(symbiotic): sync pages with source repo and add acceptance hooks Sync content/symbiotic/* with the source template repo's audited docs (openzeppelin-symbiotic-templates#132): chainlink_ccv testnet is now Supported, testnet keystore commands use --env, corrected keystore paths and CCIP selector config, JobAssigned emitter, and the Provider trait snippet. Adds the Acceptance Hooks page plus its sidebar nav entry. Signed-off-by: Dylan Kilkenny --- content/symbiotic/acceptance-hooks.mdx | 143 ++++++++++++++ content/symbiotic/architecture.mdx | 123 +++++------- content/symbiotic/chainlink-ccv.mdx | 88 +++------ content/symbiotic/cli.mdx | 245 ++++++----------------- content/symbiotic/deployment.mdx | 171 +++++----------- content/symbiotic/index.mdx | 12 +- content/symbiotic/layerzero.mdx | 85 ++++---- content/symbiotic/security.mdx | 108 +++-------- content/symbiotic/setup.mdx | 257 +++++++++++++++---------- content/symbiotic/troubleshooting.mdx | 201 +++++-------------- src/navigation/ethereum-evm.json | 5 + 11 files changed, 602 insertions(+), 836 deletions(-) create mode 100644 content/symbiotic/acceptance-hooks.mdx diff --git a/content/symbiotic/acceptance-hooks.mdx b/content/symbiotic/acceptance-hooks.mdx new file mode 100644 index 00000000..9275a603 --- /dev/null +++ b/content/symbiotic/acceptance-hooks.mdx @@ -0,0 +1,143 @@ +--- +title: Acceptance Hooks +--- + +Acceptance hooks run before a message is batched for signing. They let operators enforce local policy or call an external approval service without changing provider event decoding. + +## Decisions + +| Decision | Meaning | +| --- | --- | +| `accept` | Message proceeds to batching. | +| `reject` | Message is marked `Rejected` and is not evaluated again. | +| `defer` | Message is held until the returned RFC 3339 `until` timestamp. | + +When multiple hooks are configured, hooks run in order. The first `reject` stops evaluation. Otherwise all hooks run, and the final decision is the latest `defer` timestamp if any hook deferred, or `accept`. + +## Operator Config + +Hooks are configured in `config/environments/.json` under `operator.acceptanceHooks`: + +```json +{ + "operator": { + "acceptanceHooks": [ + { "type": "native", "name": "provider" }, + { + "type": "webhook", + "name": "approval", + "url": "http://approval-service:8088/", + "secret": "shared-secret", + "headers": { + "Authorization": { "type": "env", "value": "APPROVAL_HOOK_AUTHORIZATION" }, + "X-Operator": "operator-a" + }, + "timeout": "5s", + "errorBackoff": "30s", + "maxAttempts": 3 + } + ] + } +} +``` + +If `acceptanceHooks` is empty or omitted, the operator runs the built-in native `provider` hook for compatibility. Today the only named native hook is `provider`. + +Webhook hooks may include a `headers` object for additional outbound HTTP headers. Header values can be plain strings or `{ "type": "env", "value": "ENV_VAR_NAME" }` references; env references are resolved at operator startup and fail startup if missing or empty. `Content-Type` and `X-Hook-Signature` are reserved by the framework. + +## Native Hooks + +Native hooks are Rust code compiled into the operator. Use them for low-latency checks that need direct access to provider types or operator-local state. + +The supported native hook today is: + +| Name | Where to implement | Reference | +| --- | --- | --- | +| `provider` | Override `Provider::acceptance_hook` in the active provider implementation | `operator/src/provider/layerzero.rs` has the minimal pass-through implementation | + +The trait contract lives in `operator/src/provider/mod.rs`: + +```rust +async fn acceptance_hook( + &self, + msg: &MessageData, + context: &AcceptanceContext, +) -> Result; +``` + +Guidelines for native hook changes: + +- Return `AcceptanceDecision::accept()` when the message should proceed. +- Return `AcceptanceDecision::Reject { reason }` only for terminal policy failures. The message will not be evaluated again. +- Return `AcceptanceDecision::Defer { until, reason }` for temporary holds. The operator persists the defer state and re-evaluates on or after `until`. +- Use `context.defer_count` to give up after repeated defers and return `reject` when appropriate. This is a total framework defer count, including policy defers and operator-driven defers caused by hook errors. +- Use `context.previous_defer_reason` only for diagnostics or policy continuity; do not parse it as a stable machine contract. +- Avoid long blocking work in native hooks. If policy depends on a slow external system, prefer a webhook hook. +- Do not implement retry loops in the hook. Return the current decision and let the operator re-evaluate later. +- Avoid side effects. If unavoidable, key them by `message_id` and make them idempotent. +- Return `Err` only for true implementation failures. Native hook errors are converted into terminal `reject` decisions by the framework, so transient conditions should return `defer` instead. + +Minimal native accept: + +```rust +async fn acceptance_hook( + &self, + _msg: &MessageData, + _context: &AcceptanceContext, +) -> Result { + Ok(AcceptanceDecision::accept()) +} +``` + +Native reject/defer examples are covered by the signer tests in `operator/src/signer/mod.rs` (`RejectAllProvider` and `DeferAllProvider`). + +If a new named native hook is needed outside the provider hook, add a new `AcceptanceHookConfig::Native` name and wire it in `SignerJob::evaluate_acceptance_hooks`. + +## Webhook Contract + +The operator sends: + +```http +POST +Content-Type: application/json +X-Hook-Signature: sha256= +``` + +```json +{ + "message": { + "metadata": { + "source_chain": 1, + "destination_chain": 31338, + "block_number": 12345, + "message_id": "0x...", + "event_tx_hash": "0x..." + }, + "data": "" + }, + "context": { + "defer_count": 0, + "previous_defer_reason": null + } +} +``` + +The webhook returns `200 OK` with one of: + +```json +{ "decision": "accept" } +{ "decision": "reject", "reason": "amount above cap" } +{ "decision": "defer", "until": "2027-05-15T12:34:56Z", "reason": "awaiting approval" } +``` + +`reason` is optional. `until` is required for `defer`. + +`context.defer_count` is the total number of times the framework has deferred this message. It includes successful policy defers and operator-driven defers caused by webhook errors. Webhook unreachability is handled by `maxAttempts`, so do not treat `defer_count` as policy-only history. + +Webhook errors are not rejections. A non-2xx response, connection error, timeout, malformed JSON, or missing/invalid `until` auto-defers the message by `errorBackoff`. After `maxAttempts` consecutive errors for the same hook and message, the operator rejects the message with reason `approval service unreachable after N attempts`. + +Webhook URLs must use `http` or `https`. For production, prefer `https` or trusted in-cluster networking because message metadata is sent in the JSON body. + +Configured headers are sent with every request after startup-time validation. Use them for service-specific auth mechanisms such as `Authorization: Bearer ...` or tenant routing headers. Do not configure `Content-Type` or `X-Hook-Signature`; the operator owns those headers. + +The reference FastAPI implementation lives in `examples/webhook-hook/`. It is intentionally not Rust: webhook hooks are language-agnostic, and the example demonstrates the wire contract an external service must implement. diff --git a/content/symbiotic/architecture.mdx b/content/symbiotic/architecture.mdx index fa5993ec..913f5969 100644 --- a/content/symbiotic/architecture.mdx +++ b/content/symbiotic/architecture.mdx @@ -6,12 +6,12 @@ System overview for the Symbiotic multi-provider template. ## Core Model -1. One active provider per running stack (`config/environments/.json`). +1. One active provider per running stack, selected in `config/environments/.json`. 2. Shared off-chain runtime: - OZ Monitor for ingress - 3 operator processes - 3 Symbiotic relay sidecars for BLS signatures - - OZ Relayer for destination tx submission + - OZ Relayer for destination submission - Redis queue 3. Provider-specific on-chain contracts and calldata format. @@ -19,10 +19,8 @@ System overview for the Symbiotic multi-provider template. | Provider | Source ingress event | Destination submit call | Local | Testnet | Mainnet | | --- | --- | --- | --- | --- | --- | -| [`layerzero`](/symbiotic/layerzero) | `JobAssigned` | `SymbioticLayerZeroDVN.submitProof(...)` | Supported | Supported | Not yet | -| [`chainlink_ccv`](/symbiotic/chainlink-ccv) | `CCIPMessageSent` | `OffRamp.execute(...)` | Supported | Not yet | Not yet | - -See per-provider pages for detailed message flows and code pointers. +| [`layerzero`](/symbiotic/layerzero) | `JobAssigned` | `SymbioticLayerZeroDVN.submitProof(...)` | Supported | Supported | Verified end-to-end (operator-owned config) | +| [`chainlink_ccv`](/symbiotic/chainlink-ccv) | `CCIPMessageSent` | `OffRamp.execute(...)` | Supported | Supported | Not yet | ## Shared Off-Chain Runtime @@ -44,59 +42,27 @@ flowchart LR Relayer --> Submit ``` -All providers share the same off-chain pipeline. The provider abstraction determines: -- Which event the monitor watches for -- How operators encode the payload -- What calldata the relayer submits - -## Merkle Tree Batching +The provider abstraction decides: +- which source-chain event the monitor watches +- how operators encode the signed payload +- which destination call the relayer submits -Messages are batched into Merkle trees for gas efficiency: +## Merkle Batching -1. Multiple messages are collected into a batch -2. Each message becomes a leaf in the Merkle tree -3. The Merkle root is signed by operators -4. Proofs allow verifying individual messages against the signed root +Messages are collected into Merkle trees so one quorum signature can cover many messages: -This means: -- One signature covers many messages -- On-chain verification cost is amortized -- Individual messages can be verified independently +1. ingress events become message records +2. operators batch message leaves into a Merkle root +3. the root is signed through the Symbiotic relay sidecars +4. proofs let the destination verify individual messages against that signed root ## Symbiotic Integration Symbiotic provides the shared security layer: -- **Operator Registration**: Operators stake and register their BLS public keys -- **Settlement Contract**: Verifies BLS signatures and checks quorum -- **Slashing**: Misbehaving operators can be penalized (production) - -The Settlement contract: -1. Maintains the list of registered operators and their public keys -2. Defines the quorum threshold -3. Verifies aggregated signatures -4. Reports verification results to the provider contract - -## BLS Signing Pipeline - -1. Operators sign provider-defined payloads through Symbiotic relay sidecars. -2. Aggregation/quorum logic comes from settlement-backed Symbiotic attestation rules. -3. Provider-specific contracts decode and enforce those attestations on the destination execution path. - -## Message Status Lifecycle - -```mermaid -flowchart LR - Pending --> Processing --> Signed --> Submitted --> Confirmed -``` - -| Status | Description | -|--------|-------------| -| Pending | Received via webhook, awaiting batching | -| Processing | Batched into Merkle tree, awaiting BLS signatures | -| Signed | Quorum signatures collected, ready for submission | -| Submitted | Sent to OZ Relayer | -| Confirmed | On-chain TX confirmed | +- operators register BLS public keys +- settlement verifies quorum signatures +- voting power and epoch rules define signature validity ## Operator Internals @@ -106,12 +72,12 @@ flowchart LR | Provider | `operator/src/provider/` | Provider trait, event decoding, message storage | | SignerJob | `operator/src/signer/` | Batches messages into Merkle trees, requests BLS signatures | | RelaySubmitterJob | `operator/src/relay_submitter/` | Submits signed proofs via OZ Relayer | -| Storage | `operator/src/storage/` | redb key-value store (messages, Merkle trees, submissions) | +| Storage | `operator/src/storage/` | redb key-value store for messages, Merkle trees, and submissions | | Crypto | `operator/src/crypto/` | Merkle tree construction, leaf hashing, signing message encoding | ## Adding a New Provider -1. Create `operator/src/provider/yourprovider.rs` implementing the `Provider` trait: +1. Implement the `Provider` trait in `operator/src/provider/.rs`. ```rust #[async_trait] @@ -119,26 +85,39 @@ pub trait Provider: Send + Sync + 'static { fn name(&self) -> &'static str; async fn handle_webhook_event(&self, event: &WebhookEvent) -> Result<(), ProviderError>; - // Optional overrides: + // Required for a functional provider: the trait ships default impls for + // these that return an error, so the signing/submission path fails until + // you override all three. + fn compute_leaf_hash(&self, message: &MessageData) -> Result; + fn encode_signing_message(&self, tree: &MerkleTreeData) -> Result, ProviderError>; + fn prepare_submission( + &self, + message: &MessageData, + tree: &MerkleTreeData, + proof: &MerkleProof, + target_address: &str, + ) -> Result; + + // Optional overrides (sensible defaults provided): fn register_api_routes(&self, router: Router) -> Router { router } - async fn acceptance_hook(&self, _msg: &MessageData) -> Result<(), ProviderError> { Ok(()) } + fn max_batch_size(&self) -> usize { usize::MAX } + async fn acceptance_hook( + &self, + _msg: &MessageData, + _context: &AcceptanceContext, + ) -> Result { + Ok(AcceptanceDecision::accept()) + } + fn verifier_result_for( + &self, + _id: &B256, + ) -> Result, ProviderError> { + Ok(None) + } } ``` -2. Add configuration to `operator/src/config/mod.rs`. -3. Register in `create_provider()` in `operator/src/provider/mod.rs`. -4. Create provider-specific monitor template in `config/templates/oz-monitor/monitors/`. -5. Create `docs/.mdx` following the structure of existing provider docs (e.g., [LayerZero](/symbiotic/layerzero)). -6. Update this file's provider matrix, the [docs index](/symbiotic), and the project README. - -## Environment Comparison - -| Aspect | Local (Anvil) | Testnet | Production | -|--------|---------------|---------|------------| -| Source chain | Anvil 31337 | Base Sepolia 84532 | Mainnet | -| Dest/Settlement chain | Anvil 31338 | Sepolia 11155111 | Mainnet | -| Operators | 3 (local containers) | 3 (local containers) | 1+ (distributed) | -| Symbiotic Core | Deployed locally | Pre-deployed on Sepolia | Pre-deployed | -| BLS Keys | Deterministic | Deterministic | Hardware security | -| Quorum | 2-of-3 | 2-of-3 | Configurable | -| OZ Services | Local | Local | Hosted by OZ | +2. Add provider config to `operator/src/config/mod.rs`. +3. Register the provider in `operator/src/provider/mod.rs`. +4. Add monitor templates under `config/templates/oz-monitor/`. +5. Add `docs/.mdx` and update [the docs index](/symbiotic). diff --git a/content/symbiotic/chainlink-ccv.mdx b/content/symbiotic/chainlink-ccv.mdx index 1ccf9aad..0c38ee5a 100644 --- a/content/symbiotic/chainlink-ccv.mdx +++ b/content/symbiotic/chainlink-ccv.mdx @@ -2,15 +2,15 @@ title: Chainlink CCV --- -Symbiotic-secured Cross-Chain Verifier (CCV) for Chainlink CCIP-compatible message verification. +Symbiotic-secured Cross-Chain Verifier for Chainlink CCIP-compatible message verification. ## Overview -The CCV provider implements a Symbiotic-backed verifier compatible with Chainlink's CCIP CCV interface. When a message is sent through an OnRamp-compatible contract, a `CCIPMessageSent` event is emitted. Operators build a CCV payload, collect BLS attestations via Symbiotic relay sidecars, and the relayer submits the proof to `OffRamp.execute(...)` on the destination chain. The OffRamp calls `SymbioticCCV.verifyMessage(...)` for each message, and success is confirmed when `MessageExecuted(messageId)` is emitted. +The CCV provider watches `CCIPMessageSent` events, builds the verifier payload, collects BLS attestations through Symbiotic relay sidecars, and submits execution calldata to the destination OffRamp path. Success is confirmed when the destination emits `MessageExecuted(messageId)`. -This template supports the **Symbiotic CCV variant** only. The Chainlink auxiliary devenv stack (aggregator, indexer, verifier, executor) is not required. +This template supports the Symbiotic CCV path only. The Chainlink auxiliary devenv stack is not required. @@ -31,7 +31,7 @@ sequenceDiagram OnRamp-->>Monitor: CCIPMessageSent event Monitor->>Operators: HMAC webhook Operators->>Operators: build CCV payload + Merkle tree - Operators->>Relay: sign Merkle root (BLS) + Operators->>Relay: sign Merkle root Relay-->>Operators: aggregated signature Operators->>Relayer: OffRamp.execute calldata Relayer->>OffRamp: execute(message, ccvs, verifierResults) @@ -40,49 +40,44 @@ sequenceDiagram OffRamp-->>OffRamp: emit MessageExecuted(messageId) ``` -## Code Pointers +## Contracts and Code ### Contracts -- `contracts/src/ccv/SymbioticCCV.sol` -- CCV verifier implementation (`verifyMessage`, `forwardToVerifier`) -- `contracts/src/ccv/interfaces/` -- CCV interface definitions (`ICrossChainVerifierV1`, etc.) -- `contracts/src/ccv/libraries/` -- CCV encoding and helper libraries -- `contracts/src/symbiotic/Settlement.sol` -- BLS signature verification and quorum enforcement -- `contracts/src/symbiotic/KeyRegistry.sol` -- Operator BLS public key registry -- `contracts/src/symbiotic/Driver.sol` -- Epoch and genesis management +- `contracts/src/ccv/SymbioticCCV.sol` +- `contracts/src/ccv/interfaces/` +- `contracts/src/ccv/libraries/` +- `contracts/src/symbiotic/Settlement.sol` +- `contracts/src/symbiotic/KeyRegistry.sol` +- `contracts/src/symbiotic/Driver.sol` -### Operator (Rust) +### Operator -- `operator/src/provider/chainlink_ccv.rs` -- Decodes `CCIPMessageSent` events, builds CCV payloads -- `operator/src/provider/mod.rs` -- `Provider` trait and registration +- `operator/src/provider/chainlink_ccv.rs` +- `operator/src/provider/mod.rs` -### Config Templates +### Monitor Templates -- `config/templates/oz-monitor/monitors/ccip_message_sent.json` -- Monitor job for `CCIPMessageSent` events +- `config/templates/oz-monitor/monitors/ccip_message_sent.json` ## Configuration -Select CCV as the active provider: +Select CCV in `config/environments/.json`: ```json -// config/environments/.json { "activeProvider": "chainlink_ccv" } ``` -Chain config is shared across providers — chain IDs from `chains.source.chainId` and `chains.destination.chainId` are used as CCIP chain selectors at runtime. +CCIP chain selectors come from `chains.source.ccipChainSelector` and `chains.destination.ccipChainSelector`, which are required for non-local environments. They can be overridden at runtime via the `CCV_SOURCE_CHAIN_SELECTOR` / `CCV_DEST_CHAIN_SELECTOR` environment variables. For local Anvil only (chainId `31337`), the selector falls back to `chainId` when `ccipChainSelector` is unset. -Address resolution for CCV scripts: +Address resolution order: -1. `CCV_*` environment variables (highest priority) +1. `CCV_*` environment variables 2. `deployments/.json` -CCV settlement addresses in deployment state: - -- Destination: `destination.chainlinkCcv.settlement` in `deployments/.json` - -Available `CCV_*` override variables: +Available overrides: | Variable | Description | |----------|-------------| @@ -90,42 +85,9 @@ Available `CCV_*` override variables: | `CCV_DEST_ADDRESS` | SymbioticCCV on destination chain | | `CCV_SOURCE_ONRAMP_ADDRESS` | Source OnRamp-compatible contract | | `CCV_DEST_OFFRAMP_ADDRESS` | Destination OffRamp submit target | +| `CCV_SOURCE_CHAIN_SELECTOR` | Override the source CCIP chain selector | +| `CCV_DEST_CHAIN_SELECTOR` | Override the destination CCIP chain selector | -## Usage - -```bash -# Select chainlink_ccv provider in config/environments/local.json -# "activeProvider": "chainlink_ccv" - -# Start the stack -make start - -# Send a test message -make send MSG="hello" - -# Watch until MessageExecuted on destination -make watch - -# Or run both -make e2e -``` - -`make send` sends through the source mock `OnRamp.sendMessage(...)`, emitting `CCIPMessageSent`. - -`make watch` succeeds only when `MessageExecuted(messageId)` is found on the destination chain (not just relayer submission). - -See [CLI Reference](/symbiotic/cli) for full command options. - -## Deployment Status - -| Environment | Status | -|-------------|--------| -| Local | Supported (Symbiotic-only mock path) | -| Testnet | Not yet | -| Mainnet | Not yet | - -## Common Issues +Local and testnet are supported (testnet uses the `testnet-ccv` environment: `config/environments/testnet-ccv.json` and `deployments/testnet-ccv.json`, run via `ENV=testnet-ccv`); mainnet is not yet supported for CCV. `make watch` only succeeds once destination `MessageExecuted(messageId)` is observed. -- **EpochTooStale revert (0xf5ab0d81)** -- Settlement epoch data is stale. Refresh genesis or tune epoch timing. See [Troubleshooting](/symbiotic/troubleshooting#epochtoostale-revert-0xf5ab0d81). -- **Watch does not reach success** -- CCV requires destination `MessageExecuted(messageId)`, not just relayer submission. See [Troubleshooting](/symbiotic/troubleshooting#watch-does-not-reach-success). -- **Submission fails at estimate-gas** -- Common causes: stale epoch, incorrect CCV addresses, settlement not initialized. See [Troubleshooting](/symbiotic/troubleshooting#submission-fails-at-estimate-gas). +See [Setup](/symbiotic/setup) and [CLI & API Reference](/symbiotic/cli) for operation. diff --git a/content/symbiotic/cli.mdx b/content/symbiotic/cli.mdx index dc1607f2..ce735454 100644 --- a/content/symbiotic/cli.mdx +++ b/content/symbiotic/cli.mdx @@ -2,13 +2,15 @@ title: CLI & API Reference --- -Command-line interface and HTTP API for the Symbiotic multi-provider template. +Command-line and HTTP interfaces for the Symbiotic template. -## Make Commands +When `activeProvider` is `layerzero`, `make send` and `make e2e` use the managed `ExampleOApp` starter contract. If `layerzero.oapp.enabled` is `false`, those commands are unavailable for that environment. + +## Core Commands ### `make send` -Send one test message. Provider-aware based on `activeProvider`. +Send one test message through the active provider. ```bash make send @@ -18,7 +20,7 @@ make send ENV=testnet MSG="hello" ### `make watch` -Watch a previously sent message until it lands on the destination chain. +Watch a previously sent message until the destination path succeeds. ```bash make watch @@ -31,8 +33,8 @@ make watch TX=0x... |----------|-------------| | `ENV` | Environment (default: local) | | `TIMEOUT` | Max wait in seconds | -| `GUID` | Watch specific message by GUID | -| `TX` | Watch message by source TX hash | +| `GUID` | Watch a specific message by GUID | +| `TX` | Watch a message by source tx hash | ### `make e2e` @@ -44,22 +46,15 @@ make e2e MSG="custom message" make e2e ENV=testnet MSG="hello" TIMEOUT=180 ``` -Example output: +LayerZero starter OApp addresses are published under: -``` -[18:53:21] Operators: waiting to batch -[18:53:28] Operators: collecting BLS signatures -[18:53:30] Operators: signed (quorum reached) -[18:53:32] Relayer: submitted -[18:53:34] Relayer: confirmed (tx: 0x4617...) -[18:53:34] Destination target: verified on-chain (tx: 0x4617...) - -Message verified on destination chain! +```text +deployments/.json -> layerzero.oapp.{source,destination} ``` -## Direct xtask Commands +## Direct xtask Usage -Use the Rust CLI directly: +If you want to bypass `make`, use the Rust CLI directly: ```bash cargo xtask --env local msg send "hello" @@ -67,202 +62,84 @@ cargo xtask --env local msg watch --timeout 120 cargo xtask --env local msg e2e "hello" --timeout 120 ``` -### `cargo xtask msg send` - -```bash -cargo xtask --env local msg send "hello" -cargo xtask --env local msg send "hello" --gas 250000 -cargo xtask --env local msg send "hello" --json -``` +Global xtask overrides: -### `cargo xtask msg watch` +- `--env` selects the named environment +- `--env-config` overrides the environment JSON path +- `--deployments` overrides the deployments JSON path +- `--generated-dir` overrides the generated runtime output directory -```bash -cargo xtask --env local msg watch -cargo xtask --env local msg watch --id 0x... -cargo xtask --env local msg watch --tx 0x... -cargo xtask --env local msg watch --timeout 300 -``` +## Manual Operator Invocation -### `cargo xtask msg e2e` +`make dev-operator` is the normal entrypoint, but the raw binary requires four flags: ```bash -cargo xtask --env local msg e2e "hello" -cargo xtask --env local msg e2e "hello" --timeout 300 -cargo xtask --env local msg e2e "hello" --json +cargo run -p operator -- \ + --environment config/environments/local.json \ + --deployments deployments/local.json \ + --sidecar-address http://localhost:8081 \ + --relayer-id operator-relayer-1 ``` +| Flag | Purpose | +|------|---------| +| `--environment` | Environment JSON input | +| `--deployments` | Deployment addresses JSON input | +| `--sidecar-address` | Symbiotic relay sidecar gRPC address | +| `--relayer-id` | OZ Relayer identity to bind to the destination target | + ## Message Cache -After `send`, xtask saves message details to: +After `send`, xtask stores message details in: -``` +```text generated//msg-cache.json ``` -`watch` uses this cache when no explicit `--id` or `--tx` is provided. +`watch` uses this cache when `--id` and `--tx` are omitted. ## HTTP API -Each operator exposes HTTP endpoints on ports 3001-3003. - -### Webhook Endpoints - -#### POST /webhook/events - -Receives provider ingress events from OZ Monitor. - -**Authentication:** HMAC-SHA256 via two headers: -- `X-Signature`: Hex-encoded HMAC-SHA256 of `body + timestamp` -- `X-Timestamp`: Unix timestamp in milliseconds - -The webhook secret must match between operator (`WEBHOOK_SECRET` env var) and OZ Monitor trigger config (`config.secret.value`). +Each operator exposes HTTP endpoints on ports `3001-3003`. -#### POST /api/v1/webhooks/oz-relayer +### Ingress Webhooks -Receives transaction status updates from OZ Relayer. +| Endpoint | Purpose | Auth | +|----------|---------|------| +| `POST /webhook/events` | Provider ingress events from OZ Monitor | `X-Signature` + `X-Timestamp` using `WEBHOOK_SECRET` | +| `POST /api/v1/webhooks/oz-relayer` | Transaction status updates from OZ Relayer | `X-Signature` using `OZ_RELAYER_WEBHOOK_SECRET` | -**Authentication:** Base64-encoded HMAC-SHA256 of raw JSON body in `X-Signature` header, using `OZ_RELAYER_WEBHOOK_SECRET`. +### Debug and Health -### Debug Endpoints +| Endpoint | Purpose | +|----------|---------| +| `GET /debug/v1/messages` | List messages with processing and submission state | +| `GET /debug/v1/messages/:message_id` | Read one message by ID | +| `GET /debug/v1/pending` | List pending Merkle roots | +| `GET /healthz` | Liveness check | -#### GET /debug/v1/messages - -List messages with processing and submission status. +Example: ```bash curl -s http://localhost:3001/debug/v1/messages curl "http://localhost:3001/debug/v1/messages?status=pending&limit=10" ``` -| Parameter | Default | Description | -|-----------|---------|-------------| -| `status` | (all) | Filter: `pending`, `processing`, `signed` | -| `limit` | 50 | Max messages returned | -| `offset` | 0 | Pagination offset | - -#### GET /debug/v1/messages/:message_id - -Get a specific message by ID. - -#### GET /debug/v1/pending - -List Merkle roots awaiting BLS signatures. - -### Proof Endpoints - -#### POST /api/v1/layerzero/proof - -Retrieve Merkle proofs for processed messages. - -```bash -curl -X POST http://localhost:3001/api/v1/layerzero/proof \ - -H "Content-Type: application/json" \ - -d '{"message_ids": ["0xabc123..."]}' -``` - -Response fields: `root_hash`, `root_proof` (BLS signature), `index`, `leaf`, `siblings`, `original_list`. - -#### POST /api/v1/layerzero/verify - -Verify a Merkle proof is valid (testing only). - -### GET /healthz +### LayerZero Proof Endpoints -Returns `200 OK` if healthy. +LayerZero operators also expose: -## Webhook Configuration +| Endpoint | Purpose | +|----------|---------| +| `POST /api/v1/layerzero/proof` | Return Merkle proofs for processed messages | +| `POST /api/v1/layerzero/verify` | Verify a Merkle proof in test workflows | -OZ Monitor sends events via HMAC-SHA256 authenticated webhooks. +## Webhook Template Requirements -### Trigger Template +OZ Monitor trigger templates live under `config/templates/oz-monitor/triggers/` and are copied into `generated//oz-monitor/triggers/`. -Webhook triggers are defined in `config/templates/oz-monitor/triggers/` and copied to `generated//oz-monitor/triggers/` at startup. - -Key settings: - -| Setting | Description | -|---------|-------------| -| `url.value` | Operator endpoint (use Docker service name in compose) | -| `secret.value` | Must match `WEBHOOK_SECRET` in operator `.env` | -| `payload_mode` | Must be `"raw"` | - -### Monitor Jobs - -Provider-specific monitor templates: -- `config/templates/oz-monitor/monitors/layerzero_job_assigned.json` -- `config/templates/oz-monitor/monitors/ccip_message_sent.json` - -### Webhook Payload Format - -```json -{ - "EVM": { - "logs": [{ "address": "0x...", "topics": ["..."], "data": "0x..." }], - "matched_on_args": { - "events": [{ - "signature": "JobAssigned(address,bytes,uint256,address)", - "args": [{ "name": "dvn", "kind": "address", "value": "0x..." }] - }] - }, - "monitor": { "name": "LayerZero JobAssigned" }, - "network_slug": "anvil-source", - "transaction": { "hash": "0x...", "blockNumber": 123 } - } -} -``` - -For CCV, the event signature is `CCIPMessageSent(...)` with different args. - -## Retry Configuration - -### Symbiotic Relay (Linear Backoff) - -For gRPC calls to BLS signing sidecars. - -``` -backoff = retry_backoff x (attempt + 1) -``` - -### OZ Relayer (Exponential Backoff with Jitter) - -For HTTP calls to the transaction relayer. - -``` -base = retry_backoff x 2^attempt -jitter = random(0, base x 0.25) -backoff = min(base + jitter, 60s) -``` - -### Retry Settings - -| Setting | Description | Default | -|---------|-------------|---------| -| `max_retries` | Maximum retry attempts (0 = no retries) | 3 | -| `retry_backoff` | Base backoff duration | 1s | -| `timeout` | Request timeout (OZ Relayer only) | 30s | - -### Retryable vs Non-Retryable Errors - -**Retried:** HTTP 429, HTTP 500-504, network errors (connection refused, timeout, DNS failure). - -**Not retried:** HTTP 4xx (except 429), domain errors (chain not configured, transaction not found). - -### Tuning Examples - -```json -// Low-latency (devnet/testnet) -{ "oz_relayer": { "max_retries": 5, "retry_backoff": "100ms" } } - -// Production -{ "oz_relayer": { "max_retries": 3, "retry_backoff": "1s" } } - -// High-volume -{ "oz_relayer": { "max_retries": 5, "retry_backoff": "2s" } } -``` +Required settings: -| Config | Symbiotic Relay total | OZ Relayer total | -|--------|----------------------|------------------| -| 1s / 3 retries | ~6s | ~9s | -| 1s / 5 retries | ~15s | ~39s | -| 2s / 3 retries | ~12s | ~18s | +- `url.value`: operator webhook URL +- `secret.value`: must match the operator webhook secret +- `payload_mode: "raw"` diff --git a/content/symbiotic/deployment.mdx b/content/symbiotic/deployment.mdx index f5958481..b1100d62 100644 --- a/content/symbiotic/deployment.mdx +++ b/content/symbiotic/deployment.mdx @@ -6,163 +6,86 @@ Deploying and operating the stack beyond local development. -Testnet deployment currently supports the `layerzero` provider only. CCV is local-only for now. +Testnet supports both providers: `layerzero` (`config/environments/testnet.json`, `ENV=testnet`) and `chainlink_ccv` (`config/environments/testnet-ccv.json`, `ENV=testnet-ccv`). The workflow below uses `ENV=testnet` (LayerZero); for the CCV path substitute `ENV=testnet-ccv`. Mainnet CCV is not yet supported. -## Testnet (LayerZero) +## Testnet Workflow -- Source: Base Sepolia (`84532`) -- Destination: Sepolia (`11155111`) -- Provider: `layerzero` - -### Runtime Model +### 1. Generate keystores ```bash -make validate ENV=testnet -make deploy ENV=testnet -make refresh-genesis ENV=testnet # when validation says genesis is stale -make run-operators ENV=testnet -make e2e ENV=testnet -``` - -| Command | What it does | -|---------|-------------| -| `validate` | Read-only checks: config, chain reachability, deployment state, operator state, relayer signer safety | -| `deploy` | Deploy managed contracts, update `deployments/testnet.json` and `generated/testnet/` | -| `refresh-genesis` | Refresh settlement genesis without redeploying contracts | -| `run-operators` | Start non-local operator-side services | +cargo xtask --env testnet generate-signer \ + --name deployer --name operator-1 --name operator-2 --name operator-3 -### Environment Inputs - -Config lives in `config/environments/testnet.json`: -- Chain IDs and EIDs -- Default RPC URLs -- LayerZero predeploys -- Symbiotic Core predeploys -- Relay timing - -xtask resolves RPC URLs from the environment JSON first. `SOURCE_RPC_URL` / `DEST_RPC_URL` in `.env` are fallback overrides only. - -Required `.env` values: - -```bash -PRIVATE_KEY=0x -KEYSTORE_PASSPHRASE= -OPERATOR_1_PRIVATE_KEY=0x -OPERATOR_2_PRIVATE_KEY=0x -OPERATOR_3_PRIVATE_KEY=0x +cargo xtask --env testnet generate-signer \ + --name signer-1 --name signer-2 --name signer-3 ``` -Optional relayer bootstrap inputs: +Each command prompts for a passphrase interactively (hidden input, confirmed). Use separate passphrases for operator and relayer keystores. The `--env testnet` flag writes keys under `config/keys/testnet/`, where `config/environments/testnet.json` expects them — without it, keys default to `config/keys/local/` and `make deploy ENV=testnet` will not find them. -```bash -RELAYER_1_PRIVATE_KEY=0x -RELAYER_2_PRIVATE_KEY=0x -RELAYER_3_PRIVATE_KEY=0x -``` +Relayer keystores (`signer-1/2/3`) are generated automatically by `make deploy` for any environment if they don't already exist (using `KEYSTORE_PASSPHRASE`). Generating them manually above is optional — it only lets you set the passphrase interactively instead of reading it from `.env.testnet`. -Relayer private keys are setup-time inputs only. The steady-state runtime source is the OZ relayer keystore files under `config/oz-relayer/keys/`. +### 2. Configure environment -### Workflow +Copy `.env.example` to `.env.testnet` and fill in the RPC URLs (Base Sepolia source, Sepolia destination) and keystore passphrases. Signer definitions in `config/environments/testnet.json` reference these passphrases. -#### 1. Generate keys - -```bash -make setup -``` - - - -For public testnets: do not use known local/dev keys. Ensure deployer, operators, and relayer signers all have testnet ETH. - - - -#### 2. Validate first +### 3. Deploy and run ```bash make validate ENV=testnet -``` - -Catches: missing keys, underfunded accounts, stale genesis, relayer signer issues. - -#### 3. Deploy managed contracts - -```bash make deploy ENV=testnet +make refresh-genesis ENV=testnet # only when validation says genesis is stale +make start ENV=testnet +make e2e ENV=testnet MSG="hello" ``` -Updates `deployments/testnet.json` and `generated/testnet/`. - -#### 4. Refresh genesis when needed - -```bash -make refresh-genesis ENV=testnet -``` - -Use this instead of redeploying when contracts are already in place and only the settlement header is stale. +- `validate` runs read-only checks against config, RPC reachability, deployment state, operator state, and relayer signer safety. +- `deploy` deploys managed contracts and regenerates `deployments/testnet.json` and `generated/testnet/`. +- `refresh-genesis` repairs stale settlement genesis without redeploying contracts. +- `start` detects the environment type and starts the appropriate services. -#### 5. Start operator services +By default, the LayerZero testnet config also deploys the starter `ExampleOApp`, so `make e2e ENV=testnet` remains part of the main workflow. -```bash -make run-operators ENV=testnet -``` +If you want provider infrastructure only, set `layerzero.oapp.enabled` to `false` in `config/environments/testnet.json` before deploying. `validate` will still pass, but `make send` and `make e2e` will be unavailable. -Starts from `docker-compose.yml` (no `docker-compose.local.yml` overlay): -- Operators, OZ Monitor, OZ Relayer, Symbiotic relay sidecars +## Required Inputs -#### 6. Send and verify +Testnet settings live in `config/environments/.json`. `.env.` must provide: ```bash -make e2e ENV=testnet MSG="hello" -``` - -### How Testnet Differs From Local - -| Area | Local | Testnet | -|------|-------|---------| -| Entrypoint | `make start` | `make deploy` + `make run-operators` | -| Chains | Local Anvil | Base Sepolia + Sepolia | -| LayerZero endpoints | Local mocks | Predeployed | -| Symbiotic Core | Deployed fresh | Predeployed on destination | -| Genesis refresh | Folded into startup | Explicit `make refresh-genesis` when stale | -| Compose files | `docker-compose.yml` + `docker-compose.local.yml` | `docker-compose.yml` only | -| Operator registration | Auto-impersonate | Separate step (real chains) | - -### Testnet Architecture - -```text -Base Sepolia (84532) Sepolia (11155111) --------------------- ------------------- -LZ V2 Endpoint (pre-deployed) LZ V2 Endpoint (pre-deployed) -DVN.assignJob() DVN.submitProof() -> Settlement -TestOApp.send() TestOApp.lzReceive() - Driver, KeyRegistry, VotingPowers - - OZ Monitor -> Operators -> Symbiotic Relays -> OZ Relayer - (local Docker containers) +DEPLOYER_PASSPHRASE= +OPERATOR_1_PASSPHRASE= +OPERATOR_2_PASSPHRASE= +OPERATOR_3_PASSPHRASE= +KEYSTORE_PASSPHRASE= # used by OZ Relayer at runtime ``` -### Testnet Troubleshooting +Operator and deployer passphrases are resolved by `xtask` via the `signers` config. `KEYSTORE_PASSPHRASE` is read directly by the OZ Relayer container to decrypt relayer keystores at runtime. -**`validation failed: genesis stale`** -- Run `make refresh-genesis ENV=testnet`. +## Operational Notes -**`insufficient funds for gas`** -- Fund the deployer address from `PRIVATE_KEY`. +- Local uses `make chains && make deploy && make start`; testnet uses `make deploy ENV=testnet` plus `make start ENV=testnet`. +- Testnet uses `docker-compose.yml` only, not the local overlay. +- Canonical deployment addresses always live in `deployments/.json`. -**Sidecars fail with RPC rate limits** -- Three sidecars syncing at once can overwhelm weak testnet RPC plans. Use higher-throughput RPCs or temporarily reduce sidecar count. +## Sidecar Image Compatibility -**Keys look drained immediately** -- Do not use known local/dev keys on public testnets. Regenerate with `make setup`. +The Symbiotic relay sidecar (`symbioticfi/relay`) is still in release-candidate phase. The pinned version in `docker-compose.yml` and the `RELAY_IMAGE` default in `xtask/src/genesis.rs` must stay in sync. Known states observed during integration: -**Fresh relay deploys on shared testnet** -- Most fragile path. Prefer reusing existing environment and `make refresh-genesis` for stale-genesis repair. +| Tag | Status | Notes | +| --- | --- | --- | +| `1.0.1-rc4` | works | Current pin. First image where aggregation produces proofs against real mainnet. | +| `1.0.1-rc3.0.20260507060511-...` | broken | Starts cleanly but the aggregator never produces proofs even with full quorum signing. | +| `1.0.1-20260326074346-da0bce8ba949` | broken on non-local | Panics at config init with an `errors.As` target-type bug when multiple chains are configured. Local works because the codepath differs. | -## Mainnet +If you bump the tag, run a full testnet e2e first and verify proofs are aggregated end-to-end. The `1.0.1` GA tag has not yet shipped from Symbiotic — track upstream releases before promoting any tag past `rc4`. -Not yet supported. Key differences from testnet will include: -- Hardware security for BLS keys -- Distributed operators (not co-located containers) -- OZ-hosted services -- Configurable quorum thresholds +## Common Deployment Failures -## Address Management +- `validation failed: genesis stale`: run `make refresh-genesis ENV=`. +- `insufficient funds for gas`: fund the deployer account from the deployer keystore address. +- Sidecars hit RPC rate limits: use stronger RPCs or reduce sidecar load temporarily. +- Keys look drained immediately: do not reuse known local/dev keys on public networks. -All deployment addresses are canonical in `deployments/.json`. This file is updated by `make deploy` and read by all runtime components. +For runtime failures after deployment, see [Troubleshooting](/symbiotic/troubleshooting). diff --git a/content/symbiotic/index.mdx b/content/symbiotic/index.mdx index 706845bc..e9db01b9 100644 --- a/content/symbiotic/index.mdx +++ b/content/symbiotic/index.mdx @@ -7,12 +7,13 @@ title: Symbiotic Templates Running, configuring, and monitoring the stack. 1. [Setup](/symbiotic/setup) -- Config structure, environment setup, running locally -2. [Deployment](/symbiotic/deployment) -- Testnet and mainnet deployment +2. [Deployment](/symbiotic/deployment) -- Testnet deployment 3. Choose your provider: - [LayerZero](/symbiotic/layerzero) -- DVN for LayerZero V2 - [Chainlink CCV](/symbiotic/chainlink-ccv) -- Cross-Chain Verifier for CCIP -4. [CLI & API Reference](/symbiotic/cli) -- Commands, HTTP endpoints, webhook config -5. [Troubleshooting](/symbiotic/troubleshooting) -- Common issues and debugging +4. [Acceptance Hooks](/symbiotic/acceptance-hooks) -- Native and webhook policy checks before batching +5. [CLI & API Reference](/symbiotic/cli) -- Commands, HTTP endpoints, webhook config +6. [Troubleshooting](/symbiotic/troubleshooting) -- Common issues and debugging ## For Integrators @@ -22,5 +23,6 @@ Understanding the system and adding new providers. 2. Choose your provider: - [LayerZero](/symbiotic/layerzero) -- Message flow, contracts, code pointers - [Chainlink CCV](/symbiotic/chainlink-ccv) -- Message flow, contracts, code pointers -3. [Architecture: Adding a New Provider](/symbiotic/architecture#adding-a-new-provider) -- Provider trait, registration, templates -4. [Security](/symbiotic/security) -- Trust model, access control, invariants +3. [Acceptance Hooks](/symbiotic/acceptance-hooks) -- Hook contract and webhook wire format +4. [Architecture: Adding a New Provider](/symbiotic/architecture#adding-a-new-provider) -- Provider trait, registration, templates +5. [Security](/symbiotic/security) -- Trust model, access control, invariants diff --git a/content/symbiotic/layerzero.mdx b/content/symbiotic/layerzero.mdx index 91e4d569..1c932c8f 100644 --- a/content/symbiotic/layerzero.mdx +++ b/content/symbiotic/layerzero.mdx @@ -2,11 +2,11 @@ title: LayerZero --- -Symbiotic-secured DVN (Decentralized Verifier Network) for LayerZero V2 cross-chain messaging. +Symbiotic-secured DVN for LayerZero V2 cross-chain messaging. ## Overview -The LayerZero provider implements a DVN that uses Symbiotic shared security to verify cross-chain messages. When a message is sent through LayerZero's `SendUln302`, the DVN contract emits a `JobAssigned` event. Operators batch these jobs into Merkle trees, collect BLS signatures through Symbiotic relay sidecars, and submit the signed proof to the destination DVN contract. The destination DVN verifies the BLS quorum via the Settlement contract and forwards verification to LayerZero's `ReceiveUln302`. +The LayerZero provider watches `JobAssigned` events emitted by the `SymbioticLayerZeroDVN` source DVN (whose `assignJob` function is called by `SendUln302`), batches them into Merkle trees, collects BLS attestations through Symbiotic relay sidecars, and submits proofs to the destination DVN. The destination DVN verifies the quorum via Settlement and forwards verification into `ReceiveUln302`. ## Message Flow @@ -28,7 +28,7 @@ sequenceDiagram DVN_S-->>Monitor: JobAssigned event Monitor->>Operators: HMAC webhook Operators->>Operators: batch into Merkle tree - Operators->>Relay: sign Merkle root (BLS) + Operators->>Relay: sign Merkle root Relay-->>Operators: aggregated signature Operators->>Relayer: submitProof calldata Relayer->>DVN_D: submitProof(root, proof, signatures) @@ -37,42 +37,41 @@ sequenceDiagram DVN_D->>RecvUln: verify() ``` -## Code Pointers +## Contracts and Code ### Contracts -- `contracts/src/SymbioticLayerZeroDVN.sol` -- DVN contract handling `assignJob` (source) and `submitProof` (destination) -- `contracts/src/symbiotic/Settlement.sol` -- BLS signature verification and quorum enforcement -- `contracts/src/symbiotic/KeyRegistry.sol` -- Operator BLS public key registry -- `contracts/src/symbiotic/VotingPowers.sol` -- Operator voting power tracking -- `contracts/src/symbiotic/Driver.sol` -- Epoch and genesis management -- `contracts/src/examples/TestOApp.sol` -- Test application for sending/receiving messages +- `contracts/src/SymbioticLayerZeroDVN.sol` +- `contracts/src/symbiotic/Settlement.sol` +- `contracts/src/symbiotic/KeyRegistry.sol` +- `contracts/src/symbiotic/VotingPowers.sol` +- `contracts/src/symbiotic/Driver.sol` +- `contracts/src/examples/ExampleOApp.sol` -### Operator (Rust) +### Operator -- `operator/src/provider/layerzero.rs` -- Decodes `JobAssigned` events, stores messages -- `operator/src/provider/mod.rs` -- `Provider` trait and registration -- `operator/src/crypto/mod.rs` -- Merkle tree construction, DVN leaf hashing -- `operator/src/signer/mod.rs` -- Batches messages, requests BLS signatures -- `operator/src/relay_submitter/mod.rs` -- Submits signed proofs via OZ Relayer +- `operator/src/provider/layerzero.rs` +- `operator/src/provider/mod.rs` +- `operator/src/crypto/mod.rs` +- `operator/src/signer/mod.rs` +- `operator/src/relay_submitter/mod.rs` -### Config Templates +### Monitor Templates -- `config/templates/oz-monitor/monitors/layerzero_job_assigned.json` -- Monitor job for `JobAssigned` events -- `config/templates/oz-monitor/triggers/webhook_layerzero.json` -- Webhook trigger template +- `config/templates/oz-monitor/monitors/layerzero_job_assigned.json` +- `config/templates/oz-monitor/triggers/webhook_layerzero.json` ## Configuration -Select LayerZero as the active provider: +Select LayerZero in `config/environments/.json`: ```json -// config/environments/.json { "activeProvider": "layerzero" } ``` -Chain config is shared across providers and lives at the top level: +Shared chain config: | Field | Description | |-------|-------------| @@ -81,7 +80,7 @@ Chain config is shared across providers and lives at the top level: | `chains.source.eid` | LayerZero endpoint ID for source | | `chains.destination.eid` | LayerZero endpoint ID for destination | -LayerZero predeploys (testnet/mainnet) go in `chains..predeploys.layerzero`: +LayerZero predeploys for non-local environments live under `chains..predeploys.layerzero`: ```json { @@ -94,35 +93,25 @@ LayerZero predeploys (testnet/mainnet) go in `chains..predeploys.layerzero } ``` -`make deploy` and `make start` use these values to generate runtime configs (`destination_chains`, `chain_relayers`, `eid_to_chain_id`) under `generated//`. Validation fails if chain IDs/EIDs drift from the generated deployment state. +`make deploy` and `make start` use these values to generate the runtime chain maps under `generated//`. -## Usage +The template also manages a starter OApp by default: -```bash -# Select layerzero provider in config/environments/local.json -# "activeProvider": "layerzero" - -# Start the stack -make start - -# Send a test message -make send MSG="hello" - -# Watch until destination verification -make watch - -# Or run both in one shot -make e2e +```json +{ + "layerzero": { + "oapp": { + "enabled": true + } + } +} ``` -`make send` sends through `TestOApp.send(...)` which calls `SendUln302`, triggering `DVN.assignJob()`. - -`make watch` succeeds when destination target verification is observed on-chain. +- `true`: deploy and wire `ExampleOApp` on both chains +- `false`: skip the starter OApp and run LayerZero in provider-only mode -See [CLI Reference](/symbiotic/cli) for full command options. +Starter OApp deployments are published under `deployments/.json` at `layerzero.oapp.source` and `layerzero.oapp.destination`. -## Common Issues +Provider validation does not require the starter OApp to exist. Disabling it only removes the `make send` and `make e2e` demo flow for that environment. -- **Message stuck at "Processing"** -- BLS signatures not aggregating. Check sidecar health and operator key registration. See [Troubleshooting](/symbiotic/troubleshooting#bls-signatures-not-aggregating). -- **Quorum not reached** -- All 3 operators must be running and receiving the same events. See [Troubleshooting](/symbiotic/troubleshooting#quorum-not-reached). -- **submitProof reverts** -- Check that the OZ Relayer address is authorized as a submitter on the DVN contract, and that Settlement has correct operator keys. See [Troubleshooting](/symbiotic/troubleshooting#layerzero-issues). +Local and testnet are supported. See [Setup](/symbiotic/setup), [Deployment](/symbiotic/deployment), and [CLI & API Reference](/symbiotic/cli) for operation. diff --git a/content/symbiotic/security.mdx b/content/symbiotic/security.mdx index 7df1803e..d5ffd3ae 100644 --- a/content/symbiotic/security.mdx +++ b/content/symbiotic/security.mdx @@ -2,112 +2,66 @@ title: Security --- -Security architecture and trust assumptions for the Symbiotic multi-provider template. +Security architecture and trust assumptions for the Symbiotic template. ## Shared Trust Model | Entity | Trust Level | Notes | |--------|-------------|-------| -| **Settlement** | Trusted | Symbiotic contract for BLS signature verification | -| **Authorized Submitters** | Semi-trusted | Whitelisted addresses that submit proofs; cannot forge signatures but can grief (spam invalid proofs) | -| **Owner** | Trusted | Admin with pause/unpause, submitter management | -| **External users** | Untrusted | Cannot call privileged functions directly | +| Settlement | Trusted | Verifies BLS quorum signatures | +| Authorized submitters | Semi-trusted | Can submit proofs, but cannot forge valid signatures | +| Owner | Trusted | Handles pause, unpause, and submitter management | +| External users | Untrusted | No privileged access | -### Symbiotic Security Layer +Webhook ingress between OZ Monitor, OZ Relayer, and operators uses HMAC-SHA256 shared secrets. See [CLI & API Reference](/symbiotic/cli#http-api) for the header format. -- Operators stake and register BLS public keys via `KeyRegistry` -- Settlement contract verifies BLS quorum before accepting proofs -- Slashing handled by Symbiotic core contracts (production) +## Runtime API Security -### Webhook Authentication +Operator runtime behavior that is easy to miss: -Webhooks between OZ Monitor and operators use HMAC-SHA256: -1. Monitor computes `HMAC-SHA256(secret, body + timestamp)` -2. Signature sent in `X-Signature` header -3. Timestamp (ms since epoch) sent in `X-Timestamp` header -4. Operator rejects invalid/missing signatures or expired timestamps (HTTP 401) +- `/healthz` bypasses authentication so orchestration health checks keep working. +- `/webhook/events` verifies HMAC over the raw body plus `X-Timestamp`, and rejects requests outside a 300s timestamp window. +- `/api/v1/webhooks/oz-relayer` verifies a separate base64 HMAC signature over the raw JSON body. +- Authentication failures on ingress webhooks intentionally return generic `401` responses instead of detailed errors. +- CORS is off by default. +- Debug routes are security-gated and may be hidden by middleware before the handler runs. -Secrets must be at least 32 characters. See [CLI Reference](/symbiotic/cli#webhook-configuration) for config details. +Secrets are loaded from `WEBHOOK_SECRET` and `OZ_RELAYER_WEBHOOK_SECRET` at startup, not from the environment JSON. ## LayerZero DVN Security -Trust assumptions specific to the `SymbioticLayerZeroDVN` contract. - -| Entity | Trust Level | Notes | -|--------|-------------|-------| -| **SendUln302** | Trusted | LayerZero's send library; only caller for `assignJob` | +The LayerZero provider trusts `SendUln302` as the only valid source-chain caller for `assignJob`. ### Access Control -#### Source Chain - -| Function | Caller | Purpose | -|----------|--------|---------| -| `assignJob` | SendUln302 only | Register verification job, emit event | -| `getFee` | Anyone | Query verification fee (view) | - -#### Destination Chain - -| Function | Caller | Purpose | -|----------|--------|---------| -| `submitProof` | Authorized submitters | Submit signed Merkle proof for verification | - -#### Admin - | Function | Caller | Purpose | |----------|--------|---------| +| `assignJob` | `SendUln302` only | Register verification jobs and emit `JobAssigned` | +| `getFee` | Anyone | Quote verification fee | +| `submitProof` | Authorized submitters | Submit signed Merkle proofs | | `addSubmitter` / `removeSubmitter` | Owner | Manage submitter whitelist | -| `setBaseFee` | Owner | Update verification fee | -| `pause` / `unpause` | Owner | Emergency controls | -| `withdraw` | Owner | Recover ETH (force-sent or accidental) | -| `transferOwnership` | Owner | Transfer admin rights | +| `setBaseFee` / `pause` / `unpause` / `withdraw` / `transferOwnership` | Owner | Administrative controls | ### Invariants -1. **Leaf monotonicity**: `verifiedLeaves[leaf]` transitions `false -> true` only, never back -2. **Root monotonicity**: `verifiedRoots[root]` transitions `false -> true` only, never back -3. **Signature requirement**: Uncached roots require valid BLS quorum from Settlement -4. **Packet header integrity**: Verified packets have exactly 81 bytes and correct `dstEid` -5. **No ETH custody**: Contract does not collect fees; `assignJob` rejects `msg.value > 0` - -### Deployment Modes - -| Mode | sendUln | receiveUln | settlement | Use case | -|------|---------|------------|------------|----------| -| Source only | Set | Zero | Zero | Emit `JobAssigned` events | -| Destination only | Zero | Set | Set | Verify proofs, call ReceiveUln | -| Bidirectional | Set | Set | Set | Both functions on same chain | - -### What the DVN Does NOT Do - -- **Fee custody**: Fees handled by LayerZero's fee accounting -- **Signature generation**: BLS signing happens off-chain via Symbiotic Relay -- **Slashing**: Handled by Symbiotic core contracts +1. `verifiedLeaves[leaf]` only moves from `false` to `true`. +2. `verifiedRoots[root]` only moves from `false` to `true`. +3. Uncached roots require a valid BLS quorum from Settlement. +4. Verified packet headers must have the expected length and destination EID. +5. `assignJob` rejects `msg.value > 0`; the DVN does not custody fees. ## Chainlink CCV Security -Trust assumptions specific to the `SymbioticCCV` contract. - ### Access Control | Function | Caller | Purpose | |----------|--------|---------| -| `forwardToVerifier` | OnRamp | Source-chain hook for CCV registration | -| `verifyMessage` | OffRamp | Destination verification hook | -| `getFee` | Anyone | Quote verification fee (view) | +| `forwardToVerifier` | OnRamp | Register a message for CCV verification | +| `verifyMessage` | OffRamp | Verify the message on the destination path | +| `getFee` | Anyone | Quote verification fee | ### Invariants -CCV verification requires: -1. Valid BLS quorum signature from Settlement -2. Correct message ID derivation -3. Epoch freshness (reverts with `EpochTooStale` if settlement data is stale) - -## External Dependencies - -| Dependency | Version | Purpose | -|------------|---------|---------| -| `@openzeppelin/contracts` | 5.x | MerkleProof verification | -| `@symbioticfi/relay-contracts` | - | Settlement base contracts | -| LayerZero V2 | - | ILayerZeroDVN interface | -| Chainlink CCIP | - | CCV interfaces (ICrossChainVerifierV1) | +1. Verification requires a valid BLS quorum signature from Settlement. +2. The derived message ID must match the message payload being verified. +3. Settlement epoch data must be fresh, or verification reverts with `EpochTooStale`. diff --git a/content/symbiotic/setup.mdx b/content/symbiotic/setup.mdx index bbbba4ae..85198e11 100644 --- a/content/symbiotic/setup.mdx +++ b/content/symbiotic/setup.mdx @@ -8,17 +8,19 @@ Getting the stack running locally. - Docker and Docker Compose v2+ - [Foundry](https://book.getfoundry.sh/getting-started/installation) (`forge`, `cast`, `anvil`) -- [Rust/Cargo](https://rustup.rs/) (for `make dev-operator`) +- [pnpm](https://pnpm.io/installation) (for `make install`, which installs the contract dependencies) +- [Rust/Cargo](https://rustup.rs/) (required — `make chains`, `deploy`, `start`, `status`, `e2e`, and `validate` all run `cargo xtask`) - `jq` -## Config Structure +## Config Layout -``` +```text config/ -├── environments/ # Per-network config (local.json, testnet.json) -├── templates/ # Service config templates (oz-monitor, oz-relayer) +├── environments/ # Per-environment chain, provider, and signer config +├── keys/ # Encrypted keystores (generated by generate-signer) +├── templates/ # OZ Monitor / OZ Relayer templates ├── oz-monitor/ # Static monitor config -└── oz-relayer/ # Static relayer config and keystores +└── oz-relayer/ # Static relayer config deployments/ └── .json # Canonical deployment addresses @@ -27,55 +29,88 @@ generated/ └── / # Generated runtime config and message cache ``` -**How it works:** -1. `make deploy` deploys contracts and updates `deployments/.json` -2. `make deploy` and `make start` generate provider-specific runtime config under `generated//` -3. Docker containers mount from `generated//` +`make deploy` and `make start` read from `config/` and write runtime state into `deployments/.json` and `generated//`. + +## Signers + +Signer keys are configured in `config/environments/.json` under `signers`. Three signer types are supported: + +### Anvil (local only) + +Derives keys from Anvil's well-known mnemonic by index. Zero setup needed. -**To customize configs:** Edit templates in `config/templates/`, then rerun `make deploy` or `make start`. +```json +{ + "signers": { + "deployer": { "type": "anvil", "index": 0 }, + "operator-1": { "type": "anvil", "index": 1 } + } +} +``` -## Environment Setup +### Local (encrypted keystore) -Bootstrap local `.env` and operator keys: +Reads from an encrypted keystore file. Generate one with: ```bash -make setup +cargo xtask --env generate-signer --name deployer ``` -This generates `.env` from `.env.example` and creates BLS keys and relayer keystores. +The command prompts for a passphrase interactively (hidden input, confirmed). For CI, pass `--passphrase` explicitly. + +Then reference it in the environment config: - +```json +{ + "signers": { + "deployer": { + "type": "local", + "path": "config/keys//deployer.json", + "passphrase": { "type": "env", "value": "DEPLOYER_PASSPHRASE" } + } + } +} +``` -`make start` also auto-bootstraps `.env` and keystores if missing, so `make setup` is only needed for explicit regeneration. +### Env (environment variable) - +Reads a raw private key from an environment variable. Useful for CI/CD where keys are injected via secrets. -### Environment Variables +```json +{ + "signers": { + "deployer": { + "type": "env", + "value": "DEPLOYER_PRIVATE_KEY" + } + } +} +``` -| Variable | Description | -|----------|-------------| -| `PRIVATE_KEY` | Deployer key (default: Anvil account 0) | -| `LOG_LEVEL` | Logging level (debug, info, warn, error) | -| `WEBHOOK_SECRET` | HMAC secret for webhook auth (min 32 chars) | -| `OZ_RELAYER_WEBHOOK_SECRET` | Secret for OZ Relayer webhook auth (min 32 chars) | -| `OZ_RELAYER_API_KEY` | **Required.** Relayer API authentication | -| `SIDECAR_*_SECRET_KEYS` | BLS keys per operator (generated) | +### Relayer keystores -Generate secrets with: +The OZ Relayer needs its own encrypted keystores (`signer-1.json`, `signer-2.json`, `signer-3.json` in `config/keys//`). `make deploy` generates these automatically for any environment if they don't already exist (using `KEYSTORE_PASSPHRASE`). Generating them manually is optional — it lets you set the passphrase via an interactive prompt instead of reading it from the env file: ```bash -openssl rand -hex 32 +cargo xtask --env generate-signer --name signer-1 --name signer-2 --name signer-3 ``` +## Environment Files + +Each environment reads secrets from `.env.` (e.g. `.env.local`, `.env.testnet`). + +- **Local**: no `.env.local` needed for keys — anvil signers derive keys automatically, and relayer keystores are generated on first deploy. +- **Testnet**: copy `.env.example` to `.env.testnet` and fill in the RPC URLs and keystore passphrases (the relevant sections are commented out by default). + -The operator will fail to start if `WEBHOOK_SECRET`, `OZ_RELAYER_WEBHOOK_SECRET`, or `OZ_RELAYER_API_KEY` are missing. Secrets must be at least 32 characters. +Runtime startup requires `WEBHOOK_SECRET`, `OZ_RELAYER_WEBHOOK_SECRET`, and `OZ_RELAYER_API_KEY`. The two webhook secrets (`WEBHOOK_SECRET` and `OZ_RELAYER_WEBHOOK_SECRET`) must each be at least 32 characters. -## Provider Selection +## Select a Provider -Provider is set in `config/environments/.json`: +Set the active provider in `config/environments/.json`: ```json { @@ -83,101 +118,113 @@ Provider is set in `config/environments/.json`: } ``` -All `make` commands (`start`, `send`, `watch`, `e2e`) are provider-aware based on this field. +Supported values: +- `layerzero` +- `chainlink_ccv` -See provider-specific config in [LayerZero](/symbiotic/layerzero#configuration) or [Chainlink CCV](/symbiotic/chainlink-ccv#configuration). +Provider-specific configuration lives in [LayerZero](/symbiotic/layerzero#configuration) and [Chainlink CCV](/symbiotic/chainlink-ccv#configuration). -## Running Locally +## Environment JSON -```bash -# Start the full stack (auto-bootstrap + deploy + start services) -make start +`config/environments/.json` is the source of truth for deploy, validate, runtime rendering, and operator startup. -# Check service health -make status +### Chain Settings -# Send a test message and watch it complete -make e2e -``` +Each chain entry under `chains.source` and `chains.destination` supports: -### Common Commands +| Field | Purpose | +|-------|---------| +| `name` | Human label for logs and rendered runtime files | +| `chainId` | EVM chain ID | +| `eid` | LayerZero endpoint ID | +| `confirmations` | Confirmation depth used in rendered monitor and relayer config | +| `blockTimeMs` | Block time hint used when rendering monitor and relayer networks | +| `rpcUrls` | Non-local RPC sources; supports plain URLs or `{ "type": "env", "value": "SOURCE_RPC_URL" }` style env references | +| `predeploys` | Provider-specific or shared predeployed contract addresses | -``` -make start Start the full local stack -make deploy Deploy contracts and generate service config -make stop Stop all containers (preserve state) -make clean Full reset (stop + remove volumes + markers) - -make restart-operators Rebuild and restart all 3 operators -make restart-monitor Restart oz-monitor (config reload) -make restart-relayer Restart oz-relayer -make restart-relays Restart symbiotic-relay-1/2/3 - -make dev-operator Run operator-1 locally (cargo run) -make rebuild-operators Docker rebuild + restart all operators -make shell Interactive shell with addresses loaded - -make test Run unit tests (forge + cargo) -make test-contracts Run contract tests only - -make logs-operators Follow all 3 operator logs -make logs-operator-N Follow operator-N logs (N=1,2,3) -make logs-monitor Follow oz-monitor logs -make logs-relayer Follow oz-relayer logs -make logs-relays Follow symbiotic-relay-1/2/3 logs - -make status Show running containers and health -make help Show all available commands -``` +### Relay Timing -### Local Operator Development +The `relay` block controls settlement and genesis timing used by deploy and refresh flows: -Run operator-1 outside Docker for fast iteration: +| Field | Purpose | +|-------|---------| +| `epochDurationSeconds` | Epoch length for settlement timing | +| `slashingWindowSeconds` | Slashing window length | +| `epochStartDelaySeconds` | Delay before a new epoch becomes active | -```bash -# Start the full stack first -make start +### Monitor and Relayer Rendering -# Run operator-1 locally (replaces the Docker container) -make dev-operator +For non-local environments, these blocks are required to render runtime artifacts: + +| Block | Field | Purpose | +|-------|-------|---------| +| `ozMonitor` | `cronSchedule` | Poll cadence for the source-chain monitor | +| `ozMonitor` | `maxPastBlocks` | Backfill window for source-chain log reads | +| `ozRelayer` | `defaultSpeed` | Default submission speed tier for OZ Relayer | +| `ozRelayer` | `minBalanceWei` | Runtime threshold used by `validate` to flag low relayer-signer balances | +| `funding` | `operatorAmountWei` | Wei sent to each operator EOA during `make deploy` | +| `funding` | `signerAmountWei` | Wei sent to each explicit relayer signer during `make deploy` | +| `funding` | `minBalanceThresholdWei` | Skip funding if the target balance is already at or above this threshold. Must be ≤ `operatorAmountWei` and `signerAmountWei` (validated at config load) | + +Local environments copy static monitor and relayer config, so `ozMonitor` and `ozRelayer` mainly matter on non-local envs. `funding` is required on every env; it drives both the Solidity deploy script and the genesis-time `cast send` topups, eliminating the per-call-site defaults that previously existed for the same parameter. + +## Operator Settings + +The user-editable operator tuning lives in `config/environments/.json` under `operator`: + +```json +{ + "operator": { + "logLevel": "info", + "eventPollInterval": "15s", + "signJobInterval": "1s", + "signWorkerCount": 5, + "minBatchSize": 1, + "acceptanceHooks": [], + "enableDebugEndpoints": false + } +} ``` -This runs `cargo run` with the generated config and `RUST_LOG=debug`. The local operator connects to the same Docker services and receives the same webhooks. +If omitted, the defaults are: -## Operator Configuration +| Field | Default | Purpose | +|-------|---------|---------| +| `logLevel` | `info` | Operator log verbosity | +| `eventPollInterval` | `15s` | How often to scan for pending messages to batch | +| `signJobInterval` | `1s` | How often to retry pending signing jobs | +| `signWorkerCount` | `5` | Concurrent signing workers | +| `minBatchSize` | `1` | Minimum messages required before building a Merkle tree | +| `acceptanceHooks` | native provider hook | Ordered hooks that can accept, reject, or defer messages before batching | +| `enableDebugEndpoints` | `false` | Expose `/debug/v1/*` endpoints on operator HTTP servers | -Runtime configs are generated at `generated//operator-{n}/config.json`. +See [Acceptance Hooks](/symbiotic/acceptance-hooks) for native and webhook configuration. -Key settings: +The operator reads `config/environments/.json` directly (it is mounted into each operator container), so that file is the source of truth. Change the environment JSON, then rerun `make deploy` or `make start`. -| Setting | Default | Description | -|---------|---------|-------------| -| `signer.event_poll_interval` | 15s | How often to check for new pending messages | -| `signer.sign_job_interval` | 1s | How often to retry pending Merkle roots | -| `signer.sign_worker_count` | 5 | Concurrent signing workers | -| `signer.min_batch_size` | 1 | Minimum messages before creating a tree | -| `oz_relayer.poll_interval` | 5s | How often to check for signed trees to submit | -| `oz_relayer.status_poll_interval` | 30s | How often to poll OZ Relayer for tx status | -| `symbiotic_relay.key_tag` | 15 | BLS key identifier in the sidecar | +## Run Locally -## Contract Addresses +```bash +make install # one-time: install contract dependencies (pnpm) +make chains # start local Anvil chains +make deploy # deploy contracts +make start # start services +make status # check everything is running +make e2e # send and verify a test message +``` -After deployment, canonical addresses are in `deployments/.json`. +The workflow is split into three steps: `chains` starts Anvil, `deploy` deploys contracts, and `start` brings up services. Use `make start RESET=1` to wipe local runtime state before starting. -For manual testing, `make shell` opens an interactive shell with `.env` sourced and `ENV_CONFIG` / `DEPLOYMENTS_FILE` exported: +For faster Rust iteration, start the stack once and then run: ```bash -make shell -# Then use jq to extract addresses: -jq '.source.layerzero.dvn' $DEPLOYMENTS_FILE +make dev-operator ``` -## Service Ports +## Generated Runtime State + +- `deployments/.json` for canonical deployment addresses +- `generated//` for rendered monitor, relayer, and sidecar runtime config +- `generated//msg-cache.json` for the last message watched by xtask -| Service | Port | Purpose | -|---------|------|---------| -| anvil (source) | 8545 | Source chain RPC | -| anvil (dest) | 8546 | Destination chain RPC | -| operator-1/2/3 | 3001-3003 | Operator debug APIs | -| symbiotic-relay-1/2/3 | 8081-8083 | BLS sidecars | -| oz-relayer | 8080 | Transaction relayer | +See [CLI & API Reference](/symbiotic/cli) for command details and [Deployment](/symbiotic/deployment) for testnet operation. diff --git a/content/symbiotic/troubleshooting.mdx b/content/symbiotic/troubleshooting.mdx index f3e8f131..347616c7 100644 --- a/content/symbiotic/troubleshooting.mdx +++ b/content/symbiotic/troubleshooting.mdx @@ -2,186 +2,71 @@ title: Troubleshooting --- -Common issues and solutions for the Symbiotic multi-provider template. +Common operator-side failures and the shortest useful checks for each. -## Checking Message Status +## First Checks ```bash -# List all messages -curl http://localhost:3001/debug/v1/messages - -# Filter by status -curl "http://localhost:3001/debug/v1/messages?status=pending" - -# Get specific message -curl http://localhost:3001/debug/v1/messages/0xabc123... - -# Check pending Merkle roots -curl http://localhost:3001/debug/v1/pending -``` - -## Shared Issues - -### Webhook Not Received - -1. Check operator logs: - ```bash - make logs-operators | grep webhook - ``` -2. Verify connectivity (operators must be reachable from OZ Monitor): - ```bash - docker compose exec oz-monitor curl -s http://operator-1:3000/healthz - ``` -3. Confirm trigger is linked in the active monitor config under `generated//oz-monitor/monitors/` and rerun `make deploy` or `make start` if needed. - -### Authentication Failures (401) - -1. Verify secrets match between OZ Monitor config and operator `.env` -2. Ensure secret is at least 32 characters -3. Check for trailing whitespace in config files - -### Payload Parsing Errors - -1. Ensure `payload_mode: "raw"` is set in trigger config -2. Check operator logs for deserialization errors - -### BLS Signatures Not Aggregating - -1. Check sidecar status: - ```bash - docker compose ps symbiotic-relay-1 - docker compose logs symbiotic-relay-1 - ``` -2. Verify BLS keys in `.env` -3. Verify operator keys are registered in the Settlement contract - -### Quorum Not Reached - -1. All required operators must be running -2. Check operators are receiving the same events -3. Verify keys are registered in Settlement - -### Relayer Request Failed After All Retries - -1. Check OZ Relayer: - ```bash - curl -H "Authorization: Bearer $OZ_RELAYER_API_KEY" http://localhost:8080/api/v1/health - ``` -2. Check relayer logs: - ```bash - make logs-relayer - ``` -3. For 429 errors: increase `retry_backoff` or reduce submission rate - -### Service Won't Start - -1. Check missing env vars: - ```bash - docker compose config - ``` -2. Verify secrets are at least 32 characters -3. Check port conflicts: - ```bash - lsof -i :3001 # operator - lsof -i :8080 # relayer - ``` - -### Anvil Not Responding - -```bash -docker compose restart anvil anvil-settlement -docker compose logs anvil -``` - -### Contracts Not Found - -```bash -cat deployments/local.json -# Re-deploy if needed: -make clean && make start -``` - -### First-Run Genesis Waits - -On fresh devnet, `make start` may wait while voting power is captured. - -Symptoms: logs show `genesis not ready: totalVotingPower 0 < quorumThreshold 1`. - -This is expected on clean boot. If stuck, reset: `make clean && make start`. - -## LayerZero Issues - -### Message Stuck at "Processing" - -BLS signing not completing. Check sidecar health and logs: - -```bash -docker compose ps symbiotic-relay-1 -docker compose logs symbiotic-relay-1 +make status +make logs-operators +make logs-monitor +make logs-relayer +curl -s http://localhost:3001/debug/v1/messages +curl -s http://localhost:3001/debug/v1/pending ``` -### submitProof Reverts +## Shared Failures -Check Settlement contract state (operator keys, quorum threshold). Inspect: +### Webhook or authentication failures -```bash -make logs-relayer | grep -i "revert\|error" -``` +- Confirm operators are reachable from OZ Monitor. +- Verify `WEBHOOK_SECRET` and `OZ_RELAYER_WEBHOOK_SECRET` match the sender config exactly. +- Ensure trigger templates still use `payload_mode: "raw"`. -## Chainlink CCV Issues +### BLS signatures do not aggregate -### EpochTooStale Revert (0xf5ab0d81) +- Check sidecar health with `docker compose ps symbiotic-relay-1`. +- Check sidecar logs with `docker compose logs symbiotic-relay-1`. +- Verify operator keys are registered in Settlement and all operators saw the same ingress event. -Settlement epoch/timestamp data is stale. Relayer fails at gas estimation. +### Quorum is never reached -```bash -make logs-relayer | grep -E "estimate_gas|custom error|0xf5ab0d81" -``` +- Make sure all required operators are running. +- Confirm all operators received the same message. +- Check Settlement voting power and registered keys. -Fix: refresh genesis or tune epoch timing parameters. +### Relayer submission keeps failing -### Watch Does Not Reach Success +- Verify the relayer health endpoint responds with your API key. +- Check `make logs-relayer` for rate limits, estimate-gas failures, or permanent 4xx errors. +- For 429s, back off or use stronger RPC capacity. -CCV requires destination `MessageExecuted(messageId)`, not just relayer submission. +### Fresh local state is stuck or inconsistent -```bash -# Verify provider selection -jq -r '.activeProvider' config/environments/local.json +- If contracts, genesis, or generated config look out of sync, reset with `make clean && make deploy && make start`. +- If Anvil stopped responding, restart the relevant containers before doing a full reset. -# Check message lifecycle -make watch +## LayerZero -# If Failed, inspect relayer -make logs-relayer -``` +### `submitProof` reverts -### Submission Fails at estimate-gas +- Check that the relayer submitter address is whitelisted on the DVN. +- Check Settlement state: registered keys, quorum threshold, and current epoch assumptions. +- Search `make logs-relayer` for the revert reason before changing config. -```bash -make logs-relayer | grep -E "estimate_gas|custom error" -``` +## Chainlink CCV -Common causes: stale epoch, incorrect CCV addresses, settlement not initialized. +### `EpochTooStale` / `0xf5ab0d81` -## Log Analysis +- Settlement epoch data is stale. +- Refresh genesis or adjust the epoch timing parameters before retrying. -### Useful Commands +### `make watch` never succeeds -```bash -make logs-operators # All operator logs -make logs-operators 2>&1 | grep -i error # Errors only -make logs-operators | grep -i "retrying" # Retry activity -make logs-operators | grep "retries exhausted" -make logs-monitor # OZ Monitor -make logs-relayer # OZ Relayer -``` +- CCV success is destination `MessageExecuted(messageId)`, not just relayer submission. +- If the relayer submitted but watch still fails, inspect destination execution and relayer estimate-gas logs. -### Common Log Patterns +### Estimate-gas failures -| Pattern | Meaning | -|---------|---------| -| `webhook received` | Event received from OZ Monitor | -| `message batched` | Message added to Merkle tree | -| `signatures aggregated` | BLS quorum reached | -| `proof submitted` | Sent to OZ Relayer | -| `tx confirmed` | On-chain confirmation | +- Common causes are stale epoch data, wrong CCV addresses, or uninitialized settlement state. +- Start with `make logs-relayer | grep -E "estimate_gas|custom error"`. diff --git a/src/navigation/ethereum-evm.json b/src/navigation/ethereum-evm.json index 5644007d..f88c76ed 100644 --- a/src/navigation/ethereum-evm.json +++ b/src/navigation/ethereum-evm.json @@ -826,6 +826,11 @@ "name": "Chainlink CCV", "url": "/symbiotic/chainlink-ccv" }, + { + "type": "page", + "name": "Acceptance Hooks", + "url": "/symbiotic/acceptance-hooks" + }, { "type": "page", "name": "CLI & API Reference",