Skip to content

OPERATIONS.md

Latest commit

 

History

History
182 lines (115 loc) · 5.76 KB

OPERATIONS.md

File metadata and controls

182 lines (115 loc) · 5.76 KB

Operations Runbook

This runbook describes behavior that is implemented by the current repository.

For public paste-and-verify receipt verification, use VerifyAgent: https://github.com/commandlayer/verifyagent

Boundary reminder: this runtime executes verbs/actions and produces signed CommandLayer receipts; it does not ship the public verifier UI/demo experience.

Minimum deployment inputs

A normal boot requires all of the following unless DEV_AUTO_KEYS=1 is used for development:

  • RECEIPT_SIGNER_ID or a supported signer-id alias
  • RECEIPT_SIGNING_PRIVATE_KEY_PEM_B64 or a supported private-key alias
  • RECEIPT_SIGNING_PUBLIC_KEY_B64 or a supported public-key alias

The current .env.example uses:

  • RECEIPT_SIGNER_ID
  • RECEIPT_SIGNING_PRIVATE_KEY_PEM_B64
  • RECEIPT_SIGNING_PUBLIC_KEY_B64

If these values are missing or invalid, server.mjs exits during boot.

Health and route checks

The current server implements:

  • GET /
  • GET /health
  • GET /healthz
  • POST /verify
  • POST /<verb>/v1.1.0 for enabled verbs

It does not implement GET /ready or GET /version.

When callers omit execution on a commons verb request, the runtime fabricates entry: "https://runtime.commandlayer.org/execute", the live verb, version: "1.1.0", and class: "commons" before signing the receipt. Commercial/payment-aware behavior belongs in the separate commercial runtime and is intentionally out of scope here.

Basic checks

curl -s "$BASE_URL/health" | jq .
curl -s "$BASE_URL/" | jq .

Expected health indicators:

  • ok: true
  • signer_ok: true
  • verifier_ok: true when either a local public key is loaded or ETH_RPC_URL is configured
  • expected enabled_verbs

Debug route access

Debug routes are disabled unless both of these are set:

  • ENABLE_DEBUG=1
  • DEBUG_TOKEN=<token>

Use either header form:

curl -s "$BASE_URL/debug/env" -H "X-Debug-Token: $DEBUG_TOKEN" | jq .

or:

curl -s "$BASE_URL/debug/env" -H "Authorization: Bearer $DEBUG_TOKEN" | jq .

If debug access is not enabled correctly, the server returns 404 rather than 401 or 403.

Verification and schema warmup

POST /verify supports these operationally relevant query flags:

  • ens=1
  • strict_kid=1
  • refresh=1
  • schema=1

Schema warmup flow

When VERIFY_SCHEMA_CACHED_ONLY=1, schema verification for a cold verb returns 202 and queues warmup work. Schema validation targets the verb receipt schema under SCHEMA_HOST/schemas/v1.1.0/....

You can prewarm validators through the implemented debug route:

curl -s -X POST "$BASE_URL/debug/prewarm" \
  -H "content-type: application/json" \
  -H "X-Debug-Token: $DEBUG_TOKEN" \
  -d '{"verbs":["fetch","parse","summarize","classify"]}' | jq .

curl -s "$BASE_URL/debug/validators" \
  -H "X-Debug-Token: $DEBUG_TOKEN" | jq .

Warm state is per process. There is no shared cache across replicas.

ENS verification operations

For live ens=1 verification, the server needs:

  • ETH_RPC_URL
  • a signer ENS name in the receipt proof, or a configured runtime signer id fallback
  • TXT records for the configured keys, defaulting to:
    • cl.sig.pub
    • cl.sig.kid
    • cl.sig.canonical

Important current behavior:

  • the runtime resolves TXT records directly on the signer ENS name
  • cl.sig.kid is optional unless the request uses strict_kid=1
  • there is no VERIFIER_ENS_NAME override in the current implementation
  • there is no cl.receipt.signer delegation step in server.mjs

Troubleshooting

Boot exits with signer misconfiguration

Check that the configured private key is PKCS#8 Ed25519 PEM material and that the configured public key matches it.

The quickest built-in check is:

curl -s "$BASE_URL/health" | jq .

If the process is up but signing is broken, signer_errors in /health will show the validation errors accumulated at boot.

/verify returns no public key available

This means neither of these paths is available:

  • a local configured public key
  • ens=1 with a successful ENS lookup

For local verification, use RECEIPT_SIGNING_PUBLIC_KEY_B64 or a supported alias. RECEIPT_SIGNING_PUBLIC_KEY is not a live server variable.

/verify?schema=1 returns HTTP 202

This is expected when:

  • VERIFY_SCHEMA_CACHED_ONLY=1, and
  • the validator for that verb is not warmed yet.

Use /debug/prewarm, retry later, or disable cached-only mode.

/verify returns HTTP 502

This indicates a transient verify availability failure, not proof invalidity. The runtime emits failure_type: "availability", retryable: true, and a retry-oriented message so callers can distinguish service timeouts from cryptographic failures.

Given the current server implementation, this points more strongly to process stall/crash or an upstream proxy timeout than to cold schema loading alone, because cold validator loading under cached-only mode returns HTTP 202 instead of timing out. If the request used ens=1&refresh=1 or schema=1 with cached-only mode disabled, blocked upstream fetches or ENS/schema slowness are also plausible contributors.

ENS verification fails before signature checks

Check:

  • ETH_RPC_URL
  • signer ENS resolution
  • presence and format of cl.sig.pub
  • presence of cl.sig.canonical
  • optional cl.sig.kid if you are using strict_kid=1

fetch failures

The fetch verb is constrained by:

  • ENABLE_SSRF_GUARD
  • ALLOW_FETCH_HOSTS
  • FETCH_TIMEOUT_MS
  • FETCH_MAX_BYTES

Blocked localhost, private-network, or disallowed-host requests fail before the outbound fetch completes.

CI reality in this repo

The checked-in GitHub workflows currently run Node-based checks and a scheduled/manual ENS smoke script.

There is no repo-local Python verification test in the npm-based CI workflow defined in .github/workflows/ci.yml.