Skip to content

fix: ship CA bundle in layered image for kms-client TLS#159

Open
mpjunior92 wants to merge 2 commits into
masterfrom
fix/layered-image-ca-bundle
Open

fix: ship CA bundle in layered image for kms-client TLS#159
mpjunior92 wants to merge 2 commits into
masterfrom
fix/layered-image-ca-bundle

Conversation

@mpjunior92
Copy link
Copy Markdown
Contributor

@mpjunior92 mpjunior92 commented May 19, 2026

Summary

The kms-client binary copied into the layered image makes HTTPS calls to userapi-compute*.eigencloud.xyz/v2/attestation, but the layered image only inherits whatever CA store the user's base image happens to ship. Minimal bases (alpine without ca-certificates, scratch, distroless, custom slim images) lack one, causing kms-client to fail the TLS handshake with x509: certificate signed by unknown authority. Attestation upload is silently abandoned (29 retries, then logged as ERROR but non-fatal); the workload still starts but its attestation never reaches the user API and the verify dashboard shows "No attestations available."

Confirmed live: a vTPM app on mainnet (0x751847d2c430e3b5a843d017ea8d0a3d453377d0) — built with this CLI/SDK — has 36 releases, status=Running, but v2_count=0. Boot logs show 30 retries against the userapi /v2/attestation URL all failing with x509: certificate signed by unknown authority.

Parity with the compute-tee and ecloud-platform fixes landing in parallel.

Fix

  • Copy alpine 3.20.10's CA bundle to /usr/local/share/eigenx-ca-certs.crt in the layered image (non-standard path, does NOT touch the user's /etc/ssl/).
  • Prefix the kms-client invocation in compute-source-env.sh with SSL_CERT_FILE= pointing at that path. The env-var override is scoped to the single kms-client process — the user's runtime never sees it.
  • Alpine pinned to 3.20.10 for build determinism.

Files changed

  • packages/sdk/src/client/common/templates/Dockerfile.layered.tmpl
  • packages/sdk/src/client/common/templates/compute-source-env.sh.tmpl

Test plan

  • Build a layered image from a base without ca-certificates (e.g. alpine with no extras, or scratch/distroless) and confirm /usr/local/share/eigenx-ca-certs.crt is present. (verified locally — see comment below)
  • Confirm the user's /etc/ssl/ is unchanged inside the running container, and SSL_CERT_FILE is not set in the user workload's environment. (verified locally — see comment below)
  • Existing layered-image builds with bases that already have ca-certificates continue to work unchanged. (verified locally — see comment below)

The kms-client binary copied into the layered image makes HTTPS calls
to userapi-compute*.eigencloud.xyz/v2/attestation, but the layered
image only inherits whatever CA store the user's base image happens
to ship. Minimal bases (alpine without ca-certificates, scratch,
distroless, custom slim images) lack one, causing kms-client to fail
the TLS handshake with x509: certificate signed by unknown authority.
Attestation upload is silently abandoned (29 retries, then logged as
ERROR but non-fatal); the workload still starts but its attestation
never reaches the user API and the verify dashboard shows "No
attestations available."

Parity with the compute-tee and ecloud-platform fixes landing in
parallel.

Fix: copy alpine 3.20.10's CA bundle to /usr/local/share/eigenx-ca-certs.crt
in the layered image, and prefix the kms-client invocation in
compute-source-env.sh with SSL_CERT_FILE pointing at that path. The
user's /etc/ssl/ is never touched; the env-var override is scoped to
the single kms-client process.

Alpine pinned to 3.20.10 for reproducibility.
@mpjunior92 mpjunior92 self-assigned this May 19, 2026
@mpjunior92
Copy link
Copy Markdown
Contributor Author

Local end-to-end validation

Built a real Go binary using the same net/http stack as kms-client (statically linked ELF amd64), placed it inside a layered image rendered from this repo's actual templates, on top of a scratch+stripped-alpine base where /etc/ssl is absent. Replicates the worst-case production scenario.

Test matrix

invocation Go HTTPS POST result
patched path (this PR) SSL_CERT_FILE=/usr/local/share/eigenx-ca-certs.crt /usr/local/bin/kms-client HTTP 422 — TLS validated, server parsed body, entrypoint exits 0
bug repro (same image, env var unset) /usr/local/bin/kms-client tls: failed to verify certificate: x509: certificate signed by unknown authority

The error string in the bug repro is identical to what's in mainnet/sepolia-prod kms-client boot logs for affected vTPM apps.

Confirms

  • CA bundle lands at /usr/local/share/eigenx-ca-certs.crt (217 KB Mozilla bundle from alpine:3.20.10).
  • User's /etc/ssl/ is never touched — SSL_CERT_FILE is process-scoped to the kms-client invocation only.
  • Patch is necessary AND sufficient: failure → success on the same binary by toggling the env var alone.
  • Works on the most hostile possible base (no /etc/ssl/ at all). Distroless / alpine-without-ca-certificates / slim user images are strict subsets.

Go's SSL_CERT_FILE honoring is a crypto/x509 stdlib guarantee (root_unix.go), so the real kms-client behaves identically to the test binary.

Earlier validation layers (also passing)

  • Step 1: alpine + curl + stripped CAs reproduces and fixes via SSL_CERT_FILE.
  • Step 2: this repo's rendered template + curl on a no-CA derivative.

Real sepolia-dev redeploy still recommended as a final smoke test post-merge but no longer load-bearing.

Per code review: the layered Dockerfile comment says the bundle is for
both kms-client and tls-client, but only the kms-client invocation got
the SSL_CERT_FILE prefix. tls-client/tls-keygen makes ACME calls and
HTTPS callbacks to the user API for cert persistence, which would hit
the same x509 failure on minimal user base images. Apply the same
process-scoped override to that invocation.
@mpjunior92 mpjunior92 requested a review from mcclurejt May 21, 2026 18:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants