Experiment with Cloudflare devcontainer cache backends

[simple-agent-manager]

Summary

Implements the production Cloudflare managed Containers Registry devcontainer cache path on top of the experiment branch.

• Mints short-lived Cloudflare registry credentials in the API/control plane.
• Passes registry host, username, password, and cache ref to VM-agent workspace bootstrap.
• Keeps the existing GHCR/no-cache fallback when Cloudflare registry config is absent.
• Avoids installing Wrangler on VM-agent nodes.
• Keeps registry passwords out of logs and persisted workspace metadata.

Experiment evidence already collected on this PR branch:

• Real SAM devcontainer stress test passed against Cloudflare managed registry.
• Image size: 2,741,386,134 bytes / 2,614.4 MiB.
• Successful run: https://github.com/raphaeltm/simple-agent-manager/actions/runs/25672922644
• Push/pull phase took about 85 seconds.

Validation

• pnpm --filter @simple-agent-manager/shared build
• pnpm --filter @simple-agent-manager/providers build
• pnpm --filter @simple-agent-manager/cloud-init build
• pnpm --filter @simple-agent-manager/api test -- tests/unit/services/devcontainer-cache.test.ts
• pnpm --filter @simple-agent-manager/api typecheck
• pnpm --filter @simple-agent-manager/api build
• pnpm --filter @simple-agent-manager/api lint (passed with existing warnings, 0 errors)
• pnpm --filter @simple-agent-manager/api exec eslint src/services/devcontainer-cache.ts tests/unit/services/devcontainer-cache.test.ts src/durable-objects/task-runner/workspace-steps.ts src/services/node-agent.ts
• go test ./internal/config ./internal/cache ./internal/server
• go test ./... in packages/vm-agent
• git diff --check

Staging Verification (REQUIRED for all code changes — merge-blocking)

• Staging deployment green — not run in this session
• Live app verified via Playwright — not run in this session
• Existing workflows confirmed working — not run in this session
• New feature/fix verified on staging — not run in this session
• Infrastructure verification completed — not run in this session
• Mobile and desktop verification notes added for UI changes — N/A: no UI changes

Staging Verification Evidence

Not completed in this session. The PR must not be merged until staging deployment and infrastructure verification are completed by a human or a follow-up agent run with the required staging credentials and infrastructure access.

UI Compliance Checklist (Required for UI changes)

N/A: no UI changes.

End-to-End Verification (Required for multi-component changes)

• Data flow traced from user input to final outcome with code path citations.
• Capability test exercises the complete happy path across system boundaries.
• All spec/doc assumptions about existing behavior verified against code.
• If any gap exists between automated test coverage and full E2E, manual verification steps documented below.

Data Flow Trace

1. TaskRunner creates a full devcontainer workspace in apps/api/src/durable-objects/task-runner/workspace-steps.ts and calls getDevcontainerCacheConfig() before createWorkspaceOnNode().
2. apps/api/src/services/devcontainer-cache.ts reads Worker env, builds a Cloudflare registry ref from the GitHub repo/config path, and mints short-lived registry credentials through Cloudflare's registry credentials API.
3. apps/api/src/services/node-agent.ts serializes devcontainerCache into the VM-agent /workspaces request body.
4. packages/vm-agent/internal/server/workspaces.go accepts the optional cache credentials and stores them in runtime-only workspace state.
5. packages/vm-agent/internal/server/workspace_provisioning.go applies those credentials to bootstrap config for that workspace only.
6. packages/vm-agent/internal/bootstrap/bootstrap.go logs Docker into the provided registry and uses the explicit cache ref; if explicit Cloudflare config is absent, it falls back to the prior GitHub-token/GHCR behavior.

Untested Gaps

Automated coverage verifies API credential request construction, fallback behavior, ref construction, response validation, Go config parsing, explicit cache ref precedence, and redacted Docker login errors. Full live staging verification with real Cloudflare credentials was not run in this session.

Post-Mortem (Required for bug fix PRs)

N/A: not a bug fix.

Specialist Review Evidence (Required for agent-authored PRs)

• All dispatched reviewers completed and findings addressed before merge
• If any reviewer did NOT complete: needs-human-review label added and merge deferred to human — N/A: all local specialist reviews completed

Reviewer	Status	Outcome
security-auditor	PASS	Confirmed short-lived credentials are minted in the API plane, secrets are not persisted in workspace metadata, and Docker login errors redact supplied passwords.
cloudflare-specialist	PASS	Confirmed Cloudflare registry config is Worker-side, credential TTL/account/repo settings are configurable, and VM nodes do not require Wrangler.
env-validator	PASS	Confirmed API and VM-agent env vars are documented consistently across env examples, wrangler vars, self-hosting docs, and deploy secret mapping.
constitution-validator	PASS	Confirmed account IDs, repository prefix, registry host, credential TTL, API token, and timeouts are configurable instead of hardcoded.
go-specialist	PASS	Confirmed VM-agent request/config/runtime wiring keeps credentials in memory and preserves existing fallback behavior.
doc-sync-validator	PASS	Confirmed environment and self-hosting documentation were updated for the new production cache path.
test-engineer	PASS	Confirmed focused API and Go tests cover the new credential and fallback behavior.
task-completion-validator	PASS	Confirmed task checklist, implementation, docs, and validation evidence match the requested scope.

Exceptions (If any)

• Scope: Staging deployment/live infra verification was not completed in this session.
• Rationale: The user requested implementation and push to the existing PR, with no merge. The live Cloudflare registry experiment evidence already exists on this PR branch; production staging verification remains merge-blocking.
• Expiration: Before PR Experiment with Cloudflare devcontainer cache backends #963 is merged.

Agent Preflight (Required)

• Preflight completed before code changes

Classification

• external-api-change
• cross-component-change
• business-logic-change
• public-surface-change
• docs-sync-change
• security-sensitive-change
• ui-change
• infra-change

External References

Official documentation and prior experiment evidence were consulted before coding:

• Cloudflare Containers Registry documentation: https://developers.cloudflare.com/containers/container-registries/
• Cloudflare API Token permissions documentation: https://developers.cloudflare.com/fundamentals/api/get-started/create-token/
• Existing experiment handoff: docs/notes/2026-05-11-cloudflare-devcontainer-cache-handoff.md
• Existing experiment report: docs/notes/2026-05-11-cloudflare-devcontainer-cache-experiments.md

Codebase Impact Analysis

This change spans apps/api TaskRunner workspace provisioning, apps/api/src/services Cloudflare credential minting, packages/vm-agent workspace creation/bootstrap/cache behavior, scripts/deploy secret mapping, and docs/env references. The cache path is full-devcontainer only; lightweight workspace behavior is unchanged.

Documentation & Specs

Updated apps/api/.env.example, apps/api/wrangler.toml, packages/vm-agent/.env.example, scripts/deploy/configure-secrets.sh, docs/guides/self-hosting.md, docs/architecture/secrets-taxonomy.md, .claude/skills/env-reference/SKILL.md, and the Cloudflare devcontainer cache experiment notes.

Constitution & Risk Check

Checked Principle XI/no-hardcoded-values and the credential-safety rules. Cloudflare account ID, API token, registry host, repository prefix, credential TTL, and API timeout are configurable; registry passwords are not logged or persisted; missing Cloudflare config preserves fallback behavior.

[simple-agent-manager]

Experiment Results

The Cloudflare cache backend experiments completed successfully in GitHub Actions.

• Cloudflare managed Containers Registry worked with wrangler containers push and plain docker push / docker pull against registry.cloudflare.com.
• R2 Docker tarball storage worked: docker save, R2 upload/download, docker load, and docker run all succeeded.
• BuildKit S3 cache to R2 also worked with a docker-container Buildx builder: cache import/export completed against a temporary R2 bucket.

Recommendation: use Cloudflare managed Containers Registry for the production iteration because it preserves the VM agent’s existing registry-oriented docker pull, cacheFrom, docker tag, and docker push flow. Keep R2 BuildKit cache as a future option only if SAM later owns the Buildx/devcontainer build invocation directly.

Detailed report: docs/notes/2026-05-11-cloudflare-devcontainer-cache-experiments.md

Successful experiment run: https://github.com/raphaeltm/simple-agent-manager/actions/runs/25664066831

[simple-agent-manager]

SAM devcontainer stress test completed successfully against Cloudflare managed Containers Registry.

Run: https://github.com/raphaeltm/simple-agent-manager/actions/runs/25672922644

Key results:

• Real SAM devcontainer image size: 2,741,386,134 bytes / 2,614.4 MiB
• Full job duration: 4m18s
• Build phase: about 2m25s
• Push/pull phase: about 85s from push start to successful pull
• Digest: sha256:baeb7e14758e5b4284cd7b9b2faec8e736ed97fd1c37b153614ce06306cfc07e

This confirms Cloudflare managed registry handles the real SAM devcontainer image with plain Docker push/pull, not just the earlier 64 MiB synthetic image. I also documented this in docs/notes/2026-05-11-cloudflare-devcontainer-cache-experiments.md.

[sonarqubecloud]

Quality Gate passed

Issues
5 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.2% Duplication on New Code

See analysis details on SonarQube Cloud

[simple-agent-manager]

Staging validation completed for PR #963 on 2026-05-11.\n\nDeploy evidence:\n- deploy-staging.yml run 25687373433 succeeded on branch sam/cloudflare-devcontainer-cache-experiments-01krb4.\n- Staging API health returned healthy after deploy.\n- app.sammy.party dashboard/projects/settings returned HTTP 200 with the smoke-test session cookie.\n\nFeature validation:\n- Project: CrewAI (01KJNR9R3TEN3KX1ETE33852R8), repository serverspresentation2025/crewai, workspaceProfile=full.\n- First fresh node 01KRC349V91MW4S8959TVV0Z7B / workspace 01KRC387GS8HHM52YJHS3WT97F: debug package showed cold-cache behavior. The VM agent checked registry.cloudflare.com/c4e4aebd980b626f6af43ac6b1edcede/sam-serverspresentation2025-crewai:devcontainer-cache, found no manifest, built from scratch, marked the workspace running, then logged Cache image pushed successfully at 2026-05-11T18:06:15Z.\n- After stopping the session, the first node/workspace were deleted and confirmed deleted in D1.\n- Second fresh node 01KRC3MDN3ZFMCEX7V1G39NEWR / workspace 01KRC3RA3ED0KAAX2FK1TZW13N: debug package showed warm-cache behavior. The VM agent logged Cache image pulled successfully, Cache hit: pulled devcontainer cache image, and wrote the devcontainer override with cacheFrom=registry.cloudflare.com/c4e4aebd980b626f6af43ac6b1edcede/sam-serverspresentation2025-crewai:devcontainer-cache. Workspace reached running, then cache image pushed successfully again.\n- Second session/node/workspace were stopped/deleted and confirmed deleted in D1.\n\nFollow-up bugs filed in backlog while validating:\n- tasks/backlog/2026-05-11-conversation-task-callback-auth.md: conversation-mode task completion callback returned 401 after agent exit.\n- tasks/backlog/2026-05-11-debug-package-tar-write-too-long.md: debug package generation logged archive/tar: write too long for syslog.log.\n\nVerdict: the Cloudflare managed Containers Registry devcontainer cache path works on staging across a fresh-node cycle.