From fb3ce085272a46180f7b5b5fa0fd929a128f1a55 Mon Sep 17 00:00:00 2001 From: jsell-rh Date: Thu, 7 May 2026 18:46:41 +0000 Subject: [PATCH] spec(control-plane): add workspace init and state persistence specs Add two new spec files documenting the CP's gaps in workspace initialization (repo cloning, S3 state hydration) and ongoing state persistence (S3 sync sidecar). Both capabilities exist in the operator but are missing from the CP path, resulting in empty repo directories and ephemeral-only storage for CP-provisioned sessions. Supersedes draft PR #1477 which targeted the old file location (docs/internal/design/) before the spec reorg in #1488. Co-Authored-By: Claude Opus 4.6 --- specs/control-plane/control-plane.spec.md | 10 + specs/control-plane/state-persistence.spec.md | 187 ++++++++++++++ specs/control-plane/workspace-init.spec.md | 241 ++++++++++++++++++ 3 files changed, 438 insertions(+) create mode 100644 specs/control-plane/state-persistence.spec.md create mode 100644 specs/control-plane/workspace-init.spec.md diff --git a/specs/control-plane/control-plane.spec.md b/specs/control-plane/control-plane.spec.md index 753fc8cb1..4e3cb2680 100755 --- a/specs/control-plane/control-plane.spec.md +++ b/specs/control-plane/control-plane.spec.md @@ -100,6 +100,15 @@ The CP creates a Pod (not a Job) for each session. Key pod attributes: Each section is joined with `\n\n`. Empty sections are omitted. If all four are empty, `INITIAL_PROMPT` is not set and the runner waits for a user message via gRPC. +### Workspace Initialization and State Persistence + +Workspace setup (repo cloning, S3 state hydration) and ongoing state persistence (S3 sync sidecar) are specified in dedicated specs: + +- **[Workspace Initialization](workspace-init.spec.md)** — init container that clones repos, restores S3 state, and prepares `/workspace` before the runner starts +- **[State Persistence](state-persistence.spec.md)** — sidecar that periodically syncs workspace state to S3 and backs up git repo state + +Both are **not yet implemented** in the CP. The operator implements both patterns. Sessions created via the CP currently have ephemeral workspaces with empty repo directories. + ### Environment Variables Injected into Runner Pod | Var | Value | Purpose | @@ -116,6 +125,7 @@ Each section is joined with `\n\n`. Empty sections are omitted. If all four are | `USE_VERTEX` / `ANTHROPIC_VERTEX_PROJECT_ID` / `CLOUD_ML_REGION` | CP config | Vertex AI config (when enabled) | | `GOOGLE_APPLICATION_CREDENTIALS` | `/app/vertex/ambient-code-key.json` | Vertex service account path | | `LLM_MODEL` / `LLM_TEMPERATURE` / `LLM_MAX_TOKENS` | session fields | Per-session model config | +| `REPOS_JSON` | JSON array of `{"url","branch"}` | Repos to clone into `/workspace/repos/`. Set on the **init container** when present (see `workspace-init.spec.md`); currently set on the runner container as a stopgap | | `CREDENTIAL_IDS` | JSON map `{provider: credential_id}` | Resolved credentials for this session; runner calls `/credentials/{id}/token` per provider | --- diff --git a/specs/control-plane/state-persistence.spec.md b/specs/control-plane/state-persistence.spec.md new file mode 100644 index 000000000..e9410427f --- /dev/null +++ b/specs/control-plane/state-persistence.spec.md @@ -0,0 +1,187 @@ +# State Persistence Specification + +## Purpose + +Sessions produce workspace state (framework data, artifacts, file uploads, git repo changes) that must survive pod restarts and session resumes. The CP achieves this by adding a **state-sync sidecar** container to the runner pod that periodically uploads workspace state to S3-compatible object storage. On the next pod start, the init container (see `workspace-init.spec.md`) restores this state. The operator already implements this pattern; this spec defines the same behavior for the CP path. + +## Requirements + +### Requirement: State-Sync Sidecar Presence + +The CP SHALL add a `state-sync` sidecar container to the runner pod when S3 persistence is configured for the session's project. + +#### Scenario: Project with S3 configured + +- GIVEN a project with S3 access credentials (endpoint, bucket, access key, secret key) configured +- WHEN the CP provisions a runner pod for a session in that project +- THEN the pod spec SHALL include a `state-sync` sidecar container + +#### Scenario: Project without S3 + +- GIVEN a project with no S3 configuration +- WHEN the CP provisions a runner pod +- THEN no sidecar SHALL be added +- AND workspace state SHALL be ephemeral (lost on pod termination) + +### Requirement: Sidecar Image + +The sidecar SHALL use the same `state-sync` image as the init container (`quay.io/ambient_code/vteam_state_sync`). The image reference SHOULD be configurable via the same `STATE_SYNC_IMAGE` environment variable used for the init container. + +### Requirement: Initial Delay + +The sidecar SHALL wait 30 seconds after starting before performing its first sync cycle. This prevents syncing a partially populated workspace while the runner is still initializing. + +#### Scenario: Sidecar startup + +- GIVEN a newly started runner pod with a state-sync sidecar +- WHEN the sidecar starts +- THEN it SHALL wait 30 seconds before the first sync +- AND workspace content generated during the delay SHALL be captured in the first sync + +### Requirement: Periodic Sync + +The sidecar SHALL sync workspace state to S3 at a configurable interval (default: 60 seconds). + +Synced paths: + +| Path | Content | +|---|---| +| `/workspace//` | Framework state (e.g. `.claude/` databases, config) | +| `/workspace/artifacts/` | Session-produced artifacts | +| `/workspace/file-uploads/` | User-uploaded files | + +The sidecar SHALL NOT sync `/workspace/repos/` during periodic syncs — git repo state is handled separately via bundle backups. + +After each sync cycle, the sidecar SHALL upload a `metadata.json` file to the S3 session path containing `lastSync` timestamp, session name, namespace, and number of paths synced. + +#### Scenario: Periodic sync cycle + +- GIVEN a running session with S3 configured and `SYNC_INTERVAL=60` +- WHEN 60 seconds elapse since the last sync +- THEN the sidecar SHALL upload changed files from the synced paths to S3 +- AND the sidecar SHALL use checksum-based comparison to avoid re-uploading unchanged files + +#### Scenario: Size limit exceeded + +- GIVEN workspace content exceeding `MAX_SYNC_SIZE` (default: 1 GB) +- WHEN a sync cycle runs +- THEN the sidecar SHALL log a warning +- AND the sidecar SHALL continue syncing (best-effort, files over the limit may be skipped) + +### Requirement: Git Repo Backup + +The sidecar SHALL periodically back up git repository state to S3 at a configurable interval (default: every 5th sync cycle). For each git repository in `/workspace/repos/`: + +- A **git bundle** (`repo.bundle`) containing all refs +- An **uncommitted changes patch** (`uncommitted.patch`) +- A **staged changes patch** (`staged.patch`) +- **Metadata** (`metadata.json`) including remote URL (with credentials stripped), current branch, HEAD SHA, and local branch list + +Backups SHALL be stored at `s3://///repo-state//`. + +#### Scenario: Git repo backup cycle + +- GIVEN a session with a cloned repo at `/workspace/repos/platform` +- AND the agent has created a new branch and made uncommitted changes +- WHEN a repo backup cycle runs +- THEN the sidecar SHALL create a bundle with all refs including the new branch +- AND the sidecar SHALL capture the uncommitted changes as a patch +- AND the sidecar SHALL upload both to `repo-state/platform/` in S3 + +#### Scenario: Repo with embedded credentials in remote URL + +- GIVEN a repo cloned with `https://x-access-token:TOKEN@github.com/org/repo` +- WHEN the sidecar writes `metadata.json` +- THEN the remote URL SHALL be sanitized to `https://github.com/org/repo` +- AND the token SHALL NOT appear in any persisted metadata + +### Requirement: Graceful Shutdown + +On `SIGTERM` or `SIGINT`, the sidecar SHALL perform a final sync that includes both workspace state and a full git repo backup before exiting. This ensures state captured between the last periodic sync and pod termination is not lost. + +#### Scenario: Pod termination + +- GIVEN a running session with unsaved workspace changes +- WHEN the pod receives `SIGTERM` +- THEN the sidecar SHALL perform a final git repo backup +- AND the sidecar SHALL perform a final workspace sync +- AND the sidecar SHALL exit after both complete + +### Requirement: Sidecar Environment + +The sidecar SHALL receive the following environment variables: + +| Variable | Source | Default | +|---|---|---| +| `SESSION_NAME` | session ID | (required) | +| `NAMESPACE` | project ID | (required) | +| `S3_ENDPOINT` | project or cluster config | `http://minio.ambient-code.svc:9000` | +| `S3_BUCKET` | project or cluster config | `ambient-sessions` | +| `AWS_ACCESS_KEY_ID` | project or cluster secret | (required) | +| `AWS_SECRET_ACCESS_KEY` | project or cluster secret | (required) | +| `SYNC_INTERVAL` | CP config | `60` | +| `MAX_SYNC_SIZE` | CP config | `1073741824` (1 GB) | +| `REPO_BACKUP_INTERVAL` | CP config | `5` | +| `RUNNER_STATE_DIR` | CP config | `.claude` | + +**Note on S3 access credentials:** `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` are S3 object storage credentials, not Ambient Credential Kind entities. They are resolved from project-level or cluster-level Kubernetes Secrets. + +### Requirement: Input Sanitization + +The sidecar SHALL sanitize `SESSION_NAME` and `NAMESPACE` values to prevent path traversal in S3 paths. Only alphanumeric characters and hyphens SHALL be permitted. + +### Requirement: S3 Configuration Source + +The CP SHALL resolve S3 configuration from the session's project. The project MAY provide S3 access credentials via a Kubernetes Secret or via the api-server's project settings. When no S3 configuration is found, the CP SHALL skip both the init container's S3 hydration and the state-sync sidecar entirely — the session runs with ephemeral storage only. + +#### Scenario: Project with shared cluster S3 + +- GIVEN a project with no custom S3 config +- AND the cluster has shared MinIO credentials available +- WHEN the CP provisions a pod +- THEN the CP SHALL use the shared MinIO credentials for S3 operations + +#### Scenario: Project with custom S3 + +- GIVEN a project with a custom S3 endpoint and credentials configured +- WHEN the CP provisions a pod +- THEN the CP SHALL use the project's custom S3 config + +### Requirement: Exclude Patterns + +The sidecar SHALL exclude the following patterns from sync to avoid uploading build artifacts and caches: + +- `repos/**` (git-managed separately) +- `node_modules/**`, `.venv/**`, `__pycache__/**`, `.cache/**`, `*.pyc` +- `target/**`, `dist/**`, `build/**` +- `.git/**` +- `debug/**` (symlinks that break rclone) + +### Requirement: Volume Sharing + +The sidecar SHALL mount the same `emptyDir` volume at `/workspace` as the runner container. It SHALL also mount the framework state subdirectory (e.g., `/workspace/.claude`) at `/app/` via a subPath mount for direct access to framework databases. + +### Requirement: Security Context + +The sidecar SHALL run with a restricted security context: + +- `allowPrivilegeEscalation: false` +- `capabilities: drop: ["ALL"]` +- `readOnlyRootFilesystem: false` (required because the sidecar writes rclone config to `/tmp` and creates temporary files for git bundles) + +The sidecar does not require root privileges. Unlike the init container, it does not need `chown` access and SHOULD run as a non-root user where the pod security policy permits it. + +### Requirement: SQLite Consistency + +Before syncing framework state directories that contain SQLite databases, the sidecar SHALL issue a WAL checkpoint (`PRAGMA wal_checkpoint(TRUNCATE)`) to ensure database files are in a consistent state for upload. + +#### Scenario: Claude Code SQLite database sync + +- GIVEN the runner has an active SQLite database at `/workspace/.claude/projects.db` +- WHEN a sync cycle runs +- THEN the sidecar SHALL checkpoint the WAL before uploading +- AND the uploaded database SHALL be in a consistent, readable state + +## Status + +**Not implemented.** The CP does not create a state-sync sidecar, does not resolve S3 configuration, and does not support workspace persistence. All CP-provisioned sessions use ephemeral storage. The operator implements this in `components/operator/internal/handlers/sessions.go:1376-1419` (sidecar) and `components/operator/internal/handlers/sessions.go:2016-2090` (S3 config resolution). diff --git a/specs/control-plane/workspace-init.spec.md b/specs/control-plane/workspace-init.spec.md new file mode 100644 index 000000000..6090d2aa8 --- /dev/null +++ b/specs/control-plane/workspace-init.spec.md @@ -0,0 +1,241 @@ +# Workspace Initialization Specification + +## Purpose + +When the Control Plane provisions a runner pod, the workspace (`/workspace`) must be prepared before the runner starts. This includes creating directory structure, cloning repositories, restoring prior workspace state from object storage, and cloning workflow repositories. The CP achieves this by adding an **init container** to the pod spec that runs a hydration script. The operator already implements this pattern; this spec defines the same behavior for the CP path. + +## Requirements + +### Requirement: Init Container Presence + +The CP SHALL add an init container named `init-hydrate` to the runner pod when either: + +- The session specifies repositories (`RepoURL` or `Repos` is non-empty), OR +- S3 state persistence is configured for the session's project + +When neither condition is met, no init container SHALL be added and `/workspace` SHALL start as an empty `emptyDir` volume. + +#### Scenario: Session with a single repo URL + +- GIVEN a session where `RepoURL` is `https://github.com/org/repo` +- WHEN the CP provisions the runner pod +- THEN the pod spec SHALL include an `init-hydrate` init container +- AND the init container SHALL receive `REPOS_JSON` set to `[{"url":"https://github.com/org/repo"}]` + +#### Scenario: Session with multiple repos + +- GIVEN a session where `Repos` is `[{"url":"https://github.com/org/a","branch":"main"},{"url":"https://github.com/org/b"}]` +- WHEN the CP provisions the runner pod +- THEN the init container SHALL receive `REPOS_JSON` set to the value of `Repos` verbatim + +#### Scenario: Both RepoURL and Repos are set + +- GIVEN a session where both `RepoURL` and `Repos` are non-empty +- WHEN the CP provisions the runner pod +- THEN `Repos` SHALL take precedence +- AND `RepoURL` SHALL be ignored + +#### Scenario: No repos and no S3 configured + +- GIVEN a session with no `RepoURL`, no `Repos`, and no S3 persistence configured +- WHEN the CP provisions the runner pod +- THEN no init container SHALL be added + +#### Scenario: No repos but S3 is configured + +- GIVEN a session with no repos but S3 persistence is configured +- WHEN the CP provisions the runner pod +- THEN an `init-hydrate` init container SHALL be added +- AND the init container SHALL restore prior workspace state from S3 (if any exists) + +### Requirement: Init Container Image + +The init container SHALL use the same `state-sync` image used by the operator (`quay.io/ambient_code/vteam_state_sync`). The image reference SHOULD be configurable via a `STATE_SYNC_IMAGE` environment variable on the CP deployment, which MUST be added to the CP's `KubeReconcilerConfig`. + +#### Scenario: Image configuration + +- GIVEN the CP deployment has `STATE_SYNC_IMAGE=quay.io/ambient_code/vteam_state_sync:v2` +- WHEN the CP provisions a pod with an init container +- THEN the init container image SHALL be `quay.io/ambient_code/vteam_state_sync:v2` + +### Requirement: Init Container Authentication + +The init container SHALL authenticate to the api-server using the CP token endpoint (`GET /token`), not a pre-injected `BOT_TOKEN`. The init container calls the CP's token endpoint using the pod's mounted Kubernetes service account token (at `/var/run/secrets/kubernetes.io/serviceaccount/token`), the same mechanism the runner container uses. + +The CP SHALL inject `AMBIENT_CP_TOKEN_URL` into the init container environment. The `hydrate.sh` script SHALL check `AMBIENT_CP_TOKEN_URL` first and obtain a bearer token from the CP before calling the backend API for git credentials. When `AMBIENT_CP_TOKEN_URL` is not set (operator path), `hydrate.sh` SHALL fall back to `BOT_TOKEN` for backward compatibility. + +**Implementation note:** `hydrate.sh` does not yet support `AMBIENT_CP_TOKEN_URL` — it currently uses `BOT_TOKEN` only. The script must be updated to implement this requirement (see Status section). + +#### Scenario: Init container obtains token from CP endpoint + +- GIVEN a CP-provisioned pod with `AMBIENT_CP_TOKEN_URL` set +- WHEN `hydrate.sh` needs to fetch git credentials from the backend API +- THEN it SHALL call `GET ` with the SA token in the `Authorization` header +- AND it SHALL use the returned bearer token for subsequent backend API calls + +#### Scenario: Operator-provisioned pod with BOT_TOKEN + +- GIVEN an operator-provisioned pod with `BOT_TOKEN` set and no `AMBIENT_CP_TOKEN_URL` +- WHEN `hydrate.sh` needs to fetch git credentials +- THEN it SHALL use `BOT_TOKEN` directly (backward-compatible path) + +### Requirement: Init Container Environment + +The init container SHALL receive the following environment variables: + +| Variable | Source | Required | +|---|---|---| +| `SESSION_NAME` | session ID | always | +| `NAMESPACE` | project ID | always | +| `PROJECT_NAME` | project ID | always | +| `BACKEND_API_URL` | CP config | always | +| `AMBIENT_CP_TOKEN_URL` | CP config | always (CP path) | +| `REPOS_JSON` | session model | when repos specified | +| `ACTIVE_WORKFLOW_GIT_URL` | Workflow resource (resolved from `session.WorkflowID`) | when workflow specified | +| `ACTIVE_WORKFLOW_BRANCH` | Workflow resource | when workflow specified | +| `ACTIVE_WORKFLOW_PATH` | Workflow resource | when workflow specified | +| `S3_ENDPOINT` | project or cluster config | when S3 configured | +| `S3_BUCKET` | project or cluster config | when S3 configured | +| `AWS_ACCESS_KEY_ID` | project or cluster secret | when S3 configured | +| `AWS_SECRET_ACCESS_KEY` | project or cluster secret | when S3 configured | +| `RUNNER_STATE_DIR` | CP config | always (default: `.claude`) | + +**Note on workflow fields:** The session model has `WorkflowID` (a foreign key), not the git URL/branch/path directly. The CP MUST resolve `session.WorkflowID` by fetching the Workflow resource from the api-server and extracting its `gitUrl`, `branch`, and `path` fields. + +**Note on S3 access credentials:** `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` are S3 object storage credentials, not Ambient Credential Kind entities. They are resolved from project-level or cluster-level Kubernetes Secrets, not from the `CREDENTIAL_IDS` credential flow. + +### Requirement: Input Sanitization + +The init container SHALL sanitize `SESSION_NAME` and `NAMESPACE` values to prevent path traversal in S3 paths and local filesystem operations. Only alphanumeric characters and hyphens SHALL be permitted. + +### Requirement: Volume Sharing + +The init container and runner container SHALL share a single `emptyDir` volume mounted at `/workspace`. The init container writes to this volume; the runner reads from it after the init container exits. + +#### Scenario: Workspace volume lifecycle + +- GIVEN an init container and runner container in the same pod +- WHEN the init container clones a repo into `/workspace/repos/myrepo` +- THEN the runner container SHALL see `/workspace/repos/myrepo` with the cloned content + +### Requirement: Directory Structure + +The init container SHALL create the following directories before any clone or restore operation: + +- `/workspace/repos/` +- `/workspace/artifacts/` +- `/workspace/file-uploads/` +- `/workspace//` (default: `/workspace/.claude/`) +- `/workspace//debug/` (when `RUNNER_STATE_DIR` is `.claude`) + +Ownership SHALL be set to UID 1001 (runner user). Permissions on `/workspace/repos/` and `/workspace/file-uploads/` SHALL be world-writable (0777) because the init container and runner container may run as different UIDs. `/workspace/artifacts/` SHALL be 0755. + +### Requirement: Repo URL Normalization + +The CP SHALL normalize the session's repo specification into the `REPOS_JSON` format consumed by `hydrate.sh`: + +- `session.RepoURL` (single string) SHALL be converted to `[{"url":""}]` +- `session.Repos` (JSON array string) SHALL be passed through as-is +- If both are set, `Repos` SHALL take precedence + +**Note:** The CP currently only handles `session.RepoURL`. Support for `session.Repos` (multi-repo array) MUST be added — it is a field on the session model (`types.Session.Repos`) and the api-server data model but is not read by the CP today. + +#### Scenario: RepoURL normalization + +- GIVEN a session with `RepoURL = "https://github.com/org/repo"` +- WHEN the CP builds the init container env +- THEN `REPOS_JSON` SHALL equal `[{"url":"https://github.com/org/repo"}]` + +### Requirement: Git Credential Fetch for Private Repos + +The init container SHALL fetch git credentials at runtime by calling the backend API, not by receiving pre-injected `GITHUB_TOKEN` or `GITLAB_TOKEN` environment variables. The `hydrate.sh` script uses `BACKEND_API_URL` and its bearer token to call the credentials endpoint for each provider: + +``` +GET /projects//agentic-sessions//credentials/github +GET /projects//agentic-sessions//credentials/gitlab +``` + +If a provider credential is available, the script sets the corresponding environment variable (`GITHUB_TOKEN` or `GITLAB_TOKEN`) internally and configures a git credential helper that returns these tokens for matching hosts. + +If no credentials are available or the fetch fails, `hydrate.sh` SHALL log a distinct warning (distinguishing auth failure from clone failure) and continue — public repos will clone successfully, private repos will fail with a non-fatal warning. + +#### Scenario: Private GitHub repo with credentials + +- GIVEN a session with a `github` credential resolved for this project +- AND `RepoURL` is `https://github.com/org/private-repo` +- WHEN the init container runs +- THEN `hydrate.sh` SHALL fetch `GITHUB_TOKEN` from the backend API at runtime +- AND the clone SHALL succeed using the git credential helper + +#### Scenario: Private repo without credentials + +- GIVEN a session with no git credentials configured +- AND `RepoURL` is `https://github.com/org/private-repo` +- WHEN the init container runs +- THEN the credential fetch SHALL return empty +- AND the clone SHALL fail with a warning +- AND the init container SHALL exit with code 0 (clone failures are non-fatal; a non-zero exit would prevent the pod from starting) + +### Requirement: S3 Workspace State Hydration + +When S3 access credentials are provided, the init container SHALL check for existing workspace state in S3 at the path `s3://///` and restore it in two phases: + +**Phase 1 — Before repo cloning:** +- Framework state (`/`) to `/workspace//` +- Artifacts to `/workspace/artifacts/` +- File uploads to `/workspace/file-uploads/` + +**Phase 2 — After repo cloning:** +- Git repo state from `repo-state//` — restore bundles, checkout saved branch, apply uncommitted and staged patches + +This two-phase ordering is required because git state patches can only be applied to cloned repositories. + +#### Scenario: Resuming a session with S3 state + +- GIVEN a session that previously ran and synced workspace state to S3 +- AND S3 access credentials are configured +- WHEN the init container runs for a new pod +- THEN framework state, artifacts, and file uploads SHALL be restored first (Phase 1) +- AND repos SHALL be cloned from their remote URLs +- AND git branch, uncommitted changes, and staged changes SHALL be re-applied from S3 backup (Phase 2) + +#### Scenario: First run with no S3 state + +- GIVEN a session running for the first time +- AND S3 access credentials are configured +- WHEN the init container checks S3 +- THEN no state SHALL be found +- AND the init container SHALL proceed to clone repos and create directories normally + +### Requirement: Workflow Cloning + +When `ACTIVE_WORKFLOW_GIT_URL` is set, the init container SHALL clone the workflow repository into `/workspace/workflows/`. If `ACTIVE_WORKFLOW_PATH` is set, only the specified subdirectory SHALL be extracted to the target path. + +#### Scenario: Workflow with subpath + +- GIVEN `ACTIVE_WORKFLOW_GIT_URL = "https://github.com/org/my-workflows"` and `ACTIVE_WORKFLOW_PATH = "session-setup"` +- WHEN the init container runs +- THEN only the `session-setup/` subdirectory SHALL appear at `/workspace/workflows/my-workflows/` + +#### Scenario: Workflow subpath not found + +- GIVEN a workflow with `ACTIVE_WORKFLOW_PATH` pointing to a non-existent subdirectory +- WHEN the init container runs +- THEN the init container SHALL log a warning +- AND SHALL fall back to using the entire cloned repository + +### Requirement: Security Context + +The init container SHALL run with a restricted security context: + +- `allowPrivilegeEscalation: false` +- `capabilities: drop: ["ALL"]` +- `readOnlyRootFilesystem: false` (required because `hydrate.sh` writes rclone config to `/tmp` and creates workspace directories) + +The init container runs as root (not `runAsNonRoot`) because it must set ownership (`chown`) of workspace directories to UID 1001 before the runner container starts. This is an intentional exception to the platform's `runAsNonRoot` convention. + +## Status + +**Not implemented.** The CP currently sets `REPOS_JSON` as an env var on the runner container (for `RepoURL` only — `Repos` is not handled) but does not create an init container. Repos appear as empty directories at `/workspace/repos//`. The operator implements this correctly in `reconcileSpecReposWithPatch` (`components/operator/internal/handlers/sessions.go:944-1015`). The CP implementation should add the same `init-hydrate` container in `ensurePod` (`components/ambient-control-plane/internal/reconciler/kube_reconciler.go:394-484`). + +Additionally, `hydrate.sh` (`components/runners/state-sync/hydrate.sh`) must be updated to support the CP token endpoint (`AMBIENT_CP_TOKEN_URL`) as the primary authentication mechanism, falling back to `BOT_TOKEN` for operator compatibility.