feat(platform): cleanup api surface area and mtls flows (!39)

Closes NVIDIA#48, NVIDIA#52

## Summary
- Replace the envoy-gateway-based TLS setup with inline PKI generation during cluster bootstrap, generating CA, server, and client certificates directly in the `navigator-bootstrap` crate
- Remove all envoy gateway Helm templates (`gateway.yaml`, `gatewayclass.yaml`, `grpcroute.yaml`, PKI job, traffic policies) and the `Dockerfile.pki-job`
- Add native mTLS support to the navigator server with `tokio-rustls`, mounting client TLS certs as volumes into sandbox pods
- Update cluster entrypoint, healthcheck, and deploy scripts to work with the new direct-TLS architecture
- Add TLS security e2e test and fix formatting/clippy warnings

## Test Plan
- All unit tests pass (`cargo test --workspace`)
- Clippy clean (`cargo clippy --workspace --all-targets`)
- Format clean (`cargo fmt --all -- --check`)
- Python tests pass (`uv run pytest python/`)
- Full `mise run pre-commit` passes

Author: drew

.agent/skills/debug-navigator-cluster/SKILL.md

@@ -18,17 +18,20 @@ Diagnose why a navigator cluster failed to start after `nav cluster admin deploy
 5. Wait for k3s to generate kubeconfig (up to 60s)
 6. **Clean stale nodes**: Remove any `NotReady` k3s nodes left over from previous container instances that reused the same persistent volume
 7. **Prepare local images** (if `NAVIGATOR_PUSH_IMAGES` is set): In `internal` registry mode, bootstrap waits for the in-cluster registry and pushes tagged images there. In `external` mode, bootstrap uses legacy `ctr -n k8s.io images import` push-mode behavior.
-8. Wait for cluster health checks to pass (up to 6 min):
+7. **Reconcile TLS PKI**: Load existing TLS secrets from the cluster; if missing, incomplete, or malformed, generate fresh PKI (CA + server + client certs). Apply secrets to cluster. If rotation happened and the navigator workload is already running, rollout restart and wait for completion (failed rollout aborts deploy).
+8. **Store CLI mTLS credentials**: Persist client cert/key/CA locally for CLI authentication.
+9. Wait for cluster health checks to pass (up to 6 min):
    - k3s API server readiness (`/readyz`)
     - `navigator` statefulset ready in `navigator` namespace
    - `navigator-gateway` Gateway programmed in `navigator` namespace
-   - If TLS enabled: `navigator-cli-client` secret exists with cert data
-9. Extract mTLS credentials if TLS is enabled (up to 3 min)
+   - TLS secrets `navigator-server-tls` and `navigator-client-tls` exist
 
 For local deploys, metadata endpoint selection now depends on Docker connectivity:
 
-- default local Docker socket (`unix:///var/run/docker.sock`): `https://127.0.0.1`
-- TCP Docker daemon (`DOCKER_HOST=tcp://<host>:<port>`): `https://<host>` for non-loopback hosts
+- default local Docker socket (`unix:///var/run/docker.sock`): `https://127.0.0.1:{port}` (default port 8080)
+- TCP Docker daemon (`DOCKER_HOST=tcp://<host>:<port>`): `https://<host>:{port}` for non-loopback hosts
+
+The host port is configurable via `--port` on `nav cluster admin deploy` (default 8080) and is stored in `ClusterMetadata.gateway_port`.
 
 The TCP host is also added as an extra gateway TLS SAN so mTLS hostname validation succeeds.
 
@@ -161,23 +164,17 @@ The Envoy Gateway provides HTTP/gRPC ingress:
 # Gateway status
 docker exec navigator-cluster-<name> sh -lc 'KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n navigator get gateway/navigator-gateway'
 
-# Envoy Gateway system pods
-docker exec navigator-cluster-<name> sh -lc 'KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n envoy-gateway-system get pods -o wide'
-
-# Envoy Gateway Helm install job
-docker exec navigator-cluster-<name> sh -lc 'KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n kube-system logs -l job-name=helm-install-envoy-gateway --tail=200'
-
 # Check port bindings on the host
 docker port navigator-cluster-<name>
 ```
 
-Expected ports: `6443/tcp`, `80/tcp`, `443/tcp`, `8080/tcp` (mapped to host 30051).
+Expected ports: `6443/tcp`, `30051/tcp` (mapped to configurable host port, default 8080; set via `--port` on deploy).
 
 If ports are missing or conflicting, another process may be using them. Check with:
 
 ```bash
 # On the host (or remote host)
-ss -tlnp | grep -E ':(6443|80|443|30051)\s'
+ss -tlnp | grep -E ':(6443|8080)\s'
 ```
 
 If using Docker-in-Docker (`DOCKER_HOST=tcp://docker:2375`), verify metadata points at `https://docker` (not `https://127.0.0.1`).
@@ -223,20 +220,25 @@ If `registries.yaml` is missing or has wrong values, verify env wiring (`NAVIGAT
 
 ### Step 7: Check mTLS / PKI
 
-If TLS is enabled, the health check requires the `navigator-cli-client` secret:
+TLS certificates are generated by the `navigator-bootstrap` crate (using `rcgen`) and stored as K8s secrets before the Helm release installs. There is no PKI job or cert-manager — certificates are applied directly via `kubectl apply`.
 
 ```bash
-# Check if the secret exists
-docker exec navigator-cluster-<name> sh -lc 'KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n navigator get secret navigator-cli-client'
+# Check if the three TLS secrets exist
+docker exec navigator-cluster-<name> sh -lc 'KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n navigator get secret navigator-server-tls navigator-server-client-ca navigator-client-tls'
 
-# PKI job logs (this job creates the certificates)
-docker exec navigator-cluster-<name> sh -lc 'KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n navigator logs -l job-name=navigator-gateway-pki --tail=200'
+# Inspect server cert expiry (if openssl is available in the container)
+docker exec navigator-cluster-<name> sh -lc 'KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n navigator get secret navigator-server-tls -o jsonpath="{.data.tls\.crt}" | base64 -d | openssl x509 -noout -dates 2>/dev/null || echo "openssl not available"'
 
-# Check cert-manager pods (PKI depends on cert-manager)
-docker exec navigator-cluster-<name> sh -lc 'KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n cert-manager get pods'
+# Check if CLI-side mTLS files exist locally
+ls -la ~/.config/navigator/clusters/<name>/mtls/
 ```
 
-The PKI job often fails due to DNS issues or registry auth problems (it needs to pull its image from the distribution registry). If the job failed, check registry config (Step 6) and DNS (Step 9).
+On redeploy, bootstrap reuses existing secrets if they are valid PEM. If secrets are missing or malformed, fresh PKI is generated and the navigator workload is automatically restarted. If the rollout restart fails after rotation, the deploy aborts and CLI-side certs are not updated. Certificates use rcgen defaults (effectively never expire).
+
+Common mTLS issues:
+- **Secrets missing**: The `navigator` namespace may not have been created yet (Helm controller race). Bootstrap waits up to 2 minutes for the namespace.
+- **mTLS mismatch after manual secret deletion**: Delete all three secrets and redeploy — bootstrap will regenerate and restart the workload.
+- **CLI can't connect after redeploy**: Check that `~/.config/navigator/clusters/<name>/mtls/` contains `ca.crt`, `tls.crt`, `tls.key` and that they were updated at deploy time.
 
 ### Step 8: Check Kubernetes Events
 
@@ -293,11 +295,12 @@ If DNS is broken, all image pulls from the distribution registry will fail, as w
 | Image import fails (`k3s ctr` exit code != 0) | Corrupt tar stream or containerd not ready | Retry after k3s is fully started; check container logs |
 | Push mode images not found by kubelet | Imported into wrong containerd namespace | Must use `k3s ctr -n k8s.io images import`, not `k3s ctr images import` |
 | Gateway not `Programmed` | Envoy Gateway not ready | Check `envoy-gateway-system` pods and Helm install logs |
-| mTLS secret missing | PKI job failed (often DNS) | Check PKI job logs and DNS resolution (Step 8) |
+| mTLS secrets missing | Bootstrap couldn't apply secrets (namespace not ready, kubectl exec failure) | Check deploy logs and verify `navigator` namespace exists (Step 7) |
+| mTLS mismatch after redeploy | PKI rotated but workload not restarted, or rollout failed | Check that all three TLS secrets exist and that the navigator pod restarted after cert rotation (Step 7) |
 | Helm install job failed | Chart values error or dependency issue | Check `helm-install-navigator` job logs in `kube-system` |
 | Architecture mismatch (remote) | Built on arm64, deploying to amd64 | Cross-build the image for the target architecture |
 | SSH connection failed (remote) | SSH key/host/Docker issues | Test `ssh <host> docker ps` manually |
-| Port conflict | Another service on 6443/80/443/30051 | Stop conflicting service or change port mapping |
+| Port conflict | Another service on 6443 or the configured gateway host port (default 8080) | Stop conflicting service or use `--port` to pick a different host port |
 | gRPC connect refused to `127.0.0.1:443` in CI | Docker daemon is remote (`DOCKER_HOST=tcp://...`) but metadata still points to loopback | Verify metadata endpoint host matches `DOCKER_HOST` and includes non-loopback host |
 | DNS failures inside container | Entrypoint DNS detection failed | Check `/etc/rancher/k3s/resolv.conf` and container startup logs |
 | `metrics-server` errors in logs | Normal k3s noise, not the root cause | These errors are benign — look for the actual failing health check component |

.env.example

@@ -0,0 +1,21 @@
+# Navigator local development environment
+# Copy to .env and customise. Mise loads .env automatically.
+#
+# To run multiple Navigator clusters concurrently, give each worktree (or
+# checkout) a unique CLUSTER_NAME and GATEWAY_PORT.  For example:
+#
+#   Worktree A (.env):  CLUSTER_NAME=nav-a   GATEWAY_PORT=8080
+#   Worktree B (.env):  CLUSTER_NAME=nav-b   GATEWAY_PORT=8090
+
+# ---------- Cluster identity ----------
+
+# Name used for the Docker container, k3s volume, TLS secrets, and the
+# navigator CLI's active-cluster bookmark.  Defaults to the repo directory
+# basename (e.g. "navigator-c").
+#CLUSTER_NAME=navigator-c
+
+# ---------- Ports ----------
+
+# Host port mapped to the k3s NodePort (30051) where the Navigator gateway
+# listens.  The CLI connects here.  Must be unique per cluster.
+#GATEWAY_PORT=8080

CONTRIBUTING.md

@@ -209,7 +209,6 @@ mise run cluster          # Build and deploy local k3s cluster with Navigator
 mise run cluster:deploy   # Fast deploy: rebuild changed components and skip unnecessary helm work
 mise run cluster:push:server    # Push local server image to configured pull registry
 mise run cluster:push:sandbox   # Push local sandbox image to configured pull registry
-mise run cluster:push:pki-job   # Push local pki-job image to configured pull registry
 mise run cluster:deploy:pull    # Force full pull-mode deploy flow
 mise run cluster:push           # Legacy image-import fallback workflow
 ```

Cargo.lock

@@ -286,6 +286,7 @@ This opens an interactive SSH session into the sandbox, with all provider creden
 |---|---|
 | [Cluster Bootstrap](cluster-single-node.md) | How the platform bootstraps a Kubernetes cluster from a single Docker container, for local and remote targets. |
 | [Gateway Architecture](gateway.md) | The control plane gateway: API multiplexing, gRPC services, persistence, TLS, and sandbox orchestration. |
+| [Gateway Security](gateway-security.md) | mTLS enforcement, PKI bootstrap, certificate hierarchy, and the gateway trust model. |
 | [Sandbox Architecture](sandbox.md) | The sandbox execution environment: policy enforcement, Landlock, seccomp, network namespaces, and the network proxy. |
 | [Container Management](build-containers.md) | Container images, Dockerfiles, Helm charts, build tasks, and CI/CD. |
 | [Sandbox Connect](sandbox-connect.md) | SSH tunneling into sandboxes through the gateway. |