Remove Docker socket access from API container

[las7]

Problem

The API container mounts /var/run/docker.sock, which grants root-equivalent control over the host if the API is compromised.

Proposed change

• Split execution into a dedicated executor service (RPC-based).
• API server submits job specs; executor launches containers.
• Executor runs with minimal permissions and strict allowlists.

Acceptance criteria

• API container no longer mounts docker.sock.
• Executor enforces image allowlist.
• Executor disallows privileged containers.
• Executor disallows arbitrary mounts.

[las7]

Proposal: Separate Executor Service with gRPC

Problem

The API container currently mounts /var/run/docker.sock, which grants root-equivalent control over the host if the API is compromised.

Proposed Architecture

Split the system into two services:

┌─────────────────┐                    ┌─────────────────┐
│  API Server     │   gRPC/RPC        │  Executor       │
│  (tako-vm)     │  ───────────────► │  Service        │
│                 │                    │                 │
│  - HTTP API     │   Job Request     │  - Docker CLI   │
│  - Auth         │   ◄────────────── │  - Policy       │
│  - Queue        │   Job Result      │    Enforcement  │
│                 │                    │  - Container    │
└─────────────────┘                    │    Lifecycle    │
                                       └─────────────────┘

Communication Protocol: gRPC

Reasons for gRPC:

• Type-safe - Contract-first API with protobuf
• Streaming - Support for real-time execution updates
• Performance - HTTP/2, binary serialization
• Language-agnostic - Can generate clients for any language

Executor Service Responsibilities

1. Job Execution

   • Receive job specs via gRPC
   • Spawn Docker containers
   • Stream logs back to API
   • Return results

2. Policy Enforcement (mandatory)

   • Image allowlist (only pre-approved images)
   • Block privileged containers (--privileged)
   • Block host mounts (-v /host:/container)
   • Block dangerous capabilities
   • Resource limits enforcement

3. Isolation

   • No Docker socket access for API container
   • Executor runs with minimal permissions
   • All container configs validated against allowlists

gRPC Service Definition

service Executor {
  // Execute a job in an isolated container
  rpc Execute(ExecuteRequest) returns (ExecuteResponse);
  
  // Stream execution logs in real-time
  rpc StreamLogs(LogRequest) returns (stream LogResponse);
  
  // Cancel a running job
  rpc CancelJob(CancelRequest) returns (CancelResponse);
}

message ExecuteRequest {
  string code = 1;
  string image = 2;
  repeated string requirements = 3;
  int32 timeout_seconds = 4;
  Resources resources = 5;
}

message ExecuteResponse {
  string execution_id = 1;
  JobStatus status = 2;
  string stdout = 3;
  string stderr = 4;
  int32 exit_code = 5;
}

Implementation Phases

1. Phase 1: Define gRPC contract, create executor service skeleton
2. Phase 2: Move Docker execution logic to executor
3. Phase 3: Implement policy enforcement (allowlist, restrictions)
4. Phase 4: Update API to use gRPC instead of direct Docker
5. Phase 5: Update docker-compose, remove socket mount

Security Benefits

• API container: No Docker access at all
• Executor: Validates all container configs before running
• Defense in depth: Even if executor compromised, policies limit damage
• Auditability: All execution requests logged

Questions for Discussion

1. Should the executor be a separate Docker container or process?
2. Should we use Redis for job queuing instead of gRPC?
3. How should we handle pre-built images (image allowlist source)?
4. What's the expected latency SLA for job execution?

---

CC @las7 - Would love feedback on this approach before starting implementation.

[las7]

Additional Concerns

API Compatibility

The existing API should remain unchanged for users. The internal architecture change is transparent:

Users → HTTP API (unchanged) → [Internal: gRPC → Executor → Docker]

Users don't see the gRPC layer - it's an internal implementation detail. The API still accepts the same HTTP requests and returns the same responses.

gRPC Authentication

Several options for securing gRPC:

1. mTLS (Mutual TLS) - Both client and server present certificates

   # Server verifies client certificate
   # Client verifies server certificate
   # Strongest option but complex to set up

2. Token-based auth - Simple API key or JWT

   # API sends: metadata={"authorization": "Bearer token"}
   # Executor validates token before processing

3. Firewall/network isolation - Executor only accessible from API container

   # In docker-compose:
   executor:
     networks: [internal_network]
   api:
     networks: [internal_network]
   # No external access to executor

Recommended approach: Start with network isolation (docker-compose internal network) + simple token. mTLS can be added later.

Minimal Attack Surface

With this architecture:

• API has NO Docker access (solved!)
• Executor validates ALL requests against policies
• Network isolation: Only API can talk to executor
• Even if API is compromised, attacker still needs to get past executor policies

This is a significant security improvement over the current docker.sock approach.

[las7]

Yes, They're Complementary

These security measures stack, not replace each other:

┌─────────────────────────────────────────────────────┐
│                  Defense in Depth                   │
├─────────────────────────────────────────────────────┤
│                                                     │
│  Layer 1: Network Isolation                         │
│  ├─ Internal Docker network                         │
│  └─ Executor not exposed externally                │
│                                                     │
│  Layer 2: Token Auth                               │
│  ├─ API sends token with each request              │
│  ├─ Executor validates token                       │
│  └─ Even if network breached, need valid token    │
│                                                     │
│  Layer 3: mTLS (optional, future)                  │
│  ├─ Certificate-based mutual auth                  │
│  └─ Even if token leaked, need valid certificates │
│                                                     │
│  Layer 4: Policy Enforcement                       │
│  ├─ Image allowlist                                │
│  ├─ No privileged containers                       │
│  └─ Resource limits enforced                        │
│                                                     │
└─────────────────────────────────────────────────────┘

Recommended approach for MVP: Start with network isolation + token. Add mTLS later if needed.

[las7]

Interim Mitigations (before full executor service)

The gRPC executor split is the right long-term fix, but there are lighter-weight options that can reduce risk now:

Current situation

• Library mode (Sandbox): Server runs on host, calls docker run via subprocess. No socket sharing — no issue here.
• Server mode via docker-compose: tako-vm-server container mounts docker.sock. This is where the risk lives.

Option 1: Docker socket proxy (lowest effort)

Drop-in fix using Tecnativa/docker-socket-proxy. Whitelists specific Docker API endpoints; blocks everything else.

services:
  docker-proxy:
    image: tecnativa/docker-socket-proxy
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    environment:
      CONTAINERS: 1    # allow container create/start/stop/inspect
      IMAGES: 1        # allow image inspect
      POST: 1          # allow POST requests
      # everything else denied by default (no exec, no volumes, no privileged)
    networks:
      - internal

  tako-vm:
    environment:
      - DOCKER_HOST=tcp://docker-proxy:2375
    # remove docker.sock volume mount entirely
    networks:
      - internal
      - default

The server can only create/inspect/stop containers. It cannot:

• Mount host paths arbitrarily
• Create privileged containers
• Access the host network namespace
• docker exec into running containers

Option 2: Run server on host, only executors in Docker

Skip containerizing the server entirely. tako-vm server runs as a host process (which already has Docker access natively). Only executor containers are spawned. No socket sharing needed.

This is already the default path for pip install "tako-vm[server]" && tako-vm server.

Option 3: Rootless Docker / Podman

Run the Docker daemon in rootless mode or switch to Podman. Socket compromise gives non-root access only. Requires host-level configuration change.

Option 4: Docker API over TCP with mTLS

No socket mount. Server connects to tcp://docker-host:2376 with mutual TLS certificates. More operational complexity but eliminates the socket entirely.

---

Recommendation

Short-term: Option 1 (socket proxy) for anyone deploying via docker-compose. Minimal code changes — just update docker-compose.yaml and document DOCKER_HOST env var support.

Long-term: The gRPC executor service as already proposed in this issue. The socket proxy can serve as the bridge until that's built.