You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
feat(sandbox): log connection attempts that bypass proxy path (#326)
* feat(sandbox): log connection attempts that bypass proxy path
Add iptables LOG + REJECT rules inside the sandbox network namespace to
detect and diagnose direct connection attempts that bypass the HTTP
CONNECT proxy. This provides two improvements:
1. Fast-fail UX: applications get immediate ECONNREFUSED instead of a
30-second timeout when they bypass the proxy
2. Diagnostics: a /dev/kmsg monitor emits structured BYPASS_DETECT
tracing events with destination, protocol, process identity, and
actionable hints
Both TCP and UDP bypass attempts are covered (UDP catches DNS bypass).
The feature degrades gracefully if iptables or /dev/kmsg are unavailable.
Closes#268
* chore: track .python-version to pin Python 3.13.12 for uv
The sandbox base image runs Python 3.13. A stale venv on 3.12 causes
all exec_python E2E tests to fail because cloudpickle bytecode is not
compatible across minor versions.
* fix(cluster): preserve hostGatewayIP across fast deploys
The fast deploy's helm upgrade was missing the hostGatewayIP value that
the bootstrap entrypoint injects into the HelmChart CR. This caused
host.openshell.internal hostAliases to be lost from the gateway pod
and sandbox pods after any fast deploy, breaking host gateway routing.
Read the IP from the HelmChart CR and pass it through to helm upgrade.
* wip: fix iptables path resolution, use dmesg for kmsg, add CAP_SYSLOG
* fix(sandbox): restore NetworkNamespace Drop impl, remove dead kmsg code
The Drop impl for NetworkNamespace was accidentally deleted during the
bypass detection refactor, which would cause network namespaces and
veth interfaces to leak on every sandbox shutdown.
Also removes dead kmsg volume/mount code (bypass monitor uses dmesg
instead of direct /dev/kmsg access) and removes an accidentally
committed session transcript file.
Copy file name to clipboardExpand all lines: architecture/sandbox-custom-containers.md
+2-1Lines changed: 2 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -98,7 +98,7 @@ The `openshell-sandbox` supervisor adapts to arbitrary environments:
98
98
99
99
-**Log file fallback**: Attempts to open `/var/log/openshell.log` for append; silently falls back to stdout-only logging if the path is not writable.
100
100
-**Command resolution**: Executes the command from CLI args, then the `OPENSHELL_SANDBOX_COMMAND` env var (set to `sleep infinity` by the server), then `/bin/bash` as a last resort.
101
-
-**Network namespace**: Requires successful namespace creation for proxy isolation; startup fails in proxy mode if required capabilities (`CAP_NET_ADMIN`, `CAP_SYS_ADMIN`) or `iproute2` are unavailable.
101
+
-**Network namespace**: Requires successful namespace creation for proxy isolation; startup fails in proxy mode if required capabilities (`CAP_NET_ADMIN`, `CAP_SYS_ADMIN`) or `iproute2` are unavailable. If the `iptables` package is present, the supervisor installs OUTPUT chain rules (LOG + REJECT) inside the namespace to provide fast-fail behavior (immediate `ECONNREFUSED` instead of a 30-second timeout) and diagnostic logging when processes attempt direct connections that bypass the HTTP CONNECT proxy. If `iptables` is absent, the supervisor logs a warning and continues — core network isolation still works via routing.
102
102
103
103
## Design Decisions
104
104
@@ -114,6 +114,7 @@ The `openshell-sandbox` supervisor adapts to arbitrary environments:
114
114
| Clear `run_as_user/group` for custom images | Prevents startup failure when the image lacks the default `sandbox` user |
115
115
| Non-fatal log file init |`/var/log/openshell.log` may be unwritable in arbitrary images; falls back to stdout |
116
116
|`docker save` / `ctr import` for push | Avoids requiring a registry for local dev; images land directly in the k3s containerd store |
117
+
| Optional `iptables` for bypass detection | Core network isolation works via routing alone (`iproute2`); `iptables` only adds fast-fail (`ECONNREFUSED`) and diagnostic LOG entries. Making it optional avoids hard failures in minimal images that lack `iptables` while giving better UX when it is available. |
|`denial_aggregator.rs`|`DenialAggregator` background task -- receives `DenialEvent`s from the proxy, deduplicates by `(host, port, binary)`, drains on flush interval |
22
+
|`denial_aggregator.rs`|`DenialAggregator` background task -- receives `DenialEvent`s from the proxy and bypass monitor, deduplicates by `(host, port, binary)`, drains on flush interval |
23
23
|`mechanistic_mapper.rs`| Deterministic policy recommendation generator -- converts denial summaries to `PolicyChunk` proposals with confidence scores, rationale, and SSRF/private-IP detection |
24
24
|`sandbox/mod.rs`| Platform abstraction -- dispatches to Linux or no-op |
25
25
|`sandbox/linux/mod.rs`| Linux composition: Landlock then seccomp |
26
26
|`sandbox/linux/landlock.rs`| Filesystem isolation via Landlock LSM (ABI V1) |
27
27
|`sandbox/linux/seccomp.rs`| Syscall filtering via BPF on `SYS_socket`|
28
-
|`sandbox/linux/netns.rs`| Network namespace creation, veth pair setup, cleanup on drop |
28
+
|`bypass_monitor.rs`| Background `/dev/kmsg` reader for iptables bypass detection events |
29
+
|`sandbox/linux/netns.rs`| Network namespace creation, veth pair setup, bypass detection iptables rules, cleanup on drop |
The network namespace routes all sandbox traffic through the veth pair, but a misconfigured process that ignores proxy environment variables can still attempt direct connections to the veth gateway IP or other addresses. Bypass detection catches these attempts, providing two benefits: immediate connection failure (fast-fail UX) instead of a 30-second TCP timeout, and structured diagnostic logging that identifies the offending process.
487
+
488
+
##### iptables rules
489
+
490
+
`install_bypass_rules()` installs OUTPUT chain rules inside the sandbox network namespace using `iptables` (IPv4) and `ip6tables` (IPv6, best-effort). Rules are installed via `ip netns exec {namespace} iptables ...`. The rules are evaluated in order:
491
+
492
+
| # | Rule | Target | Purpose |
493
+
|---|------|--------|---------|
494
+
| 1 |`-d {host_ip}/32 -p tcp --dport {proxy_port}`|`ACCEPT`| Allow traffic to the proxy |
495
+
| 2 |`-o lo`|`ACCEPT`| Allow loopback traffic |
496
+
| 3 |`-m conntrack --ctstate ESTABLISHED,RELATED`|`ACCEPT`| Allow response packets for established connections |
The LOG rules use the `--log-uid` flag to include the UID of the process that initiated the connection. The log prefix `openshell:bypass:{namespace_name}:` enables the bypass monitor to filter `/dev/kmsg` for events belonging to a specific sandbox.
503
+
504
+
The proxy port defaults to `3128` unless the policy specifies a different `http_addr`. IPv6 rules mirror the IPv4 rules via `ip6tables`; IPv6 rule installation failure is non-fatal (logged as warning) since IPv4 is the primary path.
505
+
506
+
**Graceful degradation:** If iptables is not available (checked via `which iptables`), a warning is logged and rule installation is skipped entirely. The network namespace still provides isolation via routing — processes can only reach the proxy's IP, but without bypass rules they get a timeout rather than an immediate rejection. LOG rule failure is also non-fatal — if the `xt_LOG` kernel module is not loaded, the REJECT rules are still installed for fast-fail behavior.
507
+
508
+
##### /dev/kmsg monitor
509
+
510
+
`bypass_monitor::spawn()` starts a background tokio task (via `spawn_blocking`) that reads kernel log messages from `/dev/kmsg`. The monitor:
511
+
512
+
1. Opens `/dev/kmsg` in read mode and seeks to end (skips historical messages)
513
+
2. Reads lines via `BufReader`, filtering for the namespace-specific prefix `openshell:bypass:{namespace_name}:`
514
+
3. Parses iptables LOG format via `parse_kmsg_line()`, extracting `DST`, `DPT`, `SPT`, `PROTO`, and `UID` fields
515
+
4. Resolves process identity for TCP events via `procfs::resolve_tcp_peer_identity()` (best-effort — requires a valid entrypoint PID and non-zero source port)
516
+
5. Emits a structured `tracing::warn!()` event with the tag `BYPASS_DETECT`
517
+
6. Sends a `DenialEvent` to the denial aggregator channel (if available)
518
+
519
+
The `BypassEvent` struct holds the parsed fields:
520
+
521
+
```rust
522
+
pubstructBypassEvent {
523
+
pubdst_addr:String, // Destination IP address
524
+
pubdst_port:u16, // Destination port
525
+
pubsrc_port:u16, // Source port (for process identity resolution)
526
+
pubproto:String, // "tcp" or "udp"
527
+
pubuid:Option<u32>, // UID from --log-uid (if present)
528
+
}
529
+
```
530
+
531
+
##### BYPASS_DETECT tracing event
532
+
533
+
Each detected bypass attempt emits a `warn!()` log line with the following structured fields:
534
+
535
+
| Field | Type | Description |
536
+
|-------|------|-------------|
537
+
|`dst_addr`| string | Destination IP address |
538
+
|`dst_port`| u16 | Destination port |
539
+
|`proto`| string |`"tcp"` or `"udp"`|
540
+
|`binary`| string | Binary path of the offending process (or `"-"` if unresolved) |
541
+
|`binary_pid`| string | PID of the offending process (or `"-"`) |
|`hint`| string | Context-specific remediation hint (see below) |
546
+
547
+
The `hint` field provides actionable guidance:
548
+
549
+
| Condition | Hint |
550
+
|-----------|------|
551
+
| UDP + port 53 |`"DNS queries should route through the sandbox proxy; check resolver configuration"`|
552
+
| UDP (other) |`"UDP traffic must route through the sandbox proxy"`|
553
+
| TCP |`"ensure process honors HTTP_PROXY/HTTPS_PROXY; for Node.js set NODE_USE_ENV_PROXY=1"`|
554
+
555
+
Process identity resolution is best-effort and TCP-only. For UDP events or when the entrypoint PID is not yet set (PID == 0), the binary, PID, and ancestors fields are reported as `"-"`.
556
+
557
+
##### DenialEvent integration
558
+
559
+
Each bypass event sends a `DenialEvent` to the denial aggregator with `denial_stage: "bypass"`. This integrates bypass detections into the same deduplication, aggregation, and policy proposal pipeline as proxy-level denials. The `DenialEvent` fields:
560
+
561
+
| Field | Value |
562
+
|-------|-------|
563
+
|`host`| Destination IP address |
564
+
|`port`| Destination port |
565
+
|`binary`| Binary path (or `"-"`) |
566
+
|`ancestors`| Ancestor chain parsed from `" -> "` separator |
The denial aggregator deduplicates bypass events by the same `(host, port, binary)` key used for proxy denials, and flushes them to the gateway via `SubmitPolicyAnalysis` on the same interval.
573
+
574
+
##### Lifecycle wiring
575
+
576
+
The bypass detection subsystem is wired in `crates/openshell-sandbox/src/lib.rs`:
577
+
578
+
1. After `NetworkNamespace::create()` succeeds, `install_bypass_rules(proxy_port)` is called. Failure is non-fatal (logged as warning).
579
+
2. The proxy's denial channel sender (`denial_tx`) is cloned as `bypass_denial_tx` before being passed to the proxy.
580
+
3. After proxy startup, `bypass_monitor::spawn()` is called with the namespace name, entrypoint PID, and `bypass_denial_tx`. Returns `Option<JoinHandle>` — `None` if `/dev/kmsg` is unavailable.
581
+
582
+
The monitor runs for the lifetime of the sandbox. It exits when `/dev/kmsg` reaches EOF (process termination) or encounters an unrecoverable read error.
583
+
584
+
**Graceful degradation:** If `/dev/kmsg` cannot be opened (e.g., restricted container environment without access to the kernel ring buffer), the monitor logs a one-time warning and returns `None`. The iptables REJECT rules still provide fast-fail UX — the monitor only adds diagnostic visibility.
585
+
586
+
##### Dependencies
587
+
588
+
Bypass detection requires the `iptables` package for rule installation (in addition to `iproute2` for namespace management). If iptables is not installed, bypass detection degrades to routing-only isolation. The `/dev/kmsg` device is required for the monitor but not for the REJECT rules.
|`CAP_SYS_PTRACE`| Proxy reading `/proc/<pid>/fd/` and `/proc/<pid>/exe` for processes running as a different user |
485
597
486
-
The `iproute2` package must be installed (provides the `ip` command).
598
+
The `iproute2` package must be installed (provides the `ip` command). The `iptables` package is required for bypass detection rules; if absent, the namespace still provides routing-based isolation but without fast-fail rejection or diagnostic logging for bypass attempts.
487
599
488
600
If namespace creation fails (e.g., missing capabilities), startup fails in `Proxy` mode. This preserves fail-closed behavior: either network namespace isolation is active, or the sandbox does not run.
489
601
@@ -1087,6 +1199,12 @@ The sandbox uses `miette` for error reporting and `thiserror` for typed errors.
| OPA engine Mutex lock poisoned | Error on the individual evaluation |
@@ -1125,8 +1243,9 @@ Dual-output logging is configured in `main.rs`:
1125
1243
1126
1244
Key structured log events:
1127
1245
-`CONNECT`: One per proxy CONNECT request (for non-`inference.local` targets) with full identity context. Inference interception failures produce a separate `info!()` log with `action=deny` and the denial reason.
1246
+
-`BYPASS_DETECT`: One per detected direct connection attempt that bypassed the HTTP CONNECT proxy. Includes destination, protocol, process identity (best-effort), and remediation hint. Emitted at `warn` level.
1128
1247
-`L7_REQUEST`: One per L7-inspected request with method, path, and decision
1129
-
- Sandbox lifecycle events: process start, exit, namespace creation/cleanup
0 commit comments