fix(podman): avoid host-gateway on macOS machines (#1637)

Closes #1307

Default the Podman host gateway alias override to gvproxy's host-loopback IP on macOS while preserving host-gateway resolution on Linux. Wire the setting through Podman config, gateway TOML inheritance, and the standalone driver, and document the platform behavior.

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>

Author: TaylorMutch

architecture/gateway.md

@@ -367,6 +367,10 @@ table.
 - Docker-backed local gateways use Docker's `host-gateway` callback alias on
   macOS and Docker Desktop-style runtimes. Native Linux Docker may expose an
   additional bridge-gateway listener because the host can bind that bridge IP.
+- Podman-backed macOS gateways use gvproxy's host-loopback IP for sandbox host
+  aliases by default so stale Podman machine images do not need Podman's
+  `host-gateway` resolver. Linux Podman keeps the resolver unless
+  `host_gateway_ip` is configured.
 - Gateway restarts recover persisted objects from storage, but live relay
   streams must be re-established by supervisors.
 - User-facing behavior changes must update published docs in `docs/`; this file

crates/openshell-driver-podman/NETWORKING.md

@@ -223,15 +223,22 @@ The container spec configures:
 - `networks` to attach to the configured bridge, `openshell` by default.
 - `portmappings` with `host_port: 0`, `container_port: 2222`, and `protocol:
   "tcp"` to publish the SSH compatibility port on an ephemeral host port.
-- `hostadd` entries for `host.containers.internal:host-gateway` and
-  `host.openshell.internal:host-gateway`.
+- `hostadd` entries for `host.containers.internal` and
+  `host.openshell.internal`, using Podman's `host-gateway` resolver or the
+  configured `host_gateway_ip`.
 
 Pasta is not explicitly configured by the driver. The driver requests bridge
 mode and logs the network backend that Podman reports at startup.
 
 The `host.containers.internal` hostname is injected into `/etc/hosts` so the
-supervisor can reach the gateway on the host. If `OPENSHELL_GRPC_ENDPOINT` is
-empty, the driver auto-detects:
+supervisor can reach the gateway on the host. Linux defaults to
+`host-gateway`; macOS Podman machine defaults to `192.168.127.254`, gvproxy's
+host-loopback IP, because older Podman machine images can fail to resolve
+`host-gateway`. Override this with `host_gateway_ip` or
+`OPENSHELL_PODMAN_HOST_GATEWAY_IP` when a Podman machine uses a non-standard
+host-loopback address.
+
+If `OPENSHELL_GRPC_ENDPOINT` is empty, the driver auto-detects:
 
 ```rust
 if config.grpc_endpoint.is_empty() {

crates/openshell-driver-podman/README.md

@@ -167,9 +167,11 @@ Key points:
 - Port publishing: the container spec still requests `host_port: 0` for the
   configured SSH port. The gateway SSH tunnel uses the supervisor relay rather
   than connecting directly to the published port.
-- Host gateway: `host.containers.internal:host-gateway` and
-  `host.openshell.internal:host-gateway` in `/etc/hosts` allow containers to
-  reach services on the gateway host.
+- Host gateway: `host.containers.internal` and `host.openshell.internal` are
+  injected into `/etc/hosts` so containers can reach services on the gateway
+  host. Linux defaults to Podman's `host-gateway` resolver. macOS Podman
+  machine defaults to gvproxy's host-loopback IP, `192.168.127.254`, because
+  stale Podman machines may fail to resolve `host-gateway`.
 - nsenter: the supervisor uses `nsenter --net=` instead of `ip netns exec` for
   namespace operations, avoiding the sysfs remount path that fails in rootless
   containers.
@@ -291,6 +293,7 @@ Podman resources after out-of-band container removal or label drift.
 | `OPENSHELL_GRPC_ENDPOINT` | `--grpc-endpoint` | Auto-detected via `host.containers.internal` | Gateway gRPC endpoint for sandbox callbacks. |
 | `OPENSHELL_GATEWAY_PORT` | `--gateway-port` | `17670` | Gateway port used for endpoint auto-detection by the standalone binary. |
 | `OPENSHELL_NETWORK_NAME` | `--network-name` | `openshell` | Podman bridge network name. |
+| `OPENSHELL_PODMAN_HOST_GATEWAY_IP` | `--host-gateway-ip` | empty on Linux, `192.168.127.254` on macOS | Host gateway IP used for sandbox host aliases. Empty uses Podman's `host-gateway` resolver. |
 | `OPENSHELL_SANDBOX_SSH_SOCKET_PATH` | `--sandbox-ssh-socket-path` | `/run/openshell/ssh.sock` | Supervisor Unix socket path in `PodmanComputeConfig`. |
 | `OPENSHELL_STOP_TIMEOUT` | `--stop-timeout` | `10` | Container stop timeout in seconds. |
 | `OPENSHELL_SANDBOX_PIDS_LIMIT` | `--sandbox-pids-limit` | `2048` | Podman cgroup PID limit for sandbox containers. Set `0` to inherit Podman's runtime/default PID limit. |
@@ -304,10 +307,11 @@ Podman resources after out-of-band container removal or label drift.
 The Podman driver is designed for rootless operation. The following adaptations
 matter compared to cluster or rootful runtimes:
 
-1. subuid/subgid preflight check: `check_subuid_range()` in `driver.rs` warns
-   operators if `/etc/subuid` or `/etc/subgid` entries are missing for the
-   current user. This is not a hard error because some systems use LDAP or
-   other mechanisms.
+1. subuid/subgid preflight check: on non-macOS hosts, `check_subuid_range()` in
+   `driver.rs` warns operators if `/etc/subuid` or `/etc/subgid` entries are
+   missing for the current user. This is not a hard error because some systems
+   use LDAP or other mechanisms. macOS skips the check because `podman machine`
+   runs the Podman service inside a Linux VM.
 2. cgroups v2 requirement: the driver refuses to start if cgroups v1 is
    detected. Rootless Podman requires the unified cgroup hierarchy.
 3. `nsenter` for namespace operations: `openshell-sandbox` uses

crates/openshell-driver-podman/src/config.rs

@@ -2,12 +2,14 @@
 // SPDX-License-Identifier: Apache-2.0
 
 use openshell_core::config::{DEFAULT_STOP_TIMEOUT_SECS, DEFAULT_SUPERVISOR_IMAGE};
+use std::net::IpAddr;
 use std::path::PathBuf;
 use std::str::FromStr;
 
 /// Default Podman bridge network name.
 pub const DEFAULT_NETWORK_NAME: &str = "openshell";
 pub const DEFAULT_SANDBOX_PIDS_LIMIT: i64 = 2048;
+pub const MACOS_PODMAN_MACHINE_HOST_GATEWAY_IP: &str = "192.168.127.254";
 
 /// Image pull policy for sandbox and supervisor images.
 ///
@@ -90,6 +92,13 @@ pub struct PodmanComputeConfig {
     /// Name of the Podman bridge network.
     /// Created automatically if it does not exist.
     pub network_name: String,
+    /// Host gateway IP used for sandbox host aliases.
+    ///
+    /// Empty uses Podman's `host-gateway` resolver. macOS defaults to
+    /// gvproxy's host-loopback IP because stale Podman machines may fail to
+    /// resolve `host-gateway` while still serving `host.containers.internal`
+    /// through gvproxy.
+    pub host_gateway_ip: String,
     /// Container stop timeout in seconds (SIGTERM → SIGKILL).
     pub stop_timeout_secs: u32,
     /// OCI image containing the openshell-sandbox supervisor binary.
@@ -164,6 +173,33 @@ impl PodmanComputeConfig {
         Ok(())
     }
 
+    /// Validate optional host gateway override.
+    pub fn validate_host_gateway_ip(&self) -> Result<(), crate::client::PodmanApiError> {
+        let trimmed = self.host_gateway_ip.trim();
+        if trimmed.is_empty() {
+            return Ok(());
+        }
+
+        trimmed.parse::<IpAddr>().map(|_| ()).map_err(|err| {
+            crate::client::PodmanApiError::InvalidInput(format!(
+                "invalid host_gateway_ip value '{trimmed}': {err}"
+            ))
+        })
+    }
+
+    /// Resolve the default host gateway override for the current platform.
+    #[must_use]
+    pub fn default_host_gateway_ip() -> String {
+        #[cfg(target_os = "macos")]
+        {
+            MACOS_PODMAN_MACHINE_HOST_GATEWAY_IP.to_string()
+        }
+        #[cfg(not(target_os = "macos"))]
+        {
+            String::new()
+        }
+    }
+
     /// Resolve the default socket path from the environment.
     ///
     /// - **macOS**: `$HOME/.local/share/containers/podman/machine/podman.sock`
@@ -201,6 +237,7 @@ impl Default for PodmanComputeConfig {
             gateway_port: openshell_core::config::DEFAULT_SERVER_PORT,
             sandbox_ssh_socket_path: "/run/openshell/ssh.sock".to_string(),
             network_name: DEFAULT_NETWORK_NAME.to_string(),
+            host_gateway_ip: Self::default_host_gateway_ip(),
             stop_timeout_secs: DEFAULT_STOP_TIMEOUT_SECS,
             supervisor_image: DEFAULT_SUPERVISOR_IMAGE.to_string(),
             guest_tls_ca: None,
@@ -221,6 +258,7 @@ impl std::fmt::Debug for PodmanComputeConfig {
             .field("gateway_port", &self.gateway_port)
             .field("sandbox_ssh_socket_path", &self.sandbox_ssh_socket_path)
             .field("network_name", &self.network_name)
+            .field("host_gateway_ip", &self.host_gateway_ip)
             .field("stop_timeout_secs", &self.stop_timeout_secs)
             .field("supervisor_image", &self.supervisor_image)
             .field("guest_tls_ca", &self.guest_tls_ca)
@@ -275,6 +313,32 @@ mod tests {
         assert!(cfg.validate_runtime_limits().is_ok());
     }
 
+    #[test]
+    #[cfg(target_os = "macos")]
+    fn default_config_uses_gvproxy_host_gateway_ip_on_macos() {
+        let cfg = PodmanComputeConfig::default();
+        assert_eq!(cfg.host_gateway_ip, MACOS_PODMAN_MACHINE_HOST_GATEWAY_IP);
+        assert!(cfg.validate_host_gateway_ip().is_ok());
+    }
+
+    #[test]
+    #[cfg(not(target_os = "macos"))]
+    fn default_config_leaves_host_gateway_ip_empty_off_macos() {
+        let cfg = PodmanComputeConfig::default();
+        assert!(cfg.host_gateway_ip.is_empty());
+        assert!(cfg.validate_host_gateway_ip().is_ok());
+    }
+
+    #[test]
+    fn host_gateway_ip_validation_rejects_invalid_values() {
+        let cfg = PodmanComputeConfig {
+            host_gateway_ip: "not-an-ip".to_string(),
+            ..PodmanComputeConfig::default()
+        };
+        let err = cfg.validate_host_gateway_ip().unwrap_err();
+        assert!(err.to_string().contains("host_gateway_ip"));
+    }
+
     #[test]
     fn runtime_limit_validation_rejects_negative_pids_limit() {
         let cfg = PodmanComputeConfig {

crates/openshell-driver-podman/src/container.rs

@@ -537,10 +537,7 @@ pub fn build_container_spec_with_token(
         // Inject stable host aliases into /etc/hosts so sandbox containers can
         // reach services on the host. `host.openshell.internal` is the driver-
         // neutral alias used by policies and e2e tests.
-        hostadd: vec![
-            "host.containers.internal:host-gateway".into(),
-            "host.openshell.internal:host-gateway".into(),
-        ],
+        hostadd: hostadd_entries(config),
         netns: NetNS {
             nsmode: "bridge".to_string(),
         },
@@ -622,6 +619,21 @@ pub fn build_container_spec_with_token(
     serde_json::to_value(container_spec).expect("ContainerSpec serialization cannot fail")
 }
 
+fn hostadd_entries(config: &PodmanComputeConfig) -> Vec<String> {
+    let host_gateway_ip = config.host_gateway_ip.trim();
+    if host_gateway_ip.is_empty() {
+        return vec![
+            "host.containers.internal:host-gateway".into(),
+            "host.openshell.internal:host-gateway".into(),
+        ];
+    }
+
+    vec![
+        format!("host.containers.internal:{host_gateway_ip}"),
+        format!("host.openshell.internal:{host_gateway_ip}"),
+    ]
+}
+
 /// Parse a Kubernetes-style CPU quantity to cgroup quota microseconds
 /// (for a 100ms period).
 ///
@@ -1071,6 +1083,7 @@ mod tests {
             socket_path: std::path::PathBuf::from("/tmp/test.sock"),
             default_image: "test-image:latest".to_string(),
             grpc_endpoint: "http://localhost:50051".to_string(),
+            host_gateway_ip: String::new(),
             sandbox_ssh_socket_path: "/run/openshell/test-ssh.sock".to_string(),
             ..PodmanComputeConfig::default()
         }
@@ -1109,6 +1122,34 @@ mod tests {
         );
     }
 
+    #[test]
+    fn container_spec_uses_configured_host_gateway_ip() {
+        let sandbox = test_sandbox("test-id", "test-name");
+        let mut config = test_config();
+        config.host_gateway_ip = "192.168.127.254".to_string();
+        let spec = build_container_spec(&sandbox, &config);
+
+        let hostadd: Vec<&str> = spec["hostadd"]
+            .as_array()
+            .expect("hostadd should be an array")
+            .iter()
+            .filter_map(|v| v.as_str())
+            .collect();
+
+        assert!(
+            hostadd.contains(&"host.containers.internal:192.168.127.254"),
+            "missing Podman host alias with configured host gateway IP"
+        );
+        assert!(
+            hostadd.contains(&"host.openshell.internal:192.168.127.254"),
+            "missing OpenShell host alias with configured host gateway IP"
+        );
+        assert!(
+            !hostadd.contains(&"host.containers.internal:host-gateway"),
+            "configured host gateway IP should avoid Podman's host-gateway resolver"
+        );
+    }
+
     #[test]
     fn container_spec_includes_tls_mounts_when_configured() {
         let sandbox = test_sandbox("tls-id", "tls-name");

crates/openshell-driver-podman/src/driver.rs

@@ -142,6 +142,7 @@ impl PodmanComputeDriver {
         // get a clear error instead of a silent fallback to plaintext HTTP.
         config.validate_tls_config()?;
         config.validate_runtime_limits()?;
+        config.validate_host_gateway_ip()?;
 
         let client = PodmanClient::new(config.socket_path.clone());
 
@@ -195,7 +196,7 @@ impl PodmanComputeDriver {
         // Rootless pre-flight: warn if subuid/subgid ranges look missing.
         // Not a hard error because some systems configure these via LDAP or
         // other mechanisms that /etc/subuid does not reflect.
-        if rustix::process::getuid().as_raw() != 0 {
+        if !cfg!(target_os = "macos") && rustix::process::getuid().as_raw() != 0 {
             check_subuid_range();
         }
 

crates/openshell-driver-podman/src/main.rs

@@ -58,6 +58,12 @@ struct Args {
     )]
     gateway_port: u16,
 
+    /// Host gateway IP used for sandbox host aliases.
+    ///
+    /// Empty uses Podman's `host-gateway` resolver.
+    #[arg(long, env = "OPENSHELL_PODMAN_HOST_GATEWAY_IP")]
+    host_gateway_ip: Option<String>,
+
     #[arg(
         long,
         env = "OPENSHELL_SANDBOX_SSH_SOCKET_PATH",
@@ -118,6 +124,9 @@ async fn main() -> Result<()> {
         image_pull_policy: args.sandbox_image_pull_policy,
         grpc_endpoint: args.grpc_endpoint.unwrap_or_default(),
         gateway_port: args.gateway_port,
+        host_gateway_ip: args
+            .host_gateway_ip
+            .unwrap_or_else(PodmanComputeConfig::default_host_gateway_ip),
         sandbox_ssh_socket_path: args.sandbox_ssh_socket_path,
         network_name: args.network_name,
         stop_timeout_secs: args.stop_timeout,

crates/openshell-server/src/config_file.rs

@@ -275,6 +275,7 @@ fn inheritable_keys(driver: ComputeDriverKind) -> &'static [&'static str] {
         ComputeDriverKind::Podman => &[
             "default_image",
             "supervisor_image",
+            "host_gateway_ip",
             "guest_tls_ca",
             "guest_tls_cert",
             "guest_tls_key",
@@ -498,6 +499,25 @@ version = 2
         );
     }
 
+    #[test]
+    fn podman_driver_table_inherits_gateway_host_gateway_ip() {
+        let gateway = GatewayFileSection {
+            default_image: Some("ghcr.io/nvidia/openshell/sandbox:0.9".to_string()),
+            host_gateway_ip: Some("192.168.127.254".to_string()),
+            ..Default::default()
+        };
+        let merged = driver_table(ComputeDriverKind::Podman, &gateway, None);
+        let table = merged.as_table().expect("table");
+        assert_eq!(
+            table.get("default_image").and_then(|v| v.as_str()),
+            Some("ghcr.io/nvidia/openshell/sandbox:0.9")
+        );
+        assert_eq!(
+            table.get("host_gateway_ip").and_then(|v| v.as_str()),
+            Some("192.168.127.254")
+        );
+    }
+
     #[test]
     fn driver_table_specific_value_overrides_gateway_default() {
         let gateway = GatewayFileSection {

crates/openshell-server/src/lib.rs

@@ -749,6 +749,9 @@ async fn build_compute_runtime(
             if let Ok(p) = std::env::var("OPENSHELL_PODMAN_SOCKET") {
                 podman.socket_path = PathBuf::from(p);
             }
+            if let Ok(ip) = std::env::var("OPENSHELL_PODMAN_HOST_GATEWAY_IP") {
+                podman.host_gateway_ip = ip;
+            }
             apply_podman_local_tls_defaults(config, &mut podman)?;
 
             ComputeRuntime::new_podman(

docs/reference/gateway-config.mdx

@@ -233,6 +233,9 @@ grpc_endpoint           = "https://host.containers.internal:17670"
 # The gateway overwrites gateway_port from bind_address at runtime.
 gateway_port            = 17670
 network_name            = "openshell"
+# Omit for the platform default: empty on Linux, 192.168.127.254 on macOS Podman machine.
+# Set "" to force Podman's host-gateway resolver.
+# host_gateway_ip       = "192.168.127.254"
 sandbox_ssh_socket_path = "/run/openshell/ssh.sock"
 stop_timeout_secs       = 10
 supervisor_image        = "ghcr.io/nvidia/openshell/supervisor:latest"