ci: run KVM integration + external tests on GitHub-hosted runners

.github/scripts/provision-kvm-host.sh

@@ -0,0 +1,86 @@
+#!/usr/bin/env bash
+# Provision a GitHub-hosted ubuntu runner as a single-node Firecracker host
+# for the `:integration` E2E suite. Mirrors docs/cookbook/install.md; keep
+# the two in sync. Assumes: passwordless sudo, repo compiled (the firecracker
+# and suidhelper install tasks are mix tasks), MIX_ENV matching the test run.
+#
+# Deliberate CI-only deltas from install.md — do NOT remove these in a future
+# sync pass:
+#   - the 0666 udev kvm rule (install.md assumes a real host with a `kvm`
+#     group an operator is added to; the ephemeral runner has neither)
+#   - the `[budget]` stanza raising cpu_max_load to 4.0 (this host is a
+#     dedicated ephemeral runner, so the default's contention guard against
+#     other workloads doesn't apply, and compile/provision load would
+#     otherwise trip :no_capacity)
+#   - pre-creating /srv/hyper/layers (install.md documents this too, but it's
+#     load-bearing here since nothing else primes it before the suite boots
+#     the node)
+#   - DEBIAN_FRONTEND=noninteractive (install.md is written for an interactive
+#     operator session; CI has no tty to prompt on)
+set -euo pipefail
+
+export DEBIAN_FRONTEND=noninteractive
+
+# The jailed VM uid (900000+) opens /dev/kvm; 0666 is GitHub's own documented
+# rule for KVM access on hosted runners.
+echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' \
+  | sudo tee /etc/udev/rules.d/99-kvm4all.rules >/dev/null
+sudo udevadm control --reload-rules
+sudo udevadm trigger --name-match=kvm
+
+sudo apt-get update
+sudo apt-get install -y \
+  coreutils e2fsprogs libc-bin lvm2 skopeo util-linux \
+  "linux-modules-extra-$(uname -r)"
+
+# -a is load-bearing: without it modprobe reads the 2nd+ names as module
+# PARAMETERS of the first, and the load fails.
+sudo modprobe -av dm_snapshot dm_thin_pool loop
+targets="$(sudo dmsetup targets)"
+echo "dmsetup targets: ${targets}"
+grep -q thin-pool <<<"${targets}" || { echo "ERROR: thin-pool dm target missing" >&2; exit 1; }
+
+sudo mkdir -p /etc/hyper
+sudo tee /etc/hyper/config.toml >/dev/null <<'EOF'
+work_dir = "/srv/hyper"
+
+[tools]
+firecracker = "/opt/firecracker/firecracker"
+jailer = "/opt/firecracker/jailer"
+
+[jails]
+uid_gid_range = [900000, 999999]
+cgroup = "hyper"
+
+# The runner is a dedicated ephemeral host: the soft budget's default
+# cpu_max_load (0.8) refuses VMs with :no_capacity while compile/provision
+# load is still decaying, and nothing else runs here worth protecting.
+[budget]
+cpu_max_load = 4.0
+EOF
+sudo chown root:root /etc/hyper/config.toml
+sudo chmod 0644 /etc/hyper/config.toml
+
+sudo mkdir -p /sys/fs/cgroup/hyper
+echo '+cpu +memory' | sudo tee /sys/fs/cgroup/hyper/cgroup.subtree_control >/dev/null
+
+# The BEAM (the `runner` user — Hyper refuses to run as root) owns work_dir.
+# layers/ must pre-exist: boot validation (Layer.Repo.test_system) checks it,
+# and the node only creates it lazily on first image load.
+sudo mkdir -p /srv/hyper
+sudo chown "$(id -u):$(id -g)" /srv/hyper
+mkdir -p /srv/hyper/layers
+
+# firecracker.install writes into /opt/firecracker as the invoking user, then
+# the helper requires both binaries root-owned and not group/world-writable.
+sudo mkdir -p /opt/firecracker
+sudo chown "$(id -u)" /opt/firecracker
+mix firecracker.install
+sudo chown root:root /opt/firecracker/firecracker /opt/firecracker/jailer
+sudo chmod 0755 /opt/firecracker/firecracker /opt/firecracker/jailer
+
+# With passwordless sudo the task installs setuid-root itself.
+mix suidhelper.install
+[ -u /usr/local/bin/hyper-suidhelper ] || { echo "ERROR: hyper-suidhelper missing or not setuid-root" >&2; exit 1; }
+
+echo "provisioned: kvm=$(ls -l /dev/kvm)"

.github/workflows/ci.yml

@@ -254,6 +254,115 @@ jobs:
           sudo -E env "PATH=$PATH" \
             cargo nextest run --features insecure_test_seams -E 'test(/_as_root$/)'
 
+  integration:
+    name: Integration (KVM E2E)
+    # ubuntu-latest x64 exposes /dev/kvm (all Linux runners with 2+ vCPUs,
+    # per GitHub's Apr 2024 changelog); arm64 runners do NOT — keep this x64.
+    runs-on: ubuntu-latest
+    # Worst case: two 10-minute tests (vm_lifecycle + crash_recovery) plus
+    # ~4 minutes of provisioning/compile/setup. 25 could cancel the job right
+    # as it finishes, and `!cancelled()` steps then skip result uploads —
+    # exactly when flake history matters most.
+    timeout-minutes: 35
+    env:
+      MIX_ENV: test
+    services:
+      postgres:
+        image: postgres:17-alpine
+        env:
+          POSTGRES_USER: postgres
+          POSTGRES_PASSWORD: postgres
+        ports:
+          - 5432:5432
+        options: >-
+          --health-cmd "pg_isready -U postgres"
+          --health-interval 10s
+          --health-timeout 5s
+          --health-retries 5
+    steps:
+      - uses: actions/checkout@v7
+
+      # Fail fast and loudly if GitHub ever withdraws nested virt, rather
+      # than timing out inside a VM boot.
+      - name: Assert KVM is available
+        run: '[ -e /dev/kvm ]'
+
+      - uses: erlef/setup-beam@v1
+        id: beam
+        with:
+          elixir-version: "1.20"
+          otp-version: "28"
+
+      # Separate cache key from the elixir job: this _build is test-env only
+      # and the two jobs would otherwise race the same cache entry.
+      - name: Cache deps and _build
+        uses: actions/cache@v6
+        with:
+          path: |
+            deps
+            _build
+          key: ${{ runner.os }}-mix-integration-${{ steps.beam.outputs.otp-version }}-${{ steps.beam.outputs.elixir-version }}-${{ hashFiles('mix.lock') }}
+          restore-keys: |
+            ${{ runner.os }}-mix-integration-${{ steps.beam.outputs.otp-version }}-${{ steps.beam.outputs.elixir-version }}-
+
+      - name: Install dependencies
+        run: mix deps.get
+
+      # protoc + protoc-gen-elixir drive the :grpc_gen Mix compiler (same as
+      # the elixir job).
+      - name: Install protoc
+        run: sudo apt-get update && sudo apt-get install -y protobuf-compiler
+
+      - name: Install protoc-gen-elixir
+        run: mix escript.install hex protobuf 0.17.0 --force
+
+      - name: Install musl target for the guest-agent build
+        run: rustup target add x86_64-unknown-linux-musl
+
+      - name: Cache rust builds
+        uses: Swatinem/rust-cache@v2
+        with:
+          workspaces: |
+            native/guest-agent
+            native/suidhelper
+
+      - name: Compile (warnings as errors)
+        run: mix compile --warnings-as-errors
+
+      # After compile: firecracker.install / suidhelper.install are mix tasks,
+      # and the helper's checksum stamp must come from this build.
+      - name: Provision Firecracker host
+        run: .github/scripts/provision-kvm-host.sh
+
+      - name: Create and migrate test DB
+        run: |
+          mix ecto.create -r Hyper.Img.Db.Repo
+          mix ecto.migrate -r Hyper.Img.Db.Repo
+
+      # No --no-start: the app boots its real supervision tree, which runs
+      # Hyper.Node.test_system/0 host validation before any test executes.
+      # Multiple --only flags OR together: this runs ONLY the gated tags.
+      - name: Integration + external tests
+        run: mix test --only integration --only external --warnings-as-errors
+
+      - name: Upload test results to Codecov
+        if: ${{ !cancelled() }}
+        uses: codecov/codecov-action@v7
+        with:
+          token: ${{ secrets.CODECOV_TOKEN }}
+          files: _build/test/junit.xml
+          flags: integration
+          report_type: test_results
+
+      # test-results.yml globs artifacts/**/*.xml, so this artifact joins the
+      # unified Test Results check automatically.
+      - name: Upload integration test results artifact
+        if: ${{ !cancelled() }}
+        uses: actions/upload-artifact@v7
+        with:
+          name: integration-test-results
+          path: _build/test/junit.xml
+
   # Uploads the triggering event payload so the workflow_run publisher
   # (test-results.yml) can map results back to the originating PR -- required
   # for PR comments on fork PRs. Tiny job, always runs.

AGENTS.md

@@ -23,12 +23,23 @@ mix check        # THE gate. Must pass before any PR. Runs, in order:
                  #   dialyzer                    (strict; @specs required)
 mix test                              # Elixir suite (needs Postgres for DB tests)
 mix test test/unit test/controls      # pure tests, no Postgres/Firecracker needed
+mix test --only integration --only external
+                 # live E2E: boots real Firecracker VMs. Needs a provisioned
+                 # KVM host (docs/cookbook/install.md) + passwordless sudo.
+                 # CI runs this in ci.yml's `integration` job on ubuntu-latest.
 cargo nextest run                     # Rust suite (run inside native/suidhelper/)
 ```
 
 Pure tests under `test/unit` and `test/controls` need neither Postgres nor
 Firecracker. DB-backed tests need `mix ecto.create && mix ecto.migrate` first.
 
+The `:integration` suite (`test/e2e/`) is the live Firecracker E2E: it starts
+the full supervision tree (never combine with `--no-start`) and drives real
+VMs. `:external` tests pull real OCI images. Both are excluded by default and
+run in CI's `integration` job, which provisions the GitHub-hosted runner via
+`.github/scripts/provision-kvm-host.sh` — keep that script in lockstep with
+`docs/cookbook/install.md`.
+
 ## Layout
 
 - `lib/` — Elixir source. Tests in `test/` mirror this tree.

CONTRIBUTING.md

@@ -113,6 +113,8 @@ happy path is often enough. Do **not** add mechanical, one-assertion-per-getter
 tests that exist only to inflate the count -- they will be asked to be removed.
 Match the style of the tests already in `test/`.
 
+The `:integration` and `:external` test suites are excluded by default. The `:integration` suite (`test/e2e/`) contains live Firecracker E2E tests that boot real VMs and require a provisioned KVM host (see `docs/cookbook/install.md`) and passwordless sudo. The `:external` suite pulls real OCI images. Both run automatically in CI's `integration` job on every PR, so you do not need to run them locally unless you are developing them -- `mix check` runs the default test set, which is sufficient for most contributions.
+
 ## Generated code
 
 The Firecracker API bindings under

config/runtime.exs

@@ -1,22 +1,11 @@
 import Config
 
-# Per-node resource budget. Lives in runtime config because it builds `Unit.*`
-# values, which are only loadable once the app's modules are on the code path.
-config :hyper, Hyper.Cfg.Budget,
-  mem_max: Unit.Information.gib(4),
-  disk_max: Unit.Information.gib(4),
-  cpu_max_load: 0.8,
-  cpu_max_cap: 4.0,
-  disk_bw_cap: Unit.Bandwidth.gibps(1),
-  disk_bw_max_load: 0.8,
-  net_bw_cap: Unit.Bandwidth.gibps(1),
-  net_bw_max_load: 0.8
-
 # Operator overrides from a well-known location. An optional Elixir config file
 # at /etc/hyper/config.exs (override the path with HYPER_CONFIG) is merged in
-# last, so its values win over every default set above. An absent file is a
-# no-op — the normal case in dev and CI. Skipped under :test so the suite never
-# reads host state.
+# as app env, which every `Hyper.Cfg.*` module treats as the highest-priority
+# layer — above the `[budget]`-style TOML tables and above built-in defaults.
+# An absent file is a no-op — the normal case in dev and CI. Skipped under
+# :test so the suite never reads host state.
 #
 # OpenTelemetry exporter wiring is resolved through Hyper.Cfg.Otel so that the
 # operator's `config :hyper, Hyper.Cfg.Otel, ...` stanza (if present) takes

docs/cookbook/config.md

@@ -185,18 +185,19 @@ headers = { "authorization" = "Bearer YOUR_TOKEN" }
 ## Budget Configuration
 
 Hyper allows you to control the absolute maximal budgets that are available to
-all VMs on a particular node. Every field is required except `cpu_max_cap`.
-
-| Config Key         | `config.exs`        | `config.toml`       | Default | Notes |
-| ------------------ | ------------------- | ------------------- | ------- | ----- |
-| `mem_max`          | `.mem_max`          | `.mem_max`          | -       | [$\alpha$ (hard) budget](./architecture.md#budgets): total [unit](#unit) of RAM the node offers VMs. Must not exceed available system memory. |
-| `disk_max`         | `.disk_max`         | `.disk_max`         | -       | [$\alpha$ (hard) budget](./architecture.md#budgets): total [unit](#unit) of disk the node offers VMs. Must not exceed available system disk. |
-| `cpu_max_load`     | `.cpu_max_load`     | `.cpu_max_load`     | -       | [$\beta$ (soft) budget](./architecture.md#budgets): instantaneous CPU load threshold, a fraction `0.0`–`1.0` of the cap. |
-| `cpu_max_cap`      | `.cpu_max_cap`      | `.cpu_max_cap`      | `nil`   | [$\beta$ (soft) budget](./architecture.md#budgets): soft CPU capacity in cores (a float). Optional. |
-| `disk_bw_cap`      | `.disk_bw_cap`      | `.disk_bw_cap`      | -       | [$\beta$ (soft) budget](./architecture.md#budgets): soft disk-bandwidth capacity ([unit](#unit)). |
-| `disk_bw_max_load` | `.disk_bw_max_load` | `.disk_bw_max_load` | -       | [$\beta$ (soft) budget](./architecture.md#budgets): disk-bandwidth load threshold, a fraction `0.0`–`1.0` of the cap. |
-| `net_bw_cap`       | `.net_bw_cap`       | `.net_bw_cap`       | -       | [$\beta$ (soft) budget](./architecture.md#budgets): soft network-bandwidth capacity ([unit](#unit)). |
-| `net_bw_max_load`  | `.net_bw_max_load`  | `.net_bw_max_load`  | -       | [$\beta$ (soft) budget](./architecture.md#budgets): network-bandwidth load threshold, a fraction `0.0`–`1.0` of the cap. |
+all VMs on a particular node. Every field has a built-in default; set only the
+keys you need to change.
+
+| Config Key         | `config.exs`        | `config.toml`       | Default   | Notes |
+| ------------------ | ------------------- | ------------------- | --------- | ----- |
+| `mem_max`          | `.mem_max`          | `.mem_max`          | `4 GiB`   | [$\alpha$ (hard) budget](./architecture.md#budgets): total [unit](#unit) of RAM the node offers VMs. Must not exceed available system memory. |
+| `disk_max`         | `.disk_max`         | `.disk_max`         | `4 GiB`   | [$\alpha$ (hard) budget](./architecture.md#budgets): total [unit](#unit) of disk the node offers VMs. Must not exceed available system disk. |
+| `cpu_max_load`     | `.cpu_max_load`     | `.cpu_max_load`     | `0.8`     | [$\beta$ (soft) budget](./architecture.md#budgets): instantaneous CPU load threshold, a fraction `0.0`–`1.0` of the cap. |
+| `cpu_max_cap`      | `.cpu_max_cap`      | `.cpu_max_cap`      | `4.0`     | [$\beta$ (soft) budget](./architecture.md#budgets): soft CPU capacity in cores (a float). Optional — an explicit `nil` in `config.exs` disables the cap. |
+| `disk_bw_cap`      | `.disk_bw_cap`      | `.disk_bw_cap`      | `1 GiBps` | [$\beta$ (soft) budget](./architecture.md#budgets): soft disk-bandwidth capacity ([unit](#unit)). |
+| `disk_bw_max_load` | `.disk_bw_max_load` | `.disk_bw_max_load` | `0.8`     | [$\beta$ (soft) budget](./architecture.md#budgets): disk-bandwidth load threshold, a fraction `0.0`–`1.0` of the cap. |
+| `net_bw_cap`       | `.net_bw_cap`       | `.net_bw_cap`       | `1 GiBps` | [$\beta$ (soft) budget](./architecture.md#budgets): soft network-bandwidth capacity ([unit](#unit)). |
+| `net_bw_max_load`  | `.net_bw_max_load`  | `.net_bw_max_load`  | `0.8`     | [$\beta$ (soft) budget](./architecture.md#budgets): network-bandwidth load threshold, a fraction `0.0`–`1.0` of the cap. |
 
 <!-- tabs open -->
 ### `config.exs`

docs/cookbook/install.md

@@ -83,7 +83,7 @@ Hyper relies on `dm-snapshot` and `dm-thin` to build COW filesystems. Load the
 modules and confirm the targets are present:
 
 ```sh
-sudo modprobe dm_snapshot dm_thin_pool loop
+sudo modprobe -a dm_snapshot dm_thin_pool loop
 sudo dmsetup targets # must list snapshot, thin, and thin-pool
 ```
 
@@ -237,11 +237,14 @@ and the ones it deliberately does **not**.
 
 The node builds its entire on-disk tree (`jails`, `socks`, `scratch`, `layers`,
 `redist`) under `work_dir` (from `/etc/hyper/config.toml`, default `/srv/hyper`)
-**as this user**. It must therefore own that directory:
+**as this user**. It must therefore own that directory. Boot validation
+(`Hyper.Node.Layer.Repo.test_system/0`) refuses to start unless the `layers`
+subdirectory already exists — the node only creates it lazily on first image
+load, so pre-create it now:
 
 ```sh
-sudo mkdir -p /srv/hyper
-sudo chown hyper:hyper /srv/hyper
+sudo mkdir -p /srv/hyper/layers
+sudo chown -R hyper:hyper /srv/hyper
 ```
 
 ## Installation