Skip to content

ci: run KVM integration + external tests on GitHub-hosted runners#50

Merged
markovejnovic merged 22 commits into
mainfrom
test/latitude-ci
Jul 7, 2026
Merged

ci: run KVM integration + external tests on GitHub-hosted runners#50
markovejnovic merged 22 commits into
mainfrom
test/latitude-ci

Conversation

@markovejnovic

@markovejnovic markovejnovic commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Summary

  • New integration CI job on ubuntu-latest (x64 runners expose /dev/kvm): provisions the runner as a single-node Firecracker host (.github/scripts/provision-kvm-host.sh, mirrors docs/cookbook/install.md) and runs the gated suites with the real supervision tree (mix test --only integration --only external, no --no-start).
  • New live E2E tests (excluded by default): full VM lifecycle (load → boot → exec → clean stop, no dm leak) and crash reclaim (SIGKILL firecracker → monitors + Reaper reclaim the dm volume and routing entry).
  • Hyper.E2e test helpers (dm device observation, polling, exec retry).
  • Research + decision record in docs/superpowers/plans/2026-07-06-kvm-integration-ci.md: GitHub-hosted runners over latitude.sh (public repo → self-hosted runner unsafe; flintlock proves the identical stack on ubuntu-latest).

Test plan

  • mix check green locally (E2E tags excluded by default)
  • integration job green on this PR (the E2E's real verification)

@codecov

codecov Bot commented Jul 7, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ All tests successful. No failed tests found.

📢 Thoughts on this report? Let us know!

@github-actions

github-actions Bot commented Jul 7, 2026

Copy link
Copy Markdown

Test Results

  3 files  +  1  143 suites  +66   1m 31s ⏱️ + 1m 10s
433 tests +  6  433 ✅ + 7    0 💤  -   1  0 ❌ ±0 
730 runs  +303  433 ✅ + 7  297 💤 +296  0 ❌ ±0 

Results for commit 833d20d. ± Comparison against base commit d922a76.

♻️ This comment has been updated with latest results.

The teardown-on-crash contract the plan specified contradicts Core's
documented design: firecracker death cold-boots the daemon/controller
pair on the same mutable rootfs (self-heal). Decision: self-heal is the
contract. The test now proves recovery (exec answers again on the same
dm volume) and that only the explicit stop reclaims the device.
config/runtime.exs set `config :hyper, Hyper.Cfg.Budget, ...` for every
env, and app env is Cfg's highest-priority layer — so those "defaults"
permanently shadowed the [budget] TOML table for every operator and for
our own CI provisioning script's `[budget] cpu_max_load = 4.0`, which
was inert. Move the values into Hyper.Cfg.Budget as real default-layer
fallbacks passed through fetch_cfg, so the layering matches the
module's own documented contract: config.exs > TOML > default.

cpu_max_cap keeps its nilable optional_number path so an operator can
still explicitly set `cpu_max_cap: nil` to disable the cap.
Boot validation (Hyper.Node.Layer.Repo.test_system/0) refuses with
:layer_dir_not_found unless <work_dir>/layers exists — the node only
creates it lazily on first image load. The install guide only had
operators create /srv/hyper itself, which would pass ownership setup
but fail the first boot.
The dm device NAME surviving a crash/recover cycle doesn't prove it's
the SAME volume — a destroy+recreate under the same deterministic name
would pass too, silently losing guest state. Write a marker through the
guest and `sync` it (page cache dies with firecracker) before the
SIGKILL, then read it back after recovery instead of just re-running
/bin/true. The marker write doubles as the pre-crash liveness proof it
replaces; the dm-name check stays as a supplementary assertion.
- Bump the integration job timeout 25 -> 35 min: worst case is two
  10-minute E2E tests plus ~4 min of provisioning/compile/setup, which
  could hit 25 and get cancelled mid-run; cancelled steps then skip
  `!cancelled()` result uploads exactly when flake history matters.
- provision-kvm-host.sh: give the suidhelper setuid-bit check a
  diagnostic on failure, matching the existing thin-pool check, instead
  of failing silently under `set -e`.
- provision-kvm-host.sh: enumerate the deliberate CI-only deltas from
  install.md in the header comment so a future sync pass doesn't
  remove them.
@markovejnovic markovejnovic merged commit 2edb2ad into main Jul 7, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant