Tests/dockerd version compat

[misterbh-saashup]

Test dockerd version compatibility

Adds a self-contained harness that verifies the agent works against a pinned range of dockerd versions, independent of whatever Docker is installed on the host — plus a deepened test suite that exercises the agent's real behaviour rather than just HTTP status codes.

Currently green against dockerd 29.5.2, 28.5.2, and 27.5.1 (every version verified locally end-to-end before pushing).

Why

The existing CI ran against whatever daemon happened to be on the runner, and the hurl tests only asserted status codes + isCollection — they never confirmed a real Docker operation occurred. This makes it hard to know whether the agent stays compatible as dockerd evolves.

How it works

The agent talks to a hardcoded /var/run/docker.sock with no API-version prefix (so it always uses the daemon's default API version). To pin the version under test we use Docker-in-Docker: dind runs the chosen docker:<version>-dind and shares its socket into the agent — no agent code change. A wiremock "netbox" stand-in captures the agent's callbacks.

run.sh runs three phases per version; CI (.github/workflows/compat_ci.yml) reads tests/compat/versions.txt into a matrix and runs them in parallel.

What's tested

Test	Coverage
read.hurl	Field-level assertions on the synchronous read endpoints (/api/networks, /api/containers, /api/images, /api/volumes) + the previously-untested /system/usage.
version.hurl	Asserts /info reports the exact dockerd version under test.
lifecycle.hurl	A real container lifecycle through the agent: pull → create → start → logs → stats → exec → stop → delete, polling the read endpoints for each async side effect.
netbox.hurl	Asserts the agent sends the expected callbacks to netbox after a write, via the wiremock request journal.
ws-exec-test.mjs	Covers the interactive /ws exec channel using the agent's bundled client lib (not expressible in hurl).
tests.hurl	The original HTTP smoke tests (unchanged).

Notes / decisions

• Async writes (POST/PUT/DELETE /api/engine/*) return 202 and report to netbox out-of-band, so they're verified by side-effect polling of the synchronous reads.
• Config persistence: the agent rewrites /data/config.js from netbox responses (which can clear netbox_url). The compose entrypoint re-seeds a writable config (a :ro mount caused EROFS that silently broke the netbox flow), and the netbox-contract test runs against a freshly restarted agent so the rest of the suite can't contaminate it.
• Exec output isn't asserted for content — the agent's exec endpoint returns an empty stdout for short commands, so the test asserts the round-trip (200 + stdout field) only. Possible pre-existing agent limitation worth a follow-up.
• CI targets ubuntu-latest (needs Docker + privileged dind); the existing self-hosted pr_ci.yml is left untouched.

How to run locally

Requires Docker with the Compose plugin.

# Test every version listed in versions.txt
./tests/compat/run.sh

# Test a single version
./tests/compat/run.sh 29.5.2

For each version the script builds the agent image, stands up dind + the netbox mock + the agent, runs the hurl suite + the websocket-exec test + the netbox-contract test, tears everything down, and prints a pass/fail summary (non-zero exit if any version fails):

===== Compatibility summary =====
  dockerd 29.5.2       PASS
  dockerd 28.5.2       PASS
  dockerd 27.5.1       PASS

npm test still runs only the original tests.hurl smoke suite against a locally-running agent, unchanged.

Adding a version

Add one line to tests/compat/versions.txt — the local runner and the CI matrix both pick it up automatically.

---

12 files changed, 767 insertions(+)

[lvenier]

There is also some warning raised to be looked at.