Harden boot (failure fallback, PATH shims, TMPDIR on disk) + add test suite & CI#2
Open
garrytan wants to merge 2 commits into
Open
Harden boot (failure fallback, PATH shims, TMPDIR on disk) + add test suite & CI#2garrytan wants to merge 2 commits into
garrytan wants to merge 2 commits into
Conversation
Three production issues this addresses, all observed running this template as a 24/7 service on Render: 1. Crash-looping lockout. CMD ran alphaclaw directly, so any startup failure restart-looped the container and took the Render Shell tab down with it — the one tool needed to debug. Boot now goes through start.sh: it tees the boot log to /data/start.log on the persistent disk and, if alphaclaw ever exits, execs a minimal failure-status server so the deploy stays Live (/health 200) and the Shell tab stays reachable. The failure page renders no logs or env values — they can contain secrets and the page is publicly reachable. 2. spawn openclaw ENOENT. alphaclaw spawns `openclaw` by bare name and inherits PATH. The ENV prepend covers the normal path; the /usr/bin wrappers and /usr/local/bin symlinks keep the binaries resolvable even if the runtime env is munged, and the build-time `openclaw --version` smoke check fails the image build early if the install is broken. Render's runtime can also strip PATH, so start.sh re-exports it (belt and suspenders). 3. Ephemeral /tmp churn. A 24/7 service writes scratch data forever; routing TMPDIR/TEMP/TMP to /data/tmp puts it on the persistent disk instead of the container's ephemeral layer. Bare /tmp is deliberately left untouched (no symlink or bind-mount — Render containers aren't privileged, and hardcoded-/tmp code keeps working). /data is a runtime disk mount that shadows build-time mkdir, so start.sh (re)creates /data/tmp with the sticky bit on every boot. debug-start.sh is an optional inert-container mode for when even the fallback isn't enough (bind :3000, keep PID 1 alive, dump env to /data/debug.log); README documents both debug paths. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three layers, split by cost (details in tests/README.md): - unit: exercises the real failure-server.js in a subprocess — routing, /health, and a security property test that plants sentinel secrets in the env and asserts no HTTP response ever contains them. - contract: fast static locks on the load-bearing config — the PATH prepend, TMPDIR routing, sticky-bit mkdir on boot, tini/CMD wiring, never touching bare /tmp, and the exact-version alphaclaw pin (a drifting spec never changes package.json, so the Docker npm-install layer cache silently keeps shipping a stale alphaclaw). - e2e: builds the real image and boots it Render-style, twice. docker.bats uses a tmpfs over /data (empty disk, like Render's mount) and asserts every invariant at runtime. gateway-ready.bats seeds an onboarded /data (gateway.mode=local) and asserts OpenClaw reaches a genuinely working state — "[gateway] ready" in the logs with the usage-tracker plugin initialized. Render marks a service Live as soon as /health answers, which even the failure fallback satisfies, so "Live" alone proves nothing about the gateway; this makes CI catch the looks-deployed-but-dead class of failure. CI (.github/workflows/test.yml) runs unit+contract and the docker e2e on every push and PR. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
We've been running this template as a 24/7 service on Render (via a fork) and hit a few production failure modes worth fixing upstream. This PR is the generic hardening + the test suite that locks it in — no fork-specific changes.
Commit 1 — boot hardening
CMD ["alphaclaw", "start"]means any startup failure restart-loops the container, which takes the Render Shell tab down with it — the one tool you need to debug. Boot now goes throughstart.sh: it tees the boot log to/data/start.log(persistent disk) and, if alphaclaw exits, execs a tiny failure-status server so the deploy stays Live (/health200) and the Shell tab stays reachable. The failure page deliberately renders no logs/env (it's publicly reachable; logs can contain secrets).spawn openclaw ENOENT. alphaclaw spawnsopenclawby bare name. YourENV PATHprepend covers the normal case; this adds/usr/binwrappers +/usr/local/binsymlinks (Render's runtime env can strip PATH —start.shalso re-exports it), plus a build-timeopenclaw --versionsmoke check that fails the image build early if the install is broken./tmpchurn. A 24/7 agent writes scratch forever;TMPDIR/TEMP/TMP=/data/tmproutes it to the persistent disk. Bare/tmpis left untouched (no symlink/bind-mount — Render containers aren't privileged, and hardcoded-/tmpcode keeps working). Since the disk mount shadows build-timemkdir,start.shrecreates/data/tmp(sticky bit) every boot.debug-start.sh— optional inert-container mode for the worst case (binds :3000 so Render goes Live, keeps PID 1 alive, dumps env to/data/debug.log). Documented in the README troubleshooting section.Commit 2 — tests + CI
failure-server.jsin a subprocess; a security test plants sentinel secrets in the env and asserts no HTTP response ever contains them./tmp, and the exact-version alphaclaw pin — a drifting spec never changespackage.json, so Docker's npm-install layer cache silently keeps shipping a stale alphaclaw; we got bitten by this in the field).docker.batswith a tmpfs over/data(empty disk, like Render's mount) asserting every invariant at runtime;gateway-ready.batswith a seeded onboarded/dataasserting OpenClaw reaches a genuinely working state —[gateway] readyin the logs with usage-tracker initialized. Render marks a service Live as soon as/healthanswers (which even the failure fallback satisfies), so "Live" alone proves nothing about the gateway; this catches the looks-deployed-but-dead class.All verified against this repo's exact
@chrysb/alphaclaw@0.9.18npm dependency: 6 unit + 22 contract + 13 e2e passing, gateway reaches ready in the built image.🤖 Generated with Claude Code