[MAJOR] §12: Production container stage runs test executables, not a service binary

[mvillmow]

Finding

Severity: MAJOR
Section: 12
Evidence: Containerfile:119-124 (Stage 3 copies basic_delegation_tests, module_coordination_tests, component_coordination_tests)
Principle: POLA

The 'production' stage (Stage 3) of the Containerfile copies test binary executables (basic_delegation_tests, module_coordination_tests, component_coordination_tests) as its deployment artifacts and runs them as the default CMD. This is not a production service — it is a test runner. Either the service binary from src/daemon/main.cpp should be packaged, or the stage should be renamed.

---

Part of #504

[mvillmow]

Implementation Plan

Now I have everything I need to write a detailed, concrete implementation plan.

---

Implementation Plan: Issue #513 — Fix Production Stage to Ship Service Binary

1. Objective

The production stage (Stage 3) of Containerfile currently copies test executables (basic_delegation_tests, module_coordination_tests, etc.) and runs them as its CMD. It must instead package and run the real Keystone daemon binary built from src/daemon/main.cpp.

---

2. Approach

1. Add a CMake install target for the daemon binary — src/daemon/main.cpp compiles to a keystone_server (or keystone_daemon) executable but currently has no install() rule. Add one under the keystone component so cmake --install produces /usr/local/bin/keystone-server.
2. Restructure the Containerfile stages — rename the current misnamed stages and split responsibilities cleanly:
   • Stage 1 builder: compile everything (unchanged)
   • Stage 2 test: copy and run test executables (currently called runtime)
   • Stage 3 production: copy only the service binary from the builder, run it as CMD
   • Stage 4 development: unchanged
3. Fix the healthcheck — the production stage currently uses Python3 for its healthcheck. Since a minimal C++ runtime image won't have Python, use wget (or curl) instead. The daemon already serves /v1/health on port 8080.
4. Update compose files — both docker-compose.yml and podman-compose.yml reference stage targets by name; update any stage name references.
5. Verify — confirm the production image contains keystone-server and no test binaries.

---

3. Files to Modify

CMakeLists.txt

• Where: In the install() block near line 748 (the existing install rules section).
• What: Add an install rule for the daemon executable target (currently unnamed; verify target name in the daemon's add_executable() call). Add:

  install(TARGETS keystone_server
      RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR}
      COMPONENT keystone
  )

  If the daemon target is not named keystone_server, find its add_executable() call in src/daemon/CMakeLists.txt or the root CMakeLists.txt and either rename it to keystone_server there, or use the existing name in the install rule.

Containerfile

All changes are stage-level. Line references are approximate from the exploration:

• Stage 2 (runtime, lines ~90–110): Rename to test. This stage correctly copies test executables and runs them — it just needs an accurate name. Change FROM ubuntu:24.04 AS runtime → FROM ubuntu:24.04 AS test.

• Stage 3 (production, lines ~119–140): This is the broken stage. Replace its body:

  • Remove all COPY --from=builder lines that copy test executables.
  • Remove the hmas-server.sh heredoc (a wrapper that invokes tests).
  • Add a single COPY --from=builder /usr/local/bin/keystone-server /usr/local/bin/keystone-server.
  • Replace the Python3 healthcheck with a wget-based check (requires wget in the apt-get install layer of this stage).
  • Replace CMD with CMD ["/usr/local/bin/keystone-server"].
  • Keep the non-root user, port EXPOSE directives, and labels.

• Stage 4 (development, ~lines 145–165): Change FROM builder AS development (or whatever the FROM line references) — no content changes needed, but verify the FROM base stage name still resolves after the stage 2 rename.

docker-compose.yml

• Where: The production service target: key.
• What: If the compose file selects stages by name (e.g., target: production), no change is needed since we keep the stage named production. Verify the test service still references target: runtime — update to target: test after the rename.

podman-compose.yml

• Same audit as docker-compose.yml: Update target: runtime → target: test if that key exists.

docker-compose-distributed.yaml

• Audit for any target: or image: references to the renamed stages. Update accordingly.

---

4. Files to Create

None. This is a configuration-only fix — no new source files are needed.

---

5. Implementation Order

1. Locate the daemon's add_executable() target name — grep src/daemon/CMakeLists.txt (or the root) for add_executable to confirm the exact target name before touching the install rules.
2. Add install(TARGETS ...) for the daemon in CMakeLists.txt — ensures cmake --install produces the binary at ${CMAKE_INSTALL_BINDIR}/keystone-server.
3. Edit Containerfile Stage 3 (production):
   a. Replace all test-binary COPY lines with one line copying the service binary.
   b. Add wget to the apt-get install line in this stage.
   c. Replace the Python3-based HEALTHCHECK with wget -qO- http://localhost:8080/v1/health || exit 1.
   d. Replace CMD with CMD ["/usr/local/bin/keystone-server"].
   e. Remove the hmas-server.sh heredoc entirely.
4. Rename Stage 2 in Containerfile — AS runtime → AS test (so the test stage has an accurate name).
5. Update compose files — change target: runtime → target: test in docker-compose.yml, podman-compose.yml, and any distributed compose file.
6. Build and verify (see §7).

---

6. Verification

# Step 1: Confirm CMake install produces the service binary (not tests)
cmake --preset release
cmake --build --preset release
cmake --install build/x86 --prefix /tmp/keystone-staging --component keystone
ls /tmp/keystone-staging/usr/local/bin/
# Expected: keystone-server (NO test executables)

# Step 2: Build the production image
podman build --target production -t keystone-prod:test .

# Step 3: Confirm only the service binary is present (no test executables)
podman run --rm keystone-prod:test ls /usr/local/bin/
# Expected: keystone-server only — basic_delegation_tests etc. must be absent

# Step 4: Confirm the container starts and healthcheck passes
podman run --rm -d --name keystone-smoke -p 8080:8080 keystone-prod:test
sleep 2
curl -sf http://localhost:8080/v1/health && echo "HEALTHY"
podman stop keystone-smoke

# Step 5: Confirm the test stage still works under its new name
podman build --target test -t keystone-test:verify .
podman run --rm keystone-test:verify
# Expected: test executables run and exit 0

---

7. Skills Used

Invoked during planning:

• hephaestus:advise — searched the team knowledge base before planning

Team knowledge base skills referenced (from Prior Learnings):

• docker-multistage-build — Multi-stage pattern: builder compiles, runtime stage copies only declared install artifacts; verified with ls /usr/local/bin/ inside the image.
• e2e-compose-python-stub-to-cpp-server — Pattern for replacing wrong test/stub CMD with real C++ service binary; wget-based healthcheck instead of Python3.
• conan-hybrid-cpp20-turnkey-packaging — cmake --install --prefix /staging gate pattern to ensure only declared install() targets flow into the production stage; always build clean to avoid stale CMakeCache silently ignoring changes.

---

Generated by Claude Code Planner