Skip to content

fix(ci): add /root/.local/bin to rootless e2e PATH so mise is found#1642

Closed
pimlock wants to merge 3 commits into
mainfrom
fix/ci-rootless-mise-path
Closed

fix(ci): add /root/.local/bin to rootless e2e PATH so mise is found#1642
pimlock wants to merge 3 commits into
mainfrom
fix/ci-rootless-mise-path

Conversation

@pimlock
Copy link
Copy Markdown
Collaborator

@pimlock pimlock commented May 29, 2026

Summary

The rust-podman-rootless E2E suite (added in #1623) fails on every run. When the suite drops to the openshell-test user via runuser, three compounding problems prevent the run from getting past mise:

  1. mise not on PATH. The rebuilt PATH omitted /root/.local/bin, where the mise binary is installed by curl https://mise.run (see deploy/docker/Dockerfile.ci). /opt/mise/shims only holds tool shims, not the mise executable. → mise: command not found (127).
  2. /root not traversable. Root's home is mode 0700, so the non-root user cannot search into it to reach /root/.local/bin/mise or /root/.cargo/bin. Bash silently skips unsearchable PATH dirs, so this also surfaced as 127 (not 126).
  3. mise cache not writable. The image sets MISE_CACHE_DIR=/opt/mise/cache (root-owned). The rootless user inherits it but only has read/exec (a+rX), not write. mise run must write tool bin-path caches (/opt/mise/cache/<tool>/.../bin_paths-*.msgpack) and task lock files (/opt/mise/cache/lockfiles/*) even with --skip-deps. → Permission denied (os error 13) at src/lock_file.rs:37.

(Config trust is not an issue in CI: the container sets CI=true, which makes mise auto-trust the repo config.)

Related Issue

Regression introduced by #1623 (which added the rootless suite). No tracking issue; fixing directly.

Changes

  • Add /root/.local/bin to the PATH override passed to runuser so the mise binary is discoverable, and to the chmod -R a+rX loop for consistency.
  • Add chmod a+x /root to the rootless setup step so the openshell-test user can traverse root's home (0700 → 0711, search bit only, no read) to reach the already-755 /root/.local/bin and /root/.cargo/bin subtrees.
  • Override MISE_CACHE_DIR to the rootless user's home (/home/openshell-test/.cache/mise) so mise has a writable cache, keeping the shared /opt/mise tool data dir read-only. The state dir already defaults to the user's home; MISE_DATA_DIR stays inherited so the prebuilt toolchain is reused.

Why this is safe and doesn't undermine the test

  • Test integrity: This suite validates rootless Podman networking end-to-end (pasta, user-namespace isolation, subuid/subgid, the driver detecting rootless mode). None of that depends on /root perms or the mise cache location, so the changes neither mask a real rootless bug nor create a false pass.
  • Security: chmod a+x /root adds only the search bit (0711), not read. The shared /opt/mise stays read-only — the rootless user writes mise cache/state only inside its own home. This runs in an ephemeral, already --privileged CI container built from a tooling image with no secrets, torn down after the job.
  • Alternatives considered: moving permission tweaks into Dockerfile.ci (touches the shared image), provisioning a dedicated rootless user/toolchain in the image (larger change, partial duplication), chmod -R a+rwX /opt/mise (broad write on the shared tool dir), or installing mise/rust as the rootless user at job time (re-installs the toolchain per run, defeating the prebuilt image). The scoped workflow changes were chosen as the lowest-risk fix confined to the one job that needs it.

Testing

  • YAML syntax validated.
  • Reproduced and verified locally against the exact CI image digest (ghcr.io/nvidia/openshell/ci:latest, sha256:ef52bb40…), replicating the rootless setup, runuser env, CI=true, and the inherited MISE_CACHE_DIR=/opt/mise/cache:
    • mise resolves at /root/.local/bin/mise; cargo at /root/.cargo/bin/cargo.
    • No Permission denied (os error 13) and no trust error.
    • The task renders and runs through into e2e/rust/e2e-podman-rootless.sh (the actual test entrypoint).
  • mise run pre-commit passes.
  • E2E tests added/updated (if applicable) — validated by this PR's own rust-podman-rootless job.

Note: local repro validates everything up to the test script entrypoint. The script itself needs Podman rootless networking and the supervisor image, exercised only in CI — any failure beyond this point is genuine test behavior, separate from this PATH/traversal/cache regression.

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated (if applicable) — N/A

The rust-podman-rootless E2E suite introduced in #1623 rebuilds PATH when
dropping to the openshell-test user via runuser, but omitted
/root/.local/bin where the mise binary is installed (curl https://mise.run).
This caused 'mise: command not found' (exit 127) on every run.

Add /root/.local/bin to the PATH override and to the chmod -R a+rX loop so
the rootless user can discover and execute mise.

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
@pimlock pimlock added the test:e2e Requires end-to-end coverage label May 29, 2026
@github-actions
Copy link
Copy Markdown

Label test:e2e applied for 53c7b44. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

Root's home (/root) is mode 0700, so the openshell-test user could not
search into it to reach the mise binary (/root/.local/bin/mise) or cargo
(/root/.cargo/bin) listed on PATH. Bash silently skips unsearchable PATH
entries, producing 'mise: command not found' (exit 127).

Add the search bit on /root (chmod a+x) in the rootless setup step.
Verified locally against the ci:latest image: before the change mise is
not found; after it, both mise and cargo resolve and run as the rootless
user.

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
@pimlock
Copy link
Copy Markdown
Collaborator Author

pimlock commented May 29, 2026

@zredlined's branch has a fix for this already: #1528

@pimlock pimlock closed this May 29, 2026
With mise now discoverable and runnable, 'mise run' still failed with
'Permission denied (os error 13)' at src/lock_file.rs:37. The CI image sets
MISE_CACHE_DIR=/opt/mise/cache (root-owned), which the openshell-test user
inherits but can only read (a+rX), not write. mise must write tool bin-path
caches (/opt/mise/cache/<tool>/.../bin_paths-*.msgpack) and task lock files
(/opt/mise/cache/lockfiles/*) even with --skip-deps.

Override MISE_CACHE_DIR to the rootless user's home so mise has a writable
cache, keeping the shared /opt/mise tool data dir read-only. The state dir
already defaults to the user's home; MISE_DATA_DIR stays inherited so the
prebuilt toolchain is reused.

Verified locally against ci:latest with CI=true (which makes mise auto-trust
the config, matching CI): the task now renders and runs into
e2e/rust/e2e-podman-rootless.sh with no permission or trust errors.

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
@pimlock pimlock reopened this May 29, 2026
@pimlock pimlock closed this May 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test:e2e Requires end-to-end coverage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant