Skip to content

ci: make CI runnable on self-hosted runners (orphan-container fix, distutils, drop ubuntu20) #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
igorls merged 4 commits into from
Jun 6, 2026
Merged

ci: make CI runnable on self-hosted runners (orphan-container fix, distutils, drop ubuntu20) #1

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
b02376b
ci: drop ubuntu20 (Focal) build/test target
igorls Jun 6, 2026
de73d29
ci: make parallel-ctest-containers idempotent against orphaned contai…
igorls Jun 6, 2026
26993e6
package: drop python3-distutils from dev package deps
igorls Jun 6, 2026
c00a0ce
ci: don't auto-run Build & Test on pull_request
igorls Jun 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 0 additions & 22 deletions .cicd/platforms/ubuntu20.Dockerfile
View file
Open in desktop

This file was deleted.

2 changes: 1 addition & 1 deletion .github/actions/parallel-ctest-containers/dist/index.mjs
View file
Open in desktop

Large diffs are not rendered by default.

8 changes: 8 additions & 0 deletions .github/actions/parallel-ctest-containers/main.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,12 @@ const test_timeout = core.getInput('test-timeout', {required: true});
const repo_name = process.env.GITHUB_REPOSITORY.split('/')[1];

try {
// Defensively remove any leftover container from a prior/interrupted run. On a
// persistent self-hosted runner the Docker daemon survives between jobs, and the
// per-test containers below use fixed names with no --rm, so an orphan would block
// name reuse with "container name already in use". (ENF's ephemeral runners get a
// fresh daemon each job and never hit this.)
child_process.spawnSync("docker", ["rm", "-f", "base"], {stdio:"ignore"});
if(child_process.spawnSync("docker", ["run", "--name", "base", "-v", `${process.cwd()}/build.tar.zst:/build.tar.zst`, "--workdir", `/__w/${repo_name}/${repo_name}`, container, "sh", "-c", "zstdcat /build.tar.zst | tar x"], {stdio:"inherit"}).status)
throw new Error("Failed to create base container");
if(child_process.spawnSync("docker", ["commit", "base", "baseimage"], {stdio:"inherit"}).status)
Expand All @@ -29,6 +35,8 @@ try {

let subprocesses = [];
tests.forEach(t => {
// Clear any orphaned container of this name before reusing it (see note above).
child_process.spawnSync("docker", ["rm", "-f", t.name], {stdio:"ignore"});
Comment on lines 35 to +39
Copy link
Copy Markdown

@gemini-code-assist gemini-code-assist Bot Jun 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Spawning a new process synchronously inside a forEach loop blocks the Node.js event loop. If there are many tests, executing docker rm -f sequentially for each test will introduce significant synchronous overhead and delay the start of all test containers.

Since docker rm supports removing multiple containers in a single command, we can batch this operation into a single spawnSync call before the loop. This is much more efficient.

   // Clear any orphaned containers of these names before reusing them (see note above).
   const test_names = tests.map(t => t.name);
   if (test_names.length > 0) {
      child_process.spawnSync("docker", ["rm", "-f", ...test_names], {stdio:"ignore"});
   }

   let subprocesses = [];
   tests.forEach(t => {

Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot Jun 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Avoid force-removing another job's test container

When two matrix jobs using this action land on runners that share the same Docker daemon, the fixed ctest-derived container names overlap across platforms, and docker rm -f force-removes a running container (Docker documents --force as using SIGKILL). In that scenario a later NP/LR job can kill an in-progress test container from an earlier job instead of just clearing an orphan, so the self-hosted workflow can become flaky or lose logs; use job-unique container names/labels and only clean containers owned by the current run.

Useful? React with 👍 / 👎.

subprocesses.push(new Promise(resolve => {
child_process.spawn("docker", ["run", "--security-opt", "seccomp=unconfined", "-e", "GITHUB_ACTIONS=True", "--name", t.name, "--init", "baseimage", "bash", "-c", `cd build; ctest --output-on-failure -R '^${t.name}$' --timeout ${test_timeout}`], {stdio:"inherit"}).on('close', code => resolve(code));
}));
Expand Down
13 changes: 5 additions & 8 deletions .github/workflows/build.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,10 @@ on:
branches:
- main
- "release/*"
pull_request:
# NOTE: no `pull_request` trigger by design. This workflow runs on self-hosted
# runners, where auto-running PR code (especially from forks) is a security and
# cost risk. CI on a PR branch is a deliberate human action: a maintainer dispatches
# it via the "Run workflow" button or `gh workflow run build.yaml --ref <branch>`.
workflow_dispatch:
inputs:
override-cdt:
Expand Down Expand Up @@ -85,7 +88,7 @@ jobs:
strategy:
fail-fast: false
matrix:
platform: [ubuntu20, ubuntu22, ubuntu24, ubuntu26, reproducible]
platform: [ubuntu22, ubuntu24, ubuntu26, reproducible]
runs-on: ubuntu-latest
container: ${{fromJSON(needs.platform-cache.outputs.platforms)[matrix.platform].image}}
steps:
Expand Down Expand Up @@ -133,14 +136,12 @@ jobs:
fail-fast: false
matrix:
include:
- cfg: {name: 'ubuntu20', base: 'ubuntu20', builddir: 'ubuntu20'}
- cfg: {name: 'ubuntu22', base: 'ubuntu22', builddir: 'ubuntu22'}
- cfg: {name: 'ubuntu24', base: 'ubuntu24', builddir: 'ubuntu24'}
- cfg: {name: 'ubuntu26', base: 'ubuntu26', builddir: 'ubuntu26'}
- cfg: {name: 'asserton', base: 'asserton', builddir: 'asserton'}
- cfg: {name: 'ubsan', base: 'ubsan', builddir: 'ubsan'}
- cfg: {name: 'asan', base: 'asan', builddir: 'asan'}
- cfg: {name: 'ubuntu20repro', base: 'ubuntu20', builddir: 'reproducible'}
- cfg: {name: 'ubuntu22repro', base: 'ubuntu22', builddir: 'reproducible'}
runs-on: ["self-hosted", "enf-x86-hightier"]
container:
Expand Down Expand Up @@ -181,13 +182,11 @@ jobs:
fail-fast: false
matrix:
include:
- cfg: {name: 'ubuntu20', base: 'ubuntu20', builddir: 'ubuntu20'}
- cfg: {name: 'ubuntu22', base: 'ubuntu22', builddir: 'ubuntu22'}
- cfg: {name: 'ubuntu24', base: 'ubuntu24', builddir: 'ubuntu24'}
- cfg: {name: 'ubuntu26', base: 'ubuntu26', builddir: 'ubuntu26'}
- cfg: {name: 'asserton', base: 'asserton', builddir: 'asserton'}
- cfg: {name: 'ubsan', base: 'ubsan', builddir: 'ubsan'}
- cfg: {name: 'ubuntu20repro', base: 'ubuntu20', builddir: 'reproducible'}
- cfg: {name: 'ubuntu22repro', base: 'ubuntu22', builddir: 'reproducible'}
runs-on: ["self-hosted", "enf-x86-midtier"]
steps:
Expand Down Expand Up @@ -224,13 +223,11 @@ jobs:
fail-fast: false
matrix:
include:
- cfg: {name: 'ubuntu20', base: 'ubuntu20', builddir: 'ubuntu20'}
- cfg: {name: 'ubuntu22', base: 'ubuntu22', builddir: 'ubuntu22'}
- cfg: {name: 'ubuntu24', base: 'ubuntu24', builddir: 'ubuntu24'}
- cfg: {name: 'ubuntu26', base: 'ubuntu26', builddir: 'ubuntu26'}
- cfg: {name: 'asserton', base: 'asserton', builddir: 'asserton'}
- cfg: {name: 'ubsan', base: 'ubsan', builddir: 'ubsan'}
- cfg: {name: 'ubuntu20repro', base: 'ubuntu20', builddir: 'reproducible'}
- cfg: {name: 'ubuntu22repro', base: 'ubuntu22', builddir: 'reproducible'}
runs-on: ["self-hosted", "enf-x86-lowtier"]
steps:
Expand Down
1 change: 0 additions & 1 deletion .github/workflows/performance_harness_run.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,6 @@ on:
description: 'Select Platform'
type: choice
options:
- ubuntu20
- ubuntu22
override-test-params:
description: 'Override perf harness params'
Expand Down
9 changes: 0 additions & 9 deletions .github/workflows/release.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,15 +13,6 @@ jobs:
packages: write
actions: read
steps:
- name: Get ubuntu20 antelope-spring-dev.deb
uses: AntelopeIO/asset-artifact-download-action@v3
with:
owner: ${{github.repository_owner}}
repo: ${{github.event.repository.name}}
file: 'antelope-spring-dev.*amd64.deb'
target: ${{github.sha}}
artifact-name: antelope-spring-dev-ubuntu20-amd64
wait-for-exact-target: true
- name: Get ubuntu22 antelope-spring-dev.deb
uses: AntelopeIO/asset-artifact-download-action@v3
with:
Expand Down
4 changes: 3 additions & 1 deletion package.cmake
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,9 @@ set(CPACK_DEBIAN_BASE_FILE_NAME "${CPACK_DEBIAN_FILE_NAME}.deb")
string(REGEX REPLACE "^(${CMAKE_PROJECT_NAME})" "\\1-dev" CPACK_DEBIAN_DEV_FILE_NAME "${CPACK_DEBIAN_BASE_FILE_NAME}")

#deb package tooling will be unable to detect deps for the dev package. llvm is tricky since we don't know what package could have been used; try to figure it out
set(CPACK_DEBIAN_DEV_PACKAGE_DEPENDS "libgmp-dev, python3-distutils, python3-numpy, zlib1g-dev")
# NOTE: python3-distutils was dropped here — it was removed from Python 3.12 / Ubuntu 24.04+
# (so the dev package became uninstallable there) and nothing in spring or TestHarness uses it.
set(CPACK_DEBIAN_DEV_PACKAGE_DEPENDS "libgmp-dev, python3-numpy, zlib1g-dev")
Comment on lines +65 to +67
Copy link
Copy Markdown

@gemini-code-assist gemini-code-assist Bot Jun 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

While removing python3-distutils from the package dependencies solves the installation issue on Ubuntu 24.04+, scripts/postinst and scripts/prerm still import distutils when the Python version is less than 3.10.0:

if sys.version_info < (3, 10, 0):
    from distutils import sysconfig
    ...

If this .deb package is installed on an older system (such as Ubuntu 20.04, which you noted is still supported via glibc forward-compat) where python3-distutils is not installed by default, the installation or removal will fail with ModuleNotFoundError: No module named 'distutils'.

To safely drop the python3-distutils dependency, you should update scripts/postinst and scripts/prerm to use the standard sysconfig module (available since Python 3.2) instead of distutils:

import sysconfig
try:
    print(sysconfig.get_path('platlib', 'deb_system'))
except KeyError:
    print(sysconfig.get_path('platlib'))

This is compatible with all Python 3 versions and avoids any dependency on distutils.

Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot Jun 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Keep distutils available for maintainer scripts

The dev package still installs scripts/postinst/scripts/prerm, and those scripts import distutils.sysconfig whenever the target machine's Python is older than 3.10. Removing python3-distutils from the package dependencies means installs or removals on Python 3.8/3.9 Debian/Ubuntu systems can fail in the maintainer script if that module is not already present; either update the scripts to avoid distutils on those versions or keep a conditional dependency for those packages.

Useful? React with 👍 / 👎.

find_program(DPKG_QUERY "dpkg-query")
if(DPKG_QUERY AND OS_RELEASE MATCHES "\n?ID=\"?ubuntu" AND LLVM_CMAKE_DIR)
execute_process(COMMAND "${DPKG_QUERY}" -S "${LLVM_CMAKE_DIR}" COMMAND cut -d: -f1 RESULT_VARIABLE LLVM_PKG_FIND_RESULT OUTPUT_VARIABLE LLVM_PKG_FIND_OUTPUT)
Expand Down
Loading