ci: run Docker builds on a self-hosted 1ES runner pool#63
Open
benhillis wants to merge 1 commit into
Open
Conversation
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
Moves Docker build jobs to Microsoft 1ES self-hosted runners to increase disk/CPU headroom and reduce flakiness from ubuntu-latest limitations.
Changes:
- Switch
buildjob fromubuntu-latestto the 1ES hosted poolopenvmm-deps-gh-amd-westus2withubuntu2404-amd64. - Add job-level
permissionsto enable OIDC (id-token: write) for 1ES.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
855eec2 to
8fab03f
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.
Comments suppressed due to low confidence (1)
.github/workflows/build.yml:51
- The PR description calls out
aarch64continuing to build under QEMU, but this change removes the conditionaldocker/setup-qemu-actionstep and doesn't add an equivalentbinfmt/QEMU setup. Ifaarch64is still expected to run on an amd64 runner, please add QEMU/binfmt setup back; if the intent is native arm64 runners, consider updating the description (and potentially the--platformvalue) to reflect the new execution model.
- name: Install Docker Engine
uses: docker/setup-docker-action@v4
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 1 out of 1 changed files in this pull request and generated 2 comments.
Comments suppressed due to low confidence (1)
.github/workflows/build.yml:62
- Making the Docker socket world-writable (
chmod 666) allows any process/user on the runner to gain root via Docker. Prefer adding the runner user to thedockergroup (and keeping the socket at 660), or keep usingsudo docker ...in the steps that need it and avoid broadening socket permissions.
# Allow the runner user (and other actions like setup-qemu-action)
# to talk to dockerd without sudo.
sudo chmod 666 /var/run/docker.sock
docker info --format 'Docker Root Dir: {{.DockerRootDir}} / Storage Driver: {{.Driver}}'
Move the build job off ubuntu-latest onto the dedicated 1ES Hosted
Pool openvmm-deps-gh-amd-westus2 (image ubuntu2404-amd64). Both
x86_64 and aarch64 share the AMD pool; aarch64 still cross-compiles
via QEMU.
Fixes the recurring 'No space left on device' flakes on the
GitHub-hosted runners, gives us a project-owned scheduling pool, and
is a sizeable speedup (D32a has 32 vCPUs vs ubuntu-latest's 4, and
build I/O lands on a local 1.2TB SSD): x86_64 ~36m -> 9m20s,
aarch64 ~43m -> 15m34s.
The 1ES ubuntu2404-amd64 image doesn't ship with Docker, so the
workflow installs it via the get.docker.com script and pre-symlinks
/var/lib/{docker,containerd,buildkit} to /mnt before the install.
The OS disk is small (~80GB) and Docker 29's containerd snapshotter
writes to /var/lib/containerd regardless of dockerd's data-root, so
without the symlinks the OS disk fills up mid-build and the runner
agent gets killed with no log capture. The whole install step could
be dropped in the future by baking Docker into a custom 1ES image.
cgmanifest and release jobs stay on ubuntu-latest (small jobs, no
benefit from self-hosted capacity).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
c5b43c7 to
6a9f6cb
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Move the
buildjob offubuntu-latestonto the dedicated 1ES Hosted Poolopenvmm-deps-gh-amd-westus2(imageubuntu2404-amd64). Both x86_64 and aarch64 share the AMD pool; aarch64 still cross-compiles via QEMU.Fixes the recurring
No space left on deviceflakes from the GitHub-hosted runners (e.g. run 25928571548) and gives us a project-owned scheduling pool.Also a nice speedup — D32a has 32 vCPUs vs
ubuntu-latest's 4, and build I/O lands on a local 1.2TB SSD:ubuntu-latest, avg of 3 runs)Standard_D32a)The workflow grew an
Install Docker Enginestep because the 1ES image doesn't ship Docker. The OS disk is small (~80GB) so the storage paths are pre-symlinked to/mnt(~1.2TB resource SSD); without that, Docker 29's containerd snapshotter fills the OS disk and the runner agent gets killed mid-build with no log capture.Future improvement: bake Docker (and the
/mntstorage layout) directly into the 1ES image so the install step can be dropped entirely.cgmanifestandreleasestay onubuntu-latest(small jobs, no benefit).