Skip to content

feat: Updated CI for multi platform builds#28

Open
malteos wants to merge 2 commits intomasterfrom
feat/multi-platform-ci-builds
Open

feat: Updated CI for multi platform builds#28
malteos wants to merge 2 commits intomasterfrom
feat/multi-platform-ci-builds

Conversation

@malteos
Copy link
Copy Markdown
Contributor

@malteos malteos commented Feb 24, 2026

This PR enables building container images for linux/amd64 and linux/arm64 platforms. Fixes #27

Verified on MacOS:

$ uname -p
arm

$ podman pull ghcr.io/commoncrawl/cc-crawl-statistics/stats:pr-28 

$ podman inspect ghcr.io/commoncrawl/cc-crawl-statistics/stats:pr-28 | grep Architecture
          "Architecture": "arm64",

malteos and others added 2 commits February 24, 2026 11:33
Previously, pushing to a branch with an open PR triggered both the
push and pull_request workflows. Limit push to master only.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Comment thread .github/workflows/ci.yml
- name: Checkout repository
uses: actions/checkout@v4

- name: Set up QEMU
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

QEMU is OK I guess for this case, but the build will be slower than running two parallel jobs on native architectures and merging them as I mention in the issue.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is build speed an issue? I prefer this because keeps the CI yaml simple but I am happy to change as well.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it depends. 😅

If we keep running the image on our own computer and we might need to change the code if something happens, I'd prefer a faster buidl.
However, I can imagine having this docker image running on rf and provide an interface where we could pull the data and generate the statistics remotely, without having to use our laptop (took at least 30 minutes on my m2).

This would need some more work, but if so we won't care about build time IMHO.

Copy link
Copy Markdown
Contributor

@lfoppiano lfoppiano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please trigger a build so that I can test also the resulting image? I will try to process the stats for this month to compare the speed with an non-arm version 🙏

@malteos
Copy link
Copy Markdown
Contributor Author

malteos commented Mar 23, 2026

Could you please trigger a build so that I can test also the resulting image? I will try to process the stats for this month to compare the speed with an non-arm version 🙏

It's already built: https://github.com/commoncrawl/cc-crawl-statistics/pkgs/container/cc-crawl-statistics%2Fstats/703032387?tag=pr-28

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Container image is built only for linux/amd64

2 participants