Consume Docker Hub upstream SBOMs and merge with Syft

[vpetersson]

Summary

• Detect Docker Official Images (library/*) and Docker Hardened Images (dhi.io/*) — either directly via DOCKER_IMAGE or via BuildKit SLSA provenance on a user-built image — and fetch the publisher's SPDX SBOM.
• Merge that upstream SBOM with a local Syft scan of the same image: upstream wins for base-layer packages, Syft fills gaps and overlays packages installed by the user's Dockerfile (apt, pip, COPY, …). Components are tagged with a sbomify:source property (docker-hub-upstream or syft-overlay) for auditability.
• Factor the crane/cosign + in-toto walking code out of chainguard.py into a shared buildkit_provenance module so both detectors use the same primitives. Fixes a latent bug along the way: multi-arch indexes have one attestation-manifest sibling per platform, and the previous code sometimes pulled the wrong one (or nothing) when given a per-platform digest.
• Surface registry failures clearly — Docker Hub rate limits (100 anonymous pulls / 6h), 401s, and 404s now log at WARNING with the concrete remediation (docker login, docker login dhi.io, etc.) instead of silently falling back.

Full pipeline verified end-to-end with --augment (local sbomify.json) and --enrich: the merge output passes through both stages unmodified in shape, source tags preserved.

Dedup keys on the PURL's core identity, ignoring qualifiers (Docker's os_distro=trixie&os_name=debian vs Syft's arch=amd64&distro=debian-13 both collapse to the same package). A second-pass loose match (type + name + version, namespace ignored) catches the cases where the two generators disagree on namespace — for example, Amazon Linux's upstream SBOM emits pkg:rpm/amazonlinux/bash while Syft emits pkg:rpm/amzn/bash. Safe because a merge always describes a single image, so the distro is fixed.

Test plan

Unit coverage — 79 new tests, full suite 2257 passing, ruff clean:

• tests/test_buildkit_provenance.py — crane/cosign helpers, rate-limit/401/404 classifier
• tests/test_dockerhub.py — direct + provenance detection for library/*, dhi.io/*, ref classification, multi-arch sibling matching
• tests/test_sbom_merge.py — strict dedup, qualifier-ignore fallback, namespace-mismatch fallback, relationship/extracted-license carry-over
• tests/test_chainguard.py — existing tests rewired to the shared module; no behavioral change

Runnable E2E (examples/docker-hub/run-e2e.sh — 7 scenarios):

• python:3.11-slim direct, CycloneDX — 142 upstream + 2721 overlay
• python:3.11-slim direct, SPDX — 157 packages, 72 extracted licenses, 0 validation errors
• Distro sweep: alpine:3.20, ubuntu:24.04, rockylinux:9, amazonlinux:2, archlinux:latest, busybox:latest — all detected, merged, deduped across pkg:deb/pkg:apk/pkg:rpm/pkg:alpm/pkg:generic
• Built FROM python:3.11-slim + apt/pip, CycloneDX — 142 upstream + 2852 overlay, requests/click/curl/jq all present as syft-overlay
• Same, SPDX — 188 packages, 105 extracted licenses, 0 validation errors
• Full merge + --augment (local sbomify.json) + --enrich — supplier, authors, lifecycle applied on top of merged SBOM
• DHI end-to-end — skipped in this environment (dhi.io needs docker login dhi.io and account entitlement). Unit-tested cosign shape includes --key https://registry.scout.docker.com/keyring/dhi/latest.pub and --insecure-ignore-tlog=true; the keyring URL is verified publicly reachable.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Adds support for consuming Docker Hub–published upstream SPDX SBOMs (Docker Official Images and DHI), then merging them with a local Syft scan so base-layer packages prefer publisher metadata while user-installed packages are overlaid.

Changes:

• Introduces shared BuildKit attestation/provenance helpers (crane/cosign + in-toto walking) and rewires Chainguard detection to use them.
• Implements Docker Hub image detection (direct + provenance) and upstream SBOM retrieval (crane for Official Images, cosign for DHI).
• Adds CycloneDX + SPDX merge logic (upstream-wins with Syft gap-filling) and integrates it into the CLI pipeline, with extensive unit + E2E examples.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
sbomify_action/cli/main.py	Integrates Docker Hub detection/fetch + Syft overlay merge into the main generation pipeline.
sbomify_action/_generation/buildkit_provenance.py	New shared primitives for resolving platform digests and fetching BuildKit/cosign attestations.
sbomify_action/_generation/dockerhub.py	New Docker Hub Official/DHI detection and upstream SPDX SBOM fetch.
sbomify_action/_generation/sbom_merge.py	New upstream-wins merge logic for CycloneDX and SPDX documents.
sbomify_action/_generation/chainguard.py	Refactors Chainguard detector to use shared BuildKit provenance helpers.
tests/test_buildkit_provenance.py	Unit tests for shared BuildKit/crane/cosign helpers and registry error classification.
tests/test_dockerhub.py	Unit tests for Docker Hub detection paths and upstream SBOM retrieval behavior.
tests/test_sbom_merge.py	Unit tests for merge semantics (dedup, fill-empty, collision rewrites, relationship/licensing carry-over).
tests/test_chainguard.py	Updates mocks/patches to point at the extracted shared helper module.
examples/docker-hub/run-e2e.sh	Adds an end-to-end runner script covering official/provenance/DHI scenarios.
examples/docker-hub/README.md	Documents the E2E scenarios and how to run them locally.
examples/docker-hub/Dockerfile.official	Example derivative image to exercise provenance-based detection + overlay behavior.
examples/docker-hub/Dockerfile.dhi	Example derivative image for DHI scenarios (best-effort).
examples/docker-hub/hello.py	Minimal app used by the Dockerfiles/E2E runner.
README.md	Documents Docker Hub SBOM reuse/merge feature and updates required-tools notes.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.