Image exporter with non-default platform fails

[rumpl]

I am using docker 24.0.1 and have enabled the containerd image store feature.

With this Dockerfile

FROM scratch

COPY / /

If I try to build it for a platform that is not my current platform the build fails. For example I am on linux/arm64 and if I try to build an amd image I get this:

$ docker buildx build -t rumpl/test --platform=linux/amd64  .
[+] Building 0.0s (5/5) FINISHED
 => [internal] load build definition from Dockerfile                                               0.0s
 => => transferring dockerfile: 98B                                                                0.0s
 => [internal] load .dockerignore                                                                  0.0s
 => => transferring context: 2B                                                                    0.0s
 => [internal] load build context                                                                  0.0s
 => => transferring context: 1.25kB                                                                0.0s
 => CACHED [1/1] COPY / /                                                                          0.0s
 => ERROR exporting to image                                                                       0.0s
 => => exporting layers                                                                            0.0s
 => => exporting manifest sha256:12495f847e4e14c984374a13333abf65712cc8a56e57395e1dfa31f392ab2aec  0.0s
 => => exporting config sha256:71b85bc6f39a0141c5a59fea43012ae942c5fb9229bc55941cf58bd550559154    0.0s
 => => exporting attestation manifest sha256:29a43a54c9c28952fc1824c2c6f094c6dfe2f713d8e52fb12f20  0.0s
 => => exporting manifest list sha256:32dfceebd3fa6d06b49f60e42a5f672913befefa1d288e05d107dad8e55  0.0s
 => => naming to docker.io/rumpl/test:latest                                                       0.0s
 => => unpacking to docker.io/rumpl/test:latest                                                    0.0s
------
 > exporting to image:
------
ERROR: failed to solve: no match for platform in manifest sha256:32dfceebd3fa6d06b49f60e42a5f672913befefa1d288e05d107dad8e555bc0f: not found

It looks like the image exporter (the default one used when the containerd image store is enabled) is trying to only get the manifest for the current platform when unpacking:

buildkit/exporter/containerimage/export.go

Line 372 in 48e0528

manifest, err := images.Manifest(ctx, contentStore, img.Target, platforms.Default())

[tonistiigi]

@rumpl What's the expected behavior? Should it unpack all architectures? Or maybe moby should set unpack=false and issue unpack on demand when a container for a specific runtime architecture is created?

[rumpl]

I would like the build not to fail. moby now does check if the image is unpacked before running so it setting unpack=false might work

[rumpl]

Here's what happens if I force unpack=false in moby

$ docker buildx build -t rumpl/test --platform=linux/amd64  --load .
[+] Building 0.2s (5/5) FINISHED
 => [internal] load .dockerignore                                                                                           0.0s
 => => transferring context: 2B                                                                                             0.0s
 => [internal] load build definition from Dockerfile                                                                        0.0s
 => => transferring dockerfile: 60B                                                                                         0.0s
 => [internal] load build context                                                                                           0.0s
 => => transferring context: 60B                                                                                            0.0s
 => [1/1] COPY / /                                                                                                          0.0s
 => exporting to image                                                                                                      0.1s
 => => exporting layers                                                                                                     0.1s
 => => exporting manifest sha256:076fa194ad3ee4b081a235162e77f40a4ed6fc9b2b23cd07c868779d7f95f354                           0.0s
 => => exporting config sha256:094c2bc1a92a1600c7a55cc23c3085550d3a9db4220b6442f66d1b2ea22312d1                             0.0s
 => => exporting attestation manifest sha256:40c58bc71013e0b29c978bbb5a1b9b8eac035f2127f6626e49234a0c746a06be               0.0s
 => => exporting manifest list sha256:da2ffa0139834461fa0efa258c40e60959a4e3f731ee91e7ccbccfe880ec5c2a                      0.0s
 => => naming to docker.io/rumpl/test:latest                                                                                0.0s
$ docker images
REPOSITORY   TAG       IMAGE ID       CREATED         SIZE
rumpl/test   latest    da2ffa013983   3 seconds ago   1.13kB
$ sudo ctr -n moby i check
ERRO[0000] unable to check unpack for docker.io/rumpl/test:latest  error="no match for platform in manifest sha256:da2ffa0139834461fa0efa258c40e60959a4e3f731ee91e7ccbccfe880ec5c2a: not found"
REF                         TYPE                                    DIGEST                                                                  STATUS            SIZE    UNPACKED
docker.io/rumpl/test:latest application/vnd.oci.image.index.v1+json sha256:da2ffa0139834461fa0efa258c40e60959a4e3f731ee91e7ccbccfe880ec5c2a unavailable (0/?) 0.0 B/? false
ctr: unable to check unpack for docker.io/rumpl/test:latest: no match for platform in manifest sha256:da2ffa0139834461fa0efa258c40e60959a4e3f731ee91e7ccbccfe880ec5c2a: not found

Will try and see what is happening here

[jedevc]

So I had a chat with @vvoland to help me sort through what was going on 🎉

I think there are 3 distinct issues:

1. Moby failed to run images that were in the store, but were not unpacked. So if we ended up with an unpacked image from unpack=true, then moby wouldn't run it - this is fixed in c8d: Fix re-pull of an image when the snapshotter is changed moby#45647, which means that images can be unpacked on-demand on container creation. This helps resolve the issue from --load with moby containerd store should use the oci exporter docker/buildx#1813 (comment).

2. BuildKit only unpacks single-platform images. The conditional checks here only check for src.Ref,

   buildkit/exporter/containerimage/export.go

   Lines 273 to 277 in 5b9a9ce

   	if src.Ref != nil && e.unpack {
   	if err := e.unpackImage(ctx, img, src, session.NewGroup(sessionID)); err != nil {
   	return nil, nil, err
   	}
   	}

   , so multi-platform images are not unpacked. I'll open a BuildKit PR to fix this.

3. BuildKit always unpacks the default platform of the buildkit host. This doesn't always match up, which causes the no match for platform in manifest error originally reported. I'm not sure what the default should be though.

   Because of c8d: Fix re-pull of an image when the snapshotter is changed moby#45647, we can avoid throwing this error, since moby should instead unpack lazily on run, so a failure here isn't fatal. However, ideally we would still pick the "right" platform. I don't think there's a way for the buildkit server to get the client platform, so we could default to the specified platform if there is only one, otherwise select the server platform if it's in the list of platforms, and then... maybe just pick the first one? I'll try and fix this in another BuildKit PR.