Buildkit 0.20.0 may add a "variant" field, which alters the image digest

[apyrgio]

Contributing guidelines and issue reporting guide

• I've read the contributing guidelines and wholeheartedly agree. I've also read the issue reporting guide.

Well-formed report checklist

• I have found a bug that the documentation does not mention anything about my problem
• I have found a bug that there are no open or closed issues that are related to my problem
• I have provided version/information about my environment and done my best to provide a reproducer

Description of bug

Bug description

Images built with Buildkit 0.20.0 may differ from the ones built with Buildkit 0.19.0, because a new "variant" field may be added in the config blob of the image. Personally, I've seen this happen only when building ARM images.

Here's a side by side diff of two config blobs. Left is Buildkit v0.19.0, and right is Buildkit v0.20.0:

diff -y config.b19 config.b20
{                                                               {
  "architecture": "arm64",                                        "architecture": "arm64",
  "config": {                                                     "config": {
    "Env": [                                                        "Env": [
      "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin         "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
      "DEBIAN_FRONTEND=noninteractive"                                "DEBIAN_FRONTEND=noninteractive"
    ],                                                              ],
    "Cmd": [                                                        "Cmd": [
      "bash"                                                          "bash"
    ]                                                               ]
  },                                                              },
  "created": "2023-02-28T21:21:00Z",                              "created": "2023-02-28T21:21:00Z",
  "history": [                                                    "history": [
    {                                                               {
      "created": "2023-09-07T00:39:39.206776845Z",                    "created": "2023-09-07T00:39:39.206776845Z",
      "created_by": "/bin/sh -c #(nop) ADD file:fb5c8f411c4a1         "created_by": "/bin/sh -c #(nop) ADD file:fb5c8f411c4a1
    },                                                              },
    {                                                               {
      "created": "2023-09-07T00:39:39.650056081Z",                    "created": "2023-09-07T00:39:39.650056081Z",
      "created_by": "/bin/sh -c #(nop)  CMD [\"bash\"]",              "created_by": "/bin/sh -c #(nop)  CMD [\"bash\"]",
      "empty_layer": true                                             "empty_layer": true
    },                                                              },
    {                                                               {
      "created": "2023-02-28T21:21:00Z",                              "created": "2023-02-28T21:21:00Z",
      "created_by": "ENV DEBIAN_FRONTEND=noninteractive",             "created_by": "ENV DEBIAN_FRONTEND=noninteractive",
      "comment": "buildkit.dockerfile.v0",                            "comment": "buildkit.dockerfile.v0",
      "empty_layer": true                                             "empty_layer": true
    },                                                              },
    {                                                               {
      "created": "2023-02-28T21:21:00Z",                              "created": "2023-02-28T21:21:00Z",
      "created_by": "RUN /bin/sh -c repro-sources-list.sh &&          "created_by": "RUN /bin/sh -c repro-sources-list.sh && 
      "comment": "buildkit.dockerfile.v0"                             "comment": "buildkit.dockerfile.v0"
    }                                                               }
  ],                                                              ],
  "os": "linux",                                                  "os": "linux",
  "rootfs": {                                                     "rootfs": {
    "type": "layers",                                               "type": "layers",
    "diff_ids": [                                                   "diff_ids": [
      "sha256:341de903723838835820feffc6ff0ab04ea7ffddd3886cd         "sha256:341de903723838835820feffc6ff0ab04ea7ffddd3886cd
      "sha256:cdc84d7fdc4c77e4965366c040d1e0aae36b6adba12d7e2         "sha256:cdc84d7fdc4c77e4965366c040d1e0aae36b6adba12d7e2
    ]                                                               ]
  }                                                           |   },
                                                              >   "variant": "v8"
}                                                               }

Purpose of this bug report:

• Let devs know about this change, in case it's not an intended one
• Inform people that they can't reproduce bit-for-bit the same container image with the newest Buildkit version
  • I haven't tried Buildkit v0.18.0, so I don't know how stable image reproducibility is across Buildkit versions

Reproduction

Dockerfile: https://github.com/reproducible-containers/repro-sources-list.sh/blob/master/Dockerfile.debian-12

Command to build image with Buildkit 0.19.0:

$ docker buildx create \
    --name repro-build-0cdfa98a55df0f2f2a59e0f473a648f209a9d0fe83c04baa91a19931022a8d27 \
    --driver-opt image=docker.io/moby/buildkit:v0.19.0@sha256:14aa1b4dd92ea0a4cd03a54d0c6079046ea98cd0c0ae6176bdd7036ba370cbbe
$ docker buildx \
    --builder repro-build-0cdfa98a55df0f2f2a59e0f473a648f209a9d0fe83c04baa91a19931022a8d27 \
    build \
    --build-arg SOURCE_DATE_EPOCH=1677619260 \
    --provenance false \
    --output type=docker,dest=image.tar,rewrite-timestamp=true \
    -f Dockerfile.debian-12 \
    --platform linux/arm64
    .

Command to build image with Buildkit 0.20.0:

$ docker buildx create \
    --name repro-build-70d1bcdd371beedd0103a50e0d51a41627ea0ca45509fb6543675d8197a5b6bc \
    --driver-opt image=docker.io/moby/buildkit:v0.20.0
$ docker buildx \
    --builder repro-build-70d1bcdd371beedd0103a50e0d51a41627ea0ca45509fb6543675d8197a5b6bc \
    build \
    --build-arg SOURCE_DATE_EPOCH=1677619260 \
    --provenance false \
    --output type=docker,dest=image.tar,rewrite-timestamp=true \
    -f Dockerfile.debian-12 \
    --platform linux/arm64
    .

If you untar the image.tar tarballs, and check the config blog, you'll see that the "variant": "v8" field is present only in the tarball produced by Buildkit v0.20.0.

Version information

I have reproduced this both locally and in GitHub actions (docker/setup-buildx-action@v3). Here's the version information of the latter:

Docker info:

/usr/bin/docker version
  Client: Docker Engine - Community
   Version:           26.1.3
   API version:       1.45
   Go version:        go1.21.10
   Git commit:        b72abbb
   Built:             Thu May 16 08:40:13 2024
   OS/Arch:           linux/arm64
   Context:           default
  
  Server: Docker Engine - Community
   Engine:
    Version:          26.1.3
    API version:      1.45 (minimum version 1.24)
    Go version:       go1.21.10
    Git commit:       8e96db1
    Built:            Thu May 16 08:40:13 2024
    OS/Arch:          linux/arm64
    Experimental:     false
   containerd:
    Version:          1.7.25
    GitCommit:        bcc810d6b9066471b0b6fa75f557a15a1cbf31bb
   runc:
    Version:          1.2.4
    GitCommit:        v1.2.4-0-g6c52b3f
   docker-init:
    Version:          0.19.0
    GitCommit:        de40ad0
  /usr/bin/docker info
  Client: Docker Engine - Community
   Version:    26.1.3
   Context:    default
   Debug Mode: false
   Plugins:
    buildx: Docker Buildx (Docker Inc.)
      Version:  v0.20.0
      Path:     /usr/libexec/docker/cli-plugins/docker-buildx
    compose: Docker Compose (Docker Inc.)
      Version:  v2.27.1
      Path:     /usr/libexec/docker/cli-plugins/docker-compose
  
  Server:
   Containers: 0
    Running: 0
    Paused: 0
    Stopped: 0
   Images: 0
   Server Version: 26.1.3
   Storage Driver: overlay2
    Backing Filesystem: extfs
    Supports d_type: true
    Using metacopy: false
    Native Overlay Diff: false
    userxattr: false
   Logging Driver: json-file
   Cgroup Driver: systemd
   Cgroup Version: 2
   Plugins:
    Volume: local
    Network: bridge host ipvlan macvlan null overlay
    Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
   Swarm: inactive
   Runtimes: io.containerd.runc.v2 runc
   Default Runtime: runc
   Init Binary: docker-init
   containerd version: bcc810d6b9066471b0b6fa75f557a15a1cbf31bb
   runc version: v1.2.4-0-g6c52b3f
   init version: de40ad0
   Security Options:
    apparmor
    seccomp
     Profile: builtin
    cgroupns
   Kernel Version: 6.8.0-1020-azure
   Operating System: Ubuntu 24.04.1 LTS
   OSType: linux
   Architecture: aarch64
   CPUs: 4
   Total Memory: 15.58GiB
   Name: fv-az2130-671
   ID: 91b18afc-55ea-45ee-9a5e-ab3c5c3e87d5
   Docker Root Dir: /var/lib/docker
   Debug Mode: false
   Username: githubactions
   Experimental: false
   Insecure Registries:
    127.0.0.0/8
   Live Restore Enabled: false

[tonistiigi]

Fix in #5776 . This is defined in Dockerfile frontend, so if you want the previous behavior, you can use #syntax=docker/dockerfile:1.13