Combining --output type=oci and type=docker results in invalid OCI image

[SpecLad]

Create Dockerfile:

FROM scratch

Run:

# buildctl build --output type=oci,dest=/tmp/a,tar=false --output type=docker,dest=/tmp/a.tar --frontend=dockerfile.v0 --local dockerfile=. --local context=.
[+] Building 0.1s (4/4) FINISHED
 => [internal] load build definition from Dockerfile                                                               0.0s
 => => transferring dockerfile: 50B                                                                                0.0s
 => [internal] load .dockerignore                                                                                  0.0s
 => => transferring context: 2B                                                                                    0.0s
 => exporting to docker image format                                                                               0.0s
 => => exporting layers                                                                                            0.0s
 => => exporting manifest sha256:3a58ad119f0dae830e39299dd9d94100e3a764665fba8810f9a873d3ac716aec                  0.0s
 => => exporting config sha256:471a1b8817eefb6569017c1a76f288e0d4e5c8476eb199485c469d0b033168bf                    0.0s
 => => sending tarball                                                                                             0.0s
 => exporting to oci image format                                                                                  0.0s
 => => exporting manifest sha256:6cddc7c291e3d05059404a670e53a4e87166bd3f22ea50b9ae60f9d0c4423775                  0.0s
 => => exporting config sha256:471a1b8817eefb6569017c1a76f288e0d4e5c8476eb199485c469d0b033168bf

Now examine the resulting image. First, the index:

$ jq -r .manifests[0].digest /tmp/a/index.json
sha256:3a58ad119f0dae830e39299dd9d94100e3a764665fba8810f9a873d3ac716aec

Now, the blobs:

$ ls -l /tmp/a/blobs/sha256/
total 8
-r--r--r-- 1 root root 214 Nov 29 11:37 471a1b8817eefb6569017c1a76f288e0d4e5c8476eb199485c469d0b033168bf
-r--r--r-- 1 root root 288 Nov 29 11:37 6cddc7c291e3d05059404a670e53a4e87166bd3f22ea50b9ae60f9d0c4423775

As you can see, the manifest referenced by the index is nowhere to be found. This makes the image invalid.

Version information:

# buildctl --version
buildctl github.com/moby/buildkit v0.18.0 95d190ef4f18b57c717eaad703b67cb2be781ebb

For reference, if you omit --output type=docker from the original command, the index is different and references an existing blob:

$ jq -r .manifests[0].digest /tmp/a/index.json
sha256:6cddc7c291e3d05059404a670e53a4e87166bd3f22ea50b9ae60f9d0c4423775

[tonistiigi]

@jedevc @a-palchikov Any idea what this is?

[a-palchikov]

@tonistiigi I'll look into this.

[a-palchikov]

Regarding the original issue - the problem is a known limitation of exporter responses (to be more precise - a single response for all exporters) when image exporters collide on the image descriptor (last write wins). So, the parallel export is sort of incomplete in this respect.

I'll work on improving this.

[tonistiigi]

Looked into this as well and how it possibly relates to #5572

The workaround for this specific issue is changing --output type=docker,dest=/tmp/a.tar to --output type=docker,dest=/tmp/a.tar,oci-mediatypes=true. Current difference is that type=docker exporter defaults to Docker mediatypes, type=oci defaults to OCI mediatypes. That creates a difference in image digest and then due to the issue described by @a-palchikov the digest from the last exporter gets returned as result that confuses the index.json written by tar=false mode. It seems that it even works if you just reverse the order of the outputs as the tar exporter does not care about the result digest as whole tarball is generated on the daemon side.

@crazy-max Maybe switching to OCI mediatypes by default is ok for type=docker exporter as well? @thaJeztah Any idea how old Docker versions would support that? This doesn't fix the whole issue though as there are legitimate ways how different exporters can return different values, but still better to have as few changes as possible between the defaults.

@LaurentGoderre FYI, this might have an impact on your efforts for returning more descriptors for build results as it is possible for different exporters to return different sets of descriptors.