Build command doesn't work in some scenarios with containerd image store

[mthalman]

While attempting to fix #1516 I was trying to run my fix locally with the configuration that would verify it was working (i.e. image caching was enabled). It ended up not working correctly because some of the logic in Image Builder was not getting the expected results from the locally built image.

This logic expects that a locally built image will not have a digest associated with it from the docker inspect command:

docker-tools/src/Microsoft.DotNet.ImageBuilder/src/IManifestService.cs

Lines 41 to 47 in 008a7cf

	IEnumerable<string> digests = DockerHelper.GetImageDigests(image, isDryRun);
	// A digest will not exist for images that have been built locally or have been manually installed
	if (!digests.Any())
	{
	return null;
	}

This turns out not to be the case for my version of Docker Desktop. In other words, the image does have a digest for the locally built image. This causes issues later on in that method because it will attempt to verify the digest reflects what is the registry (because it assumes it is not a locally built image at this point):

docker-tools/src/Microsoft.DotNet.ImageBuilder/src/IManifestService.cs

Lines 49 to 62 in 008a7cf

	string digestSha = await GetManifestDigestShaAsync(image, isDryRun);
	if (digestSha is null)
	{
	return null;
	}
	string digest = DockerHelper.GetDigestString(DockerHelper.GetRepo(image), digestSha);
	if (!digests.Contains(digest))
	{
	throw new InvalidOperationException(
	$"Found published digest '{digestSha}' for tag '{image}' but could not find a matching digest value from " +
	$"the set of locally pulled digests for this tag: { string.Join(", ", digests) }. This most likely means that " +
	"this tag has been updated since it was last pulled.");
	}

This then throws the exception because the digests don't match which makes sense.

So at some point, a new version of Docker (not sure if this is in the client or server) caused the digest to be created. You can see this in the build output:

This differs from the behavior of the version of Docker installed on the build agents. But this will become an issue whenever they get updated to whichever version this behavior was introduced.

For reference, here's my docker info output for my Docker Desktop installation:

Client:
 Version:    27.3.1
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  ai: Ask Gordon - Docker Agent (Docker Inc.)
    Version:  v0.1.0
    Path:     C:\Users\mthalman\.docker\cli-plugins\docker-ai.exe
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.18.0-desktop.2
    Path:     C:\Users\mthalman\.docker\cli-plugins\docker-buildx.exe
  compose: Docker Compose (Docker Inc.)
    Version:  v2.30.3-desktop.1
    Path:     C:\Users\mthalman\.docker\cli-plugins\docker-compose.exe
  debug: Get a shell into any image or container (Docker Inc.)
    Version:  0.0.37
    Path:     C:\Users\mthalman\.docker\cli-plugins\docker-debug.exe
  desktop: Docker Desktop commands (Alpha) (Docker Inc.)
    Version:  v0.0.15
    Path:     C:\Users\mthalman\.docker\cli-plugins\docker-desktop.exe
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.2
    Path:     C:\Users\mthalman\.docker\cli-plugins\docker-dev.exe
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.27
    Path:     C:\Users\mthalman\.docker\cli-plugins\docker-extension.exe
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.5
    Path:     C:\Users\mthalman\.docker\cli-plugins\docker-feedback.exe
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.4.0
    Path:     C:\Users\mthalman\.docker\cli-plugins\docker-init.exe
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     C:\Users\mthalman\.docker\cli-plugins\docker-sbom.exe
  scout: Docker Scout (Docker Inc.)
    Version:  v1.15.0
    Path:     C:\Users\mthalman\.docker\cli-plugins\docker-scout.exe

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 22
 Server Version: 27.3.1
 Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 nvidia runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 472731909fa34bd7bc9c087e4c27943f9835f111
 runc version: v1.1.13-0-g58aa920
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
 Kernel Version: 5.15.167.4-microsoft-standard-WSL2
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 31.32GiB
 Name: docker-desktop
 ID: 853347f3-b3ff-446d-a665-c790a3d9585d
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=npipe://\\.\pipe\docker_cli
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[lbussell]

I have a few questions:

1. Doesn't every image have a digest?
2. Are you using containerd image store on Docker Desktop? I think it may behave more like a "local registry".
3. Are we really, actually using BuildKit in CI? Can we confirm this? Or, does disabling BuildKit locally change the behavior you see?

[mthalman]

Doesn't every image have a digest?

The behavior of Docker on the build agents is that a digest will not show up in the docker inspect output until the image gets pushed to a registry.

Are you using containerd image store on Docker Desktop? I think it may behave more like a "local registry".

Nice. That's the cause of this. I disabled it and I no longer get that behavior.

[lbussell]

[Triage] We should look into if containerd image store will affect our Docker Linux installations in our build agents (or the ImageBuilder container image) in the future.

[lbussell]

I looked into this. containerd image store is not specific to Docker Desktop. It is a replacement for the overlay2 Docker storage driver. It looks like it will eventually be the default driver in Docker Engine, and it is already the default in new installations of Docker Desktop.

You can see the current storage driver used in our builds in CI thanks to our logging: https://dev.azure.com/dnceng/internal/_build/results?buildId=2602863&view=logs&j=fc59f0f2-c1bd-58ae-b870-833d1e8a924c&t=8d53baa0-ff68-5ea5-041a-af0e08303d7f&l=101

Server:
...
 Server Version: 27.3.1-1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: false
  userxattr: false
...

Notably, this depends on the host OS and not our ImageBuilder image.

We should plan for this eventual transition and migrate away from using the presence of a digest for determining if an image was locally built or not.