devcontainer up not working with podman and docker-compose file

[Panaetius]

With this devcontainer.json, devcontainer cli 0.66.0 and podman the image seems to build fine mostly but then the cli will prepend localhost to the image name and the build fails.

Command executed: devcontainer up --buildkit never --workspace-folder .
CLI version: 0.66.0 (also with 0.59.1)
System: Arch Linux
Podman: 5.1.2

Logs:

[...]
[4/4] STEP 32/32: LABEL "com.docker.compose.image.builder"="classic"
[4/4] COMMIT docker.io/library/vsc-renku-data-services-34f67da2ae2e898466e7b938e43785c2322033e1d9fbed1ea7c91813ff9ad829:latest
--> b93856cae65a
[Warning] one or more build args were not consumed: [_DEV_CONTAINERS_FEATURE_CONTENT_SOURCE]
Successfully tagged docker.io/library/vsc-renku-data-services-34f67da2ae2e898466e7b938e43785c2322033e1d9fbed1ea7c91813ff9ad829:latest
b93856cae65a07925056a02be038297c9959607ad139bf21dd84ed722cfb90ee
Successfully built b93856cae65a
Successfully tagged vsc-renku-data-services-34f67da2ae2e898466e7b938e43785c2322033e1d9fbed1ea7c91813ff9ad829
STEP 1/9: FROM localhost/vsc-renku-data-services-34f67da2ae2e898466e7b938e43785c2322033e1d9fbed1ea7c91813ff9ad829
Trying to pull localhost/vsc-renku-data-services-34f67da2ae2e898466e7b938e43785c2322033e1d9fbed1ea7c91813ff9ad829:latest...
WARN[0000] Failed, retrying in 2s ... (1/3). Error: initializing source docker://localhost/vsc-renku-data-services-34f67da2ae2e898466e7b938e43785c2322033e1d9fbed1ea7c91813ff9ad829:latest: pinging container registry localhost: Get "https://localhost/v2/": dial tcp [::1]:443: connect: connection refused
WARN[0002] Failed, retrying in 2s ... (2/3). Error: initializing source docker://localhost/vsc-renku-data-services-34f67da2ae2e898466e7b938e43785c2322033e1d9fbed1ea7c91813ff9ad829:latest: pinging container registry localhost: Get "https://localhost/v2/": dial tcp [::1]:443: connect: connection refused
WARN[0004] Failed, retrying in 2s ... (3/3). Error: initializing source docker://localhost/vsc-renku-data-services-34f67da2ae2e898466e7b938e43785c2322033e1d9fbed1ea7c91813ff9ad829:latest: pinging container registry localhost: Get "https://localhost/v2/": dial tcp [::1]:443: connect: connection refused
Error: creating build container: initializing source docker://localhost/vsc-renku-data-services-34f67da2ae2e898466e7b938e43785c2322033e1d9fbed1ea7c91813ff9ad829:latest: pinging container registry localhost: Get "https://localhost/v2/": dial tcp [::1]:443: connect: connection refused
Error: Command failed: docker build -f /tmp/devcontainercli-zenon/updateUID.Dockerfile-0.66.0 -t vsc-renku-data-services-34f67da2ae2e898466e7b938e43785c2322033e1d9fbed1ea7c91813ff9ad829-uid --platform linux/amd64 --build-arg BASE_IMAGE=localhost/vsc-renku-data-services-34f67da2ae2e898466e7b938e43785c2322033e1d9fbed1ea7c91813ff9ad829 --build-arg REMOTE_USER=vscode --build-arg NEW_UID=1000 --build-arg NEW_GID=1000 --build-arg IMAGE_USER=root /tmp/devcontainercli-zenon/empty-folder
    at utA (/usr/lib/node_modules/@devcontainers/cli/dist/spec-node/devContainersSpecCLI.js:410:3925)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async KtA (/usr/lib/node_modules/@devcontainers/cli/dist/spec-node/devContainersSpecCLI.js:482:3944)
    at async eB (/usr/lib/node_modules/@devcontainers/cli/dist/spec-node/devContainersSpecCLI.js:482:4886)
    at async hrA (/usr/lib/node_modules/@devcontainers/cli/dist/spec-node/devContainersSpecCLI.js:663:200)
    at async lrA (/usr/lib/node_modules/@devcontainers/cli/dist/spec-node/devContainersSpecCLI.js:662:13452)
{"outcome":"error","message":"Command failed: docker build -f /tmp/devcontainercli-zenon/updateUID.Dockerfile-0.66.0 -t vsc-renku-data-services-34f67da2ae2e898466e7b938e43785c2322033e1d9fbed1ea7c91813ff9ad829-uid --platform linux/amd64 --build-arg BASE_IMAGE=localhost/vsc-renku-data-services-34f67da2ae2e898466e7b938e43785c2322033e1d9fbed1ea7c91813ff9ad829 --build-arg REMOTE_USER=vscode --build-arg NEW_UID=1000 --build-arg NEW_GID=1000 --build-arg IMAGE_USER=root /tmp/devcontainercli-zenon/empty-folder","description":"An error occurred setting up the container."}

I think the key line in the log it where it tags vsc-renku-data-services-34f67da2ae2e898466e7b938e43785c2322033e1d9fbed1ea7c91813ff9ad829 but then tries to pull from localhost/vsc-renku-data-services-34f67da2ae2e898466e7b938e43785c2322033e1d9fbed1ea7c91813ff9ad829 (I don't have a local registry).

I tried with 0.66.0, 0.65.0, 0.59.1 and all have this issue.

maybe related to microsoft/vscode-remote-release#9748 ? at least one user there had the same issue, but they actually had a custom registry.

[samruddhikhandale]

Looping in @chrmarti for his thoughts, thanks!

[chrmarti]

I agree, the fix for microsoft/vscode-remote-release#9748 seems to cause this.

Could you try with v0.58.0 to confirm it works without that fix?

Maybe that fix is only needed when using buildx (seeing that you are disabling it). Does the latest version work when you allow BuildKit? (Or does that not change behavior with Podman?)

[mfontana-elem]

I'm seeing the same issue here, but when building in vscode via the extension. As far as I know I don't have any specific builder options except format docker for buildah (as an envar). Reverting to vscode-devcontainers 0.348 seems to fix the build issue.

As far as I can tell, the problem is that when I execute podman compose build the images are prepended the docker.io/library registry, whereas if I just do podman build the prepended domain is localhost. I am using docker's compose parser (not the package podman-compose), which I think it's a somewhat common set up nowadays.

My guess is that docker's compose engine is manually tagging with docker's default registry, hence the fix for podman build breaks podman compose build.

[Panaetius]

I disable buildkit due it failing otherwise (I think buildkit is not supported in podman, see containers/podman#17836 , but there's conflicting information online on podman and buildkit support).

I tried with 0.58 but that fails due to an unrelated error, which I think was fixed in 0.59. I can try again and post the log but doubt it sheds a light on this issue.

I'm not currently on that machine (still using regular Docker on my work machine) but can give try and post relevant logs later.

[Panaetius]

Here it is with buildkit:

$ DOCKER_BUILDKIT=1 devcontainer build --workspace-folder .
[...]
[8404 ms] Docker Compose override file for building image:
version: '3.8'

services:
  data_service:
    image: vsc-renku-data-services-34f67da2ae2e898466e7b938e43785c2322033e1d9fbed1ea7c91813ff9ad829
    build:
      dockerfile: /tmp/devcontainercli-zenon/container-features/0.66.0-1723021184360/Dockerfile-with-features
      context: /tmp/devcontainercli-zenon/empty-folder
      args:
        - BUILDKIT_INLINE_CACHE=1
        - _DEV_CONTAINERS_BASE_IMAGE=dev_container_auto_added_stage_label
        - _DEV_CONTAINERS_IMAGE_USER=root
        - _DEV_CONTAINERS_FEATURE_CONTENT_SOURCE=dev_container_feature_content_temp
      additional_contexts:
        - dev_containers_feature_content_source=/tmp/devcontainercli-zenon/container-features/0.66.0-1723021184360

[8405 ms] Start: Run: docker compose --project-name renku-data-services_devcontainer -f /home/zenon/DEV/ETH/renku-data-services/.devcontainer/docker-compose.yml -f /tmp/devcontainercli-zenon/docker-compose/docker-compose.devcontainer.build-1723021191395.yml build data_service
>>>> Executing external compose provider "/usr/bin/docker-compose". Please refer to the documentation for details. <<<<

WARN[0000] /home/zenon/DEV/ETH/renku-data-services/.devcontainer/docker-compose.yml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion
WARN[0000] /tmp/devcontainercli-zenon/docker-compose/docker-compose.devcontainer.build-1723021191395.yml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion
the classic builder doesn't support additional contexts, set DOCKER_BUILDKIT=1 to use BuildKit
Error: executing /usr/bin/docker-compose --project-name renku-data-services_devcontainer -f /home/zenon/DEV/ETH/renku-data-services/.devcontainer/docker-compose.yml -f /tmp/devcontainercli-zenon/docker-compose/docker-compose.devcontainer.build-1723021191395.yml build data_service: exit status 1
Error: Command failed: docker compose --project-name renku-data-services_devcontainer -f /home/zenon/DEV/ETH/renku-data-services/.devcontainer/docker-compose.yml -f /tmp/devcontainercli-zenon/docker-compose/docker-compose.devcontainer.build-1723021191395.yml build data_service
    at Km (/usr/lib/node_modules/@devcontainers/cli/dist/spec-node/devContainersSpecCLI.js:430:525)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async FrA (/usr/lib/node_modules/@devcontainers/cli/dist/spec-node/devContainersSpecCLI.js:665:3021)
    at async yrA (/usr/lib/node_modules/@devcontainers/cli/dist/spec-node/devContainersSpecCLI.js:664:4994)
{"outcome":"error","message":"Command failed: docker compose --project-name renku-data-services_devcontainer -f /home/zenon/DEV/ETH/renku-data-services/.devcontainer/docker-compose.yml -f /tmp/devcontainercli-zenon/docker-compose/docker-compose.devcontainer.build-1723021191395.yml build data_service","description":"An error occurred building the Docker Compose images."}

~/DEV/ETH/renku-data-services main 10s
[8404 ms] Docker Compose override file for building image:
version: '3.8'

services:
  data_service:
    image: vsc-renku-data-services-34f67da2ae2e898466e7b938e43785c2322033e1d9fbed1ea7c91813ff9ad829
    build:
      dockerfile: /tmp/devcontainercli-zenon/container-features/0.66.0-1723021184360/Dockerfile-with-features
      context: /tmp/devcontainercli-zenon/empty-folder
      args:
        - BUILDKIT_INLINE_CACHE=1
        - _DEV_CONTAINERS_BASE_IMAGE=dev_container_auto_added_stage_label
        - _DEV_CONTAINERS_IMAGE_USER=root
        - _DEV_CONTAINERS_FEATURE_CONTENT_SOURCE=dev_container_feature_content_temp
      additional_contexts:
        - dev_containers_feature_content_source=/tmp/devcontainercli-zenon/container-features/0.66.0-1723021184360

[8405 ms] Start: Run: docker compose --project-name renku-data-services_devcontainer -f /home/zenon/DEV/ETH/renku-data-services/.devcontainer/docker-compose.yml -f /tmp/devcontainercli-zenon/docker-compose/docker-compose.devcontainer.build-1723021191395.yml build data_service
>>>> Executing external compose provider "/usr/bin/docker-compose". Please refer to the documentation for details. <<<<

WARN[0000] /home/zenon/DEV/ETH/renku-data-services/.devcontainer/docker-compose.yml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion
WARN[0000] /tmp/devcontainercli-zenon/docker-compose/docker-compose.devcontainer.build-1723021191395.yml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion
the classic builder doesn't support additional contexts, set DOCKER_BUILDKIT=1 to use BuildKit
Error: executing /usr/bin/docker-compose --project-name renku-data-services_devcontainer -f /home/zenon/DEV/ETH/renku-data-services/.devcontainer/docker-compose.yml -f /tmp/devcontainercli-zenon/docker-compose/docker-compose.devcontainer.build-1723021191395.yml build data_service: exit status 1
Error: Command failed: docker compose --project-name renku-data-services_devcontainer -f /home/zenon/DEV/ETH/renku-data-services/.devcontainer/docker-compose.yml -f /tmp/devcontainercli-zenon/docker-compose/docker-compose.devcontainer.build-1723021191395.yml build data_service
    at Km (/usr/lib/node_modules/@devcontainers/cli/dist/spec-node/devContainersSpecCLI.js:430:525)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async FrA (/usr/lib/node_modules/@devcontainers/cli/dist/spec-node/devContainersSpecCLI.js:665:3021)
    at async yrA (/usr/lib/node_modules/@devcontainers/cli/dist/spec-node/devContainersSpecCLI.js:664:4994)
{"outcome":"error","message":"Command failed: docker compose --project-name renku-data-services_devcontainer -f /home/zenon/DEV/ETH/renku-data-services/.devcontainer/docker-compose.yml -f /tmp/devcontainercli-zenon/docker-compose/docker-compose.devcontainer.build-1723021191395.yml build data_service","description":"An error occurred building the Docker Compose images."}

I tried again with 0.58.0 to reproduce the error, but it actually works! 🤷
Tried with devcontainer up --buildkit never --build-no-cache --workspace-folder ..
So can confirm that 0.58.0 does not have this issue.

[viktorianer]

I started working with Podman today, and I can confirm that version 0.58.0 works perfectly with my Rails project on macOS, while the latest version 0.70.0 does not.

Here’s the error log when trying to build with version 0.70.0:

[3094 ms] Start: Run: docker compose --project-name example_devcontainer -f ~/Documents/projects/example/.devcontainer/compose.yaml -f /var/folders/vv/m5djvkbs6t1fr96k4l3cp66w0000gn/T/devcontainercli/docker-compose/docker-compose.devcontainer.build-1725915103822.yml build app
the classic builder doesn't support additional contexts, set DOCKER_BUILDKIT=1 to use BuildKit
Error: executing /usr/local/bin/docker-compose --project-name example_devcontainer -f ~/Documents/projects/example/.devcontainer/compose.yaml -f /var/folders/vv/m5djvkbs6t1fr96k4l3cp66w0000gn/T/devcontainercli/docker-compose/docker-compose.devcontainer.build-1725915103822.yml build app: exit status 1
Error: Command failed: docker compose --project-name example_devcontainer -f ~/Documents/projects/example/.devcontainer/compose.yaml -f /var/folders/vv/m5djvkbs6t1fr96k4l3cp66w0000gn/T/devcontainercli/docker-compose/docker-compose.devcontainer.build-1725915103822.yml build app
    at Ym (/opt/homebrew/Cellar/devcontainer/0.70.0/libexec/lib/node_modules/@devcontainers/cli/dist/spec-node/devContainersSpecCLI.js:430:525)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async MrA (/opt/homebrew/Cellar/devcontainer/0.70.0/libexec/lib/node_modules/@devcontainers/cli/dist/spec-node/devContainersSpecCLI.js:665:3021)
    at async krA (/opt/homebrew/Cellar/devcontainer/0.70.0/libexec/lib/node_modules/@devcontainers/cli/dist/spec-node/devContainersSpecCLI.js:664:4962)
{"outcome":"error","message":"Command failed: docker compose --project-name example_devcontainer -f ~/Documents/projects/example/.devcontainer/compose.yaml -f /var/folders/vv/m5djvkbs6t1fr96k4l3cp66w0000gn/T/devcontainercli/docker-compose/docker-compose.devcontainer.build-1725915103822.yml build app","description":"An error occurred building the Docker Compose images."}

To fix the issue, I uninstalled version 0.70.0 and rolled back to 0.58.0 using the following steps:

devcontainer --version
0.70.0
brew uninstall devcontainer

npm install -g @devcontainers/[email protected]
devcontainer --version
0.58.0

After downgrading, everything works as expected again.