Image build with secret mounts stopped working v1.32

[JustKiddingCode]

Description

We have a gitlab-ci with a gitlab-runner (running with podman). The runner has CAP_SYS_ADMIN privileges.
Building the containers worked for buildah version from 1.29 to 1.31 but stopped working with buildah 1.32.
The CI fails with:
error running subprocess: remounting "/var/tmp/buildah4082641103/mnt/rootfs/run/secrets/Test1" in mount namespace with flags 0x1 instead of 0x0: operation not permitted

Steps to reproduce the issue:

1. Dockerfile

FROM docker.io/library/debian:bullseye
MAINTAINER Konstantin Zangerle <[email protected]>

CMD [ "/bin/bash" ]
ENTRYPOINT []
ENV DEBIAN_FRONTEND=noninteractive

RUN --mount=type=secret,id=Test1 --mount=type=secret,id=Test2 findmnt && ls -lahZ /run/secrets;  \
	cat /run/secrets/Test1

3. Gitlab-CI

image: quay.io/buildah/stable:v1.32

#before_script:
#  - podman login -u "${HUB_USER}" -p "${HUB_PASS}" hub.scc.kit.edu:5000

build:
  stage: build
  script:
    - buildah --version
    - buildah info
    - findmnt
    - echo -n "einStringstehtindergitlabci" > /builds/REPO_USER
    - echo $Test2 > /builds/REPO_PW
    - head /builds/REPO*
    - buildah bud --log-level debug
      --secret id=Test1,src=Test1,type=env
      --secret id=Test2,src=Test2,type=env
      --no-cache .
    - head /builds/REPO*

Describe the results you received:

Buildah fails with error running subprocess: remounting "/var/tmp/buildah4082641103/mnt/rootfs/run/secrets/Test1" in mount namespace with flags 0x1 instead of 0x0: operation not permitted

Describe the results you expected:

Buildah builds image like v1.31 without failing.

Buildah information

$ buildah --version
buildah version 1.32.2 (image-spec 1.1.0-rc.4, runtime-spec 1.1.0)
$ buildah info
{
    "host": {
        "CgroupVersion": "v2",
        "Distribution": {
            "distribution": "fedora",
            "version": "39"
        },
        "MemFree": 1488[18](https://git.scc.kit.edu/KIT-CERT/test-gitlab-ci-buildah/-/jobs/1131210#L18)28864,
        "MemTotal": 33670135808,
        "OCIRuntime": "crun",
        "SwapFree": 0,
        "SwapTotal": 0,
        "arch": "amd64",
        "cpus": 16,
        "hostname": "runner-rxsnej93-project-34890-concurrent-0",
        "kernel": "6.1.0-13-cloud-amd64",
        "os": "linux",
        "rootless": true,
        "uptime": "890h 48m 16.69s (Approximately 37.08 days)",
        "variant": ""
    },
    "store": {
        "ContainerStore": {
            "number": 4
        },
        "GraphDriverName": "overlay",
        "GraphOptions": [
            "overlay.imagestore=/var/lib/shared",
            "overlay.mount_program=/usr/bin/fuse-overlayfs",
            "overlay.mountopt=nodev,fsync=0"
        ],
        "GraphRoot": "/var/lib/containers/storage",
        "GraphStatus": {
            "Backing Filesystem": "extfs",
            "Native Overlay Diff": "false",
            "Supports d_type": "true",
            "Supports shifting": "true",
            "Supports volatile": "true",
            "Using metacopy": "false"
        },
        "ImageStore": {
            "number": [29](https://git.scc.kit.edu/KIT-CERT/test-gitlab-ci-buildah/-/jobs/1131210#L29)
        },
        "RunRoot": "/run/containers/storage"
    }
}

[rhatdan]

@flouthoc PTAL

[flouthoc]

@JustKiddingCode Could you please try upstream version of buildah quay.io/buildah/upstream:latest, I suspect this involves a fix which was made here #5083

[JustKiddingCode]

@JustKiddingCode Could you please try upstream version of buildah quay.io/buildah/upstream:latest, I suspect this involves a fix which was made here #5083

Still same issue with:

$ buildah --version
buildah version 1.33.2-dev (image-spec 1.1.0-rc.5, runtime-spec 1.1.0)

[blue42u]

I'm also experiencing this issue, it seems Buildah 1.32.0 does not have this issue while 1.32.2 does. Which would point to #5083 being the cause of this regression.

Command to reproduce the error locally:

podman run --rm -it --device /dev/fuse -v ./Dockerfile:/Dockerfile:ro quay.io/buildah/stable:v1.32.2 \
  buildah build --secret id=Test1,src=Test1,type=env --secret id=Test2,src=Test2,type=env

[flouthoc]

I think @nalind can help here if you have spotted that #5083 is causing regression.

[poettig]

Is there any progress with this? This is still an issue for us which forces us to keep using buildah 1.31.

[wesleyeleazar]

I can also confirm the same failure message, running buildah 1.33.7.
No such error on 1.31.5. Can't try any releases in between while following Red Hat's rhel8/buildah container releases.

[jonahbull]

I put up #5544 which has an attempt at fixing this along with a test that reproduces the issue.