Bootc Disk Generation fails with Logically Bound Images

[oglok]

cat /etc/redhat-release
Red Hat Enterprise Linux release 9.4 (Plow)

cat /etc/redhat-release
Red Hat Enterprise Linux release 9.4 (Plow)
[oglok@maxwell bootc-flightctl]$ podman version
Client:       Podman Engine
Version:      4.9.4-rhel
API Version:  4.9.4-rhel
Go Version:   go1.21.11 (Red Hat 1.21.11-1.el9_4)
Built:        Thu Aug  8 10:51:41 2024
OS/Arch:      linux/amd64

sudo podman images
REPOSITORY                                TAG                IMAGE ID      CREATED         SIZE
quay.io/flightctl/flightctl-api           latest             71f76706f68e  11 minutes ago  90.6 MB
quay.io/flightctl/flightctl-periodic      latest             f067c8c9f39a  11 minutes ago  85.5 MB
quay.io/flightctl/flightctl-worker        latest             10529348efc4  11 minutes ago  94.1 MB
quay.io/oglok/bootc-app-flightctl        latest             9c59c35fa727  19 hours ago    2.26 GB
quay.io/sclorg/postgresql-16-c9s          latest             6ffd15073b8c  33 hours ago    384 MB
quay.io/centos-bootc/bootc-image-builder  latest             767a10b2f63a  2 days ago      744 MB
registry.redhat.io/rhel9/rhel-bootc       9.4                7477926aeead  4 days ago      1.48 GB
quay.io/flightctl/flightctl-ui            0.2.2-19-g8eadecf  3a752147eae0  2 weeks ago     66 MB
docker.io/library/rabbitmq                3.13-management    c7383e9ad93d  4 weeks ago     258 MB
quay.io/sclorg/postgresql-12-c8s          latest             054a44a2f9ce  2 months ago    519 MB
quay.io/prometheus/prometheus             v2.54.0            34665e733a53  2 months ago    276 MB
quay.io/keycloak/keycloak                 25.0.1             79fc80eb59b4  4 months ago    435 MB

Containerfile:

FROM registry.redhat.io/rhel9/rhel-bootc:9.4

ARG USHIFT_VER=4.17
RUN dnf config-manager \
        --set-enabled rhocp-${USHIFT_VER}-for-rhel-9-$(uname -m)-rpms \
        --set-enabled fast-datapath-for-rhel-9-$(uname -m)-rpms
RUN dnf install -y microshift && \
    systemctl enable microshift && \
    dnf clean all

# Create a default 'redhat' user with the specified password.
# Add it to the 'wheel' group to allow for running sudo commands.
ARG USER_PASSWD
RUN if [ -z "${USER_PASSWD}" ] ; then \
        echo USER_PASSWD is a mandatory build argument && exit 1 ; \
    fi
RUN useradd -m -d /var/home/redhat -G wheel redhat && \
    echo "redhat:${USER_PASSWD}" | chpasswd

# Create a systemd unit to recursively make the root filesystem subtree
# shared as required by OVN images
RUN cat > /etc/systemd/system/microshift-make-rshared.service <<'EOF'
[Unit]
Description=Make root filesystem shared
Before=microshift.service
ConditionVirtualization=container
[Service]
Type=oneshot
ExecStart=/usr/bin/mount --make-rshared /
[Install]
WantedBy=multi-user.target
EOF
RUN systemctl enable microshift-make-rshared.service
RUN curl -L https://mirror.openshift.com/pub/openshift-v4/clients/helm/latest/helm-linux-amd64 -o /usr/local/bin/helm && \
    chmod +x /usr/local/bin/helm && \
    helm version

COPY usr/ usr/

RUN for i in $(ls usr/share/containers/systemd/); do ln -sr /usr/share/containers/systemd/$i /usr/lib/bootc/bound-images.d; done

ll usr/share/containers/systemd/
total 36
-rw-r--r--. 1 oglok oglok 251 oct 23 13:47 flightctl-flightctl-api.container
-rw-r--r--. 1 oglok oglok 255 oct 23 15:27 flightctl-flightctl-periodic.container
-rw-r--r--. 1 oglok oglok 260 oct 23 15:27 flightctl-flightctl-ui.container
-rw-r--r--. 1 oglok oglok 253 oct 23 15:27 flightctl-flightctl-worker.container
-rw-r--r--. 1 oglok oglok 244 oct 23 15:28 flightctl-keycloak.container
-rw-r--r--. 1 oglok oglok 251 oct 23 15:28 flightctl-postgresql-12-c8s.container
-rw-r--r--. 1 oglok oglok 251 oct 23 15:28 flightctl-postgresql-16-c9s.container
-rw-r--r--. 1 oglok oglok 249 oct 23 15:29 flightctl-prometheus.container
-rw-r--r--. 1 oglok oglok 254 oct 23 15:29 flightctl-rabbitmq.container

[oglok@maxwell bootc-flightctl]$ cat usr/share/containers/systemd/flightctl-flightctl-api.container
[Unit]
Description=Flightctl API

[Container]
PodmanArgs=--storage-opt=additionalimagestore=/usr/lib/bootc/storage
Image=quay.io/flightctl/flightctl-api:latest
# This is just a generally good practice
ReadOnly=true

[Install]
WantedBy=default.target

Podman build works fine, and the bootc image is generated. When building the qcow2 image, I get this:

⏱  Duration: 0s
org.osbuild.bootc.install-to-filesystem: 62a150870705243e880ebc4b7d898da78ccba5f5da4b61a0c3b9e2bb105b2f73 {
  "kernel-args": [
    "rw",
    "console=tty0",
    "console=ttyS0"
  ],
  "target-imgref": "quay.io/oglok/bootc-app-flightctl:latest"
}
device/disk (org.osbuild.loopback): loop3 acquired (locked: False)
mount/- (org.osbuild.xfs): mounting /dev/loop3p4 -> /store/tmp/buildroot-tmp-ymtc9fkr/mounts/
mount/boot (org.osbuild.xfs): mounting /dev/loop3p3 -> /store/tmp/buildroot-tmp-ymtc9fkr/mounts/boot
mount/boot-efi (org.osbuild.fat): mounting /dev/loop3p2 -> /store/tmp/buildroot-tmp-ymtc9fkr/mounts/boot/efi
Mount transient overlayfs for /etc/containers
Host kernel does not have SELinux support, but target enables it by default; this is less well tested.  See https://github.com/containers/bootc/issues/419
Installing image: docker://quay.io/oglok/bootc-app-flightctl:latest
ERROR Installing to filesystem: Failed to invoke skopeo proxy method OpenImage: remote error: reference "[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev,metacopy=on]quay.io/flightctl/flightctl-api:latest" does not resolve to an image ID: identifier is not an image
Traceback (most recent call last):
  File "/run/osbuild/bin/org.osbuild.bootc.install-to-filesystem", line 53, in <module>
    r = main(args["options"], args["inputs"], args["paths"])
  File "/run/osbuild/bin/org.osbuild.bootc.install-to-filesystem", line 48, in main
    subprocess.run(pargs, env=env, check=True)
  File "/usr/lib64/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['bootc', 'install', 'to-filesystem', '--source-imgref', 'containers-storage:[overlay@/run/osbuild/containers/storage+/run/containers/storage]9c59c35fa727dbf58214a3279387cb07246ce25642ffec3414a9c1c603b822ff', '--skip-fetch-check', '--generic-image', '--karg', 'rw', '--karg', 'console=tty0', '--karg', 'console=ttyS0', '--target-imgref', 'quay.io/oglok/bootc-app-flightctl:latest', '/run/osbuild/mounts']' returned non-zero exit status 1.
mount/boot-efi (org.osbuild.fat): umount: /store/tmp/buildroot-tmp-ymtc9fkr/mounts/boot/efi unmounted
mount/boot (org.osbuild.xfs): umount: /store/tmp/buildroot-tmp-ymtc9fkr/mounts/boot unmounted
mount/- (org.osbuild.xfs): umount: /store/tmp/buildroot-tmp-ymtc9fkr/mounts/ unmounted

⏱  Duration: 2s
manifest - failed
Failed
2024/10/24 11:07:05 error: cannot run osbuild: running osbuild failed: exit status 1

[cgwalters]

Hm we do test this in bootc upstream CI. It looks like you're using 9.4, but it'd be a good idea to update to 9.5 which is getting more active fixes.

What version of bootc-image-builder is in use here?