BIB failed to create centos9s anaconda-iso : No space left on device

[LeoBras]

Error found on a system with 120GB free space:

time="2024-11-27T19:27:44Z" level=fatal msg="reading blob sha256:9d82b174fe65f4d0aaf3fadb96af43b1fc738185a152e11daf123407a7b87570: write /var/tmp/container_images_739548886: no space left on device"
Traceback (most recent call last):
  File "/run/osbuild/bin/org.osbuild.skopeo", line 53, in <module>
    r = main(args["inputs"], args["tree"], args["options"])
  File "/run/osbuild/bin/org.osbuild.skopeo", line 36, in main
    subprocess.run(cmd, check=True)
  File "/usr/lib64/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['skopeo', 'copy', '--remove-signatures', 'containers-storage:[overlay@/run/osbuild/containers/storage+/run/containers/storage]e598260069df1fd8e6d66ef18667333f0e97eec803d9b22ce45ab81b0918ed83', 'oci:/run/osbuild/tree/container']' returned non-zero exit status 1.

⏱  Duration: 16s
manifest - failed
Failed
2024/11/27 19:27:52 error: cannot run osbuild: running osbuild failed: exit status 1

How to reproduce:

git clone https://gitlab.com/LeoBras/rt-pretuned-bootc-example
cd rt-pretuned-bootc-example

echo '
img_name="centos-rt-pretuned-bootc"
target="host"        # host (virt pkgs) / guest
rtcpus="1-3"           # format : "1-3,5-7,9"
extra_kargs="idle=poll"
extra_pkg="tmux"
extra_cmd="echo test > /etc/hostname"
' > tst_variables.sh

bash build_container.sh -c init 
bash build_container.sh -c container -v tst_variables.sh
bash build_container.sh -c rootfs -t anaconda-iso -v tst_variables.sh

which runs in the rootfs command:

sudo podman run \
  --rm \
  -it \
  --privileged \
  --pull=newer \
  --security-opt label=type:unconfined_t \
  -v $(pwd)/config.toml:/config.toml:ro \
  -v $(pwd)/output:/output \
  -v /var/lib/containers/storage:/var/lib/containers/storage \
  quay.io/centos-bootc/bootc-image-builder:latest \
  --type $1 \
  localhost/${img_name} \
  --local

[LeoBras]

@achilleas-k ^

[achilleas-k]

Some context from quickly digging around: I think this might be related to the size of the rootfs that we create for the ISO. I'm not entirely sure that's the root cause but it's a good place to start looking for issues.
If that's the case, we should be smarter about the size selection. Currently it's fixed at 4 GiB.

[ondrejbudai]

I forgot that we have the intermediate EXT4 partition inside the ISO. Can we just get rid of it? It seems like Fedora did this a while ago: https://fedoraproject.org/wiki/Changes/OptimizeSquashFSOnDVDByRemovingEXT4FilesystemImageLayer

[ondrejbudai]

(note that RHEL 10 might actually go one step-further, and use erofs instead of squashfs)

[ondrejbudai]

We are working on removing the intermediate ext4 partition, which will fix this issue: osbuild/images#1105