donotmerge: IBX changes #958

omertuc · 2024-12-10T23:45:41Z

Some (trash, temporary) changes I had to perform on bootc to get IBX to work

./Containerfile builds a bootable image from an existing normal IBX seed
./build_seed.sh is just a helper script

To install on recipient:

sudo podman run --privileged -v $PWD/authkeys:/authkeys --env RUST_LOG=trace -v /var/tmp:/var/tmp -v /var/lib/containers/storage:/var/lib/containers/storage --pid=host -it quay.io/otuchfel/bootc:seed bootc install to-existing-root --acknowledge-destructive --stateroot omeroot --root-ssh-authorized-keys /authkeys --bound-images pull

This PR only exists to start discussions around concrete issues, it's not meant to be merged.

Current state:

Installation of the seed image on new stateroot completes successfully
Host reboots successfully and connects to the network

But...

bootc only allows us to set authorized keys for the root user, not the core user as we would expect from OCP
For some reason, I can't SSH even to the root user? sshd is running, /root/.authorized_keys is populated with the key I expect... yet:

Even though I'm using ssh-ed25519, so it's rather strange... Will look into it once I come back from vacation

Future work:

Fix SSH bug
Integrate a PoC that uses lifecycle-agent and actually does a proper upgrade

omertuc · 2024-12-10T23:47:15Z

Containerfile

@@ -0,0 +1,26 @@
+FROM quay.io/centos/centos:stream9 as build


We must build bootc because it doesn't ship with RHCOS yet

omertuc · 2024-12-10T23:47:44Z

Containerfile

+RUN --mount=type=cache,target=/build/target --mount=type=cache,target=/var/roothome make test-bin-archive && mkdir -p /out && cp target/bootc.tar /out
+RUN mkdir -p /build/target/dev-rootfs  # This can hold arbitrary extra content
+
+FROM quay.io/otuchfel/ostbackup:serv1 as seed


Your normal seed image goes here.

omertuc · 2024-12-10T23:48:24Z

Containerfile

In an ideal future all seeds will simply be "bootc seeds", but for now we have this Containerfile to make a "bootc seed" out of a normal seed

hmm not sure. we can generate new seeds I guess. no need to deal with backwards compatibility at this point. but whatever is easier for the POC of course

omertuc · 2024-12-10T23:48:50Z

Makefile

@@ -39,10 +39,10 @@ install-all: install install-ostree-hooks
 	install -D -m 0755 target/release/tests-integration $(DESTDIR)$(prefix)/bin/bootc-integration-tests 

 bin-archive: all
-	$(MAKE) install DESTDIR=tmp-install && $(TAR_REPRODUCIBLE) --zstd -C tmp-install -cf target/bootc.tar.zst . && rm tmp-install -rf


-zstd unsupported on quay.io/centos/centos:stream9

Wait that's definitely not true. Do you have a really old image? Builds don't pull by default so unfortunately you need to explicitly e.g. podman pull quay.io/centos-bootc/centos-bootc:stream9 or so.

I don't believe I have an old image

omertuc · 2024-12-10T23:49:01Z

build_seed.sh

Some helper script, ignore

omertuc · 2024-12-10T23:49:19Z

hack/Containerfile

This file isn't really used in this PoC, ignore those changes

omertuc · 2024-12-10T23:50:04Z

lib/src/install.rs

@@ -946,7 +946,7 @@ pub(crate) fn reexecute_self_for_selinux_if_needed(
    override_disable_selinux: bool,
 ) -> Result<SELinuxFinalState> {
    // If the target state has SELinux enabled, we need to check the host state.
-    if srcdata.selinux {


The re-exec is failing for some reason, I skipped it and despite that things seems to work alright (maybe because I sudo setenforce permissive)

update: it works even without setenforce

Failing with what error message?

ERROR Installing to filesystem: Ensuring selinux install_t type: Ensuring selinux install_t type: fsetxattr(security.selinux): Operation not supported (os error 95)

But I'm now starting to think this just might be due to my removal of setup_tmp_mounts

Temporarily working around the setup_tmp_mounts error mentioned in the other comment below using -v /tmp:/tmp -v /var/tmp:/var/tmp

But now the re-exec fails with a new error

ERROR Installing to filesystem: Ensuring selinux install_t type: Failed to enter install_t (running as unconfined_u:unconfined_r:spc_t:s0) - use BOOTC_SETENFORCE0_FALLBACK=1 to override

Weirdly the error seems to disappear when -v /tmp:/tmp is removed, leaving only the /var/tmp mount. Confirming that this is true

Yep that is indeed the case.

To summarize:

No tmp mounts at all leads to ERROR Installing to filesystem: Linking tmp mounts to host: Exchanging /var/tmp <=> /var/tmp.tmp: Invalid cross-device link (os error 18)

Both /tmp/ and /var/tmp mounts leads to ERROR Installing to filesystem: Ensuring selinux install_t type: Failed to enter install_t (running as unconfined_u:unconfined_r:spc_t:s0) - use BOOTC_SETENFORCE0_FALLBACK=1 to override

Only /var/tmp mount works

omertuc · 2024-12-10T23:50:49Z

lib/src/install.rs

@@ -1222,7 +1162,7 @@ async fn prepare_install(
    crate::mount::ensure_mirrored_host_mount("/dev")?;
    crate::mount::ensure_mirrored_host_mount("/var/lib/containers")?;
    ensure_var()?;
-    setup_tmp_mounts()?;


This was failing for some reason so I had to take it out, I'm not quite sure what purpose it serves anyway, but need to dig into why it was failing

Again what error message?

ERROR Installing to filesystem: Linking tmp mounts to host: Exchanging /var/tmp <=> /var/tmp.tmp: Invalid cross-device link (os error 18)

Looking into it

omertuc · 2024-12-11T00:31:51Z

While the install completes "successfully", SSHing after rebooting doesn't work

ssh -o IdentityFile=./bip-orchestrate-vm/ssh-key/key -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no [email protected]
ssh: connect to host 192.168.126.10 port 22: Connection refused
make: *** [Makefile:425: ssh] Error 255

omertuc · 2024-12-11T00:43:17Z

omertuc · 2024-12-11T00:46:10Z

Host SSH keys were not created... Suspecting selinux?

EDIT: neither did the key I added with --root-ssh-authorized-keys

cgwalters · 2024-12-11T00:47:32Z

Containerfile

+LABEL com.openshift.lifecycle-agent.seed_format_version=3
+
+RUN mkdir -p /usr/lib/bootc/install
+RUN echo -e '[install.filesystem.root]\ntype = "xfs"' > /usr/lib/bootc/install/00-bootc.toml


I think this is unnecessary after FCOS starts inheriting from fedora-bootc, you just haven't picked up that change in the base image here right?

I've added that because I was just starting off from a very old PoC by @!javipolo, it might be completely unnecessary. I'll try to do without it

Yeah it works fine without it

cgwalters · 2024-12-11T00:49:31Z

Makefile

@@ -39,10 +39,10 @@ install-all: install install-ostree-hooks
 	install -D -m 0755 target/release/tests-integration $(DESTDIR)$(prefix)/bin/bootc-integration-tests 

 bin-archive: all
-	$(MAKE) install DESTDIR=tmp-install && $(TAR_REPRODUCIBLE) --zstd -C tmp-install -cf target/bootc.tar.zst . && rm tmp-install -rf


Wait that's definitely not true. Do you have a really old image? Builds don't pull by default so unfortunately you need to explicitly e.g. podman pull quay.io/centos-bootc/centos-bootc:stream9 or so.

cgwalters · 2024-12-11T00:50:04Z

lib/src/install.rs

@@ -946,7 +946,7 @@ pub(crate) fn reexecute_self_for_selinux_if_needed(
    override_disable_selinux: bool,
 ) -> Result<SELinuxFinalState> {
    // If the target state has SELinux enabled, we need to check the host state.
-    if srcdata.selinux {


Failing with what error message?

cgwalters · 2024-12-11T00:50:25Z

lib/src/install.rs

@@ -1222,7 +1162,7 @@ async fn prepare_install(
    crate::mount::ensure_mirrored_host_mount("/dev")?;
    crate::mount::ensure_mirrored_host_mount("/var/lib/containers")?;
    ensure_var()?;
-    setup_tmp_mounts()?;


Again what error message?

Some (trash, temporary) changes I had to perform on bootc to get IBX to work - `./Containerfile` builds a bootable image from an existing normal IBX seed - `./build_seed.sh` is just a helper script To install on recipient ``` sudo podman run --privileged -v $PWD/authkeys:/authkeys --env RUST_LOG=trace -v /var/tmp:/var/tmp -v /var/lib/containers/storage:/var/lib/containers/storage --pid=host -it quay.io/otuchfel/bootc:seed bootc install to-existing-root --acknowledge-destructive --stateroot omeroot --root-ssh-authorized-keys /authkeys --bound-images pull ``` This PR only exists to start discussions around concrete issues, it's not meant to be merged.

openshift-ci bot added the do-not-merge/work-in-progress label Dec 10, 2024

github-actions bot added the area/install Issues related to `bootc install` label Dec 10, 2024

omertuc commented Dec 10, 2024

View reviewed changes

build_seed.sh Outdated

Copy link

Contributor Author

omertuc Dec 10, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some helper script, ignore

omertuc commented Dec 10, 2024

View reviewed changes

hack/Containerfile Outdated

Copy link

Contributor Author

omertuc Dec 10, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file isn't really used in this PoC, ignore those changes

omertuc commented Dec 10, 2024

View reviewed changes

cgwalters reviewed Dec 11, 2024

View reviewed changes

omertuc force-pushed the seed branch from 86a1f73 to 715d97a Compare December 14, 2024 02:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

donotmerge: IBX changes #958

donotmerge: IBX changes #958

omertuc commented Dec 10, 2024 •

edited

Loading

omertuc Dec 10, 2024

omertuc Dec 10, 2024

omertuc Dec 10, 2024

romfreiman Dec 16, 2024

omertuc Dec 10, 2024

cgwalters Dec 11, 2024

omertuc Dec 14, 2024

omertuc Dec 10, 2024

omertuc Dec 10, 2024

omertuc Dec 10, 2024 •

edited

Loading

omertuc Dec 11, 2024

cgwalters Dec 11, 2024

omertuc Dec 14, 2024

omertuc Dec 14, 2024

omertuc Dec 14, 2024 •

edited

Loading

omertuc Dec 14, 2024

omertuc Dec 10, 2024

cgwalters Dec 11, 2024

omertuc Dec 14, 2024 •

edited

Loading

omertuc commented Dec 11, 2024 •

edited

Loading

omertuc commented Dec 11, 2024

omertuc commented Dec 11, 2024 •

edited

Loading

cgwalters Dec 11, 2024

omertuc Dec 14, 2024

omertuc Dec 14, 2024

cgwalters Dec 11, 2024

cgwalters Dec 11, 2024

cgwalters Dec 11, 2024

donotmerge: IBX changes #958

Are you sure you want to change the base?

donotmerge: IBX changes #958

Conversation

omertuc commented Dec 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

omertuc Dec 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

omertuc Dec 14, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

omertuc Dec 14, 2024 • edited Loading

Choose a reason for hiding this comment

omertuc commented Dec 11, 2024 • edited Loading

omertuc commented Dec 11, 2024

omertuc commented Dec 11, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

omertuc commented Dec 10, 2024 •

edited

Loading

omertuc Dec 10, 2024 •

edited

Loading

omertuc Dec 14, 2024 •

edited

Loading

omertuc Dec 14, 2024 •

edited

Loading

omertuc commented Dec 11, 2024 •

edited

Loading

omertuc commented Dec 11, 2024 •

edited

Loading