Ubuntu as an alternative OS

[jr-dimedis]

I have to admit, I almost don't dare to open a thread with this topic, because this project explicitly relies on MicroOS and already excludes Ubuntu in the README.

I have great respect for the work that is done here and we have been using kube-hetzner productively for over a year. One major problem, however, was the OS upgrades, which failed so often that we switched them off completely. We realised that the technology that is supposed to make MicroOS so stable (ready only root, snapshots, automatic rollback) is apparently so complex that it was repeatedly responsible for errors that almost led to complete cluster losses. It was only because we were very careful and because Kubernetes is so incredibly resilient that we were able to avoid the worst.

We have been using Debian and Ubuntu based servers for over 20 years and have never had any problems with upgrades that would have resulted in server loss. Here, too, there are automatic software updates that can be coordinated with Kured. With LTS releases, the core remains stable for many years and runs absolutely reliably, while MicroOS with its rolling releases and cutting-edge software gives us the feeling of constantly dancing on the edge.

That's the reason why I started an experiment and extended kube-hetzner so that either MicroOS or Ubuntu can be used, even mixed. Ultimately, only very few interventions in the existing architecture were necessary. Essentially, there is now an os property for the node pools and a few variables in the "Ubuntu" variant when it comes to creating the servers.

There are still some issues to clarify here, e.g. there is currently no SELinux support at all, but basically I was able to set up MicroOS & Ubuntu Cluster with it and also migrate from MicroOS to Ubuntu during operation.

I haven't opened a pull request yet because I first wanted to check here whether such a request would have any chance of being accepted at all, as it runs counter to the declared aim of this project.

For all those who are interested in the code or would like to experiment themselves, here is the link to the branch:

https://github.com/jr-dimedis/terraform-hcloud-kube-hetzner/tree/ubuntu-nodes?tab=readme-ov-file#ubuntu-based-nodes-experimental

Thank you very much for reading. I look forward to any feedback.

[pat-s]

That is similar to other requests aiming to make the OS choice flexible (e.g. #1329). I guess overall a flexible way to specify an OS of choice would be very welcome, see also #1329 (While I'd argue personally that Ubuntu is definitely not an OS one should use in a kubernetes cluster).

I also had issues with the microOS updates but it might have gotten them under control now but restricting the updates to two days per week and using a force-reboot logic via kured to amplify the issues with Longhorn PDBs preventing reboots.