Features to use only private IP on worker nodes

[jodhi]

Hello, I just saw the news on hetzner page regarding flexible networking

And since the kube-hetzner already using private ip by default, i think that should be straightforward to implement worker nodes only with private ip (still not sure).
What do you guys think about worker nodes without public ip?

ps: I can try to create PR if that make sense

[phaer]

Good idea, PR more than welcome I think!

[jodhi]

cool, i will try this one

[ricristian]

An idea would be to extend this from worker nodes also to control plane ones due to the fact that there might be other possibilities to connect to control planes maybe via a VPN and so on . In this way we can basically close completely any chance from receiving requests from outside ...

[jodhi]

An idea would be to extend this from worker nodes also to control plane ones due to the fact that there might be other possibilities to connect to control planes maybe via a VPN and so on . In this way we can basically close completely any chance from receiving requests from outside ...

Yeah, make sense as well, this one will require bastion/jumphost to access the control plane right.
Now you mention it, we need to access the worker node to bootstrap them as well 😓. Looks like we need internal bastion/jumphost to bootstrap the node 🤔

[ricristian]

Or at initial bootstrap of the cluster we use public IPv4 and after we finish the job drop those public IP's ?

[jodhi]

@ricristian that is smart idea. really.
but it will requre two step right? first create machine with public IPs with terraform plan+apply.
Then remove the ip definition on terraform module. cmiiw

[ricristian]

I'm not 100% sure how terraform works but I don't think that we can run in a sequence manner tf modules so a 2nd step can be implemented. I don't see that much hassle for that instead of trying to use jump hosts and so on ...

[jodhi]

yeah right, looks like terraform cannot create machine with public ip, provision, then destroy public ip with one go.

what i have in mind

1. 2-step (the configuration will be tricky)

we put  the machine as usual with public ip in tfvars
terraform apply
then we change the tfvars to remove public ip
terraform apply

2. 1-step (this is the simplest)

create control plane nodes with public ip, then create worker nodes without public ip, then get inside the worker nodes through control planes nodes to bootstrap

3. 1-step (this one possible as well)
   Use jumphost, then the terraform module can access all nodes internally. some docs

cmiiw. feel free to add

[ricristian]

Hmmm this might be something but maybe this flow might be a little bit easier ?

1st step:
As you mentioned:
we put the machine as usual with public ip in tfvars
terraform apply
then we change the tfvars to remove public ip
terraform apply

2. Create both control plane & worker nodes with public IP
   SSH Normal to all those nodes via public IP configure stuff & so on ...
3. Drop all IPv4 from all VM's ?

Why I'm trying to avoid jumphosts is because there are a lot of ways to have some kind of jumposts ( VPN, proxy command, Cloudflare, another VM inside Hetzner and so on ) and each one of us might have different use case so maybe it is a global option just to add Public IPv4 when we need to do some stuff on that cluster ( bootrap it, change flavour ) and after that drop those public IP's ( yeah there is a drawback for those few minutes when terraform is doing stuff port 6443 & 22 will be opened in internet if we don't limit it from firewall but still I can afford to have for 2 minutes 2 ports exposed ... it will require some Tesla processing power for someone to break in in under 3 minutes ) ...

But those are my thoughts hope that it adds some value

[jodhi]

thanks @ricristian. nice input, lemme check and try the approach

[phaer]

Or at initial bootstrap of the cluster we use public IPv4 and after we finish the job drop those public IP's ?

I think such approaches run a bit against terraforms spirit. But if someone gets that working with a single terraform apply invocation in a somewhat maintainable form, I'd gladly merge it 👍

In theory, I'd try to implement by using a bastion host for ssh on tcp/22 and kubernetes api on tcp/6443.

• First step would be a flag for control nodes which provisions the correct sshd config there (probably a bunch of ProxyJump directives pointing to all each private ip for nodes in "kubectl get nodes"?).
• At this point you should only need a public ip for one control node of your choice.
• You should already be able to optionally set the flag and add a public ip address for more than one control node. SSH ProxyJump should be stateless server side and the kube api in sync, so you have high availability for "free".
• Second step could be an option to add a tcp load balancer for ssh and kubernetes api which points to private ips of control hosts with the flag set. With that option you'd need to pay for a load-balancer, but get a single public ip which you could firewall off as you like (i.e. only allow your workstation/office ip).

What would you think about such an approach? Is there anything I am missing?
Sadly, I won't find time to implement this myself anytime soon, but I'd be very willing to review a PR and help with working out the details as we go.
Thanks for raising the topic and your input so far!

[mysticaltech]

Thanks, @jodhi, very good idea, and great input @ricristian.

For what it's worth, I really like the last iteration on this idea by @phaer. On top of having the nodes only on the private network, it would provide a load balancer for the control plane Kubernetes API, which is a feature I was hoping to have for a long time.

[mysticaltech]

Folks, FYI, slightly related to the above, starting in 1.4.3 we now have the possibility to load-balance CP nodes, thanks to @PurpleBooth!

[PurpleBooth]

I started putting together a spike as to the limitations of public interface-less hosts, and it looks a bit more complicated than simply allowing access via a bastion, which requires a public interface, we enable the recovery mode to install MicroOS, so we need to change how we install MicroOS in order to allow installation without a public interface, see #261 for my spike.

[mysticaltech]

Having given this a second thought, I see two more or less easy ways to implement this:

1. After the agents are deployed and running, access them with a local-exec. In it, SSH in with a background process on the server, or through a control-plane node, to permanently disable the public interfaces, meaning edit the networking config file.
2. A little more involved. After the agent nodes run, create snapshots of them, one for each nodepool kind (those tagged with an attributed of private=true, for instance), kill them, and redeploy the snapshots but without a public interface at the Hetzner level.

I would have chosen method 1. What do you think, @jodhi? Would that satisfy your tension?

[mysticaltech]

Moving this to a discussion as I do not see it as a priority for the time being.

[mysticaltech]

@oujonny All the nodes get created in parallel, so passing via a snapshot would almost double the creation time. It would surely help with the autoscaling, but that is already being discussed in #235 and will happen separately from the node creation.

About the network config file modification, no, actually, the public IP is not billed; it's included in the node price, so that won't change anything!

[phaer]

@mysticaltech You can save € 0.6 per host if you don't need a public ipv4 address since this summer, see https://www.hetzner.com/news/06-22-flexibles-networking/ and https://www.hetzner.com/cloud#pricing ("ipv6 only")

[mysticaltech]

Ok, thanks for sharing @phaer; I had no idea! Apparently, IPv6 is free, so we can still have connectivity if needed and disable it if we want to. Beautiful!

[mysticaltech]

So first, we would need to add IPv6 for the external access (@phaer, if you have the bandwidth, you are probably the most qualified to implement this), which would allow us to save some cost on the cluster.

As for the private networking, we don't need IPv6, and the CNIs are already configured just to consider eth1, so private IPv4.

[phaer]

Going to look into it whenever I'll spend development time on kube-hetzner, but can't give you a schedule yet - probably at least a few month. So if anyone wants to try: feel free, happy to review a PR!

[mysticaltech]

Something important to realize, folks, is that with private IPs only on the agent's nodes, go away automatic upgrades of nodes, k3s, and fetching of containers! Which is kind of bad; please correct me if I am missing something.

[phaer]

I'd hope that hetzner would just NAT outgoing traffic from private IPs? So you get no public ip but can talk to external services just as with your home internet connection (normally). But I haven't tested that

[mysticaltech]

Maybe, I hadn't thought of that. I would assume that is the case, as they have a gateway IP, and that's how the cluster connects to the web through the subnets. My bad, so most probably all of the above would work.

[ifeulner]

No, not out of the box, you have to set a route and a router, to handle the outgoing (internet) traffic. See this tutorial.
This introduces unfortunately another SPOF.

[mysticaltech]

@ifeulner The Hetzner CCM does take care of setting out the route properly if I remember correctly. It would be good testing it, but thanks for the info above, could come in very handy!

[clickersmudge]

Hello,
The idea of ​​putting all machines on a private network is great. This significantly increases security. A domain A record points to Load Balancer, and it distributes the traffic. We had a similar problem, but set up is Cloudflare, one Haproxy server, and several servers inside a private network. OVHcloud servers. The solution was not based on Kubernetes. Disconnecting from the public network meant that the machine had no access to the Internet and could not perform the update. The solution was to set the Firewall to allow outbound traffic but deny inbound traffic. On a private network, the machines communicated by themselves. This means that the firewall did not work inside this network. The machines in the private network were accessed via the Load Balancer HaProxy.

[mysticaltech]

Thanks for sharing @clickersmudge, interesting setup!

[janus-reith]

I'd really like to have my whole cluster, both controlplanes and workers, to only have private IPs assigned.
Ingress could still work over a Loadbalancer, and an LB could also loadbalance the API within the private network.

Regarding egress, I would assume there would be no need for any two-step terraform setup as mentioned in earlier posts.
A separate instance with public (floating) IP within that private network which acts as NAT could be deployed first, the setup to act as NAT could be part of its cloud-init config and the remaining cluster specific setup would just need some config to use that NAT.
I assume a bastion host would not be required in that case.

That mainly would leave two concerns:

• Maintaining HA of the NAT (Not specific to K8s but wonder what options there might be)
• Instead of the "static" NAT I wonder if there is some way to make the NAT native to the Cluster aswell
  • Maybe that could also resolve the HA concern if the cluster knows about the NAT nodes? )
  • So far found Cilium has an egress feature, but I guess we can't really set that up without the cluster existing yet, also HA seems to be an EE only feature

Regarding a bastion host, I assume there might be other usecases, e.g. with an LB placed only within the private network for the API and a bastion with public IP so the Kubernetes API is not public at all. I would however not put too much thought into that, since for daily operations I could rather create a tunnel from within the cluster (or e.g. use Tailscale) and just spin up a bastion instance for rescue purposes if that tunnel went down for some reason.

[mysticaltech]

@janus-reith Please check our latest version that shipped yesterday. We added the possibility to have the LB without public interface, and now the firewall can how all incoming ports closed. Basically no in traffic allowed, you can configure that now. See the kube.tf.example for more details.

@kube-hetzner/core FYI, if you want to add something else.

[janus-reith]

Awesome, thank you I'll check that out right away! 👌

[defyjoy]

Did this issue ever got concluded ? I do not see we still have any way to setup nodes with a proper Private IPs instead of public . I still see the worker nodes and control plane nodes having public ips

[mysticaltech]

@defyjoy At this moment, setup requires public IPs, but after setup, we added ways to block all access to the Public IPs via the firewall. Please see the firewall section in the kube.tf.example. It's basically the same as having private IP only nodes.

[oujonny]

As far as I understood the public ip is required to execute code on the remote machines with terraform, right?

What's about using a bastion host? I mean terraform remote_exec support this. See: https://developer.hashicorp.com/terraform/language/resources/provisioners/connection#connecting-through-a-bastion-host-with-ssh

[valkenburg-prevue-ch]

This is indeed possible using the bastion settings in remote_exec. I've been thinking about that approach too, but one reason why I haven't started building that functionality is the added complexity of the bastion host (which should also serve as a NAT, right?). That would be a single machine for which you need to setup monitoring and maintenance, and if it goes down, you have a problem. There is no simple solution with a highly available NAT, for as far as I understand.

[ifeulner]

You also need a router then to give the private-ip-only nodes internet accesss. This router needs to be highly available, if you do not want to create a new single point of failure.
That increases complexity and maintenance a lot... but doable...

[mysticaltech]

Exactly, so not absolutely needed since isolation is possible via the firewall.

[mikeywuu]

I have encountered the same desire to eliminate the use of public IPs. As I began refactoring the code to handle private-only control and agent nodes through configuration, I learned two key points: a) Hetzner requires NAT/public IPs to communicate with the internet, and b) more parts of the code need modification.

Since I have already invested some effort into this, and I can envision running a single point of failure (SPOF) pfSense in my network (which also supports remote access) until some high-availability (HA) options are feasible, I wanted to ask if there is any interest in extending the project. Specifically, this extension would make the assignment of public/private IPs configurable for those who manage the NAT component themselves.

[mikeywuu]

Thats sounds awesome.
Just two questions about that:

• Is there an estimate / timeline for the refactoring?
• As im curious, whats your solution to avoid the SPOF?

[valkenburg-prevue-ch]

Cool!

[xavierleune]

Hi everyone, I've been working on this feature, may you please give some feedback on #1567 ? Thanks !

[mysticaltech]

It's a big change, I'm reviewing it now! Amazing work @xavierleune