Solution for multiple networks for nodes

[roberto-sebastiano]

Description

Hello,

In my setup on Hetzner, I use two private networks, one for kubernetes cluster and one for other service VMS.
For security reasons, I needed the Kubernetes nodes to access the service VMs via a private network, not via the public one.
In this setup, it's necessary for all the VMs, including the Kubernetes nodes and service VMs, to be members of both networks for the private network access to work.
I couldn't find a way to do it in the project. I only see existing_network_id that is unfortunately a list of length 1
I think this should allow multiple networks.

After looking at the hcloud documentation, it seems this is hardcoded (I can't find the line right now, sorry).

SOLUTION:
I've made a custom solution that works quite well for me and may benefit others too, hetzner-aux-node-operator
As a simple operator, it loops over nodes (all the kubernetes one + other VMs you can specify) and adds them to the required extra networks.
As a plus, it also enforces a firewall (or any number of them) to be always applied to all nodes.
If you manually remove a node from a network or firewall by mistake, the operator kicks in and enforces the attachments again.

This is a sample output of the operator in action after adding some nodes:

hetzner-aux-node-operator-7bf6478c9d-mk4j7 hetzner-aux-node-operator Starting Hetzner Cloud Aux Network Operator..
hetzner-aux-node-operator-7bf6478c9d-mk4j7 hetzner-aux-node-operator INFO: STARTUP CONFIG: firewall_check_interval: 10
hetzner-aux-node-operator-7bf6478c9d-mk4j7 hetzner-aux-node-operator INFO: STARTUP CONFIG: required_network_ids: []
hetzner-aux-node-operator-7bf6478c9d-mk4j7 hetzner-aux-node-operator INFO: STARTUP CONFIG: required_network_names: ['other-vms']
hetzner-aux-node-operator-7bf6478c9d-mk4j7 hetzner-aux-node-operator INFO: STARTUP CONFIG: enforce_firewall_ids: []
hetzner-aux-node-operator-7bf6478c9d-mk4j7 hetzner-aux-node-operator INFO: STARTUP CONFIG: enforce_firewall_names: ['book-test']
hetzner-aux-node-operator-7bf6478c9d-mk4j7 hetzner-aux-node-operator INFO: STARTUP CONFIG: enforce_firewall_extra_hosts: ['share-docker-test-v4']
hetzner-aux-node-operator-7bf6478c9d-mk4j7 hetzner-aux-node-operator Network Enforcer Found Server ID: 46801522 Server Name: book-test-control-plane-1-nurem-dry Private Networks: {4187883: 'book-test', 3993510: 'other-vms'}
hetzner-aux-node-operator-7bf6478c9d-mk4j7 hetzner-aux-node-operator Firewall Enforcer Found Server ID: 46801522 Server Name: book-test-control-plane-1-nurem-dry Firewalls: {1376676: 'book-test', 1430000: 'scale-job-temp-ssh'}
hetzner-aux-node-operator-7bf6478c9d-mk4j7 hetzner-aux-node-operator Firewall Enforcer Found Server ID: 44674171 Server Name: share-docker-test-v4 Firewalls: {1376676: 'book-test'}
hetzner-aux-node-operator-7bf6478c9d-mk4j7 hetzner-aux-node-operator Network Enforcer Found Server ID: 48375733 Server Name: book-test-agent-cax11-nurem-cik Private Networks: {4187883: 'book-test'}
hetzner-aux-node-operator-7bf6478c9d-mk4j7 hetzner-aux-node-operator REMEDIATION: Node book-test-agent-cax11-nurem-cik Required network NAME other-vms is not attached. Attaching ..
hetzner-aux-node-operator-7bf6478c9d-mk4j7 hetzner-aux-node-operator REMEDIATION: Node book-test-agent-cax11-nurem-cik Required network NAME other-vms ATTACHED SUCCESSFULLY.
hetzner-aux-node-operator-7bf6478c9d-mk4j7 hetzner-aux-node-operator Network Enforcer Found Server ID: 48375735 Server Name: book-test-agent-cax11-falk-gdb Private Networks: {4187883: 'book-test'}
hetzner-aux-node-operator-7bf6478c9d-mk4j7 hetzner-aux-node-operator REMEDIATION: Node book-test-agent-cax11-falk-gdb Required network NAME other-vms is not attached. Attaching ..
hetzner-aux-node-operator-7bf6478c9d-mk4j7 hetzner-aux-node-operator REMEDIATION: Node book-test-agent-cax11-falk-gdb Required network NAME other-vms ATTACHED SUCCESSFULLY.

You can find it here https://github.com/roberto-sebastiano/hetzner-aux-node-operator
If you think this could be accomplished in a different way or do you want to write this somewhere, you are welcome

Thank you,
Roberto Sebastiano

Kube.tf file

<snippet> 
# If you want to create the private network before calling this module,
  # you can do so and pass its id here. For example if you want to use a proxy
  # which only listens on your private network. Advanced use case.
  #
  # NOTE1: make sure to adapt network_ipv4_cidr, cluster_ipv4_cidr, and service_ipv4_cidr accordingly.
  #        If your network is created with 10.0.0.0/8, and you use subnet 10.128.0.0/9 for your
  #        non-k3s business, then adapting `network_ipv4_cidr = "10.0.0.0/9"` should be all you need.
  #
  # NOTE2: square brackets! This must be a list of length 1.
  #
  existing_network_id = ["12345"]

Screenshots

No response

Platform

Mac

[mysticaltech]

@roberto-sebastiano This is amazing, thanks for sharing! Wow.

@kube-hetzner/core @maaft FYI