k3s hetzner additional routes for hetzner network

[CroutonDigital]

Hi,

I deployed k3s on hetzner and I created additional prometheus instance for monitoring and attach to the same network.
From pods I can ping

PING 10.0.0.254 (10.0.0.254): 56 data bytes
64 bytes from 10.0.0.254: seq=0 ttl=62 time=0.814 ms
64 bytes from 10.0.0.254: seq=1 ttl=62 time=0.513 ms
64 bytes from 10.0.0.254: seq=2 ttl=62 time=0.388 ms

Bot from prometheus instance can't.
When I try manual add additional route to hetzner network after 10 sec it removed.

Any cases make route from VMs attached to the subnet and cluster pods?

thank you!

[M4t7e]

Hey @CroutonDigital can you provide your kube.tf config? This may help better understand your setup, specifically I have no idea what CNI you are using. Please also provide more information about which route you would like to add.

Your prometheus instance is a native VM and connected directly to the K3s cluster? My guess is that your Pod IP is masqueraded (SNATed) with the Node IP. That could be the reason why you can ping IPs outside the cluster but not vice versa. If you are looking for native routed setup, please have a look here: #911

[CroutonDigital]

module "kube-hetzner" {
  providers = {
    hcloud = hcloud
  }
  hcloud_token = var.hcloud_token != "" ? var.hcloud_token : local.hcloud_token
  source = "kube-hetzner/kube-hetzner/hcloud"
   ssh_port = 2222
  ssh_public_key = file("${path.module}/ssh/k8s-hetzner.pub")
  ssh_private_key = file("${path.module}/ssh/k8s-hetzner")
  network_region = "eu-central"
  network_ipv4_cidr = "10.0.0.0/8"
  cluster_ipv4_cidr = "10.42.0.0/16"
  control_plane_nodepools = [
    {
      name        = "control-plane-fsn1",
      server_type = "cpx11",
      location    = "fsn1",
      labels      = [],
      taints      = [],
      count       = 1

      # Enable automatic backups via Hetzner (default: false)
      # backups = true
    },
    {
      name        = "control-plane-nbg1",
      server_type = "cpx11",
      location    = "nbg1",
      labels      = [],
      taints      = [],
      count       = 1

      # Enable automatic backups via Hetzner (default: false)
      # backups = true
    },
#    {
#      name        = "control-plane-hel1",
#      server_type = "cpx11",
#      location    = "hel1",
#      labels      = [],
#      taints      = [],
#      count       = 1
#
#      # Enable automatic backups via Hetzner (default: false)
#      # backups = true
#    }
  ]

  agent_nodepools = [
    {
      name        = "agent-small",
      server_type = "cpx11",
      location    = "fsn1",
      labels      = [],
      taints      = [],
      count       = 2

      # Enable automatic backups via Hetzner (default: false)
      # backups = true
    },
    {
      name        = "agent-large",
      server_type = "cpx21",
      location    = "nbg1",
      labels      = [],
      taints      = [],
      count       = 0

      # Enable automatic backups via Hetzner (default: false)
      # backups = true
    },
    {
      name        = "storage",
      server_type = "cpx21",
      location    = "fsn1",
      # Fully optional, just a demo.
      labels      = [
        "node.kubernetes.io/server-usage=storage"
      ],
      taints      = [],
      count       = 1
      
    },
    {
      name        = "egress",
      server_type = "cpx11",
      location    = "fsn1",
      labels = [
        "node.kubernetes.io/role=egress"
      ],
      taints = [
        "node.kubernetes.io/role=egress:NoSchedule"
      ],
      floating_ip = true
      count = 0
    },
    {
      name        = "agent-arm-small",
      server_type = "cax11",
      location    = "fsn1",
      labels      = [],
      taints      = [],
      count       = 0
    }
  ]
  load_balancer_type     = "lb11"
  load_balancer_location = "fsn1"
  ingress_controller = "traefik"

  traefik_additional_options = ["--log.level=DEBUG"]
  initial_k3s_channel = "stable"
  cluster_name = "k3s"

  restrict_outbound_traffic = true

   extra_firewall_rules = [
     {
       description = "ELK logs"
       direction       = "out"
       protocol        = "tcp"
       port            = "9200"
       source_ips      = [] # Won't be used for this rule
       destination_ips = ["0.0.0.0/0", "::/0"]
     },
     {
       description = "To Allow ArgoCD access to resources via SSH"
       direction       = "out"
       protocol        = "tcp"
       port            = "22"
       source_ips      = [] # Won't be used for this rule
       destination_ips = ["0.0.0.0/0", "::/0"]
     }
   ]

   cni_plugin = "calico"
   calico_version = "v3.25.0"
   enable_cert_manager = false
   dns_servers = ["1.1.1.1", "8.8.8.8", "9.9.9.9"]

 traefik_values = <<EOT
deployment:
  replicas: 1
globalArguments: []
service:
  enabled: true
  type: LoadBalancer
  annotations:
    "load-balancer.hetzner.cloud/name": "k3s"
    "load-balancer.hetzner.cloud/use-private-ip": "true"
    "load-balancer.hetzner.cloud/disable-private-ingress": "true"
    "load-balancer.hetzner.cloud/location": "nbg1"
    "load-balancer.hetzner.cloud/type": "lb11"
    "load-balancer.hetzner.cloud/uses-proxyprotocol": "true"

ports:
  web:
    redirectTo: websecure

    proxyProtocol:
      trustedIPs:
        - 127.0.0.1/32
        - 10.0.0.0/8
    forwardedHeaders:
      trustedIPs:
        - 127.0.0.1/32
        - 10.0.0.0/8
  websecure:
    proxyProtocol:
      trustedIPs:
        - 127.0.0.1/32
        - 10.0.0.0/8
    forwardedHeaders:
      trustedIPs:
        - 127.0.0.1/32
        - 10.0.0.0/8

tlsOptions: {}
tlsStore: {}

certResolvers:
  letsencrypt:
    email: [email protected]
    tlsChallenge: true
    httpChallenge:
      entryPoint: "web"
    #     # It has to match the path with a persistent volume
    storage: /data/acme.json

  EOT
} 

provider "hcloud" {
  token = var.hcloud_token != "" ? var.hcloud_token : local.hcloud_token
}

terraform {
  required_version = ">= 1.3.3"
  required_providers {
    hcloud = {
      source  = "hetznercloud/hcloud"
      version = ">= 1.39.0"
    }
  }
}

Yes, I use different instance for prometheus and want monitor pods.

[M4t7e]

This could be related to an independent IPAM from the CCM and Calico. I described the problem here:

• Update handling of custom cluster-cidr #902 (comment)
• Update handling of custom cluster-cidr #902 (comment)

A fully integrated network model would solve your problem, like I sketched for Cilium here: #911
This basically means that everything (including Pod IPs) can be reached within a bigger network range, even outside of the cluster, without tunnel overlays. I am very close to file a PR to achieve that for Cilium. If you would like to switch to Cilium, that can be a solution for your use case. Otherwise someone would have to try to do the same for Calico.

If you would to give it a try with Cilium, you can already do it with this config (Cilium >= 1.14 is required):

cni_plugin = "cilium"

cilium_values = <<EOT
ipam:
  mode: kubernetes
k8s:
  requireIPv4PodCIDR: true
kubeProxyReplacement: true
routingMode: native
ipv4NativeRoutingCIDR: "10.0.0.0/8"
endpointRoutes:
  enabled: true
loadBalancer:
  acceleration: native
bpf:
  masquerade: true
egressGateway:
  enabled: true
MTU: 1450
EOT

With this snippet Cilium honors the CCM IPAM and the routes to the Pods are set accordingly by the CCM. This is a native routed setup and ipv4NativeRoutingCIDR defines where native routing is done without masquerading. Btw, the CCM is the one that deletes your manually set routes (the interval of that should be 10 seconds).

[M4t7e]

Now I also created the announced PR: #930

[M4t7e]

Hey @CroutonDigital, my PR has been merged and is available in release v2.6.0. See also the announcement here: #932

In case you don't want to maintain your own cilium_values configuration, you can now use the following configuration to achieve the same result:

  cni_plugin = "cilium"
  cilium_routing_mode = "native"
  cilium_ipv4_native_routing_cidr = "10.0.0.0/8"
  cilium_egress_gateway_enabled = true

[CroutonDigital]

I recreated cluster and switch cni to cilium and added cilium_values.
Now I can ping pods and can monitoring.

ping from additional VM

root@monitoring:~# ping 10.42.2.19
PING 10.42.2.19 (10.42.2.19) 56(84) bytes of data.
64 bytes from 10.42.2.19: icmp_seq=1 ttl=62 time=3.15 ms
64 bytes from 10.42.2.19: icmp_seq=2 ttl=62 time=0.592 ms
64 bytes from 10.42.2.19: icmp_seq=3 ttl=62 time=2.31 ms
64 bytes from 10.42.2.19: icmp_seq=4 ttl=62 time=0.918 ms
64 bytes from 10.42.2.19: icmp_seq=5 ttl=62 time=0.539 ms
64 bytes from 10.42.2.19: icmp_seq=6 ttl=62 time=1.17 ms
64 bytes from 10.42.2.19: icmp_seq=7 ttl=62 time=0.617 ms
64 bytes from 10.42.2.19: icmp_seq=8 ttl=62 time=0.478 ms
64 bytes from 10.42.2.19: icmp_seq=9 ttl=62 time=0.610 ms
64 bytes from 10.42.2.19: icmp_seq=10 ttl=62 time=2.04 ms
^C
--- 10.42.2.19 ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9094ms
rtt min/avg/max/mdev = 0.478/1.242/3.151/0.884 ms

ping from pod:

PING 10.0.0.254 (10.0.0.254): 56 data bytes
64 bytes from 10.0.0.254: seq=0 ttl=62 time=4.437 ms
64 bytes from 10.0.0.254: seq=1 ttl=62 time=0.622 ms
64 bytes from 10.0.0.254: seq=2 ttl=62 time=0.591 ms
64 bytes from 10.0.0.254: seq=3 ttl=62 time=0.510 ms
64 bytes from 10.0.0.254: seq=4 ttl=62 time=0.573 ms
64 bytes from 10.0.0.254: seq=5 ttl=62 time=0.573 ms
64 bytes from 10.0.0.254: seq=6 ttl=62 time=0.573 ms
64 bytes from 10.0.0.254: seq=7 ttl=62 time=0.581 ms
64 bytes from 10.0.0.254: seq=8 ttl=62 time=0.543 ms
64 bytes from 10.0.0.254: seq=9 ttl=62 time=0.477 ms
^C
--- 10.0.0.254 ping statistics ---
10 packets transmitted, 10 packets received, 0% packet loss
round-trip min/avg/max = 0.477/0.948/4.437 ms

why different response time, up 3ms from additional VM to pod?

Thank you!

[M4t7e]

Is there some time between your ping tests? I can't reproduce this different latency based on ping direction. A few ms difference can occur which may depend on the underlying Hetzner SDN. They also need to perform network maintenance from time to time, which can result in higher latency. Under normal circumstances, there shouldn't be much difference between pod to server latency and vice versa.

Pod to dedicated Server:

# ping 10.0.2.1 -c 15
PING 10.0.2.1 (10.0.2.1) 56(84) bytes of data.
64 bytes from 10.0.2.1: icmp_seq=1 ttl=62 time=0.427 ms
64 bytes from 10.0.2.1: icmp_seq=2 ttl=62 time=0.551 ms
64 bytes from 10.0.2.1: icmp_seq=3 ttl=62 time=0.458 ms
64 bytes from 10.0.2.1: icmp_seq=4 ttl=62 time=0.572 ms
64 bytes from 10.0.2.1: icmp_seq=5 ttl=62 time=0.444 ms
64 bytes from 10.0.2.1: icmp_seq=6 ttl=62 time=0.413 ms
64 bytes from 10.0.2.1: icmp_seq=7 ttl=62 time=0.414 ms
64 bytes from 10.0.2.1: icmp_seq=8 ttl=62 time=0.436 ms
64 bytes from 10.0.2.1: icmp_seq=9 ttl=62 time=0.400 ms
64 bytes from 10.0.2.1: icmp_seq=10 ttl=62 time=0.704 ms
64 bytes from 10.0.2.1: icmp_seq=11 ttl=62 time=0.574 ms
64 bytes from 10.0.2.1: icmp_seq=12 ttl=62 time=0.503 ms
64 bytes from 10.0.2.1: icmp_seq=13 ttl=62 time=0.436 ms
64 bytes from 10.0.2.1: icmp_seq=14 ttl=62 time=0.473 ms
64 bytes from 10.0.2.1: icmp_seq=15 ttl=62 time=0.476 ms

--- 10.0.2.1 ping statistics ---
15 packets transmitted, 15 received, 0% packet loss, time 194ms
rtt min/avg/max/mdev = 0.400/0.485/0.704/0.082 ms

Dedicated Server to Pod:

# ping 10.0.133.123 -c 15
PING 10.0.133.123 (10.0.133.123) 56(84) bytes of data.
64 bytes from 10.0.133.123: icmp_seq=1 ttl=62 time=0.526 ms
64 bytes from 10.0.133.123: icmp_seq=2 ttl=62 time=0.501 ms
64 bytes from 10.0.133.123: icmp_seq=3 ttl=62 time=0.418 ms
64 bytes from 10.0.133.123: icmp_seq=4 ttl=62 time=0.382 ms
64 bytes from 10.0.133.123: icmp_seq=5 ttl=62 time=0.391 ms
64 bytes from 10.0.133.123: icmp_seq=6 ttl=62 time=0.516 ms
64 bytes from 10.0.133.123: icmp_seq=7 ttl=62 time=0.396 ms
64 bytes from 10.0.133.123: icmp_seq=8 ttl=62 time=0.436 ms
64 bytes from 10.0.133.123: icmp_seq=9 ttl=62 time=0.515 ms
64 bytes from 10.0.133.123: icmp_seq=10 ttl=62 time=0.595 ms
64 bytes from 10.0.133.123: icmp_seq=11 ttl=62 time=0.459 ms
64 bytes from 10.0.133.123: icmp_seq=12 ttl=62 time=0.501 ms
64 bytes from 10.0.133.123: icmp_seq=13 ttl=62 time=0.447 ms
64 bytes from 10.0.133.123: icmp_seq=14 ttl=62 time=0.488 ms
64 bytes from 10.0.133.123: icmp_seq=15 ttl=62 time=0.438 ms

--- 10.0.133.123 ping statistics ---
15 packets transmitted, 15 received, 0% packet loss, time 14330ms
rtt min/avg/max/mdev = 0.382/0.467/0.595/0.057 ms

[CroutonDigital]

iperf speed check result

/ # iperf -c 10.0.0.254
------------------------------------------------------------
Client connecting to 10.0.0.254, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  1] local 10.42.2.19 port 44230 connected with 10.0.0.254 port 5001
[ ID] Interval       Transfer     Bandwidth
[  1] 0.00-10.03 sec  4.85 GBytes  4.16 Gbits/sec

[CroutonDigital]

ok, thank's a lot

[CroutonDigital]

I found little bug, with network, before i tested ping with pods. But now I can't connect to service after when we switch CNI to cilium.

#979

[laurenz-glueck]

Hey @M4t7e
I have just deployed a cluster with the settings you have suggested:

  cni_plugin = "cilium"
  cilium_routing_mode = "native"
  cilium_ipv4_native_routing_cidr = "10.0.0.0/8"
  cilium_egress_gateway_enabled = true

But I noticed that the kube-probe requests in my cluster are now coming from the public IP of the agent node where the pod is running on.
This was not the case in a different cluster where I use "flannel" as the CNI. So I guess this is related to cilium?

Do you have any idea why this is the case? And how I can fix this?