external IP changes?

[TimHeckel]

Hey all - I noticed that in the magic of the kured/microos world my traefik deployment was updated overnight (at least, I assume that's what happened, and the traefik deploy had a creation timestamp of a couple hours older). The issue is that none of my previous ingress routes were working (they were before), and all the service->endpoint->pod connections looked good and served the pages when I port forwarded. Then I saw that the hetzner public IP changed for my traefik load balancer, so I had to go back and update the DNS settings to get the ingresses to work again.

That said, is this known? Is there a way to force the IP to stay the same when traefik is updated? thanks!

[mysticaltech]

Hey @TimHeckel, please verify the upgrade with these commands on your node last reboot, uptime, and also the kured logs!

About DNS, external-dns would take care of automatically adjusting. But that does not mean it would not inquire 1 or 2 minutes downtime before the automatic adjustment takes place.

However, if you have doubts, just turn off automatic upgrades for both the node and k3s, see https://github.com/kube-hetzner/kube-hetzner#automatic-upgrade.

I have yet to try this myself in a production-like setting (will do ASAP), so at this time, have no other ideas.

[TimHeckel]

Thanks! The uptime and last reboot commands on both the agent and control plane nodes do not show a reboot at all. I'm not entirely sure why the system destroyed and recreated the loadbalancer, but the IP certainly changed. It would appear that kured did not reboot any of the nodes.

I am going to "enable protection" on the hetzner load balancer to try to ensure the IP doesn't change and that it can't be deleted - it's still a mystery as to why the traefik deployment redeployed 7 hours ago.

Happy to try to find more information out as needed. Not quite sure where else to look. I'll reply to this thread if I see the IP change again after marking the lb protected.

[mysticaltech]

Thanks for investigating this deeper! Good to know it was not caused by a reboot. The only way I have seen the load-balancer redeploy is when the Traekik config gets manually changed.

So have a look at the Traefik logs carefully, and for the IP, just use external-dns (I install it via helm) to just take care of the DNS for you automatically, God forbid that was to happen again! :)

And, yes, any more info you find, please do share! 🙏

[phaer]

I am not entirely sure why the system destroyed and recreated the loadbalancer

Did you look into cloud controller managers logs? Thats the component thats responsible for managing the load-balancer
kubectl logs -n kube-system -l app=hcloud-cloud-controller-manager

[TimHeckel]

Thank you! The logs are below. This happened today 2/21, and you can see the log right before the deletion was back from 2/18, so I'm not sure what actually precipitated the deletion...

I0218 14:42:35.755635       1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"traefik", UID:"cd0b152a-bb6a-4283-8201-7a3e420e4333", APIVersion:"v1", ResourceVersion:"2237", FieldPath:""}): type: 'Normal' reason: 'UpdatedLoadBalancer' Updated load balancer with new hosts
I0221 12:41:33.528906       1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"traefik", UID:"cd0b152a-bb6a-4283-8201-7a3e420e4333", APIVersion:"v1", ResourceVersion:"631645", FieldPath:""}): type: 'Normal' reason: 'DeletingLoadBalancer' Deleting load balancer
I0221 12:41:33.899316       1 load_balancers.go:284] "delete Load Balancer" op="hcloud/loadBalancers.EnsureLoadBalancerDeleted" loadBalancerID=622869
I0221 12:41:34.524381       1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"traefik", UID:"cd0b152a-bb6a-4283-8201-7a3e420e4333", APIVersion:"v1", ResourceVersion:"631645", FieldPath:""}): type: 'Normal' reason: 'DeletedLoadBalancer' Deleted load balancer
I0221 12:41:35.682677       1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"traefik", UID:"e5fb77cf-1380-4839-a783-d29b43e89f2d", APIVersion:"v1", ResourceVersion:"631706", FieldPath:""}): type: 'Normal' reason: 'EnsuringLoadBalancer' Ensuring load balancer
I0221 12:41:35.699187       1 load_balancers.go:104] "ensure Load Balancer" op="hcloud/loadBalancers.EnsureLoadBalancer" service="traefik" nodes=[k3s-agent-2 k3s-agent-1 k3s-agent-0]
I0221 12:41:38.578289       1 load_balancer.go:636] "add service" op="hcops/LoadBalancerOps.ReconcileHCLBServices" port=80 loadBalancerID=625385
I0221 12:41:39.510365       1 load_balancer.go:636] "add service" op="hcops/LoadBalancerOps.ReconcileHCLBServices" port=443 loadBalancerID=625385
I0221 12:41:40.460209       1 load_balancer.go:557] "add target" op="hcops/LoadBalancerOps.ReconcileHCLBTargets" service="traefik" targetName="k3s-agent-2"
I0221 12:41:41.587793       1 load_balancer.go:557] "add target" op="hcops/LoadBalancerOps.ReconcileHCLBTargets" service="traefik" targetName="k3s-agent-1"
I0221 12:41:42.776651       1 load_balancer.go:557] "add target" op="hcops/LoadBalancerOps.ReconcileHCLBTargets" service="traefik" targetName="k3s-agent-0"
I0221 12:41:44.575019       1 load_balancers.go:154] "reload HC Load Balancer" op="hcloud/loadBalancers.EnsureLoadBalancer" loadBalancerID=625385
I0221 12:41:45.011200       1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"traefik", UID:"e5fb77cf-1380-4839-a783-d29b43e89f2d", APIVersion:"v1", ResourceVersion:"631706", FieldPath:""}): type: 'Normal' reason: 'EnsuredLoadBalancer' Ensured load balancer

[mysticaltech]

So now we understand I guess. When you add or delete hosts, it deletes the LB service in Kube and recreates it. So basically you need to setup external-dns for peace of mind @TimHeckel, especially if you are going to scale up or down often.