metrics-server unable to bind port 10250, already in use by k3s-server

[edmondburnett]

Setting up a brand new cluster today, and having issues with metrics-server unable to start, due to a port conflict. I'm using the default install script as per the docs. Fresh cluster.

The pod logs kubectl logs metrics-server-77f857b947-6tj2s --namespace kube-system shows:

panic: failed to create listener: failed to listen on 0.0.0.0:10250: listen tcp 0.0.0.0:10250: bind: address already in use

the k3s service logs show:

Nov 24 18:55:51 fuji k3s[4698]: E1124 18:55:51.317739    4698 pod_workers.go:1298] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"metrics-server\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=metrics-server pod=metrics-server-6587897fdb-zhfg2_kube-system(8181fac7-efc2-46fa-91df-66610623cf7e)\"" pod="kube-system/metrics-server-6587897fdb-zhfg2" podUID="8181fac7-efc2-46fa-91df-66610623cf7e"

Netstat shows that k3s-server is already using the port that metrics-server is trying to use:

tcp6       0      0 :::10250                :::*                    LISTEN      4698/k3s server

tcp   LISTEN 0      4096                                          *:10250            *:*    users:(("k3s-server",pid=816,fd=195))

It seems metrics-server is trying to bind to the host machine's main interface but the k3s main service is already using that port there.

If I edit the port number of the deployment using kubectl edit deploy -n kube-system metrics-server, the change gets automatically overwritten again after a while. The docs seem to indicate that this behaviour is intentional. So what's the correct way to utilize the included metrics-server?

I added a second node, also with a fresh OS install, and it happens there too. Since it's crashing, the metrics-service bounces back and forth between the nodes.

[edmondburnett]

Well, I uninstalled k3s on all of the nodes and recreated the cluster, and now that pod is still reporting errors, but at least the binding issue is now gone. I guess I will troubleshoot this one for a while.