[Release-1.29] - Secondary etcd-only nodes do not reconnect to apiserver after outage if joined against an etcd-only node

[brandond]

Backport fix for Secondary etcd-only nodes do not reconnect to apiserver after outage if joined against an etcd-only node

• Secondary etcd-only nodes do not reconnect to apiserver after outage if joined against an etcd-only node #11311

[aganesh-suse]

Validated on release-1.29 branch with commit b91bdd3

Environment Details

Infrastructure

• Cloud
• Hosted

Node(s) CPU architecture, OS, and Version:

$ cat /etc/os-release
PRETTY_NAME="Ubuntu 24.04 LTS"

$ uname -m
x86_64

Cluster Configuration:

HA: 3 server/ 1 agent - 2 etcd, 1 cp, 1 agent node config.
or 
3 etcd, 2 cp, 1 agent configuration.

P.S - all nodes pointing to the main/first etcd server

Config.yaml:

etcd only node config.yaml:

token: xxxx
disable-apiserver: true
disable-controller-manager: true
disable-scheduler: true
node-taint:
- node-role.kubernetes.io/etcd:NoExecute
cluster-init: true
write-kubeconfig-mode: "0644"
node-external-ip: 1.1.1.1
node-label:
- k3s-upgrade=server
debug: true

Control plane only node config.yaml:

$ cat /etc/rancher/k3s/config.yaml 
token: xxxx
server: https://1.1.1.1:6443
disable-etcd: true
node-taint:
- node-role.kubernetes.io/control-plane:NoSchedule
write-kubeconfig-mode: "0644"
node-external-ip: 2.2.2.2
node-label:
- k3s-upgrade=server
debug: true

Testing Steps

1. Copy config.yaml

$ sudo mkdir -p /etc/rancher/k3s && sudo cp config.yaml /etc/rancher/k3s

2. Install k3s

curl -sfL https://get.k3s.io | sudo INSTALL_K3S_COMMIT='b91bdd32d6dc817bc98017aa15b5dc9ac46f541d' sh -s - server

3. Verify Cluster Status:

kubectl get nodes -o wide
kubectl get pods -A

4. Restart control plane node. Wait for 5 minutes.
5. Verify cluster status - that all nodes are in Ready state:

kubectl get nodes -o wide

Replication Results:

• k3s version used for replication:

$ k3s -v 
k3s version v1.29.10+k3s1 (ae4df311)
go version go1.22.8

$ kubectl get nodes
time="2024-11-15T04:23:15Z" level=debug msg="Asset dir /var/lib/rancher/k3s/data/9f4d91d896c15e475c3d62297a5940f714a93339400de6381dc3bb48257dc23a"
time="2024-11-15T04:23:15Z" level=debug msg="Running /var/lib/rancher/k3s/data/9f4d91d896c15e475c3d62297a5940f714a93339400de6381dc3bb48257dc23a/bin/kubectl [kubectl get nodes]"
NAME               STATUS     ROLES                  AGE   VERSION
ip-172-31-0-37     NotReady   <none>                 13m   v1.29.10+k3s1
ip-172-31-12-163   Ready      etcd                   15m   v1.29.10+k3s1
ip-172-31-4-1      Ready      control-plane,master   15m   v1.29.10+k3s1
ip-172-31-9-130    NotReady   etcd                   15m   v1.29.10+k3s1

Validation Results:

• k3s version used for validation:

$ k3s -v 
k3s version v1.29.10+k3s-b91bdd32 (b91bdd32)
go version go1.22.8

$ sudo /usr/local/bin/kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml get nodes 
time="2024-11-15T03:52:01Z" level=debug msg="Asset dir /var/lib/rancher/k3s/data/732a81448bad0b7f939e5009e5a4fdb7dfd5b0f3db1bb177345e28ef5ea2290f"
time="2024-11-15T03:52:01Z" level=debug msg="Running /var/lib/rancher/k3s/data/732a81448bad0b7f939e5009e5a4fdb7dfd5b0f3db1bb177345e28ef5ea2290f/bin/kubectl [kubectl get nodes]"
NAME               STATUS   ROLES                  AGE   VERSION
ip-172-31-14-14    Ready    <none>                 13m   v1.29.10+k3s-b91bdd32
ip-172-31-15-153   Ready    control-plane,master   14m   v1.29.10+k3s-b91bdd32
ip-172-31-15-186   Ready    etcd                   14m   v1.29.10+k3s-b91bdd32
ip-172-31-7-89     Ready    etcd                   14m   v1.29.10+k3s-b91bdd32