The vip-manager on "replica" won't create the vip after "leader" reboot

[oraclepeople]

I've set up a PostgreSQL cluster with Patroni, an etcd server, and vip-manager. Everything works fine until recently.

The "leader" node reboots at 4:15 am every Monday for weekly updates. However, the vip-manager on the "replica" node doesn't bring up the desired VIP after this.

In the log below, the leader node is "tomato" and the replica is "green." The VIP address (10.70.253.129) is used for Postgres.

Initially, the status of 10.70.253.129 was correct (state is false, desired false) as "green" was a replica. After "tomato" rebooted, Patroni on "green" promoted it to leader. I expected vip-manager to assign the VIP 10.70.253.129 to "green."

However, this didn't happen. Its status remained incorrect (state is false, desired false) instead of the expected (state is true, desired true).

If I restart the vip-manager service ([email protected]) on the new leader "green," it correctly assigns the VIP 10.70.253.129 to "green."

Can anyone shed some light on this issue? Thanks!

Jun 17 04:16:01 green.abc.com patroni[1313]: 2024-06-17 04:16:01,931 INFO: no action. I am (green), a secondary, and following a leader (tomato)
Jun 17 04:16:03 green.abc.com vip-manager[1166]: 2024/06/17 04:16:03 IP address 10.70.253.129/24 state is false, desired false

# leader node rebooting and "green" promotedto "leader"

Jun 17 04:16:05 green.abc.com patroni[1313]: 2024-06-17 04:16:05,659 WARNING: Request failed to tomato: GET https://tomato.abc.com:8008/patroni (HTTPSConnectionPool(host='tomato.abc.com', port=8008): Max retries exceeded with url: /patroni (Caused by SSLError(SSLError("bad handshake: SysCallError(104, 'ECONNRESET')"))))
Jun 17 04:16:05 green.abc.com vip-manager[1167]: 2024/06/17 04:16:05 Current Leader from DCS: green
Jun 17 04:16:05 green.abc.com patroni[1313]: 2024-06-17 04:16:05,708 WARNING: Could not activate Linux watchdog device: Can't open watchdog device: [Errno 2] No such file or directory: '/dev/watchdog'
Jun 17 04:16:05 green.abc.com patroni[1313]: 2024-06-17 04:16:05,754 INFO: promoted self to leader by acquiring session lock
Jun 17 04:16:05 green.abc.com patroni[2014617]: server promoting
...

# Patroni knew "green" was leader but vip-manager didn't think so

Jun 17 04:16:06 green.abc.com patroni[1313]: 2024-06-17 04:16:06,917 INFO: no action. I am (green), the leader with the lock

Jun 17 04:16:43 green.abc.com vip-manager[1166]: 2024/06/17 04:16:43 IP address 10.70.253.129/24 state is false, desired false

Environment information
RHEL 9.4
etcd Version: 3.5.13
patroni version 3.3.0
vip-manager 2.5.0

vip-manager configuration

more /etc/vip-manager/10.70.25.241.yml
# This file is Ansible managed. Manual changes to it will be lost at some point. Template: patroni/templates/vip-manager/vip-manager.yml
# leader vip configuration file
# config for vip-manager by Cybertec Schönig & Schönig GmbH

# time (in milliseconds) after which vip-manager wakes up and checks if it needs to register or release ip addresses.
interval: 1000

# the etcd or consul key which vip-manager will regularly poll.
trigger-key: "/abc-dev/abc-dev-16/leader"
# if the value of the above key matches the trigger-value (often the hostname of this host), vip-manager will try to add the virtual ip address to the interface specified in Iface
trigger-value: "green"

ip: 10.70.254.241 # the virtual ip address to manage
netmask: 24 # netmask for the virtual ip
interface: eth0 #interface to which the virtual ip will be added

# how the virtual ip should be managed. we currently support "ip addr add/remove" through shell commands or the Hetzner api
manager-type: basic # possible values: basic, hetzner .

dcs-type: etcd # etcd or consul
# a list that contains all endpoints to which etcd could talk.
dcs-endpoints:
  - https://beige.abc.com:2379
  - https://coral.abc.com:2379
  - https://snow.abc.com:2379
  # A single list-item is also fine.
  # consul will always only use the first entry from this list.
  # For consul, you'll obviously need to change the port to 8500. Unless you're using a different one. Maybe you're a rebel and are running consul on port 2379? Just to confuse people? Why would you do that? Oh, I get it.

# etcd_user: "patroni"
# etcd_password: "Julian's secret password"
# don't worry about parameter with a prefix that doesn't match the endpoint_type. You can write anything there, I won't even look at it.
# consul_token: "Julian's secret token"

# how often things should be retried and how long to wait between retries. (currently only affects arpClient)
retry-num: 2
retry-after: 250  #in milliseconds

The etcd server configuration:

# This file is Ansible managed. Manual changes to it will be lost at some point. Template: etcd/templates/etcd.conf
# Member
ETCD_NAME="salmon.abc.com"
ETCD_LISTEN_PEER_URLS="https://10.70.13.5:2380"
ETCD_LISTEN_CLIENT_URLS="https://10.70.13.5:2379"
ETCD_DATA_DIR=/var/lib/etcd/
ETCD_HEARTBEAT_INTERVAL=500
ETCD_ELECTION_TIMEOUT=2500

# Cluster
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://salmon.abc.com:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://salmon.abc.com:2379"
ETCD_INITIAL_CLUSTER="salmon.abc.com=https://salmon.abc.com:2380,khaki.abc.com=https://khaki.abc.com:2380,sienna.abc.com=https://sienna.abc.com:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd_hapg_prod"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_STRICT_RECONFIG_CHECK="true"
ETCD_ENABLE_V2="true"

ETCD_AUTO_COMPACTION_MODE='periodic'
ETCD_AUTO_COMPACTION_RETENTION=1

# Proxy
ETCD_PROXY="off"

# Secruity
ETCD_CERT_FILE="/etc/pki/tls/certs/STAR_abc.com_etcd.crt"
ETCD_KEY_FILE="/etc/pki/tls/private/STAR_abc.com_etcd.key"
ETCD_CLIENT_CERT_AUTH="false"
#ETCD_TRUSTED_CA_FILE=""
ETCD_AUTO_TLS="false"
ETCD_PEER_CERT_FILE="/etc/pki/tls/certs/STAR_abc.com_etcd.crt"
ETCD_PEER_KEY_FILE="/etc/pki/tls/private/STAR_abc.com_etcd.key"
ETCD_PEER_CLIENT_CERT_AUTH="false"
ETCD_PEER_TRUSTED_CA_FILE="/etc/pki/tls/certs/ca-bundle.crt"
ETCD_PEER_AUTO_TLS="false"

[pashagolub]

Hello. vip-manager uses etcd as the source of truth. If etcd still holds the wrong leader (for whatever reason) vip-manager will do nothing.

Can you confirm that etcd changes the leader value after the primary reboots?