Alerts duplication in HA mode

[nip-was-here]

What did you do?

Stack used: multiple vmalert (VictoriaMetrics) and several Alertmanager instances in HA cluster mode.

When there is a large number of alerts, duplicates of some alerts begin to appear on the receivers of Alertmanager.

What did you expect to see?

Stable work of deduplication in HA cluster mode with only one alert received per unique alert from all vmalert instances.

What did you see instead? Under which circumstances?

Some of alerts received multiple times

Environment

• System information:

The issue occurs both on VMs with Debian 12 amd64 and within K8s.

• Alertmanager version:

Found at 0.22.2, reproduced at 0.25.0 too.

• Alertmanager configuration file:

Part of systemd service template in Ansible:

    --web.listen-address=":{{ alertmanager_web_listen_port }}"  \
    --storage.path="{{ alertmanager_bin_dir }}/data" \
    --cluster.advertise-address="{{ ansible_default_ipv4.address }}:{{ alertmanager_cluster_listen_port }}" \
    --cluster.listen-address="{{ ansible_default_ipv4.address }}:{{ alertmanager_cluster_listen_port }}" \
    {% for node in alertmanager_cluster_nodes | sort %}
    --cluster.peer="{{ node.split(':')[0] }}:{{ alertmanager_cluster_listen_port }}" \
    {% endfor %}
    --config.file="{{ alertmanager_conf_dir }}/alertmanager.yml" \
    --web.config.file="{{ alertmanager_conf_dir }}/webconfig.yml" \
    --log.level="{{ alertmanager_log_level }}"