Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Kube-vip does not have ability to set FQDN address in kube-vip settings #11750

Open
vtlrazin opened this issue Nov 27, 2024 · 0 comments
Open

Kube-vip does not have ability to set FQDN address in kube-vip settings #11750

vtlrazin opened this issue Nov 27, 2024 · 0 comments
Labels
kind/bug Categorizes issue or PR as related to a bug.

Comments

@vtlrazin
Copy link

vtlrazin commented Nov 27, 2024 •
edited
Loading

What happened?

Hello,
I'm created in my DNS server A record with:
10.0.110.10 kube-vip.mydomain.org

After set up kube-vip parameters in the inventory/mycluster/group_vars/k8s_cluster/addons.yml

# Kube VIP
kube_vip_enabled: true
kube_vip_arp_enabled: true
kube_vip_controlplane_enabled: true
kube_vip_address: 10.0.110.10
loadbalancer_apiserver:
  address: "{{ kube_vip_address }}"
  port: 6443
# kube_vip_interface: eth0
kube_vip_services_enabled: true
# kube_vip_dns_mode: first
# kube_vip_cp_detect: false
# kube_vip_leasename: plndr-cp-lock
# kube_vip_enable_node_labeling: false

I got kubeconfig with unresolvable by my DNS server FQDN address:

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: .......
    server: https://lb-apiserver.kubernetes.local:6443
  name: cluster.local
contexts:
- context:
    cluster: cluster.local
    user: kubernetes-admin
  name: [email protected]
current-context: [email protected]
kind: Config
preferences: {}
......

What did you expect to happen?

I would like to get kubeconfig file with my FQDN kube-vip address like:

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: .......
    server: https://kube-vip.mydomain.org:6443
  name: cluster.local
contexts:
- context:
    cluster: cluster.local
    user: kubernetes-admin
  name: [email protected]
current-context: [email protected]
kind: Config
preferences: {}
...........

How can we reproduce it (as minimally and precisely as possible)?

Standard deployment K8s cluster by kubespray with enabled kube-vip
Example of my deployment hosts.yaml configuration file:

all:
  hosts:
    master1:
      ansible_host: 10.0.110.1
      ip: 10.0.110.1
      access_ip: 10.0.110.1
    master2:
      ansible_host: 10.0.110.2
      ip: 10.0.110.2
      access_ip: 10.0.110.2
    master3:
      ansible_host: 10.0.110.3
      ip: 10.0.110.3
      access_ip: 10.0.110.3
    worker1:
      ansible_host: 10.0.110.21
      ip: 10.0.110.21
      access_ip: 10.0.110.21
    worker2:
      ansible_host: 10.0.110.22
      ip: 10.0.110.22
      access_ip: 10.0.110.22
  children:
    kube_control_plane:
      hosts:
        master1:
        master2:
        master3:
    kube_node:
      hosts:
        worker1:
        worker2:
    etcd:
      hosts:
        master1:
        master2:
        master3:
    k8s_cluster:
      children:
        kube_control_plane:
        kube_node:
    calico_rr:
      hosts: {}

OS

Linux 6.8.0-49-generic x86_64
PRETTY_NAME="Ubuntu 24.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.1 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo

Version of Ansible

ansible [core 2.16.13]
config file = /home/depuser/kubespray/kubespray-2.26.0/ansible.cfg
configured module search path = ['/home/depuser/kubespray/kubespray-2.26.0/library']
ansible python module location = /home/depuser/kubespray/kubespray-2.26.0/venv/lib64/python3.12/site-packages/ansible
ansible collection location = /home/depuser/.ansible/collections:/usr/share/ansible/collections
executable location = /home/depuser/kubespray/kubespray-2.26.0/venv/bin/ansible
python version = 3.12.1 (main, Oct 7 2024, 00:00:00) [GCC 11.4.1 20231218 (Red Hat 11.4.1-3)] (/home/depuser/kubespray/kubespray-2.26.0/venv/bin/python3)
jinja version = 3.1.4
libyaml = True

Version of Python

Python 3.12.1

Version of Kubespray (commit)

2.26.0

Network plugin used

custom_cni

Full inventory with variables

master1 | SUCCESS => {
"hostvars[inventory_hostname]": {
"access_ip": "10.0.110.1",
"allow_unsupported_distribution_setup": false,
"ansible_check_mode": false,
"ansible_config_file": "/home/depuser/kubespray/kubespray-2.26.0/ansible.cfg",
"ansible_diff_mode": false,
"ansible_facts": {},
"ansible_forks": 5,
"ansible_host": "10.0.110.1",
"ansible_inventory_sources": [
"/home/depuser/kubespray/kubespray-2.26.0/inventory/mycluster/hosts.yaml"
],
"ansible_playbook_python": "/home/depuser/kubespray/kubespray-2.26.0/venv/bin/python3",
"ansible_verbosity": 0,
"ansible_version": {
"full": "2.16.13",
"major": 2,
"minor": 16,
"revision": 13,
"string": "2.16.13"
},
"argocd_enabled": false,
"auto_renew_certificates": false,
"bin_dir": "/usr/local/bin",
"calico_cni_name": "k8s-pod-network",
"calico_pool_blocksize": 26,
"cephfs_provisioner_enabled": false,
"cert_manager_enabled": false,
"cert_manager_namespace": "cert-manager",
"cert_manager_tolerations": [
{
"effect": "NoSchedule",
"key": "node-role.kubernetes.io/control-plane"
}
],
"cilium_l2announcements": false,
"cluster_name": "cluster.local",
"container_manager": "containerd",
"coredns_k8s_external_zone": "k8s_external.local",
"credentials_dir": "/home/depuser/kubespray/kubespray-2.26.0/inventory/mycluster/credentials",
"default_kubelet_config_dir": "/etc/kubernetes/dynamic_kubelet_dir",
"deploy_netchecker": false,
"dns_domain": "cluster.local",
"dns_mode": "coredns",
"docker_bin_dir": "/usr/bin",
"docker_container_storage_setup": false,
"docker_daemon_graph": "/var/lib/docker",
"docker_dns_servers_strict": false,
"docker_iptables_enabled": "false",
"docker_log_opts": "--log-opt max-size=50m --log-opt max-file=5",
"docker_rpm_keepcache": 1,
"enable_coredns_k8s_endpoint_pod_names": false,
"enable_coredns_k8s_external": false,
"enable_dual_stack_networks": false,
"enable_nat_default_gateway": true,
"enable_nodelocaldns": false,
"enable_nodelocaldns_secondary": false,
"etcd_data_dir": "/var/lib/etcd",
"etcd_deployment_type": "host",
"event_ttl_duration": "1h0m0s",
"gateway_api_enabled": false,
"group_names": [
"etcd",
"k8s_cluster",
"kube_control_plane"
],
"groups": {
"all": [
"master1",
"master2",
"master3",
"worker1",
"worker2"
],
"calico_rr": [],
"etcd": [
"master1",
"master2",
"master3"
],
"k8s_cluster": [
"master1",
"master2",
"master3",
"worker1",
"worker2"
],
"kube_control_plane": [
"master1",
"master2",
"master3"
],
"kube_node": [
"worker1",
"worker2"
],
"ungrouped": []
},
"helm_enabled": true,
"ingress_alb_enabled": false,
"ingress_nginx_enabled": false,
"ingress_publish_status_address": "",
"inventory_dir": "/home/depuser/kubespray/kubespray-2.26.0/inventory/mycluster",
"inventory_file": "/home/depuser/kubespray/kubespray-2.26.0/inventory/mycluster/hosts.yaml",
"inventory_hostname": "master1",
"inventory_hostname_short": "master1",
"ip": "10.0.110.1",
"k8s_image_pull_policy": "IfNotPresent",
"kata_containers_enabled": false,
"krew_enabled": false,
"krew_root_dir": "/usr/local/krew",
"kube_api_anonymous_auth": true,
"kube_apiserver_ip": "10.233.0.1",
"kube_apiserver_port": 6443,
"kube_cert_dir": "/etc/kubernetes/ssl",
"kube_cert_group": "kube-cert",
"kube_config_dir": "/etc/kubernetes",
"kube_encrypt_secret_data": false,
"kube_log_level": 2,
"kube_manifest_dir": "/etc/kubernetes/manifests",
"kube_network_node_prefix": 24,
"kube_network_node_prefix_ipv6": 120,
"kube_network_plugin": "cni",
"kube_network_plugin_multus": false,
"kube_ovn_default_gateway_check": true,
"kube_ovn_default_logical_gateway": false,
"kube_ovn_default_vlan_id": 100,
"kube_ovn_dpdk_enabled": false,
"kube_ovn_enable_external_vpc": true,
"kube_ovn_enable_lb": true,
"kube_ovn_enable_np": true,
"kube_ovn_enable_ssl": false,
"kube_ovn_encap_checksum": true,
"kube_ovn_external_address": "8.8.8.8",
"kube_ovn_external_address_ipv6": "2400:3200::1",
"kube_ovn_external_dns": "alauda.cn",
"kube_ovn_hw_offload": false,
"kube_ovn_ic_autoroute": true,
"kube_ovn_ic_dbhost": "127.0.0.1",
"kube_ovn_ic_enable": false,
"kube_ovn_ic_zone": "kubernetes",
"kube_ovn_network_type": "geneve",
"kube_ovn_node_switch_cidr": "100.64.0.0/16",
"kube_ovn_node_switch_cidr_ipv6": "fd00:100:64::/64",
"kube_ovn_pod_nic_type": "veth_pair",
"kube_ovn_traffic_mirror": false,
"kube_ovn_tunnel_type": "geneve",
"kube_ovn_vlan_name": "product",
"kube_owner": "kube",
"kube_pods_subnet": "10.233.64.0/18",
"kube_pods_subnet_ipv6": "fd85:ee78:d8a6:8607::1:0000/112",
"kube_proxy_mode": "ipvs",
"kube_proxy_nodeport_addresses": [],
"kube_proxy_remove": true,
"kube_proxy_strict_arp": true,
"kube_read_only_port": 10255,
"kube_script_dir": "/usr/local/bin/kubernetes-scripts",
"kube_service_addresses": "10.233.0.0/18",
"kube_service_addresses_ipv6": "fd85:ee78:d8a6:8607::1000/116",
"kube_token_dir": "/etc/kubernetes/tokens",
"kube_version": "v1.30.4",
"kube_vip_address": "10.0.110.10",
"kube_vip_arp_enabled": true,
"kube_vip_controlplane_enabled": true,
"kube_vip_enabled": true,
"kube_vip_services_enabled": true,
"kube_webhook_token_auth": false,
"kube_webhook_token_auth_url_skip_tls_verify": false,
"kubeadm_certificate_key": "440893082aca80c3740a2c94c7bc1a1bef80adcaa2f7f2c5ff20ca94dad51a8e",
"kubeadm_patches": {
"dest_dir": "/etc/kubernetes/patches",
"enabled": false,
"source_dir": "/home/depuser/kubespray/kubespray-2.26.0/inventory/mycluster/patches"
},
"kubernetes_audit": false,
"loadbalancer_apiserver": {
"address": "10.0.110.10",
"port": 6443
},
"loadbalancer_apiserver_healthcheck_port": 8081,
"loadbalancer_apiserver_port": 6443,
"local_path_provisioner_enabled": false,
"local_release_dir": "/tmp/releases",
"local_volume_provisioner_enabled": false,
"macvlan_interface": "eth1",
"metallb_enabled": false,
"metallb_namespace": "metallb-system",
"metallb_speaker_enabled": false,
"metrics_server_enabled": false,
"ndots": 2,
"no_proxy_exclude_workers": false,
"node_feature_discovery_enabled": false,
"node_labels": {
"k8s.ovn.org/zone-name": "master1"
},
"nodelocaldns_bind_metrics_host_ip": false,
"nodelocaldns_health_port": 9254,
"nodelocaldns_ip": "169.254.25.10",
"nodelocaldns_second_health_port": 9256,
"nodelocaldns_secondary_skew_seconds": 5,
"ntp_enabled": false,
"ntp_manage_config": false,
"ntp_servers": [
"0.pool.ntp.org iburst",
"1.pool.ntp.org iburst",
"2.pool.ntp.org iburst",
"3.pool.ntp.org iburst"
],
"omit": "__omit_place_holder__720105600654639cccb72727e5e6a48b9fc8299f",
"persistent_volumes_enabled": false,
"playbook_dir": "/home/depuser/kubespray/kubespray-2.26.0",
"populate_inventory_to_hosts_file": false,
"rbd_provisioner_enabled": false,
"registry_enabled": false,
"remove_anonymous_access": false,
"resolvconf_mode": "host_resolvconf",
"retry_stagger": 5,
"skydns_server": "10.233.0.3",
"skydns_server_secondary": "10.233.0.4",
"systemd_resolved_disable_stub_listener": true,
"unsafe_show_logs": false,
"volume_cross_zone_attachment": false
}
}
master3 | SUCCESS => {
"hostvars[inventory_hostname]": {
"access_ip": "10.0.110.3",
"allow_unsupported_distribution_setup": false,
"ansible_check_mode": false,
"ansible_config_file": "/home/depuser/kubespray/kubespray-2.26.0/ansible.cfg",
"ansible_diff_mode": false,
"ansible_facts": {},
"ansible_forks": 5,
"ansible_host": "10.0.110.3",
"ansible_inventory_sources": [
"/home/depuser/kubespray/kubespray-2.26.0/inventory/mycluster/hosts.yaml"
],
"ansible_playbook_python": "/home/depuser/kubespray/kubespray-2.26.0/venv/bin/python3",
"ansible_verbosity": 0,
"ansible_version": {
"full": "2.16.13",
"major": 2,
"minor": 16,
"revision": 13,
"string": "2.16.13"
},
"argocd_enabled": false,
"auto_renew_certificates": false,
"bin_dir": "/usr/local/bin",
"calico_cni_name": "k8s-pod-network",
"calico_pool_blocksize": 26,
"cephfs_provisioner_enabled": false,
"cert_manager_enabled": false,
"cert_manager_namespace": "cert-manager",
"cert_manager_tolerations": [
{
"effect": "NoSchedule",
"key": "node-role.kubernetes.io/control-plane"
}
],
"cilium_l2announcements": false,
"cluster_name": "cluster.local",
"container_manager": "containerd",
"coredns_k8s_external_zone": "k8s_external.local",
"credentials_dir": "/home/depuser/kubespray/kubespray-2.26.0/inventory/mycluster/credentials",
"default_kubelet_config_dir": "/etc/kubernetes/dynamic_kubelet_dir",
"deploy_netchecker": false,
"dns_domain": "cluster.local",
"dns_mode": "coredns",
"docker_bin_dir": "/usr/bin",
"docker_container_storage_setup": false,
"docker_daemon_graph": "/var/lib/docker",
"docker_dns_servers_strict": false,
"docker_iptables_enabled": "false",
"docker_log_opts": "--log-opt max-size=50m --log-opt max-file=5",
"docker_rpm_keepcache": 1,
"enable_coredns_k8s_endpoint_pod_names": false,
"enable_coredns_k8s_external": false,
"enable_dual_stack_networks": false,
"enable_nat_default_gateway": true,
"enable_nodelocaldns": false,
"enable_nodelocaldns_secondary": false,
"etcd_data_dir": "/var/lib/etcd",
"etcd_deployment_type": "host",
"event_ttl_duration": "1h0m0s",
"gateway_api_enabled": false,
"group_names": [
"etcd",
"k8s_cluster",
"kube_control_plane"
],
"groups": {
"all": [
"master1",
"master2",
"master3",
"worker1",
"worker2"
],
"calico_rr": [],
"etcd": [
"master1",
"master2",
"master3"
],
"k8s_cluster": [
"master1",
"master2",
"master3",
"worker1",
"worker2"
],
"kube_control_plane": [
"master1",
"master2",
"master3"
],
"kube_node": [
"worker1",
"worker2"
],
"ungrouped": []
},
"helm_enabled": true,
"ingress_alb_enabled": false,
"ingress_nginx_enabled": false,
"ingress_publish_status_address": "",
"inventory_dir": "/home/depuser/kubespray/kubespray-2.26.0/inventory/mycluster",
"inventory_file": "/home/depuser/kubespray/kubespray-2.26.0/inventory/mycluster/hosts.yaml",
"inventory_hostname": "master3",
"inventory_hostname_short": "master3",
"ip": "10.0.110.3",
"k8s_image_pull_policy": "IfNotPresent",
"kata_containers_enabled": false,
"krew_enabled": false,
"krew_root_dir": "/usr/local/krew",
"kube_api_anonymous_auth": true,
"kube_apiserver_ip": "10.233.0.1",
"kube_apiserver_port": 6443,
"kube_cert_dir": "/etc/kubernetes/ssl",
"kube_cert_group": "kube-cert",
"kube_config_dir": "/etc/kubernetes",
"kube_encrypt_secret_data": false,
"kube_log_level": 2,
"kube_manifest_dir": "/etc/kubernetes/manifests",
"kube_network_node_prefix": 24,
"kube_network_node_prefix_ipv6": 120,
"kube_network_plugin": "cni",
"kube_network_plugin_multus": false,
"kube_ovn_default_gateway_check": true,
"kube_ovn_default_logical_gateway": false,
"kube_ovn_default_vlan_id": 100,
"kube_ovn_dpdk_enabled": false,
"kube_ovn_enable_external_vpc": true,
"kube_ovn_enable_lb": true,
"kube_ovn_enable_np": true,
"kube_ovn_enable_ssl": false,
"kube_ovn_encap_checksum": true,
"kube_ovn_external_address": "8.8.8.8",
"kube_ovn_external_address_ipv6": "2400:3200::1",
"kube_ovn_external_dns": "alauda.cn",
"kube_ovn_hw_offload": false,
"kube_ovn_ic_autoroute": true,
"kube_ovn_ic_dbhost": "127.0.0.1",
"kube_ovn_ic_enable": false,
"kube_ovn_ic_zone": "kubernetes",
"kube_ovn_network_type": "geneve",
"kube_ovn_node_switch_cidr": "100.64.0.0/16",
"kube_ovn_node_switch_cidr_ipv6": "fd00:100:64::/64",
"kube_ovn_pod_nic_type": "veth_pair",
"kube_ovn_traffic_mirror": false,
"kube_ovn_tunnel_type": "geneve",
"kube_ovn_vlan_name": "product",
"kube_owner": "kube",
"kube_pods_subnet": "10.233.64.0/18",
"kube_pods_subnet_ipv6": "fd85:ee78:d8a6:8607::1:0000/112",
"kube_proxy_mode": "ipvs",
"kube_proxy_nodeport_addresses": [],
"kube_proxy_remove": true,
"kube_proxy_strict_arp": true,
"kube_read_only_port": 10255,
"kube_script_dir": "/usr/local/bin/kubernetes-scripts",
"kube_service_addresses": "10.233.0.0/18",
"kube_service_addresses_ipv6": "fd85:ee78:d8a6:8607::1000/116",
"kube_token_dir": "/etc/kubernetes/tokens",
"kube_version": "v1.30.4",
"kube_vip_address": "10.0.110.10",
"kube_vip_arp_enabled": true,
"kube_vip_controlplane_enabled": true,
"kube_vip_enabled": true,
"kube_vip_services_enabled": true,
"kube_webhook_token_auth": false,
"kube_webhook_token_auth_url_skip_tls_verify": false,
"kubeadm_certificate_key": "440893082aca80c3740a2c94c7bc1a1bef80adcaa2f7f2c5ff20ca94dad51a8e",
"kubeadm_patches": {
"dest_dir": "/etc/kubernetes/patches",
"enabled": false,
"source_dir": "/home/depuser/kubespray/kubespray-2.26.0/inventory/mycluster/patches"
},
"kubernetes_audit": false,
"loadbalancer_apiserver": {
"address": "10.0.110.10",
"port": 6443
},
"loadbalancer_apiserver_healthcheck_port": 8081,
"loadbalancer_apiserver_port": 6443,
"local_path_provisioner_enabled": false,
"local_release_dir": "/tmp/releases",
"local_volume_provisioner_enabled": false,
"macvlan_interface": "eth1",
"metallb_enabled": false,
"metallb_namespace": "metallb-system",
"metallb_speaker_enabled": false,
"metrics_server_enabled": false,
"ndots": 2,
"no_proxy_exclude_workers": false,
"node_feature_discovery_enabled": false,
"node_labels": {
"k8s.ovn.org/zone-name": "master3"
},
"nodelocaldns_bind_metrics_host_ip": false,
"nodelocaldns_health_port": 9254,
"nodelocaldns_ip": "169.254.25.10",
"nodelocaldns_second_health_port": 9256,
"nodelocaldns_secondary_skew_seconds": 5,
"ntp_enabled": false,
"ntp_manage_config": false,
"ntp_servers": [
"0.pool.ntp.org iburst",
"1.pool.ntp.org iburst",
"2.pool.ntp.org iburst",
"3.pool.ntp.org iburst"
],
"omit": "__omit_place_holder__720105600654639cccb72727e5e6a48b9fc8299f",
"persistent_volumes_enabled": false,
"playbook_dir": "/home/depuser/kubespray/kubespray-2.26.0",
"populate_inventory_to_hosts_file": false,
"rbd_provisioner_enabled": false,
"registry_enabled": false,
"remove_anonymous_access": false,
"resolvconf_mode": "host_resolvconf",
"retry_stagger": 5,
"skydns_server": "10.233.0.3",
"skydns_server_secondary": "10.233.0.4",
"systemd_resolved_disable_stub_listener": true,
"unsafe_show_logs": false,
"volume_cross_zone_attachment": false
}
}
worker2 | SUCCESS => {
"hostvars[inventory_hostname]": {
"access_ip": "10.0.110.22",
"allow_unsupported_distribution_setup": false,
"ansible_check_mode": false,
"ansible_config_file": "/home/depuser/kubespray/kubespray-2.26.0/ansible.cfg",
"ansible_diff_mode": false,
"ansible_facts": {},
"ansible_forks": 5,
"ansible_host": "10.0.110.22",
"ansible_inventory_sources": [
"/home/depuser/kubespray/kubespray-2.26.0/inventory/mycluster/hosts.yaml"
],
"ansible_playbook_python": "/home/depuser/kubespray/kubespray-2.26.0/venv/bin/python3",
"ansible_verbosity": 0,
"ansible_version": {
"full": "2.16.13",
"major": 2,
"minor": 16,
"revision": 13,
"string": "2.16.13"
},
"argocd_enabled": false,
"auto_renew_certificates": false,
"bin_dir": "/usr/local/bin",
"calico_cni_name": "k8s-pod-network",
"calico_pool_blocksize": 26,
"cephfs_provisioner_enabled": false,
"cert_manager_enabled": false,
"cert_manager_namespace": "cert-manager",
"cert_manager_tolerations": [
{
"effect": "NoSchedule",
"key": "node-role.kubernetes.io/control-plane"
}
],
"cilium_l2announcements": false,
"cluster_name": "cluster.local",
"container_manager": "containerd",
"coredns_k8s_external_zone": "k8s_external.local",
"credentials_dir": "/home/depuser/kubespray/kubespray-2.26.0/inventory/mycluster/credentials",
"default_kubelet_config_dir": "/etc/kubernetes/dynamic_kubelet_dir",
"deploy_netchecker": false,
"dns_domain": "cluster.local",
"dns_mode": "coredns",
"docker_bin_dir": "/usr/bin",
"docker_container_storage_setup": false,
"docker_daemon_graph": "/var/lib/docker",
"docker_dns_servers_strict": false,
"docker_iptables_enabled": "false",
"docker_log_opts": "--log-opt max-size=50m --log-opt max-file=5",
"docker_rpm_keepcache": 1,
"enable_coredns_k8s_endpoint_pod_names": false,
"enable_coredns_k8s_external": false,
"enable_dual_stack_networks": false,
"enable_nat_default_gateway": true,
"enable_nodelocaldns": false,
"enable_nodelocaldns_secondary": false,
"etcd_data_dir": "/var/lib/etcd",
"etcd_deployment_type": "host",
"event_ttl_duration": "1h0m0s",
"gateway_api_enabled": false,
"group_names": [
"k8s_cluster",
"kube_node"
],
"groups": {
"all": [
"master1",
"master2",
"master3",
"worker1",
"worker2"
],
"calico_rr": [],
"etcd": [
"master1",
"master2",
"master3"
],
"k8s_cluster": [
"master1",
"master2",
"master3",
"worker1",
"worker2"
],
"kube_control_plane": [
"master1",
"master2",
"master3"
],
"kube_node": [
"worker1",
"worker2"
],
"ungrouped": [
"worker4",
"worker5",
"worker6"
]
},
"helm_enabled": true,
"ingress_alb_enabled": false,
"ingress_nginx_enabled": false,
"ingress_publish_status_address": "",
"inventory_dir": "/home/depuser/kubespray/kubespray-2.26.0/inventory/mycluster",
"inventory_file": "/home/depuser/kubespray/kubespray-2.26.0/inventory/mycluster/hosts.yaml",
"inventory_hostname": "worker2",
"inventory_hostname_short": "worker2",
"ip": "10.0.110.22",
"k8s_image_pull_policy": "IfNotPresent",
"kata_containers_enabled": false,
"krew_enabled": false,
"krew_root_dir": "/usr/local/krew",
"kube_api_anonymous_auth": true,
"kube_apiserver_ip": "10.233.0.1",
"kube_apiserver_port": 6443,
"kube_cert_dir": "/etc/kubernetes/ssl",
"kube_cert_group": "kube-cert",
"kube_config_dir": "/etc/kubernetes",
"kube_encrypt_secret_data": false,
"kube_log_level": 2,
"kube_manifest_dir": "/etc/kubernetes/manifests",
"kube_network_node_prefix": 24,
"kube_network_node_prefix_ipv6": 120,
"kube_network_plugin": "cni",
"kube_network_plugin_multus": false,
"kube_ovn_default_gateway_check": true,
"kube_ovn_default_logical_gateway": false,
"kube_ovn_default_vlan_id": 100,
"kube_ovn_dpdk_enabled": false,
"kube_ovn_enable_external_vpc": true,
"kube_ovn_enable_lb": true,
"kube_ovn_enable_np": true,
"kube_ovn_enable_ssl": false,
"kube_ovn_encap_checksum": true,
"kube_ovn_external_address": "8.8.8.8",
"kube_ovn_external_address_ipv6": "2400:3200::1",
"kube_ovn_external_dns": "alauda.cn",
"kube_ovn_hw_offload": false,
"kube_ovn_ic_autoroute": true,
"kube_ovn_ic_dbhost": "127.0.0.1",
"kube_ovn_ic_enable": false,
"kube_ovn_ic_zone": "kubernetes",
"kube_ovn_network_type": "geneve",
"kube_ovn_node_switch_cidr": "100.64.0.0/16",
"kube_ovn_node_switch_cidr_ipv6": "fd00:100:64::/64",
"kube_ovn_pod_nic_type": "veth_pair",
"kube_ovn_traffic_mirror": false,
"kube_ovn_tunnel_type": "geneve",
"kube_ovn_vlan_name": "product",
"kube_owner": "kube",
"kube_pods_subnet": "10.233.64.0/18",
"kube_pods_subnet_ipv6": "fd85:ee78:d8a6:8607::1:0000/112",
"kube_proxy_mode": "ipvs",
"kube_proxy_nodeport_addresses": [],
"kube_proxy_remove": true,
"kube_proxy_strict_arp": true,
"kube_read_only_port": 10255,
"kube_script_dir": "/usr/local/bin/kubernetes-scripts",
"kube_service_addresses": "10.233.0.0/18",
"kube_service_addresses_ipv6": "fd85:ee78:d8a6:8607::1000/116",
"kube_token_dir": "/etc/kubernetes/tokens",
"kube_version": "v1.30.4",
"kube_vip_address": "10.0.110.10",
"kube_vip_arp_enabled": true,
"kube_vip_controlplane_enabled": true,
"kube_vip_enabled": true,
"kube_vip_services_enabled": true,
"kube_webhook_token_auth": false,
"kube_webhook_token_auth_url_skip_tls_verify": false,
"kubeadm_certificate_key": "440893082aca80c3740a2c94c7bc1a1bef80adcaa2f7f2c5ff20ca94dad51a8e",
"kubeadm_patches": {
"dest_dir": "/etc/kubernetes/patches",
"enabled": false,
"source_dir": "/home/depuser/kubespray/kubespray-2.26.0/inventory/mycluster/patches"
},
"kubernetes_audit": false,
"loadbalancer_apiserver": {
"address": "10.0.110.10",
"port": 6443
},
"loadbalancer_apiserver_healthcheck_port": 8081,
"loadbalancer_apiserver_port": 6443,
"local_path_provisioner_enabled": false,
"local_release_dir": "/tmp/releases",
"local_volume_provisioner_enabled": false,
"macvlan_interface": "eth1",
"metallb_enabled": false,
"metallb_namespace": "metallb-system",
"metallb_speaker_enabled": false,
"metrics_server_enabled": false,
"ndots": 2,
"no_proxy_exclude_workers": false,
"node_annotations": {
"k8s.ovn.org/remote-zone-migrated": "worker2"
},
"node_feature_discovery_enabled": false,
"node_labels": {
"k8s.ovn.org/dpu-host": "",
"k8s.ovn.org/zone-name": "worker2",
"node-role.kubernetes.io/worker": ""
},
"nodelocaldns_bind_metrics_host_ip": false,
"nodelocaldns_health_port": 9254,
"nodelocaldns_ip": "169.254.25.10",
"nodelocaldns_second_health_port": 9256,
"nodelocaldns_secondary_skew_seconds": 5,
"ntp_enabled": false,
"ntp_manage_config": false,
"ntp_servers": [
"0.pool.ntp.org iburst",
"1.pool.ntp.org iburst",
"2.pool.ntp.org iburst",
"3.pool.ntp.org iburst"
],
"omit": "__omit_place_holder__720105600654639cccb72727e5e6a48b9fc8299f",
"persistent_volumes_enabled": false,
"playbook_dir": "/home/depuser/kubespray/kubespray-2.26.0",
"populate_inventory_to_hosts_file": false,
"rbd_provisioner_enabled": false,
"registry_enabled": false,
"remove_anonymous_access": false,
"resolvconf_mode": "host_resolvconf",
"retry_stagger": 5,
"skydns_server": "10.233.0.3",
"skydns_server_secondary": "10.233.0.4",
"systemd_resolved_disable_stub_listener": true,
"unsafe_show_logs": false,
"volume_cross_zone_attachment": false
}
}
master2 | SUCCESS => {
"hostvars[inventory_hostname]": {
"access_ip": "10.0.110.2",
"allow_unsupported_distribution_setup": false,
"ansible_check_mode": false,
"ansible_config_file": "/home/depuser/kubespray/kubespray-2.26.0/ansible.cfg",
"ansible_diff_mode": false,
"ansible_facts": {},
"ansible_forks": 5,
"ansible_host": "10.0.110.2",
"ansible_inventory_sources": [
"/home/depuser/kubespray/kubespray-2.26.0/inventory/mycluster/hosts.yaml"
],
"ansible_playbook_python": "/home/depuser/kubespray/kubespray-2.26.0/venv/bin/python3",
"ansible_verbosity": 0,
"ansible_version": {
"full": "2.16.13",
"major": 2,
"minor": 16,
"revision": 13,
"string": "2.16.13"
},
"argocd_enabled": false,
"auto_renew_certificates": false,
"bin_dir": "/usr/local/bin",
"calico_cni_name": "k8s-pod-network",
"calico_pool_blocksize": 26,
"cephfs_provisioner_enabled": false,
"cert_manager_enabled": false,
"cert_manager_namespace": "cert-manager",
"cert_manager_tolerations": [
{
"effect": "NoSchedule",
"key": "node-role.kubernetes.io/control-plane"
}
],
"cilium_l2announcements": false,
"cluster_name": "cluster.local",
"container_manager": "containerd",
"coredns_k8s_external_zone": "k8s_external.local",
"credentials_dir": "/home/depuser/kubespray/kubespray-2.26.0/inventory/mycluster/credentials",
"default_kubelet_config_dir": "/etc/kubernetes/dynamic_kubelet_dir",
"deploy_netchecker": false,
"dns_domain": "cluster.local",
"dns_mode": "coredns",
"docker_bin_dir": "/usr/bin",
"docker_container_storage_setup": false,
"docker_daemon_graph": "/var/lib/docker",
"docker_dns_servers_strict": false,
"docker_iptables_enabled": "false",
"docker_log_opts": "--log-opt max-size=50m --log-opt max-file=5",
"docker_rpm_keepcache": 1,
"enable_coredns_k8s_endpoint_pod_names": false,
"enable_coredns_k8s_external": false,
"enable_dual_stack_networks": false,
"enable_nat_default_gateway": true,
"enable_nodelocaldns": false,
"enable_nodelocaldns_secondary": false,
"etcd_data_dir": "/var/lib/etcd",
"etcd_deployment_type": "host",
"event_ttl_duration": "1h0m0s",
"gateway_api_enabled": false,
"group_names": [
"etcd",
"k8s_cluster",
"kube_control_plane"
],
"groups": {
"all": [
"master1",
"master2",
"master3",
"worker1",
"worker2"
],
"calico_rr": [],
"etcd": [
"master1",
"master2",
"master3"
],
"k8s_cluster": [
"master1",
"master2",
"master3",
"worker1",
"worker2"
],
"kube_control_plane": [
"master1",
"master2",
"master3"
],
"kube_node": [
"worker1",
"worker2"
],
"ungrouped": []
},
"helm_enabled": true,
"ingress_alb_enabled": false,
"ingress_nginx_enabled": false,
"ingress_publish_status_address": "",
"inventory_dir": "/home/depuser/kubespray/kubespray-2.26.0/inventory/mycluster",
"inventory_file": "/home/depuser/kubespray/kubespray-2.26.0/inventory/mycluster/hosts.yaml",
"inventory_hostname": "master2",
"inventory_hostname_short": "master2",
"ip": "10.0.110.2",
"k8s_image_pull_policy": "IfNotPresent",
"kata_containers_enabled": false,
"krew_enabled": false,
"krew_root_dir": "/usr/local/krew",
"kube_api_anonymous_auth": true,
"kube_apiserver_ip": "10.233.0.1",
"kube_apiserver_port": 6443,
"kube_cert_dir": "/etc/kubernetes/ssl",
"kube_cert_group": "kube-cert",
"kube_config_dir": "/etc/kubernetes",
"kube_encrypt_secret_data": false,
"kube_log_level": 2,
"kube_manifest_dir": "/etc/kubernetes/manifests",
"kube_network_node_prefix": 24,
"kube_network_node_prefix_ipv6": 120,
"kube_network_plugin": "cni",
"kube_network_plugin_multus": false,
"kube_ovn_default_gateway_check": true,
"kube_ovn_default_logical_gateway": false,
"kube_ovn_default_vlan_id": 100,
"kube_ovn_dpdk_enabled": false,
"kube_ovn_enable_external_vpc": true,
"kube_ovn_enable_lb": true,
"kube_ovn_enable_np": true,
"kube_ovn_enable_ssl": false,
"kube_ovn_encap_checksum": true,
"kube_ovn_external_address": "8.8.8.8",
"kube_ovn_external_address_ipv6": "2400:3200::1",
"kube_ovn_external_dns": "alauda.cn",
"kube_ovn_hw_offload": false,
"kube_ovn_ic_autoroute": true,
"kube_ovn_ic_dbhost": "127.0.0.1",
"kube_ovn_ic_enable": false,
"kube_ovn_ic_zone": "kubernetes",
"kube_ovn_network_type": "geneve",
"kube_ovn_node_switch_cidr": "100.64.0.0/16",
"kube_ovn_node_switch_cidr_ipv6": "fd00:100:64::/64",
"kube_ovn_pod_nic_type": "veth_pair",
"kube_ovn_traffic_mirror": false,
"kube_ovn_tunnel_type": "geneve",
"kube_ovn_vlan_name": "product",
"kube_owner": "kube",
"kube_pods_subnet": "10.233.64.0/18",
"kube_pods_subnet_ipv6": "fd85:ee78:d8a6:8607::1:0000/112",
"kube_proxy_mode": "ipvs",
"kube_proxy_nodeport_addresses": [],
"kube_proxy_remove": true,
"kube_proxy_strict_arp": true,
"kube_read_only_port": 10255,
"kube_script_dir": "/usr/local/bin/kubernetes-scripts",
"kube_service_addresses": "10.233.0.0/18",
"kube_service_addresses_ipv6": "fd85:ee78:d8a6:8607::1000/116",
"kube_token_dir": "/etc/kubernetes/tokens",
"kube_version": "v1.30.4",
"kube_vip_address": "10.0.110.10",
"kube_vip_arp_enabled": true,
"kube_vip_controlplane_enabled": true,
"kube_vip_enabled": true,
"kube_vip_services_enabled": true,
"kube_webhook_token_auth": false,
"kube_webhook_token_auth_url_skip_tls_verify": false,
"kubeadm_certificate_key": "440893082aca80c3740a2c94c7bc1a1bef80adcaa2f7f2c5ff20ca94dad51a8e",
"kubeadm_patches": {
"dest_dir": "/etc/kubernetes/patches",
"enabled": false,
"source_dir": "/home/depuser/kubespray/kubespray-2.26.0/inventory/mycluster/patches"
},
"kubernetes_audit": false,
"loadbalancer_apiserver": {
"address": "10.0.110.10",
"port": 6443
},
"loadbalancer_apiserver_healthcheck_port": 8081,
"loadbalancer_apiserver_port": 6443,
"local_path_provisioner_enabled": false,
"local_release_dir": "/tmp/releases",
"local_volume_provisioner_enabled": false,
"macvlan_interface": "eth1",
"metallb_enabled": false,
"metallb_namespace": "metallb-system",
"metallb_speaker_enabled": false,
"metrics_server_enabled": false,
"ndots": 2,
"no_proxy_exclude_workers": false,
"node_feature_discovery_enabled": false,
"node_labels": {
"k8s.ovn.org/zone-name": "master2"
},
"nodelocaldns_bind_metrics_host_ip": false,
"nodelocaldns_health_port": 9254,
"nodelocaldns_ip": "169.254.25.10",
"nodelocaldns_second_health_port": 9256,
"nodelocaldns_secondary_skew_seconds": 5,
"ntp_enabled": false,
"ntp_manage_config": false,
"ntp_servers": [
"0.pool.ntp.org iburst",
"1.pool.ntp.org iburst",
"2.pool.ntp.org iburst",
"3.pool.ntp.org iburst"
],
"omit": "__omit_place_holder__720105600654639cccb72727e5e6a48b9fc8299f",
"persistent_volumes_enabled": false,
"playbook_dir": "/home/depuser/kubespray/kubespray-2.26.0",
"populate_inventory_to_hosts_file": false,
"rbd_provisioner_enabled": false,
"registry_enabled": false,
"remove_anonymous_access": false,
"resolvconf_mode": "host_resolvconf",
"retry_stagger": 5,
"skydns_server": "10.233.0.3",
"skydns_server_secondary": "10.233.0.4",
"systemd_resolved_disable_stub_listener": true,
"unsafe_show_logs": false,
"volume_cross_zone_attachment": false
}
}
worker1 | SUCCESS => {
"hostvars[inventory_hostname]": {
"access_ip": "10.0.110.21",
"allow_unsupported_distribution_setup": false,
"ansible_check_mode": false,
"ansible_config_file": "/home/depuser/kubespray/kubespray-2.26.0/ansible.cfg",
"ansible_diff_mode": false,
"ansible_facts": {},
"ansible_forks": 5,
"ansible_host": "10.0.110.21",
"ansible_inventory_sources": [
"/home/depuser/kubespray/kubespray-2.26.0/inventory/mycluster/hosts.yaml"
],
"ansible_playbook_python": "/home/depuser/kubespray/kubespray-2.26.0/venv/bin/python3",
"ansible_verbosity": 0,
"ansible_version": {
"full": "2.16.13",
"major": 2,
"minor": 16,
"revision": 13,
"string": "2.16.13"
},
"argocd_enabled": false,
"auto_renew_certificates": false,
"bin_dir": "/usr/local/bin",
"calico_cni_name": "k8s-pod-network",
"calico_pool_blocksize": 26,
"cephfs_provisioner_enabled": false,
"cert_manager_enabled": false,
"cert_manager_namespace": "cert-manager",
"cert_manager_tolerations": [
{
"effect": "NoSchedule",
"key": "node-role.kubernetes.io/control-plane"
}
],
"cilium_l2announcements": false,
"cluster_name": "cluster.local",
"container_manager": "containerd",
"coredns_k8s_external_zone": "k8s_external.local",
"credentials_dir": "/home/depuser/kubespray/kubespray-2.26.0/inventory/mycluster/credentials",
"default_kubelet_config_dir": "/etc/kubernetes/dynamic_kubelet_dir",
"deploy_netchecker": false,
"dns_domain": "cluster.local",
"dns_mode": "coredns",
"docker_bin_dir": "/usr/bin",
"docker_container_storage_setup": false,
"docker_daemon_graph": "/var/lib/docker",
"docker_dns_servers_strict": false,
"docker_iptables_enabled": "false",
"docker_log_opts": "--log-opt max-size=50m --log-opt max-file=5",
"docker_rpm_keepcache": 1,
"enable_coredns_k8s_endpoint_pod_names": false,
"enable_coredns_k8s_external": false,
"enable_dual_stack_networks": false,
"enable_nat_default_gateway": true,
"enable_nodelocaldns": false,
"enable_nodelocaldns_secondary": false,
"etcd_data_dir": "/var/lib/etcd",
"etcd_deployment_type": "host",
"event_ttl_duration": "1h0m0s",
"gateway_api_enabled": false,
"group_names": [
"k8s_cluster",
"kube_node"
],
"groups": {
"all": [
"master1",
"master2",
"master3",
"worker1",
"worker2"
],
"calico_rr": [],
"etcd": [
"master1",
"master2",
"master3"
],
"k8s_cluster": [
"master1",
"master2",
"master3",
"worker1",
"worker2"
],
"kube_control_plane": [
"master1",
"master2",
"master3"
],
"kube_node": [
"worker1",
"worker2"
],
"ungrouped": []
},
"helm_enabled": true,
"ingress_alb_enabled": false,
"ingress_nginx_enabled": false,
"ingress_publish_status_address": "",
"inventory_dir": "/home/depuser/kubespray/kubespray-2.26.0/inventory/mycluster",
"inventory_file": "/home/depuser/kubespray/kubespray-2.26.0/inventory/mycluster/hosts.yaml",
"inventory_hostname": "worker1",
"inventory_hostname_short": "worker1",
"ip": "10.0.110.21",
"k8s_image_pull_policy": "IfNotPresent",
"kata_containers_enabled": false,
"krew_enabled": false,
"krew_root_dir": "/usr/local/krew",
"kube_api_anonymous_auth": true,
"kube_apiserver_ip": "10.233.0.1",
"kube_apiserver_port": 6443,
"kube_cert_dir": "/etc/kubernetes/ssl",
"kube_cert_group": "kube-cert",
"kube_config_dir": "/etc/kubernetes",
"kube_encrypt_secret_data": false,
"kube_log_level": 2,
"kube_manifest_dir": "/etc/kubernetes/manifests",
"kube_network_node_prefix": 24,
"kube_network_node_prefix_ipv6": 120,
"kube_network_plugin": "cni",
"kube_network_plugin_multus": false,
"kube_ovn_default_gateway_check": true,
"kube_ovn_default_logical_gateway": false,
"kube_ovn_default_vlan_id": 100,
"kube_ovn_dpdk_enabled": false,
"kube_ovn_enable_external_vpc": true,
"kube_ovn_enable_lb": true,
"kube_ovn_enable_np": true,
"kube_ovn_enable_ssl": false,
"kube_ovn_encap_checksum": true,
"kube_ovn_external_address": "8.8.8.8",
"kube_ovn_external_address_ipv6": "2400:3200::1",
"kube_ovn_external_dns": "alauda.cn",
"kube_ovn_hw_offload": false,
"kube_ovn_ic_autoroute": true,
"kube_ovn_ic_dbhost": "127.0.0.1",
"kube_ovn_ic_enable": false,
"kube_ovn_ic_zone": "kubernetes",
"kube_ovn_network_type": "geneve",
"kube_ovn_node_switch_cidr": "100.64.0.0/16",
"kube_ovn_node_switch_cidr_ipv6": "fd00:100:64::/64",
"kube_ovn_pod_nic_type": "veth_pair",
"kube_ovn_traffic_mirror": false,
"kube_ovn_tunnel_type": "geneve",
"kube_ovn_vlan_name": "product",
"kube_owner": "kube",
"kube_pods_subnet": "10.233.64.0/18",
"kube_pods_subnet_ipv6": "fd85:ee78:d8a6:8607::1:0000/112",
"kube_proxy_mode": "ipvs",
"kube_proxy_nodeport_addresses": [],
"kube_proxy_remove": true,
"kube_proxy_strict_arp": true,
"kube_read_only_port": 10255,
"kube_script_dir": "/usr/local/bin/kubernetes-scripts",
"kube_service_addresses": "10.233.0.0/18",
"kube_service_addresses_ipv6": "fd85:ee78:d8a6:8607::1000/116",
"kube_token_dir": "/etc/kubernetes/tokens",
"kube_version": "v1.30.4",
"kube_vip_address": "10.0.110.10",
"kube_vip_arp_enabled": true,
"kube_vip_controlplane_enabled": true,
"kube_vip_enabled": true,
"kube_vip_services_enabled": true,
"kube_webhook_token_auth": false,
"kube_webhook_token_auth_url_skip_tls_verify": false,
"kubeadm_certificate_key": "440893082aca80c3740a2c94c7bc1a1bef80adcaa2f7f2c5ff20ca94dad51a8e",
"kubeadm_patches": {
"dest_dir": "/etc/kubernetes/patches",
"enabled": false,
"source_dir": "/home/depuser/kubespray/kubespray-2.26.0/inventory/mycluster/patches"
},
"kubernetes_audit": false,
"loadbalancer_apiserver": {
"address": "10.0.110.10",
"port": 6443
},
"loadbalancer_apiserver_healthcheck_port": 8081,
"loadbalancer_apiserver_port": 6443,
"local_path_provisioner_enabled": false,
"local_release_dir": "/tmp/releases",
"local_volume_provisioner_enabled": false,
"macvlan_interface": "eth1",
"metallb_enabled": false,
"metallb_namespace": "metallb-system",
"metallb_speaker_enabled": false,
"metrics_server_enabled": false,
"ndots": 2,
"no_proxy_exclude_workers": false,
"node_annotations": {
"k8s.ovn.org/remote-zone-migrated": "worker1"
},
"node_feature_discovery_enabled": false,
"node_labels": {
"k8s.ovn.org/dpu-host": "",
"k8s.ovn.org/zone-name": "worker1",
"node-role.kubernetes.io/worker": ""
},
"nodelocaldns_bind_metrics_host_ip": false,
"nodelocaldns_health_port": 9254,
"nodelocaldns_ip": "169.254.25.10",
"nodelocaldns_second_health_port": 9256,
"nodelocaldns_secondary_skew_seconds": 5,
"ntp_enabled": false,
"ntp_manage_config": false,
"ntp_servers": [
"0.pool.ntp.org iburst",
"1.pool.ntp.org iburst",
"2.pool.ntp.org iburst",
"3.pool.ntp.org iburst"
],
"omit": "__omit_place_holder__720105600654639cccb72727e5e6a48b9fc8299f",
"persistent_volumes_enabled": false,
"playbook_dir": "/home/depuser/kubespray/kubespray-2.26.0",
"populate_inventory_to_hosts_file": false,
"rbd_provisioner_enabled": false,
"registry_enabled": false,
"remove_anonymous_access": false,
"resolvconf_mode": "host_resolvconf",
"retry_stagger": 5,
"skydns_server": "10.233.0.3",
"skydns_server_secondary": "10.233.0.4",
"systemd_resolved_disable_stub_listener": true,
"unsafe_show_logs": false,
"volume_cross_zone_attachment": false
}
}

Command used to invoke ansible

ansible-playbook -i inventory/mycluster/hosts.yaml --become --become-user=root cluster.yml

Output of ansible run

Deployment finished successfully.

Anything else we need to know

No response
@vtlrazin vtlrazin added the kind/bug Categorizes issue or PR as related to a bug. label Nov 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@vtlrazin