Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Etcd cluster keeps getting corrupted on Talos since version 1.8.x upgrade ( etcd 3.5.16) #18933

Open
4 tasks done
rkoosaar opened this issue Nov 21, 2024 · 9 comments
Open
4 tasks done

Etcd cluster keeps getting corrupted on Talos since version 1.8.x upgrade ( etcd 3.5.16) #18933

rkoosaar opened this issue Nov 21, 2024 · 9 comments
Labels
type/bug

Comments

@rkoosaar
Copy link

Bug report criteria

What happened?

Hi folks, I have a really odd issue that I'm troubleshooting. I have a 3 node Talos (1.8.3) cluster at home where etcd (3.5.16) keeps getting corrupted after a while. Initially I thought it could be a disk related issue. So I bought brand new disks and swapped them around. I installed a new cluster last night (around 8pm) and when I woke up this morning (8am) cluster was not working and etcd was reporting cluster corrupted.
Looking at the logs, it seems something happened around 6am, but I'm unable to work out what the cause is.
So far I have redeployed the cluster in the past week 4 times and every time etcd has ended up corrupted.
Any help/guidance to troubleshoot this would be much appreciated.

What did you expect to happen?

Cluster not the get corrupted

How can we reproduce it (as minimally and precisely as possible)?

I'm not 100% sure how this can be reproduced in your env as I don't fully understand why this happens

Anything else we need to know?

I have actually saved a log bundles from all 3 cluster nodes using talosctl -n node_ip support
I'm just not sure which log files would be helpful. If you could advise which logs are needed I can provide them:
the log bundle has folders:
kubernetes-logs
service-logs (etcd.log file here, I pasted it in the relevant log section)
and separately log files:
controller-runtime.log
dmesg.log

Etcd version (please run commands below)

here is the output of EtcdConfigs.etcd.talos.dev file from node1

$ etcd --version
# paste output here
metadata:
    namespace: etcd
    type: EtcdConfigs.etcd.talos.dev
    id: etcd
    version: 1
    owner: etcd.ConfigController
    phase: running
    created: 2024-11-20T20:09:34Z
    updated: 2024-11-20T20:09:34Z
spec:
    advertiseValidSubnets:
        - 10.1.1.0/24
    advertiseExcludeSubnets:
        - 10.1.1.30
    listenValidSubnets:
        - 10.1.1.0/24
    listenExcludeSubnets: []
    image: gcr.io/etcd-development/etcd:v3.5.16
    extraArgs:
        listen-metrics-urls: http://0.0.0.0:2381

$ etcdctl version
# paste output here

Etcd configuration (command line flags or environment variables)

paste your configuration here

metadata:
namespace: etcd
type: EtcdConfigs.etcd.talos.dev
id: etcd
version: 1
owner: etcd.ConfigController
phase: running
created: 2024-11-20T20:09:34Z
updated: 2024-11-20T20:09:34Z
spec:
advertiseValidSubnets:
- 10.1.1.0/24
advertiseExcludeSubnets:
- 10.1.1.30
listenValidSubnets:
- 10.1.1.0/24
listenExcludeSubnets: []
image: gcr.io/etcd-development/etcd:v3.5.16
extraArgs:
listen-metrics-urls: http://0.0.0.0:2381

Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

I'm not 100% sure how I can run the below commands on talos

$ etcdctl member list -w table
# paste output here
talosctl -n 10.1.1.32 etcd members
NODE        ID                 HOSTNAME    PEER URLS                CLIENT URLS              LEARNER
10.1.1.32   05f2a341182982bc   nectarine   https://10.1.1.33:2380   https://10.1.1.33:2379   false
10.1.1.32   4ce3ddbc6cc0469b   melon       https://10.1.1.32:2380   https://10.1.1.32:2379   false
10.1.1.32   d9983876898d0723   mango       https://10.1.1.31:2380   https://10.1.1.31:2379   false
$ etcdctl --endpoints=<member list> endpoint status -w table
# paste output here

Relevant log output

id:0723934b321f01b0>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:39.708327Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"2.25µs","request":"header:<ID:5087822155244138456 > lease_revoke:<id:02bc934b32ef139b>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:39.708355Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"02bc934b32ef139b","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:39.708372Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"0723934b321f01b0","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:39.709189Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"4.598µs","request":"header:<ID:5087822155244138455 > lease_revoke:<id:0723934b321e383c>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:39.709267Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"1.711µs","request":"header:<ID:5087822155244138457 > lease_revoke:<id:02bc934b32ef5746>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:39.709318Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"02bc934b32ef5746","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:39.709336Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"0723934b321e383c","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:39.840207Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"9.109µs","request":"header:<ID:9420566471738703565 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/network/ingress-nginx-internal-leader\" mod_revision:482768 > success:<request_put:<key:\"/registry/leases/network/ingress-nginx-internal-leader\" value_size:439 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:39.846801Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"6.293µs","request":"header:<ID:514416733623534472 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-system/spegel-leader-election\" mod_revision:482773 > success:<request_put:<key:\"/registry/leases/kube-system/spegel-leader-election\" value_size:450 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:39.922691Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"8.287µs","request":"header:<ID:5087822155244138460 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/services/endpoints/openebs-system/openebs.io-local\" mod_revision:482782 > success:<request_put:<key:\"/registry/services/endpoints/openebs-system/openebs.io-local\" value_size:542 >> failure:<request_range:<key:\"/registry/services/endpoints/openebs-system/openebs.io-local\" > >>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:39.969917Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"5.089µs","request":"header:<ID:514416733623534475 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-system/spegel-leader-election\" mod_revision:482773 > success:<request_put:<key:\"/registry/leases/kube-system/spegel-leader-election\" value_size:450 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:39.988979Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"7.809µs","request":"header:<ID:514416733623534478 username:\"kube-apiserver\" auth_revision:1 > lease_grant:<ttl:15-second id:0723934b3225c38d>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.039835Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"8.076µs","request":"header:<ID:5087822155244138461 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/longhorn.io/engines/storage/pvc-ea1044b5-360f-4ad6-a190-a6c1a4190ab6-e-0\" mod_revision:474638 > success:<request_put:<key:\"/registry/longhorn.io/engines/storage/pvc-ea1044b5-360f-4ad6-a190-a6c1a4190ab6-e-0\" value_size:8401 >> failure:<request_range:<key:\"/registry/longhorn.io/engines/storage/pvc-ea1044b5-360f-4ad6-a190-a6c1a4190ab6-e-0\" > >>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.207388Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"9.406µs","request":"header:<ID:5087822155244138465 > lease_revoke:<id:0723934b321bc809>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.207492Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"3.201µs","request":"header:<ID:5087822155244138466 > lease_revoke:<id:02bc934b32ef3ecb>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.207531Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"02bc934b32ef3ecb","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.207598Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"0723934b321bc809","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.208243Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"6.962µs","request":"header:<ID:5087822155244138468 > lease_revoke:<id:0723934b321e790a>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.208313Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"2.534µs","request":"header:<ID:5087822155244138467 > lease_revoke:<id:02bc934b32ef3db5>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.208366Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"20.283µs","request":"header:<ID:5087822155244138469 > lease_revoke:<id:0723934b321c69c9>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.208351Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"0723934b321e790a","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.208410Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"2.463µs","request":"header:<ID:5087822155244138470 > lease_revoke:<id:02bc934b32ef09ab>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.208424Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"02bc934b32ef3db5","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.208448Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"0723934b321c69c9","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.208448Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"02bc934b32ef09ab","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.218575Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"8.363µs","request":"header:<ID:514416733623534481 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-system/kube-scheduler\" mod_revision:482780 > success:<request_put:<key:\"/registry/leases/kube-system/kube-scheduler\" value_size:412 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.246152Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"7.879µs","request":"header:<ID:9420566471738703566 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-system/apiserver-4n6dbydlfglfbkoitpmtm5hhra\" mod_revision:482697 > success:<request_put:<key:\"/registry/leases/kube-system/apiserver-4n6dbydlfglfbkoitpmtm5hhra\" value_size:600 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.254819Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"8.528µs","request":"header:<ID:9420566471738703567 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-system/apiserver-4n6dbydlfglfbkoitpmtm5hhra\" mod_revision:482697 > success:<request_put:<key:\"/registry/leases/kube-system/apiserver-4n6dbydlfglfbkoitpmtm5hhra\" value_size:600 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.263370Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"7.398µs","request":"header:<ID:9420566471738703568 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-system/apiserver-4n6dbydlfglfbkoitpmtm5hhra\" mod_revision:482697 > success:<request_put:<key:\"/registry/leases/kube-system/apiserver-4n6dbydlfglfbkoitpmtm5hhra\" value_size:600 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.275016Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"4.819µs","request":"header:<ID:9420566471738703569 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-system/apiserver-4n6dbydlfglfbkoitpmtm5hhra\" mod_revision:482697 > success:<request_put:<key:\"/registry/leases/kube-system/apiserver-4n6dbydlfglfbkoitpmtm5hhra\" value_size:600 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.283479Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"3.384µs","request":"header:<ID:9420566471738703570 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-system/apiserver-4n6dbydlfglfbkoitpmtm5hhra\" mod_revision:482697 > success:<request_put:<key:\"/registry/leases/kube-system/apiserver-4n6dbydlfglfbkoitpmtm5hhra\" value_size:601 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.296399Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"5.681µs","request":"header:<ID:9420566471738703572 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-system/apiserver-4n6dbydlfglfbkoitpmtm5hhra\" mod_revision:482697 > success:<request_put:<key:\"/registry/leases/kube-system/apiserver-4n6dbydlfglfbkoitpmtm5hhra\" value_size:601 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.304905Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"7.013µs","request":"header:<ID:9420566471738703573 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-system/apiserver-4n6dbydlfglfbkoitpmtm5hhra\" mod_revision:482697 > success:<request_put:<key:\"/registry/leases/kube-system/apiserver-4n6dbydlfglfbkoitpmtm5hhra\" value_size:601 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.314273Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"5.867µs","request":"header:<ID:9420566471738703574 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-system/apiserver-4n6dbydlfglfbkoitpmtm5hhra\" mod_revision:482697 > success:<request_put:<key:\"/registry/leases/kube-system/apiserver-4n6dbydlfglfbkoitpmtm5hhra\" value_size:601 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.322761Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"6.973µs","request":"header:<ID:9420566471738703575 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-system/apiserver-4n6dbydlfglfbkoitpmtm5hhra\" mod_revision:482697 > success:<request_put:<key:\"/registry/leases/kube-system/apiserver-4n6dbydlfglfbkoitpmtm5hhra\" value_size:601 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.332116Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"6.185µs","request":"header:<ID:9420566471738703576 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-system/apiserver-4n6dbydlfglfbkoitpmtm5hhra\" mod_revision:482697 > success:<request_put:<key:\"/registry/leases/kube-system/apiserver-4n6dbydlfglfbkoitpmtm5hhra\" value_size:601 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.472883Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"8.726µs","request":"header:<ID:9420566471738703578 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/database/db9c8771.cnpg.io\" mod_revision:482784 > success:<request_put:<key:\"/registry/leases/database/db9c8771.cnpg.io\" value_size:428 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.478827Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"8.75µs","request":"header:<ID:5087822155244138473 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/longhorn.io/engines/storage/pvc-0f2da8c6-25ff-4fdb-83f1-de2df988b575-e-0\" mod_revision:456502 > success:<request_put:<key:\"/registry/longhorn.io/engines/storage/pvc-0f2da8c6-25ff-4fdb-83f1-de2df988b575-e-0\" value_size:7837 >> failure:<request_range:<key:\"/registry/longhorn.io/engines/storage/pvc-0f2da8c6-25ff-4fdb-83f1-de2df988b575-e-0\" > >>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.498891Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"4.983µs","request":"header:<ID:514416733623534485 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/pods/flux-system/helm-controller-64cd59b777-l22bw\" mod_revision:2901 > success:<request_put:<key:\"/registry/pods/flux-system/helm-controller-64cd59b777-l22bw\" value_size:5391 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.520308Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"4.01µs","request":"header:<ID:514416733623534487 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/pods/openebs-system/openebs-localpv-provisioner-56d8447b9-kbxm7\" mod_revision:4743 > success:<request_put:<key:\"/registry/pods/openebs-system/openebs-localpv-provisioner-56d8447b9-kbxm7\" value_size:4789 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.544561Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"7.851µs","request":"header:<ID:514416733623534491 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-system/kube-controller-manager\" mod_revision:482767 > success:<request_put:<key:\"/registry/leases/kube-system/kube-controller-manager\" value_size:432 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.549218Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"8.825µs","request":"header:<ID:514416733623534492 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/pods/flux-system/kustomize-controller-54f65db458-fg22s\" mod_revision:3124 > success:<request_put:<key:\"/registry/pods/flux-system/kustomize-controller-54f65db458-fg22s\" value_size:5343 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.582441Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"7.94µs","request":"header:<ID:5087822155244138474 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/longhorn.io/engines/storage/pvc-9750c179-d6c3-40d0-b89f-6dc1e6fd45c4-e-0\" mod_revision:474686 > success:<request_put:<key:\"/registry/longhorn.io/engines/storage/pvc-9750c179-d6c3-40d0-b89f-6dc1e6fd45c4-e-0\" value_size:8387 >> failure:<request_range:<key:\"/registry/longhorn.io/engines/storage/pvc-9750c179-d6c3-40d0-b89f-6dc1e6fd45c4-e-0\" > >>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.708052Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"8.388µs","request":"header:<ID:5087822155244138475 > lease_revoke:<id:02bc934b32ef1d03>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.708127Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"02bc934b32ef1d03","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.709060Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"6.758µs","request":"header:<ID:5087822155244138476 > lease_revoke:<id:0723934b321deb0c>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.709134Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"2.642µs","request":"header:<ID:5087822155244138478 > lease_revoke:<id:02bc934b32ef3bcf>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.709165Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"2.181µs","request":"header:<ID:5087822155244138477 > lease_revoke:<id:469b934b33aa80b1>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.709191Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"1.744µs","request":"header:<ID:5087822155244138479 > lease_revoke:<id:02bc934b32ef1fa0>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.709214Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"1.752µs","request":"header:<ID:5087822155244138480 > lease_revoke:<id:0723934b321d2570>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.709222Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"0723934b321deb0c","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.709286Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"02bc934b32ef3bcf","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.709313Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"02bc934b32ef1fa0","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.709293Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"0723934b321d2570","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.709333Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"469b934b33aa80b1","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.852623Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"9.431µs","request":"header:<ID:514416733623534495 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-system/kube-scheduler\" mod_revision:482780 > success:<request_put:<key:\"/registry/leases/kube-system/kube-scheduler\" value_size:410 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:40.993053Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"3.448µs","request":"header:<ID:9420566471738703580 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/storage/external-resizer-driver-longhorn-io\" mod_revision:482772 > success:<request_put:<key:\"/registry/leases/storage/external-resizer-driver-longhorn-io\" value_size:410 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.028156Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"3.859µs","request":"header:<ID:9420566471738703582 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/storage/driver-longhorn-io\" mod_revision:482778 > success:<request_put:<key:\"/registry/leases/storage/driver-longhorn-io\" value_size:404 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.028822Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"2.164µs","request":"header:<ID:5087822155244138481 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/longhorn.io/engines/storage/pvc-a37c368b-f784-476d-a0e4-be4db6645b7e-e-0\" mod_revision:361515 > success:<request_put:<key:\"/registry/longhorn.io/engines/storage/pvc-a37c368b-f784-476d-a0e4-be4db6645b7e-e-0\" value_size:7850 >> failure:<request_range:<key:\"/registry/longhorn.io/engines/storage/pvc-a37c368b-f784-476d-a0e4-be4db6645b7e-e-0\" > >>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.044577Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"3.882µs","request":"header:<ID:5087822155244138482 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/longhorn.io/engines/storage/pvc-0beb7017-cc7c-42c5-9509-4c1bb1fa4cc1-e-0\" mod_revision:475880 > success:<request_put:<key:\"/registry/longhorn.io/engines/storage/pvc-0beb7017-cc7c-42c5-9509-4c1bb1fa4cc1-e-0\" value_size:8399 >> failure:<request_range:<key:\"/registry/longhorn.io/engines/storage/pvc-0beb7017-cc7c-42c5-9509-4c1bb1fa4cc1-e-0\" > >>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.044630Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"1.259µs","request":"header:<ID:5087822155244138483 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/longhorn.io/engines/storage/pvc-05d7cd83-dcd5-4385-913e-0ec1d3d8c06e-e-0\" mod_revision:481730 > success:<request_put:<key:\"/registry/longhorn.io/engines/storage/pvc-05d7cd83-dcd5-4385-913e-0ec1d3d8c06e-e-0\" value_size:8400 >> failure:<request_range:<key:\"/registry/longhorn.io/engines/storage/pvc-05d7cd83-dcd5-4385-913e-0ec1d3d8c06e-e-0\" > >>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.045521Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"2.235µs","request":"header:<ID:5087822155244138484 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/longhorn.io/engines/storage/pvc-85502885-8e7a-4d4a-a58e-0dd33bd54cc5-e-0\" mod_revision:482731 > success:<request_put:<key:\"/registry/longhorn.io/engines/storage/pvc-85502885-8e7a-4d4a-a58e-0dd33bd54cc5-e-0\" value_size:8401 >> failure:<request_range:<key:\"/registry/longhorn.io/engines/storage/pvc-85502885-8e7a-4d4a-a58e-0dd33bd54cc5-e-0\" > >>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.056059Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"2.953µs","request":"header:<ID:9420566471738703585 username:\"kube-apiserver\" auth_revision:1 > lease_grant:<ttl:15-second id:02bc934b32f0bee0>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.139893Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"8.315µs","request":"header:<ID:514416733623534498 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-system/kubelet-csr-approver\" mod_revision:482781 > success:<request_put:<key:\"/registry/leases/kube-system/kubelet-csr-approver\" value_size:451 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.187110Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"9.253µs","request":"header:<ID:514416733623534500 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/flux-system/source-controller-leader-election\" mod_revision:482775 > success:<request_put:<key:\"/registry/leases/flux-system/source-controller-leader-election\" value_size:458 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.195634Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"7.015µs","request":"header:<ID:514416733623534502 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/flux-system/kustomize-controller-leader-election\" mod_revision:482740 > success:<request_put:<key:\"/registry/leases/flux-system/kustomize-controller-leader-election\" value_size:468 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.208502Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"7.898µs","request":"header:<ID:5087822155244138486 > lease_revoke:<id:0723934b321bfe07>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.208593Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"0723934b321bfe07","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.209357Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"7.243µs","request":"header:<ID:5087822155244138487 > lease_revoke:<id:0723934b321eb240>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.209427Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"2.749µs","request":"header:<ID:5087822155244138488 > lease_revoke:<id:02bc934b32ef52fd>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.209463Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"2.023µs","request":"header:<ID:5087822155244138489 > lease_revoke:<id:0723934b321ec88f>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.209488Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"1.756µs","request":"header:<ID:5087822155244138490 > lease_revoke:<id:0723934b321dfb59>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.209513Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"1.696µs","request":"header:<ID:5087822155244138492 > lease_revoke:<id:02bc934b32ef453e>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.209517Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"0723934b321eb240","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.209544Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"1.952µs","request":"header:<ID:5087822155244138491 > lease_revoke:<id:02bc934b32ef2ba3>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.209565Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"02bc934b32ef453e","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.209592Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"0723934b321dfb59","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.209588Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"0723934b321ec88f","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.209628Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"02bc934b32ef2ba3","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.209631Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"02bc934b32ef52fd","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.477676Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"8.433µs","request":"header:<ID:5087822155244138493 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/longhorn.io/engines/storage/pvc-d34b0c22-6951-4be9-b742-37bff4cd2efc-e-0\" mod_revision:482626 > success:<request_put:<key:\"/registry/longhorn.io/engines/storage/pvc-d34b0c22-6951-4be9-b742-37bff4cd2efc-e-0\" value_size:8390 >> failure:<request_range:<key:\"/registry/longhorn.io/engines/storage/pvc-d34b0c22-6951-4be9-b742-37bff4cd2efc-e-0\" > >>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.494209Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"6.532µs","request":"header:<ID:514416733623534506 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/flux-system/image-reflector-controller-leader-election\" mod_revision:482755 > success:<request_put:<key:\"/registry/leases/flux-system/image-reflector-controller-leader-election\" value_size:488 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.710090Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"8.661µs","request":"header:<ID:5087822155244138494 > lease_revoke:<id:469b934b33aa516c>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.710191Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"469b934b33aa516c","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.710658Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"6.44µs","request":"header:<ID:5087822155244138495 > lease_revoke:<id:02bc934b32ef4e33>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.710740Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"3.006µs","request":"header:<ID:5087822155244138497 > lease_revoke:<id:02bc934b32ef5148>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.710799Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"02bc934b32ef4e33","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.710820Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"2.853µs","request":"header:<ID:5087822155244138496 > lease_revoke:<id:0723934b321c30b9>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.710847Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"02bc934b32ef5148","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.710896Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"7.145µs","request":"header:<ID:5087822155244138498 > lease_revoke:<id:469b934b33aab5f6>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.710938Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"2.95µs","request":"header:<ID:5087822155244138499 > lease_revoke:<id:0723934b321d9310>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.710978Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"2.44µs","request":"header:<ID:5087822155244138500 > lease_revoke:<id:0723934b321d6665>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.710963Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"0723934b321c30b9","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.710978Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"469b934b33aab5f6","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.711009Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"0723934b321d6665","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.711018Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"0723934b321d9310","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.796843Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"8.747µs","request":"header:<ID:514416733623534509 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-system/cilium-operator-resource-lock\" mod_revision:482771 > success:<request_put:<key:\"/registry/leases/kube-system/cilium-operator-resource-lock\" value_size:408 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.837363Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"9.5µs","request":"header:<ID:514416733623534511 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/storage/external-snapshotter-leader-driver-longhorn-io\" mod_revision:482745 > success:<request_put:<key:\"/registry/leases/storage/external-snapshotter-leader-driver-longhorn-io\" value_size:428 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.897227Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"9.009µs","request":"header:<ID:514416733623534513 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-system/kube-controller-manager\" mod_revision:482767 > success:<request_put:<key:\"/registry/leases/kube-system/kube-controller-manager\" value_size:428 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.912839Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"9.143µs","request":"header:<ID:514416733623534515 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/pods/storage/csi-attacher-787d99954-j6tt7\" mod_revision:12987 > success:<request_put:<key:\"/registry/pods/storage/csi-attacher-787d99954-j6tt7\" value_size:3913 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.934714Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"7.311µs","request":"header:<ID:514416733623534517 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/pods/cert-manager/cert-manager-cainjector-c879b86d6-w7tg9\" mod_revision:130011 > success:<request_put:<key:\"/registry/pods/cert-manager/cert-manager-cainjector-c879b86d6-w7tg9\" value_size:4165 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.957608Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"7.483µs","request":"header:<ID:514416733623534519 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/pods/security/authentik-worker-6c9b654886-bj8vj\" mod_revision:478106 > success:<request_put:<key:\"/registry/pods/security/authentik-worker-6c9b654886-bj8vj\" value_size:4627 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:41.982359Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"8.64µs","request":"header:<ID:514416733623534521 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/pods/flux-system/notification-controller-556b8867f8-79nbc\" mod_revision:2849 > success:<request_put:<key:\"/registry/pods/flux-system/notification-controller-556b8867f8-79nbc\" value_size:5424 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:42.000088Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"9.055µs","request":"header:<ID:514416733623534523 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-node-lease/nectarine\" mod_revision:482715 > success:<request_put:<key:\"/registry/leases/kube-node-lease/nectarine\" value_size:475 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:42.009692Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"9.335µs","request":"header:<ID:514416733623534524 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/pods/storage/volsync-8466895bb6-mmlrf\" mod_revision:12652 > success:<request_put:<key:\"/registry/pods/storage/volsync-8466895bb6-mmlrf\" value_size:6091 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:42.011609Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"7.855µs","request":"header:<ID:514416733623534525 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-node-lease/nectarine\" mod_revision:482715 > success:<request_put:<key:\"/registry/leases/kube-node-lease/nectarine\" value_size:473 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:42.019020Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"8.332µs","request":"header:<ID:5087822155244138501 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/longhorn.io/engines/storage/pvc-38a643b6-abcd-457b-94e0-b75d357d9864-e-0\" mod_revision:480448 > success:<request_put:<key:\"/registry/longhorn.io/engines/storage/pvc-38a643b6-abcd-457b-94e0-b75d357d9864-e-0\" value_size:8399 >> failure:<request_range:<key:\"/registry/longhorn.io/engines/storage/pvc-38a643b6-abcd-457b-94e0-b75d357d9864-e-0\" > >>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:42.020536Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"7.689µs","request":"header:<ID:514416733623534526 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-node-lease/nectarine\" mod_revision:482715 > success:<request_put:<key:\"/registry/leases/kube-node-lease/nectarine\" value_size:474 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:42.029988Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"11.455µs","request":"header:<ID:514416733623534528 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-node-lease/nectarine\" mod_revision:482715 > success:<request_put:<key:\"/registry/leases/kube-node-lease/nectarine\" value_size:474 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:42.038232Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"7.985µs","request":"header:<ID:514416733623534529 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/pods/database/cloudnative-pg-586475f9d5-drx9k\" mod_revision:29569 > success:<request_put:<key:\"/registry/pods/database/cloudnative-pg-586475f9d5-drx9k\" value_size:5602 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:42.040470Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"7.149µs","request":"header:<ID:514416733623534530 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-node-lease/nectarine\" mod_revision:482715 > success:<request_put:<key:\"/registry/leases/kube-node-lease/nectarine\" value_size:474 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:42.043611Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"14.402µs","request":"header:<ID:5087822155244138502 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/longhorn.io/engines/storage/pvc-7c5c589f-0e6d-4e70-81ce-12c0d4c349c4-e-0\" mod_revision:482081 > success:<request_put:<key:\"/registry/longhorn.io/engines/storage/pvc-7c5c589f-0e6d-4e70-81ce-12c0d4c349c4-e-0\" value_size:7851 >> failure:<request_range:<key:\"/registry/longhorn.io/engines/storage/pvc-7c5c589f-0e6d-4e70-81ce-12c0d4c349c4-e-0\" > >>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:42.053946Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"7.008µs","request":"header:<ID:514416733623534533 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-node-lease/nectarine\" mod_revision:482715 > success:<request_put:<key:\"/registry/leases/kube-node-lease/nectarine\" value_size:474 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:42.059050Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"7.624µs","request":"header:<ID:514416733623534534 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/pods/cert-manager/cert-manager-659dbd9c7f-cmm4j\" mod_revision:130005 > success:<request_put:<key:\"/registry/pods/cert-manager/cert-manager-659dbd9c7f-cmm4j\" value_size:4795 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:42.064973Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"6.353µs","request":"header:<ID:514416733623534535 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-node-lease/nectarine\" mod_revision:482715 > success:<request_put:<key:\"/registry/leases/kube-node-lease/nectarine\" value_size:474 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:42.067134Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"5.292µs","request":"header:<ID:9420566471738703587 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/storage/driver-longhorn-io\" mod_revision:482778 > success:<request_put:<key:\"/registry/leases/storage/driver-longhorn-io\" value_size:404 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:42.073474Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"6.807µs","request":"header:<ID:514416733623534538 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-node-lease/nectarine\" mod_revision:482715 > success:<request_put:<key:\"/registry/leases/kube-node-lease/nectarine\" value_size:474 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:42.076837Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"5.029µs","request":"header:<ID:514416733623534539 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-system/kube-scheduler\" mod_revision:482780 > success:<request_put:<key:\"/registry/leases/kube-system/kube-scheduler\" value_size:408 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:42.083302Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"5.122µs","request":"header:<ID:514416733623534540 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/pods/flux-system/image-reflector-controller-7cdc88c5c7-vjdng\" mod_revision:2937 > success:<request_put:<key:\"/registry/pods/flux-system/image-reflector-controller-7cdc88c5c7-vjdng\" value_size:5448 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:42.083940Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"5.07µs","request":"header:<ID:514416733623534541 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-node-lease/nectarine\" mod_revision:482715 > success:<request_put:<key:\"/registry/leases/kube-node-lease/nectarine\" value_size:474 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:42.091384Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"6µs","request":"header:<ID:514416733623534542 username:\"kube-apiserver\" auth_revision:1 > txn:<compare:<target:MOD key:\"/registry/leases/kube-node-lease/nectarine\" mod_revision:482715 > success:<request_put:<key:\"/registry/leases/kube-node-lease/nectarine\" value_size:474 >> failure:<>>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:42.210758Z","caller":"etcdserver/util.go:123","msg":"failed to apply request","took":"8.282µs","request":"header:<ID:5087822155244138506 > lease_revoke:<id:0723934b321bb66b>","response":"","error":"etcdserver: corrupt cluster"}
{"level":"warn","ts":"2024-11-21T10:50:42.210836Z","caller":"etcdserver/server.go:1198","msg":"failed to revoke lease","lease-id":"0723934b321bb66b","error":"etcdserver: corrupt cluster"}
@serathius
Copy link
Member

Logs you provided are just from request failures due to past corruption alarm. Can you provide the logs from time the corruption happen?

@rkoosaar
Copy link
Author

This is what the log bundle had from the time I generated it. Not sure what happened then or why it starts from that time. Maybe it got overwritten since its constantly saying cluster corrupted. do you think any other log file from the bundle might help? or me uploading the whole bundle?

@ahrtr
Copy link
Member

ahrtr commented Nov 22, 2024

This is what the log bundle had from the time I generated it.

Could you provide the complete log of all etcd instances?

or me uploading the whole bundle?

Yes, please. Is it possible to upload all the db files (under the ${data_dir}/member/snap/db) If it doesn't have any sensitive data?

@rkoosaar
Copy link
Author

Sorry no I only have log bundles from the nodes (nodes themselves have been reset). I'll see if there's any sensitive data in the log bundles and take that out and then provide those.
I have a fresh cluster running again, same 1.8.3 with same etcd version.

@rkoosaar
Copy link
Author

here are the support log bundles i got from nodes running - talosctl -n node_ip support -
support-31-33.zip

@ahrtr
Copy link
Member

ahrtr commented Nov 23, 2024

here are the support log bundles i got from nodes running - talosctl -n node_ip support - support-31-33.zip

Unfortunately, the log in this bundle isn't complete either. The first log entry is the same (etcdserver: corrupt cluster) as you already posted in this ticket.

I also do not see the data files as mentioned in #18933 (comment). If you can provide that files, I can analyze the db file directly.

@rkoosaar
Copy link
Author

I'm not sure how I could get the etcd db of talos nodes as there is no direct shell access. just commands via talosctl

@ahrtr
Copy link
Member

ahrtr commented Nov 23, 2024

I'm not sure how I could get the etcd db of talos nodes as there is no direct shell access. just commands via talosctl

You might want to raise the question in talos community?

@samfili
Copy link

samfili commented Dec 19, 2024 •
edited
Loading

Hi folks, I have a really odd issue that I'm troubleshooting. I have a 3 node Talos (1.8.3) cluster at home where etcd (3.5.16) keeps getting corrupted after a while. Initially I thought it could be a disk related issue. So I bought brand new disks and swapped them around. I installed a new cluster last night (around 8pm) and when I woke up this morning (8am) cluster was not working and etcd was reporting cluster corrupted. Looking at the logs, it seems something happened around 6am, but I'm unable to work out what the cause is. So far I have redeployed the cluster in the past week 4 times and every time etcd has ended up corrupted. Any help/guidance to troubleshoot this would be much appreciated.

I have the same problem, in addition, I have it happening on the VMware software platform: cluster - 3 control plane, 3 working nodes, everything works fine after a clean installation for about 4-6 hours etcd then goes into the error "etcdserver: no leader", this is the last thing I could diagnose, build an image of Talos in Factory-installed with extensions :

customization:
systemExtensions:
officialExtensions:
- siderolabs/drbd
- siderolabs/iscsi-tools
- siderolabs/util-linux-tools
- siderolabs/vmtoolsd-guest-agent
- siderolabs/zfs

Interestingly, if you completely turn off the virtual machines with Talos, then wait a few seconds and start them back one by one starting from the first control plane, everything starts working again, then after a few hours everything repeats again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type/bug
Milestone
No milestone
Development

No branches or pull requests
4 participants
@samfili @serathius @ahrtr @rkoosaar