Cluster-autoscaler doesn't work, fails on get request to get key because of TLS error

[virtuman]

Description

error looks like this:

❯ kubectl logs -f cluster-autoscaler-587474c8dd-bzlqq
I1214 13:51:10.554558 1 main.go:600] Cluster Autoscaler 1.28.0
I1214 13:51:10.658982 1 leaderelection.go:250] attempting to acquire leader lease kube-system/cluster-autoscaler...
I1214 13:51:10.667091 1 leaderelection.go:260] successfully acquired lease kube-system/cluster-autoscaler
I1214 13:51:10.667312 1 event_sink_logging_wrapper.go:48] Event(v1.ObjectReference{Kind:"Lease", Namespace:"kube-system", Name:"cluster-autoscaler", UID:"bcf726c9-709e-4019-b38c-624ed3af52d7", APIVersion:"coordination.k8s.io/v1", ResourceVersion:"265610483", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' cluster-autoscaler-587474c8dd-bzlqq became leader
I1214 13:51:10.757784 1 cloud_provider_builder.go:29] Building hetzner cloud provider.
F1214 13:51:10.773745 1 hetzner_cloud_provider.go:186] Failed to create Hetzner manager: failed to get ssh key error: Get "https://api.hetzner.cloud/v1/ssh_keys?name=alexduman": tls: failed to verify certificate: x509: failed to load system roots and no roots provided; open /etc/ssl/certs: permission denied

upgraded cluster, erased deployment, all to no avail. any suggestions?

Kube.tf file

all cluster_autoscaler options are set to default, only 1 pool added similar to example

Screenshots

No response

Platform

Linux

[mysticaltech]

@virtuman Please make sure your images are new enough, if they are too old they won't work. Follow the upgrade methodology here #794.

[Coennie]

I've got the exact same issue.

"Failed to create Hetzner manager: failed to get ssh key error: Get "https://api.hetzner.cloud/v1/ssh_keys?name=my-key": tls: failed to verify certificate: x509: failed to load system roots and no roots provided; open /etc/ssl/certs: permission denied"

Tried the recreation of the images, but that makes no difference. It's complaining about the verification of the x509 certificate

[Coennie]

@mysticaltech thanks for your reply,but that does not make any difference.

in mykube.tf file I have:
ssh_public_key = file("~/.ssh/id_ed25519.pub")
ssh_private_key = file("~/.ssh/id_ed25519")
hcloud_ssh_key_id = "ssh-ed25519-coen"

and these are commented
cluster_autoscaler_image
cluster_autoscaler_version

But after a:
terraform init -upgrade
terraform apply

I do see its deploying a new autoscaler, but it still gives back the error:
Failed to create Hetzner manager: failed to get ssh key error: Get "https://api.hetzner.cloud/v1/ssh_keys?name=ssh-ed25519-coen": tls: failed to verify certificate: x509: failed to load system roots and no roots provided; open /etc/ssl/certs: permission denied

[brain-dev-null]

The Issue is with SELinux blocking reads of /etc/ssl/certs on the host.
I only managed to confirm this by disabling SELinux but did not investigate further as I am not familiar with SELinux and how to change its configuration accordingly.

[mysticaltech]

Ok folks, issue identified in #1271, working on a fix.

[mysticaltech]

Fixed in v2.13.3

[Coennie]

Hey @mysticaltech,
thanks for your quick update! I've followed these steps:

1. hcloud image list
2. hcloud image delete
3. rm hcloud-microos-snapshots.pkr.hcl
4. createkh, answer yes to create snapshots (this will create the snapshots)
5. terraform init -upgrade
6. terrafom apply

After I do see a new autoscaler being deployed, but it still shows the same error... Did I miss a step? Or should I follow different steps?

[mysticaltech]

@Coennie Please create an issue with your kube.tf so that I can reproduce, and descrbe your issue in great details there too please. I need a complete picture otherwise I can't help.