Autoscaler failed to verify certificate: x509

[Coennie]

Description

When the autoscaler is deployed the logs show:
Failed to create Hetzner manager: failed to get ssh key error: Get "https://api.hetzner.cloud/v1/ssh_keys?name=id_ed25519_hetzner": tls: failed to verify certificate: x509: failed to load system roots and no roots provided; open /etc/ssl/certs: permission denied

Even after the #1130 fix and following the steps:

1. hcloud image list
2. hcloud image delete
3. rm hcloud-microos-snapshots.pkr.hcl
4. createkh, answer yes to create snapshots (this will create the snapshots)
5. terraform init -upgrade
6. terrafom apply

I get the same result

Kube.tf file

locals {
  hcloud_token = "***"
}

module "kube-hetzner" {
  providers = {
    hcloud = hcloud
  }
  hcloud_token = var.hcloud_token != "" ? var.hcloud_token : local.hcloud_token

  source = "kube-hetzner/kube-hetzner/hcloud"
  
  ssh_public_key = file("~/.ssh/id_ed25519_hetzner.pub")
  ssh_private_key = file("~/.ssh/id_ed25519_hetzner")
  hcloud_ssh_key_id = "id_ed25519_hetzner"

  network_region = "eu-central" # change to `us-east` if location is ash

  control_plane_nodepools = [
    {
      name        = "control-plane-fsn1",
      server_type = "cax11",
      location    = "fsn1",
      labels      = [],
      taints      = [],
      count       = 0
    },
    {
      name        = "control-plane-nbg1",
      server_type = "cax11",
      location    = "nbg1",
      labels      = [],
      taints      = [],
      count       = 0
    },
    {
      name        = "control-plane-hel1",
      server_type = "cax11",
      location    = "hel1",
      labels      = [],
      taints      = [],
      count       = 0
    },
    {
      name        = "control-plane-fsn1-1",
      server_type = "cax11",
      location    = "fsn1",
      labels      = [],
      taints      = [],
      count       = 1
    },
    {
      name        = "control-plane-nbg1-1",
      server_type = "cax11",
      location    = "nbg1",
      labels      = [],
      taints      = [],
      count       = 1
    },
    {
      name        = "control-plane-hel1-1",
      server_type = "cax11",
      location    = "hel1",
      labels      = [],
      taints      = [],
      count       = 1
    }
  ]

  agent_nodepools = [
    {
      name        = "agent-large",
      server_type = "cpx21",
      location    = "fsn1",
      labels      = [],
      taints      = [],
      count       = 0
    },
    {
      name        = "agent-large-1",
      server_type = "cpx21",
      location    = "fsn1",
      labels      = [],
      taints      = [],
      count       = 3
    }
  ]

  # * LB location and type, the latter will depend on how much load you want it to handle, see https://www.hetzner.com/cloud/load-balancer
  load_balancer_type     = "lb11"
  load_balancer_location = "fsn1"

  base_domain = "kubernetes.mydomain.com"

  autoscaler_nodepools = [
    {
      name        = "autoscaled-small"
      server_type = "cpx21"
      location    = "fsn1"
      min_nodes   = 0
      max_nodes   = 5
      labels      = {
        "node.kubernetes.io/role": "peak-workloads"
      }
      taints      = [{
         key: "node.kubernetes.io/role"
         value: "peak-workloads"
         effect: "NoExecute"
      }]
    }
  ]

  cluster_autoscaler_log_level = 4
  cluster_autoscaler_log_to_stderr = true
  cluster_autoscaler_stderr_threshold = "INFO"

  initial_k3s_channel = "stable"
  cluster_name = "mycluster"

  dns_servers = [
    "1.1.1.1",
    "8.8.8.8",
    "2606:4700:4700::1111",
  ]
}

provider "hcloud" {
  token = var.hcloud_token != "" ? var.hcloud_token : local.hcloud_token
}

terraform {
  required_version = ">= 1.5.0"
  required_providers {
    hcloud = {
      source  = "hetznercloud/hcloud"
      version = ">= 1.43.0"
    }
  }
}

output "kubeconfig" {
  value     = module.kube-hetzner.kubeconfig
  sensitive = true
}

variable "hcloud_token" {
  sensitive = true
  default   = ""
}

Screenshots

No response

Platform

linux

[mysticaltech]

@Coennie Now I understand. As explained in the kube.tf.example, the first control-plane nodepool should not have a count of zero. It's important to have it to at least count 1.

[Coennie]

Ok, I did that because of a guide to update the controllers to the latest version, where I needed to double the controllers and scale back to 0 again after the upgrade. Must have misread that then. When I change the first back to 1 again, it wants to replace all current controllers as well:
module.kube-hetzner.null_resource.control_plane_config["3-0-control-plane-fsn1-1"] must be replaced
I assume this will destroy of disrupt my current cluster right?

[mysticaltech]

That's the null ressource that does the config module.kube-hetzner.null_resource.control_plane_config, not the node themselves!

[Coennie]

Trying to restore everything in de kube.tf file did kill my cluster. Deployed a fresh one, now all (including the autoscaler) does work as expected. Thanks for your assistance on this.