Best/easiest way to duplicate a complete cluster to test changes

[Taronyuu]

I've been using this repository to manage my own k3s cluster on Hetzner for over a year now and it has been great, recently I've noticed that my control planes haven't been upgrading because of #1163. Now, I've updated terraform and a few changes have been applied most likely solving my issue. The problem is that 5 resources will be destroyed (I've added the log below just in case someone wants to see which ones). Most likely this isn't a problem, but I am scared to run it on my production cluster. Preferably I would like to run this change first on an exact duplicate to make sure nothing breaks and everything works.

Now the question: what would be the absolute best and easiest way to create an exact duplicate of the current cluster?

My own suggestion would be to create a duplicate of my kube.tf, fix the version of the terraform module to the old one and use a new Hetzner project and token to deploy it. Then upgrade and make sure everything still works as expected. Possibly lower some data planes and deploy a test project, but that should be it. But I want to make sure that I am not missing something and possibly messing up my production cluster.

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
-/+ destroy and then create replacement

Terraform will perform the following actions:

  # module.kube-hetzner.local_file.kustomization_backup[0] must be replaced
-/+ resource "local_file" "kustomization_backup" {
      ~ content              = <<-EOT # forces replacement
            "apiVersion": "kustomize.config.k8s.io/v1beta1"
            "kind": "Kustomization"
            "patches":
            - "patch": |
                apiVersion: apps/v1
                kind: Deployment
                metadata:
                  name: system-upgrade-controller
                  namespace: system-upgrade
                spec:
                  template:
                    spec:
                      containers:
                        - name: system-upgrade-controller
                          volumeMounts:
                            - name: ca-certificates
                              mountPath: /var/lib/ca-certificates
                      volumes:
                        - name: ca-certificates
                          hostPath:
                            path: /var/lib/ca-certificates
                            type: Directory
              "target":
                "group": "apps"
                "kind": "Deployment"
                "name": "system-upgrade-controller"
                "namespace": "system-upgrade"
                "version": "v1"
          + - "patch": |
          +     - op: replace
          +       path: /spec/template/spec/containers/0/image
          +       value: rancher/system-upgrade-controller:v0.13.4
          +   "target":
          +     "group": "apps"
          +     "kind": "Deployment"
          +     "name": "system-upgrade-controller"
          +     "namespace": "system-upgrade"
          +     "version": "v1"
            - "path": "kured.yaml"
            - "path": "ccm.yaml"
            "resources":
            - "https://github.com/hetznercloud/hcloud-cloud-controller-manager/releases/download/v1.20.0/ccm-networks.yaml"
            - "https://github.com/kubereboot/kured/releases/download/1.15.1/kured-1.15.1-dockerhub.yaml"
          - - "https://raw.githubusercontent.com/rancher/system-upgrade-controller/master/manifests/system-upgrade-controller.yaml"
          + - "https://raw.githubusercontent.com/rancher/system-upgrade-controller/9e7e45c1bdd528093da36be1f1f32472469005e6/manifests/system-upgrade-controller.yaml"
            - "hcloud-csi.yml"
            - "nginx_ingress.yaml"
            - "longhorn.yaml"
            - "cert_manager.yaml"
        EOT
      ~ content_base64sha256 = "jlGYoTn9VwkGJP2hRr8FPaHfbAdmI0WJwfvM9QSCbNg=" -> (known after apply)
      ~ content_base64sha512 = "i/pN0mkPQo0GeI4OIAyddRrrHWJ4XmHgbEa5tiUncLEi2RC2b9zaQS5vj4T1dT2lNkCRG99R2PaClPU3uQn0YA==" -> (known after apply)
      ~ content_md5          = "096d7416bc7a809d0ab7758bdfd751b7" -> (known after apply)
      ~ content_sha1         = "73d0da9142cd1a0ca732f4cff9e1589d65dc5e4e" -> (known after apply)
      ~ content_sha256       = "8e5198a139fd57090624fda146bf053da1df6c0766234589c1fbccf504826cd8" -> (known after apply)
      ~ content_sha512       = "8bfa4dd2690f428d06788e0e200c9d751aeb1d62785e61e06c46b9b6252770b122d910b66fdcda412e6f8f84f5753da53640911bdf51d8f68294f537b909f460" -> (known after apply)
      ~ id                   = "73d0da9142cd1a0ca732f4cff9e1589d65dc5e4e" -> (known after apply)
        # (3 unchanged attributes hidden)
    }

  # module.kube-hetzner.null_resource.control_plane_config["0-0-cp1"] must be replaced
-/+ resource "null_resource" "control_plane_config" {
      ~ id       = "7227763833626197108" -> (known after apply)
      ~ triggers = { # forces replacement
          ~ "config"           = (sensitive value)
            # (1 unchanged element hidden)
        }
    }

  # module.kube-hetzner.null_resource.control_plane_config["0-1-cp1"] must be replaced
-/+ resource "null_resource" "control_plane_config" {
      ~ id       = "1291983469705802030" -> (known after apply)
      ~ triggers = { # forces replacement
          ~ "config"           = (sensitive value)
            # (1 unchanged element hidden)
        }
    }

  # module.kube-hetzner.null_resource.control_plane_config["1-0-cp2"] must be replaced
-/+ resource "null_resource" "control_plane_config" {
      ~ id       = "7314723119774852608" -> (known after apply)
      ~ triggers = { # forces replacement
          ~ "config"           = (sensitive value)
            # (1 unchanged element hidden)
        }
    }

  # module.kube-hetzner.null_resource.kustomization must be replaced
-/+ resource "null_resource" "kustomization" {
      ~ id       = "8059785587357245696" -> (known after apply)
      ~ triggers = { # forces replacement
          ~ "helm_values_yaml" = (sensitive value)
          ~ "options"          = <<-EOT
              - lock-ttl=30m
                period=5m
                post-reboot-node-labels=kured=done
                pre-reboot-node-labels=kured=rebooting
                reboot-command=/usr/bin/systemctl reboot
              + reboot-sentinel=/sentinel/reboot-required
            EOT
          ~ "versions"         = <<-EOT
              - v1.28
              - 20231027
              + v1.29
              + v1.30.1
                N/A
                N/A
                N/A
                N/A
              - 1.14.4
              + 1.15.1
                N/A
                N/A
              + N/A
            EOT
        }
    }

Plan: 5 to add, 0 to change, 5 to destroy.

[schlichtanders]

I myself had similar issues and figured out how to do a complete disaster recovery of a whole cluster from an S3 backup.

Take a look at the backup and restore documentation at the README. (I've written that)