Wordpress charm creates too many k8s secrets and never cleans up

[kot0dama]

Bug Description

The wordpress charm seems to create a lot of k8s secrets and never clean them up. All our wordpress deployments seem affected and hold more than 3000 secrets in their respective namespaces each.

This greatly increases the load on the k8s cluster and affects its stability.

Also, since we are collecting metrics through kube-state-metrics, it also yields 5 high-cardinality metrics in our Prometheus instance (kube_secret_metadata_resource_version, kube_secret_type, kube_secret_created, kube_secret_info, kube_secret_labels currently have 127180 cardinality which is way too much).

It seems some event triggers creating 3 tokens modeloperator-token-RANDOMBITS model-exec-token-RANDOMBITS wordpress-operator-token-RANDOMBITS.

I am unsure at this point what creates these k8s secrets, but if these are triggered through juju actions or other automated actions, I believe this is a bad idea to store forever such tokens in k8s secrets. Sadly, it doesn't look like k8s secrets can be set with an expiry time.

Could you please elaborate on what these secrets would be used for, and ideally come up with a solution to either not use k8s secrets as storage, or at least prune these secrets after a while?

Thank you!

To Reproduce

Deploy wordpress-k8s charm with listed versions and wait for it to create secrets.

Environment

Juju 2.9.49
Charm: wordpress-k8s
Channel: stable
Revs where this happens: 7, 25, 114

Relevant log output

This is the full list of 3k+ secrets on a single deployment over 3 years+ (for Canonical employees only): https://pastebin.canonical.com/p/fr6DGKKw6Q/

Additional context

No response

[amandahla]

@alithethird Can you have a look, please? Thanks.

[weiiwang01]

I believe this is a Juju/Kubernetes problem, not a charm issue, as the secrets that aren't cleaned up are all of type kubernetes.io/service-account-token. I think this is the same problem described in this issue, caused by juju updating the service account.

kubernetes/kubernetes#84642