Enable k8s client cache for Secrets and ConfigMaps in the single namespace mode

[tfujiwar]

WHAT

• Allow the controller to watch Secrets / ConfigMaps in the single namespace mode.
• Enable the k8s API client cache for Secrets / ConfigMaps in the single namespace mode.

WHY

To reduce latency of runner pod creation (#3276).

The EphemeralRunnerReconciler retrieves a JITConfig Secret for every reconciliation once the Secret is created. It can be a performance bottleneck because the k8s API client cache is disabled for Secrets.

The cache has been disabled for Secrets because it requires cluster-wide list/watch permissions in the default mode. But in the single namespace mode, we can narrow down the permissions only to the single namespace and the controller namespace, which would be acceptable.

This change is aligned with ADR 2023-04-11: Limit Permissions for Service Accounts in Actions-Runner-Controller.

In this mode, you will end up with a manager Role that has all Get/List/Create/Delete/Update/Patch/Watch permissions on resources we need, and a RoleBinding to bind the Role with the controller ServiceAccount in the watched single namespace and the controller namespace

TESTING

We have tested the change with a controller that manages 300-400 EphemeralRunners at peak times. We confirmed that the duration of getting k8s Secrets from the EphemeralRunnerReconciler has decreased due to the caching.

We also confirmed that the number of Pending EphemeralRunners has decreased because they are reconciled faster.

Before

After