Scheduler does not clean up routes on passive cluster if all services have been removed

[rustyjux]

Scheduler does not clean up routes on passive cluster if all services have been removed.

Cause

This is because gwa-scheduler calls routes/sync on kube-api for all the namespaces in the transform_data_by_ns() output dictionary. However, transform_data_by_ns() builds a dictionary of existing namespaces based on the existing routes in Kong on the active cluster (more specifically, the select_tags on those routes).

Solution

A different source for existing namespaces to call routes/sync on is required. Keycloak is one option - at the start of scheduler's main function sync_routes() we could also call Keycloak, get all active (non-decommissioned) NS's with Gold perm-data-plane (stored in env var DATA_PLANE for scheduler) and call sync on those.

Note: This is an opportunity to speed up scheduler - Currently, we call sync on all NSs with routes, regardless of the namespace's data plane - it takes 4 s per namespace but almost none of these NSs are on Gold / DR.

Test case

TESTCASE

• Create a Gateway on Gold data plane
• Add Gateway Services
• Allow Gateway Services to sync to DR (can restart the gwa-scheduler deployment to trigger sync, or wait or call API)
• Delete all Gateway Services in the Gateway (pg with services: [])
• Allow gwa-scheduler to sync to DR again

EXPECTED

• k8s routes are deleted on Gold and Gold DR

ACTUAL

• k8s routes remain on DR