Migrate from webhook to CRD CEL expressions, ValidatingWebhookPolicy and ValidatingWebhookPolicyBinding

[pmalek]

Problem statement

Creating and managing webhook resources like certificates and coordinating the kube-webhook-certgen Job to properly update the webhook's caBundle has proven to be difficult and unreliable.

This setup has already caused issues for a number of users (internal and external):

• Admission webhook Job sporadically fails causes false positive alerts #428
• Consider using Fail failure policy in ControlPlane's validating webhook configuration #122
• Cannot install KGO on OpenShift #921

Proposed solution

Migrate away from ValidatingWebhookConfiguration and managing certificates and kube-webhook-certgen Job, towards using ValidatingWebhookPolicy and ValidatingWebhookPolicyBindings which will be deployed as part of KGO's helm chart.

All the rules that are currently defined in

• https://github.com/Kong/gateway-operator/blob/d6b69e3366089fda45386e3289138b5832b97c96/internal/validation/dataplane/validation.go
• and https://github.com/Kong/gateway-operator/blob/3876430e09e61edce58bd8464989e33236bd1872/internal/validation/controlplane/validation.go

are either trivial to implement at CEL expressions on CRDs or through CEL expressions in ValidatingAdmissionPolicy.

Some rules would be more complex than the others and would require to be added as the latter, e.g. validation of KONG_PORT_MAPS and KONG_PROXY_LISTEN against DataPlane's `spec.network.services.ingress.ports, when added as CRD CEL expression:

// +kubebuilder:validation:XValidation:message="Each port from spec.network.services.ingress.ports has to have an accompanying port in KONG_PORT_MAPS env",rule="self.spec.network.services.ingress.ports.all(p, self.spec.deployment.podTemplateSpec.spec.containers.filter(c, c.name == 'proxy')[0].env.filter(e, e.name == 'KONG_PORT_MAPS')[0].value.split(',').exists(pm, pm.split(':')[1] == string(p.targetPort)))"

yields the following

The CustomResourceDefinition "dataplanes.gateway-operator.konghq.com" is invalid:
* spec.validation.openAPIV3Schema.x-kubernetes-validations[1].rule: Forbidden: estimated rule cost exceeds budget by factor of more than 100x (try simplifying the rule, or adding maxItems, maxProperties, and maxLength where arrays, maps, and strings are declared)
* spec.validation.openAPIV3Schema.x-kubernetes-validations[1].rule: Forbidden: contributed to estimated rule cost total exceeding cost limit for entire OpenAPIv3 schema
* spec.validation.openAPIV3Schema: Forbidden: x-kubernetes-validations estimated rule cost total for entire OpenAPIv3 schema exceeds budget by factor of more than 100x (try simplifying the rule, or adding maxItems, maxProperties, and maxLength where arrays, maps, and strings are declared)

But works when implemented as:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: ports.dataplane.gateway-operator.konghq.com
spec:
  matchConstraints:
    resourceRules:
    - apiGroups:   ["gateway-operator.konghq.com"]
      apiVersions: ["v1beta1"]
      operations:  ["CREATE", "UPDATE"]
      resources:   ["dataplanes"]
  variables:
  - name: ingressPorts
    expression: object.spec.network.services.ingress.ports
  - name: proxyContainer
    expression: "object.spec.deployment.podTemplateSpec.spec.containers.exists(c, c.name == 'proxy') ? object.spec.deployment.podTemplateSpec.spec.containers.filter(c, c.name == 'proxy')[0] : null"
  # Using string functions from: https://pkg.go.dev/github.com/google/cel-go/ext
  validations:
  - expression: |
      variables.proxyContainer != null &&
        variables.ingressPorts.all(p, variables.proxyContainer.env.filter(e, e.name == "KONG_PORT_MAPS")[0].value.
                  split(",").exists(pm,
                      pm.split(":")[1] == string(p.targetPort)
                      )
                  )
    messageExpression: "'Each port from spec.network.services.ingress.ports has to have an accompanying port in KONG_PORT_MAPS env'"
    reason: Invalid
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: binding-ports.dataplane.gateway-operator.konghq.com
spec:
  policyName: ports.dataplane.gateway-operator.konghq.com
  validationActions: [Deny]

validatingadmissionpolicy.admissionregistration.k8s.io/ports.dataplane.gateway-operator.konghq.com configured
validatingadmissionpolicybinding.admissionregistration.k8s.io/binding-ports.dataplane.gateway-operator.konghq.com unchanged
The dataplanes "dataplane-example" is invalid: : ValidatingAdmissionPolicy 'ports.dataplane.gateway-operator.konghq.com' with binding 'binding-ports.dataplane.gateway-operator.konghq.com' denied request: Each ports from spec.network.services.ingress.ports has to have an accompanying port in KONG_PORT_MAPS env