diff --git a/.github/styles/kong/dictionary.txt b/.github/styles/kong/dictionary.txt index 6d8ecb08862a..4c9dfc07251d 100644 --- a/.github/styles/kong/dictionary.txt +++ b/.github/styles/kong/dictionary.txt @@ -258,6 +258,7 @@ Github glibc globbing Gluu +goecho gojira Golang Goroutine diff --git a/app/_data/docs_nav_kic_3.4.x.yml b/app/_data/docs_nav_kic_3.4.x.yml index 7d74625c229c..e379b0cc3bd1 100644 --- a/app/_data/docs_nav_kic_3.4.x.yml +++ b/app/_data/docs_nav_kic_3.4.x.yml @@ -145,6 +145,8 @@ items: url: /guides/security/client-ip - text: Kubernetes Secrets in Plugins url: /guides/security/plugin-secrets + - text: Verifying Upstream TLS + url: /guides/security/verify-upstream-tls #- text: Service Mesh # items: # - text: Kong Mesh diff --git a/app/_includes/md/kic/ca-certificates-note.md b/app/_includes/md/kic/ca-certificates-note.md new file mode 100644 index 000000000000..ee1d27043f80 --- /dev/null +++ b/app/_includes/md/kic/ca-certificates-note.md @@ -0,0 +1,13 @@ +{:.note} +> CA certificates in Kong are provisioned by creating a `Secret` resource in Kubernetes. CA certificate secrets must +> have the following properties: +> - the `konghq.com/ca-cert: "true"` label applied. +> - a`cert` data property which contains a valid CA certificate in PEM format. +> - a `kubernetes.io/ingress.class` annotation whose value matches the value of the controller's `--ingress-class` + argument. By default, that value is `kong`. +> - an `id` data property which contains a random UUID. +> +> Each CA certificate that you create needs a unique ID. Any random UUID should suffice here and it doesn't have a +> security implication. You can use [uuidgen](https://linux.die.net/man/1/uuidgen) (Linux, OS X) +> or [New-Guid](https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/new-guid) (Windows) to +> generate an ID. diff --git a/app/_src/kubernetes-ingress-controller/guides/security/verify-upstream-tls.md b/app/_src/kubernetes-ingress-controller/guides/security/verify-upstream-tls.md new file mode 100644 index 000000000000..4cf1d8402f33 --- /dev/null +++ b/app/_src/kubernetes-ingress-controller/guides/security/verify-upstream-tls.md @@ -0,0 +1,392 @@ +--- +title: TLS Verification of Upstream Services +type: how-to +purpose: | + How to configure Kong to verify TLS certificates when connecting to upstream services +--- + +From version 3.4.0 {{ site.kic_product_name }} allows you to configure {{ site.base_gateway }} to run the TLS +verification of upstream services. If you have an upstream service that handles TLS, you can configure {{ +site.base_gateway }} to verify the certificate it presents by attaching a CA certificate to a service. This guide +shows how to make this happen using the `BackendTLSPolicy` (when using Gateway API) or Kubernetes Service annotations +(when using Ingress API). + +{% include /md/kic/prerequisites.md release=page.release disable_gateway_api=false%} + +## Set up an upstream service with TLS + +Before we configure {{ site.kic_product_name }} to verify the certificate of the upstream service, we need to set up an +upstream service that handles TLS. We can use the `kong/goecho` service as an example. It is a simple echo server that +can be configured to serve HTTPS on an arbitrary port using a TLS certificate and key pair. + +### Create a certificate chain + +To showcase all the possible configurations, we will create a certificate chain with a root CA, an intermediate CA, and +a leaf server certificate. We will use `openssl` to generate the certificates. + +```shell +# Create a directory to store the certificates +mkdir certs +cd certs + +# Generate Root CA +openssl req -new -newkey rsa:2048 -nodes -keyout root.key -subj "/CN=root" -x509 -days 365 -out root.crt + +# Generate Intermediate CA +openssl req -new -newkey rsa:2048 -nodes -keyout inter.key -subj "/CN=inter" -out inter.csr +openssl x509 -req -in inter.csr -CA root.crt -CAkey root.key -CAcreateserial -days 365 -out inter.crt -extfile <(echo "basicConstraints=CA:TRUE") + +# Generate Leaf Certificate +openssl req -new -newkey rsa:2048 -nodes -keyout leaf.key -subj "/CN=kong.example" -out leaf.csr +openssl x509 -req -in leaf.csr -CA inter.crt -CAkey inter.key -CAcreateserial -days 365 -out leaf.crt -extfile <(printf "subjectAltName=DNS:kong.example") + +# Create a certificate chain +cat leaf.crt inter.crt > chain.crt + +# Cleanup intermediate files +rm -f *.csr *.srl +cd .. +``` + +Running this script will generate the following files in `certs` directory: + +- `root.key` and `root.crt`: Root CA key and certificate +- `inter.key`, `inter.crt`: Intermediate CA key and certificate +- `leaf.key`, `leaf.crt`: Server key and certificate (valid for `kong.example` SAN) +- `chain.crt`: Server certificate chain + +### Deploy the goecho service + +We will deploy the `kong/goecho` service with the generated certificates. We will use the `leaf.key` and +`chain.crt` files to configure the service to serve HTTPS. + +First, deploy the standard `kong/goecho` service (no HTTPS support). + +```bash +kubectl apply -f {{site.links.web}}/assets/kubernetes-ingress-controller/examples/echo-service.yaml +``` + +The results should look like this. + +```text +service/echo created +deployment.apps/echo created +``` + +Now, let's configure it to serve HTTPS. Let's create a secret with the server key and the certificate chain (including +the intermediate certificate and the leaf certificate). + +```bash +kubectl create secret tls goecho-tls --key ./certs/leaf.key --cert ./certs/chain.crt +``` + +Next, patch the `echo` deployment to use the secret and serve HTTPS using it. + +```bash +kubectl patch deployment echo -p '{ + "spec": { + "template": { + "spec": { + "containers": [ + { + "name": "echo", + "ports": [ + { + "containerPort": 443 + } + ], + "env": [ + { + "name": "HTTPS_PORT", + "value": "443" + }, + { + "name": "TLS_CERT_FILE", + "value": "/etc/tls/tls.crt" + }, + { + "name": "TLS_KEY_FILE", + "value": "/etc/tls/tls.key" + } + ], + "volumeMounts": [ + { + "mountPath": "/etc/tls", + "name": "tls" + } + ] + } + ], + "volumes": [ + { + "name": "tls", + "secret": { + "secretName": "goecho-tls" + } + } + ] + } + } + } +}' +``` + +Also, patch the service to use HTTPS by adding the `konghq.com/protocol: https` annotation and the `spec.ports` +entry. + +```shell +kubectl patch service echo -p '{ + "metadata": { + "annotations": { + "konghq.com/protocol": "https" + } + }, + "spec": { + "ports": [ + { + "name": "https", + "port": 443, + "targetPort": 443 + } + ] + } +}' +``` + +The results should look like this: + +```text +deployment.apps/echo patched +service/echo patched +``` + +## Expose the goecho service + +Now that the `kong/goecho` service is serving HTTPS, we need to expose it. + +{% include /md/kic/http-test-routing-resource.md service=echo port=443 %} + +Verify connectivity by issuing an HTTP request to proxy. The service serves HTTPS but {{ site.base_gateway }} initiates +the connection and proxies it as HTTP in this case, thus the request should be made over HTTP. The `Host` header is +required to match the hostname of the service. + +```shell +curl -H Host:kong.example $PROXY_IP/echo +``` + +You should see a response similar to this: + +```text +Welcome, you are connected to node orbstack. +Running on Pod echo-bd94b7dcc-qxs2b. +In namespace default. +With IP address 192.168.194.9. +Through HTTPS connection. +``` + +That means the service is up and running and {{ site.base_gateway }} connects to it successfully over HTTPS, _without +verification_. + +## Configure {{ site.base_gateway }} to verify the upstream TLS certificate + +### Enable TLS verification + +To configure {{ site.base_gateway }} to verify the certificate of the upstream service, we need to annotate the +service accordingly. + +```shell +kubectl annotate service echo konghq.com/tls-verify=true +``` + +The results should look like this. + +```text +service/echo annotated +``` + +Now, when you issue the same request as before, you should see an error similar to this. + +```shell +curl -H Host:kong.example $PROXY_IP/echo +``` + +```text +{ + "message":"An invalid response was received from the upstream server", + "request_id":"e2b3182856c96c23d61e880d0a28012f" +} +``` + +If you inspect {{ site.base_gateway }}'s container logs, you should see an error indicating an issue with TLS handshake. + +```shell + kubectl logs -n kong deploy/kong-gateway | grep "GET /echo" +``` + +```text +2024/11/28 18:04:05 [error] 1280#0: *20004 upstream SSL certificate verify error: (20:unable to get local issuer certificate) while SSL handshaking to upstream, client: 192.168.194.1, server: kong, request: "GET /echo HTTP/1.1", upstream: "https://192.168.194.9:443/", host: "kong.example", request_id: "e2b3182856c96c23d61e880d0a28012f" +192.168.194.1 - - [28/Nov/2024:18:04:05 +0000] "GET /echo HTTP/1.1" 502 126 "-" "curl/8.7.1" kong_request_id: "e2b3182856c96c23d61e880d0a28012f" +``` + +{{ site.base_gateway }} is now verifying the certificate of the upstream service and rejecting the connection because +the certificate is not trusted. + +### Add the root CA certificate to {{ site.base_gateway }} + +That can be fixed by adding the root CA certificate to the {{ site.base_gateway }}'s CA certificates and associating +it with the service. + +First, create a secret with the root CA certificate. + +{% include /md/kic/ca-certificates-note.md %} + +```shell +echo ' +apiVersion: v1 +kind: Secret +metadata: + name: root-ca + labels: + konghq.com/ca-cert: "true" # This label is required for the CA certificate to be recognized by Kong + annotations: + kubernetes.io/ingress.class: kong +data: + cert: '$(base64 -w0 ./certs/root.crt)' + # An arbitrary ID for the certificate + id: '$(printf "bf6e0f14-78cd-45ad-9325-87ec7ef7b890" | base64 -w0)' +' | kubectl apply -f - +``` + +The results should look like this. + +```text +secret/root-ca created +``` + +Now, associate the root CA certificate with the service passing its name to `konghq.com/ca-certificates` annotation. + +{:.note} +> The `konghq.com/ca-certificates` annotation is a comma-separated list of CA certificate names. You can add multiple +> CA certificates to the list. + +```shell +kubectl annotate service echo konghq.com/ca-certificates='root-ca' +``` + +The results should look like this. + +```text +service/echo annotated +``` + +Now, when you issue the same request as before, you should see a successful response. + +```shell +curl -H Host:kong.example $PROXY_IP/echo +``` + +```text +Welcome, you are connected to node orbstack. +Running on Pod echo-bd94b7dcc-gdnhl. +In namespace default. +With IP address 192.168.194.18. +Through HTTPS connection. +``` + +{{ site.base_gateway }} is now verifying the certificate of the upstream service and accepting the connection because +the certificate is trusted. + +### Configure verification depth + +By default, {{ site.base_gateway }} verifies the certificate chain up to the root CA certificate with no depth limit. +You can configure the verification depth by annotating the service with the `konghq.com/tls-verify-depth` annotation. + +For example, to limit the verification depth to 1 (i.e., only verify one intermediate certificate), you can annotate the +service like this: + +```shell +kubectl annotate service echo konghq.com/tls-verify-depth=1 +``` + +The results should look like this. + +```text +service/echo annotated +``` + +Now, when you issue the same request as before, you should still see a successful response. + +```shell +curl -H Host:kong.example $PROXY_IP/echo +``` + +```text +Welcome, you are connected to node orbstack. +Running on Pod echo-bd94b7dcc-gdnhl. +In namespace default. +With IP address +Through HTTPS connection. +``` + +You can also set the verification depth to 0 to not allow any intermediate certificates. + +```shell +kubectl annotate --overwrite service echo konghq.com/tls-verify-depth=0 +``` + +The results should look like this. + +```text +service/echo annotated +``` + +Now, when you issue the same request as before, you should see an error similar to this. + +```shell +curl -H Host:kong.example $PROXY_IP/echo +``` + +```text +{ + "message":"An invalid response was received from the upstream server", + "request_id":"e2b3182856c96c23d61e880d0a28012f" +} +``` + +You can inspect {{ site.base_gateway }}'s container logs to see the error. + +```shell +kubectl logs -n kong deploy/kong-gateway | grep "GET /echo" +``` + +```text +2024/11/29 11:41:46 [error] 1280#0: *45531 upstream SSL certificate verify error: (22:certificate chain too long) while SSL handshaking to upstream, client: 192.168.194.1, server: kong, request: "GET /echo HTTP/1.1", upstream: "https://192.168.194.19:443/", host: "kong.example", request_id: "678281372fb8907ed06d517cf515de78" +192.168.194.1 - - [29/Nov/2024:11:41:46 +0000] "GET /echo HTTP/1.1" 502 126 "-" "curl/8.7.1" kong_request_id: "678281372fb8907ed06d517cf515de78" +``` + +{{ site.base_gateway }} is now rejecting the connection because the certificate chain is too long. Changing the +verification depth to 1 should allow the connection to succeed again. + +```shell +kubectl annotate --overwrite service echo konghq.com/tls-verify-depth=1 +``` + +The results should look like this. + +```text +service/echo annotated +``` + +Now, when you issue the same request as before, you should see a successful response. + +```shell +curl -H Host:kong.example $PROXY_IP/echo +``` + +```text +Welcome, you are connected to node orbstack. +Running on Pod echo-bd94b7dcc-9lvmf. +In namespace default. +With IP address 192.168.194.19. +Through HTTPS connection. +``` diff --git a/app/_src/kubernetes-ingress-controller/plugins/mtls.md b/app/_src/kubernetes-ingress-controller/plugins/mtls.md index 7d3a7a3bf40a..b894c01568ec 100644 --- a/app/_src/kubernetes-ingress-controller/plugins/mtls.md +++ b/app/_src/kubernetes-ingress-controller/plugins/mtls.md @@ -18,17 +18,7 @@ Configure the {{site.kic_product_name}} to verify client certificates using CA c 1. Add the generated certificates to Kong. - {:.note} - > CA certificates in Kong are provisioned by creating a `Secret` resource in Kubernetes. - > - > CA certificate secrets must have the following properties: - > - the `konghq.com/ca-cert: "true"` label applied. - > - a`cert` data property which contains a valid CA certificate in PEM format. - > - a `kubernetes.io/ingress.class` annotation whose value matches the value of the controller's `--ingress-class` argument. By default, that value is `kong`. - > - an `id` data property which contains a random UUID. - > - > Each CA certificate that you create needs a unique ID. Any random UUID should suffice here and it doesn't have a security implication. You can use [uuidgen](https://linux.die.net/man/1/uuidgen) (Linux, OS X) or [New-Guid](https://docs.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/new-guid) (Windows) to generate an ID. - + {% include /md/kic/ca-certificates-note.md %} ```bash $ kubectl create secret generic my-ca-cert --from-literal=id=cce8c384-721f-4f58-85dd-50834e3e733a --from-file=cert=./cert.pem diff --git a/app/_src/kubernetes-ingress-controller/reference/annotations.md b/app/_src/kubernetes-ingress-controller/reference/annotations.md index 4915bf5ee443..557c84f375da 100644 --- a/app/_src/kubernetes-ingress-controller/reference/annotations.md +++ b/app/_src/kubernetes-ingress-controller/reference/annotations.md @@ -4,6 +4,7 @@ type: reference purpose: | What annotatations are available and what do they do? --- + The {{site.kic_product_name}} supports these annotations on various resources. @@ -33,8 +34,8 @@ Following annotations are supported on Ingress resources: {% if_version gte:3.2.x %} | [`konghq.com/headers-separator`](#konghqcomheaders-separator) | Separator for header values, other than default `,` | {% endif_version %} -| [`konghq.com/rewrite`](#konghqcomrewrite) | Rewrite the path of a URL | -| [`konghq.com/tags`](#konghqcomtags) | Assign custom tags to Kong entities generated out of this Ingress | +| [`konghq.com/rewrite`](#konghqcomrewrite) | Rewrite the path of a URL | +| [`konghq.com/tags`](#konghqcomtags) | Assign custom tags to Kong entities generated out of this Ingress | `kubernetes.io/ingress.class` is required, and its value should match the value of the `--ingress-class` controller argument (`kong` by default). @@ -58,6 +59,11 @@ These annotations are supported on Service resources. | [`konghq.com/write-timeout`](#konghqcomwritetimeout) | Set the timeout for writing data | | [`konghq.com/retries`](#konghqcomretries) | Set the number of times to retry requests that failed | | [`konghq.com/tags`](#konghqcomtags) | Assign custom tags to Kong entities generated out of this Service | +{% if_version gte:3.4.x %} +| [`konghq.com/tls-verify`](#konghqcomtls-verify) | Enable or disable verification of the upstream service's TLS certificates | +| [`konghq.com/tls-verify-depth`](#konghqcomtls-verify-depth) | Set the maximal depth of a certificate chain when verifying the upstream service's TLS certificates | +| [`konghq.com/ca-certificates`](#konghqcomca-certificates) | Assign CA certificates to be used for the upstream service's TLS certificates verification | +{% endif_version %} ## KongConsumer resource @@ -92,7 +98,7 @@ Kong resources (KongConsumer, TCPIngress, etc.) still use the `kubernetes.io/ingress.class` annotation. If you have multiple Ingress controllers in a single cluster, -you can pick one by specifying the `ingress.class` annotation. +you can pick one by specifying the `ingress.class` annotation. Here is an example to create an Ingress with an annotation: ```yaml @@ -663,3 +669,41 @@ The value of the annotation is the name of the `KongUpstreamPolicy` object in th same namespace as the `Service`. Please refer to the [KongUpstreamPolicy reference](/kubernetes-ingress-controller/{{page.release}}/reference/custom-resources/#kongupstreampolicy) for details on how to configure the `KongUpstreamPolicy` resource. + +{% if_version gte:3.4.x %} + +### konghq.com/tls-verify + +> Available since controller 3.4 + +This annotation can be used to enable or disable verification of the upstream service's TLS certificates. +The value of the annotation should be either `true` or `false`. By default, the verification is disabled. + +See [TLS verification of Upstream Service](/kubernetes-ingress-controller/{{page.release}}/guides/security/verify-upstream-tls) +guide for more information. + +### konghq.com/tls-verify-depth + +> Available since controller 3.4 + +This annotation can be used to set the maximal depth of a certificate chain when verifying the upstream service's TLS +certificates. +The value of the annotation should be an integer. If not set, a system default value is used. + +See [TLS verification of Upstream Service](/kubernetes-ingress-controller/{{page.release}}/guides/security/verify-upstream-tls) +guide for more information. + +### konghq.com/ca-certificates + +> Available since controller 3.4 + +This annotation can be used to assign CA certificates to be used for the upstream service's TLS certificates +verification. +The value of the annotation should be a comma-separated list of CA certificate names. + +{% include /md/kic/ca-certificates-note.md %} + +See [TLS verification of Upstream Service](/kubernetes-ingress-controller/{{page.release}}/guides/security/verify-upstream-tls) +guide for more information. + +{% endif_version %}