diff --git a/blog/config/nav.yml b/blog/config/nav.yml index 22b9d7ab87..1adb925b2e 100644 --- a/blog/config/nav.yml +++ b/blog/config/nav.yml @@ -49,6 +49,7 @@ nav: - releases/announcing-knative-v0-3-release.md - releases/announcing-knative-v0-2-release.md - Articles: + - articles/consuming_sqs_data_with_integrationsource.md - articles/knative-backstage-security.md - articles/Knative-Serving-WebSockets.md - articles/Building-Stateful-applications-with-Knative-and-Restate.md diff --git a/blog/docs/articles/consuming_sqs_data_with_integrationsource.md b/blog/docs/articles/consuming_sqs_data_with_integrationsource.md new file mode 100644 index 0000000000..2c1fc630da --- /dev/null +++ b/blog/docs/articles/consuming_sqs_data_with_integrationsource.md @@ -0,0 +1,146 @@ +# Consuming AWS SQS Events with Knative Eventing + +**Author: Matthias Weßendorf, Senior Principal Software Engineer @ Red Hat** + +_In a [previous post](/blog/articles/consuming_s3_data_with_knative){:target="_blank"} we discussed the consumption of notifications from an AWS S3 bucket using Apache Camel K. While this is a good approach for getting data from cloud providers, like AWS, into Knative, the Knative Eventing team is aiming to integrate this at the core of its offering with a new CRD, the `IntegrationSource`. This post will describe howto receive SQS notifications and forward them to a regular Knative Broker for further processing._ + +## Installation + +The `IntegrationSource` will be part of Knative Eventing in a future release. Currently it is under development but already included in the `main` branch. For installing Knative Eventing from the sources you can follow the [develpoment guide](https://github.com/knative/eventing/blob/main/DEVELOPMENT.md){:target="_blank"}. + +!!! note + + Installing Knative Eventing from the source repository is not recommended for production cases. The purpose of this blog post is to give an early introduction to the new `IntegrationSource` CRD.. + +## Creating a Knative Broker instance + +Once the `main` branch of Knative Eventing is installed we are using a Knative Broker as the heart of our system, acting as an [Event Mesh](https://knative.dev/docs/eventing/event-mesh/){:target="_blank"} for both event producers and event consumers: + +```yaml +apiVersion: eventing.knative.dev/v1 +kind: Broker +metadata: + namespace: default + name: my-broker +``` + +Now event producers can send events to it and event consumers can receive events. + +## Using IntegrationSource for AWS SQS + +In order to send data from AWS SQS to a Knative component, like `my-broker` we created above, we are using the new `IntegrationSource` CRD. It basically allows to declaratively move data from a system, like AWS SQS, _towards_ a Knative resource, like our above Broker: + +```yaml +apiVersion: sources.knative.dev/v1alpha1 +kind: IntegrationSource +metadata: + name: aws-sqs-source +spec: + aws: + sqs: + queueNameOrArn: "my-queue" + region: "my-queue" + visibilityTimeout: 20 + auth: + secret: + ref: + name: "my-secret" + sink: + ref: + apiVersion: eventing.knative.dev/v1 + kind: Broker + name: my-broker +``` + +The `IntegrationSource` has an `aws` field, for defining different Amazon Web Services, such as `s3`, `ddb-streams` or like in this case `sqs`. Underneath the `aws` property is also a reference to a _Kubernetes Secret_, which contains the credentials for connecting to AWS. All SQS notifications are processed by the source and being forwarded as CloudEvents to the provided `sink` + +!!! note + + If you compare the `IntegrationSource` to the `Pipe` from Apache Camel on the [previous article](/blog/articles/consuming_s3_data_with_knative){:target="_blank"} you will notice the new resource is less verbose and is directly following established Knative development principles, like any other Knative Event Source. + +## Creating the Kubernetes Secret for the IntegrationSource + +For connecting to any AWS service the `IntegrationSource` uses regular Kubernetes `Secret`s, present in the namespace of the resource. The `Secret` can be created like: + +``` +$ kubectl -n create secret generic --from-literal=aws.accessKey=<...> --from-literal=aws.secretKey=<...> +``` + +## Setting up the Consumer application + +Now that we have the `Broker` and the `IntegrationSource` connected to it, it is time to define an application that is receiving _and_ processing the SQS notifications: + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: log-receiver + labels: + app: log-receiver +spec: + containers: + - name: log-receiver + image: gcr.io/knative-releases/knative.dev/eventing/cmd/event_display + imagePullPolicy: Always + ports: + - containerPort: 8080 + protocol: TCP + name: log-receiver +--- +apiVersion: v1 +kind: Service +metadata: + name: log-receiver +spec: + selector: + app: log-receiver + ports: + - port: 80 + protocol: TCP + targetPort: log-receiver + name: http +``` + +Here we define a simple `Pod` and its `Service`, which points to an HTTP-Server, that receives the CloudEvents. As you can see, this is **not** a AWS SQS specific consumer. Basically _any_ HTTP Webserver, in any given language, can be used for processing the CloudEvents coming from a Knative Broker. + +## Subscribe the Consumer application to AWS SQS events + +In order to be able to receive the SQS event notifications, we need to create a `Trigger` for our _Consumer application_: + +```yaml +apiVersion: eventing.knative.dev/v1 +kind: Trigger +metadata: + name: aws-sqs-trigger +spec: + broker: my-broker + subscriber: + ref: + apiVersion: v1 + kind: Service + name: log-receiver +``` + +For debugging purpose we create a `Trigger` without any `filters` so it will forward _all_ CloudEvents to the `log-receiver` application. Once deployed, in the log of the `log-receiver` pod we should be seeing the following for any produced SQS notification on our queue: + +``` +☁️ cloudevents.Event +Context Attributes, + specversion: 1.0 + type: dev.knative.connector.event.aws-sqs + source: dev.knative.eventing.aws-sqs-source + subject: aws-sqs-source + id: 9CC70D09569020C-0000000000000001 + time: 2024-11-08T07:34:16.413Z + datacontenttype: application/json +Extensions, + knativearrivaltime: 2024-11-08T07:34:16.487697262Z +Data, + +``` + +## Conclusion and Outlook + +With the new `IntegrationSource` we will have a good way to integrate services from public cloud providers like AWS, by leveraging Apache Camel Kamelets behind the scenes. The initial set of services is on AWS, like `s3`, `sqs` or `ddb-streams`. However we are planning to add support for different services and providers. + +Since Apache Camel Kamelets can also act as `Sink`s, the team is working on providing a `IntegrationSink`, following same principles.