Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[binatmi/schema-registry] Add support for Amazon MSK IAM auth #74014

Closed
wants to merge 1 commit into from
Closed

[binatmi/schema-registry] Add support for Amazon MSK IAM auth #74014

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions bitnami/schema-registry/7.7/debian-12/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
FROM docker.io/bitnami/minideb:bookworm

ARG DOWNLOADS_URL="downloads.bitnami.com/files/stacksmith"
ARG AWS_MSK_IAM_AUTH_VERSION="2.2.0"
ARG JAVA_EXTRA_SECURITY_DIR="/bitnami/java/extra-security"
ARG TARGETARCH

Expand Down Expand Up @@ -42,6 +43,16 @@ RUN mkdir -p /tmp/bitnami/pkg/cache/ ; cd /tmp/bitnami/pkg/cache/ ; \
tar -zxf "${COMPONENT}.tar.gz" -C /opt/bitnami --strip-components=2 --no-same-owner --wildcards '*/files' ; \
rm -rf "${COMPONENT}".tar.gz{,.sha256} ; \
done

RUN mkdir -p /tmp/bitnami/aws-msk-iam-auth/ ; cd /tmp/bitnami/aws-msk-iam-auth/ ; \
AWS_MSK_IAM_AUTH_DOWNLOAD_URL="https://github.com/aws/aws-msk-iam-auth/releases/download/v${AWS_MSK_IAM_AUTH_VERSION}" ; \
AWS_MSK_IAM_AUTH_FILE="aws-msk-iam-auth-${AWS_MSK_IAM_AUTH_VERSION}-all.jar" ; \
curl -SsLf "${AWS_MSK_IAM_AUTH_DOWNLOAD_URL}/${AWS_MSK_IAM_AUTH_FILE}" -O ; \
curl -SsLf "${AWS_MSK_IAM_AUTH_DOWNLOAD_URL}/${AWS_MSK_IAM_AUTH_FILE}.sha256" -O ; \
echo "$(cat ${AWS_MSK_IAM_AUTH_FILE}.sha256)" "${AWS_MSK_IAM_AUTH_FILE}" | sha256sum -c ; \
rm -f "${AWS_MSK_IAM_AUTH_DOWNLOAD_URL}.sha256" ; \
cp "${AWS_MSK_IAM_AUTH_FILE}" /opt/bitnami/schema-registry/share/java/schema-registry/

RUN apt-get autoremove --purge -y curl && \
apt-get update && apt-get upgrade -y && \
apt-get clean && rm -rf /var/lib/apt/lists /var/cache/apt/archives
Expand Down
18 changes: 13 additions & 5 deletions bitnami/schema-registry/7.7/debian-12/rootfs/opt/bitnami/scripts/libschemaregistry.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -144,12 +144,12 @@ schema_registry_validate() {

if [[ -n "$SCHEMA_REGISTRY_KAFKA_BROKERS" ]]; then
if brokers_auth_protocol="$(schema_registry_brokers_auth_protocol)"; then
if [[ "$brokers_auth_protocol" =~ SSL ]]; then
if [[ "$brokers_auth_protocol" =~ SSL ]] && [[ -z "$SCHEMA_REGISTRY_KAFKASTORE_CLIENT_AUTH_DISABLED" ]]; then
if [[ ! -f ${SCHEMA_REGISTRY_CERTS_DIR}/schema-registry.keystore.jks ]] || [[ ! -f ${SCHEMA_REGISTRY_CERTS_DIR}/schema-registry.truststore.jks ]]; then
print_validation_error "In order to configure the TLS encryption for communication with Kafka brokers, you must mount your schema-registry.keystore.jks and schema-registry.truststore.jks certificates to the ${SCHEMA_REGISTRY_CERTS_DIR} directory."
fi
fi
if [[ "$brokers_auth_protocol" =~ SASL ]]; then
if [[ "$brokers_auth_protocol" =~ SASL ]] && [[ "$SCHEMA_REGISTRY_KAFKASTORE_SASL_MECHANISM" != "AWS_MSK_IAM" ]]; then
if [[ -z "$SCHEMA_REGISTRY_KAFKA_SASL_USERS" ]] || [[ -z "$SCHEMA_REGISTRY_KAFKA_SASL_PASSWORDS" ]]; then
print_validation_error "In order to configure SASL authentication for Kafka, you must provide the SASL credentials. Set the environment variables SCHEMA_REGISTRY_KAFKA_SASL_USERS and SCHEMA_REGISTRY_KAFKA_SASL_PASSWORDs"
fi
Expand Down Expand Up @@ -291,7 +291,7 @@ schema_registry_initialize() {
brokers_auth_protocol="$(schema_registry_brokers_auth_protocol)"
[[ -n "$SCHEMA_REGISTRY_KAFKA_BROKERS" ]] && schema_registry_conf_set "kafkastore.bootstrap.servers" "${SCHEMA_REGISTRY_KAFKA_BROKERS/%,/}"
schema_registry_conf_set "kafkastore.security.protocol" "$brokers_auth_protocol"
if [[ "$brokers_auth_protocol" =~ SASL ]]; then
if [[ "$brokers_auth_protocol" =~ SASL ]] && [[ "$SCHEMA_REGISTRY_KAFKASTORE_SASL_MECHANISM" != "AWS_MSK_IAM" ]]; then
read -r -a users <<< "$(tr ',;' ' ' <<< "${SCHEMA_REGISTRY_KAFKA_SASL_USERS}")"
read -r -a passwords <<< "$(tr ',;' ' ' <<< "${SCHEMA_REGISTRY_KAFKA_SASL_PASSWORDS}")"
aux_string="org.apache.kafka.common.security.plain.PlainLoginModule required username=\"${users[0]:-}\" password=\"${passwords[0]:-}\";"
Expand All @@ -302,15 +302,23 @@ schema_registry_initialize() {
schema_registry_conf_set "kafkastore.sasl.jaas.config" "$aux_string"
fi

if [[ "$brokers_auth_protocol" =~ SSL ]]; then
if [[ "$brokers_auth_protocol" =~ SSL ]] && [[ -z "$SCHEMA_REGISTRY_KAFKASTORE_CLIENT_AUTH_DISABLED" ]]; then
schema_registry_conf_set "kafkastore.ssl.keystore.location" "${SCHEMA_REGISTRY_CERTS_DIR}/schema-registry.keystore.jks"
[[ -n "$SCHEMA_REGISTRY_KAFKA_KEYSTORE_PASSWORD" ]] && schema_registry_conf_set "kafkastore.ssl.keystore.password" "$SCHEMA_REGISTRY_KAFKA_KEYSTORE_PASSWORD"
schema_registry_conf_set "kafkastore.ssl.truststore.location" "${SCHEMA_REGISTRY_CERTS_DIR}/schema-registry.truststore.jks"
[[ -n "$SCHEMA_REGISTRY_KAFKA_KEY_PASSWORD" ]] && schema_registry_conf_set "kafkastore.ssl.key.password" "$SCHEMA_REGISTRY_KAFKA_KEY_PASSWORD"
[[ -n "$SCHEMA_REGISTRY_KAFKA_TRUSTSTORE_PASSWORD" ]] && schema_registry_conf_set "kafkastore.ssl.truststore.password" "$SCHEMA_REGISTRY_KAFKA_TRUSTSTORE_PASSWORD"
[[ -n "$SCHEMA_REGISTRY_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM" ]] && schema_registry_conf_set "kafkastore.ssl.endpoint.identification.algorithm" "$SCHEMA_REGISTRY_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM"
fi

if [[ -n "$SCHEMA_REGISTRY_KAFKASTORE_SECURITY_PROTOCOL" ]]; then
schema_registry_conf_set "kafkastore.security.protocol" "$SCHEMA_REGISTRY_KAFKASTORE_SECURITY_PROTOCOL"
[[ -n "$SCHEMA_REGISTRY_KAFKASTORE_SASL_MECHANISM" ]] && schema_registry_conf_set "kafkastore.sasl.mechanism" "$SCHEMA_REGISTRY_KAFKASTORE_SASL_MECHANISM"
[[ -n "$SCHEMA_REGISTRY_KAFKASTORE_SASL_JAAS_CONFIG" ]] && schema_registry_conf_set "kafkastore.sasl.jaas.config" "$SCHEMA_REGISTRY_KAFKASTORE_SASL_JAAS_CONFIG"
[[ -n "$SCHEMA_REGISTRY_KAFKASTORE_SASL_CLIENT_CALLBACK_HANDLER_CLASS" ]] && schema_registry_conf_set "kafkastore.sasl.client.callback.handler.class" "$SCHEMA_REGISTRY_KAFKASTORE_SASL_CLIENT_CALLBACK_HANDLER_CLASS"
fi

[[ -n "$SCHEMA_REGISTRY_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM" ]] && schema_registry_conf_set "kafkastore.ssl.endpoint.identification.algorithm" "$SCHEMA_REGISTRY_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM"

# Listeners settings
if [[ -n "$SCHEMA_REGISTRY_LISTENERS" ]]; then
schema_registry_conf_set "listeners" "$SCHEMA_REGISTRY_LISTENERS"
Expand Down
5 changes: 5 additions & 0 deletions bitnami/schema-registry/7.7/debian-12/rootfs/opt/bitnami/scripts/schema-registry-env.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,11 @@ export SCHEMA_REGISTRY_SSL_TRUSTSTORE_PASSWORD="${SCHEMA_REGISTRY_SSL_TRUSTSTORE
export SCHEMA_REGISTRY_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM="${SCHEMA_REGISTRY_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM:-}"
export SCHEMA_REGISTRY_CLIENT_AUTHENTICATION="${SCHEMA_REGISTRY_CLIENT_AUTHENTICATION:-}"
export SCHEMA_REGISTRY_AVRO_COMPATIBILY_LEVEL="${SCHEMA_REGISTRY_AVRO_COMPATIBILY_LEVEL:-}"
export SCHEMA_REGISTRY_KAFKASTORE_SECURITY_PROTOCOL="${SCHEMA_REGISTRY_KAFKASTORE_SECURITY_PROTOCOL:-}"
export SCHEMA_REGISTRY_KAFKASTORE_SASL_MECHANISM="${SCHEMA_REGISTRY_KAFKASTORE_SASL_MECHANISM:-}"
export SCHEMA_REGISTRY_KAFKASTORE_SASL_JAAS_CONFIG="${SCHEMA_REGISTRY_KAFKASTORE_SASL_JAAS_CONFIG:-}"
export SCHEMA_REGISTRY_KAFKASTORE_SASL_CLIENT_CALLBACK_HANDLER_CLASS="${SCHEMA_REGISTRY_KAFKASTORE_SASL_CLIENT_CALLBACK_HANDLER_CLASS:-}"
export SCHEMA_REGISTRY_KAFKASTORE_CLIENT_AUTH_DISABLED="${SCHEMA_REGISTRY_KAFKASTORE_CLIENT_AUTH_DISABLED:-}"
export SCHEMA_REGISTRY_DEBUG="${SCHEMA_REGISTRY_DEBUG:-}"

# Custom environment variables may be defined below
2 changes: 1 addition & 1 deletion ...mi/schema-registry/7.7/debian-12/rootfs/opt/bitnami/scripts/schema-registry/postunpack.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,4 +99,4 @@ done

# Copy all initially generated configuration files to the default directory
# (this is to avoid breaking when entrypoint is being overridden)
cp -r "${SCHEMA_REGISTRY_CONF_DIR}/"* "$SCHEMA_REGISTRY_DEFAULT_CONF_DIR"
cp -r "${SCHEMA_REGISTRY_CONF_DIR}/"* "$SCHEMA_REGISTRY_DEFAULT_CONF_DIR"
58 changes: 40 additions & 18 deletions bitnami/schema-registry/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,24 +60,30 @@ docker build -t bitnami/APP:latest .

#### Customizable environment variables

| Name | Description | Default Value |
|---------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------|
| `SCHEMA_REGISTRY_MOUNTED_CONF_DIR` | Directory for including custom configuration files (that override the default generated ones) | `${SCHEMA_REGISTRY_VOLUME_DIR}/etc` |
| `SCHEMA_REGISTRY_KAFKA_BROKERS` | List of Kafka brokers to connect to. | `nil` |
| `SCHEMA_REGISTRY_ADVERTISED_HOSTNAME` | Advertised hostname in ZooKeeper. | `nil` |
| `SCHEMA_REGISTRY_KAFKA_KEYSTORE_PASSWORD` | Password to access the keystore. | `nil` |
| `SCHEMA_REGISTRY_KAFKA_KEY_PASSWORD` | Password to be able to used ssl secured kafka broker with SR | `nil` |
| `SCHEMA_REGISTRY_KAFKA_TRUSTSTORE_PASSWORD` | Password to access the truststore. | `nil` |
| `SCHEMA_REGISTRY_KAFKA_SASL_USER` | SASL user to authenticate with Kafka. | `nil` |
| `SCHEMA_REGISTRY_KAFKA_SASL_PASSWORD` | SASL password to authenticate with Kafka. | `nil` |
| `SCHEMA_REGISTRY_LISTENERS` | Comma-separated list of listeners that listen for API requests over either HTTP or HTTPS. | `nil` |
| `SCHEMA_REGISTRY_SSL_KEYSTORE_PASSWORD` | Password to access the SSL keystore. | `nil` |
| `SCHEMA_REGISTRY_SSL_KEY_PASSWORD` | Password to access the SSL key. | `nil` |
| `SCHEMA_REGISTRY_SSL_TRUSTSTORE_PASSWORD` | Password to access the SSL truststore. | `nil` |
| `SCHEMA_REGISTRY_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM` | Endpoint identification algorithm to validate the server hostname using the server certificate. | `nil` |
| `SCHEMA_REGISTRY_CLIENT_AUTHENTICATION` | Client authentication configuration. Valid options: none, requested, over required. | `nil` |
| `SCHEMA_REGISTRY_AVRO_COMPATIBILY_LEVEL` | The Avro compatibility type. Valid options: none, backward, backward_transitive, forward, forward_transitive, full, or full_transitive | `nil` |
| `SCHEMA_REGISTRY_DEBUG` | Enable Schema Registry debug logs. Valid options: true or false | `nil` |
| Name | Description | Default Value |
|-----------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------|
| `SCHEMA_REGISTRY_MOUNTED_CONF_DIR` | Directory for including custom configuration files (that override the default generated ones) | `${SCHEMA_REGISTRY_VOLUME_DIR}/etc` |
| `SCHEMA_REGISTRY_KAFKA_BROKERS` | List of Kafka brokers to connect to. | `nil` |
| `SCHEMA_REGISTRY_ADVERTISED_HOSTNAME` | Advertised hostname in ZooKeeper. | `nil` |
| `SCHEMA_REGISTRY_KAFKA_KEYSTORE_PASSWORD` | Password to access the keystore. | `nil` |
| `SCHEMA_REGISTRY_KAFKA_KEY_PASSWORD` | Password to be able to used ssl secured kafka broker with SR | `nil` |
| `SCHEMA_REGISTRY_KAFKA_TRUSTSTORE_PASSWORD` | Password to access the truststore. | `nil` |
| `SCHEMA_REGISTRY_KAFKA_SASL_USER` | SASL user to authenticate with Kafka. | `nil` |
| `SCHEMA_REGISTRY_KAFKA_SASL_PASSWORD` | SASL password to authenticate with Kafka. | `nil` |
| `SCHEMA_REGISTRY_KAFKASTORE_SECURITY_PROTOCOL` | The security protocol for Kafka connections. | `nil` |
| `SCHEMA_REGISTRY_KAFKASTORE_SASL_MECHANISM` | The SASL mechanism used for Kafka connections. | `nil` |
| `SCHEMA_REGISTRY_KAFKASTORE_SASL_JAAS_CONFIG` | JAAS configuration for Kafka connections. | `nil` |
| `SCHEMA_REGISTRY_KAFKASTORE_SASL_CLIENT_CALLBACK_HANDLER_CLASS` | Callback handlers that obtain username and password from an external source. | `nil` |
| `SCHEMA_REGISTRY_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM` | The endpoint identification algorithm used by clients to validate server host name. | `nil` |
| `SCHEMA_REGISTRY_KAFKASTORE_CLIENT_AUTH_DISABLED` | To disable TLS client authentication when connecting to Kafka. | `nil` |
| `SCHEMA_REGISTRY_LISTENERS` | Comma-separated list of listeners that listen for API requests over either HTTP or HTTPS. | `nil` |
| `SCHEMA_REGISTRY_SSL_KEYSTORE_PASSWORD` | Password to access the SSL keystore. | `nil` |
| `SCHEMA_REGISTRY_SSL_KEY_PASSWORD` | Password to access the SSL key. | `nil` |
| `SCHEMA_REGISTRY_SSL_TRUSTSTORE_PASSWORD` | Password to access the SSL truststore. | `nil` |
| `SCHEMA_REGISTRY_SSL_ENDPOINT_IDENTIFICATION_ALGORITHM` | Endpoint identification algorithm to validate the server hostname using the server certificate. | `nil` |
| `SCHEMA_REGISTRY_CLIENT_AUTHENTICATION` | Client authentication configuration. Valid options: none, requested, over required. | `nil` |
| `SCHEMA_REGISTRY_AVRO_COMPATIBILY_LEVEL` | The Avro compatibility type. Valid options: none, backward, backward_transitive, forward, forward_transitive, full, or full_transitive | `nil` |
| `SCHEMA_REGISTRY_DEBUG` | Enable Schema Registry debug logs. Valid options: true or false | `nil` |

#### Read-only environment variables

Expand Down Expand Up @@ -155,6 +161,22 @@ schema-registry:
- ./truststore.jks:/opt/bitnami/schema-registry/certs/truststore.jks:ro
```

#### IAM authentication for Amazon MSK

This image allows the use of Simple Authentication and Security Layer (SASL) mechanism called `AWS_MSK_IAM`
See [`aws-msk-iam-auth`](https://github.com/aws/aws-msk-iam-auth) for more details.

Here is an example of what environment variables need to be set:

```yaml
SCHEMA_REGISTRY_KAFKA_BROKERS="SASL_SSL://kafka:9098"
SCHEMA_REGISTRY_KAFKASTORE_SECURITY_PROTOCOL="SASL_SSL"
SCHEMA_REGISTRY_KAFKASTORE_SASL_MECHANISM="AWS_MSK_IAM"
SCHEMA_REGISTRY_KAFKASTORE_SASL_JAAS_CONFIG="software.amazon.msk.auth.iam.IAMLoginModule required;"
SCHEMA_REGISTRY_KAFKASTORE_SASL_CLIENT_CALLBACK_HANDLER_CLASS="software.amazon.msk.auth.iam.IAMClientCallbackHandler"
SCHEMA_REGISTRY_KAFKASTORE_CLIENT_AUTH_DISABLED="true"
```

## Using `docker-compose.yaml`

Please be aware this file has not undergone internal testing. Consequently, we advise its use exclusively for development or testing purposes. For production-ready deployments, we highly recommend utilizing its associated [Bitnami Helm chart](https://github.com/bitnami/charts/tree/main/bitnami/schema-registry).
Expand Down
Loading