[binatmi/schema-registry] Add support for Amazon MSK IAM auth

[bpesics]

Description of the change

Add Amazon MSK IAM authentication support.

Benefits

Allows Schema Registry to use AWS IAM for authentication and authorization against Amazon MSK clusters that have AWS IAM enabled as an authentication mechanism. This is obviously more desired than no authentication or plain text with static credentials.

Possible drawbacks

N/A

Applicable issues

• implements: [bitnami/schema-registry] Support IAM auth way #70769
• related to: [bitnami/schema-registry] Support AWS_MSK_IAM charts#28766

Additional information

As a side issue, this image insisted on mounting a keystore file for any SECURITY_PROTOCOL which includes SSL or SASL, presumably for TLS client authentication.
In order to avoid breaking the existing behaviour I added SCHEMA_REGISTRY_KAFKASTORE_CLIENT_AUTH_DISABLED to provide a way to disable validation when mounting keystore files isn't really necessary.

[migruiz4]

Hi @bpesics,

Thank you very much for your contribution! Before I can accept this feature, I would need to ask you about the following information:

• Could you please provide Schema Registry documentation where support for AWS MSK is mentioned?
• What would be compatibility restrictions between the AWS MSK jar and Schema registry versions?

I'm sorry but without that information, this would need to remain as a customization on users' side.

[migruiz4]

We could accept adding support for the SCHEMA_REGISTRY_KAFKASTORE_* environment variables, but we need the information requested in order to add the AWS_MSK jar file to the source image.

[bpesics]

@migruiz4

Hi @bpesics,

Thank you very much for your contribution! Before I can accept this feature, I would need to ask you about the following information:

• Could you please provide Schema Registry documentation where support for AWS MSK is mentioned?

I think the support of additional authentication mechanisms is an implicit consequence of SASL support.

SASL (Simple Authentication Security Layer) is a framework that provides developers of applications and shared libraries with mechanisms for authentication, data integrity-checking, and encryption.

The SASL mechanism can be selected by kafkastore.sasl.mechanism as documented here. This interface is a native feature of the Java Kafka client used by Schema Registry and other Kafka related Confluent and non-Confluent products. AWS_MSK_IAM implements this interface org.apache.kafka.common.security.auth.AuthenticateCallbackHandler and the library itself is supported by Amazon.

I'd also point out that I think the users of this Bitnami image most probably aren't subscribers of the Confluent platform (otherwise they'd use the image confluentinc/cp-schema-registry) and this change makes this image a lot more useful.

• What would be compatibility restrictions between the AWS MSK jar and Schema registry versions?

As AWS_MSK_IAM depends on Interface AuthenticateCallbackHandler, the current Kafka (client) major version 3.x should provide compatibility for the foreseeable future and there are no other restrictions.

[bpesics]

@migruiz4 @carrodher is there anything more I can do to get this merged?

[michalmisiewicz]

I encountered a similar issue with Google Cloud Managed Kafka and proposed a provider-agnostic solution in #74972. I believe the best way to support additional sasl.mechanism is by providing a configuration file specified in SCHEMA_REGISTRY_CONF_FILE.

[bpesics]

I encountered a similar issue with Google Cloud Managed Kafka and proposed a provider-agnostic solution in #74972. I believe the best way to support additional sasl.mechanism is by providing a configuration file specified in SCHEMA_REGISTRY_CONF_FILE.

@michalmisiewicz Nice, thanks for the heads-up! Your PR addressed what I refer to here as "client authentication" issue.

I encountered a similar issue with Google Cloud Managed Kafka and proposed a provider-agnostic solution in #74972. I believe the best way to support additional sasl.mechanism is by providing a configuration file specified in SCHEMA_REGISTRY_CONF_FILE.

Thanks! Your PR addressed what I referred to here as "client authentication" issue, that should simplify things.

On the other hand, SCHEMA_REGISTRY_CONF_FILE is a read-only variable and the only way to pass in configuration from outside is mounting a file with the same relative path as SCHEMA_REGISTRY_CONF_FILE via SCHEMA_REGISTRY_MOUNTED_CONF_DIR. Using this injected file pretty much disables most of the configuration processing of this image, rendering it not too useful (ie. the config variables are ignored). In addition to that, the Amazon MSK Library still has to be baked in somehow. So I still think adding the support would benefit others.

[github-actions]

This Pull Request has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thank you for your contribution.

[github-actions]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Pull Request. Do not hesitate to reopen it later if necessary.