🔊 New container images security assessments

[carrodher]

On December 10th, 2024, we released a minor version for every Helm chart in the Bitnami catalog. This minor version introduced a new security mechanism to detect the usage of non-standard containers while installing or upgrading Helm chart releases.

Replacing the original containers that are shipped with the charts will result in installation errors like the one below:

⚠ ERROR: Original containers have been substituted for unrecognized ones. Deploying this chart with non-standard containers is likely to cause degraded security and performance, broken chart features, and missing environment variables.

Unrecognized images:

- [REGISTRY]/[REPOSITORY]:[TAG]
(...)

FAQ

Why did Bitnami introduce this mechanism?

There are several reasons to introduce this change:

1. Bitnami charts are designed, tested, and validated on multiple platforms using a specific set of Bitnami and Tanzu Application Catalog containers. Therefore, Bitnami cannot guarantee the same quality standards levels when the original container images are replaced.
2. Non-standard container images are likely to cause degraded security and performance, broken chart features, and missing environment variables.
3. Incapacity to provide efficient support for non-standard container images given the Bitnami support team's limited bandwidth.

Can I use different image tags?

Yes, replacing default image tags is considered a valid use case. For instance, a user might be interested in using the Bitnami PostgreSQL chart using Bitnami PostgreSQL 15.x container images (note the chart uses 16.x images by default).

That said, it is expected that you see some warnings on the chart notes given that the specific chart & image combination wasn't tested and validated by Bitnami on any platform.

Can I skip the security verification?

Yes, we introduced a new global parameter on every chart that can be used for that purpose, see:

Name	Description	Value
global.security.allowInsecureImages	By default, this chart verifies that the original container images that were designed, tested, and validated are used. This option makes the chart skip the verification step and proceed.	false

At your own risk, you bypass the verification specifying the parameter adding the --set global.security.allowInsecureImages=true argument to helm install or helm upgrade commands or, alternatively, adding the block below to your values YAML file:

global: 
  security: 
    allowInsecureImages: true

Should I expect any issues upgrading my existing chart releases?

No issue is expected if you upgrade your existing releases using original container images. If you replace them, you'll have to skip the security verification.

Should I expect errors consuming Bitnami charts from mirror registries?

Yes. It is expected that you face installation errors while consuming Bitnami chart from mirror registries given the security mechanism will detect original images are not used. To continue using mirror registries, you'll have to skip the security verification.

I built my custom image extending a Bitnami container image, should I expect errors?

Yes. Custom images, even if they're extensions of original Bitnami container images, would be detected by the security mechanism and it is expected to face installation errors. To use your custom image, you'll have to skip the security verification at your own risk.

[robcresswell]

I'll take the bait and ask the obvious question; why was this a minor version release? You've probably broken thousands of dev workflows and pipelines. I understand the why and I appreciate the easy workaround, but I dont understand the version scheme here.

[perfectra1n]

Bitnami: introduces breaking changes
Bitnami: marks as minor version update

[cglacet]

I must be stupid, but eventhough this post is long, I don't understand what I'm supposed to do. So I just ignored the error and set global.security.allowInsecureImages to true. I guess its not so much of an issue because I only use bitnami images in development, but that's a bit confusing.

[robcresswell]

They basically just don't want you to use non-bitnami images in their helm charts because its too annoying to support. This is totally fine and reasonable, but they're masquerading it as both a security issue (very questionable) and a minor version (absolutely not), which is disappointing.

That said, the vast majority of us are probably just using all this for free so... 🤷🏻‍♂️

[ggogel]

It is expected that you face installation errors while consuming Bitnami chart from mirror registries given the security mechanism will detect original images are not used.

Bitnami team, are you aware that pulling through a mirror or remote cache is a standard practice in enterprise environments? Could you clarify what the security mechanism is actually verifying? The repository and image name?

[RangerRick]

Bitnami team, are you aware that pulling through a mirror or remote cache is a standard practice in enterprise environments? Could you clarify what the security mechanism is actually verifying? The repository and image name?

Yeah, we're overriding the images (which values.yaml explicitly supports...) to point to a pull-through proxy, but it is actually the bitnami images still. Seems like extra validation is good idea in general, but the implementation breaks a pretty major usecase which should be supported without having to just turn off validation entirely.

[pofl]

Can confirm that we pull our images through a proxy and our Continuous Deilvery fails now.

[hrobertson]

✋ using a pull-through cache.

Even for those genuinely using different images, that doesn't mean those images are 'insecure'!

1. The only thing being validated here is image names/tags, not checksums or anything more rigorous (which I appreciate is beyond the capability of a Helm chart).
2. global.acceptUnsupportedImagesDisclaimer would have been a far more reasonable key to use than global.security.allowInsecureImages! i.e. this is nothing about security.
3. The message should be a warning, not an error.
4. "Deploying this chart with non-standard containers is likely to cause degraded security and performance, broken chart features, and missing environment variables." sed 's/is likely to/may/'

What sort of review process did this go through?

[portswigger-tim]

Should this not work with public ECR bitnami images too?

EDIT: I see the commercial intent behind this to encourage uptake in Bitnami commercial subscriptions ❤️ - luckily I use shared Github Action workflows, so only needed to --set global.security.allowInsecureImages=true in a couple of places 😸

[carrodher]

Thanks for your feedback. We hear your comments and suggestions, and as always, will take them into account for future changes. We work to make this change as simple and smooth as possible, with an easy-to-follow approach to bypass the error.

Why did we make this a minor version release?

Bitnami's policy for versioning Helm charts is to release a major version for anything that could break the service upgrade or persistent data:

• The major version of the application
• A configuration change in the application (e.g., configuration parameter) that breaks compatibility with the previously deployed configuration.

The supply chain integrity check only affects an optional property of the chart for users who are overwriting the path of the images. It does not affect the previously deployed application during the upgrade; the error is thrown when evaluating the template.

Why we introduced a supply chain integrity check

The Helm charts have included a warning since May ‘24. We have seen multiple instances where users substituting non-Bitnami or modified images experienced serious security problems, which is why we introduced this check and a simple workaround that ensures awareness when non-standard images are substituted.

Commercial Bitnami images are available for business use cases.

We made this change concurrently with the release of our commercial product, Bitnami Premium. Bitnami Premium raises the security level for Bitnami Premium and Tanzu Application Catalog with SLSA level 3 in-toto attestations and cryptographic signatures for images and charts. Those security improvements make it easier to mirror or cache commercial Bitnami containers and Helm charts without setting a flag during deployment. You can read more in our blog post.

[perfectra1n]

Bitnami's policy for versioning Helm charts is to release a major version for anything that could break the service upgrade

A configuration change in the application (e.g., configuration parameter) that breaks compatibility with the previously deployed configuration.

This change breaks the service upgrade, as the the Helm chart's templates no longer complete evaluation when using values that previously compatible with the deployed configuration. This isn't a minor version update of the Helm chart.

[ggogel]

We made this change concurrently with the release of our commercial product, Bitnami Premium.

It appears that this breaking change was introduced to align with, and potentially promote, your new commercial offering rather than serving a purely technical or user-driven purpose.

We have seen multiple instances where users substituting non-Bitnami or modified images experienced serious security problems

Claiming that this change enhances security is not entirely accurate. Using the default image does not inherently guarantee the integrity of the deployed image, as true verification can only be achieved by validating its hash. Likewise, using a different image does not necessarily result in a less secure deployment. I therefore see the name of this parameter as counterintuitive and misleading.

[viceice]

what about the official public.ecr.aws/bitnami/* images? they should also be accepted. 🤔