[bitnami/elasticsearch] Enabling readOnlyFileSystem with a multi-node elasticsearch cluster fails on startup

[amitonlentra]

Name and Version

bitnami/elasticsearch: 19.13.15

What architecture are you using?

amd64

What steps will reproduce the bug?

We are deploying elasticsearch v8.9.2 through zeebe (open source workflow engine from camunda team). The helm chart used is version: 19.13.15 by camunda.

To tighten security our requirement is to run elasticsearch with minimal privileges and on read only file system. We configured it with the below helm config but the deployment fails with an startup error "Exception in thread "main" java.nio.file.FileSystemException: /opt/bitnami/elasticsearch/config/elasticsearch.keystore.tmp: Read-only file system". The complete stacktrace is shared below.

We looked into a few previously reported issues but could not find a solution. Kindly suggest a solution that works.

Are you using any custom parameters or values?

Below is the helm config to tighten security.

elasticsearch:
  enabled: true
  metrics:
    enabled: true
    resources:
      requests:
        cpu: 500m
        memory: 512Mi
      limits:
        cpu: 1
        memory: 512Mi
    containerSecurityContext:
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
      seccompProfile:
        type: RuntimeDefault
      runAsNonRoot: true
  master:
    masterOnly: false
    replicaCount: 2
    containerSecurityContext:
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
      seccompProfile:
        type: RuntimeDefault
      runAsNonRoot: true
  image:
    repository: "bitnami/elasticsearch"
    tag: "8.9.2"
 resources:
    requests:
      cpu: 1.5
      memory: 2Gi
    limits:
      cpu: 1.5
      memory: 2Gi
  volumeClaimTemplate:
    accessModes: ["ReadWriteOnce"]
    storageClassName: "local-path"
    resources:
      requests:
        storage: "20Gi"

What is the expected behavior?

Elasticsearch should be deployed and executed without errors

What do you see instead?

Complete stacktrace

Defaulted container "elasticsearch" out of: elasticsearch, sysctl (init)
elasticsearch 12:35:18.05 
elasticsearch 12:35:18.06 Welcome to the Bitnami elasticsearch container
elasticsearch 12:35:18.06 Subscribe to project updates by watching https://github.com/bitnami/containers
elasticsearch 12:35:18.07 Submit issues and feature requests at https://github.com/bitnami/containers/issues
elasticsearch 12:35:18.07 
elasticsearch 12:35:18.07 INFO  ==> ** Starting Elasticsearch setup **
elasticsearch 12:35:18.17 WARN  ==> Setting ELASTICSEARCH_IS_DEDICATED_NODE is disabled.
elasticsearch 12:35:18.18 WARN  ==> ELASTICSEARCH_NODE_ROLES will be ignored and Elasticsearch will asume all different roles.
elasticsearch 12:35:18.19 INFO  ==> Configuring/Initializing Elasticsearch...
elasticsearch 12:35:18.27 INFO  ==> Custom configuration file detected, using it...
elasticsearch 12:35:18.27 WARN  ==> The JVM options configuration files are not writable. Configurations based on environment variables will not be applied
elasticsearch 12:35:18.36 INFO  ==> ** Elasticsearch setup finished! **

elasticsearch 12:35:18.41 INFO  ==> ** Starting Elasticsearch **
warning: ignoring JAVA_HOME=/opt/bitnami/java; using ES_JAVA_HOME
Exception in thread "main" java.nio.file.FileSystemException: /opt/bitnami/elasticsearch/config/elasticsearch.keystore.tmp: Read-only file system
    at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)
    at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
    at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
    at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:218)
    at java.base/java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:484)
    at java.base/java.nio.file.Files.newOutputStream(Files.java:228)
    at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:394)
    at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:387)
    at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:220)
    at org.elasticsearch.common.settings.KeyStoreWrapper.save(KeyStoreWrapper.java:438)
    at org.elasticsearch.common.settings.KeyStoreWrapper.save(KeyStoreWrapper.java:428)
    at org.elasticsearch.common.settings.KeyStoreWrapper.bootstrap(KeyStoreWrapper.java:233)
    at org.elasticsearch.server.cli.KeyStoreLoader.bootstrap(KeyStoreLoader.java:37)
    at org.elasticsearch.server.cli.ServerCli.execute(ServerCli.java:88)
    at org.elasticsearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:54)
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:85)
    at org.elasticsearch.cli.Command.main(Command.java:50)
    at org.elasticsearch.launcher.CliToolLauncher.main(CliToolLauncher.java:64)

[carrodher]

Hi, the issue may not be directly related to the Bitnami container image/Helm chart, but rather to how the application is being utilized, configured in your specific environment, or tied to a particular scenario that is not easy to reproduce on our side.

If you think that's not the case and want to contribute a solution, we'd like to invite you to create a pull request. The Bitnami team is excited to review your submission and offer feedback. You can find the contributing guidelines here.

Your contribution will greatly benefit the community. Please feel free to contact us if you have any questions or need assistance.

Suppose you have any questions about the application, customizing its content, or technology and infrastructure usage. In that case, we highly recommend that you refer to the forums and user guides provided by the project responsible for the application or technology.

With that said, we'll keep this ticket open until the stale bot automatically closes it, in case someone from the community contributes valuable insights.

[amitonlentra]

Thanks @carrodher for the reply. I have a follow up question

Hi, the issue may not be directly related to the Bitnami container image/Helm chart, but rather to how the application is being utilized, configured in your specific environment, or tied to a particular scenario that is not easy to reproduce on our side.

I am not sure if I follow you completely.

1. Since these attributes readOnlyRootFilesystem, allowPrivilegeEscalation are defined by the bitnami helm chart, I am assuming they should be tested and supported by the bitnami helm chart team. Please correct me if my understanding is incorrect.
2. There is no specific environment or scenario needed to reproduce this. If you just use the above config, you would be able to see the error.

[javsalgar]

Hi,

I believe the issue has to do with the bitnami/elasticsearch image you are using. ReadOnlyRootFilesystem support was added in the 8.12.2 image, so the 8.9.2 does not have it. You will need to either use a newer Elasticsearch image or disable readOnlyRootFilesystem and set runAsGroup: 0.

[amitonlentra]

@javsalgar - can you please share the link that states that ReadOnlyRootFilesystem support was added in elasticsearch version 8.12.2? I tried searching but I couldn't find it. I looked into the release notes here but they don't mention about this support either

[javsalgar]

Hi,

The support of readOnlyRootFilesystem was added not by the upstream project, but in how we changed our bitnami/elasticsearch container. And it was in that version where we added that support

f499109

[amitonlentra]

Ok so just to confirm my understanding - we need to use bitnami/elasticsearch chart version > 20.0.0 to resolve the issue instead of 19.13.15 currently used, right?

[carrodher]

Yes

[amitonlentra]

I tried with chart version 20.0.0 but the copy-default-plugins init container is crashing with an error /opt/bitnami/scripts/libfs.sh: line 61: 1: missing directory

Could you guide me if something in the config (below) is incorrect?

My elasticsearch helm config is as below

  metrics:
    enabled: true
    resources:
      requests:
        cpu: 500m
        memory: 512Mi
      limits:
        cpu: 1
        memory: 512Mi
    containerSecurityContext:
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
      seccompProfile:
        type: RuntimeDefault
      runAsNonRoot: true
  master:
    masterOnly: false
    replicaCount: 2
    containerSecurityContext:
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
      seccompProfile:
        type: RuntimeDefault
      runAsNonRoot: true
  image:
    repository: "bitnami/elasticsearch"
    tag: "8.9.2"
  extraVolumes:
    - name: custom-log-config
      configMap:
        name: elasticsearch-log-config   
  extraVolumeMounts:
    - name: custom-log-config
      mountPath: /opt/bitnami/elasticsearch/config/log4j2.properties
      subPath: log4j2.properties
  resources:
    requests:
      cpu: 1.5
      memory: 2Gi
    limits:
      cpu: 1.5
      memory: 2Gi
  volumeClaimTemplate:
    accessModes: ["ReadWriteOnce"]
    storageClassName: "local-path"
    resources:
      requests:
        storage: "20Gi"

[carrodher]

Hi, the issue may not be directly related to the Bitnami container image/Helm chart, but rather to how the application is being utilized, configured in your specific environment, or tied to a particular scenario that is not easy to reproduce on our side.

If you think that's not the case and want to contribute a solution, we'd like to invite you to create a pull request. The Bitnami team is excited to review your submission and offer feedback. You can find the contributing guidelines here.

Your contribution will greatly benefit the community. Please feel free to contact us if you have any questions or need assistance.

Suppose you have any questions about the application, customizing its content, or technology and infrastructure usage. In that case, we highly recommend that you refer to the forums and user guides provided by the project responsible for the application or technology.

With that said, we'll keep this ticket open until the stale bot automatically closes it, in case someone from the community contributes valuable insights.

[github-actions]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[github-actions]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.