[bitnami/redis-cluster] BUG: not working with healthcheck

[apachler]

Name and Version

bitnami/redis-cluster:7.2

What architecture are you using?

amd64

What steps will reproduce the bug?

Activating HEALTHCHECK in Docker Compose will stop container from working.

    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 10s
      timeout: 30s
      retries: 10
      start_period: 30s

What is the expected behavior?

Container is starting and working as expected.

What do you see instead?

The container's initialization stops at Setting Redis config file or while Changing old IP.

redis-cluster 07:37:14.77 INFO  ==>
redis-cluster 07:37:14.77 INFO  ==> Welcome to the Bitnami redis-cluster container
redis-cluster 07:37:14.77 INFO  ==> Subscribe to project updates by watching https://github.com/bitnami/containers
redis-cluster 07:37:14.78 INFO  ==> Submit issues and feature requests at https://github.com/bitnami/containers/issues
redis-cluster 07:37:14.78 INFO  ==> Upgrade to Tanzu Application Catalog for production environments to access custom-configured and pre-packaged software components. Gain enhanced features, including Software Bill of Materials (SBOM), CVE scan result reports, and VEX documents. To learn more, visit https://bitnami.com/enterprise
redis-cluster 07:37:14.78 INFO  ==>
redis-cluster 07:37:14.79 INFO  ==> ** Starting Redis setup **
redis-cluster 07:37:14.82 INFO  ==> Initializing Redis
redis-cluster 07:37:14.86 INFO  ==> Setting Redis config file
redis-cluster 07:37:14.99 INFO  ==> Changing old IP xxx by the new one yyy
redis-cluster 07:39:31.31 INFO  ==>

[javsalgar]

Hi,

Maybe the startup process is slower. Could you try increasing the number of retries? Also, could you try setting the BITNAMI_DEBUG=true environment variable?

[apachler]

Thanks for answering. I already have increased the retries to 1000. The container is not killed but did not finish startup.
Sure, i will give the BITNAMI_DEBUG=true env var a try and report back ...

[apachler]

Started the container with debug enabled with following logs

redis-cluster 08:29:37.85 INFO  ==>
redis-cluster 08:29:37.85 INFO  ==> Welcome to the Bitnami redis-cluster container
redis-cluster 08:29:37.86 INFO  ==> Subscribe to project updates by watching https://github.com/bitnami/containers
redis-cluster 08:29:37.86 INFO  ==> Submit issues and feature requests at https://github.com/bitnami/containers/issues
redis-cluster 08:29:37.86 INFO  ==> Upgrade to Tanzu Application Catalog for production environments to access custom-configured and pre-packaged software components. Gain enhanced features, including Software Bill of Materials (SBOM), CVE scan result reports, and VEX documents. To learn more, visit https://bitnami.com/enterprise
redis-cluster 08:29:37.86 INFO  ==>
redis-cluster 08:29:37.87 DEBUG ==> Copying files from /opt/bitnami/redis/etc.default to /opt/bitnami/redis/etc
redis-cluster 08:29:37.88 INFO  ==> ** Starting Redis setup **
redis-cluster 08:29:37.91 DEBUG ==> Validating settings in REDIS_* env vars..
redis-cluster 08:29:37.92 INFO  ==> Initializing Redis
redis-cluster 08:29:37.93 DEBUG ==> Ensuring expected directories/files exist
redis-cluster 08:29:37.96 INFO  ==> Setting Redis config file

It seems that the container is not able to finish initialization as soon as a healthcheck is defined...

[javsalgar]

Hi,

This is strange. This is the readiness command we execute in our side. But it's true that we provide an initialDelay, which would explain that there are no conflicts.

bitnami/charts@f499109

[apachler]

Ok, so is there any chance to get that fixed?

[jotamartos]

Hi @apachler,

Sorry for the late response here. I just tried to configure the healthcheck in the Bitnami Redis Cluster solution and everything worked as expected. I configured the same code you pasted above in all containers, and they all ran as expected.

➜  redis-cluster git:(main) docker ps -a 
CONTAINER ID   IMAGE                                                                      COMMAND                  CREATED              STATUS                        PORTS      NAMES
a1ef3b7ca749   bitnami/redis-cluster:7.4                                                  "/opt/bitnami/script…"   About a minute ago   Up About a minute (healthy)   6379/tcp   redis-cluster-redis-node-5-1
db572426a9fc   bitnami/redis-cluster:7.4                                                  "/opt/bitnami/script…"   About a minute ago   Up About a minute (healthy)   6379/tcp   redis-cluster-redis-node-4-1
f836dede68c5   bitnami/redis-cluster:7.4                                                  "/opt/bitnami/script…"   About a minute ago   Up About a minute (healthy)   6379/tcp   redis-cluster-redis-node-3-1
c6f70a306e67   bitnami/redis-cluster:7.4                                                  "/opt/bitnami/script…"   About a minute ago   Up About a minute (healthy)   6379/tcp   redis-cluster-redis-node-2-1
f5dfb99d0bf6   bitnami/redis-cluster:7.4                                                  "/opt/bitnami/script…"   About a minute ago   Up About a minute (healthy)   6379/tcp   redis-cluster-redis-node-0-1
fb9f96c198ad   bitnami/redis-cluster:7.4                                                  "/opt/bitnami/script…"   About a minute ago   Up About a minute (healthy)   6379/tcp   redis-cluster-redis-node-1-1

➜  redis-cluster git:(main) docker inspect --format "{{json .State.Health }}" a1ef3b7ca749 | jq

{
  "Status": "healthy",
  "FailingStreak": 0,
  "Log": [
    {
      "Start": "2024-12-23T07:59:07.844566709Z",
      "End": "2024-12-23T07:59:07.87803975Z",
      "ExitCode": 0,
      "Output": "PONG\n"
    },
    {
      "Start": "2024-12-23T07:59:17.87957313Z",
      "End": "2024-12-23T07:59:17.91265688Z",
      "ExitCode": 0,
      "Output": "PONG\n"
    },
    {
      "Start": "2024-12-23T07:59:27.917526968Z",
      "End": "2024-12-23T07:59:27.949895468Z",
      "ExitCode": 0,
      "Output": "PONG\n"
    },
    {
      "Start": "2024-12-23T07:59:37.953836959Z",
      "End": "2024-12-23T07:59:37.982182167Z",
      "ExitCode": 0,
      "Output": "PONG\n"
    },
    {
      "Start": "2024-12-23T07:59:47.982666422Z",
      "End": "2024-12-23T07:59:48.011520547Z",
      "ExitCode": 0,
      "Output": "PONG\n"
    }
  ]
}

➜  redis-cluster git:(main) docker logs a1ef3b7ca749
redis-cluster 07:57:52.60 INFO  ==> 
redis-cluster 07:57:52.61 INFO  ==> Welcome to the Bitnami redis-cluster container
redis-cluster 07:57:52.61 INFO  ==> Subscribe to project updates by watching https://github.com/bitnami/containers
redis-cluster 07:57:52.61 INFO  ==> Submit issues and feature requests at https://github.com/bitnami/containers/issues
redis-cluster 07:57:52.61 INFO  ==> Upgrade to Tanzu Application Catalog for production environments to access custom-configured and pre-packaged software components. Gain enhanced features, including Software Bill of Materials (SBOM), CVE scan result reports, and VEX documents. To learn more, visit https://bitnami.com/enterprise
redis-cluster 07:57:52.61 INFO  ==> 
redis-cluster 07:57:52.61 INFO  ==> ** Starting Redis setup **
redis-cluster 07:57:52.63 INFO  ==> Initializing Redis
redis-cluster 07:57:52.64 INFO  ==> Setting Redis config file
Storing map with hostnames and IPs
redis-cluster 07:57:52.70 INFO  ==> ** Redis setup finished! **

121:C 23 Dec 2024 07:57:52.718 # WARNING: Changing databases number from 16 to 1 since we are in cluster mode
121:C 23 Dec 2024 07:57:52.719 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
121:C 23 Dec 2024 07:57:52.719 * Redis version=7.4.1, bits=64, commit=00000000, modified=1, pid=121, just started
121:C 23 Dec 2024 07:57:52.719 * Configuration loaded
121:M 23 Dec 2024 07:57:52.719 * monotonic clock: POSIX clock_gettime
121:M 23 Dec 2024 07:57:52.720 * Running mode=cluster, port=6379.
121:M 23 Dec 2024 07:57:52.720 * No cluster configuration found, I'm 19f759c2d60cc55992a436b9dd36693a995aaf6f
121:M 23 Dec 2024 07:57:52.722 * Server initialized
121:M 23 Dec 2024 07:57:52.723 * Creating AOF base file appendonly.aof.1.base.rdb on server start
121:M 23 Dec 2024 07:57:52.725 * Creating AOF incr file appendonly.aof.1.incr.aof on server start
121:M 23 Dec 2024 07:57:52.725 * Ready to accept connections tcp

This is strange. This is the readiness command we execute in our side. But it's true that we provide an initialDelay, which would explain that there are no conflicts.

As @javsalgar mentioned above, you can try increasing the delay (start_period parameter) and see if that solves the issue. You can find more information here

[apachler]

Thanks for sharing your experience.

I already tried to increase the start_period but it does not help since docker is executing the healthcheck command even within the start_period but does not count negative results.

I have seen that sometimes the container finished initalization with a configured healtcheck but fails again on the next startup...

[jotamartos]

Hi @apachler,

The issue seems to be related to your environment. I just launched a fresh instance in the cloud, and got the same results than before (in my local M1-based Mac OS X machine)

admin@ip-172-31-54-137:~$ docker ps -a 
CONTAINER ID   IMAGE                       COMMAND                  CREATED         STATUS                   PORTS      NAMES
c4615b477931   bitnami/redis-cluster:7.4   "/opt/bitnami/script…"   2 minutes ago   Up 2 minutes (healthy)   6379/tcp   admin-redis-node-5-1
d595a8b47014   bitnami/redis-cluster:7.4   "/opt/bitnami/script…"   2 minutes ago   Up 2 minutes (healthy)   6379/tcp   admin-redis-node-1-1
4ad42d509ade   bitnami/redis-cluster:7.4   "/opt/bitnami/script…"   2 minutes ago   Up 2 minutes (healthy)   6379/tcp   admin-redis-node-3-1
20d4fcc8cebf   bitnami/redis-cluster:7.4   "/opt/bitnami/script…"   2 minutes ago   Up 2 minutes (healthy)   6379/tcp   admin-redis-node-2-1
e86dd6967903   bitnami/redis-cluster:7.4   "/opt/bitnami/script…"   2 minutes ago   Up 2 minutes (healthy)   6379/tcp   admin-redis-node-4-1
12d468c0a7ca   bitnami/redis-cluster:7.4   "/opt/bitnami/script…"   2 minutes ago   Up 2 minutes (healthy)   6379/tcp   admin-redis-node-0-1
admin@ip-172-31-54-137:~$ docker inspect --format "{{json .State.Health }}" c4615b477931
{"Status":"healthy","FailingStreak":0,"Log":[{"Start":"2024-12-23T08:56:47.486660655Z","End":"2024-12-23T08:56:47.54162141Z","ExitCode":0,"Output":"PONG\n"},{"Start":"2024-12-23T08:56:57.542747068Z","End":"2024-12-23T08:56:57.617571569Z","ExitCode":0,"Output":"PONG\n"},{"Start":"2024-12-23T08:57:07.618206925Z","End":"2024-12-23T08:57:07.678010316Z","ExitCode":0,"Output":"PONG\n"},{"Start":"2024-12-23T08:57:17.679746117Z","End":"2024-12-23T08:57:17.797774734Z","ExitCode":0,"Output":"PONG\n"},{"Start":"2024-12-23T08:57:27.799175018Z","End":"2024-12-23T08:57:27.883858861Z","ExitCode":0,"Output":"PONG\n"}]}

[apachler]

I am running the containers on Docker Swarm, maybe that causes the issue?

[jotamartos]

That's probably it. Can you try without using Docker Swarm?