Failure to add an external Hidden instance to an externally accessed ReplicaSet

[bslpzk]

Name and Version

bitnami/mongodb 13.10.2

What architecture are you using?

None

What steps will reproduce the bug?

On two separate clusters with external IPs, run replicaset and hidden mongodb instance.

Add a hidden mongodb instance from one cluster to the replicaset of the other cluster.

Are you using any custom parameters or values?

## First_k8s_cluster_values.yaml
arbiter:
  enabled: false
architecture: replicaset
auth:
  enabled: true
  replicaSetKey: bIaAS0ZRNR8ODFd5Hs81suNdxkpuBacF2izTfGs9
  rootPassword: woohahnohdish5oHia4aak3agaeyeim1
externalAccess:
  enabled: true
  service:
    loadBalancerIPs:
      - <Firs_cluster_ip>
    portName: mongodb
    ports:
      mongodb: 27017
    type: LoadBalancer
image:
  debug: true
persistence:
  enabled: true
  size: 1Gi
rbac:
  create: true
replicaCount: 1
replicaSetConfigurationSettings:
  enabled: false
replicaSetName: rs0
tls:
  enabled: false

## Second_k8s_cluster_values.yaml
arbiter:
  enabled: false
architecture: replicaset
auth:
  enabled: true
  replicaSetKey: bIaAS0ZRNR8ODFd5Hs81suNdxkpuBacF2izTfGs9
  rootPassword: woohahnohdish5oHia4aak3agaeyeim1
externalAccess:
  hidden:
    enabled: true
    service:
      loadBalancerIPs:
        - <Second_k8s_cluster_IP>
      loadBalancerSourceRanges:
        - <admin_static_ip>/32
        - <firs_cluster_ip>/32
      portName: mongodb
      ports:
        mongodb: 27017
      type: LoadBalancer
hidden:
  containerPorts:
    mongodb: 27017
  enabled: true
  persistence:
    accessModes:
      - ReadWriteOnce
    enabled: true
    size: 1Gi
  replicaCount: 1
  service:
    portName: mongodb
    ports:
      mongodb: 27017
image:
  debug: true
persistence:
  enabled: true
  size: 1Gi
rbac:
  create: true
replicaCount: 0
replicaSetName: rs0
tls:
  enabled: false

What is the expected behavior?

To be able to run a hidden instance to connect to a replicaset located in another cluster via external access

What do you see instead?

N/A

Additional information

Hello colleagues,

I'm having difficulties adding a hidden MongoDB instance from another Kubernetes cluster while maintaining external access.

The initial goal was simple: add another hidden MongoDB instance to an existing replica set running in Azure AKS. This new instance would be located in a different AKS cluster and serve as a read-only copy.

The inter-cluster communication via IP is established. However, when deploying the bitnami/mongodb chart on each cluster, two separate replica sets are created, even if the names are the same. Consequently, when attempting to execute rs.add("<second_cluster_ip>:27017"), an error occurs due to different replica set IDs. These IDs are generated upon initialization of the primary MongoDB instance in each cluster and cannot be changed. Here's an example log entry from heartbeat:

lastHeartbeatMessage: "replica set IDs do not match, ours: 645d3dbb64c2adc0dc1891c7; remote node's: 645d3395c20febe7d45240c9"

Similarly, the methods to add initdbScripts do not work, because they run after the replicaset is initialized.

If I try to deploy the chart on the second cluster with replicaCount: 0 and hidden.replicaCount: 1, the replica set initialization is skipped, but the hidden replica profile is not configured either. Attempting to manually adjust its settings via the console results in the following expected error:

use admin
db.createUser({user: 'root', pwd: passwordPrompt(), roles: ["root"]})

MongoServerError: not primary

I need help to solve this problem and achieve the desired configuration. I appreciate any advice from those who can assist me. Thank you.

[carrodher]

It seems a very specific use case difficult to reproduce on our side and very tied to your scenario.

For information regarding the application itself, customization of the content within the application, or questions about the use of the technology or infrastructure; we highly recommend checking forums and user guides made available by the project behind the application or the technology.

That said, we will keep this ticket open until the stale bot closes it just in case someone from the community adds some valuable info.

[dtrts]

Hi @bslpzk

I have also been looking at something similar when it comes to deploying nodes using a new chart but wanting them to attach to an existing replica set.

From my understanding your hidden node is running through initialization whilst attempting to register with a primary node which has a DNS address relative to the hidden chart, and not your original chart.

I would look at updating the hidden chart to overwrite the environment variable: MONGODB_INITIAL_PRIMARY_HOST to point towards the first charts original pimary host.

It will be used in this function: mongodb_configure_hidden

---

I have also found a workaround if you want to add all nodes, and not just hidden nodes, to an existing cluster.

The description is buried in this issue: #16719
Search for "BONUS ISSUE".

The idea is to edit the setup.sh script which is defined in replicaset/scripts-configmap.yaml #Line 119

There is a line where it searched for a current primary. If you're able to alter this search to target the host names of your existing cluster then it will add itself to that cluster.

It might still be worth overwriting MONGODB_INITIAL_PRIMARY_HOST here which will prevent the -0 node from trying to configure a new replica set and set itself as primary.

[bslpzk]

Hi @dtrts.
Thank you for your kind assistance.

In fact, I also moved away from the idea of a hidden node and added a normal secondary node in the same way - using variables from the initialization scripts. And to change replicaset configuration it's convenient to use initdbScriptsConfigMap value, by putting into the configmap the changed replicaset configuration as rs-config.js I managed to get the desired result.

[github-actions]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[github-actions]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.