[improve][pip] PIP-398: Subscription replication on the namespace and topic levels

[nodece]

Motivation

Enhance the subscription replication.

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

[lhotari]

@nodece would it be possible to cover #23769 feature too? That enables subscription replication for all subscriptions at the broker level if I understand it correctly. Could you work together with @yyj8 to cover that?

[nodece]

@nodece would it be possible to cover #23769 feature too? That enables subscription replication for all subscriptions at the broker level if I understand it correctly. Could you work together with @yyj8 to cover that?

#23769 aims to overwrite the subscription replication state from the consumer, which is different from this PIP. Actually, I also need an overwrite feature, and then the users use the admin API to manage the subscription replication.

The downstream users sometimes enable/disable the subscription replication on the consumer, this lead to the namespace or topic level is ignored.

We can also merge #23769 to this PIP. What's your opinion? @lhotari @yyj8

[lhotari]

@nodece would it be possible to cover #23769 feature too? That enables subscription replication for all subscriptions at the broker level if I understand it correctly. Could you work together with @yyj8 to cover that?

#23769 aims to overwrite the subscription replication state from the consumer, which is different from this PIP. Actually, I also need an overwrite feature, and then the users use the admin API to manage the subscription replication.

The downstream users sometimes enable/disable the subscription replication on the consumer, this lead to the namespace or topic level is ignored.

We can also merge #23769 to this PIP. What's your opinion? @lhotari @yyj8

@nodece @yyj8 Please work together to combine both cases since they are both valuable and useful. I think it would make
sense to combine the work in a single PIP to ensure that we achieve consistency across the changes.

[yyj8]

@nodece would it be possible to cover #23769 feature too? That enables subscription replication for all subscriptions at the broker level if I understand it correctly. Could you work together with @yyj8 to cover that?

#23769 aims to overwrite the subscription replication state from the consumer, which is different from this PIP. Actually, I also need an overwrite feature, and then the users use the admin API to manage the subscription replication.
The downstream users sometimes enable/disable the subscription replication on the consumer, this lead to the namespace or topic level is ignored.
We can also merge #23769 to this PIP. What's your opinion? @lhotari @yyj8

@nodece @yyj8 Please work together to combine both cases since they are both valuable and useful. I think it would make sense to combine the work in a single PIP to ensure that we achieve consistency across the changes.

@nodece would it be possible to cover #23769 feature too? That enables subscription replication for all subscriptions at the broker level if I understand it correctly. Could you work together with @yyj8 to cover that?

#23769 aims to overwrite the subscription replication state from the consumer, which is different from this PIP. Actually, I also need an overwrite feature, and then the users use the admin API to manage the subscription replication.
The downstream users sometimes enable/disable the subscription replication on the consumer, this lead to the namespace or topic level is ignored.
We can also merge #23769 to this PIP. What's your opinion? @lhotari @yyj8

@nodece @yyj8 Please work together to combine both cases since they are both valuable and useful. I think it would make sense to combine the work in a single PIP to ensure that we achieve consistency across the changes.

This is a great idea and suggestion. Can we consider a strategy that divides into three dimensions: cluster dimension, namespace dimension, and topic dimension.
(1)PR #23769 As a cluster dimension strategy, merge it into this PIP #23770;
(2)PR #23769 add the function of dynamically configuring parameter replicateAllSubscriptionState values through the command bin/pulse admin brokers update-dynamic-config --config replicateAllSubscriptionState --value true/false

@nodece @lhotari Can you see if this method is feasible?

[nodece]

@yyj8

cluster dimension, namespace dimension, and topic dimension.

Once the consumer level is configured, it is the highest priority, the cluster, namespace, and topic levels will be ignored, please see #23769 (comment) for details.

(2)PR #23769 add the function of dynamically configuring parameter replicateAllSubscriptionState values through the command bin/pulse admin brokers update-dynamic-config --config replicateAllSubscriptionState --value true/false

This is feasible.

[yyj8]

@yyj8

cluster dimension, namespace dimension, and topic dimension.

Once the consumer level is configured, it is the highest priority, the cluster, namespace, and topic levels will be ignored, please see #23769 (comment) for details.

(2)PR #23769 add the function of dynamically configuring parameter replicateAllSubscriptionState values through the command bin/pulse admin brokers update-dynamic-config --config replicateAllSubscriptionState --value true/false

This is feasible.

The default configuration for the client is replicateSubscriptionState=false. In the scenario of cluster migration, we hope that there is no need for business code modification, and the server will control the replication of the state of all subscriptions in the cluster uniformly. If the configuration priority of the client is the highest, and the client does not display the configuration replicateSubscriptionState=true, this goes against our original intention of not requiring business code modification.

My suggestion is to configure the cluster dimension. If repliceAllSubscriptState=true is enabled, it will be a mandatory configuration to overwrite the client. If repliceAllSubscriptState=false, then use the client's configuration.

[nodece]

@yyj8

The default configuration for the client is replicateSubscriptionState=false.

The latest client defaults to null, please see #23757.

In the scenario of cluster migration, we hope that there is no need for business code modification, and the server will control the replication of the state of all subscriptions in the cluster uniformly. If the configuration priority of the client is the highest, and the client does not display the configuration replicateSubscriptionState=true, this goes against our original intention of not requiring business code modification.
My suggestion is to configure the cluster dimension. If repliceAllSubscriptState=true is enabled, it will be a mandatory configuration to overwrite the client. If repliceAllSubscriptState=false, then use the client's configuration.

I understand your idea, this can make all subscriptions replicated when repliceAllSubscriptState=true, when true, this equals the consumer with replicateSubscriptionState=true, and they have the highest.

However, this idea conflicts with PIP-398, which assumes that the consumer-level configuration for replicateSubscriptionState is not set, and instead, replication behavior is driven by namespace and topic policies. If the replicateSubscriptionState is set, it means that the specified subscription will be replicated, namespace and topic levels cannot change this behavior. If changed, it will result in a breaking change. This is worth considering.

In our case, I don't want to configure the replicateSubscriptionState on the consumer, but the user has been configured, I cannot change the user code. We have the same case, so I also want to ignore the replicateSubscriptionState.

My idea

To combine your case and my case, I suggest introducing a broker configuration for overwriting the consume configuration, and assuming the namespace and topic level are set, the final result will be so like this:

• overwirteConsumerReplicateSubscriptionState=true: The subscription will be replicated.
• overwirteConsumerReplicateSubscriptionState=false: The subscription will be replicated.
• overwirteConsumerReplicateSubscriptionState=null: This means the consumer level is null, and then the broker depends on the namespace and topic level to check the subscription replication.

We can also introduce replicateSubscriptionsState=true/false/empty configuration on the broker level, when the namespace/topic/consumer levels are not set, we use the broker level configuration to check if replicates the subscription.

[yyj8]

@yyj8

The default configuration for the client is replicateSubscriptionState=false.

The latest client defaults to null, please see #23757.

In the scenario of cluster migration, we hope that there is no need for business code modification, and the server will control the replication of the state of all subscriptions in the cluster uniformly. If the configuration priority of the client is the highest, and the client does not display the configuration replicateSubscriptionState=true, this goes against our original intention of not requiring business code modification.
My suggestion is to configure the cluster dimension. If repliceAllSubscriptState=true is enabled, it will be a mandatory configuration to overwrite the client. If repliceAllSubscriptState=false, then use the client's configuration.

I understand your idea, this can make all subscriptions replicated when repliceAllSubscriptState=true, when true, this equals the consumer with replicateSubscriptionState=true, and they have the highest.

However, this idea conflicts with PIP-398, which assumes that the consumer-level configuration for replicateSubscriptionState is not set, and instead, replication behavior is driven by namespace and topic policies. If the replicateSubscriptionState is set, it means that the specified subscription will be replicated, namespace and topic levels cannot change this behavior. If changed, it will result in a breaking change. This is worth considering.

In our case, I don't want to configure the replicateSubscriptionState on the consumer, but the user has been configured, I cannot change the user code. We have the same case, so I also want to ignore the replicateSubscriptionState.

My idea

To combine your case and my case, I suggest introducing a broker configuration for overwriting the consume configuration, and assuming the namespace and topic level are set, the final result will be so like this:

• overwirteConsumerReplicateSubscriptionState=true: The subscription will be replicated.
• overwirteConsumerReplicateSubscriptionState=false: The subscription will be replicated.
• overwirteConsumerReplicateSubscriptionState=null: This means the consumer level is null, and then the broker depends on the namespace and topic level to check the subscription replication.

We can also introduce replicateSubscriptionsState=true/false/empty configuration on the broker level, when the namespace/topic/consumer levels are not set, we use the broker level configuration to check if replicates the subscription.

@nodece I agree with your suggestion.
Before using cluster level configuration, we first check if there are topic and namespace dimensions. If there are none, we apply cluster level configuration; The priority of the three dimensions from high to low is: topic -> namespace -> cluster.

Then we can make unified code adjustments on your pip, or you can directly merge my code into your code, or after you submit the code, I can synchronize your code and modify the cluster level configuration before submitting.

[nodece]

Before using cluster level configuration, we first check if there are topic and namespace dimensions. If there are none, we apply cluster level configuration; The priority of the three dimensions from high to low is: topic -> namespace -> cluster.

@yyj8 Correct. The replicateSubscriptionState is used to auto-create a subscription. When set to false, the broker doesn't persist that to the cursor property in the old version. Therefore, we only need to add replicateSubscriptionState=true/false/null at the cluster level.

Breaking Change: For an existing subscription where replicateSubscriptionState=false is set at the consumer level, the new broker will use the subscription replication policy from the topic/namespace/cluster level.

The new subscription will follow the above priority.

/cc @lhotari