How does the number of pulsar consumers consume when it exceeds the number of partitions

[kansnow]

for example 3 partion 4 consumer, How to allocate partition consumption

[Loahrs]

Could you further describe your use case?

From my understanding, partitions in pulsar are a detail that is mostly abstracted away for consumers and producers.
You could create a partitioned topic with 3 partitions for persistent://tenant/namespace/topic. Internally that would create 3 topics:

 persistent://tenant/namespace/topic-partition-0
 persistent://tenant/namespace/topic-partition-1
 persistent://tenant/namespace/topic-partition-2

Now you can create a consumer on with topic = persistent://tenant/namespace/topic. This consumer will be able to receive messages in each partition.

If you have a producer that produces to topic = persistent://tenant/namespace/topic (without specyfing any partitions) Pulsar will decide in which partition the message is sent internally. You can also specify the rule for that. The default is a RoundRobin-Style distribution.

So producers and consumers don't have to care about the partitions as this detail is abstracted away by Pulsar.

You could however create a consumer with topic = persistent://tenant/namespace/topic-partition-1 if you want to specifically listen to only that partition.
When there are 2 or more consumers on a topic, they will consume the messages in a RoundRobin way I believe, except you define it another way.

This also depends on the type of subscription used, maybe this is helpful: https://pulsar.apache.org/docs/next/client-libraries-consumers/

I hope that answers your question 🙂

[kansnow]

Thank you very much for answering this question.

In Kafka, if the number of consumers in a consumer group exceeds the number of partitions, some consumers will be idle because each partition can only be consumed by one consumer.

What I am concerned about is the shared subscription model in Pulsar. If there are a total of 3 partitions and 4 consumers, is it possible for 3 consumers to consume a fixed partition and the other 1 to consume data from all partitions in Robin mode? I want to know the internal principles.

[Loahrs]

In Pulsar we have Topics. Topics can be partitioned or non-partitioned. A partitioned Topic is a virtual concept since it can be used in the same way as a non-partitioned Topic. Pulsar will "wrap" the partitions away. (A consumer will be implicitly subscribed to each partition). However, you can decide to explicitly listen to a specific partition.

Consumers don't subscribe to Topics directly, they do it through a Subscription. A Topic can have n - Subscriptions. If a Topic has 2 Subscriptions, Pulsar will send the message to each Subscription simultaneously. Once an acknowledgment of the message was sent for each Subscription, the message will be deleted.

A Subscription has a type (explicit, shared, ...) and depending on that type it can have 1 - n active consumers. A subscription is identified by it's name.

So for your case, where you have a consumer explicitly for each partition and also one that is subscribed to each topic, you could build your consumers like that:

A consumer for each partition explicitly:

consumer1 = client.subscribe("persistent://test/test/test-partition-0", "test-subscription", consumer_type=pulsar.ConsumerType.Shared)

consumer2 = client.subscribe("persistent://test/test/test-partition-1", "test-subscription", consumer_type=pulsar.ConsumerType.Shared)

consumer3 = client.subscribe("persistent://test/test/test-partition-2", "test-subscription", consumer_type=pulsar.ConsumerType.Shared)

And one for the topic, without explicitly stating a partition. Pulsar will implicitly connect this consumer to each partition.
consumer4 = client.subscribe("persistent://test/test/test", "test-subscription", consumer_type=pulsar.ConsumerType.Shared)

Consumer4 will participate in each partition, while the others only react to their partition.
So:
if "Hello" is put into partition-0 then the message will be received by either consumer1 or consumer4 (round-robin style).
if "World" is put into partition-1 then the message will be received by either consumer2 or consumer 4 and so on...

I hope that answers your question 😊

[kansnow]

It's a good answer, concise and clear，think you very much.
I apologize for not watching GitHub during the holiday and only seeing the answer now

[mkueh]

Thanks! For the nice and friendly answer!