Replies: 2 comments 1 reply
-
max.in.flight is automatically adjusted to 5 when transactions or the idempotent producer is enabled. |
Beta Was this translation helpful? Give feedback.
0 replies
-
Could you point me to where this is set? I cannot find it for the life of me and our librdkafka app also logs
at startup. As it is a producer config, this has to be somewhere in librdkafka itself; is this line of reasoning correct? |
Beta Was this translation helpful? Give feedback.
1 reply
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
According to the Kafka documentation of
enable.idempotence
This is because the leader keeps updating the five last sequence IDs every time a new message is produced. However if more than five messages are inflight, it is possible that messages are duplicated because their sequence ID ran out of cache.
Since apache/kafka#11788 which landed in Kafka 3.2 and was backported to 3.0 and 3.1 a
max.in.fight.request > 5
idempotence is disabled implicitly in the Java client itself:https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java#L545
The librdkafka default is 1.000.000. Looking through old issues I found #831 (comment)
which is older than idempotence and does not appear to have changed since. In a "fire-and-forget" scenario I would arguably prefer setting
acks=0
if data loss is okay or tuningbatch.size
before changingmax.inflight.requests
, but I can agree with the setting. Since idempotence is a thing though, would it be possible to either change the default to 5 or through some kind of error?Beta Was this translation helpful? Give feedback.
All reactions