Durability (Avoiding Message Loss)

By default, Kafunk is configured to either prevent data (message) loss or to fail fast in case a potential loss of data has been detected. The applicable Kafunk configuration points are:

• ProducerConfig.requiredAcks = RequiredAcks.AllInSync: this setting works in conjunction with the server side min.insync.replicas setting to determine the number of replicas which acknowledge a produce request. By default, Kafunk requires all in-sync replicas (ISR) to acknowledge a produce request. By default, Kafka configures min.insync.replicas = 1 so you may want to increase this setting. Note that this comes as the cost of increased produce latency since more synchronization is required.

• ConsumerConfig.autoOffsetReset = AutoOffsetReset.TryStartFromCommittedOffsets: this setting controls the behavior of the consumer when it doesn't have offsets to start from and when it detects an out of range offset from a fetch request. The former can happen if the consumer is running for the first time or if the consumer's offsets have been lost due to offset retention. The latter can happen if the consumer is falling behind the topic retention window, or a lagging broker has been elected as the leader for a partition.

• ConsumerConfig.offsetRetentionTime: this setting overrides the server side offset retention setting. This can be used to extend the offset retention window to allow consumers to be offline for longer periods.

The applicable server side configuration points are:

• min.insync.replicas
• offsets.retention.minutes
• offsets.topic.replication.factor
• unclean.leader.election.enable
• log.retention.*
• log.flush.*