Durability (Avoiding Message Loss)

Data loss can occur in two ways - brokers losing data due to system failures, and due to retention policies. The former can be addressed by adjusting replication factors, in-sync replica size and producer side synchronization settings. The latter can be addressed by tuning the retention policies and consumer offset reset settings to fail in case of out of range offsets. By default, Kafunk is configured to either prevent data (message) loss or to fail fast in case a potential loss of data has been detected. The applicable Kafunk configuration points are:

• ProducerConfig.requiredAcks = RequiredAcks.AllInSync: this setting works in conjunction with the server side min.insync.replicas setting to determine the number of replicas which acknowledge a produce request. By default, Kafunk requires all in-sync replicas (ISR) to acknowledge a produce request. By default, Kafka configures min.insync.replicas = 1 so you may want to increase this setting. Note that this comes as the cost of increased produce latency since more synchronization is required.

• ConsumerConfig.autoOffsetReset = AutoOffsetReset.TryStartFromCommittedOffsets: this setting controls the behavior of the consumer when it doesn't have offsets to start from and when it detects an out of range offset from a fetch request. The former can happen if the consumer is running for the first time or if the consumer's offsets have been lost due to offset retention. The latter can happen if the consumer is falling behind the topic retention window, or a lagging broker has been elected as the leader for a partition.

• ConsumerConfig.offsetRetentionTime: this setting overrides the server side offset retention setting. This can be used to extend the offset retention window to allow consumers to be offline for longer periods.

The applicable server side configuration points are:

• min.insync.replicas
• offsets.retention.minutes
• offsets.topic.replication.factor
• unclean.leader.election.enable
• log.retention.*
• log.flush.*