Compaction verification enhancements #60

WillemKauf · 2024-11-14T18:11:24Z

Adds new flags and validation enhancements to kgo-verifier:

--tombstone-probability, which allows for random production of tombstone records per the input parameter.

--compacted, which indicates that the topic to be verified is compacted. A warning about gaps in consumed offsets will not be shown if this is true

--validateLatestValues, which if true, performs verification of the consumed keys against the latest values produced by a worker. The log should be fully compacted before the consumer is started if this flag is passed as true.

Allows the `producer_worker` and `repeater_worker` to generate tombstone records.

cmd/kgo-repeater/main.go

WillemKauf · 2024-11-14T18:12:33Z

pkg/worker/verifier/producer_worker.go

@@ -108,7 +109,10 @@ func (pw *ProducerWorker) newRecord(producerId int, sequence int64) *kgo.Record
 		pw.Status.AbortedTransactionMessages += 1
 	}

-	payload := make([]byte, pw.config.messageSize)
+	payload := pw.config.valueGenerator.Generate()


Not sure why valueGenerator wasn't used here before instead of an empty make([]byte)?

nvartolomei

Please make a PR to redpanda and check that it works as expect by pointing kgo at this commit sha before merging this PR. Otherwise if we merge and detect a problem we get a "head of line blocking" on making concurrent changes to kgo.

Suggestion: For testing compacted topics I think we can do more. kgo-verifier should keep track of the keys and and the end checkpoint the expected set of keys and associated values (last values produced for a key minus tombstones). Then during consumption we can track the same and compare at the end.

Or maybe we have a test like this already?

nvartolomei · 2024-11-14T19:55:23Z

pkg/worker/verifier/producer_worker.go

@@ -108,7 +109,10 @@ func (pw *ProducerWorker) newRecord(producerId int, sequence int64) *kgo.Record
 		pw.Status.AbortedTransactionMessages += 1
 	}

-	payload := make([]byte, pw.config.messageSize)
+	payload := pw.config.valueGenerator.Generate()


nvartolomei · 2024-11-14T19:57:55Z

pkg/worker/verifier/validator_status.go

@@ -53,7 +53,7 @@ type ValidatorStatus struct {
 	lastLeaderEpoch map[int32]int32
 }

-func (cs *ValidatorStatus) ValidateRecord(r *kgo.Record, validRanges *TopicOffsetRanges) {
+func (cs *ValidatorStatus) ValidateRecord(r *kgo.Record, validRanges *TopicOffsetRanges, compacted bool) {


shall this be a field on the ValidatorStatus struct instead? it doesn't change for the lifetime entire lifetime anyway why pass it as a param then?

WillemKauf · 2024-11-14T20:44:32Z

Please make a PR to redpanda and check that it works as expect by pointing kgo at this commit sha before merging this PR. Otherwise if we merge and detect a problem we get a "head of line blocking" on making concurrent changes to kgo.

Ack, will do with an upcoming PR.

Suggestion: For testing compacted topics I think we can do more. kgo-verifier should keep track of the keys and and the end checkpoint the expected set of keys and associated values (last values produced for a key minus tombstones). Then during consumption we can track the same and compare at the end.

Yes, this is possible. Will push some follow up commits to this PR.

Offset gaps exist in compacted topics. Suppress a warning log in the read workers if the user passes a `--compacted` flag as a parameter.

WillemKauf · 2024-11-15T15:09:12Z

Force push to:

Move compacted flag to the ValidatorStatus struct
Add LatestValueProduced and LatestValueConsumed to the ProducerWorkerStatus and ValidatorStatus structs respectively. These nested maps track the latest (and expected) key-value pair per partition for compacted topic verification.
Because these are nested map structures, JSON output may grow very large. Alter the String() function for both ProducerWorkerStatus and ValidatorStatus so that they are excluded from output. There will have to be changes on the ducktape side in the redpanda repo to ensure this output isn't in the logs there as well.

See redpanda-data/kgo-verifier#60, which added `--tombstone-probability`, `--compacted` as input parameters, and `LatestValueProduced`/`LatestValueConsumed` as return values in producer/worker status structs.

See redpanda-data/kgo-verifier#60, which added `--tombstone-probability`, `--compacted` as input parameters.

See redpanda-data/kgo-verifier#60, which added `--tombstone-probability`, `--compacted` as input parameters. Also, use `errors='replace'` in `kafka_cat.py`, to avoid UTF-8 decoding issues with randomly generated bytes in `kgo-verifier` records.

See redpanda-data/kgo-verifier#60, which added `--tombstone-probability`, `--compacted` as input parameters.

See redpanda-data/kgo-verifier#60, which added `--tombstone-probability`, `--compacted`, `--validate-latest-values` as input parameters.

For verification of compacted topics, we can track the last expected key-value pair that will be seen after the log has been fully compacted.

As a means to verify the results for a compacted topic. Use the flag `--validate-latest-values` to trigger validation.

See redpanda-data/kgo-verifier#60, which added `--tombstone-probability`, `--compacted`, `--validate-latest-values` as input parameters.

pkg/worker/verifier/producer_worker.go

And use it in the verifier workers. In the case that the topic being consumed from had tombstones produced, the high watermark may be given by a tombstone record that has been removed. In trying to consume until this point, sequential readers will become stuck polling for new records. Persist the last consumable offset in order to adjust the offset we attempt to read up to in the read workers.

There was a race-y access to a map within the producer worker. Add `OnAcked()` to allow for proper access of the producer's lock during reading and writing.

See redpanda-data/kgo-verifier#60, which added `--tombstone-probability`, `--compacted`, `--validate-latest-values` as input parameters.

See redpanda-data/kgo-verifier#60, which added `--tombstone-probability`, `--compacted`, `--validate-latest-values` as input parameters. (cherry picked from commit 85bea2b)

worker: add tombstone_probability as parameter

3ae88aa

Allows the `producer_worker` and `repeater_worker` to generate tombstone records.

WillemKauf requested review from nvartolomei and andrwng November 14, 2024 18:11

WillemKauf commented Nov 14, 2024

View reviewed changes

cmd/kgo-repeater/main.go Show resolved Hide resolved

WillemKauf commented Nov 14, 2024

View reviewed changes

nvartolomei previously approved these changes Nov 14, 2024

View reviewed changes

WillemKauf added 2 commits November 14, 2024 16:56

worker: add compacted parameter

583032e

Offset gaps exist in compacted topics. Suppress a warning log in the read workers if the user passes a `--compacted` flag as a parameter.

worker: add TombstonesProduced and TombstonesConsumed

98b0a59

WillemKauf force-pushed the tombstones branch from b13a42a to 28a4a05 Compare November 15, 2024 15:05

WillemKauf requested a review from nvartolomei November 15, 2024 15:09

WillemKauf mentioned this pull request Nov 15, 2024

[Do Not Merge]: tests: bump kgo-verifier version and support new features redpanda-data/redpanda#24141

Closed

7 tasks

WillemKauf added a commit to WillemKauf/redpanda that referenced this pull request Nov 18, 2024

tests: bump kgo-verifier version and support new features

615d04d

See redpanda-data/kgo-verifier#60, which added `--tombstone-probability`, `--compacted` as input parameters.

WillemKauf force-pushed the tombstones branch from 28a4a05 to 937e857 Compare November 18, 2024 16:09

WillemKauf added a commit to WillemKauf/redpanda that referenced this pull request Nov 18, 2024

tests: bump kgo-verifier version and support new features

7d23c5e

See redpanda-data/kgo-verifier#60, which added `--tombstone-probability`, `--compacted` as input parameters.

WillemKauf changed the title ~~Add --tombstone-probability and --compacted~~ Compaction verification enhancements Nov 18, 2024

WillemKauf force-pushed the tombstones branch from 937e857 to 3d37b2b Compare November 18, 2024 22:13

WillemKauf force-pushed the tombstones branch 2 times, most recently from 0d2c91a to df9a29a Compare November 19, 2024 15:05

WillemKauf added a commit to WillemKauf/redpanda that referenced this pull request Nov 19, 2024

tests: bump kgo-verifier version and support new features

f0a2298

See redpanda-data/kgo-verifier#60, which added `--tombstone-probability`, `--compacted` as input parameters.

WillemKauf added a commit to WillemKauf/redpanda that referenced this pull request Nov 19, 2024

tests: bump kgo-verifier version and support new features

2d94c80

See redpanda-data/kgo-verifier#60, which added `--tombstone-probability`, `--compacted` as input parameters.

WillemKauf added a commit to WillemKauf/redpanda that referenced this pull request Nov 19, 2024

tests: bump kgo-verifier version and support new features

9476f1a

See redpanda-data/kgo-verifier#60, which added `--tombstone-probability`, `--compacted` as input parameters.

WillemKauf added 3 commits November 21, 2024 15:58

worker: add LatestValueProduced

e41fa40

For verification of compacted topics, we can track the last expected key-value pair that will be seen after the log has been fully compacted.

worker: use LatestValueProduced in workers

ca1fdd7

As a means to verify the results for a compacted topic. Use the flag `--validate-latest-values` to trigger validation.

worker: generate valid UTF-8 payloads

90de28d

WillemKauf force-pushed the tombstones branch from ca5a89b to b552221 Compare November 21, 2024 21:04

WillemKauf force-pushed the tombstones branch 2 times, most recently from 5ef813d to 65fa3e8 Compare November 22, 2024 14:39

nvartolomei previously approved these changes Nov 25, 2024

View reviewed changes

pkg/worker/verifier/producer_worker.go Outdated Show resolved Hide resolved

WillemKauf added 2 commits November 25, 2024 09:26

producer_worker: add OnAcked()

300f32e

There was a race-y access to a map within the producer worker. Add `OnAcked()` to allow for proper access of the producer's lock during reading and writing.

WillemKauf dismissed nvartolomei’s stale review via 300f32e November 25, 2024 14:27

WillemKauf force-pushed the tombstones branch from 65fa3e8 to 300f32e Compare November 25, 2024 14:27

nvartolomei approved these changes Nov 25, 2024

View reviewed changes

WillemKauf merged commit 27986ea into redpanda-data:main Nov 25, 2024
2 checks passed

This was referenced Dec 12, 2024

producer_worker: revert use of ValueGenerator::Generate() #63

Merged

rptest: bump kgo-verifier version redpanda-data/redpanda#24558

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Compaction verification enhancements #60

Compaction verification enhancements #60

WillemKauf commented Nov 14, 2024 •

edited

Loading

WillemKauf Nov 14, 2024

nvartolomei Nov 14, 2024

nvartolomei left a comment

nvartolomei Nov 14, 2024

nvartolomei Nov 14, 2024

WillemKauf commented Nov 14, 2024 •

edited

Loading

WillemKauf commented Nov 15, 2024

Compaction verification enhancements #60

Compaction verification enhancements #60

Conversation

WillemKauf commented Nov 14, 2024 • edited Loading

WillemKauf Nov 14, 2024

Choose a reason for hiding this comment

nvartolomei Nov 14, 2024

Choose a reason for hiding this comment

nvartolomei left a comment

Choose a reason for hiding this comment

nvartolomei Nov 14, 2024

Choose a reason for hiding this comment

nvartolomei Nov 14, 2024

Choose a reason for hiding this comment

WillemKauf commented Nov 14, 2024 • edited Loading

WillemKauf commented Nov 15, 2024

WillemKauf commented Nov 14, 2024 •

edited

Loading

WillemKauf commented Nov 14, 2024 •

edited

Loading