checksum: Use XXH3 algorithm instead of md5 #1325
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
XXH3 is substantially faster than md5. Test: - Used the Linux source code at v6.2.12 - Had a taskfile configured with all source directories as `sources` - Ran on a AMD Ryzen 3900XT CPU - NVME storage - Host system was Linux Fedora 38 using kernel 6.4.13 - Tests were ran in several configurations: - Full page cache results were gathered by running the test task several times in a row until the results became consistent, indicating that the entire source tree was in the page cache. Then the test was ran 5 more times and those results were averaged. This is a best-case result, most of the time go-task will be close to this result. - Empty page cache results were gathered by running `sync; echo 3 > /proc/sys/vm/drop_caches` after each run in order to empty out the page cache. Each test was ran 5 times and the results averaged. This is a worst case result, such as if the user runs go-task after booting their machine. Results: md5: Full: 184.65ms Empty: 483.34ms XXH3: Full: 90.29ms (-51.1% over md5) Empty: 398.70ms (-17.5% over md5) BLAKE3 (This is another option that has the property of providing cryptographic hashes at a slightly reduced performance over XXH3. Go-task however likely does not benefit from cryptographically secure hashes) Full: 106.21ms (-42.4% over md5) Empty: 407.78ms (-15.6% over md5)
|
I doubt that it matters, but on Linux the binary size increased by 132KB over the md5 binary, or 1.6% (the BLAKE3 binary increased by 103KB instead, or 1.3%). Note also that this change will invalidate all existing checksum files, and so deserves a release note accordingly. |
|
Using a 128KB buffer improved the benchmarks a bit more: XXH3:
|
This seems to perform well against a micro-benchmark reading the Linux kernel, while using an amount of memory that should be acceptable on almost all systems.
andreynering
added
the
type: performance
|
Hi @ReillyBrogan, Great idea, thanks for your contribution! I'll certainly mention this in the change log. This is a small breaking change, and I don't think it's a problem to release as a minor version. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
XXH3 is substantially faster than md5.
Test:
sourcessync; echo 3 > /proc/sys/vm/drop_cachesafter each run in order to empty out the page cache. Each test was ran 5 times and the results averaged. This is a worst case result, such as if the user runs go-task after booting their machine.Results:
md5:
XXH3:
BLAKE3 (This is another option that has the property of providing cryptographic hashes at a slightly reduced performance over XXH3. Go-task however likely does not benefit from cryptographically secure hashes)