removeItem performance
#2933
Comments
|
Thanks for filling this! This is definitely something that we would take a patch for and parallelise the removal similar to how we did for copying in #2806. |
FranzBusch
added
kind/enhancement
This PR refactors the logic to remove items. Instead of sequentially
discovering and deleting items one by one, we concurrently traverse all
directories that we find and then sequentially issue file deletions.
This new deletion logic is currently unbounded, and will spawn a bunch
of tasks no matter the underlying system.
Benchmark using `--configration=release` for both the `main` and
`parallel-removal` branches:
```
$ rm -rf .build && git switch main && swift build --configuration=release && cp .build/release/NIOTestCLI NIOTestCLI_main
[...]
Building for production...
[134/134] Linking NIOPerformanceTester
Build complete! (32.65s)
$ rm -rf .build && git switch parallel-removal && swift build --configuration=release
[...]
Building for production...
[134/134] Linking NIOPerformanceTester
Build complete! (32.19s)
$ hyperfine --max-runs 10 --prepare "rm -rf /tmp/swift-nio && cp -r $(pwd) /tmp/swift-nio" "./NIOTestCLI_main /tmp/swift-nio" ".build/release/NIOTestCLI /tmp/swift-nio" "rm -rf /tmp/swift-nio"
Benchmark 1: ./NIOTestCLI_main /tmp/swift-nio
Time (mean ± σ): 3.019 s ± 0.032 s [User: 0.601 s, System: 11.123 s]
Range (min … max): 2.985 s … 3.072 s 10 runs
Benchmark 2: .build/release/NIOTestCLI /tmp/swift-nio
Time (mean ± σ): 2.295 s ± 0.065 s [User: 0.673 s, System: 7.677 s]
Range (min … max): 2.212 s … 2.370 s 10 runs
Benchmark 3: rm -rf /tmp/swift-nio
Time (mean ± σ): 1.517 s ± 0.037 s [User: 0.015 s, System: 1.442 s]
Range (min … max): 1.480 s … 1.590 s 10 runs
Summary
rm -rf /tmp/swift-nio ran
1.51 ± 0.06 times faster than .build/release/NIOTestCLI /tmp/swift-nio
1.99 ± 0.05 times faster than ./NIOTestCLI_main /tmp/swift-nio
```
Resolves apple#2933
|
@pepicrft I've raised a PR (#3008) to address the issue of handling removal of items sequentially. I seem to be unable to reproduce your results though. The existing logic in If you still have your test setup, would you mind sharing that, so I can try and reproduce that? |
Thanks a lot, @mimischi 🙏🏼. I unfortunately don't have the setup anymore, but I remember it was a directory containing all the dependencies of a project that had been resolved by the SPM. Perhaps you can try with a project that has many dependencies |
This PR refactors the logic to remove items. Instead of sequentially
discovering and deleting items one by one, we concurrently traverse all
directories that we find and then sequentially issue file deletions.
This new deletion logic is currently unbounded, and will spawn a bunch
of tasks no matter the underlying system.
Benchmark using `--configration=release` for both the `main` and
`parallel-removal` branches:
```
$ rm -rf .build && git switch main && swift build --configuration=release && cp .build/release/NIOTestCLI NIOTestCLI_main
[...]
Building for production...
[134/134] Linking NIOPerformanceTester
Build complete! (32.65s)
$ rm -rf .build && git switch parallel-removal && swift build --configuration=release
[...]
Building for production...
[134/134] Linking NIOPerformanceTester
Build complete! (32.19s)
$ hyperfine --max-runs 10 --prepare "rm -rf /tmp/swift-nio && cp -r $(pwd) /tmp/swift-nio" "./NIOTestCLI_main /tmp/swift-nio" ".build/release/NIOTestCLI /tmp/swift-nio" "rm -rf /tmp/swift-nio"
Benchmark 1: ./NIOTestCLI_main /tmp/swift-nio
Time (mean ± σ): 3.019 s ± 0.032 s [User: 0.601 s, System: 11.123 s]
Range (min … max): 2.985 s … 3.072 s 10 runs
Benchmark 2: .build/release/NIOTestCLI /tmp/swift-nio
Time (mean ± σ): 2.295 s ± 0.065 s [User: 0.673 s, System: 7.677 s]
Range (min … max): 2.212 s … 2.370 s 10 runs
Benchmark 3: rm -rf /tmp/swift-nio
Time (mean ± σ): 1.517 s ± 0.037 s [User: 0.015 s, System: 1.442 s]
Range (min … max): 1.480 s … 1.590 s 10 runs
Summary
rm -rf /tmp/swift-nio ran
1.51 ± 0.06 times faster than .build/release/NIOTestCLI /tmp/swift-nio
1.99 ± 0.05 times faster than ./NIOTestCLI_main /tmp/swift-nio
```
Resolves apple#2933
|
I've never followed-up here. I've done a few more tests on my PR, but never increased the number of files/directories above five-digits. The benchmarks showed that the new |
This is fantastic news. Thanks a lot @mimischi 🙏🏼 |
👋🏼 Hi,
First of all, thanks for this amazing work.
We are using the file-system IO components at Tuist, and I'd like to bring up something that we notice regarding the performance of the
NIOFileSystem.FileSystem.removeItemfunction.When we run it against a large directory with many nested directories and and files, we noticed the performance is much worse than using
FileManageror justrm -rfin a Linux environment. We benchmarked the performance with hyperfine and the results are below.Looking at the implementation, it seems that we are not parallelising at all, so I was wondering if this is something intentional, or if it's ok to refactor that code to run as much as possible in parallel.
Bechmarks
With NIOFileSystem
Benchmark
With Foundation's FileManager
Benchmark:
With
rm -rfBenchmark:
Time (mean ± σ): 3.150 s ± 0.386 s [User: 0.022 s, System: 1.666 s]
Range (min … max): 2.225 s … 3.585 s 10 runs
The text was updated successfully, but these errors were encountered: