Best MergerFS Options for Rsync

[kyleinprogress]

All my new HDDs have been delivered as a result of Black Friday sales, so I'm starting the process of swapping new and old HDDs.

Using the documentation found here: Backup & Recovery

1. Drives have been burned in and formatted: Setup EXT4 Drive
2. Created temp pool:

sudo mergerfs -o cache.files=off,dropcacheonclose=true,category.create=mfs,fsname=tmp_pool /mnt/disks/s01:/mnt/disks/s02 /mnt/tmp

3. Sync data using rsync

DISK -> DISK:
rsync -avxhHAXWES --numeric-ids --info=progress2 /mnt/disks/s14/ /mnt/disks/s02/

DISK -> TEMP POOL:
rsync -avxhHAXWES --numeric-ids --info=progress2 /mnt/disks/s14/ /mnt/tmp/

If I use rsync directly between disks, I get ~ 150MB/s. If I rsync from disk to temp pool, I get about half the speed.

Not a huge deal, just curious if there's an optimal set of options to use when setting up the temporary pool to maximize throughput. Figured I didn't need the mmap support for this process since it's just file copies.

[trapexit]

I'll have to look over the arguments and see how they impact mergerfs. What kernel version are you running because that could impact things. I've seen situations where a truncate gets converted into lots of sparse seek and writes which harms file growth. Could also be some xattr thing. Those both increase the number of requests and given the sequential and blocking nature of the transfer slow things down. But IIRC that behavior can differ based on filesystem and kernel.

[kyleinprogress]

Here's the details:

Operating System: Ubuntu 20.04.6 LTS
Kernel: Linux 5.4.0-1312304060810-generic
Architecture: x86-64
mergerfs v2.38.0
rsync  version 3.1.3  protocol version 31

[trapexit]

It it is -S

Remove that. It causes rsync to write 1KB rather than 256KB chunks of data and that murders perf. Even if you enable writeback caching it doesn't help much. In fact... I'm surprised it only cut your speeds in half. My test setup went from >150MB/s to less than 5MB/s.

[kyleinprogress]

That worked. Back up to almost ~200MB/s copying. Just odd that it only surfaced when running over MergerFS and not on disk to disk.

Either way, looks to be fixed! Thanks again for the help and you're awesome software.

[trapexit]

As I mention in the performance section of mergerfs docs the block size can have serious impacts on throughput on any device but due to the synchronous nature of file IO the higher the latency the worse the throughput. The more back and forth it has to do the more the increased latency affects it relatively speaking. mergerfs adds latency, the block size was small, much more serious impact.

It is also possible the filesystem may have an impact. You could double check by doing a rsync -S but stracing it. See if the write size is just 1024. Could also depend on the version of rsync.

[kyleinprogress]

Good to know. I may play with it a little later this week. I've got a couple of extra drives now, so I can run copies "disk to disk" and "disk to pool" and see what the block size is. But for now, just removing -S from the rsync command seems to make a huge difference.

[trapexit]

BTW... I've updated the docs in question and removed -S.