Pull backup from a TrueNAS system to Synology

It’s never going to get a whole lot quicker unless you replace rsync with replication, which would require a ZFS filesystem on both ends. You are correct that rsync must grovel over each individual file looking for potential differences; it also has to chunk the changed files on both ends, individually hash each chunk, then compare the hashes between source and destination to figure out which bits to move or not move.

On a 1gbps LAN, the last bit–chunking and hashing the files so you don’t need to move the entire file, just the changes–is usually a lose. You can make rsync skip that part–which decreases disk load significantly–with the ‘W’ flag (whole-file), which is safe as houses.

Using --inplace is dangerous on non-CoW filesystems, because if your replication gets interrupted partway through, you’re left with a corrupt file. (Rsync’s default behavior is to create an entirely new file, and work with that, then unlink the old and rename the new into place after it’s finished.)

If you can stomach ditching the Synology in favor of another ZFS system, you can start doing replication, which can be entire orders of magnitude faster than rsync–replication neither needs to grovel over the filesystem looking for potentially changed metadata, nor does it need to grovel over the insides of changed files. Replication already knows exactly which bits of which files need updating, and immediately begins chucking them down the wire at the target.

As an example, my /opt directory has 540,767 files in it (because that’s where my Steam repo lives). It gets backed up via replication every night, across a 1Gbps LAN like yours. Actually that’s not true… it gets backed up via replication every hour across a 1Gbps LAN:

root@jrs-dr0:/# crontab -l | grep opt
10 * * * * /usr/local/bin/syncoid -r --no-sync-snap root@banshee:banshee/opt    data/backup/jrs/banshee/opt

And it does not take very long when it does back up, despite the target being simple rust drives, thanks to ZFS replication:

root@jrs-dr0:/# /usr/local/bin/syncoid -r --no-sync-snap root@banshee:banshee/opt    data/backup/jrs/banshee/opt
NEWEST SNAPSHOT: autosnap_2023-07-31_16:00:01_hourly
Sending incremental banshee/opt@autosnap_2023-07-30_04:00:03_hourly ... autosnap_2023-07-31_16:00:01_hourly (~ 314.2 MB):
 313MiB 0:00:12 [25.1MiB/s] [================================> ] 99%            

That twelve second backup run was actually painfully slow by my standards… because when I manually ran it just now, three or four other backup jobs were also running, to the same rust target pool. :slight_smile:

1 Like