Hi,
I have about 4TB of various data on two ZFS pools I want to replicate to a faraway server running TrueNAS Scale. However, when using a single ssh connection, the throughput is quite poor at around 4-8 MB/s, which would take a long time to do this initial send.
I noticed that rclone is able to achieve a much higher combined throughput when sending many files, around 20-30 MB/s, which I suspect is due to it using parallel connections. I wonder if there is a way to do something similar.
So I came up with the following plan:
- First send to a local file on disk
- Split the file
- Use rclone to transfer it to the destination
- Reassemble it on destination or otherwise stream it into zfs recv
But this seems unnecessarily complicated. Is there a better way of doing it? I can conceive a tool that works like ssh for sending streams but uses multiple connections in the background.
Also, for future syncoid runs, will syncoid recognize that some of the snapshots are already present and do the rest incrementally?