Trying to work out system to mirror/sync between two servers/pools in different locations

OpenZFS
1

I have two Proxmox PVE servers, each of which has a NVME for Proxmox and VMs/LXCs which backup to a PBS server, and a 16TB USB HDD for data which is LUKS encrypted with ZFS on top. The zpool on the machine that will be at my house (server1) is named z16TB-DM, and the one that will be located at my Dad’s house (server2) is named z16TB-AM.

I have datasets for three users (del, user2, user3), and for media, software and NVR, and I’ve done a snapshot for each dataset and done zfs send/receive with each one to copy it to the other drive, so the contents are identical at present.

Once the servers are deployed, I’ll be backing up my PC’s data to my dataset (del), and my Dad and Mum will do likewise with theirs (user2 and user3).

What I want to do is have my dataset (del) mirror to my Dad’s server every night, and have the user2 and user3 datasets mirror from his server to mine, so we have an offsite copy of each other’s backed up PC data.

For the media and software datasets it’s a bit more complicated, because new files are likely to be added on either server, so I guess that needs to do an “update with new files only” both ways, otherwise if a file is added to server 1 and server 2 initiated a sync job first, it would delete the file from server 1.

The servers will be connected via a secure Tailscale connection, so I won’t need to use any extra encryption in transit.

I’ve created a user “syncoid” and added it to the sudo group; added “syncoid ALL=NOPASSWD: </usr/sbin/zfs>” to /etc/sudoers; generated a key with ssh-keygen and copied it to both servers under /home/syncoid/.ssh/ and copied the pubkey into authorized keys, and tested that I can connect either way using ssh (using the LAN addresses 10.10.18.198 and 10.10.55.198).

I’ve configured sanoid on my server and that’s creating hourly, daily and monthly snapshots automatically.

syncoid isn’t working at the moment though. One of the errors it gave was about sudo not being found, which didn’t surprise me as sudo isn’t installed in PVE, so I installed it but it still gives the same error.

syncoid z16TB-DM/del syncoid@10.10.55.198:z16TB-AM/del --sshport=2325 -sshkey=/home/syncoid/.ssh/id_rsa
WARN: ZFS resume feature not available on target machine - sync will continue without resume support.
INFO: Sending oldest full snapshot z16TB-DM/del@copy (~ 624.0 GB) to new target filesystem:
bash: line 1: sudo: command not found
mbuffer: error: outputThread: error writing to at offset 0x110000: Broken pipe
mbuffer: warning: error during output to : Broken pipe
2.27MiB 0:00:00 [3.09MiB/s] [> ] 0%
CRITICAL ERROR: zfs send ‘z16TB-DM/del’@‘copy’ | pv -p -t -e -r -b -s 670047542304 | lzop | mbuffer -q -s 128k -m 16M 2>/dev/null | ssh -p 2325 -i /home/syncoid/.ssh/id_rsa -S /tmp/syncoid-syncoid@10.10.55.198-1739064523 syncoid@10.10.55.198 ’ mbuffer -q -s 128k -m 16M 2>/dev/null | lzop -dfc | sudo zfs receive -F ‘"’“‘z16TB-AM/del’”‘"’’ failed: 32512 at /usr/sbin/syncoid line 492.

I’m not even sure if syncoid is the best thing for what I need to do, and as I’m using Tailscale, ssh is probably an unnecessary overhead, but if the tools that I need to use rely on it, it’s probably not worth worrying about.

I’m also not sure what I need to do about permissions for the datasets/folders. At the moment each users dataset is owned by user1:user1, etc. and the media folder is owned by media:media, but maybe for syncoid and the like to work I need to create a “syncoid” group and recursively set the group permissions for each dataset to that and group, with rw group permissions?

2

OK, after reading this page Improving Replication Security With OpenZFS Delegation - Klara Systems I understand that I my approach was wrong.

What I needed to do is just have my user’s dataset on my server, and have my parents’ datasets on my Dad’s server, and then create a “backup” dataset on each server, and then use zfs send/receive to copy the snapshots of the datasets on each server to the backup dataset on the other server.

I’ve done that now, but something seems a bit funky with the backup of my dataset, as on my server it says Used 654G Refer 519G, but the backup on my Dad’s server says Used and Refer are both 570G.

I’m also still not sure what to do with the media and software datasets, as they both need to exist and be snapshotted regularly on both servers, so I can’t use the zfs send/receive to a backup dataset approach but I want to ensure that any new files added to either server get backed up to the other one.

3

You can’t do this with ZFS replication. ZFS replication updates the ENTIRE dataset, and must destroy any data created locally on the target each time new data is replicated in from source.

This isn’t simply “syncoid won’t do this,” this is “ZFS replication cannot do this, regardless of what orchestration tool based on it you try.”

1 Like
4

Hmm, OK I’ll have to find another way to achieve that. As the intention is that the media and software folders will be synced on both servers, I guess I can just snapshot them locally and there’s no need to send the snapshots to the other server’s backup dataset.

Any idea why my dataset shows 654G Used on my server, but only 570G Used on the backup?

5

There are endless ways for that to happen, and you haven’t given us any concrete data. Most frequently, you’ve got snapshots on one side that aren’t on the other. Sometimes, you’ve got compression set differently from one side to the other. Sometimes it’s recordsize, sometimes it’s ashift, sometimes (I am repeating what Allan Jude told me recently; I do not fully understand this myself) it’s uncommitted write queues on one side vs the queues empty and committed on the other side, when one side is mounted and the other is not.

1 Like
6

If I understand your requirements correctly, you can use SyncThing to keep both sides identical above the filesystem level and set up your ZFS snapshots independently of that. I’ve been using it for a couple of years now and I wholeheartedly recommend it.

1 Like
7

Sorry, I got very busy with work.

So on server 1, the ‘del’ dataset says its using 664GB of which 529GB is referred:
z16TB-DM/del 664G 9.54T 529G /mnt/z16TB-DM/del

and it has these snapshots:

> z16TB-DM/del@copy                                          135G      -   570G  -
> z16TB-DM/del@autosnap_2025-02-08_13:46:32_monthly            0B      -   519G  -
> z16TB-DM/del@autosnap_2025-02-08_13:46:32_daily              0B      -   519G  -
> z16TB-DM/del@syncoid_pve_2025-02-08:14:45:12-GMT00:00        0B      -   519G  -
> z16TB-DM/del@syncoid_pve_2025-02-08:14:47:58-GMT00:00        0B      -   519G  -
> z16TB-DM/del@syncoid_pve_2025-02-08:14:48:48-GMT00:00        0B      -   519G  -
> z16TB-DM/del@syncoid_pve_2025-02-08:14:51:21-GMT00:00        0B      -   519G  -
> z16TB-DM/del@autosnap_2025-02-09_00:00:01_daily              0B      -   519G  -
> z16TB-DM/del@syncoid_pve_2025-02-09:01:28:44-GMT00:00        0B      -   519G  -
> z16TB-DM/del@syncoid_pve_2025-02-09:02:02:41-GMT00:00        0B      -   519G  -
> z16TB-DM/del@autosnap_2025-02-10_00:00:02_daily              0B      -   519G  -
> z16TB-DM/del@autosnap_2025-02-11_00:00:04_daily            408K      -   520G  -
> z16TB-DM/del@autosnap_2025-02-12_00:00:08_daily            460K      -   520G  -
> z16TB-DM/del@autosnap_2025-02-13_00:00:08_daily            472K      -   521G  -
> z16TB-DM/del@autosnap_2025-02-14_00:00:08_daily            492K      -   521G  -
> z16TB-DM/del@autosnap_2025-02-15_00:00:08_daily            512K      -   521G  -
> z16TB-DM/del@autosnap_2025-02-16_00:00:08_daily            544K      -   522G  -
> z16TB-DM/del@autosnap_2025-02-17_00:00:08_daily            584K      -   522G  -
> z16TB-DM/del@autosnap_2025-02-18_00:00:08_daily            612K      -   523G  -
> z16TB-DM/del@autosnap_2025-02-19_00:00:08_daily            640K      -   524G  -
> z16TB-DM/del@autosnap_2025-02-20_00:00:08_daily            676K      -   524G  -
> z16TB-DM/del@autosnap_2025-02-21_00:00:08_daily            704K      -   525G  -
> z16TB-DM/del@autosnap_2025-02-22_00:00:08_daily           1.41M      -   525G  -
> z16TB-DM/del@syncoid_pve-DM_2025-02-22:22:52:51-GMT00:00     0B      -   525G  -
> z16TB-DM/del@autosnap_2025-02-23_00:00:08_daily              0B      -   525G  -
> z16TB-DM/del@autosnap_2025-02-24_00:00:08_daily            168K      -   526G  -
> z16TB-DM/del@autosnap_2025-02-25_00:00:08_daily            168K      -   526G  -
> z16TB-DM/del@autosnap_2025-02-26_00:00:08_daily            824K      -   526G  -
> z16TB-DM/del@autosnap_2025-02-27_00:00:08_daily            840K      -   527G  -
> z16TB-DM/del@autosnap_2025-02-28_00:00:08_daily              0B      -   528G  -
> z16TB-DM/del@autosnap_2025-03-01_00:00:08_monthly            0B      -   528G  -
> z16TB-DM/del@autosnap_2025-03-01_00:00:08_daily              0B      -   528G  -
> z16TB-DM/del@autosnap_2025-03-02_00:00:08_daily            964K      -   529G  -

@copy was the first one that I created manually, and the rest were mostly created with sanoid, other than a few that were created when I was trying to get syncoid working.

On server 2, there’s only the del@copy snapshot, which I manually sent across with send/receive.

z16TB-AM/backups/del@copy 0B - 570G

and I don’t understand why that only contains 570GB, when the del@copy on server 1 seems to refer to 570GB and use an additional 135GB.

I’ve tried to send the later snapshots to server 2, and ChatGPT advised me to do

zfs send z16TB-DM/del@autosnap_2025-03-02_00:00:08_daily | ssh root@10.10.55.198 -p 2325 zfs receive z16TB-AM/backups/del

but that returns ‘cannot receive new filesystem stream: destination ‘z16TB-AM/backups/del’ exists must specify -F to overwrite it’ and I don’t think I want to overwrite the backup every time I update it with the latest snapshot.

Is there a way to consolidate all the data in the snapshots on server 1 into the original ‘del’ dataset and then start the snapshots afresh?

8

Thanks, SyncThing looks good but my concern is that if some or all of the data in the media folder gets deleted on one server, then SyncThing will automatically delete it on the other server, which I want to avoid.

I guess if I’m snapshotting the media folder on each server locally, without sending those snapshots to the other server (which would just waste a load of space, because it would mean having two full copies of the data on both servers), then it will be fairly easy to roll back from any such disasters though.

It would seem safer if there was a way to have SyncThing only sync new files to the other server though, and not delete anything. Then maybe once in a while I could run a manual full sync/delete after checking that the folder is intact first?

9

I see. There’s an advanced option to prevent deletion but it isn’t recommended because it interferes with the fundamental model of syncing.

ST’s versioning is very flexible, though. You can have it keep deleted files indefinitely if you want, and/or keep multiple versions. (It also supports versioning through an external program but I’ve never used that.) I’m not sure any more that it fits your intended use case, but perhaps you could take a look at the docs on versioning to see if that would allay your concerns. And yes, ZFS snapshots seem like a good way to avoid any potential data loss.

1 Like
10

Thanks. Yes, the versioning is probably sufficient so I won’t mess with that advanced option to prevent deletion.

Now I just need to get the zfs send and receive working properly for the other datasets, so I can replicate them between the servers.

I think I need to use the -R and -I flags to do incremental replications, so the command would be:

zfs send -R -I z16TB-DM/del@copy z16TB-DM/del@autosnap_2025-03-02_00:00:08_daily | ssh root@10.10.55.198 -p 2325 zfs receive z16TB-AM/backups/del

but that’s taking ages, even with both machines on the same LAN, and as you can see below, the total size of all the snapshots after the first @copy one isn’t very much at all, so shouldn’t that complete very quickly? Also, why is the Refer size for each snapshot less than the @copy one? Shouldn’t that be the same or greater each time?

> z16TB-DM/del@copy                                          135G      -   570G  -
> z16TB-DM/del@autosnap_2025-02-08_13:46:32_monthly            0B      -   519G  -
> z16TB-DM/del@autosnap_2025-02-08_13:46:32_daily              0B      -   519G  -
> z16TB-DM/del@syncoid_pve_2025-02-08:14:45:12-GMT00:00        0B      -   519G  -
> z16TB-DM/del@syncoid_pve_2025-02-08:14:47:58-GMT00:00        0B      -   519G  -
> z16TB-DM/del@syncoid_pve_2025-02-08:14:48:48-GMT00:00        0B      -   519G  -
> z16TB-DM/del@syncoid_pve_2025-02-08:14:51:21-GMT00:00        0B      -   519G  -
> z16TB-DM/del@autosnap_2025-02-09_00:00:01_daily              0B      -   519G  -
> z16TB-DM/del@syncoid_pve_2025-02-09:01:28:44-GMT00:00        0B      -   519G  -
> z16TB-DM/del@syncoid_pve_2025-02-09:02:02:41-GMT00:00        0B      -   519G  -
> z16TB-DM/del@autosnap_2025-02-10_00:00:02_daily              0B      -   519G  -
> z16TB-DM/del@autosnap_2025-02-11_00:00:04_daily            408K      -   520G  -
> z16TB-DM/del@autosnap_2025-02-12_00:00:08_daily            460K      -   520G  -
> z16TB-DM/del@autosnap_2025-02-13_00:00:08_daily            472K      -   521G  -
> z16TB-DM/del@autosnap_2025-02-14_00:00:08_daily            492K      -   521G  -
> z16TB-DM/del@autosnap_2025-02-15_00:00:08_daily            512K      -   521G  -
> z16TB-DM/del@autosnap_2025-02-16_00:00:08_daily            544K      -   522G  -
> z16TB-DM/del@autosnap_2025-02-17_00:00:08_daily            584K      -   522G  -
> z16TB-DM/del@autosnap_2025-02-18_00:00:08_daily            612K      -   523G  -
> z16TB-DM/del@autosnap_2025-02-19_00:00:08_daily            640K      -   524G  -
> z16TB-DM/del@autosnap_2025-02-20_00:00:08_daily            676K      -   524G  -
> z16TB-DM/del@autosnap_2025-02-21_00:00:08_daily            704K      -   525G  -
> z16TB-DM/del@autosnap_2025-02-22_00:00:08_daily           1.41M      -   525G  -
> z16TB-DM/del@syncoid_pve-DM_2025-02-22:22:52:51-GMT00:00     0B      -   525G  -
> z16TB-DM/del@autosnap_2025-02-23_00:00:08_daily              0B      -   525G  -
> z16TB-DM/del@autosnap_2025-02-24_00:00:08_daily            168K      -   526G  -
> z16TB-DM/del@autosnap_2025-02-25_00:00:08_daily            168K      -   526G  -
> z16TB-DM/del@autosnap_2025-02-26_00:00:08_daily            824K      -   526G  -
> z16TB-DM/del@autosnap_2025-02-27_00:00:08_daily            840K      -   527G  -
> z16TB-DM/del@autosnap_2025-02-28_00:00:08_daily              0B      -   528G  -
> z16TB-DM/del@autosnap_2025-03-01_00:00:08_monthly            0B      -   528G  -
> z16TB-DM/del@autosnap_2025-03-01_00:00:08_daily              0B      -   528G  -
> z16TB-DM/del@autosnap_2025-03-02_00:00:08_daily            964K      -   529G  -
11

After that incremental update completed, I ran syncoid and it said:

DEBUG: syncing source z16TB-DM/del to target z16TB-AM/backups/del.
DEBUG: getting current value of syncoid:sync on z16TB-DM/del…
zfs get -H syncoid:sync ‘z16TB-DM/del’
DEBUG: checking to see if z16TB-AM/backups/del on ssh -p 2325 -S /tmp/syncoid-root@10.10.55.198-1741549923 root@10.10.55.198 is already in zfs receive using ssh -p 2325 -S /tmp/syncoid-root@10.10.55.198-1741549923 root@10.10.55.198 ps -Ao args= …
DEBUG: checking to see if target filesystem exists using “ssh -p 2325 -S /tmp/syncoid-root@10.10.55.198-1741549923 root@10.10.55.198 zfs get -H name ‘’”‘“‘z16TB-AM/backups/del’”’“‘’ 2>&1 |”…
DEBUG: getting current value of receive_resume_token on z16TB-AM/backups/del…
ssh -p 2325 -S /tmp/syncoid-root@10.10.55.198-1741549923 root@10.10.55.198 zfs get -H receive_resume_token ‘’“'”‘z16TB-AM/backups/del’“'”‘’
DEBUG: no receive token found
DEBUG: getting list of snapshots on z16TB-DM/del using zfs get -Hpd 1 -t snapshot guid,creation ‘z16TB-DM/del’ |…
DEBUG: getting list of snapshots on z16TB-AM/backups/del using ssh -p 2325 -S /tmp/syncoid-root@10.10.55.198-1741549923 root@10.10.55.198 zfs get -Hpd 1 -t snapshot guid,creation ‘’“'”‘z16TB-AM/backups/del’“'”‘’ |…
DEBUG: creating sync snapshot using " zfs snapshot ‘z16TB-DM/del’@syncoid_pve-DM_2025-03-09:19:52:07-GMT00:00
“…
DEBUG: getting current value of -p used on z16TB-AM/backups/del…
ssh -p 2325 -S /tmp/syncoid-root@10.10.55.198-1741549923 root@10.10.55.198 zfs get -H -p used ‘’”‘“‘z16TB-AM/backups/del’”’“‘’
DEBUG: checking to see if z16TB-AM/backups/del on ssh -p 2325 -S /tmp/syncoid-root@10.10.55.198-1741549923 root@10.10.55.198 is already in zfs receive using ssh -p 2325 -S /tmp/syncoid-root@10.10.55.198-1741549923 root@10.10.55.198 ps -Ao args= …
DEBUG: rolling back target to z16TB-AM/backups/del@autosnap_2025-03-02_00:00:08_daily…
ssh -p 2325 -S /tmp/syncoid-root@10.10.55.198-1741549923 root@10.10.55.198 zfs rollback -R ‘z16TB-AM/backups/del’@‘autosnap_2025-03-02_00:00:08_daily’
DEBUG: getting estimated transfer size from source using " zfs send -nvP -I ‘z16TB-DM/del@autosnap_2025-03-02_00:00:08_daily’ ‘z16TB-DM/del@syncoid_pve-DM_2025-03-09:19:52:07-GMT00:00’ 2>&1 |”…
DEBUG: sendsize = 75782487576
Sending incremental z16TB-DM/del@autosnap_2025-03-02_00:00:08_daily … syncoid_pve-DM_2025-03-09:19:52:07-GMT00:00 (~ 70.6 GB):
DEBUG: zfs send -I ‘z16TB-DM/del’@‘autosnap_2025-03-02_00:00:08_daily’ ‘z16TB-DM/del’@‘syncoid_pve-DM_2025-03-09:19:52:07-GMT00:00’ | pv -p -t -e -r -b -s 75782487576 | lzop | mbuffer -q -s 128k -m 16M 2>/dev/null | ssh -p 2325 -S /tmp/syncoid-root@10.10.55.198-1741549923 root@10.10.55.198 ’ mbuffer -q -s 128k -m 16M 2>/dev/null | lzop -dfc | zfs receive -s -F ‘"’“‘z16TB-AM/backups/del’”‘"’ 2>&1’
70.6GiB 0:10:50 [ 111MiB/s] [=======================================================================================================================================>] 100%

So the snapshot that syncoid created was 70.6GB (which is reasonable because the previous snapshot was 7 days ago) and took about 11 minutes to transfer on a 1Gb LAN, but the list of snapshots shows:

z16TB-DM/del@autosnap_2025-03-02_00:00:08_daily            972K      -   529G  -
z16TB-DM/del@syncoid_pve-DM_2025-03-09:19:52:07-GMT00:00     0B      -   599G  -

I don’t understand why the additional 70GB is reflected in the Refer column rather than the Used column, as surely Refer is meant to show how much data was in the previous snapshot and is referred to by this one, whilst Used is supposed to show the additional data contained in this one?