Can this be solved? (Syncoid - "target exists, but has no snapshots matching...")

vortexsurfer · September 6, 2023, 6:37pm

Hi,
I have a setup with a zpool that gets snapshotted by sanoid regularly, and once a day (at 2 AM) syncoid replicates it over to a zpool in my backup server.

But I recently discovered a few datasets weren’t replicating anymore, due to the “target exists, but has no snapshots matching” critical error.

I realized the reason is I deleted some snapshots (on the source) a month ago, to free up some space, without thinking much… I guess I managed to delete the common syncoid snapshot

So on my backup server, the newest snapshots are:

backup01/data@autosnap_2023-08-05_22:00:05_hourly               0B      -      765G  -
backup01/data@autosnap_2023-08-05_23:00:02_hourly               0B      -      765G  -
backup01/data@autosnap_2023-08-06_00:00:08_daily                0B      -      765G  -
backup01/data@autosnap_2023-08-06_00:00:08_hourly               0B      -      765G  -
backup01/data@autosnap_2023-08-06_01:00:02_hourly               0B      -      765G  -
backup01/data@autosnap_2023-08-06_02:00:01_hourly               0B      -      765G  -
backup01/data@syncoid_backup01_2023-08-06:02:02:57-GMT02:00     0B      -      765G  -

And on the source zpool, the oldest snapshots are:

thepool/data@syncoid_backup01_2023-08-07:02:08:49-GMT02:00   116K      -      776G  -
thepool/data@syncoid_backup01_2023-08-08:02:02:41-GMT02:00    84K      -      776G  -
thepool/data@syncoid_backup01_2023-08-09:02:04:16-GMT02:00    76K      -      776G  -
thepool/data@syncoid_backup01_2023-08-10:02:13:25-GMT02:00   104K      -      794G  -
thepool/data@syncoid_backup01_2023-08-11:02:04:40-GMT02:00   124K      -      808G  -
thepool/data@syncoid_backup01_2023-08-12:02:03:34-GMT02:00    64K      -      845G  -
thepool/data@syncoid_backup01_2023-08-13:02:02:45-GMT02:00    64K      -      845G  -

I’ve used ZFS for quite a while, but still haven’t fully wrapped my head around the details of how incremental replication works, so my question is: can I somehow get syncoid to start replicating this dataset again?
From what I’ve read, I’m quite sure the answer is “no”, but I wanted to ask the experts to be sure.

If need be I can delete all snapshots on the source, there’s nothing important there. But I do want to keep the snapshots I currently have on the backup server, at least for a while.

What are my options here?
I guess the “worst case solution” is something like: rename backup01/data to backup01/data-before-i-messed-up and run a fresh replication to backup01/data from the source. It would waste some disk space, but I can live with that.

Thanks for any help/advice!

muay_throwaway · September 7, 2023, 4:14am

You can use an existing snapshot on the target as an origin by specifying -o origin=snapshot_name . This treats the incoming snapshot as a clone of snapshot_name . You can then promote the new clone (zfs promote ). Here is a better explanation by ahrens, one of the maintainers.

EDIT:
The snapshot would have to be received to a separate filesystem.

I should note that this relies on nopwrite, so it will only work with compatible checksum algorithms (not the default, fletcher4) and requires compression. If your data is not already using one of the nopwrite-supporting checksum algorithms and compressed, it may not be worth going through the trouble (credit to @rincebrain for pointing this out). Also, changing the algorithm only affects new writes, so the snapshot data would have to be rewritten.

vortexsurfer · September 8, 2023, 4:01pm

Thanks all! I have experimented - a bit with the origin flag (and -n, to not actually receive anything).

I came up with this, running on the backup server (I use the pull strategy, source doesn’t have direct network access to the backup server):

ssh user@source zfs send -v -w -i thepool/data@<oldest snap> thepool/data@<newest snap> | zfs receive backup01/data-new -v -n -o origin=backup01/data@<newest snap>

Output (before it starts sending):

total estimated size is 87.3G
using provided clone origin backup01/data@<newest snap>
would receive incremental stream of thepool/data@<newest snap> into backup01/data-new@<newest snap>

87.3G is a lot less than the full dataset, so this seems reasonable?

I haven’t had time to try to run it without the -n flag yet though.

Bandwidth is of no concern, it’s all on my local network.

mercenary_sysadmin · September 8, 2023, 4:41pm

It doesn’t look like your proposed commands will work for you tbh. You said:

ssh user@source zfs send -v -w -i thepool/data@ thepool/data@ | zfs receive backup01/data-new -v -n -o origin=backup01/data@

When I tried it out in practice, it failed:

root@elden:/# zfs create rpool/origintest
root@elden:/# zfs snapshot rpool/origintest@1
root@elden:/# zfs send rpool/origintest@1 | zfs receive rpool/targettest
root@elden:/# zfs snapshot rpool/origintest@2
root@elden:/# zfs send -I rpool/origintest@1 rpool/origintest@2 | zfs receive rpool/targettest
root@elden:/# zfs snapshot rpool/origintest@3
root@elden:/# zfs destroy rpool/origintest@1
root@elden:/# zfs destroy rpool/origintest@2
root@elden:/# zfs snapshot rpool/origintest@4
root@elden:/# zfs send -v -w -i rpool/origintest@3 rpool/origintest@4 | zfs receive rpool/targettest -v -n -o origin=rpool/targettest@2
send from rpool/origintest@3 to rpool/origintest@4 estimated size is 624B
total estimated size is 624
using provided clone origin rpool/targettest@2
cannot receive new filesystem stream: destination 'rpool/targettest' exists
must specify -F to overwrite it

So, that doesn’t look good… what happens if we use the -F flag?

root@elden:/# zfs send -v -w -i rpool/origintest@3 rpool/origintest@4 | zfs receive rpool/targettest -v -n -o origin=rpool/targettest@2 -F
send from rpool/origintest@3 to rpool/origintest@4 estimated size is 624B
total estimated size is 624
using provided clone origin rpool/targettest@2
cannot receive new filesystem stream: destination has snapshots (eg. rpool/targettest@2)
must destroy them to overwrite it

Perhaps there’s some way to get this done with a different command syntax, or with the version of OpenZFS in the master repos… but it doesn’t appear to be a functional approach with the syntax given on OpenZFS 2.1.5, at least.