Can this be solved? (Syncoid - "target exists, but has no snapshots matching...")

vortexsurfer · September 6, 2023, 6:37pm

Hi,
I have a setup with a zpool that gets snapshotted by sanoid regularly, and once a day (at 2 AM) syncoid replicates it over to a zpool in my backup server.

But I recently discovered a few datasets weren’t replicating anymore, due to the “target exists, but has no snapshots matching” critical error.

I realized the reason is I deleted some snapshots (on the source) a month ago, to free up some space, without thinking much… I guess I managed to delete the common syncoid snapshot

So on my backup server, the newest snapshots are:

backup01/data@autosnap_2023-08-05_22:00:05_hourly               0B      -      765G  -
backup01/data@autosnap_2023-08-05_23:00:02_hourly               0B      -      765G  -
backup01/data@autosnap_2023-08-06_00:00:08_daily                0B      -      765G  -
backup01/data@autosnap_2023-08-06_00:00:08_hourly               0B      -      765G  -
backup01/data@autosnap_2023-08-06_01:00:02_hourly               0B      -      765G  -
backup01/data@autosnap_2023-08-06_02:00:01_hourly               0B      -      765G  -
backup01/data@syncoid_backup01_2023-08-06:02:02:57-GMT02:00     0B      -      765G  -

And on the source zpool, the oldest snapshots are:

thepool/data@syncoid_backup01_2023-08-07:02:08:49-GMT02:00   116K      -      776G  -
thepool/data@syncoid_backup01_2023-08-08:02:02:41-GMT02:00    84K      -      776G  -
thepool/data@syncoid_backup01_2023-08-09:02:04:16-GMT02:00    76K      -      776G  -
thepool/data@syncoid_backup01_2023-08-10:02:13:25-GMT02:00   104K      -      794G  -
thepool/data@syncoid_backup01_2023-08-11:02:04:40-GMT02:00   124K      -      808G  -
thepool/data@syncoid_backup01_2023-08-12:02:03:34-GMT02:00    64K      -      845G  -
thepool/data@syncoid_backup01_2023-08-13:02:02:45-GMT02:00    64K      -      845G  -

I’ve used ZFS for quite a while, but still haven’t fully wrapped my head around the details of how incremental replication works, so my question is: can I somehow get syncoid to start replicating this dataset again?
From what I’ve read, I’m quite sure the answer is “no”, but I wanted to ask the experts to be sure.

If need be I can delete all snapshots on the source, there’s nothing important there. But I do want to keep the snapshots I currently have on the backup server, at least for a while.

What are my options here?
I guess the “worst case solution” is something like: rename backup01/data to backup01/data-before-i-messed-up and run a fresh replication to backup01/data from the source. It would waste some disk space, but I can live with that.

Thanks for any help/advice!

muay_throwaway · September 6, 2023, 10:57pm

You can use an existing snapshot on the target as an origin by specifying -o origin=snapshot_name. This treats the incoming snapshot as a clone of snapshot_name. You can then promote the new clone (zfs promote). Here is a better explanation by ahrens, one of the maintainers.

muay_throwaway · September 7, 2023, 4:14am

You can use an existing snapshot on the target as an origin by specifying -o origin=snapshot_name . This treats the incoming snapshot as a clone of snapshot_name . You can then promote the new clone (zfs promote ). Here is a better explanation by ahrens, one of the maintainers.

EDIT:
The snapshot would have to be received to a separate filesystem.

I should note that this relies on nopwrite, so it will only work with compatible checksum algorithms (not the default, fletcher4) and requires compression. If your data is not already using one of the nopwrite-supporting checksum algorithms and compressed, it may not be worth going through the trouble (credit to @rincebrain for pointing this out). Also, changing the algorithm only affects new writes, so the snapshot data would have to be rewritten.

vortexsurfer · September 8, 2023, 4:01pm

Thanks all! I have experimented - a bit with the origin flag (and -n, to not actually receive anything).

I came up with this, running on the backup server (I use the pull strategy, source doesn’t have direct network access to the backup server):

ssh user@source zfs send -v -w -i thepool/data@<oldest snap> thepool/data@<newest snap> | zfs receive backup01/data-new -v -n -o origin=backup01/data@<newest snap>

Output (before it starts sending):

total estimated size is 87.3G
using provided clone origin backup01/data@<newest snap>
would receive incremental stream of thepool/data@<newest snap> into backup01/data-new@<newest snap>

87.3G is a lot less than the full dataset, so this seems reasonable?

I haven’t had time to try to run it without the -n flag yet though.

Bandwidth is of no concern, it’s all on my local network.

mercenary_sysadmin · September 8, 2023, 4:41pm

It doesn’t look like your proposed commands will work for you tbh. You said:

ssh user@source zfs send -v -w -i thepool/data@ thepool/data@ | zfs receive backup01/data-new -v -n -o origin=backup01/data@

When I tried it out in practice, it failed:

root@elden:/# zfs create rpool/origintest
root@elden:/# zfs snapshot rpool/origintest@1
root@elden:/# zfs send rpool/origintest@1 | zfs receive rpool/targettest
root@elden:/# zfs snapshot rpool/origintest@2
root@elden:/# zfs send -I rpool/origintest@1 rpool/origintest@2 | zfs receive rpool/targettest
root@elden:/# zfs snapshot rpool/origintest@3
root@elden:/# zfs destroy rpool/origintest@1
root@elden:/# zfs destroy rpool/origintest@2
root@elden:/# zfs snapshot rpool/origintest@4
root@elden:/# zfs send -v -w -i rpool/origintest@3 rpool/origintest@4 | zfs receive rpool/targettest -v -n -o origin=rpool/targettest@2
send from rpool/origintest@3 to rpool/origintest@4 estimated size is 624B
total estimated size is 624
using provided clone origin rpool/targettest@2
cannot receive new filesystem stream: destination 'rpool/targettest' exists
must specify -F to overwrite it

So, that doesn’t look good… what happens if we use the -F flag?

root@elden:/# zfs send -v -w -i rpool/origintest@3 rpool/origintest@4 | zfs receive rpool/targettest -v -n -o origin=rpool/targettest@2 -F
send from rpool/origintest@3 to rpool/origintest@4 estimated size is 624B
total estimated size is 624
using provided clone origin rpool/targettest@2
cannot receive new filesystem stream: destination has snapshots (eg. rpool/targettest@2)
must destroy them to overwrite it

Perhaps there’s some way to get this done with a different command syntax, or with the version of OpenZFS in the master repos… but it doesn’t appear to be a functional approach with the syntax given on OpenZFS 2.1.5, at least.

muay_throwaway · January 3, 2024, 10:53am

I actually verified some of these details some time ago but hadn’t shared. In any case, destroying and recovering from backup would be the more straightforward solution. Or making sure that there is a bookmark prepared beforehand for a send/recv source. However, I thought it might be helpful to clarify details of nopwrite for future travelers just for reference purposes.

The nopwrite feature relies on a cryptographically secure (nondefault) hash algorithm and compression to effectively “deduplicate” at the receive level (not actually deduplication in te ZFS sense), so it would not work on a defaultly configured ZFS dataset. Presumably, with the more advanced hash algorithms, ZFS knows it is safe to assume that two data blocks having the same checksum correspond to the same data and can therefore save space during the receive. (This is similar to the requirements for deduplication without the same performance costs of deduplication.) With the default hash algorithm (fletcher4), there is better performance but a higher risk of collision, so this assumption is disallowed. Compression is also required, as ZFS avoids space-saving measures without it.

Thus, the nopwrite feature is available if a compatible checksum algorithm and compression were used. It allows one to create a de facto incremental clone based on an existing snapshot with common data blocks. This is somewhat esoteric and nondefault, but it may be a more widely used/acknowledged feature someday (similar to how compression is treated), namely if the default hash were to change.

The following verified that this works:

zfs set checksum=sha512 apool
zfs set compression=on apool
zfs set checksum=sha512 bpool
zfs set compression=on bpool
zfs create apool/fs
# I separately created a large lorem ipsum file and copied to apool/fs
cp -v lorem_2.txt /apool/fs/
# create snapshots
zfs snapshot apool/fs@a
zfs snapshot apool/fs@b
zfs snapshot apool/fs@c
# send/receive
zfs send apool/fs@b | zfs receive bpool/fs
zfs send -I apool/fs@b apool/fs@c | zfs receive bpool/fs
zfs send -v apool/fs@a | zfs receive -v -o origin=bpool/fs@c bpool/fs2

The cloned filesystem has only KiB-level space consumption:

NAME                   AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
apool                   803M  29.2M        0B     25K             0B      29.1M
apool/fs                803M  29.0M        0B   29.0M             0B         0B
bpool                   803M  29.2M        0B     25K             0B      29.2M
bpool/fs                803M  29.0M        0B   29.0M             0B         0B
bpool/fs2               803M    64K        0B     64K             0B         0B

This is a relatively new feature (so there is a greater risk of issues), but it may come in handy.