Good evening,
I have syncoid backups going to more than one destination and as a consequence I see
could not find any snapshots to destroy; check snapshot names.
WARNING: zfs destroy 'pool/data/set'@syncoid_host_timestamp; zfs destroy 'pool/data/set'@syncoid_host_timestamp failed: 256 at /sbin/syncoid line 1380.
Or worse yet
CRITICAL ERROR: Target pool/data/set exists but has no snapshots matching with pool/data/set!
I dug into that a bit and uncovered the issue. In short, the syncoid operation to the second target resulted in removing the snapshots created when sending to the first target. Adding:
--identifier=,EXTRA/ can be used to disambiguate the snapshots between the two destinations. But snapshots for the “other” target accumulate and are not otherwise managed.
--no-stream causes the snapshots for the “other” target to not be transported.
Feel free to review this and point out possible problems or errors or better ways to manage this. I should probably test this between two hosts and not just pools on one host as host names are included in the snapshot names so they might matter.
One way to manage this (and it works for me) is something like this:
Host A (where your “live” data lives and needs backing up) - run sanoid to take (and prune) snapshots based whatever retention needs you have. The key is to ensure that pruning here happens such that there is enough overlap to ensure that syncoid is able to replicate the snapshots to Hosts B and C (your “backup” hosts). If you take snapshots just prior to replication, you can get away without this “common” overlap window.
Hosts B and C (“backup” hosts that maintain replicated snapshots) - run sanoid here to prune snapshots (and ONLY prune) on the replicated pool/datasets. Ensure that pruning always keeps some common snapshots so that sycoid from A to B (or C) will work properly - unless you snapshot before replication - see (1).
Run syncoid periodically to replicate snapshots from A to B and A to C (this can be push or pull based depending on your needs).
Run sanoid (to prune) on Hosts B and C on some schedule. Alternatively you can also run sanoid via SSH from some other host (including A) so that the prunes happen after the replications.
Depending on how you have setup your sanoid configuration files and location, you can have a single sanoid config on Host A that is also replicated (as part of the replication process) to Hosts B and C. You can use that common sanoid config across all hosts (may make things easier to maintain).
Hopefully you get some ideas to customize this to suit your workflow.
Thanks for the suggestion. My concern with using sanoid snapshots for syncoid backups was that snapshots could get out of sync. Some of my hosts can be off line for days and weeks and that opens the door for snapshots on the destination to be destroyed while they are still needed for a returning client. I believe that having syncoid directly manage the snapshots when the backups happen should prevent this.