Good afternoon,
I’m running into a snapshot management issue on a stream whereby workstations backup to a local server and then the local server is backed up to a remote (all using syncoid of course.) The issue is that after the first “stream” (A → B → C) the second operation fails due to no common snapshot between B and C. It seems that the second syncoid backup from A to B destroys the snapshot created in the first backup of B → C.
I suspect there’s a combination of flags that will resolve this but after perusing the man page for syncoid, it’s not obvious to me which they should be. Even more “interesting” is that of four hosts that backup to the local server, only two exhibit this issue. I’ve developed a simple test case that can be run on a local host that demonstrates the problem: https://github.com/HankB/Fun-with-ZFS/tree/main/chained-nested-syncoid and the output is:
hbarta@rocinante:~/Programming/Fun-with-ZFS/chained-nested-syncoid$ ./chaining.sh
/sbin/sanoid version 2.3.0
(Getopt::Long::GetOptions version 2.57, Perl version 5.40.1)
filesystem successfully created, but it may only be mounted by root
INFO: --no-stream selected; sending newest full snapshot A/a@syncoid_A_to_B_rocinante_2026-06-18:16:43:12-GMT-05:00 to new target filesystem B/a: (~ 12 KB)
44.2KiB 0:00:00 [2.58MiB/s] [==================================================================================================================] 350%
cannot mount 'B/a': Insufficient privileges
snapshots after 1st syncoid A/a B/a
A/a@syncoid_A_to_B_rocinante_2026-06-18:16:43:12-GMT-05:00
B/a@syncoid_A_to_B_rocinante_2026-06-18:16:43:12-GMT-05:00
INFO: --no-stream selected; sending newest full snapshot B/a@syncoid_B_to_C_rocinante_2026-06-18:16:43:12-GMT-05:00 to new target filesystem C/a: (~ 12 KB)
44.2KiB 0:00:00 [2.46MiB/s] [==================================================================================================================] 350%
cannot mount 'C/a': Insufficient privileges
snapshots after 1st syncoid B/a C/a
A/a@syncoid_A_to_B_rocinante_2026-06-18:16:43:12-GMT-05:00
B/a@syncoid_A_to_B_rocinante_2026-06-18:16:43:12-GMT-05:00
B/a@syncoid_B_to_C_rocinante_2026-06-18:16:43:12-GMT-05:00
C/a@syncoid_B_to_C_rocinante_2026-06-18:16:43:12-GMT-05:00
INFO: Sending incremental A/a@syncoid_A_to_B_rocinante_2026-06-18:16:43:12-GMT-05:00 ... syncoid_A_to_B_rocinante_2026-06-18:16:43:14-GMT-05:00 to B/a (~ 4 KB):
624 B 0:00:00 [31.2KiB/s] [================> ] 15%
snapshots after 2nd syncoid A/a B/a
A/a@syncoid_A_to_B_rocinante_2026-06-18:16:43:14-GMT-05:00
B/a@syncoid_A_to_B_rocinante_2026-06-18:16:43:14-GMT-05:00
C/a@syncoid_B_to_C_rocinante_2026-06-18:16:43:12-GMT-05:00
CRITICAL ERROR:
Target C/a exists but has no snapshots matching with B/a!
Replication to target would require destroying existing
target. Cowardly refusing to destroy your existing target.
snapshots after 2nd syncoid B/a C/a
A/a@syncoid_A_to_B_rocinante_2026-06-18:16:43:14-GMT-05:00
B/a@syncoid_A_to_B_rocinante_2026-06-18:16:43:14-GMT-05:00
B/a@syncoid_B_to_C_rocinante_2026-06-18:16:43:15-GMT-05:00
C/a@syncoid_B_to_C_rocinante_2026-06-18:16:43:12-GMT-05:00
hbarta@rocinante:~/Programming/Fun-with-ZFS/chained-nested-syncoid$
The arguments I’m using for syncoid are --no-stream --identifier=[ID] --no-privilege-elevation and for the local to remote server I add --recursive. sudo is needed to run the example on Linux. (Not sure about BSD.)
Suggestions for what I’m doing wrong are most welcome.
I think if you want consistent replications & not snapshot spaghetti like you are experiencing, it’s best to snapshot A, then replicate A —> B, and then A —> C. This is what I do. Do these via cron job so your replications are always consistent.
Just putting out a couple of points that you could look into:
bookmarks - this should allow you to delete a snapshot but still run zfs sends. See --create-bookmark option in syncoid.
Make the deletion of snapshots (“prune” via sanoid) on B run AFTER all the replications (B->C) are done. Since you’re using --no-stream syncoid is not creating (or deleting) any snapshots. So something else is deleting them. Assuming that you’re using syncoid+sanoid, ensure that sanoid doesn’t delete any snapshots on B out of sequence (resulting in the inability to sync A->B or B-C).
Irrespective of bookmarks, the culprit is something deleting/pruning snapshots on B before the latest B->C replication is run.
You can’t rely on sync snapshots for A->B->C. Incoming replication from A will always wipe out sync snapshots locally created on B. That’s just how replication works.
If you want to use multiple targets, you need to create snapshots on A with sanoid (or similar orchestrator, if you really don’t like sanoid for some reason) and use the --no-sync-snap argument on syncoid for both A->B and B->C replication.
I see that as a fallback option. There’s more going on on the local server than just that including git server storage and photo storage. That complicates things because I can’t fully eliminate B → C. But I think it could work.
I think it might be helpful to run the scripts and see what it does. syncoid is most certainly creating snapshots. If it normally relies on existing snapshots, perhaps it is doing that in this case because they don’t otherwise exist. Perhaps that’s the difference between hosts where this works and doesn’t work. I need to look at that.
Is that a policy choice encoded in syncoid or is it a fundamental characteristic of ZFS? I’m trying to wrap my head around that before my first cup of coffee. I do have an example where this works in https://github.com/HankB/Fun-with-ZFS/blob/main/chained-nested/chaining.sh using zfs snap, zfs send and zfs recv. The second A → B send does not wipe out the first B → C snapshots. I could script something to automate that but I’d prefer to leverage someone else’ effort in that regard.
I’m not averse to using sanoid and --no-sync-snap but I am concerned about hosts that may not be available for days or weeks pruning the snapshots that syncoid chose and resulting in the same issue. Is that a possibility for hosts that are occasionally offline?
Fundamental. The snapshot tree cannot branch. It goes straight up, and absolutely no other way.
If you zfs send source/dataset@1 | zfs receive target/dataset, then you zfs snapshot target/dataset@lollerskates, then zfs snapshot source/dataset@2 ; zfs send -I source/dataset@1 source/dataset@2 | zfs receive target/dataset, the only way that can succeed is to destroy target/dataset@lollerskates.
That’s not a syncoid thing, that’s literally just how ZFS works.
With respect, no, you don’t. What you might have an example of is a setup where the sync snapshots are taken on A but not on B, and B replicates to C without taking a sync snapshot.
In that case, things are working as they must for replication–new snapshots are only taken on the origin, and then replicated from node to node downstream.
No matter what you do, you cannot take snapshots locally on downstream nodes and expect them to survive new incoming replication. No way, no how, not today, not tomorrow, not ever–by the time you redesigned ZFS sufficiently to allow branching snapshot trees, you might as well wrap the alphabet and call it AFS.
No stupid questions. And I shall try to avoid stupid answers.
My concern with that is that at times the remote is not available. It’s on a farm about 5 hours away. Because it’s out in the country it seems to be more susceptible to power outages which can last up to days. Also my son is doing considerable renovation at hos house there which might necessitate taking the server down from time to time.
My fear is that local backups will proceed and the remote ones will not, resulting in snapshots on the remote getting stale. Do you think this will not happen?
However, if I drive the backups from the remote server, if it is not available, the local backups will not happen. That solves the problem with snapshots getting out of sync but exposes me to no backups for my desktop and laptop and other local hosts.
Set up Sanoid on the source (production) host, using the production template. It will take local snapshots every hour, every day, and every month, keeping 30 of the former two categories and 3 of the last.
use Syncoid with --no-sync-snap to keep target (backup or hotspare) locations up to date. I generally recommend hourly replication if local (same site) or daily if offsite.
Sanoid on the target (backup or hotspare) machines should use the hotspare template if doing hourly replication, or the backup template if doing daily replication. Both templates give you the same 30/30/3 retention that the production template did, but they DON’T take snapshots locally–they only prune them locally when they become stale, and they are VERY CAREFUL: eg, if your oldest daily snapshot is 583 days old, it STILL won’t get pruned unless you’ve got 30 NEWER daily snapshots. Seriously… sanoid is safe as freaking HOUSES.
As long as your remote locations don’t go more than three full months without a successful replication, you’ll be fine. If you’re worried you might go three full months between successful replications, you’re going to need to keep more monthlies on the source (production), and/or use bookmarks on the source (which are created from snapshots, but still serve as viable replication points even if the associated snapshot is destroyed).
If you REALLY REALLY WANT TO, you CAN do A->B->C replication, but I do not recommend it. This leaves B as a single point of failure which can take out C as well. It’s generally safer and a better idea to do B<–A–>C (both B and C do pull replication directly from A, which makes them independent of one another in terms of getting backups).
If you’ve got any more direct questions after absorbing all of that, hit me up. =)
Many thanks Jim for providing the detailed suggestions for an A → B → C backup stream. I’ve been doing that for about a week now and it’s working w/out a hitch. Wife’s laptop is often suspended resulting in failed pulls but when she is on line, it catches up.
If it looks like either of ‘B’ or ‘C’ is going to be off line for longer than 3 months, I’ll update the sanoid monthly retention policies.
In an unrelated note, I’m still chasing the HDD reset issue on ‘B’. I’m close to pulling out my hair but hair is not yet on fire. I have sufficient H/W spares to bring something up in an emergency, but I’m cautiously optimistic that I’ve identified the culprit. I’m just not ready to declare victory until I haven’t seen the issue for at a week minimum.
