Replicating snapshots via syncoid from unencrypted -> encrypted local -> encrypted offsite

OpenZFS Sanoid
1

Hello all, ZFS Newbie here. I’m sure this has been asked in many ways before…but I’ve been running ZFS in Proxmox for a while and moved my main storage to TrueNAS Scale. I’m finally getting my 2+1 backup strategy going and I want a sanity check on if what I’m trying to do is craziness or a reasonable setup. I have tried to do my research but I’m a bit stuck

The internets have already told me that it’s not possible to send/sync an unencrypted dataset to an untrusted (key not loaded) encrypted dataset using raw send. So I imagined/plagarized this scenario:

  • A - hot storage - unencrypted (raidz mirror)
  • B - local backup - encrypted (mounted with key)
  • C - remote backup - encrypted (untrusted/unmounted, no key)

B runs syncoid on A to pull snapshots, B runs syncoid to push raw snapshots to C.

As such, I used Jim’s wonderful advice from Ars and created some test datasets using raw files on 3 local machines to test the setup before I replicated my real data. I can pull the data from A to B, then push the encrypted dataset from B to C, but when I add more data to A and pull from B, I get this message:

NEWEST SNAPSHOT: syncoid_xeon_2024-06-22:22:39:49-GMT-05:00

CRITICAL ERROR: Target zoffsite/crypt1/evectra exists but has no snapshots matching with zbackup/crypt1/evectra!
                Replication to target would require destroying existing
                target. Cowardly refusing to destroy your existing target.

What am I doing wrong?

This is how I’m doing the initial sync:
(force because I don’t want to blow away the dataset every time I get snapshots out of sync)

B: syncoid -r --force --dumpsnaps root@storage.lan:zroot/evectra zbackup/crypt1/evectra
B: syncoid -r  --sendoptions=wrp --dumpsnaps --no-sync-snap  zbackup/crypt1/evectra root@offsite:zoffsite/crypt1/evectra

This is how I’m running the 2nd sync:

B: syncoid -r --dumpsnaps root@storage.lan:zroot/evectra zbackup/crypt1/evectra
B: ERROR, plus there aren't common snapshots between A and B anymore

I’ve tried adding --no-sync-snap to both commands, but I haven’t tried using a raw zfs send from B->C after A->B

I’d paste more detail, including the commands I used to create and encrypt the datasets on A, B, and C if needed, but this post is already too long.

As an addendum, if I can get past this, I already have an unencrypted sync of A->B and want to do a one-time mount and sync from A->C so I can be protected while I wipe B’s dataset sync it as encrypted; then I’ll do the A (une) → B (e) → C (e) sync. I’d love to not screw this up as C is going to a family members’ house in another state (losing a common snapshot would be nigh-impossible to resync in a timely fashion without shipping drives)

References:
Ars - using file vdevs to test ZFS
comment describing similar setup, but mostly discussing feasibility

2

If I remember correctly, it’s been mentioned that you can send from unencrypted to encrypted by setting up the target dataset as the child dataset of the target, and making the parent encrypted, with inheritance turned on.

I haven’t tested this myself.

I would also be careful about using that force flag in your syncoid commands. If it’s just for while you’re testing, fine, but I recommend against running that in production.

3

Just to confirm, you have decrypted/mounted B before running the 2nd send?

Also, do you mean you have compared the snapshots from A and B using

zfs list -t snapshot

and confirmed there are no longer any shared snapshots? Do you have sanoid-prune running on the source for those test datasets?

4

I hadn’t tried using child datasets. I’m trying to hit the “untrusted remote” use case.
I added --force only for testing because I didn’t want to recreate the datasets every time I found a way that wouldn’t work.

5

B is decrypted and mounted before the 2nd send.

I mostly fell back on --dumpsnaps because that lists the snapshots on src/dest and makes it easier to compare. I did use zfs list -t snapshot a couple of times.

I’m not using sanoid yet, I started with syncoid to do a proof of concept of the strategy.