Zfs sanoid and syncoid question

OpenZFS
1

Hi All,

I have a question and i am hoping someone can help me with this. I have a source and a target machine. I use sanoid to take snapshots and syncoid to replicate it to the target machine.

On my source machine i have some vm’s running. Now when i do a zfs rollback -r on the source machine i am always creating a problem for the target machine.

Syncoid wont sync anymore because my source machine is missing some snapshots. How can i prevent this? or how can i fix this so that syncoid will run again.

2

Sounds like you need more archive depth.

Replication requires a common snapshot. So, if you destroy all of your common snapshots–such as by rolling back to a snapshot that’s older than your oldest common snapshot–you’ll break the chain, and the only way out is to destroy the target (or rename it) and begin again with a full replication.

To avoid this, maintain a greater archive depth on both sides. If you keep, for example, 30 dailies on both sides–repeat, both sides–you’d have to roll back to some snapshot even older than that before you broke the chain.

So for example, you encounter a problem, and you roll back a week on the source. Your next replication gets based on the snapshot from eight days ago, which is now the most recent common snapshot–so the target itself effectively rolls back to that eight day old snapshot automatically, the next time you replicate in.

If you don’t want that to happen either, you need to make some special arrangements after rolling back on the source, before its next replication to the target.

Alternately, avoid rolling back at all, and use a tool like rsync --inplace against the snapshot you’d like to “roll back” to. It will be slower than a rollback, but it will simply patch the existing files in the existing filesystem with each block that’s different in the snapshot, leaving you with a completely unbroken chain and no missing data on EITHER side.

zfs rollback -r pool/ds@old – instantaneous but destructive; all snapshots newer than @old irrevocably destroyed; replication will destroy those snapshots on target as well (assuming a common snapshot still exists)

zfs snapshot pool/ds@before; zfs clone pool/ds@old pool/clonetmp ; rsync -hav --inplace --delete --progress /pool/clonetmp/ /pool/ds/ ; zfs destroy pool/clonetmp – slower, patches files in place with changed blocks from the referenced snapshot. Does not destroy snapshots on either source OR target.

So for example say you got hit with ransomware: a rollback instantly repairs the damage but permanently destroys the altered data, making it impossible to investigate HOW you got ransomwared unless you shell into your target to salvage that data before the next incoming replication wipes it out.

But an rsync --inplace with the steps shown above, although potentially much much slower, repairs your data while leaving copies of the ransomwared data intact–both in any automatically taken snapshots while the ransomware was active, AND in the manually created snapshot you took @before… And upon your next replication, @before will be backed up to the target as well as any new snapshots you take afterward.

3

Hi,

Thank you for your reply. So just to understand what happens.
I do raw send recursive (syncoid)
When i do a rollback on the source machine (say 2 days) and i will do nothng else will my target machine remove the snapshots so its the same as the source machine?

4

Sort of, but not quite.

Replication works like this:

  • find the most recent common snapshot (a snapshot present on both machines)
  • send a list of all blocks more recent than that snapshot’s creation date to the remote machine, along with snapshots they belong to
  • on the remote machine, take all those new blocks and add them in
  • now the remote machine has everything the local machine did

OK, so that’s how replication works in a nutshell. But notice that very first step: find the most recent common snapshot and base from that.

The next thing to understand is, what happens if there are changes present on the remote machine that are newer than that most recent common snapshot? Well, if you want to replicate, the remote machine must effectively roll itself back to that most recent common snapshot first. (It’s not, technically, a rollback. But it has the same effect: snapshots and blocks newer than the ones in the common snapshot must all be destroyed in the process.)

So: if you roll back your source machine, you’ll effectively have to roll back your target machine also. If you roll back your source machine to a snapshot that no longer exists on the target, you will not have any common snapshots any more, and incremental replication will be broken until you begin again with another full.

If you don’t want those things to happen, your choices are to either find an alternative to literal rollbacks (like the rsync method I explained), or take action to prevent losing data on the target BEFORE the next replication occurs. For example, you might make a separate copy of the dataset on the target, so that the data destroyed with the incoming, post-rollback replication is still available elsewhere. Or you might rename the dataset on the target entirely, then just let a brand-new full replication happen on the next attempt, then eventually manually destroy your renamed original copy later, when you were certain you’d no longer need the data in the snapshots you’d rolled back past on the source.

5

Thank you so much for your explanation! my final question. When i do this

zfs snapshot pool/ds@before; zfs clone pool/ds@old pool/clonetmp ; rsync -hav --inplace --delete --progress /pool/clonetmp/ /pool/ds/ ; zfs destroy pool/clonetmp

you are telling me it wont destroy snapshots. So if i understand it correct i can also revert back to the latest snapshot? so undo it again i mean.

Could you please explain what the command does exactly? you create a snapshot. then you clone the existing pool. rsync the changes back to the pool. then delete the clone.

am i correct or am i total wrong.

Again thank you for the info you are giving me. i learn a lot from this.

6

zfs snapshot pool/ds@before – this creates a snapshot of the dataset name ds, under the pool named pool, and names that snapshot “before,” short for “before I started messing with this.” This preserves the data that you’re “rolling back,” which would otherwise be gone for good after either rolling back for real, or otherwise replacing your current data with the old data from an old snapshot (see next step).

zfs clone pool/ds@old pool/clonetmp – this presumes that you have a snapshot of pool/ds named before, which was already taken before whatever happened that made you want to roll back in the first place. It clones that snapshot into a new dataset named 'clonetmp`. This new dataset can be read or written to, and contains exactly what the snapshot itself did, in the condition that snapshot was in.

rsync -hav --inplace --delete --progress /pool/clonetmp/ /pool/ds/ – this isn’t ZFS specific at all. This invocation of rsync will compare the contents of pool/clonetmp and pool/ds on a block by block level. When it finds differences, it will update the contents of pool/ds to match what’s in pool/clonetmp, and when it finds differences, it will try to change ONLY the blocks that need changing, instead of rsync’s default behavior (which would replace entire files completely). Doing it this was keeps the storage space used by your snapshots low–without --inplace, changing a single 4K block in a 10G file would mean rewriting the entire file, but by using --inplace, we only have to write the one 4K block! This will still take a while to run if you have endless small files or a few enormous files, though, because it has to read every single block of every file that has been changed.

zfs destroy pool/clonetmp – this just destroys the temporary clone that you created so that you could rsync from the snapshot back into pool/ds. You don’t want to leave it lying around once you’re done with it; as long as the clone exists, the parent snapshot it was cloned from cannot be deleted.

====================================

If you do all these steps, you have essentially the same outcome as a rollback–everything goes back to just how it was in the older snapshot–but you don’t lose any data in the process. Nothing gets destroyed on the source or rolled back on the target; you have all of your existing snapshots PLUS the new one that you created in the first step.

That new snapshot you took manually will also be replicated to the target, the next time you replicate. You’ll eventually need to manually destroy that snapshot yourself; sanoid won’t destroy it for you since it’s not a snapshot taken or managed by sanoid.

Everything clear now? This is something I would advise practicing BEFORE you need it for real. :slight_smile:

7

Hi,.
Yes perfect thank you for this information. I will test this with a test machine.

But i could skip the creation of a new anapshot and use the snapshot that sanoid allredeay created for me and yes that instead. Am i wright.