Corruption potentially triggered by syncoid (discussion on ZFS issue.))

OpenZFS Sanoid
1

Good evening,
I’d like to draw Jim’s attention to a discussion at ZFS corruption related to snapshots post-2.0.x upgrade · Issue #12014 · openzfs/zfs · GitHub regarding corruption. A number of commenters have identified that the issue seems to be triggered by syncoid and that is my experience as well. My situation is as follows:

I perform single filesystem transfers from an encrypted pool to a file system on a regular basis. (daily.) The target filesystem is not encrypted and I do not use raw sends. The commands are

for f in Programming Documents Archive
do
    time -p /sbin/syncoid --no-privilege-elevation --recursive \
        hbarta@rocinante:rpool/home/hbarta/$f \
        tank/srv/rocinante/$f
done

These backups never seem to provoke any problems.

From time to time I enable full pool backups to my desktop using the following command:

/bin/time -p /sbin/syncoid --recursive --no-privilege-elevation rocinante:rpool olive:ST8TB-ZA20HR7B/rocinante-backup/rpool

I’ve enabled this over about a year and eventually I see “permanent errors” and that prompts me to to disable this full pool backup. I tried this recently with ZVS 2.2.6 and the problem still happens. Between snapshots rollong off and scrubs, these permanent errors always go away. There doesn’t seem to be any issue with the receiving end and no errors are ever reported at the receiving end.

ISTR that syncoid doesn’t use a recursive send but walks the filesystem tree and sends each filesystem in a separate operation. My suspicion is that something about the way syncoid operates exposes a race condition in the underlying ZFS code. I also haven’t done anything to prevent concurrent operation of these scripts and/or sanoid snapshots, so that could be leading to the issue as well.

Anyway, if there is anything you can add to the conversation, please do so! This corruption bug has been around for too long.

2

Typically, that’s correct, because zfs-recursive sends and receives break in all KINDS of interesting ways when you go adding new datasets to one side or the other after you’ve already begun a replication chain.

If you think the difference between:

  • walking the tree and doing a zfs send -I on each dataset individually

and

  • doing a single zfs send -RI on the parent of them all

is the difference between corruption and no corruption, you can test that for yourself pretty easily, by using syncoid --sendoptions=R source target instead of syncoid -r source target.

Just be warned, if you’ve got no experience doing zfs-recursive sends and receives, you will encounter some weird and frustrating shit eventually when you start adding or removing datasets from one side or the other. :slight_smile:

1 Like
3

Thanks for the reply. I have no experience with “zfs send -RI on the parent of them all” for the most part since I generally don’t want to save all filesystems. (Perhaps only for ad-hoc backups.) My routine usage is either specific filesystems or using the syncoid recursive option. I have no desire to explore “weird and frustrating”. The latter (entire pool via syncoid recursive) seems to reveal the corruption bug.

best,

4

Well, I’m not seeing the bug regardless of usage, so I’m unable to help beyond pointing out a way to explore for certain whether walking the tree of datasets and replicating them in succession is triggering the bug or not.

1 Like