Problems with zfs send/receive: destination [...] has been modified since most recent snapshot

I was debating whether to post this in Proxmox or OpenZFS - it ended here - please move it to proxmox instead :slight_smile:

I am in the process of setting up my hyperconverged proxmox/storage box. I have a bunch of data in a ZFS pool (fatagnus) that I need to move to a temporary pool (tank) and then rebuild the new pool (fatagnus v2) from some of the original HDDs and move the data from the temp pool to the permanent pool. I am at stage one of the rebuild - moving data from the original pool to the temp pool.

The original fatagnus pool had only top-level datasets and I wanted to divi out especially the media dataset into sub datasets. This data has been rsync’ed. Now what is left is my home dataset.

I have managed to send/receive the initial snapshot and then do a zfs send -i fatagnus/home@alpha fatagnus/home@omega tank/home without problems - but it did not have the interim snapshots … So now I am trying to do zfs send -i fatagnus/home@alpha fatagnus/home@omega tank/home (after having done a zfs destroy tank/home and zfs send fatagnus/home@alpha) but it fails after a couple of interim snapshots with:

cannot receive incremental stream: destination tank/home has been modified since most recent snapshot .

I have even tried to zfs rollback to that interim snapshot and then zfs send -I interim omega but that also fails with the above error. I am not using either of the two pools as there are no containers or vms running yet.

What am I doing wrong?

Even something as simple as ls’ing a directory can update atimes, which in turn means updating metadata, which means receive will fail if not fed the -F argument.

It might be worth looking at using syncoid to orchestrate your replication. It fixes a lot of this sort of replication papercut.

Thanks @mercenary_sysadmin! The plan is to setup sanoid/syncoid when everything is moved to the v2 fatagnus pool. The error occurs during the zfs send/receive operation which throws me, and I am doing nothing in tank.

Maybe I just need to do -F to get it over with …