Error syncing some datasets: "Invalid argument" from zfs send

OpenZFS
1

When trying to sync some datasets, syncoid fails because zfs send fails with the error “Invalid argument”. Example:

# zfs snapshot revivajxo2/scott/Videos@`date -I`
# zfs send -w revivajxo2/scott/Videos@`date -I` | mbuffer -q -s 128k -m 16M | pv -pterb | zfs receive -s -F revivajxo2_dup/scott/Videos
43.6GiB 0:04:04 [ 182MiB/s] [                                                                         <=>         ]
# syncoid --sendoptions w revivajxo2/scott/Videos revivajxo2_dup/scott/VideosSending incremental revivajxo2/scott/Videos@2024-10-29 ... syncoid_revivajxo_2024-10-29:08:04:00-GMT-04:00 (~ 4 KB):
warning: cannot send 'revivajxo2/scott/Videos@syncoid_revivajxo_2024-10-29:07:49:32-GMT-04:00': Invalid argument
2.53KiB 0:00:00 [6.11KiB/s] [=========================================>                          ]  63%            
cannot receive incremental stream: most recent snapshot of revivajxo2_dup/scott/Videos does not
match incremental source
CRITICAL ERROR:  zfs send -w  -I 'revivajxo2/scott/Videos'@'2024-10-29' 'revivajxo2/scott/Videos'@'syncoid_revivajxo_2024-10-29:08:04:00-GMT-04:00' | mbuffer  -q -s 128k -m 16M | pv -p -t -e -r -b -s 4096 |  zfs receive  -s -F 'revivajxo2_dup/scott/Videos' 2>&1 failed: 256 at /nix/store/7hg0qbn9j81zd5d982ifiwh4a0dkaxjj-sanoid-2.2.0/bin/.syncoid-wrapped line 889.
#  zfs send -w  -I 'revivajxo2/scott/Videos'@'2024-10-29' 'revivajxo2/scott/Videos'@'syncoid_revivajxo_2024-10-29:08:04:00-GMT-04:00' | mbuffer  -q -s 128k -m 16M | pv -p -t -e -r -b -s 4096 |  zfs receive  -s -F 'revivajxo2_dup/scott/Videos'
warning: cannot send 'revivajxo2/scott/Videos@syncoid_revivajxo_2024-10-29:07:49:32-GMT-04:00': Invalid argument
2.53KiB 0:00:00 [5.79KiB/s] [=========================================>                          ]  63%            
cannot receive incremental stream: most recent snapshot of revivajxo2_dup/scott/Videos does not
match incremental source

For background:

# zfs --version
zfs-2.2.6-1
zfs-kmod-2.2.4-1

# syncoid --version
/nix/store/7hg0qbn9j81zd5d982ifiwh4a0dkaxjj-sanoid-2.2.0/bin/.syncoid-wrapped version 2.2.0
(Getopt::Long::GetOptions version 2.54; Perl version 5.38.2)

I guess there’s a hint with “cannot receive incremental stream: most recent snapshot of revivajxo2_dup/scott/Videos does not match incremental source” but I’m not sure how to get it to match?

2

No, that’s your hint. ZFS receive doesn’t come into it, because you’re actually failing to send at all.

My first suspicion is a mismatch between the kernel module and the userland libraries, on the source system.

3

Okay, I’ll try updating and rebooting and see if that does anything, thanks.

1 Like
4

Looks like that fixed it, now we have

$ zfs --version           
zfs-2.2.6-1
zfs-kmod-2.2.6-1

The error has cleared up. Thanks, Jim.

1 Like