Syncoid trying to mount dataset with canmount=noauto

OverpricedEternity · December 7, 2024, 4:36pm

I have two syncoid jobs running on my main NAS machine, N
(real machine names elided for a false sense of privacy)

The first pulls the full filesystem from E, my elasticsearch/logstash/kibana
machine.

The second sends everything (including E’s backups) to a hot spare backup machine, B.

The setup looks like this:

E ← N → B

The systemd units look like the following:
syncoid-E.service

[Unit]
Description=Syncoid ZFS Recieve from E
Wants=syncoid-E.timer

[Service]
Type=oneshot
ExecStart=syncoid --no-privilege-elevation --no-sync-snap --sendoptions=p -r N@E:E-zroot N-pool/srv/machines/E

syncoid-B.service

[Unit]
Description=Syncoid ZFS Send to B
Wants=syncoid-B.timer

[Service]
Type=oneshot
ExecStart=syncoid --no-privilege-elevation --sendoptions=p -r N-pool/srv N@B:B-pool/srv

E is running ZFSBootMenu and has a dataset setup that closely follows their
recommended setup:

E# zfs list
NAME                     USED  AVAIL     REFER  MOUNTPOINT
E-zroot              3.62G   212G       96K  none
E-zroot/ROOT         3.57G   212G       96K  none
E-zroot/ROOT/debian  3.57G   212G     3.17G  /
E-zroot/home          204K   212G      140K  /home

Importantly, E’s datasets have the following canmount settings:

root@E:~# zfs get canmount E-zroot/ROOT/debian
NAME                    PROPERTY  VALUE     SOURCE
E-zroot/ROOT/debian  canmount  noauto    local

root@E:~# zfs get canmount E-zroot/home
NAME             PROPERTY  VALUE     SOURCE
E-zroot/home  canmount  on        default

When I first pulled E to N, I ended up in a situation where syncoid had mounted
over N’s /home, which I eventually resolved via crossing my fingers and rebooting,
then setting canmount=noauto on N-pool/srv/machines/E/home.

N# zfs get canmount N-pool/srv/machines/E/home
NAME                          PROPERTY  VALUE     SOURCE
N-pool/srv/machines/E/home  canmount  noauto    local

However, when N pushes to B, syncoid fails with the following error:

...
Dec 07 10:55:27 N syncoid[306103]: Sending incremental N-pool/srv/machines/E@autosnap_2024-12-06_03:00:05_hourly ... syncoid_N_2024-12-07:10:54:40-GMT-05:00 (~ 38 KB):
Dec 07 10:55:58 N syncoid[306107]: could not find any snapshots to destroy; check snapshot names.
Dec 07 10:55:58 N syncoid[306103]: WARNING: ssh     -S /tmp/syncoid-N@B-1733586221 N@B ' zfs destroy '"'"'B-pool/srv/machines/E'"'"'@syncoid_N_2024-12-07:09:08:47-GMT-05:00' failed: 256 at /usr/sbin/syncoid line 1380.
Dec 07 10:56:43 N syncoid[306103]: Sending incremental N-pool/srv/machines/E/ROOT@autosnap_2024-12-06_03:00:05_hourly ... syncoid_N_2024-12-07:10:55:59-GMT-05:00 (~ 37 KB):
Dec 07 10:56:44 N syncoid[310622]: cannot receive org.zfsbootmenu:commandline property on B-pool/srv/machines/E/ROOT: permission denied
[REPEATED MANY TIMES, ELIDED]
Dec 07 10:57:13 N syncoid[306107]: could not find any snapshots to destroy; check snapshot names.
Dec 07 10:57:13 N syncoid[306103]: WARNING: ssh     -S /tmp/syncoid-N@B-1733586221 N@B ' zfs destroy '"'"'B-pool/srv/machines/E/ROOT'"'"'@syncoid_N_2024-12-07:09:08:57-GMT-05:00' failed: 256 at /usr/sbin/syncoid line 1380.
Dec 07 10:57:58 N syncoid[306103]: Sending incremental N-pool/srv/machines/E/ROOT/debian@autosnap_2024-12-06_03:00:05_hourly ... syncoid_N_2024-12-07:10:57:14-GMT-05:00 (~ 558.8 MB):
Dec 07 10:58:51 N syncoid[311232]: cannot mount 'B-pool/srv/machines/E/ROOT/debian': Insufficient privileges
Dec 07 10:58:51 N syncoid[306103]: CRITICAL ERROR:  zfs send -p  -I 'N-pool/srv/machines/E/ROOT/debian'@'autosnap_2024-12-06_03:00:05_hourly' 'N-pool/srv/machines/E/ROOT/debian'@'syncoid_N_2024-12-07:10:57:14-GMT-05:00' | pv -p -t -e -r -b -s 585988952 | lzop  | mbuffer  -q -s 128k -m 16M 2>/dev/null | ssh     -S /tmp/syncoid-N@B-1733586221 N@B ' mbuffer  -q -s 128k -m 16M 2>/dev/null | lzop -dfc |  zfs receive  -s -F '"'"'B-pool/srv/machines/E/ROOT/debian'"'"' 2>&1' failed: 256 at /usr/sbin/syncoid line 817.
Dec 07 10:59:36 N syncoid[306103]: Sending incremental N-pool/srv/machines/E/home@autosnap_2024-12-06_03:00:05_hourly ... syncoid_N_2024-12-07:10:58:52-GMT-05:00 (~ 65 KB):
Dec 07 11:00:02 N syncoid[311878]: cannot mount 'B-pool/srv/machines/E/home': Insufficient privileges
Dec 07 11:00:02 N syncoid[306103]: CRITICAL ERROR:  zfs send -p  -I 'N-pool/srv/machines/E/home'@'autosnap_2024-12-06_03:00:05_hourly' 'N-pool/srv/machines/E/home'@'syncoid_N_2024-12-07:10:58:52-GMT-05:00' | pv -p -t -e -r -b -s 67360 | lzop  | mbuffer  -q -s 128k -m 16M 2>/dev/null | ssh     -S /tmp/syncoid-N@B-1733586221 N@B ' mbuffer  -q -s 128k -m 16M 2>/dev/null | lzop -dfc |  zfs receive  -s -F '"'"'B-pool/srv/machines/E/home'"'"' 2>&1' failed: 256 at /usr/sbin/syncoid line 817.

Permissions on N’s send user on B:

B# zfs allow B-pool/srv
---- Permissions on B-pool/srv ----------------------------------------
Local+Descendent permissions:
	user N acltype,canmount,compression,create,destroy,mount,mountpoint,quota,receive,refquota,relatime,rollback,xattr

canmount on B:

B# zfs get canmount B-pool/srv/machines/E/ROOT/debian
NAME                                 PROPERTY  VALUE     SOURCE
B-pool/srv/machines/E/ROOT/debian  canmount  noauto    local

B# zfs get canmount B-pool/srv/machines/E/home
NAME                          PROPERTY  VALUE     SOURCE
B-pool/srv/machines/E/home  canmount  noauto    local

How can I ensure that syncoid does not try to mount any of E’s datasets on B?
Do I need to setup a separate dataset (instead of using srv) for zfsbootmenu
machines (I intend to have more in the future)?

How can I avoid the property error with org.zfsbootmenu:commandline?
I was able to share all the normal ZFS properties as distinct permissions,
but I couldn’t find a permission for properties more generally.

mercenary_sysadmin · December 7, 2024, 7:36pm

Syncoid hasn’t attempted to mount, unmount, or change read-only properties in datasets for a very long time; how old is the version that you have? If it’s not several years old, there’s probably some other root cause for the mount behavior you’re seeing.

This is what you get if you grep mount syncoid:

	# until the filesystem is zfs umounted and zfs remounted.
		# dyking this functionality out for the time being due to buggy mount/unmount behavior
				# dyking this functionality out for the time being due to buggy mount/unmount behavior
				# dyking this functionality out for the time being due to buggy mount/unmount behavior
			# dyking this functionality out for the time being due to buggy mount/unmount behavior
				# dyking this functionality out for the time being due to buggy mount/unmount behavior

… nothing but comments. No actual mount or unmount commands.

OverpricedEternity · December 7, 2024, 7:57pm

Weird! Both B and N are running syncoid 2.1.0 as distributed in Debian 12, which does look to be a few years old, but appears to have the same lack of references to mounting.

N:~$ dpkg -l sanoid
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version      Architecture Description
+++-==============-============-============-==========================================================
ii  sanoid         2.1.0-1.1    all          Policy-driven ZFS snapshot management and replication tool

N:~$ /usr/sbin/syncoid --version
/usr/sbin/syncoid version 2.1.0
(Getopt::Long::GetOptions version 2.52; Perl version 5.36.0)
N:~$ grep mount /usr/sbin/syncoid
	# until the filesystem is zfs umounted and zfs remounted.
		# dyking this functionality out for the time being due to buggy mount/unmount behavior
				# dyking this functionality out for the time being due to buggy mount/unmount behavior
				# dyking this functionality out for the time being due to buggy mount/unmount behavior
			# dyking this functionality out for the time being due to buggy mount/unmount behavior
				# dyking this functionality out for the time being due to buggy mount/unmount behavior

OverpricedEternity · December 7, 2024, 8:30pm

Ok turns out the answer is pretty simple, and not syncoid’s fault at all.
It appears zfs recv defaults to mounting any filesystems it receives, and this behavior can be disabled simply by passing -u to zfs recv.
A simple --recvoptions=u in my syncoid-B.service fixed the issue.

I’m not entirely sure why zfs recv doesn’t respect canmount but that’s a problem for another day, that side of this issue is resolved.

I would still like to copy over the org.zfsbootmenu:commandline property if there’s some permission I can grant my receive user to let it copy all properties when sending.