Syncoid CRITICAL ERROR: no datasets found

kinghat · May 23, 2024, 3:55am

new to the *oids and im trying to set up my first replication between two hosts using the pull method and im getting:

$ syncoid --no-privilege-elevation --no-sync-snap --recursive send-server:services tank/backup/services
WARN: ZFS resume feature not available on source machine - sync will continue without resume support.
cannot open ‘send-server:services’: dataset does not exist
CRITICAL ERROR: no datasets found at /usr/sbin/syncoid line 144.

this is being initiated from backup-server:

$ zfs --version
zfs-2.1.6-0york1~20.04
zfs-kmod-2.1.6-0york1~20.04

$ syncoid --version
/usr/sbin/syncoid version 2.0.3
(Getopt::Long::GetOptions version 2.50; Perl version 5.30.0)

$ zfs list
NAME             USED  AVAIL     REFER  MOUNTPOINT
tank             677G  1.97T       24K  none
tank/backup       24K  1.97T       24K  none

the send-server deets:

$ zfs --version
zfs-2.1.5-1ubuntu6~22.04.4
zfs-kmod-2.1.5-1ubuntu6~22.04.3

$ syncoid --version
/usr/sbin/syncoid version 2.1.0
(Getopt::Long::GetOptions version 2.52; Perl version 5.34.0)

$ zfs list
NAME            USED  AVAIL     REFER  MOUNTPOINT
services       60.9G   400G       96K  /services
services/data  60.9G   400G     60.8G  /mnt/services/

both machines are user kinghat. i applied the following permissions on the receiving side and send side respectively:

backup-server:

sudo zfs allow -u kinghat compression,mountpoint,create,mount,receive,rollback,destroy tank/backup

send-server:

sudo zfs allow -u kinghat send,hold services

im probably just missing a piece to the puzzle but hopefully ive provided the proper information that someone can point it out. thanks for your time

mercenary_sysadmin · May 23, 2024, 4:53am

I am, to be honest, still a bit new and “fuck with it til it works” about ZFS delegation myself. As a troubleshooting step, would you mind using a root key and trying replication as root one time, to see if it works?

This is not me telling you “just be root,” I’m just asking you to do that as a bisection tool, so we can see whether that’s where the problem lies or not, before I start breaking my brain looking over old docs and trying to figure out whether that’s ACTUALLY where the problem lies.

kinghat · May 23, 2024, 5:58am

ok i setup the keys for root on both hosts and made sure i could ssh from backup-server to send-server and get the same thing:

root@backup-server:~# syncoid --no-sync-snap --recursive send-server:services tank/backup/services
WARN: ZFS resume feature not available on source machine - sync will continue without resume support.
cannot open ‘send-server:services’: dataset does not exist
CRITICAL ERROR: no datasets found at /usr/sbin/syncoid line 144.

i want to note that i havent yet created the dataset: tank/backup/services on backup-server yet as i thought i read you shouldnt before your first replication. also that im trying to replicate the pools root dataset recursively. hopefully those things are correct.

hernil · May 23, 2024, 8:16am

I see that your backup host has an older version of syncoid than your sender. Not saying that’s the/an issue, but in case you start troubleshooting the script it’s nice to be aware so you look at the correct lines of code in the correct version.

I would maybe try without the --no-sync-snap flag as well just to see if it gets going. Otherwise you have to be sure that there are snapshots of the services dataset (at least I’m pretty sure that’s the case).

I’ll just drop my own docs for doing this here in case you want to compare. They are heavily inspired by the Klara article on the topic.

Also trying a manual send/receive without syncoid in the middle might reveal more information?

mercenary_sysadmin · May 23, 2024, 1:50pm

Yes, correct.

Ok, next step: when you say this:

root@backup-server:~# syncoid --no-sync-snap --recursive send-server:services tank/backup/services

Is that the exact command, and are you relying on an /etc/hosts entry to resolve “send-server” to an IP address, or is that a substitution you’re typing in here in order to make it more clear which is which?

If that’s the actual command, please check your /etc/hosts to make certain that the alias resolves.

Whether that’s the actual command or not, please try specifying the username in your syncoid command, eg:

root@backup-server:~# syncoid --no-sync-snap --recursive root@send-server:services tank/backup/services

If you’re having trouble seeing the difference, it’s directly specifying root@send-server:services, as opposed to merely send-server:services.

kinghat · May 23, 2024, 2:53pm

im sorry for the confusion i was trying to make it more clear but also im using an ssh config file. on “backup-server”:

root@backup-server:~# sudo cat /root/.ssh/config
Host send-server
HostName send-server.lan
User root
IdentityFile /root/.ssh/id_ed25519

i tried it by specifying the user and i get this now:

root@backup-server:~# syncoid --no-sync-snap --recursive root@send-server.lan:services tank/backup/services
WARN: --no-sync-snap is set, and getnewestsnapshot() could not find any snapshots on source for current dataset. Continuing.
CRITICAL: no snapshots exist on source services, and you asked for --no-sync-snap.
NEWEST SNAPSHOT: autosnap_2024-05-23_14:00:06_hourly
INFO: Sending oldest full snapshot services/data@autosnap_2024-05-22_19:45:15_monthly (~ 42 KB) to new target filesystem:
44.2KiB 0:00:00 [2.02MiB/s] [============================================================================================================================================================] 103%
cannot open ‘tank/backup/services’: dataset does not exist
cannot receive new filesystem stream: unable to restore to destination
CRITICAL ERROR: ssh -S /tmp/syncoid-root-root@send-server.lan-1716475600 root@send-server.lan ’ zfs send ‘"’“‘services/data’”‘"’@‘"’“‘autosnap_2024-05-22_19:45:15_monthly’”‘"’ | lzop | mbuffer -q -s 128k -m 16M 2>/dev/null’ | mbuffer -q -s 128k -m 16M 2>/dev/null | lzop -dfc | pv -s 43632 | zfs receive -s -F ‘tank/backup/services/data’ failed: 256 at /usr/sbin/syncoid line 464.

reading through some issues: Randomly failing syncoid pulls with `CRITICAL ERROR ... failed: 256 at /usr/sbin/syncoid line 889.` · Issue #885 · jimsalterjrs/sanoid · GitHub could it be that my send-server:services is mounted and my backup-server:tank/backup isnt?

or it might be this: Syncoid failes initial send: CRITICAL ERROR · Issue #630 · jimsalterjrs/sanoid · GitHub where i have to create the tank/backup/services dataset before i can apply the proper allow permissions to it:

kinghat@backup-server:~$ sudo zfs allow -u kinghat compression,mountpoint,create,mount,receive,rollback,destroy tank/backup/services
cannot open ‘tank/backup/services’: dataset does not exist
Failed to open dataset: tank/backup/services

that was tried as user vs as root and root wouldnt have needed the permissions so probably not the issue.

your link isnt working for me.

this worked! i gave the 3 extra permissions(mount, snapshot, destroy) noted in the wiki and dropped the --no-sync-snap flag. is this a zfs or syncoid thing and should i open a bug or make a change to the wiki noting it?

kinghat@backup-server:~$ syncoid --no-privilege-elevation --recursive kinghat@send-server.lan:services tank/backup/services
INFO: Sending oldest full snapshot services@syncoid_backup-server_2024-05-23:11:02:03 (~ 42 KB) to new target filesystem:
44.2KiB 0:00:00 [2.81MiB/s] [============================================================================================================================================================] 103%
INFO: Sending oldest full snapshot services/data@autosnap_2024-05-22_19:45:15_monthly (~ 42 KB) to new target filesystem:
44.2KiB 0:00:00 [2.55MiB/s] [============================================================================================================================================================] 103%
INFO: Updating new target filesystem with incremental services/data@autosnap_2024-05-22_19:45:15_monthly … syncoid_backup-server_2024-05-23:11:02:04 (~ 65.0 GB):
65.2GiB 0:13:19 [83.4MiB/s] [===========================================================================================================================================================>] 100%

kinghat@backup-server:~$ zfs list
NAME                        USED  AVAIL     REFER  MOUNTPOINT
tank                        739G  1.91T       24K  none
tank/backup                61.0G  1.91T       24K  none
tank/backup/services       61.0G  1.91T       24K  none
tank/backup/services/data  61.0G  1.91T     60.9G  none

mercenary_sysadmin · May 23, 2024, 5:03pm

OK, now we’re back on familiar territory. Your remaining issue is that services does not exist as a dataset on the remote server, and has no snapshots on the source server. I recommend including services in your sanoid.conf on the source server, so that it gets regular snapshots. If you don’t keep anything in that dataset itself, only in children, that’s fine–just means the snapshots won’t take up any appreciable space.

There are several other potential workarounds, but that’s by FAR the cleanest. It’s not worth trying to chase “but I don’t ever want to snapshot this because it’s empty” as a target.

kinghat · May 23, 2024, 5:51pm

im currently just doing this config on the source:

[services/data]
        use_template = production
        recursive = zfs

should i just do a singular, top down snapshot? like:

[services]
        use_template = production
        recursive = zfs

mercenary_sysadmin · May 23, 2024, 5:53pm

Yep, that’s the way. I’d recommend recursive=yes over recursive=zfs, personally, though. recursive=zfs can cause you some weird issues if you add child datasets later, after you’ve already begun creating zfs-recursive snapshots on the parent and existing children.

hernil · May 23, 2024, 6:39pm

Sorry, fixed the link. Glad to see Jim got you sorted out!