Hi all,
For whatever reason, I can’t seem to get --exclude-snaps to work.
I don’t use perl, but I know it’s kind of the kind of regex, so I assume that a standard regex should work.
I’m trying to avoid syncing my zfs-auto-snap_frequent and zfs-auto-snap_hourly snapshots.
I’ve tried --exclude-snaps=zfs-auto-snap_hourly --exclude-snaps=zfs-auto-snap_frequent, being careful with my hyphens and underscores, but they still get synced.
I thought perhaps I need a full match, so I changed to --exclude-snaps="^zfs-auto-snap_hourly.*", but I still get these snaps synced over.
Am I misunderstanding this option?
Given no --dry-run flag, is there a better way for me to debug this other than just waiting 15 minutes and re-running to see if the latest _frequent was sent?
TIA for any suggestions, much appreciated.
Full command:
syncoid \
--delete-target-snapshots \
--recvoptions="uo canmount=noauto" \
--exclude-datasets=cache \
--exclude-snaps="^zfs-auto-snap_frequent-.*" \
--exclude-snaps="^zfs-auto-snap_hourly-.*" \
-r \
--sshkey /my/key_ed25519 \
--sshport=42 \
--compress=none \
--exclude-datasets=local \
--no-privilege-elevation \
syncoid@ip_address:zroot tank/zfs-backups/hostname/zroot
I did not write the code for and have not used this particular argument, but I’d try getting rid of the special characters. See if --exclude-snaps="zfs-auto-snap_frequent" works.
That appears to do the trick in my testing:
root@elden:/# zfs list -rt snap testpool/source
NAME USED AVAIL REFER MOUNTPOINT
testpool/source@testing 0B - 24K -
testpool/source@excludeme 0B - 24K -
testpool/source@syncoid_elden_2024-12-16:18:44:42-GMT-05:00 0B - 24K -
Let’s try using syncoid to replicate, but exclude the “excludeme” snap:
root@elden:/# syncoid --exclude-snaps="excludeme" testpool/source testpool/target
INFO: Sending oldest full snapshot testpool/source@testing to new target filesystem testpool/target (~ 12 KB):
44.2KiB 0:00:00 [9.93MiB/s] [===============================================================================================================================================] 350%
INFO: --no-stream is omitted but snaps are filtered. Simulating -I with filtered snaps
INFO: Sending incremental testpool/source@testing ... syncoid_elden_2024-12-16:18:44:42-GMT-05:00 to testpool/target (~ 4 KB):
624 B 0:00:00 [ 143KiB/s] [====================> ] 15%
INFO: Sending incremental testpool/source@syncoid_elden_2024-12-16:18:44:42-GMT-05:00 ... syncoid_elden_2024-12-16:18:45:30-GMT-05:00 to testpool/target (~ 4 KB):
624 B 0:00:00 [ 163KiB/s] [====================> ] 15%
Looks good so far…
root@elden:/# zfs list -rt snap testpool/target
NAME USED AVAIL REFER MOUNTPOINT
testpool/target@testing 0B - 24K -
testpool/target@syncoid_elden_2024-12-16:18:44:42-GMT-05:00 0B - 24K -
testpool/target@syncoid_elden_2024-12-16:18:45:30-GMT-05:00 0B - 24K -
Huzzah? 
Thanks for your time. I could have been more clear, but what you suggested was my original attempt, which wasn’t working (which is why I tried to make it more regex-y).
For the below, I reverted back to --exclude-snaps="zfs-auto-snap_frequent-"
Pre:
# zfs list -t snapshot -r tank/zfs-backups/HOSTNAME | grep frequent
tank/zfs-backups/HOSTNAME/zroot@zfs-auto-snap_frequent-2024-12-16-2030 0B - 140K -
tank/zfs-backups/HOSTNAME/zroot@zfs-auto-snap_frequent-2024-12-16-2045 0B - 140K -
tank/zfs-backups/HOSTNAME/zroot@zfs-auto-snap_frequent-2024-12-16-2100 0B - 140K -
tank/zfs-backups/HOSTNAME/zroot@zfs-auto-snap_frequent-2024-12-16-2115 0B - 140K -
tank/zfs-backups/HOSTNAME/zroot/ROOT@zfs-auto-snap_frequent-2024-12-16-2030 0B - 140K -
tank/zfs-backups/HOSTNAME/zroot/ROOT@zfs-auto-snap_frequent-2024-12-16-2045 0B - 140K -
tank/zfs-backups/HOSTNAME/zroot/ROOT@zfs-auto-snap_frequent-2024-12-16-2100 0B - 140K -
tank/zfs-backups/HOSTNAME/zroot/ROOT@zfs-auto-snap_frequent-2024-12-16-2115 0B - 140K -
during:
Dec 16 17:13:28 nas syncoid[50526]: INFO: --no-stream is omitted but snaps are filtered. Simulating -I with filtered snaps
Dec 16 17:13:28 nas syncoid[50526]: INFO: Performing an incremental sync between 'syncoid_nas_2024-12-16:14:20:57-GMT-07:00' and 'zfs-auto-snap_hourly-2024-12-16-2217'
Dec 16 17:13:28 nas syncoid[50526]: INFO: Sync size: ~4 KB
Dec 16 17:13:29 nas syncoid[50526]: INFO: Performing an incremental sync between 'zfs-auto-snap_hourly-2024-12-16-2217' and 'zfs-auto-snap_frequent-2024-12-16-2315'
Dec 16 17:13:29 nas syncoid[50526]: INFO: Sync size: ~4 KB
Dec 16 17:13:30 nas syncoid[50526]: INFO: Performing an incremental sync between 'zfs-auto-snap_frequent-2024-12-16-2315' and 'zfs-auto-snap_hourly-2024-12-16-2317'
Dec 16 17:13:30 nas syncoid[50526]: INFO: Sync size: ~4 KB
Dec 16 17:13:30 nas syncoid[50526]: INFO: Performing an incremental sync between 'zfs-auto-snap_hourly-2024-12-16-2317' and 'zfs-auto-snap_frequent-2024-12-16-2330'
Dec 16 17:13:30 nas syncoid[50526]: INFO: Sync size: ~4 KB
Dec 16 17:13:31 nas syncoid[50526]: INFO: Performing an incremental sync between 'zfs-auto-snap_frequent-2024-12-16-2330' and 'zfs-auto-snap_frequent-2024-12-16-2345'
Dec 16 17:13:31 nas syncoid[50526]: INFO: Sync size: ~4 KB
Dec 16 17:13:31 nas syncoid[50526]: INFO: Performing an incremental sync between 'zfs-auto-snap_frequent-2024-12-16-2345' and 'zfs-auto-snap_frequent-2024-12-17-0000'
Dec 16 17:13:32 nas syncoid[50526]: INFO: Sync size: ~4 KB
Dec 16 17:13:32 nas syncoid[50526]: INFO: Performing an incremental sync between 'zfs-auto-snap_frequent-2024-12-17-0000' and 'syncoid_nas_2024-12-16:17:13:28-GMT-07:00'
Dec 16 17:13:32 nas syncoid[50526]: INFO: Sync size: ~4 KB
post:
# zfs list -t snapshot -r tank/zfs-backups/HOSTNAME | grep zfs-auto-snap_frequent
tank/zfs-backups/HOSTNAME/zroot@zfs-auto-snap_frequent-2024-12-16-2315 0B - 140K -
tank/zfs-backups/HOSTNAME/zroot@zfs-auto-snap_frequent-2024-12-16-2330 0B - 140K -
tank/zfs-backups/HOSTNAME/zroot@zfs-auto-snap_frequent-2024-12-16-2345 0B - 140K -
tank/zfs-backups/HOSTNAME/zroot@zfs-auto-snap_frequent-2024-12-17-0000 0B - 140K -
tank/zfs-backups/HOSTNAME/zroot/ROOT@zfs-auto-snap_frequent-2024-12-16-2315 0B - 140K -
tank/zfs-backups/HOSTNAME/zroot/ROOT@zfs-auto-snap_frequent-2024-12-16-2330 0B - 140K -
tank/zfs-backups/HOSTNAME/zroot/ROOT@zfs-auto-snap_frequent-2024-12-16-2345 0B - 140K -
tank/zfs-backups/HOSTNAME/zroot/ROOT@zfs-auto-snap_frequent-2024-12-17-0000 0B - 140K -
So clearly it’s still syncing them over for some reason.
Was running from 2259625b082af82b33959b7f51965573d4a3d889, will try updating to master.
1 Like
No luck updating to master.
Will try removing the double quotes around the snapshot name.
Master is what I’ve got on this system, and it’s working as intended as far as I can see:
root@elden:/# syncoid --exclude-snaps="zfs-auto-snap_frequent-" testpool/source testpool/target
INFO: Sending oldest full snapshot testpool/source@testing to new target filesystem testpool/target (~ 12 KB):
44.2KiB 0:00:00 [13.6MiB/s] [===============================================================================================================================================] 350%
INFO: --no-stream is omitted but snaps are filtered. Simulating -I with filtered snaps
INFO: Sending incremental testpool/source@testing ... excludeme to testpool/target (~ 4 KB):
624 B 0:00:00 [ 159KiB/s] [====================> ] 15%
INFO: Sending incremental testpool/source@excludeme ... final to testpool/target (~ 4 KB):
624 B 0:00:00 [ 149KiB/s] [====================> ] 15%
INFO: Sending incremental testpool/source@final ... syncoid_elden_2024-12-16:19:26:55-GMT-05:00 to testpool/target (~ 4 KB):
624 B 0:00:00 [ 227KiB/s] [====================> ] 15%
INFO: Sending incremental testpool/source@syncoid_elden_2024-12-16:19:26:55-GMT-05:00 ... syncoid_elden_2024-12-16:19:27:22-GMT-05:00 to testpool/target (~ 4 KB):
624 B 0:00:00 [ 162KiB/s] [====================> ] 15%
root@elden:/# zfs list -rt snap testpool/source | grep frequent
testpool/source@zfs-auto-snap_frequent-2024-12-16-2030 0B - 24K -
root@elden:/# zfs list -rt snap testpool/target | grep frequent
root@elden:/#
I’d advise double and triple checking your commands for typos. The mixture of dashes and underscores can be kinda tricky to navigate from memory. I speak from very direct experience, gained literally as I was doing this testing
1 Like
No doubt! I’ll keep debugging and report back when I eventually discover my silly mistake. Don’t think it’s a typo – I’ve been trying to sort this out on-and-off for 6 months or so, just finally got around to asking.
…and… before even hitting post, I think I got it. In my case, it was the double quotes.
I’m running NixOS, which automatically generates the systemd file from its configuration language.
In this case, it is single-quoting all of the flags / arguments, so --exclude-snaps=foo becomes '--exclude-snaps=foo' in my systemd service.
But that means that it is probably passing the double quotes literally: '--exclude-snaps="foo"'
That must be it. Both --exclude-snaps=^zfs-auto-snap_hourly.* and --exclude-snaps=zfs-auto-snap_hourly work fine (in my nix config), because they are getting enclosed in single quotes.
culprit: nixpkgs/nixos/modules/services/backup/syncoid.nix at 28b4ffa230761a1ee2d566ac376b77f5f35efad5 · NixOS/nixpkgs · GitHub
nix-repl> lib.escapeShellArgs [''--foo="bar"'']
"'--foo=\"bar\"'"
Why is it that you can waste 6 months on a problem, but as soon as you post about it you find the answer in a day?
Sorry to waste your time! And thanks for your responses and all the great 2.5A content.
1 Like
Not a waste at all! I learned something–however fragmentary–about NixOS today, and I know a lot of people are using my code on that platform, so, hey! I consider it a win for us both.
1 Like