Syncoid target "does not match incremental source"

pgesting · June 3, 2024, 9:39am

I have a source that sends sanoid snaps to a target. Two targets actually.

For two of my zfs datasets, I have been getting errors when syncoid tries to replicate to the target. And it is happening for the same two data sets. I do not think it has replicated successfully for about 3 weeks. At this point, there are no daily or hourly snapshots in common between production and backup. But there are still about 10 snapshots in common (monthlies, yearly, etc).

When I try to run syncoid, I get the following error:

Sending incremental rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-05-05_00:00:21_daily ... autosnap_2024-06-03_09:01:06_hourly (~ 142.7 GB):
warning: cannot send 'rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-05-06_00:00:09_daily': Input/output error
cannot receive incremental stream: most recent snapshot of backup/encrypt/rpool/data/encrypt_lxc/subvol-109-disk-0 does not                                 
match incremental source                                                                                                                                    
mbuffer: error: outputThread: error writing to <stdout> at offset 0x2dc000: Broken pipe                                                                     
mbuffer: warning: error during output to <stdout>: Broken pipe                                                                                              
27.6MiB 0:00:00 [74.4MiB/s] [>                                                                                                             ]  0%            
CRITICAL ERROR:  zfs send  -I 'rpool/data/encrypt_lxc/subvol-109-disk-0'@'autosnap_2024-05-05_00:00:21_daily' 'rpool/data/encrypt_lxc/subvol-109-disk-0'@'autosnap_2024-06-03_09:01:06_hourly' | pv -p -t -e -r -b -s 153252971272 | lzop  | mbuffer  -q -s 128k -m 16M 2>/dev/null | ssh     -S /tmp/syncoid-pgesting@192.168.1.85-1717406808 pgesting@192.168.1.85 ' mbuffer  -q -s 128k -m 16M 2>/dev/null | lzop -dfc |  zfs receive  -s -F '"'"'backup/encrypt/rpool/data/encrypt_lxc/subvol-109-disk-0'"'"' 2>&1' failed: 256 at /usr/sbin/syncoid line 817.

Looks like it says that the erorr is that the most recent snapshot on the backup does not match the incremental source. Well, that is true. The most recent snapshot on the backup is an hourly backup that has since been pruned in production. But that should be fine, correct? Shouldn’t syncoid go look at the most recent snapshot that is in common?

Here are all the snapshots on production:

NAME                                                                            USED  AVAIL     REFER  MOUNTPOINT
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2023-12-01_00:00:25_monthly  4.42G      -     6.15G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-01-01_00:00:16_monthly  4.05G      -     5.92G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-02-01_00:00:16_monthly  4.75G      -     7.38G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-03-01_00:00:15_monthly  2.98G      -     7.22G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-04-01_00:00:18_monthly  2.38G      -     7.50G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-05-01_00:00:20_monthly  1.61G      -     6.92G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-05-05_00:00:21_daily    1.77G      -     7.24G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-05-06_00:00:09_daily    1.38G      -     6.85G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-05-07_00:00:14_daily    1.76G      -     7.23G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-05-08_00:00:28_daily    1.37G      -     6.84G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-05-09_00:00:05_daily    1.37G      -     6.84G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-05-10_00:00:16_daily    1.76G      -     7.22G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-05-11_00:00:55_daily     130M      -     6.84G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-05-12_00:01:04_daily    27.2M      -     6.74G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-05-13_00:00:04_daily    1.35G      -     6.82G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-05-14_00:00:03_daily    1.76G      -     7.22G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-05-15_00:00:06_daily    1.37G      -     6.83G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-05-16_00:01:30_daily    1.37G      -     6.83G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-05-17_00:00:17_daily    1.37G      -     6.84G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-05-18_00:00:09_daily    1.38G      -     6.85G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-05-19_00:00:07_daily    1.75G      -     7.22G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-05-20_00:00:09_daily     767M      -     7.21G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-05-21_00:01:07_daily    2.61G      -     6.88G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-05-22_00:00:58_daily    3.57G      -     7.42G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-05-23_00:00:04_daily    4.62G      -     7.69G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-05-24_00:00:04_daily    1.90G      -     7.99G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-05-25_00:00:07_daily    1.87G      -     7.99G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-05-26_00:00:06_daily    1.85G      -     7.99G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-05-27_00:00:04_daily    1.85G      -     7.99G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-05-28_00:01:12_daily    4.93G      -     8.13G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-05-29_00:00:07_daily    1.84G      -     8.10G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-05-30_00:01:18_daily    1.83G      -     8.10G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-05-31_00:01:02_daily    1.83G      -     8.09G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-06-01_00:01:07_monthly     0B      -     8.00G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-06-01_00:01:07_daily       0B      -     8.00G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-06-01_22:00:12_hourly   2.26M      -     8.01G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-06-01_23:00:07_hourly   2.63M      -     8.01G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-06-02_00:00:55_daily       0B      -     8.02G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-06-02_00:00:55_hourly      0B      -     8.02G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-06-02_01:00:07_hourly   1.96M      -     8.02G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-06-02_02:00:06_hourly   1.96M      -     8.02G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-06-02_03:00:04_hourly   2.07M      -     8.02G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-06-02_04:00:05_hourly   2.24M      -     8.02G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-06-02_05:00:05_hourly   1.72M      -     8.02G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-06-02_06:00:01_hourly   1.88M      -     8.02G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-06-02_07:00:05_hourly   2.09M      -     8.02G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-06-02_08:00:03_hourly   2.12M      -     8.02G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-06-02_09:00:04_hourly   2.08M      -     8.02G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-06-02_10:00:08_hourly   2.07M      -     8.02G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-06-02_11:00:09_hourly   1.89M      -     8.02G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-06-02_12:00:06_hourly   1.64M      -     8.02G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-06-02_13:00:01_hourly   1.85M      -     8.02G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-06-02_14:00:07_hourly   2.15M      -     8.02G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-06-02_15:00:05_hourly   1.98M      -     8.02G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-06-02_16:00:04_hourly   1.96M      -     8.02G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-06-02_17:00:10_hourly   1.89M      -     8.02G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-06-02_18:00:05_hourly   1.88M      -     8.02G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-06-02_19:00:03_hourly   2.11M      -     8.02G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-06-02_20:00:06_hourly   2.09M      -     8.02G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-06-02_21:00:05_hourly   2.09M      -     8.02G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-06-02_22:00:05_hourly   2.15M      -     8.02G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-06-02_23:00:59_hourly   5.45M      -     8.03G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-06-03_00:01:24_daily     956K      -     8.16G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-06-03_00:01:24_hourly    892K      -     8.16G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-06-03_01:00:59_hourly   2.15M      -     8.05G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-06-03_02:01:13_hourly   2.31M      -     8.05G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-06-03_03:00:06_hourly   2.16M      -     8.05G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-06-03_04:00:07_hourly   2.20M      -     8.05G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-06-03_05:01:42_hourly   2.21M      -     8.05G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-06-03_06:01:38_hourly   2.22M      -     8.05G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-06-03_07:00:08_hourly   2.25M      -     8.06G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-06-03_08:01:46_hourly   2.14M      -     8.06G  -
rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-06-03_09:01:06_hourly   1.82M      -     8.06G  -

Here are all the snapshots on backup:

NAME                                                                                                USED  AVAIL     REFER  MOUNTPOINT
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2023-06-01_18:21:31_monthly  4.27G      -     5.85G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2023-07-10_08:48:24_monthly  7.35G      -     9.22G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2023-08-01_00:00:19_monthly  3.63G      -     5.98G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2023-09-01_00:00:05_monthly  3.56G      -     6.12G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2023-10-04_15:41:22_monthly  4.23G      -     6.75G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2023-11-01_00:00:53_monthly  3.22G      -     6.85G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2023-12-01_00:00:25_monthly  3.64G      -     7.34G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-01-01_00:00:16_monthly  4.55G      -     7.11G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-02-01_00:00:16_monthly  2.72G      -     9.08G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-02-06_00:00:19_daily    2.49G      -     9.08G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-02-07_00:00:04_daily    2.37G      -     9.40G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-02-08_00:00:11_daily    1.86G      -     8.89G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-02-09_00:00:06_daily    2.38G      -     9.39G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-02-10_00:00:07_daily    1.85G      -     8.89G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-02-11_00:02:37_daily    2.24G      -     9.28G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-02-12_00:00:13_daily    1.85G      -     8.89G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-02-13_00:00:11_daily    1.86G      -     8.90G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-02-14_00:00:15_daily    2.37G      -     9.41G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-02-15_00:00:16_daily    1.85G      -     8.89G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-02-16_00:00:25_daily    1.85G      -     8.89G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-02-17_00:00:20_daily    1.85G      -     8.89G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-02-18_00:01:00_daily    1.87G      -     8.90G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-02-19_00:00:14_daily     401M      -     8.92G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-02-20_00:00:53_daily    2.98G      -     8.97G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-02-21_00:00:04_daily    2.95G      -     8.99G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-02-22_00:01:12_daily    2.86G      -     8.91G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-02-23_00:00:23_daily    4.58G      -     9.11G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-02-24_00:00:55_daily    4.54G      -     9.12G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-02-25_00:00:11_daily    2.40G      -     9.55G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-02-26_00:00:02_daily    1.10G      -     9.50G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-02-27_00:01:05_daily     734M      -     8.96G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-02-28_00:00:10_daily    2.45G      -     9.57G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-02-29_00:00:17_daily    2.45G      -     9.55G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-03-01_00:00:15_monthly  23.0M      -     9.07G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-03-01_00:00:15_daily    23.1M      -     9.07G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-03-02_00:00:11_daily    1.91G      -     9.07G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-03-03_00:00:24_daily    2.30G      -     9.46G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-03-04_00:00:13_daily    2.42G      -     9.57G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-03-05_00:00:15_daily    2.41G      -     9.54G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-03-06_00:00:21_daily    1.90G      -     9.04G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-03-07_00:00:19_daily    2.36G      -     9.51G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-03-08_00:01:11_daily    2.40G      -     9.55G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-03-09_00:00:11_daily    1.89G      -     9.05G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-03-10_00:00:55_daily    2.41G      -     9.57G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-03-11_00:00:10_daily    2.37G      -     9.52G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-03-12_00:00:05_daily    1.87G      -     9.02G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-03-13_00:00:14_daily    2.34G      -     9.49G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-03-14_00:01:13_daily    2.35G      -     9.49G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-03-15_00:00:03_daily    2.34G      -     9.49G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-03-16_00:00:07_daily    1.86G      -     9.01G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-03-17_00:00:16_daily    2.33G      -     9.47G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-03-18_00:00:09_daily    2.33G      -     9.47G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-03-19_00:00:06_daily    2.34G      -     9.46G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-03-20_00:00:14_daily    2.32G      -     9.45G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-03-21_00:00:13_daily    1.84G      -     8.97G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-03-22_00:00:18_daily    1.84G      -     8.97G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-03-23_00:00:12_daily    2.32G      -     9.43G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-03-24_00:00:03_daily    1.85G      -     8.97G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-03-25_00:00:14_daily    2.33G      -     9.44G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-03-26_00:00:23_daily    1.84G      -     8.95G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-03-27_00:00:19_daily    2.31G      -     9.43G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-03-28_00:00:14_daily    2.30G      -     9.41G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-03-29_00:00:05_daily    2.14G      -     9.25G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-03-30_00:00:28_daily    1.81G      -     8.92G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-03-31_00:00:24_daily    1.82G      -     8.94G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-04-01_00:00:18_monthly  14.6M      -     9.41G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-04-01_00:00:18_daily    14.6M      -     9.41G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-04-02_00:00:08_daily    2.28G      -     9.39G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-04-03_00:00:06_daily    2.27G      -     9.38G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-04-04_00:00:20_daily    2.29G      -     9.38G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-04-05_00:00:08_daily    2.26G      -     9.37G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-04-06_00:00:56_daily    2.25G      -     9.36G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-04-07_00:00:20_daily    1.77G      -     8.88G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-04-08_00:00:21_daily    1.77G      -     8.88G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-04-09_00:00:19_daily    2.24G      -     9.34G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-04-10_00:00:13_daily    2.24G      -     9.34G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-04-11_00:00:22_daily    1.75G      -     8.86G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-04-12_00:00:22_daily    2.22G      -     9.32G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-04-13_00:01:07_daily    2.16G      -     9.32G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-04-14_00:00:45_daily    2.02G      -     9.19G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-04-15_00:00:12_daily    1.74G      -     8.85G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-04-16_00:00:23_daily    2.19G      -     9.30G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-04-17_00:00:13_daily    2.18G      -     9.28G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-04-18_00:00:18_daily    2.17G      -     9.27G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-04-19_00:00:13_daily    2.15G      -     9.25G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-04-20_00:00:09_daily    1.72G      -     8.83G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-04-21_00:00:21_daily    1.70G      -     8.81G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-04-22_00:00:16_daily    2.13G      -     9.23G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-04-23_00:00:20_daily    1.70G      -     8.80G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-04-24_00:00:07_daily    1.69G      -     8.79G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-04-25_00:00:06_daily    2.13G      -     9.23G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-04-26_00:00:12_daily    2.11G      -     9.20G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-04-27_00:00:08_daily    1.69G      -     8.79G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-04-28_00:00:23_daily    2.10G      -     9.19G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-04-29_00:00:20_daily    2.09G      -     9.18G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-04-30_00:00:14_daily    2.08G      -     9.16G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-05-01_00:00:20_monthly  6.05M      -     8.79G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-05-01_00:00:20_daily    6.07M      -     8.79G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-05-02_00:00:16_daily     171M      -     8.75G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-05-02_02:00:09_hourly   4.05M      -     8.62G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-05-02_03:00:05_hourly   3.66M      -     8.62G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-05-02_04:00:05_hourly   3.92M      -     8.62G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-05-02_05:01:09_hourly   3.62M      -     8.62G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-05-02_06:00:04_hourly   3.67M      -     8.62G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-05-02_07:00:06_hourly   4.67M      -     8.62G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-05-02_08:00:04_hourly   4.70M      -     8.62G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-05-02_09:00:08_hourly   3.64M      -     8.63G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-05-02_10:00:04_hourly   3.98M      -     8.63G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-05-02_11:00:07_hourly   3.44M      -     8.63G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-05-02_12:00:05_hourly   3.44M      -     8.63G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-05-02_13:00:02_hourly   3.25M      -     8.63G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-05-02_14:00:00_hourly   3.27M      -     8.63G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-05-02_15:00:01_hourly   3.79M      -     8.63G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-05-02_16:00:04_hourly   3.88M      -     8.63G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-05-02_17:00:07_hourly   3.68M      -     8.63G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-05-02_18:00:06_hourly   3.76M      -     8.63G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-05-02_19:00:07_hourly   3.47M      -     8.63G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-05-02_20:00:04_hourly   3.79M      -     8.63G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-05-02_21:00:04_hourly   3.77M      -     8.63G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-05-02_22:00:08_hourly   3.60M      -     8.63G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-05-02_23:00:05_hourly   4.20M      -     8.63G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-05-03_00:00:18_daily    2.06G      -     9.16G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-05-04_00:00:24_daily    2.05G      -     9.13G  -
tank/encrypt/backup/rpool/data/encrypt_lxc/subvol-109-disk-0@autosnap_2024-05-05_00:00:21_daily     131M      -     9.13G  -

You can see that they have some in common.

What do I need to do to fix this?

mercenary_sysadmin · June 4, 2024, 2:54pm

Are you running sanoid on the target as well as the source?
Are you using the production template on the target?

pgesting · June 5, 2024, 4:52am

Yes

No, I am running backup template on the target.

I tried to follow the instructions for sanoid as closely as possible. Here is my production template:

[template_production] 
    frequently = 0
    hourly = 36 
    daily = 30
    monthly = 6
    yearly = 0 
    autosnap = yes
    autoprune = yes

Here is the backup:

[template_backup]
    autoprune = yes
    frequently = 0
    hourly = 30
    daily = 90
    monthly = 12
    yearly = 2

You can tell I am not running production on the target as well from the snapshots available in my original post. There are a lot more and older montlies than on the source.

Although, I just realized, that since production has no yearlies, and since I turned off autosnap on backup, the target has no yearlies as well, haha. I will change production to have 1 yearly so that the target can keep the last two.

pgesting · June 10, 2024, 5:44am

@mercenary_sysadmin Any advice as to what I should do to get them back in sync?

mercenary_sysadmin · June 10, 2024, 11:31am

To the best of my knowledge, the only way to get that message is to have a snapshot on the target with the same name as one on the source, but which does not actually match. In which case, the solution is to destroy the offending snapshot on either source or target.

This usually happens when somebody naively sets sanoid up with the production template on both sides. But ultimately, however it happened, that’s the issue; syncoid isn’t really what’s even telling you about the problem: zfs receive is.

pgesting · June 10, 2024, 12:14pm

Ah, interesting. So something happened wrong during a previous sync? It happened for the same dataset on two different targets, so I have no idea how that happened.

From looking at the error log, it is trying to send all the incremental snapshots from daily on May 5 until the hourly on June 3 at 9am. But it is crashing on the May 6 daily.

Based upon the snapshots, the May 6 daily is the next snapshot after May 5 daily, so what you are saying is that the May 5 daily is the one that is wrong? I should delete that from the target and then try to run syncoid again?

mercenary_sysadmin · June 10, 2024, 10:18pm

I’m not sure off the top of my head if the problem is the fifth or sixth and I’m too tired to scroll up and look, but whichever one ZFS told you “didn’t match” is the problem and needs to be destroyed on the target.