Can multiple syncoid jobs run in parallel pulling snapshots from the same remote?

It should work fine with multiple syncoid processes pulling from the same source, but when I investigate now I’m seeing issues, at least when called extremely quickly:

for i in {1..4} ; do 
    syncoid --no-sync-snap rpool/demo/0 rpool/demo/$i & sleep 1 ; 
done

This results in what appears to be a crash in the ZFS send or receive process itself: syncoid terminates, but the send and receive processes themselves are still hanging around (and not actually moving data between each other).

I’ll have to investigate further when I get a chance; this was a very unexpected result since I’ve got plenty of machines in trio sets, where C and B both pull from A, and have not seen any issues there. So I’m not sure what’s different now, whether this is a regression since older versions, or whether this is an edge case that only crops up with VERY rapid invocations.

edit: it looks like this is a bug that crops up when not using --quiet; when I specify --quiet I get the expected result:

root@elden:/# zfs list -r rpool/demo
NAME           USED  AVAIL     REFER  MOUNTPOINT
rpool/demo    1.00G  1.11T       96K  /demo
rpool/demo/0  1.00G  1.11T     1.00G  /demo/0

We have a dataset with 1GiB of data in it, which we will replicate using four parallel syncoid processes to four separate targets.

root@elden:/# for i in {1..4} ; do 
                  syncoid --no-sync-snap --quiet rpool/demo/0 rpool/demo/$i &
              done
[15] 897559
[16] 897560
[17] 897561
[18] 897562
[11]   Done                    syncoid --no-sync-snap --quiet rpool/demo/0 rpool/demo/$i
[12]   Done                    syncoid --no-sync-snap --quiet rpool/demo/0 rpool/demo/$i
[13]   Done                    syncoid --no-sync-snap --quiet rpool/demo/0 rpool/demo/$i
[14]   Done                    syncoid --no-sync-snap --quiet rpool/demo/0 rpool/demo/$i

We used a simple for loop to do our four parallel syncoid runs… but crucially, this time we used --quiet in our argument list.

root@elden:/# zfs list -r rpool/demo
NAME           USED  AVAIL     REFER  MOUNTPOINT
rpool/demo    5.00G  1.10T       96K  /demo
rpool/demo/0  1.00G  1.10T     1.00G  /demo/0
rpool/demo/1  1.00G  1.10T     1.00G  /demo/1
rpool/demo/2  1.00G  1.10T     1.00G  /demo/2
rpool/demo/3  1.00G  1.10T     1.00G  /demo/3
rpool/demo/4  1.00G  1.10T     1.00G  /demo/4

And we can see that our four parallel processes worked properly. I don’t know what bug we’re encountering during the print output phase, but in the meantime, this should help you work around it. If you can confirm that this works for you, would you mind opening up a bug report outlining what you’re encountering over at Github? Thanks!