Weird slowness issue

I just can across this issue the first couple times using syncoid. My setup is an HP Gen 9 which I will refer to as exodus on the receiving end, and the sending end is a Dell R510 called revelation on the sending end. exodus is empty, and I’m sending the initial stream of data to exodus/vm-bhyve from tank/vm-bhyve on revelation. Revelation contains 6 VMs, all around 20-40 GB in size. I initiated the send using the recursive option, and the no privilege elevation option. The send took painstakingly long (24 hours) and since the servers are both connected to the same switch, obviously there must be some issue.

Any thoughts?

The only thing that was different about the setup is that each dataset and vol had 5 snapshots that were taken by another replication program that date to the 10th of October and the following 4 days after that.

The initial sends are around 110MB/s on the progress bar, but then it seems to take forever to process the intermediate snapshots or something.

I’ve since deleted the snaps from the other program and have restarted the sync, so I will see how long that takes.

I’m beginning to suspect the HP Gen 9 server which has a P440 card in HBA mode. I’m finding some stuff online saying it doesn’t exactly play well with zfs.

One other thing is that I inadvertently overlooked that the defaults file takes quite a few hourly snaps by default, so I had 3-4 days worth of those.