Hi
I have a sanoid/syncoid setup (on three nodes in a Proxmox cluster) where the backup server (the third node) pulls the snapshots from the production datasets to a separate backup disk in a separate zpool. Today I noticed the backup server suddenly ran at 100% CPU and has according to the Proxmox stats it has been doing so for almost 18 hours. Looking at htop the culprit command was pv -p -t -e -r -b (three processes). I searched through the syncoid script and found the exact reference so I thought it has to do something with syncing my snapshots
Here is the crontab on my backup server. I have three proxmox nodes and one pool on the third node is the backup server.
I first restarted the server but a few hours later I came back to see the load was up again to 100%. I then killed the processes and the CPU load, of course, disappeared. I will keep track of this the coming days but I am curious what can be the case of this, and if somebody has had this issue before and can help me troubleshoot it. I can post the output of zfs list if necessary.
# From PVE1
3 * * * * root /usr/sbin/syncoid -r --skip-parent --no-sync-snap --recvoptions=u --delete-target-snapshots root@pve1.lan:rpool backup/data_crypt/pve1/rpool
3 * * * * root /usr/sbin/syncoid -r --skip-parent --no-sync-snap --recvoptions=u --delete-target-snapshots root@pve1.lan:syncpool backup/data_crypt/pve1/syncpool
# From PVE2
18 * * * * root /usr/sbin/syncoid -r --skip-parent --no-sync-snap --recvoptions=u --delete-target-snapshots root@pve2.lan:rpool backup/data_crypt/pve2/rpool
18 * * * * root /usr/sbin/syncoid -r --skip-parent --no-sync-snap --recvoptions=u --delete-target-snapshots root@pve2.lan:syncpool backup/data_crypt/pve2/syncpool
18 * * * * root /usr/sbin/syncoid -r --skip-parent --no-sync-snap --recvoptions=u --delete-target-snapshots root@pve2.lan:storage backup/data_crypt/storage
# From PVE3
33 * * * * root /usr/sbin/syncoid -r --skip-parent --no-sync-snap --recvoptions=u --delete-target-snapshots rpool backup/data_crypt/pve3/rpool
33 * * * * root /usr/sbin/syncoid -r --skip-parent --no-sync-snap --recvoptions=u --delete-target-snapshots syncpool backup/data_crypt/pve3/syncpool
# From PVE4
48 * * * * root /usr/sbin/syncoid -r --skip-parent --no-sync-snap --recvoptions=u --delete-target-snapshots root@pve4.lan:rpool backup/data_crypt/pve4/rpool