I'm working on an improved progress bar for Syncoid

For YEARS now it’s been bothering me that syncoid can only tell you the progress on the current dataset, not the entire recursive replication task. That’s been a hard limit because of the use of an external program (Andrew Wood’s excellent pv) to provide the meter.

That’s gonna change soon! I’ve already written a pv alternative in perl as a proof of concept, and I’m going to start working on incorporating that proof of concept directly into syncoid, so it can show both progress on the current dataset AND the entire replication task.

If you want to get a sneak peek of the proof of concept: GitHub - jimsalterjrs/perlpv: A proof-of-concept pipe viewer similar to Andrew Wood's pv in perl, for eventual use internally in syncoid


For measuring durations you want want Time::HiRes::clock_gettime(CLOCK_MONOTONIC) over Time::HiRes::time, as this isn’t affected by changes to the system clock.

For FreeBSD/Rust uutils dd status=progress I used a SIGALRM/sleep thread with a boolean flag to signal the copy loop to handle progress updates at a fixed interval, without involving timekeeping calls for every block. Looks like you could do something similar with Time::HiRes::setitimer.


Well, it helped a bit, the numbers went in the right direction:

    Benchmark 1: ./perlpv testfile /dev/null
      Time (mean ± σ):      2.564 s ±  0.090 s    [User: 0.693 s, System: 1.870 s]
      Range (min … max):    2.382 s …  2.654 s    10 runs

    Benchmark 2: ./perlpv.orig testfile /dev/null
      Time (mean ± σ):      2.780 s ±  0.098 s    [User: 0.930 s, System: 1.850 s]
      Range (min … max):    2.600 s …  2.889 s    10 runs

      './perlpv testfile /dev/null' ran
        1.08 ± 0.05 times faster than './perlpv.orig testfile /dev/null
1 Like

Well, I’m not going to complain about an 8% performance boost. Thanks!

I got an extra ~1GiB/sec out of your improvements, on this i7-1200K system.

jim@elden:~/git/perlpv$ ../perlpv.orig /dev/zero /dev/null
    Transferred 140.0GiB (at average rate 23.9GiB/sec)
jim@elden:~/git/perlpv$ ./perlpv /dev/zero /dev/null
    Transferred 264.9GiB (at average rate 25.1GiB/sec)

Another neat point of comparison…

jim@elden:~/git/perlpv$ pv < /dev/zero > /dev/null
 276GiB 0:00:11 [25.1GiB/s] [                                       <=>                                          ]

Thanks to your improvements, perlpv is now every bit as fast as Andrew’s original (which is written in pure C). Awesome!

1 Like

AMD 5700X on FreeBSD 13.2:

❯ pv testfile >/dev/null
 300GiB 0:00:11 [25.3GiB/s] [======================================================>] 100%
11.870 real, 0.331 user, 11.539 sys;  page: 0 hard/248 soft, swap: 0, I/O: 0/0
Mem: 2532KB (43KB shared + 138KB data/stack = 182KB), VCSW: 64 IVCSW: 213
❯ ./perlpv testfile /dev/null
    Transferred 300.0GiB of 300.0GiB (at average rate 21.3GiB/sec)
19.060 real, 5.117 user, 13.942 sys;  page: 0 hard/1093 soft, swap: 0, I/O: 0/0
Mem: 8192KB (7KB shared + 130KB data/stack = 138KB), VCSW: 102 IVCSW: 328
❯ ./perlpv.orig testfile /dev/null
    Transferred 300.0GiB of 300.0GiB (at average rate 23.8GiB/sec)
30.237 real, 16.243 user, 13.992 sys;  page: 0 hard/27278 soft, swap: 0, I/O: 0/0
Mem: 113180KB (7KB shared + 130KB data/stack = 138KB), VCSW: 120 IVCSW: 615

Something sketchy about the numbers there - the original was a couple of GB/s faster but took longer?

Also FreeBSD appears to have a very fast /dev/zero:

❯ pv /dev/zero >/dev/null
 415GiB 0:00:10 [40.5GiB/s]

Alas not reflected by perlpv, and again questionable transfer rate figures involved:

❯ ./perlpv /dev/zero /dev/null
    Transferred 124.3GiB (at average rate 14.9GiB/sec)
11.778 real, 5.487 user, 6.290 sys;  page: 0 hard/1094 soft, swap: 0, I/O: 0/0
❯ ./perlpv.orig /dev/zero /dev/null
    Transferred 127.9GiB (at average rate 15.1GiB/sec)
14.202 real, 7.717 user, 6.481 sys;  page: 0 hard/14020 soft, swap: 0, I/O: 0/0

Replacing print with syswrite appears to help a lot:

❯ ./perlpv /dev/zero /dev/null
    Transferred 125.1GiB (at average rate 32.5GiB/sec)
5.980 real, 0.181 user, 5.798 sys;  page: 0 hard/1095 soft, swap: 0, I/O: 0/0

❯ ./perlpv testfile /dev/null
    Transferred 300.0GiB of 300.0GiB (at average rate 35.4GiB/sec)
11.589 real, 0.425 user, 11.163 sys;  page: 0 hard/1100 soft, swap: 0, I/O: 0/0

Repeating the initial test (a 40GB sparse file):

Benchmark 1: ./perlpv testfile /dev/null
  Time (mean ± σ):      1.555 s ±  0.030 s    [User: 0.094 s, System: 1.461 s]
  Range (min … max):    1.521 s …  1.610 s    10 runs

Benchmark 2: ./perlpv.orig testfile /dev/null
  Time (mean ± σ):      2.789 s ±  0.085 s    [User: 0.911 s, System: 1.877 s]
  Range (min … max):    2.629 s …  2.879 s    10 runs

  './perlpv testfile /dev/null' ran
    1.79 ± 0.06 times faster than './perlpv.orig testfile /dev/null'

Reduced user CPU time by an order of magnitude.

1 Like

The apparent rate anomalies are likely because perlpv uses an algorithm which averages together instantaneous transfer rate, ten second average rate, one minute average rate, and five minute average rate in its estimator… And I didn’t think to replace that with the simple overall mean in the final output line. Oops!

Also: this is why it’s not the worst thing to create proof of concept stuff and put it on GitHub, somebody like @Freaky might happen along and improve it for you even BEFORE it goes into production. :pinched_fingers:


In your face, pv. And Linux /dev/zero.

❯ ./perlpv /dev/zero /dev/null
    Transferred 513.6GiB (AVG 46.6GiB/sec CUR 48.1GiB/sec)
11.169 real, 0.843 user, 10.325 sys;  page: 0 hard/1000 soft, swap: 0, I/O: 0/0
❯ pv /dev/zero >/dev/null
^C93GiB 0:00:12 [42.5GiB/s]
12.600 real, 1.268 user, 11.331 sys;  page: 0 hard/152 soft, swap: 0, I/O: 0/0
1 Like

Maybe it’s not your /dev/zero that’s faster than Linux’s. Maybe it’s your /dev/null! :upside_down_face:

Wow, this sounds great!
By chance would it also enable a summary of total bytes transferred?

Yes, it will. But keep in mind, it still won’t be any more accurate in estimating sizes than it ever was–that’s an upstream issue in ZFS itself, not something I can fix here.

That won’t generally matter much on large jobs, but little ones with lots of snapshots but very little actual data will still tend to show very inaccurate estimates.

1 Like