ZFS write performance on Host vs VM

thmzz · July 1, 2024, 2:49pm

I have been checking performance on the host and vm, and getting around ~1Gb/s on the host and around ~250MB/s with command:
dd if=/dev/zero of=/nvme/testfile1 bs=1G count=1 oflag=dsync

NVME is 2 x 2TB Samsung PM9A3 with a Supermicro PCI Riser
ZFS ARC Max is 50 GB
There is no limitations on read/write speed on the VM.
Config of VM:
boot: order=scsi0;net0
cores: 12
memory: 65536
meta: creation-qemu=6.2.0,ctime=1657351461
net0: virtio=CE:35:68:5F:52:36,bridge=vmbr0,firewall=1
numa: 0
onboot: 1
ostype: l26
scsi0: nvme:vm-113-disk-0,size=800G
scsihw: virtio-scsi-pci
smbios1: uuid=20114d8c-12d9-4cc9-84cb-ab720ef2cde7
sockets: 2
vmgenid: 625e3afc-7468-4bcf-b263-d45a6670b529

Is the speed on the VM to low? Any tips or advice for increasing performance if thats possible?

mercenary_sysadmin · July 1, 2024, 9:19pm

This is not a good way to test your storage performance, and will not give you valuable insights for actual real-world scenarios.

What you want for this job is fio. See here: How fast are your disks? Find out the open source way, with fio | Ars Technica

If you just want to run one single test, random 64K writes with numjobs=8 is probably the one to run.

thmzz · July 3, 2024, 8:02pm

Hi,

I ran the test and this is the output.

fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=64k --size=4g --numjobs=8 --runtime=60 --time_based --end_fsync=1
random-write: (g=0): rw=randwrite, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=posixaio, iodepth=1
...
fio-3.19
Starting 8 processes
random-write: Laying out IO file (1 file / 4096MiB)
random-write: Laying out IO file (1 file / 4096MiB)
random-write: Laying out IO file (1 file / 4096MiB)
random-write: Laying out IO file (1 file / 4096MiB)
random-write: Laying out IO file (1 file / 4096MiB)
random-write: Laying out IO file (1 file / 4096MiB)
random-write: Laying out IO file (1 file / 4096MiB)
Jobs: 8 (f=8): [F(8)][100.0%][eta 00m:00s]                         
Jobs: 1 (f=1): [F(1),_(7)][100.0%][eta 00m:00s]          
random-write: (groupid=0, jobs=1): err= 0: pid=676991: Wed Jul  3 19:24:37 2024
  write: IOPS=429, BW=26.9MiB/s (28.2MB/s)(2677MiB/99672msec); 0 zone resets
    slat (usec): min=3, max=246, avg= 6.08, stdev= 3.95
    clat (usec): min=3, max=71488, avg=1386.41, stdev=6821.03
     lat (usec): min=48, max=71492, avg=1392.49, stdev=6820.96
    clat percentiles (usec):
     |  1.00th=[   50],  5.00th=[   52], 10.00th=[   54], 20.00th=[   58],
     | 30.00th=[   63], 40.00th=[   70], 50.00th=[   76], 60.00th=[   85],
     | 70.00th=[  102], 80.00th=[  130], 90.00th=[  190], 95.00th=[  441],
     | 99.00th=[39584], 99.50th=[44827], 99.90th=[57410], 99.95th=[61604],
     | 99.99th=[67634]
   bw (  KiB/s): min=21418, max=519025, per=18.71%, avg=44898.10, stdev=75007.89, samples=119
   iops        : min=  334, max= 8109, avg=701.16, stdev=1171.97, samples=119
  lat (usec)   : 4=0.01%, 50=1.71%, 100=67.72%, 250=23.26%, 500=2.50%
  lat (usec)   : 750=0.43%, 1000=0.19%
  lat (msec)   : 2=0.24%, 4=0.12%, 10=0.14%, 20=0.30%, 50=3.11%
  lat (msec)   : 100=0.28%
  cpu          : usr=0.37%, sys=0.53%, ctx=42855, majf=0, minf=30
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,42836,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
random-write: (groupid=0, jobs=1): err= 0: pid=676992: Wed Jul  3 19:24:37 2024
  write: IOPS=484, BW=30.3MiB/s (31.8MB/s)(2976MiB/98267msec); 0 zone resets
    slat (usec): min=3, max=1639, avg= 5.98, stdev= 8.57
    clat (usec): min=41, max=74854, avg=1252.32, stdev=6526.08
     lat (usec): min=45, max=74859, avg=1258.30, stdev=6526.07
    clat percentiles (usec):
     |  1.00th=[   47],  5.00th=[   50], 10.00th=[   51], 20.00th=[   55],
     | 30.00th=[   60], 40.00th=[   66], 50.00th=[   71], 60.00th=[   77],
     | 70.00th=[   89], 80.00th=[  116], 90.00th=[  165], 95.00th=[  326],
     | 99.00th=[39060], 99.50th=[43779], 99.90th=[56886], 99.95th=[60556],
     | 99.99th=[69731]
   bw (  KiB/s): min=21291, max=828087, per=20.46%, avg=49120.98, stdev=102400.01, samples=119
   iops        : min=  332, max=12938, avg=767.13, stdev=1599.97, samples=119
  lat (usec)   : 50=6.62%, 100=68.01%, 250=19.30%, 500=2.02%, 750=0.36%
  lat (usec)   : 1000=0.12%
  lat (msec)   : 2=0.13%, 4=0.04%, 10=0.09%, 20=0.25%, 50=2.78%
  lat (msec)   : 100=0.27%
  cpu          : usr=0.41%, sys=0.28%, ctx=47641, majf=0, minf=30
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,47623,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
random-write: (groupid=0, jobs=1): err= 0: pid=676993: Wed Jul  3 19:24:37 2024
  write: IOPS=503, BW=31.5MiB/s (33.0MB/s)(2941MiB/93429msec); 0 zone resets
    slat (usec): min=3, max=454, avg= 5.89, stdev= 4.17
    clat (usec): min=40, max=73140, avg=1267.20, stdev=6555.24
     lat (usec): min=44, max=73144, avg=1273.08, stdev=6555.18
    clat percentiles (usec):
     |  1.00th=[   47],  5.00th=[   50], 10.00th=[   51], 20.00th=[   55],
     | 30.00th=[   62], 40.00th=[   68], 50.00th=[   73], 60.00th=[   80],
     | 70.00th=[   93], 80.00th=[  119], 90.00th=[  167], 95.00th=[  318],
     | 99.00th=[39060], 99.50th=[43779], 99.90th=[57410], 99.95th=[61080],
     | 99.99th=[65274]
   bw (  KiB/s): min=21972, max=774219, per=20.27%, avg=48642.93, stdev=98713.71, samples=119
   iops        : min=  343, max=12097, avg=759.67, stdev=1542.36, samples=119
  lat (usec)   : 50=6.71%, 100=66.52%, 250=20.83%, 500=1.87%, 750=0.32%
  lat (usec)   : 1000=0.11%
  lat (msec)   : 2=0.15%, 4=0.06%, 10=0.09%, 20=0.25%, 50=2.84%
  lat (msec)   : 100=0.26%
  cpu          : usr=0.43%, sys=0.28%, ctx=47085, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,47063,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
random-write: (groupid=0, jobs=1): err= 0: pid=676994: Wed Jul  3 19:24:37 2024
  write: IOPS=487, BW=30.5MiB/s (31.0MB/s)(3008MiB/98676msec); 0 zone resets
    slat (usec): min=2, max=205, avg= 6.07, stdev= 3.88
    clat (usec): min=38, max=79278, avg=1238.08, stdev=6487.39
     lat (usec): min=42, max=79283, avg=1244.15, stdev=6487.33
    clat percentiles (usec):
     |  1.00th=[   47],  5.00th=[   49], 10.00th=[   51], 20.00th=[   55],
     | 30.00th=[   60], 40.00th=[   66], 50.00th=[   71], 60.00th=[   78],
     | 70.00th=[   91], 80.00th=[  114], 90.00th=[  165], 95.00th=[  306],
     | 99.00th=[39060], 99.50th=[43254], 99.90th=[56886], 99.95th=[61080],
     | 99.99th=[70779]
   bw (  KiB/s): min=21418, max=773818, per=20.75%, avg=49810.71, stdev=104949.98, samples=119
   iops        : min=  334, max=12090, avg=777.90, stdev=1639.82, samples=119
  lat (usec)   : 50=7.82%, 100=67.15%, 250=19.22%, 500=1.87%, 750=0.31%
  lat (usec)   : 1000=0.11%
  lat (msec)   : 2=0.11%, 4=0.06%, 10=0.10%, 20=0.23%, 50=2.77%
  lat (msec)   : 100=0.25%
  cpu          : usr=0.43%, sys=0.28%, ctx=48148, majf=0, minf=27
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,48131,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
random-write: (groupid=0, jobs=1): err= 0: pid=676995: Wed Jul  3 19:24:37 2024
  write: IOPS=481, BW=30.1MiB/s (31.6MB/s)(2959MiB/98276msec); 0 zone resets
    slat (usec): min=3, max=1091, avg= 5.89, stdev= 6.23
    clat (usec): min=38, max=72245, avg=1259.93, stdev=6538.51
     lat (usec): min=42, max=72248, avg=1265.82, stdev=6538.45
    clat percentiles (usec):
     |  1.00th=[   47],  5.00th=[   50], 10.00th=[   51], 20.00th=[   55],
     | 30.00th=[   61], 40.00th=[   67], 50.00th=[   72], 60.00th=[   79],
     | 70.00th=[   91], 80.00th=[  116], 90.00th=[  163], 95.00th=[  326],
     | 99.00th=[39060], 99.50th=[43779], 99.90th=[56361], 99.95th=[60556],
     | 99.99th=[66847]
   bw (  KiB/s): min=21461, max=728414, per=20.44%, avg=49054.72, stdev=100246.31, samples=119
   iops        : min=  335, max=11381, avg=766.10, stdev=1566.33, samples=119
  lat (usec)   : 50=7.23%, 100=67.05%, 250=19.61%, 500=1.99%, 750=0.33%
  lat (usec)   : 1000=0.14%
  lat (msec)   : 2=0.18%, 4=0.07%, 10=0.11%, 20=0.24%, 50=2.81%
  lat (msec)   : 100=0.26%
  cpu          : usr=0.42%, sys=0.27%, ctx=47356, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,47337,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
random-write: (groupid=0, jobs=1): err= 0: pid=676996: Wed Jul  3 19:24:37 2024
  write: IOPS=485, BW=30.4MiB/s (31.8MB/s)(2907MiB/95772msec); 0 zone resets
    slat (usec): min=3, max=428, avg= 6.11, stdev= 4.48
    clat (usec): min=40, max=71527, avg=1281.96, stdev=6594.41
     lat (usec): min=44, max=71530, avg=1288.07, stdev=6594.36
    clat percentiles (usec):
     |  1.00th=[   48],  5.00th=[   50], 10.00th=[   52], 20.00th=[   56],
     | 30.00th=[   62], 40.00th=[   68], 50.00th=[   73], 60.00th=[   81],
     | 70.00th=[   95], 80.00th=[  121], 90.00th=[  172], 95.00th=[  334],
     | 99.00th=[39060], 99.50th=[44303], 99.90th=[57410], 99.95th=[61080],
     | 99.99th=[68682]
   bw (  KiB/s): min=21461, max=682633, per=20.11%, avg=48260.18, stdev=95392.93, samples=119
   iops        : min=  335, max=10666, avg=753.70, stdev=1490.49, samples=119
  lat (usec)   : 50=5.71%, 100=66.96%, 250=21.17%, 500=2.00%, 750=0.34%
  lat (usec)   : 1000=0.12%
  lat (msec)   : 2=0.14%, 4=0.06%, 10=0.11%, 20=0.25%, 50=2.85%
  lat (msec)   : 100=0.28%
  cpu          : usr=0.44%, sys=0.25%, ctx=46527, majf=0, minf=29
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,46511,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
random-write: (groupid=0, jobs=1): err= 0: pid=676997: Wed Jul  3 19:24:37 2024
  write: IOPS=489, BW=30.6MiB/s (32.1MB/s)(2931MiB/95722msec); 0 zone resets
    slat (usec): min=3, max=191, avg= 6.01, stdev= 3.74
    clat (nsec): min=1707, max=71359k, avg=1270990.13, stdev=6570440.44
     lat (usec): min=42, max=71363, avg=1277.00, stdev=6570.40
    clat percentiles (usec):
     |  1.00th=[   47],  5.00th=[   50], 10.00th=[   51], 20.00th=[   55],
     | 30.00th=[   61], 40.00th=[   67], 50.00th=[   72], 60.00th=[   80],
     | 70.00th=[   94], 80.00th=[  117], 90.00th=[  163], 95.00th=[  322],
     | 99.00th=[39060], 99.50th=[43779], 99.90th=[57410], 99.95th=[61080],
     | 99.99th=[69731]
   bw (  KiB/s): min=21291, max=746197, per=20.23%, avg=48557.45, stdev=97727.99, samples=119
   iops        : min=  332, max=11659, avg=758.34, stdev=1526.97, samples=119
  lat (usec)   : 2=0.01%, 50=6.71%, 100=66.51%, 250=20.82%, 500=1.77%
  lat (usec)   : 750=0.39%, 1000=0.13%
  lat (msec)   : 2=0.15%, 4=0.06%, 10=0.09%, 20=0.24%, 50=2.84%
  lat (msec)   : 100=0.28%
  cpu          : usr=0.41%, sys=0.30%, ctx=46909, majf=0, minf=30
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,46892,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
random-write: (groupid=0, jobs=1): err= 0: pid=676998: Wed Jul  3 19:24:37 2024
  write: IOPS=495, BW=30.0MiB/s (32.5MB/s)(2964MiB/95726msec); 0 zone resets
    slat (usec): min=3, max=221, avg= 5.94, stdev= 3.63
    clat (usec): min=40, max=70949, avg=1257.52, stdev=6529.32
     lat (usec): min=43, max=70953, avg=1263.46, stdev=6529.26
    clat percentiles (usec):
     |  1.00th=[   47],  5.00th=[   49], 10.00th=[   51], 20.00th=[   55],
     | 30.00th=[   60], 40.00th=[   67], 50.00th=[   72], 60.00th=[   78],
     | 70.00th=[   89], 80.00th=[  113], 90.00th=[  159], 95.00th=[  310],
     | 99.00th=[39060], 99.50th=[43779], 99.90th=[56361], 99.95th=[60556],
     | 99.99th=[68682]
   bw (  KiB/s): min=21418, max=735097, per=20.46%, avg=49115.02, stdev=100888.51, samples=119
   iops        : min=  334, max=11485, avg=767.03, stdev=1576.35, samples=119
  lat (usec)   : 50=7.97%, 100=67.55%, 250=18.65%, 500=1.79%, 750=0.32%
  lat (usec)   : 1000=0.10%
  lat (msec)   : 2=0.12%, 4=0.06%, 10=0.10%, 20=0.26%, 50=2.80%
  lat (msec)   : 100=0.27%
  cpu          : usr=0.42%, sys=0.27%, ctx=47431, majf=0, minf=28
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,47419,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=234MiB/s (246MB/s), 26.9MiB/s-31.5MiB/s (28.2MB/s-33.0MB/s), io=22.8GiB (24.5GB), run=93429-99672msec

Disk stats (read/write):
    dm-0: ios=38/460181, merge=0/0, ticks=22/1979186, in_queue=1979208, util=97.80%, aggrios=545/226794, aggrmerge=0/234520, aggrticks=623/974274, aggrin_queue=974898, aggrutil=85.39%
  sda: ios=545/226794, merge=0/234520, ticks=623/974274, in_queue=974898, util=85.39%

From what I understand is 246MB/s. I should probably get more since there is not limit on the VPS?

mercenary_sysadmin · July 3, 2024, 8:06pm

What does it look like when you run that test on the host?

edit: oh, I just realized this is Proxmox, which means you’re using zvols not datasets and your recordsize is going to be very low.

You’re going to get very deep in the weeds very quickly. Perhaps the most important question to ask, really, is a very simple one: is your VM perceptibly “too slow” or are you just worrying about benchmarks in the absence of an actual problem?

Proxmox’s approach does not result in the most performant storage, but if you don’t actually need your storage to go faster than it is, you may not want to dive down an extremely deep rabbit hole chasing the details.