Troubleshooting poor performance of zvols over iscsi

OpenZFS
7

Recently I’ve been testing zvol performance on a zfs SSD raid0 stripe.

  • I compared performance of qcow2 / zvol & lvm + the various volblocksize options on the zvols:
    • 16K / 32K / 64K / 128K
  • I also tested virtio-blk versus virtio-scsi
  • The zvol performance is so good I’m going to stop passing through my 2nd nvme to Windows & instead put the os on a zvol on the nvme
  • the configuration below showed basically bare metal read performance in the builtin winsat tool :
> Disk  Random 16.0 Read                       1155.78 MB/s          8.9
> Disk  Sequential 64.0 Read                   8193.54 MB/s          9.9
> Disk  Sequential 64.0 Write                  5185.20 MB/s          9.7

I am more interested in speed than redundancy. The 2 x SSDs are a striped pool for a SSD Steam Library but are also partitioned to provide special devices for my main mirrored pool on spinning sata + new sata stripe for Windows / Steam

  • I created the striped (raid0) SSD pool with:
zpool create -f -o ashift=12 -m /mnt/ssd1 ssd1 \
    ata-INTEL_SSDSC2KG019T8_PHYG9170009C1P9DGN-part4 \
    ata-SAMSUNG_MZ7KH1T9HAJR-00005_S47PNE0M508088-part4
  • I created a -s (sparse) zvol with a 64K volblocksize:
zfs create -o volblocksize=64K -s -V 3.16TB ssd1/windows

On the zvol I created:

  • 16MB partition type Microsoft reserved (type 10 in fdisk)
  • balance of the space as partition type Microsoft basic data (type 11 in fdisk)
  • inside my Windows vm in Disk Management I formatted the zvol (which I think uses a 64K cluster size by default)
  • tested performance with the built in Windows tool winsat

First round of testing:

  • SSD => qcow2 / zvol / lvm
  • Existing SCSI whole device passthrough of a Toshiba Enterprise SATA as a comparision (which is in the process of being migrated to a zvol on a SATA stripe with a striped special device on the SSD (I have approx 9.9 petabytes of writes left on the SSDs)
2 x 1.92TB Enterprise SSD tests
===============================

QCOW2:
========================
Z:\>winsat disk -drive Q

Windows System Assessment Tool
> Running: Feature Enumeration ''
> Run Time 00:00:00.00
> Running: Storage Assessment '-drive Q -ran -read'
> Run Time 00:00:00.41
> Running: Storage Assessment '-drive Q -seq -read'
> Run Time 00:00:01.16
> Running: Storage Assessment '-drive Q -seq -write'
> Run Time 00:00:00.73
> Running: Storage Assessment '-drive Q -flush -seq'
> Run Time 00:00:00.45
> Running: Storage Assessment '-drive Q -flush -ran'
> Run Time 00:00:00.42
> Dshow Video Encode Time                      0.00000 s
> Dshow Video Decode Time                      0.00000 s
> Media Foundation Decode Time                 0.00000 s
> Disk  Random 16.0 Read                       59.18 MB/s          6.7
> Disk  Sequential 64.0 Read                   3156.85 MB/s          9.3
> Disk  Sequential 64.0 Write                  2859.21 MB/s          9.2
> Average Read Time with Sequential Writes     0.097 ms          8.8
> Latency: 95th Percentile                     0.172 ms          8.9
> Latency: Maximum                             2.063 ms          8.8
> Average Read Time with Random Writes         0.099 ms          8.9
> Total Run Time 00:00:03.33

-------------------------------------------------------------------------------------------

LVM STRIPE
========================

Z:\>winsat disk -drive R
Windows System Assessment Tool
> Running: Feature Enumeration ''
> Run Time 00:00:00.00
> Running: Storage Assessment '-drive R -ran -read'
> Run Time 00:00:00.13
> Running: Storage Assessment '-drive R -seq -read'
> Run Time 00:00:01.47
> Running: Storage Assessment '-drive R -seq -write'
> Run Time 00:00:01.33
> Running: Storage Assessment '-drive R -flush -seq'
> Run Time 00:00:00.66
> Running: Storage Assessment '-drive R -flush -ran'
> Run Time 00:00:00.66
> Dshow Video Encode Time                      0.00000 s
> Dshow Video Decode Time                      0.00000 s
> Media Foundation Decode Time                 0.00000 s
> Disk  Random 16.0 Read                       433.61 MB/s          8.2
> Disk  Sequential 64.0 Read                   757.50 MB/s          8.3
> Disk  Sequential 64.0 Write                  672.60 MB/s          8.2
> Average Read Time with Sequential Writes     0.222 ms          8.6
> Latency: 95th Percentile                     0.416 ms          8.7
> Latency: Maximum                             3.967 ms          8.6
> Average Read Time with Random Writes         0.242 ms          8.8
> Total Run Time 00:00:04.34


ZVOL:
=========================

Z:\>winsat disk -drive V
Windows System Assessment Tool
> Running: Feature Enumeration ''
> Run Time 00:00:00.00
> Running: Storage Assessment '-drive V -ran -read'
> Run Time 00:00:00.11
> Running: Storage Assessment '-drive V -seq -read'
> Run Time 00:00:01.19
> Running: Storage Assessment '-drive V -seq -write'
> Run Time 00:00:00.75
> Running: Storage Assessment '-drive V -flush -seq'
> Run Time 00:00:00.70
> Running: Storage Assessment '-drive V -flush -ran'
> Run Time 00:00:00.47
> Dshow Video Encode Time                      0.00000 s
> Dshow Video Decode Time                      0.00000 s
> Media Foundation Decode Time                 0.00000 s
> Disk  Random 16.0 Read                       844.00 MB/s          8.6
> Disk  Sequential 64.0 Read                   2533.42 MB/s          9.1
> Disk  Sequential 64.0 Write                  2757.42 MB/s          9.2
> Average Read Time with Sequential Writes     0.097 ms          8.8
> Latency: 95th Percentile                     0.133 ms          8.9
> Latency: Maximum                             0.421 ms          8.9
> Average Read Time with Random Writes         0.097 ms          8.9
> Total Run Time 00:00:03.31


SATA HARD DRIVE PASSTHROUGH
===========================

Z:\>winsat disk -drive E
Windows System Assessment Tool
> Running: Feature Enumeration ''
> Run Time 00:00:00.00
> Running: Storage Assessment '-drive E -ran -read'
> Run Time 00:00:07.44
> Running: Storage Assessment '-drive E -seq -read'
> Run Time 00:00:03.25
> Running: Storage Assessment '-drive E -seq -write'
> Run Time 00:00:03.92
> Running: Storage Assessment '-drive E -flush -seq'
> Run Time 00:00:06.39
> Running: Storage Assessment '-drive E -flush -ran'
> Run Time 00:00:07.41
> Dshow Video Encode Time                      0.00000 s
> Dshow Video Decode Time                      0.00000 s
> Media Foundation Decode Time                 0.00000 s
> Disk  Random 16.0 Read                       2.13 MB/s          4.3
> Disk  Sequential 64.0 Read                   136.18 MB/s          7.0
> Disk  Sequential 64.0 Write                  155.47 MB/s          7.1
> Average Read Time with Sequential Writes     6.563 ms          5.5
> Latency: 95th Percentile                     19.058 ms          4.7
> Latency: Maximum                             90.300 ms          7.7
> Average Read Time with Random Writes         7.283 ms          5.2
> Total Run Time 00:00:28.52

With zvol the clear performance winner I then experimented with various blocksizes (16K / 32K / 64K / 128K):

  • TLDR: 64K block sizes won as I let Windows format the drive with the default NTFS cluster size of 64K
16K BLOCKSIZE
=============

PS C:\WINDOWS\system32> winsat disk -drive G
Windows System Assessment Tool
> Running: Feature Enumeration ''
> Run Time 00:00:00.00
> Running: Storage Assessment '-drive G -ran -read'
> Run Time 00:00:00.30
> Running: Storage Assessment '-drive G -seq -read'
> Run Time 00:00:08.41
> Running: Storage Assessment '-drive G -seq -write'
> Run Time 00:00:33.24
> Running: Storage Assessment '-drive G -flush -seq'
> Run Time 00:00:00.72
> Running: Storage Assessment '-drive G -flush -ran'
> Run Time 00:00:00.45
> Dshow Video Encode Time                      0.00000 s
> Dshow Video Decode Time                      0.00000 s
> Media Foundation Decode Time                 0.00000 s
> Disk  Random 16.0 Read                       870.43 MB/s          8.7
> Disk  Sequential 64.0 Read                   4390.17 MB/s          9.5
> Disk  Sequential 64.0 Write                  2799.89 MB/s          9.2
> Average Read Time with Sequential Writes     0.084 ms          8.8
> Latency: 95th Percentile                     0.123 ms          8.9
> Latency: Maximum                             0.561 ms          8.9
> Average Read Time with Random Writes         0.090 ms          8.9
> Total Run Time 00:00:43.22


32K BLOCKSIZE
=============

PS C:\WINDOWS\system32> winsat disk -drive H
Windows System Assessment Tool
> Running: Feature Enumeration ''
> Run Time 00:00:00.00
> Running: Storage Assessment '-drive H -ran -read'
> Run Time 00:00:00.11
> Running: Storage Assessment '-drive H -seq -read'
> Run Time 00:00:01.16
> Running: Storage Assessment '-drive H -seq -write'
> Run Time 00:00:00.76
> Running: Storage Assessment '-drive H -flush -seq'
> Run Time 00:00:00.42
> Running: Storage Assessment '-drive H -flush -ran'
> Run Time 00:00:00.41
> Dshow Video Encode Time                      0.00000 s
> Dshow Video Decode Time                      0.00000 s
> Media Foundation Decode Time                 0.00000 s
> Disk  Random 16.0 Read                       917.66 MB/s          8.7
> Disk  Sequential 64.0 Read                   4330.72 MB/s          9.5
> Disk  Sequential 64.0 Write                  2933.63 MB/s          9.2
> Average Read Time with Sequential Writes     0.084 ms          8.8
> Latency: 95th Percentile                     0.144 ms          8.9
> Latency: Maximum                             0.278 ms          8.9
> Average Read Time with Random Writes         0.087 ms          8.9
> Total Run Time 00:00:02.97


64K BLOCKSIZE
=============

PS C:\WINDOWS\system32> winsat disk -drive I
Windows System Assessment Tool
> Running: Feature Enumeration ''
> Run Time 00:00:00.00
> Running: Storage Assessment '-drive I -ran -read'
> Run Time 00:00:00.11
> Running: Storage Assessment '-drive I -seq -read'
> Run Time 00:00:01.14
> Running: Storage Assessment '-drive I -seq -write'
> Run Time 00:00:00.75
> Running: Storage Assessment '-drive I -flush -seq'
> Run Time 00:00:00.45
> Running: Storage Assessment '-drive I -flush -ran'
> Run Time 00:00:00.42
> Dshow Video Encode Time                      0.00000 s
> Dshow Video Decode Time                      0.00000 s
> Media Foundation Decode Time                 0.00000 s
> Disk  Random 16.0 Read                       918.52 MB/s          8.7
> Disk  Sequential 64.0 Read                   4442.18 MB/s          9.5
> Disk  Sequential 64.0 Write                  2990.89 MB/s          9.2
> Average Read Time with Sequential Writes     0.086 ms          8.8
> Latency: 95th Percentile                     0.115 ms          8.9
> Latency: Maximum                             0.375 ms          8.9
> Average Read Time with Random Writes         0.081 ms          8.9
> Total Run Time 00:00:03.00

128K BLOCKSIZE
==============

PS C:\WINDOWS\system32> winsat disk -drive J
Windows System Assessment Tool
> Running: Feature Enumeration ''
> Run Time 00:00:00.00
> Running: Storage Assessment '-drive J -ran -read'
> Run Time 00:00:00.11
> Running: Storage Assessment '-drive J -seq -read'
> Run Time 00:00:01.22
> Running: Storage Assessment '-drive J -seq -write'
> Run Time 00:00:00.74
> Running: Storage Assessment '-drive J -flush -seq'
> Run Time 00:00:00.76
> Running: Storage Assessment '-drive J -flush -ran'
> Run Time 00:00:00.47
> Dshow Video Encode Time                      0.00000 s
> Dshow Video Decode Time                      0.00000 s
> Media Foundation Decode Time                 0.00000 s
> Disk  Random 16.0 Read                       873.73 MB/s          8.7
> Disk  Sequential 64.0 Read                   2704.19 MB/s          9.2
> Disk  Sequential 64.0 Write                  2791.04 MB/s          9.2
> Average Read Time with Sequential Writes     0.099 ms          8.8
> Latency: 95th Percentile                     0.137 ms          8.9
> Latency: Maximum                             0.605 ms          8.9
> Average Read Time with Random Writes         0.096 ms          8.9
> Total Run Time 00:00:03.41

As a final test:

  • compare zvol performance on virto-blk versus virtio-scsi on 64K volblocksize (NB: in my case the underlying NTFS filesystem is using the default 64K cluster size)
  • TLDR: virto-blk is still faster by about 9-10%:
VIRTIO BLK
==========

PS C:\WINDOWS\system32> winsat disk -drive G
Windows System Assessment Tool
> Running: Feature Enumeration ''
> Run Time 00:00:00.00
> Running: Storage Assessment '-drive G -ran -read'
> Run Time 00:00:00.26
> Running: Storage Assessment '-drive G -seq -read'
> Run Time 00:00:01.11
> Running: Storage Assessment '-drive G -seq -write'
> Run Time 00:00:00.66
> Running: Storage Assessment '-drive G -flush -seq'
> Run Time 00:00:00.28
> Running: Storage Assessment '-drive G -flush -ran'
> Run Time 00:00:00.28
> Dshow Video Encode Time                      0.00000 s
> Dshow Video Decode Time                      0.00000 s
> Media Foundation Decode Time                 0.00000 s
> Disk  Random 16.0 Read                       1155.78 MB/s          8.9
> Disk  Sequential 64.0 Read                   8193.54 MB/s          9.9
> Disk  Sequential 64.0 Write                  5185.20 MB/s          9.7
> Average Read Time with Sequential Writes     0.048 ms          8.9
> Latency: 95th Percentile                     0.116 ms          8.9
> Latency: Maximum                             0.324 ms          8.9
> Average Read Time with Random Writes         0.046 ms          8.9
> Total Run Time 00:00:02.75


VIRTIO SCSI
===========

PS C:\WINDOWS\system32> winsat disk -drive H
Windows System Assessment Tool
> Running: Feature Enumeration ''
> Run Time 00:00:00.00
> Running: Storage Assessment '-drive H -ran -read'
> Run Time 00:00:00.11
> Running: Storage Assessment '-drive H -seq -read'
> Run Time 00:00:01.11
> Running: Storage Assessment '-drive H -seq -write'
> Run Time 00:00:00.74
> Running: Storage Assessment '-drive H -flush -seq'
> Run Time 00:00:00.41
> Running: Storage Assessment '-drive H -flush -ran'
> Run Time 00:00:00.39
> Dshow Video Encode Time                      0.00000 s
> Dshow Video Decode Time                      0.00000 s
> Media Foundation Decode Time                 0.00000 s
> Disk  Random 16.0 Read                       1048.02 MB/s          8.9
> Disk  Sequential 64.0 Read                   4305.40 MB/s          9.5
> Disk  Sequential 64.0 Write                  2939.35 MB/s          9.2
> Average Read Time with Sequential Writes     0.078 ms          8.8
> Latency: 95th Percentile                     0.132 ms          8.9
> Latency: Maximum                             1.088 ms          8.9
> Average Read Time with Random Writes         0.083 ms          8.9
> Total Run Time 00:00:02.86
  • Hopefully these results are useful to others, for details of the special devices see Level1
  • NB: special devices cannot be removed from a RAIDZ pool - yet another reason to always use mirrored vdevs
  • During my research I saw an SLOG helps with iscsi performance on the TrueNAS forum (have hit my link limit here)
  • I’m also going to test / move my SLOG from ssd => zram (my system runs on a UPS)