Troubleshooting poor performance of zvols over iscsi

OpenZFS
10

After a bit more testing I’ve got acceptable performance again (200 fps in Hitman 3)

Steam downloads

  • virtio-blk = 60mb avg / 100mb max
  • virtio-scsi (with 8 iothread queues) = the same as virtio-blk except also after a while 125mb avg / 250mb max WRITE (from queues ?) / 430mb READ (when Hitman 3 was finishing installing)

ZVOL configuration

  • When Windows creates a NTFS filesystem it does not align the disk sectors in the same way Linux does (2048 by default):
fdisk /dev/zvol/ssd1/windows

Disk /dev/zvol/ssd1/windows: 2.98 TiB, 3276544671744 bytes, 6399501312 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 65536 bytes
I/O size (minimum/optimal): 65536 bytes / 65536 bytes
Disklabel type: gpt
Disk identifier: 95A45633-38E3-4236-A04D-41EE98435A40

Device                  Start        End    Sectors Size Type
/dev/zvol/ssd1/windows1    34      32767      32734  16M Microsoft reserved
/dev/zvol/ssd1/windows2 32768 6399498239 6399465472   3T Microsoft basic data

Partition 1 does not start on physical sector boundary.
  • I created the zol as a normal device (not sparse):
  • zfs create -o volblocksize=64K -V 2.98TB ssd1/windows (the default option nowadays in zfs is compression=on which gives you lz4 compresson)
  • I partitioned the zvol manually with a single Microsoft basic data partition (i.e without with a Microsoft reserved partition of 16mb - which I don’t need as I’m not creating dynamic volumes in Windows) :
$ fdisk /dev/zvol/ssd1/windows

Disk /dev/zvol/ssd1/windows: 2.98 TiB, 3276544671744 bytes, 6399501312 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 65536 bytes
I/O size (minimum/optimal): 65536 bytes / 65536 bytes
Disklabel type: gpt
Disk identifier: 95A45633-38E3-4236-A04D-41EE98435A40

Device                  Start        End    Sectors Size Type
/dev/zvol/ssd1/windows1  2048 6399500287 6399498240   3T Microsoft basic data

virtio-scsi configuration

  • attached the zvol to virtio-scsi (add a Controller VirtIO SCSI to virt-manager)
  • add the disk as type scsi
  • set the passed through zvol as an SSD (rotation_rate='1') :
<target dev='sda' bus='scsi' rotation_rate='1'/>
  • configure queues on the virtio-scsi
<controller type="scsi" index="0" model="virtio-scsi">
  <driver queues="8" iothread="1"/>
  <address type="pci" domain="0x0000" bus="0x08" slot="0x00" function="0x0"/>
</controller>

iothread configuration

  • my CPU configuration (as it relates to the iothread)
<vcpu placement="static">16</vcpu>
  <iothreads>1</iothreads>
  <cputune>
    <vcpupin vcpu="0" cpuset="8"/>
    <vcpupin vcpu="1" cpuset="24"/>
    <vcpupin vcpu="2" cpuset="9"/>
    <vcpupin vcpu="3" cpuset="25"/>
    <vcpupin vcpu="4" cpuset="10"/>
    <vcpupin vcpu="5" cpuset="26"/>
    <vcpupin vcpu="6" cpuset="11"/>
    <vcpupin vcpu="7" cpuset="27"/>
    <vcpupin vcpu="8" cpuset="12"/>
    <vcpupin vcpu="9" cpuset="28"/>
    <vcpupin vcpu="10" cpuset="13"/>
    <vcpupin vcpu="11" cpuset="29"/>
    <vcpupin vcpu="12" cpuset="14"/>
    <vcpupin vcpu="13" cpuset="30"/>
    <vcpupin vcpu="14" cpuset="15"/>
    <vcpupin vcpu="15" cpuset="31"/>
    <emulatorpin cpuset="0-3"/>
    <iothreadpin iothread="1" cpuset="4,20"/>
    <vcpusched vcpus="0" scheduler="rr" priority="1"/>
    <vcpusched vcpus="1" scheduler="rr" priority="1"/>
    <vcpusched vcpus="2" scheduler="rr" priority="1"/>
    <vcpusched vcpus="3" scheduler="rr" priority="1"/>
    <vcpusched vcpus="4" scheduler="rr" priority="1"/>
    <vcpusched vcpus="5" scheduler="rr" priority="1"/>
    <vcpusched vcpus="6" scheduler="rr" priority="1"/>
    <vcpusched vcpus="7" scheduler="rr" priority="1"/>
    <vcpusched vcpus="8" scheduler="rr" priority="1"/>
    <vcpusched vcpus="9" scheduler="rr" priority="1"/>
    <vcpusched vcpus="10" scheduler="rr" priority="1"/>
    <vcpusched vcpus="11" scheduler="rr" priority="1"/>
    <vcpusched vcpus="12" scheduler="rr" priority="1"/>
    <vcpusched vcpus="13" scheduler="rr" priority="1"/>
    <vcpusched vcpus="14" scheduler="rr" priority="1"/>
    <vcpusched vcpus="15" scheduler="rr" priority="1"/>
    <iothreadsched iothreads="1" scheduler="fifo" priority="98"/>
  </cputune>
  • I experimented with a zram device of 5gb as a shared SLOG (I run on a UPS) - pools don’t import automatically after a reboot - so am creating a SSD stripe for the SLOG (to be used for my main SATA pool & the new SATA stripe)
  • performance is now good enough & I’ve had no problems moving everything to the new striped SSD pool:
VIRTIO-SCSI 8 queues 64K volblocksize
=====================================

PS C:\WINDOWS\system32> winsat disk -drive G
Windows System Assessment Tool
....
> Disk  Random 16.0 Read                       1187.85 MB/s          9.0
> Disk  Sequential 64.0 Read                   8900.57 MB/s          9.9
> Disk  Sequential 64.0 Write                  4473.51 MB/s          9.5
> Average Read Time with Sequential Writes     0.048 ms          8.9
> Latency: 95th Percentile                     0.061 ms          8.9
> Latency: Maximum                             12.453 ms          7.9
> Average Read Time with Random Writes         0.042 ms          8.9
> Total Run Time 00:00:02.69

  • TODO:

    • Move the Windows vm from nvme passthrough => zvol. The zvol performance is good enough / I don’t login to bare metal Windows
    • SLOG / special devices on nvme then become possible for the new SSD pool
    • Running Linux as btrfs raid1 / Windows vm as mirrored zvol on nvme also becomes possible
1 Like