Experience with running a large Windows server VM on zfs?

OpenZFS
1

Hey there,

has anyone experience in running a rather large Windows server VM disk with classical KVM on a zfs dataset? I’m talking a separate data disk in the orders of tenths of terabytes.

Current idea is to use qcow2 for the system and the datadisk. I’m a bit worried about the shear size of this virtual disk, and wanted to ask if someone has experience in doing so. Known caveats?

The VM is Windows Server 2022 and as said, separate system and data disk, both formatted to NTFS because I didn’t bother to try ReFS yet.

2

I’m doing this right now. The client with the largest one also has a 10Gbps LAN that they need to get as close to saturating as possible, so the second drive for the VM (the big one) is a raw file instead of qcow2, on a dataset with recordsize=256K (because the throughout bumps from increasing recordsize farther than that were miniscule, not worth it).

1 Like
3

If possible I would like to use qcow2 because I can expose the snapshotting feature of virt-manager to other users but I cannot expose the zfs snapshots due to user unwillingness to work on the terminal.

A mild performance maulus would be acceptable, but I’m not sure how cow-on-cow scales on really larger datasets/qcow2 images.

4

You’ll get a pretty serious performance debuff while the qcow2 is still sparse. And to the person who points out that “sparse” is kind of less of a thing on ZFS than on non-copy-on-write operating sytems: yes, that’s entirely true, which doesn’t change the fact that even on ZFS, you can see up to a 50% performance hit on qcow2s until they’re fully allocated.

I would not recommend qcow2 if you’re doing 10Gbps networking and hoping to get the most out of it. If you’re only doing 1Gbps, though, you’re unlikely to notice any difference.

1 Like
5

Will likely go on with fully pre-allocated qcow2 then (--preallocation=full) and see if that works. This is a 1Gbps scenario, so while we’re not expecting to fully utilize the full server specs, the aim is to have a somehow decent and consistently performing system.

6

Long story short, QCOW2 snapshots are great for recovery, but don’t underestimate the performance hit of QCOW2 snapshots.

Short story long, the following fio benchmarks from almost five years ago do NOT include ZFS. It was for me to see the worst case scenario of KVM guest disk configuration with the best case scenario of an otherwise unused KVM host. In this case, both the host storage and quest storage were using the following:
dmcrypt/LUKS
LVM
ext4

Add mdraid RAID-6 for the host.

The QCOW2 disks were configured using ESS (Empty Sector Suppression)…the worst for performance, but the best for keeping the QCOW2 sizes smaller later for rsync.

There are multiple fio runs to demonstrate that the first run after creating the snapshot was the slowest, the second run was about five times faster than the first run, and the third run (with the QCOW2 snapshot destroyed) was about ninety times faster than the first run.

Until ZoL gets Native Encryption working reliably with ZFS snapshots, QCOW2 snapshots are what I will keep using, but there is a significant performance hit.

Exact command used during benchmarking of KVM guests:
fio --name=random-write --ioengine=posixaio --rw=randwrite --bs=4k --numjobs=1 --size=4g --iodepth=1 --runtime=60 --time_based --end_fsync=1

Supermicro X10SRA / LSI Logic SAS2008 PCI-Express Fusion-MPT SAS-2 / mdraid RAID-6 of four Samsung SSD 860 EVO 1TB:
QEMU-KVM guest / Virtio Block Device / ESS QCOW2 with live snapshots:
2020-02-07 - WRITE: bw=978KiB/s (1001kB/s), 978KiB/s-978KiB/s (1001kB/s-1001kB/s), io=341MiB (358MB), run=357516-357516msec (1st run)
2020-02-07 - WRITE: bw=4586KiB/s (4696kB/s), 4586KiB/s-4586KiB/s (4696kB/s-4696kB/s), io=663MiB (695MB), run=147988-147988msec (2nd run)
2020-02-08 - WRITE: bw=88.9MiB/s (93.2MB/s), 88.9MiB/s-88.9MiB/s (93.2MB/s-93.2MB/s), io=5547MiB (5817MB), run=62425-62425msec (3rd run - snapshots destroyed)
2020-02-08 - WRITE: bw=500KiB/s (512kB/s), 500KiB/s-500KiB/s (512kB/s-512kB/s), io=366MiB (384MB), run=749747-749747msec (4th run)
2020-02-08 - WRITE: bw=2804KiB/s (2871kB/s), 2804KiB/s-2804KiB/s (2871kB/s-2871kB/s), io=222MiB (233MB), run=80980-80980msec (5th run)
2020-02-08 - WRITE: bw=84.7MiB/s (88.8MB/s), 84.7MiB/s-84.7MiB/s (88.8MB/s-88.8MB/s), io=5543MiB (5812MB), run=65431-65431msec (6th run - snapshots destroyed)

As always, thanks to “mercenary_sysadmin” for recommending fio!

2 Likes
7

Thank you for the input. My main take-away is that the performance suffers in unpredictable ways, the best of all ways! /s.

What I also take is that the cow nature of zfs as underlying filesystem appears to have a minor (if any) impact on the macro level, given that the performance malus from unallocated qcow2 images is much worse.