Raw Sparse Files vs. zVols

[SinisterPisces]

Reference:

Samba Performance for MacOS Clients (TrueNAS Scale)

This isn’t going to really make an impact, if APFS is being run in a VM on top of ZFS storage (or on ZFS storage via iSCSI), because ZFS will nod its head sagely at the “location” APFS asks to put sectors/clusters, and go right the hell about writing them wherever ZFS would normally write them anyway, since ZFS is essentially presenting the APFS filesystem with a fake, virtual block address map that ZFS itself translates down to the actual physical sectors of the drives in question. With that …

tl;dr: Is there actually a way to get Proxmox to use ZFS sparse files for VM storage, or are we stuck with zVols there?

(I’m going to ask this on the Proxmox forum, too, but I wanted to undestand the ZFS side of it first before diving into the Proxmox Way of Things.)

Hello,

Looking through this thread again, I noticed a comment from @mercenary_sysadmin that I realize I don’t quite understand (emphasis added):

With that said, if you’re using zvols, you’re going to lose a LOT of performance right there. Zvol performance sucks, a lot. You’re generally much better off using a sparse file (eg truncate -s 100G hundredgig.raw) rather than a zvol.

My understanding is that Proxmox stores VM disks as zVols by default when using the RAW disk image format. See: Proxmox VE Administration Guide

Image Format

On each controller you attach a number of emulated hard disks, which are backed by a file or a block device residing in the configured storage. The choice of a storage type will determine the format of the hard disk image. Storages which present block devices (LVM, ZFS, Ceph) will require the raw disk image format, whereas files based storages (Ext4, NFS, CIFS, GlusterFS) will let you to choose either the raw disk image format or the QEMU image format.

• the QEMU image format is a copy on write format which allows snapshots, and thin provisioning of the disk image.
• the raw disk image is a bit-to-bit image of a hard disk, similar to what you would get when executing the dd command on a block device in Linux. This format does not support thin provisioning or snapshots by itself, requiring cooperation from the storage layer for these tasks. It may, however, be up to 10% faster than the QEMU image format. [35]
• the VMware image format only makes sense if you intend to import/export the disk image to other hypervisors.

I realized as I’m writing this out that the above must refer to QEMU Raw format which is implemented on top of a zVol, not a ZFS sparse file of the type mentioned by @mercenary_sysadmin , which I don’t think I’ve ever worked with before.

[mercenary_sysadmin]

@SinisterPisces

tl;dr: Is there actually a way to get Proxmox to use ZFS sparse files for VM storage, or are we stuck with zVols there?

If you create a sparse file on the command line, it’s possible to get Proxmox to host a VM based on that sparse file. But at that point, you’ve really got to ask yourself why you’re still bothering with Proxmox in the first place.

If you actually want it to still feel like Proxmox and do everything in the web GUI, yeah, you’re stuck with zvols.

[SinisterPisces]

Thanks for clarifying this.

is there documentation that you’ve seen that actually covers using .raw files instead of zVols in Proxmox? I’d still like to learn about it, but I’ve had some trouble finding anything written. All the search results I’ve gotten so far are for QEMU .raw disk images and I haven’t figured out how to filter those out yet.

[mercenary_sysadmin]

No, sorry, I’ve simply heard from proxmox users who say that it can be done on the command line, and once done, you can at least manage the resulting VM in the Web interface.

[SinisterPisces]

No need to apologize. That’s really useful information to have.

If they can be managed from the web GUI, that means they (apparently) don’t break the Proxmox Experience®, and could be enabled on the back-end someday.

From the PVE dev mailing list, there’s some sort of huge overhaul of the storage infrastructure happening right now, likely targeting PVE 9. Maybe something interesting will happen in this direction. It would be cool to just have a more performant storage system just … show up.