tl;dr: Is there actually a way to get Proxmox to use ZFS sparse files for VM storage, or are we stuck with zVols there?
(I’m going to ask this on the Proxmox forum, too, but I wanted to undestand the ZFS side of it first before diving into the Proxmox Way of Things.)
Hello,
Looking through this thread again, I noticed a comment from @mercenary_sysadmin that I realize I don’t quite understand (emphasis added):
With that said, if you’re using zvols, you’re going to lose a LOT of performance right there. Zvol performance sucks, a lot. You’re generally much better off using a sparse file (eg truncate -s 100G hundredgig.raw) rather than a zvol.
My understanding is that Proxmox stores VM disks as zVols by default when using the RAW disk image format. See: Proxmox VE Administration Guide
Image Format
On each controller you attach a number of emulated hard disks, which are backed by a file or a block device residing in the configured storage. The choice of a storage type will determine the format of the hard disk image. Storages which present block devices (LVM, ZFS, Ceph) will require the raw disk image format, whereas files based storages (Ext4, NFS, CIFS, GlusterFS) will let you to choose either the raw disk image format or the QEMU image format.
the QEMU image format is a copy on write format which allows snapshots, and thin provisioning of the disk image.
the raw disk image is a bit-to-bit image of a hard disk, similar to what you would get when executing the dd command on a block device in Linux. This format does not support thin provisioning or snapshots by itself, requiring cooperation from the storage layer for these tasks. It may, however, be up to 10% faster than the QEMU image format. [35]
the VMware image format only makes sense if you intend to import/export the disk image to other hypervisors.
I realized as I’m writing this out that the above must refer to QEMU Raw format which is implemented on top of a zVol, not a ZFS sparse file of the type mentioned by @mercenary_sysadmin , which I don’t think I’ve ever worked with before.
If you create a sparse file on the command line, it’s possible to get Proxmox to host a VM based on that sparse file. But at that point, you’ve really got to ask yourself why you’re still bothering with Proxmox in the first place.
If you actually want it to still feel like Proxmox and do everything in the web GUI, yeah, you’re stuck with zvols.
is there documentation that you’ve seen that actually covers using .raw files instead of zVols in Proxmox? I’d still like to learn about it, but I’ve had some trouble finding anything written. All the search results I’ve gotten so far are for QEMU .raw disk images and I haven’t figured out how to filter those out yet.
No, sorry, I’ve simply heard from proxmox users who say that it can be done on the command line, and once done, you can at least manage the resulting VM in the Web interface.
No need to apologize. That’s really useful information to have.
If they can be managed from the web GUI, that means they (apparently) don’t break the Proxmox Experience®, and could be enabled on the back-end someday.
From the PVE dev mailing list, there’s some sort of huge overhaul of the storage infrastructure happening right now, likely targeting PVE 9. Maybe something interesting will happen in this direction. It would be cool to just have a more performant storage system just … show up.
A really easy way to accomplish this is to add a “Directory” storage type to the cluster and point it at the pool mount point. It then treats everything as if it were just a normal filesystem, allowing .raw or .qcow2 storage instead of using zvols.
I go a little further and manually create datasets on all nodes for the pve vm numbers, e.g. /images/33029 so that I have more granular per-VM control with replication or rollback at the OS/zfs level. I do it on all nodes so that I’m covered if I migrate the VM to another node either temporarily or permanently.
