Recommendations for Storing VM data on host ZFS pool (dilemmas!)

OpenZFS
1

I’m setting up a new VM on my system to run all of my container apps. The VM exists as a raw sparse image file in a ZFS dataset on the host. All well and good, but it occurred to me that because the VM is running uBlue uCore (a Fedora-based OS derivative) with XFS as the default filesystem, I lose the benefits of ZFS for my container data living in /var/lib/data/containers. I don’t get features like granular ZFS snapshots or rollbacks for individual containers.

I started looking into ways to keep my VM’s container data on the host ZFS pool. Ideally, I would like a separate dataset for each container’s data. The challenge is allowing the VM to access that data safely and with decent performance.

  1. ZVOLs as block storage directly attached to the VM: Everything I’ve read warns against this, saying there are very few valid use cases and to generally stay away from it.

  2. NFS and VirtioFS: My containers host self-hosted apps with databases (SQLite, Postgres, Redis, etc.), so I am worried about database corruption over network shares. I could try a hybrid approach, storing bulk data (like MP4 files for Audiobookshelf) in a host dataset via VirtioFS (preferred?), but this seems messy and full of trade-offs.

  3. Splitting the Pool: I even considered rebuilding my 3‑disk RAIDZ1 pool by partitioning the disks to create two separate pools: one for the host and one passed directly to the VM. However, that seems like a lot of work and introduces its own performance trade offs.

VirtioFS seemed interesting, but I’d have to have a separate mount for every container data folder, that probably doesn’t scale well from the standpoint of management.

Am I wrong about ZVOLs? I’m considering creating individual ZVOLs, attaching them as VirtIO block devices, and mounting them for specific databases. Then I could use VirtioFS for bulk media. This seems tricky because the ZVOLs would need to be tuned (e.g., block sizes) to accommodate XFS running on top of ZFS.

There doesn’t seem to be a clean solution, just trade-offs. Thoughts? Any suggestions are greatly appreciated! Maybe I’ll learn something with the upcoming “12 days of ZFS” that might help my issue..

2

What is your hypervisor? The hypervisor may give you better options for mounting a host directory directly into the VM.

Also, is there a reason you’re running containers inside a VM instead of next to it on the host? If you can run containers directly on the hypervisor (Proxmox definitely can, ESXi will if you have $), then you can have all the goodness ZFS does for containers without adding the layers you’re talking about.