General ZFS Question's

OpenZFS
1

System Architecture

  • My main machine (server located @ home)

    • I plan on running proxmox
      • VM 1: (my homelab OS + the containers & software for my homelab)
      • VM 2: (completely isolated VM separate from my personal stuff

  • I have 2 Main NVMe Drives + 1 external SATA SSD for my homelab file backups

  • I want to configure ZFS via Nix & get individual FS level snapshots on my homelab VM and 2nd VM

  • I dont mind moving away from proxmox & using my homelab OS as my host machine, I could make the 2nd VM via libvirt QEMU on the host homelab OS

Question

  1. Where do I put the ZFS filesystem?
    Should my Proxmox FS be ZFS (closer to baremetal)
    OR
    Should my VM FS be ZFS (which is a VM) - I dont mind passing a whole NVMe drive to my homelab VM
2

Someone will probably provide a better answer but I’ll share some thoughts.

  • I have zero experience with Proxmox but have heard it makes decisions about ZFS that can be hard to modify. I don’t know if that has changed.
  • In general, you want ZFS to work directly with the storage H/W. Nevertheless, it can be implemented using files as storage and I do that for testing. (GitHub - HankB/Fun-with-ZFS: Explore simple ZFS operations)
  • I have heard that it is a Bad Thing to pass drives to a VM. A better approach is to pass the storage controller. I pass two drives to a Debian VM on a Debian host and have not had any problems. The boot drive for the host and storage drives are on the same controller so I cannot pass it to the VM.

I’ve used MD RAID and ZFS on Debian for years (decades?) and have been happy with that. I’ve never felt the urge to try Proxmox but if you prefer a GUI to CLI it’s probably the way to go.

3

I think that Proxmox is probably more than you need for two VMs, if your homelab OS has good support for ZFS I’d run that on bare metal and create ZFS datasets for each VM and container, then you also don’t have to do nested virtualization with the containers.

4

I’v experimented with all the permutations and my verdict: 100% ZFS on the host. Don’t run it inside the VM.

ZFS wants to talk to the disks directly. If you put it in a VM it’s sitting on a virtual disk, so it can’t see the hardware properly, and you end up with copy-on-write on top of copy-on-write.

My Proxmox box is ZFS root and that’s where all the real data lives. Each service gets its own dataset, rpool/sage//, which gets mounted into an LXC at /opt/services/-, and the Docker container mounts that. So it goes ZFS to LXC to container, and every service gets its own snapshots without the guest knowing ZFS exists. Sanoid runs recursively over rpool/sage, so new datasets get picked up on their own.

My other machines are bare metal linux boxes with a disposable root and OS and persistent data for services in the same ZFS layout. I backup all the config and .files. The important datasets snapshot locally and syncoid pushes to the Proxmox box every night.

For your setup:

VMs can’t bind mount datasets the way LXCs can. For a VM you’d use a zvol (whole-disk snapshots only) or virtiofs. I use virtuofs on my GPU VM (I pass the GPU through wholesale).

LXC is the easy way to get per-service snapshots.You just bind the mount for the dataset into the container. Irritatingly, you cant pass an empty root dataset, you have to pass each individual dataset in. All this forces you to manage the snapshots yourself - proxmox won’t snapshot and lxc with binds. And you need to be comfortable with making drive-access containers privileged.

Passing a whole NVMe to the VM works, but then the host and your second VM can’t use it. I wouldn’t bother unless that VM is basically your NAS.

If you want ZFS managed in Nix, honestly I’d skip Proxmox. Run NixOS on bare metal with ZFS set up in your config (sanoid and syncoid are both NixOS modules), and run VM 2 in libvirt on its own zvol. Same idea, ZFS on the host, and the whole thing lives in one config you can rebuild from.

It took a while to get here, but this system, for me at least, is totally seamless. Its definately overkill for 2 containers/VMs, hence the last paragraph, but leaves a lot of room to grow.