Best Practice for using ZFS for containers & VMs under proxmox

Proxmox
1

Background:
I am about to migrate my Nextcloud server from a TurnKey Linux LXC container to a Docker setup. I will create an LXC container that will run my docker setup. My reason for doing so is that the TurnKey LXC container I have been using so far has been a hassle to keep up to date, especially the “support” parts like MariaDB, PHP, etc.

Question:
What is the best practice for carving out ZFS Datasets on proxmox for a container set like this. I am planning to have differentiated datasets for different parts in order to be able to tune ZFS and the underlying storage layers to suit their uses. While this split is partly specific to Nextcloud, I am of the opinion, that it does have a wider use across other (self-hosted) server installations, like Immich, etc.

My planned datasets are:

dataset storage type use ZFS tuning
rpool/[…]/vmdatanvme/nextcloud nvme nextcloud parent dataset compression=zstd, atime=off, recordsize=128K
rpool/[…]/vmdatanvme/nextcloud/rootfs(*) nvme nextcloud container boot disk (inherited tuning)
rpool/[…]/vmdatanvme/nextcloud/db-mariadb nvme nextcloud database (inherited tuning), recordsize=16K, logbias=throughput
rpool/[…]/vmdatanvme/nextcloud/redis nvme nextcloud redis database cache (inherited tuning), recordsize=64K
tank/[…]/vmdatahdd/nextcloud/storage hdd, possibly ssd nextcloud file storage (inherited tuning), recordsize=1M

If I create all but the container rootfs dataset, they are all in a neat package, and I can add storage datasets for Collabora and other supporting server containers under the nextcloud dataset package(s)

creating the rootfs during LXC container creation, using the Web UI, it will create the rootfs dataset under rpool/[…]/vmdatanvme/subvol-1001-disk-0 and thus outside of my dataset “package”. I could create an extra zfs-based storage for rpool/[…]/vmdatanvme/nextcloud, but that scales badly to other hosts and other containers.

If I create the rootfs dataset upfront, i break the way proxmox sets up disks - will that give problems when I have a proxmox cluster?

2

You could use the default local-zfs storage and set the ZFS properties you want after creation of the virtual disk(s)?
If you do create specific storages with specific properties I’d recommend to keep them general, not app/guest specific. I.e database, storage, media, whatever.
I do this myself (podman) but docker inside a CT can sometimes break and major updates and proxmox does not recommend this.

3

The reason that I want to do it as I described it, is to keep all storage in one “package”. Currently I only have one Proxmox host - well I actually have two, but not in the same cluster, and I plan to decommision the other as a proxmox host - it may become a Proxmox Backupserver on bare metal, or just a storage/backup host with storage upon ZFS. But when I add a second host, I would like to bundle the containers into one package, storage wise. This package may be split across several pools (i.e. NVMe, SSD, HDD) but by naming them from their “native placement” and then setting up those datasets on what is available on the specific host, it should be possible to fairly easily migrate them - or at least that was my hope.

I understand your suggestion about more general storages, which might be simpler in proxmox, they would end up in a heap, so to speak. Immich DB, Jellyfin DB, and Nextcloud DB would end up in the same place, rather than separete.