Background:
I am about to migrate my Nextcloud server from a TurnKey Linux LXC container to a Docker setup. I will create an LXC container that will run my docker setup. My reason for doing so is that the TurnKey LXC container I have been using so far has been a hassle to keep up to date, especially the “support” parts like MariaDB, PHP, etc.
Question:
What is the best practice for carving out ZFS Datasets on proxmox for a container set like this. I am planning to have differentiated datasets for different parts in order to be able to tune ZFS and the underlying storage layers to suit their uses. While this split is partly specific to Nextcloud, I am of the opinion, that it does have a wider use across other (self-hosted) server installations, like Immich, etc.
My planned datasets are:
| dataset |
storage type |
use |
ZFS tuning |
| rpool/[…]/vmdatanvme/nextcloud |
nvme |
nextcloud parent dataset |
compression=zstd, atime=off, recordsize=128K |
| rpool/[…]/vmdatanvme/nextcloud/rootfs(*) |
nvme |
nextcloud container boot disk |
(inherited tuning) |
| rpool/[…]/vmdatanvme/nextcloud/db-mariadb |
nvme |
nextcloud database |
(inherited tuning), recordsize=16K, logbias=throughput |
| rpool/[…]/vmdatanvme/nextcloud/redis |
nvme |
nextcloud redis database cache |
(inherited tuning), recordsize=64K |
| tank/[…]/vmdatahdd/nextcloud/storage |
hdd, possibly ssd |
nextcloud file storage |
(inherited tuning), recordsize=1M |
If I create all but the container rootfs dataset, they are all in a neat package, and I can add storage datasets for Collabora and other supporting server containers under the nextcloud dataset package(s)
creating the rootfs during LXC container creation, using the Web UI, it will create the rootfs dataset under rpool/[…]/vmdatanvme/subvol-1001-disk-0 and thus outside of my dataset “package”. I could create an extra zfs-based storage for rpool/[…]/vmdatanvme/nextcloud, but that scales badly to other hosts and other containers.
If I create the rootfs dataset upfront, i break the way proxmox sets up disks - will that give problems when I have a proxmox cluster?
You could use the default local-zfs storage and set the ZFS properties you want after creation of the virtual disk(s)?
If you do create specific storages with specific properties I’d recommend to keep them general, not app/guest specific. I.e database, storage, media, whatever.
I do this myself (podman) but docker inside a CT can sometimes break and major updates and proxmox does not recommend this.
The reason that I want to do it as I described it, is to keep all storage in one “package”. Currently I only have one Proxmox host - well I actually have two, but not in the same cluster, and I plan to decommision the other as a proxmox host - it may become a Proxmox Backupserver on bare metal, or just a storage/backup host with storage upon ZFS. But when I add a second host, I would like to bundle the containers into one package, storage wise. This package may be split across several pools (i.e. NVMe, SSD, HDD) but by naming them from their “native placement” and then setting up those datasets on what is available on the specific host, it should be possible to fairly easily migrate them - or at least that was my hope.
I understand your suggestion about more general storages, which might be simpler in proxmox, they would end up in a heap, so to speak. Immich DB, Jellyfin DB, and Nextcloud DB would end up in the same place, rather than separete.
After a long time battling this question I can answer clearly! Let Proxmox create and own the rootfs. Don’t try to fit it into your dataset layout.
I treat the root and OS as disposable everywhere, both in LXCs and on my bare metal boxes. Everything I care about lives in datasets I create myself and bind mount into the container. Proxmox doesn’t manage those, so it doesn’t matter where it puts subvol-whatever-disk-0.
My layout is one dataset per service, rpool/sage/[host]/[service], mounted into the LXC at /opt/services/[host]-[service]. The compose file and all the container data live in there. If the LXC dies I make a new one, mount the dataset back in, and run compose up.
You can still group things the way you want, with a parent dataset for Nextcloud and children for the database and the files. A few things I learned painfully:
Binding the parent doesn’t bring the children with it. Each child dataset needs its own mount point in the LXC config. This gets a but messy (especially b/c PM reformats my hand-written config masterpieces and 12 comes before 3!)
In Proxmox, pct snapshot won’t touch an LXC with bind mounts, and vzdump backups skip them, so you handle snapshots yourself. I run sanoid recursively over rpool/sage, so new datasets get picked up without any setup.
Changes to an LXC’s mount config don’t take effect until you restart the container. If Docker starts before the mount is there, it writes into the rootfs and your snapshots never see that data. Check the mount inside the container before you start anything.
I’d skip the dataset for Redis. It’s a cache, so there’s nothing in it worth snapshotting. I only create datasets for persistant data and I have all the docker composes and .files in a repo.