Best Practices for Setting Up OpenZFS on Ubuntu

OpenZFS
1

Hi everyone,

I am new to OpenZFS and looking to set it up on my Ubuntu system. I have read some documentation but I would appreciate some guidance on best practices from those with experience.

  1. What are the essential steps I should follow to ensure a smooth installation.
  2. Are there any common pitfalls I should avoid when configuring OpenZFS.
  3. Can anyone recommend specific resources or tutorials for beginners.

Additionally I am curious about the most effective ways to manage storage pools and snapshots. Any tips or advice would be greatly appreciated.
I am also read this topic on this site https://www.reddit.com/r/zfs/comments/1dbszym/ubuntu_blue-prism but I could not find anything helpful.

Thank you in advance for your help.

Cheers
[James]

2
  • I would not recommend setting up ZFS root on Ubuntu using the Canonical installer
  • If you really want a ZFS root, I’d recommend the ZFSBootmenu route https://zfsbootmenu.org/
  • If you don’t really NEED a ZFS root, consider just doing the OS itself on a 100GiB ext4 partition (or ext4 on an mdraid1 of 100GiB partitions), and allocating the rest of the space to a pool. Then you can zfs create mypool/home, rsync your user directories into it (or individual datasets beneath it, if you’d like to be able to replicate and/or roll them back independently of one another), then zfs set mountpoint=/home mypool/home and you’re off to the races. If something happens to the base OS… just wipe it out and reinstall; only takes about 15 minutes and you get your home directory(ies) back as soon as you reimport the pool.
3

root on zfs can be awesome. Just something as simple as snapshot’ing before doing updates can be a lifesaver. Combined with zfsbootmenu to allow easy rollbacks or cloning, and it’s - what’s the phrase ? Slicker-n snot on a doorknob :stuck_out_tongue:

It’s definitely more work to get set up though. And agreed, the Canonical way of root-on-zfs feels cumbersome.

There are a few ways of handling backups/replication. Sanoid/Syncoid is one that seems highly regarded, though I haven’t used it myself. Older setup of zfs-auto-snapshot (zfsonlinux/zfs-auto-snapshot) and zfs-backup (adaugherity/zfs-backup) works well, and I use that on my main media box. I’ve become fond of zrepl (zrepl/zrepl) and have that working on most systems.

Shameless plug - I have my own opinionated root-on-zfs setup (GitHub - Halfwalker/ZFS-root: Set up root-on-zfs using whole disk, with dracut and zfsbootmenu) that builds a system using zfsbootmenu, dracut (instead of initramfstools), rEFInd (instead of efibootmgr), with optional LUKS or native zfs encryption. This is what I use to build all my own systems, from servers to workstations to laptops. There are several other similar examples out there - check them out and pick one if it feels right.

As for datasets, think about the type of data to go in them. Large files like videos ? Use a larger blockside. Running a database ? There are specific tuning params to consider. Datasets are cheap - break up your data in logical chunks, keeping in mind what kind of data, how important, how often to snapshot/replication etc., what kind of throughput you might need (VMs ? docker/kubernetes ?)

For example, my home dataset is snapshot’d every 15mins.
Keep 4x 15min snaps (so the last hour at 15min intervals)
Keep 24x hourly snaps
Keep 30x daily snaps
Keep 12x monthly snaps

Root dataset is also snapshot’d every 15mins but only if more than 2meg has changed on the dataset. That keeps the number of snaps down somewhat. Also, everytime apt install is called, a hook run a snapshot before the install, making it easy to rollback to pre-install of any new/updated stuff.

zrepl manages those snapshots and replicates them to a 2nd device on the box.
A neat utility httm makes it easy to search back through snapshots.

I’ve had instances when Sonarr has “updated” some shows, totally borking them in the process. And I don’t find out for 6 months or more. Mount the snapshot from 7 months ago with sudo mount -t zfs <snapshot> /mnt and copy the old files back. Boom, done.

The backup box with 24 old drives has 2 years of replication backups. I’ve only had to pull from that twice in the last 6 years or so. But boy I was glad it was there …

4

Thank you, Jim, for always providing a different perspective based on your experience. I have a question regarding the third solution you mentioned.

If you install the root on ext4, do you mean that all the installed software and configurations (e.g., /etc) also end up on ext4? In this case, how do you restore a fully functional system? Do you reinstall all the software and redo all the configurations?

5

Correct, reinstall. If something is too complex for me to want to restore it with a few apt install commands and minimal off-the-top-of-my-head configuration, then I don’t do it on the host operating system at all–i do it in a VM, and the VM itself is stored in a dataset (as is the folder that contains all the XML files that define the VMs).

1 Like
6

I thought so! Recently, I’ve been using ZFSBootMenu a lot, both on workstations and servers, and I have to say it works really well. Recently, on my laptop running Fedora, a system update went wrong, resulting in the loss of all programs and configurations. A rollback, and in 10 seconds, the magic happened… and voilà, my computer was back to how it was before!

7

I’ve got a slightly off topic question: How do you manage to relocate the VM definition files? Can this be configured or are the paths symlinked? The only answer I could find is that symlinking doesn’t work because the networks subfolder needs to be accessible at boot, when the dataset isn’t available yet. I haven’t tried myself yet but would appreciate the ideal solution :slight_smile:

8

You can’t get away with putting the entirety of /etc under a mountpoint that won’t be available until after the boot sequence gets around to loading the kernel module, but you CAN relocate just the QEMU directory itself to one.

zfs set mountpoint=/etc/libvirt/qemu poolname/images/qemu

It’s technically kinda goofy putting the dataset for the image definitions directly beneath poolname/images along with the datasets for the actual images themselves, but doing it that way means my sanoid and syncoid configs don’t need to account for any datasets above poolname/images, which they will DEFINITELY already be configured to maintain.

1 Like