Linux home directory on ZFS

OpenZFS
1

Just finished listening to 2.5 admins 192 and it got me curious how other people have been handling this.

How I’ve been going about this is I go through the normal Ubuntu installer, but I limit the size of the root partition to 50GB or so.

Then, once I’ve booted into Ubuntu I install zfsutils-linux, create my pool from another partition on the same disk, create a dataset for my home directory, move the content of /home to it, and then just set the mountpoint of the dataset I created to /home.

Bit of an unwieldy process as installations go, though definitely worth it for the features of ZFS.

I’m still currently dual booting with Windows (damn you, anti cheat!), so I doubt that ZFSBootMenu is currently an option for me.

What does your ZFS home directory installation look like?

2

ZFSBootMenu can handle dual booting just fine, though you would generally handle the selection process between multiple operating systems with your UEFI firmware boot selector, rather than one built into either grub or Windows.

You also have the option of never installing a boot loader from ZBM onto the drive at all, instead using a USB bootloader to boot into ZBM.

What does your ZFS home directory installation look like?

I use a separate dataset, but mounted beneath my ZBM root for the distro. So, eg:

pool
  |---ROOT
       |---Ubuntu
       |     |---home
       |
       |---Fedora
             |---home

This way, I have cleanly separated home directories so that one distro doesn’t conflict with another, even when both have my home directory mounted at /home/jrs–but I can also still choose to replicate, roll forward, clone, etc my home directory independently from the base distribution (and vice versa).

3
  • Laptop - Debian with root on ZFS (bpool/rpool config) and using native encryption. Single NVME SSD.
  • Desktop - Debian with root on ZFS (bpool/rpool config) on a single NVME SSD. I should probably mirror the system SSD but didn’t have the right size 2nd NVME SSD on hand. I have an 8TB HDD installed and run frequent ZFS send/recv both pools to the HDD to support recovery should that be needed.
  • X86_64 server - Debian with root on ZFS on a mirrored pair of cheap SATA SSDs. Storage pool is a 5 drive RAIDZ2 consisting of a mix of 4TB and 6TB HDDs. (I’m in a long drawn out process of upgrading storage space to all 6TB HDDs) My laptop and desktop have backups of “important stuff” going to the storage pool.
  • Remote X86_64 server boots Debian from a single SSD with no ZFS. A VM runs Debian with two SATA HDDs passed through and configured as ZFS Mirror. I put ZFS in a VM because a couple years ago an upgrade required console input (unattended-upgrades) and left the system unbootable following the next reboot. It was 5 months before I was on site to fix this. Putting ZFS in a VM gives me remote console access to that host should this ever happen again.)
  • Various Raspberry Pi 3/3B/4B/CM4/5 Debian (except Pi 5) with root on EXT4 since I haven’t figured out how to put root on ZFS. I resize the root (EXT4) filesystem and use the remaining space for a ZFS pool. I put things like storage for Docker containers and active directories like /var in the ZFS pool and of course, my home directory. The Raspberry Pis have an annoying tendency to render themselves unable to boot following updates so I copy the MBR, boot and root partitions to a ZFS pool and then copy either that or the entire pool to a Pi 4B based file server. Most of these boot and run from (USB connected) SATA or (PCIe connected) NVME SSDs. I don’t tolerate slow storage well. :smiley:
  • Pi 4B boots from an SD card and has attached a drive bay with 2x 8TB HDDs. The entire 8TB drives are used for ZFS and contain backups (including a complete copy of the storage pool on my X86_64 server.) I’ve also put “busy” directories in the pool to reduce wear on the SD boot device. I’m not thrilled about using USB to connect storage, but this has been going well for nearly two years now. The only issue I’ve run into is that if I connect a third drive, things start falling over pretty quickly, even if I put the third drive on a different host USB3 port. A couple months ago I replaced 2 6TB HDDs with 8TB HDDs and I could not do that with all three drives connected.

Ask away if any of this does not make sense. :wink:

4

I’ve always kept /home on the same partition or dataset as the rest of the OS. I don’t share /home with other distro installs. If I’m (re)installing the OS, I start fresh with /home too. I keep all of my personal data on a separate partition or dataset mounted under /home.

I have root on ZFS with native encryption. I use plain old grub and haven’t had any compatibility issues since the initrd is doing the work of booting into the OS.

5

All my systems - laptops included - are built with my installer

/home/user is a dataset separate from root. It uses zfsbootmenu (awesome btw) as well.

I have dedicated backup boxes with zfs send/recv going between them for backups. Mostly zfs-auto-snapshot and zfs-backup, but moving to zrepl. Unfortunately zrepl seems slow - no new commits in 4 months :frowning:

1 Like