Linux home directory on ZFS

OpenZFS
1

Just finished listening to 2.5 admins 192 and it got me curious how other people have been handling this.

How I’ve been going about this is I go through the normal Ubuntu installer, but I limit the size of the root partition to 50GB or so.

Then, once I’ve booted into Ubuntu I install zfsutils-linux, create my pool from another partition on the same disk, create a dataset for my home directory, move the content of /home to it, and then just set the mountpoint of the dataset I created to /home.

Bit of an unwieldy process as installations go, though definitely worth it for the features of ZFS.

I’m still currently dual booting with Windows (damn you, anti cheat!), so I doubt that ZFSBootMenu is currently an option for me.

What does your ZFS home directory installation look like?

2

ZFSBootMenu can handle dual booting just fine, though you would generally handle the selection process between multiple operating systems with your UEFI firmware boot selector, rather than one built into either grub or Windows.

You also have the option of never installing a boot loader from ZBM onto the drive at all, instead using a USB bootloader to boot into ZBM.

What does your ZFS home directory installation look like?

I use a separate dataset, but mounted beneath my ZBM root for the distro. So, eg:

pool
  |---ROOT
       |---Ubuntu
       |     |---home
       |
       |---Fedora
             |---home

This way, I have cleanly separated home directories so that one distro doesn’t conflict with another, even when both have my home directory mounted at /home/jrs–but I can also still choose to replicate, roll forward, clone, etc my home directory independently from the base distribution (and vice versa).

3
  • Laptop - Debian with root on ZFS (bpool/rpool config) and using native encryption. Single NVME SSD.
  • Desktop - Debian with root on ZFS (bpool/rpool config) on a single NVME SSD. I should probably mirror the system SSD but didn’t have the right size 2nd NVME SSD on hand. I have an 8TB HDD installed and run frequent ZFS send/recv both pools to the HDD to support recovery should that be needed.
  • X86_64 server - Debian with root on ZFS on a mirrored pair of cheap SATA SSDs. Storage pool is a 5 drive RAIDZ2 consisting of a mix of 4TB and 6TB HDDs. (I’m in a long drawn out process of upgrading storage space to all 6TB HDDs) My laptop and desktop have backups of “important stuff” going to the storage pool.
  • Remote X86_64 server boots Debian from a single SSD with no ZFS. A VM runs Debian with two SATA HDDs passed through and configured as ZFS Mirror. I put ZFS in a VM because a couple years ago an upgrade required console input (unattended-upgrades) and left the system unbootable following the next reboot. It was 5 months before I was on site to fix this. Putting ZFS in a VM gives me remote console access to that host should this ever happen again.)
  • Various Raspberry Pi 3/3B/4B/CM4/5 Debian (except Pi 5) with root on EXT4 since I haven’t figured out how to put root on ZFS. I resize the root (EXT4) filesystem and use the remaining space for a ZFS pool. I put things like storage for Docker containers and active directories like /var in the ZFS pool and of course, my home directory. The Raspberry Pis have an annoying tendency to render themselves unable to boot following updates so I copy the MBR, boot and root partitions to a ZFS pool and then copy either that or the entire pool to a Pi 4B based file server. Most of these boot and run from (USB connected) SATA or (PCIe connected) NVME SSDs. I don’t tolerate slow storage well. :smiley:
  • Pi 4B boots from an SD card and has attached a drive bay with 2x 8TB HDDs. The entire 8TB drives are used for ZFS and contain backups (including a complete copy of the storage pool on my X86_64 server.) I’ve also put “busy” directories in the pool to reduce wear on the SD boot device. I’m not thrilled about using USB to connect storage, but this has been going well for nearly two years now. The only issue I’ve run into is that if I connect a third drive, things start falling over pretty quickly, even if I put the third drive on a different host USB3 port. A couple months ago I replaced 2 6TB HDDs with 8TB HDDs and I could not do that with all three drives connected.

Ask away if any of this does not make sense. :wink:

4

I’ve always kept /home on the same partition or dataset as the rest of the OS. I don’t share /home with other distro installs. If I’m (re)installing the OS, I start fresh with /home too. I keep all of my personal data on a separate partition or dataset mounted under /home.

I have root on ZFS with native encryption. I use plain old grub and haven’t had any compatibility issues since the initrd is doing the work of booting into the OS.

5

All my systems - laptops included - are built with my installer

/home/user is a dataset separate from root. It uses zfsbootmenu (awesome btw) as well.

I have dedicated backup boxes with zfs send/recv going between them for backups. Mostly zfs-auto-snapshot and zfs-backup, but moving to zrepl. Unfortunately zrepl seems slow - no new commits in 4 months :frowning:

1 Like
6

Sorry for (mildly) necroposting, but how do you mount your home dataset on boot? They have both the same mountpoint, am I right?

7

Hi! No worries about necro anything, it’s a valid question. And a bit confusing to figure out!

What you do is, you don’t have an explicit mountpoint for the home dataset in each of your operating systems at all. You simply nest the home dataset for each operating system as a child of the root dataset for that OS.

So, let’s return to that diagram earlier:

pool
  |---ROOT
       |---Ubuntu
       |     |---home
       |
       |---Fedora
             |---home

If you boot Fedora, then pool/ROOT/Fedora gets mounted (by ZFSBootMenu) on /. This, in turn, means that pool/ROOT/Fedora/home gets mounted on /home, not because it has an explicitly set mountpoint (it doesn’t!) but because it’s automatically mounted directly beneath /pool/ROOT/Fedora as its child dataset.

Similarly, if you boot Ubuntu, ZBM mounts pool/ROOT/Ubuntu as /, and therefore /pool/ROOT/Ubuntu/home gets mounted beneath it as /home.

When you boot Fedora, Ubuntu’s datasets aren’t mounted at all, anywhere (unless you explicitly mount them at the command line temporarily) and vice versa.

This is in contrast to the ZBM-documentation-suggested method of keeping a separate home dataset under pool/home with an explictly ZFS-set mountpoint of /home. Doing it that way means you have the same “home” dataset regardless of which OS you boot, which can lead to problems, which is why I’m recommending not doing it their way in the first place. :slight_smile:

When and why MIGHT you want /home to be the same dataset, no matter which OS you boot? Well, let’s say you are running Ubuntu 22.04, and you want to upgrade it to 24.04. You can clone your existing Ubuntu 22.04 root into a new dataset, boot from that, and upgrade it without affecting your /home at all, or having to maintain two separate versions of /home. This appeals to a lot of folks.

With that said, I wouldn’t personally recommend doing that either. While it’s a lot LESS likely for your home directory to get you into trouble if shared between two versions of Ubuntu, it can absolutely still happen–what if your upgraded apps in your upgraded 22.04 version of Ubuntu rewrite the dotfiles they keep under your home directory, then you decide you actually need to go back to 22.04? Now you may have severely broken applications that don’t understand directives in their own dotfiles, with no clear and simple path back.

8

Yes, I wasn’t thinking of setting the mountpoint explicitly, but I didn’t say it :slight_smile:
But both pool/ROOT/Ubuntu/home and pool/ROOT/Fedora/home will have the same effective mountpoint of /home and canmount=on. It is not a problem? For the root filesystem, as per ZBM documentation, canmount is set to noauto, so there is no conflict there, but I’m not sure how to manage the children datasets

9

I think you’re not quite getting what I’m saying. There is no explicit mountpoint set on ROOT/fedora/home or ROOT/ubuntu/home.

If ZBM mounts pool/ROOT/fedora on /, then pool/ROOT/fedora/home is automounted as /home, and pool/ROOT/ubuntu/home is not mounted anywhere at all, just as pool/ROOT/ubuntu is not mounted anywhere at all.

If you import the pool into a different system and zfs get mountpoint on both pool/ROOT/ubuntu/home and pool/ROOT/fedora/home, you’ll see both are set to “inherit” rather than to any explicit mountpoint.

By contrast, if you set up a system the way ZBM recommends, with pool/home as a dataset with explicit mountpoint of /home, then should you import that pool into a different system (rather than booting into it with ZBM) you’ll see that zfs get mountpoint pool/home returns /home.

If you wanted TEMPORARY access to Fedora’s home directories while Ubuntu is booted and mounted, you’d do something along the lines of zfs mount pool/ROOT/fedora/home /tmp/home. That would mount all of Fedora’s home directories TEMPORARILY at /tmp/home, WITHOUT changing the ZFS property “mountpoint” permanently.

10

Yeah, I get what you’re saying. What is not clear to me is who is mounting what and when.

The ZBM doc says (emphasys mine)

It is important to set the property canmount=noauto on any file systems with mountpoint=/ (that is, on any additional boot environments you create). Without this property, the OS will attempt to automount all ZFS file systems and fail when multiple file systems attempt to mount at /; this will prevent your system from booting. Automatic mounting of / is not required because the root file system is explicitly mounted in the boot process.

Now, the doc is explicit about /, but the the same reasoning is valid for /home, is it not?
It appears that ZBM will have no problem booting, but the actual OS might: if I were to boot pool/root/ubuntu, pool/root/fedora, having canmount=noauto will not be mounted by ZBM, but the OS will try to mount all filesystems regardless of the nesting levels, so pool/root/ubuntu/home and pool/root/fedora/home will clash.
Am I wrong?

11

No. It is not correct. I really can’t explain it any better than I already have. If you still don’t understand it, I think you’re going to need to set up an experimental system and just poke at it until you do.

12

Ahw, but that’s work!!! :slight_smile:
Ok, I’ll try on an old laptop.
Thank you

1 Like
13

Could also use a VM if that’s easier.

14

Just for reference: The standard Ubuntu installer, at least for 22.04 when choosing ZFS-on-root (and I suppose this is regardless if encrypting or not), has the home directories under rpool/USERDATA. Canonical did something to the adduser command, since new home directory datasets are created under USERDATA both when adding via the settings dialogs, and when using sudo adduser at the prompt. (Edit: I tested this today by creating two temporary dummy users - But I also have a VM running Ubuntu 24.04, installed as 22.04 but upgraded via do-release-upgrade, that did not add those ZFS datasets for new users, so perhaps that is a recent addition.)

And I agree that it would be safer to have unique home directories per OS that is being booted, with the other home directories mounted underneath if needed, but I don’t want to mess with the Ubuntu installer’s way-of-working, since I want as hassle-free an upgrade as I can.

15

Isn’t this just because the mountpoint of rpool/USERDATA is set to /home?

16

No, the mountpoint of rpool/USERDATA is actually set to /, but canmount=off.
I suppose the /home directory is a normal directory in the ZFS dataset mounted at / (and that is under rpool/ROOT/ubuntu_XXXXXX).