Migrating a root EXT4 file system to OpenZFS - looking for people who have done it

OpenZFS
1

Edit: original post seemed disjoined…so I redid it.

I’ve been thinking about migrating my root file system, currently on my Debian 12 desktop/server, over to OpenZFS with zfsbootmenu. Snapshots and rollbacks would be incredibly cool and useful. I’ve already been running OpenZFS on my big storage pool for a while, but not on root. Looking back, if I’d been more forward thinking, I would have set this up during the initial Debian 12 install years ago, but alas, I didn’t.

Now, I’m considering how to migrate a root file system (and boot) without completely messing things up. Has anyone here done this and not hated themselves for the hassle? My initial research suggests it’s a pretty risky and heavy lift. I wouldn’t even dream of doing an in place conversion. That’s just crazy talk in my mind. Instead, I’d buy another SSD, create the root and boot pools there, and then attempt to copy everything over.

I’m also wondering if it’s worth the effort. I know that’s subjective. EXT4 isn’t a bad file system, but I can’t create a mirrored pool with it, which is what I’d really like for my root/boot SSDs. My plan is to migrate to a single pool root/boot SSD first, then buy another one to add for mirroring.

Thanks for any experiences or insights you might share if you’ve attempted this!

1 Like
2

It’s very difficult and I generally wouldn’t recommend it. You really need to be able to recreate your basic operating system environment on demand anyway, so consider this an exercise of your basic backup and recovery skills! :slight_smile:

Even once you’ve got a zfs root that you’re happy with and you can start replicating that, I’d strongly advise keeping your basic OS installation and manual recreation of the local environment skills sharp. I’ve had one or two zfsbootmenu systems suddenly refuse to boot, and ended up finding it easier to just reinstall the OS than successfully repair the boot environment… And I did try to rescue the boot environment, to be clear.

1 Like
3

It’s doable. ZFSBootMenu’s guides for Debian and Fedora assume you’re copying files from a live image, but you can substitute it with an existing working installation.

I’ve done it twice and have one home server running Fedora Server and a laptop running a desktop Fedora. I regret not making notes of that process, because there are always issues once you stray from ZBM’s guides. I think Fedora Server went relatively smooth, but I couldn’t get Fedora’s Sway spin to install directly from either live image or working installation. I’ve probably ended up installing Workstation, moving it to ZFS root and then installing Sway separately. And all of that booted from Workstation live image.

I’d say go ahead, make backup of your system, plug a second drive, boot from live image and try to follow ZBM’s guide except from mounting and copying your existing system.

4

When I switched my Debian install from ext4 to zfs, I installed the necessary packages for initramfs to boot from zfs. I used a live CD with ZFS baked-in to setup the pool and rsync’d the existing install to a new root dataset. I might have had to chroot into the dataset to update fstab and rebuild the initfamfs. The SSD has a small EFI partition, and a partition for zfs. I just use plain old Grub. I don’t have any experience with ZBM, but it sounds like it’s finicky from what I’ve read.

I’m sure this isn’t the recommended way, but it was straight forward for me, and I’ve had no issues since.

5

Thanks Jim. That was the other option I’m considering, a full re-install with re-installation of apps, vms, flatpaks, home directories, containers, etc. I’ve not taken the time to document everything I’ve done to the system like I would at work, so it would mean “rediscovering” what I’ve done to the system.

I certainly need to be able to do that, but as you know..it’s a PAIN :wink:

1 Like
6

This is the how and the why of tech debt accumulation: because the pain now tends to seem more important than the pain in the foreseeable future. “Future me? That’s somebody else, and they can sort their own problems out.” :slight_smile:

7

I have a fairly opinionated script to build a root-on-zfs system based on Ubuntu, rEFInd, zfsbootmenu, Dracut. I use it extensively for all my own systems, of various flavors … laptop, workstation, general server, media box etc.

It’s easy to test, works fine under virtualbox. Also has a packer setup so you can build a qcow2 image that can be booted under many different virtualization platforms for messing around with.

I have the process for integrating UEFI SecureBoot working now, by signing both the rEFInd binary and the zfsbootmenu unified kernel/initramfs. Just needs to be made a bit more robust and I’ll push the commits in a week or so (if that’s of any interest).

Listen to Jim, he knows all. I am constantly begrudging past-halfwalker for being a jerk and not doing enough for present-me. Then again, present-me, although I try, I sometimes fail to set things up better for future-halfwalker. And I know he’ll think me a jerk …

1 Like
8

Every night, I get my coffee ready for the next morning, and most nights, I literally tell myself that I’m doing a kind thing for tomorrow-me. And in the morning, I silently thank last-night-me. It’s weird, maybe, but I’ve found that I’m more motivated to do a kindness for someone “else” who will appreciate it than I am to just “plan for tomorrow” for myself.

Without the little ritual, I’m a LOT more likely to just be like “ah I’m too tired to bother” and then have to face the next morning with no coffee!

9

Not quite off topic - if it’s raid on boot/os you’d like, mdraid works just fine… No snapshots or roll backs, but easy to implement and well supported

1 Like
10

That’s usually the direction I go. 100GiB partition on the first two drives is an mdraid1 root, rest of the drives are the first mirror vdev of the pool.

11

Long time lurker here. I thought I chime in because I just did that transition for my main server in the home lab.

Background: the server (running Debian) has been serving zfs datasets for 10 years now (mainboards and pool drives replaced at some time). No problems ever. A drive in one of the pools (raidz2 at the time) died, was replaced, no sweat. OS had always been on ext4 (Samsung SATA ssd).

Then, a couple of weeks ago, in a separate machine, a Samsung nvme ssd died out of the blue. It was one drive of two in a mirror. That scared me enough to replace the Samsung nvmes with Intel drives. In the meantime, smartctl showed 90% health remaining on the boot drive of the main server. I couldn’t tell how many hours remaining life (or MBs written) this would translate to but it provided the necessary kick to change the boot drive(s).

For learning, I used a spare machine to install a minimum debian system on a mirror. The instructions on zfsbootmenu are good, it really helped me to go through them a couple of times (first with a single drive zfs boot), then with a mirror. It also helped me to read up on efibootmgr. This here helped:

I suppose you can do some of the learning with VMs, but for me there was quite a bit of hassle getting some of my boards (Supermicro X10) to boot uefi, so practicing on real hardware proved helpful (though painful).

For the actual migration, on the main server, I saved the list of installed packages:

sudo dpkg --get-selections > pkg_list

I also copied all config files to the data zfs pool to have them available:

  • etc/network/interfaces

  • etc/crontab

  • etc/exports

  • etc/samba/smb.conf

  • all ssh keys etc.

  • exported VM definitions

  • and other configs you might use

on the new machine, I prepared for the package installation like this:

avail=`mktemp`
sudo apt-cache dumpavail > "$avail"
sudo dpkg --merge-avail "$avail"
rm -f "$avail"

Then updated the dpkg selections:

sudo dpkg --set-selections < pkg_list

and ask to install the selected packages:

sudo apt-get dselect-upgrade

Note: you’ll likely have to manually curate the list. In my case there were some old linux headers and other packages present in the previous install that weren’t available or needed anymore.

After this, you have a system with all packages from the previous system installed.

I created a usb stick for zfs boot menu. Powered off the server, removed the previous boot drive (stuck it into a how-swap usb enclosure to mount on /tmp if needed), installed the new mirror ssds (setup on the test machine), booted the machine using the zbm usb stick, from there into the os on the new mirror; then ran efibootmgr to install efi boot on the new mirror, ran efibootmgr again to set correct boot order, rebooted and was pleasantly surprised that it actually booted.

From there, all I had to do was copy the config files for the various services. All in all it took me less than an hour to get the server back up running including NFS, Samba and a few VMs.

In summary, it’s a lot of work, definitely takes some practice to wrap your head around the boot manager.

For the two mirror drives I use, I actually installed a boot partition with zfs boot menu on each of the two mirror drives for added safety. The previous experience of a failed drive proved enough incentive to have a mirror boot dataset, for which I can hopefully replace a single drive before catastrophic failure. Right now I’m doing test runs with sanoid and syncoid the backup the snapshots of the main server to a backup server.

Sorry for the super long post, hoping this proves helpful

1 Like
12

I used to back up systems exactly this way, back in the day when I ran real workloads on bare metal! (I know, my “restore” command is slightly different than yours, but it does the same thing.)

These days, I almost never need to actually use that technique, because my real workloads are on VMs, replicated automatically to multiple backup systems, and therefore are essentially immortal. And I keep the hosts simple enough that I just don’t care about recreating them: all I need to reinstall manually is ZFS + the KVM stack + WireGuard + nagios-nrpe-server for monitoring, and I’m good to go.