Ubuntu 22.04 LTS, KVM host on bare metal. Root install is on a drive in several pools:
bpool
rpool
hpool
kvmpool
These pools live on a drive as part of a two disc mirror vdev.
The snapshots are taken daily, and also replicated to another drive on a different host using cron.
I have everything backed up just fine, in multiple places.
I know I am about to upgrade to a more recent release and kernel. This next time I think I am going to put root on EXT4, and perhaps a software raid mirror for root. I will definitely leave my VMs on ZFS pools, with each VM in its own dataset.
However, I have the itch to break this ZFS root & see if I am able to successfully restore root from past snapshots. Perhaps roll it back a month and see if it works. Perhaps, this is the best way to break it in the first place,
I would think the datasets could all be replicated back from an older snapshot without any trouble. My guess is that boot will probably not work out of the box.
Has anyone tried to restore a root install; or have any suggestions?
It can certainly be done, but getting the boot environment to work can be a hitch in your giddyup.
Apart from that, the important thing is to realize that you need a bootable OS that can import your boot pool, list its snapshots, roll it back to the one you want, and then exit. You can’t use the OS you want to roll back to roll itself back–certainly not without causing massive issues that would require an immediate reboot, at absolute best (and in practice, I think you’ll typically simply find it impossible because you can’t unmount the root filesystem, and Busybox et al don’t generally have their own ZFS functionality either).
Does Ubuntu provide a good way to put your VM configs on a dataset? Then the root shouldn’t even matter - just install the newer LTS, import your/mount your datasets including the one with your configs, and you’re good to go!
I personally treat all my rootfs as ephemeral throwaway things. My datasets are pets, rootfs are cattle.
This is what I do. The XML files that live in /etc/libvirt/qemu are what define your VMs. Each of my VMs has its own dataset beneath data/images; by putting the dataset mounted at /etc/libvirt/qemu beneath data/images, I ensure that any backup (eg syncoid -r data/images root@target:data/images) will pick up the VM definitions also.
My solution to this is using ZFSBootMenu on a dedicated SSD. ZBM is just a tiny installation of Void Linux running ZFS, so you can snapshot and rollback from within your bootloader. (As long as you can import the pool of course.)
It works well for my needs. I haven’t tried restoring from remote snapshots yet, but I haven’t needed to. Testing that is on my list for January since I already have Syncoid sending snapshots to remote hosts. The ZBM recovery images have an SSH client and networking support. Theoretically this enables replicating an entire hierarchy of datasets back, pool by pool. (I’m going to be testing that soon. I’d be happy to report back about how it worked.)
I can relate to the desire to re-do ZFS-on-Root installations. I have a few systems I want to cleanup and standardize. At least with ZFS, it’s relatively easy if you can abide some host downtime.
My instructions for setting up ZBM are to first get a bootable ZBM thumb drive working, then install the ZBM bootloader onto the main system after ensuring that your ZBM thumb drive works.
After that, it’s same-same. You’re still just booting into the ZBM bootloader, grabbing a shell, and doing what you need to do in there (and, again, need to deal with stuff like chrooting into your real OS dataset before doing any necessary apt update / apt upgrade or similar tasks that need to update the root file system of the real OS).
That’s kind of what I do with my Ubuntu 22.04 host with Qemu/KVM VMs - But you also have a lot of stuff under /var/lib/libvirt (that I also put in a ZFS filesystem), for example all UEFI data (or perhaps I should say EFI variables, in the nvram directory) for each VM that uses it.
@Alfra Sorry if I steered the conversation in a unhelpful direction. Do any of the solutions from @mercenary_sysadmin or @JeffreyA solve your current restoration needs?
Thank you all !
Sorry for the delay. I was able to work on it today. I rolled back by root & boot until November (I had been procrastinating). That brought me back to a working system without the hitch that was causing some problem.
I did it manually since I do not have ZBM. Perhaps I will try that next time.
After chrooting in I was able to run a: zfs rollback -r on all of the datasets. I had to run a zfs list to find the snapshot in each dataset that I needed.
After all that, I shutdown and restarted and it all worked.
EDIT: and I took good notes in case I need to do this in the future
