Recreating a natively-encrypted zpool (and encrypted zpool organization in general)?

OpenZFS
1

Hi all, first-time Discourse user, and about five years of ZFS on Linux user here.

I wonder if there is a way to recreate a ZFS-natively encrypted zpool from a backup, using exactly the same encryption IVs (initialization vectors) as before?

For example, if a primary zpool (encrypted, with the root filesystem encryptionroot for all children) fails due to broken hardware, but snapshots have been replicated to a backup storage, the original zpool can be recreated on a fresh drive and data can be restored - But one cannot use raw sends from backup to get exactly the same state as before, since the root is now encrypted differently. (This is solvable if letting all second-level filesystems become their own encryptionroots.)

If I were to create a new encrypted pool, now that I have more ZFS experience, I would create an unencrypted zpool with canmount=off/mountpoint=none on the root filesystem, then create an encrypted child dataset with canmount=on and mountpoint=/, and place all other datasets under that one. And never bother with snapshoting or backing up the “formal” top-level zfs filesystem.

However, that is not possible if (for example) installing Ubuntu with encrypted ZFS-on-root, since the rpool root filesystem is encrypted. Recreating that kind of “rpool” means re-encrypting everything (not using raw sends) if restoring from a backup.

Have I misunderstood something in this description/discussion?

(To clarify: I could use the same encryption key on the new/recreated zpool as the original one, but that is not the same as storing the same ciphertext on disk. Also, using non-raw sends when restoring from backup means that the encryption key must be loaded on the backup zpool first, which negates a major security advantage: Namely, if the backup snapshots are stored on an untrusted system.)

2

While I am a ZFS user on Linux, I do not yet use ZFS-on-root, ZFS snapshots, or ZFS send. I am sorry, but I cannot directly answer your questions.

Let me offer an analogy, along with my solution.

My systems are built with the Linux Mint installer (inherited from Ubuntu’s Ubiquity I think), using LVM and dmcrypt-LUKS encryption. The only modification I do to that part is to shrink the LVM volume for root, increase the size of the volume for swap, and leave a good chunk of free space available for future use or SSD trim. I perform file-level backups using Duplicity and create partition images of the non-LVM partitions. My backup script works very well. I have tested restores successfully, and been forced to really perform restores when there was a regression in SystemD that broke the display manager.

What does this have to do with ZFS Native Encryption, ZFS-on-root, snapshots and backups, you are probably asking.

I had to solve the same problem with initialization vectors that you described; however, I am neither a cryptographer, nor did I care to go down the path of doing all that manually for a restore, which for most people will be the most stressful task they will attempt. Also, I share my installation process/checklist with family members, and I did not want to over-complicate things.

My solution was one-time disk imaging with DD.

Basic one-time steps:

  1. Install the OS with LVM and dmcrypt-LUKS.
  2. Use DD to image the non-LVM partitions and Duplicity to create a backup of the contents of root, perform a backup.
  3. Use DD to create an image of the entire disk.

Step three is key to maintaining the cryptography. Yes, DD’ing the entire disk is time consuming (approximately one hour for a 1 TB SSD with compression across the network), but this is a one-time operation. All those zeroed/trimmed blocks compress to next to nothing, but even that takes time. The DD destination needs approximately 35 GB free for that 1 TB drive image.

To restore from a trashed OS, restore the full-disk image to the SSD, then restore using Duplicity, and finally overwrite with the latest partition images.

How does this help you, you are probably asking.

Perhaps you could do something similar with ZFS-on-root with Native Encryption. Install the OS with ZFS-on-root and Native Encryption, and then take a DD image.

I mount an SSHFS share on TrueNAS for the destination, and I create the DD image from the Linux Mint live environment (booted from UFD/ISO/DVD) compressing and encrypting with GPG:

sudo sshfs <UserName>@<ServerName>.<Domain>:/mnt/tank/backup /media/backup
sudo lsblk -a -fs
time sudo nice -n 19 ionice -c 3 bash -c 'export GPG_TTY=$(tty) && dd if=/dev/sdX bs=4096 conv=sparse status=progress | gpg --symmetric >/media/backup/Temp/<hostname>_00.img_-_<date>_<time>.gpg'

Hope that helps!

3

That’s a good idea, to take a complete raw image of the relatively empty zpool. I’ll do that the next time I create a zpool from scratch whose root is encrypted (my current ones are too full for it to be meaningful).

Having a raw disk image (as a compressed, sparse file) of an empty pool should make it very easy to restore it to a disk or partition of the same size as the original.

Speaking of “same size”: Will ZFS mount a zpool based on a raw dump of a partition created as you wrote above (i.e., the dd command)? I know that zpools can be created using normal files as the backing storage, even if that is mainly for testing purposes, but are there some differences in the zpool data if it’s dumped raw from a disk partition or created from a file?