System Architecture, Home Lab Reengineering, and Other offtopic thoughts

Offtopic Chat
1

Hello,

This is my first post here, I’m a former system and security admin, who is now in management, but continues to sysadmin at home and run Linux and BSD. I’ve used BSD and various forms of Linux since the mid 90s. My installations are now… aged and I’m in my once a decade time to reconsider everything research. I’ve been attempting to over-engineer this and have ended up with the following questions that are mostly about getting reliable, repeatable, installs of systems that support ZFS that I can generally ignore for another decade (sans regular upgrades, but not tinkering with the general setup). For my rearchitecture, I’m working on the following systems: a few laptops, a desktop (NAS/Gaming rig/Oversubscribed), DIY router, and a Raspberry Pi.

Now, the too many (mostly offtopic for ZFS) questions I’ve wrapped myself around the axle with:

  • Distro preference? (Ubuntu, Arch, EndeavorOS (dubious ZFS support), MXLinux (same)?)
  • Will it run Steam?

  • Are snaps really that bad?

  • Do you also hate systemd? Or learn to love it?

  • Is it stable and secure? (Am I overvaluing Canonical's support vs. community support?)
  • SecureBoot, is it worth the pain? Or does everyone just turn SecureBoot off? Everything works with it off, but I feel like I’m stripping out critical security. There’s dubious documentation to getting SecureBoot working with ZFS reliably. I’ve got shiny new hardware and have a security background and have spent way too much time trying to make it work…
  • Related, preferences for Refind vs. UEFI?
  • Swap file on disk vs. swap partition? Encrypted or not?
  • I've seen and noticed that without a swap partition, that performance sucks, even with lots of physical RAM. Thoughts on how best to address this?
  • ZFS Boot Menu (Seems preferably vs. default ubuntu installs, yes?)
  • Preference for Sithuk script vs. manual installs

  • Setting changes to use the downloaded ISO files vs. network download (I've got the live install right there... any way to bypass the debootstrap to the Internet) 

  • Repeatable installs - I'm attempting to build and rebuild to regain familiarity with newer means of doing things until I have a "clean" install. I've limped old systems along with installation faults and too many upgrades because I didn't have an easy means of rebuilding without losing all of my random customizations.

  • ZBM isn't signed by default, right? Which may loop back to my SecureBoot question.
  • Different encrypted datasets for data types?
  • Personal documents (financial, small file size, etc.)

  • Photos & Videos (from phones/cameras)

  • Plex content: DVDs, Bluray, Music
  • Encrypted dataset questions:
  • Do you do encrypt at the pool, dataset, or nested dataset? 

  •     Depending on which you do, where do you do your establish your raw send for backup? 

  • Can you do nested dataset encryption? (or is this inadvisable, and where do you store the different keys?)
  • Seperate Pools for NVMe and Rust?
  • I've heard/read Jim's comments about NVMe and regret my storage purchases but am past return windows... but I had these spare NVMe PCIe channels available. :) 

  • Do you allocate spare devices to your pools / mirrors?

I would appreciate any thoughts, opinions, random rants that you have available on any of these topics. It might help me stop rebuilding my systems and settle on a stable configuration and get on with things instead of engineering it to death.

Thank you!

2

You’ll probably get more engagement if you split these questions up into small, related groups and open separate threads for each small group of questions. As is, this is a bit overwhelming. :slight_smile:

3

Ah, I wondered. :smiley:

I saw that I’d missed a thread about selecting your very favorite ZFS distro.

Thanks.

4

About a year ago I switched all of my remote servers from Ubuntu to Opensuse’s MicroOS - it’s been completely hands off so far :

  • it’s a rolling release so I never have to reinstall
  • I can build custom iso’s with Kiwi which install themselves & reboot
  • I run services in podman which can be easily configured to auto update images & run as quadlet services (with the podlet tool)

You can run zfs in Opensuse - but cannot enforce signed kernel modules as zfs is not from an official repo.

Ubuntu would be my first choice for zfs if you need signed modules & official support. Podman runs ok in Ubuntu.

MicroOS is a good choice for a server that “just works” once it’s configured (it auto updates itself by default - & is also immutable) - great for security. You also get RKE2 / Longhorn if you want to play around with Kubernetes (although I’m also looking at Talos Linux for this on my 2nd cluster)

For my workstation I’ve run Arch Linux on btrfs for 10 years & use podman for some services (custom primary dns / custom mail that does mx lookups of my non public private domain - basically 4 x remote relays - to give HA DNS & Mail to my servers acting as secondary DNS) + a zfs mirror for data. Knot DNS & Postfix in podman runs very well. I run my private DNS / Mail over netbird (wireguard that also “just works”) - my workstation is fully encrypted with LUKS (system) + encrypted zfs.

For gaming I do hardware passthrough of a nvidia gpu / 2nd nvme / intel I350 in kvm & have not tinkered with proton so far.

1 Like