NAS OS Considerations

OpenZFS
1

I’m putting together a pair of NAS machines and could use some guidance. Opinions and anecdotes are welcome. If I’ve missed any important considerations, my pros/cons are inaccurate, or if anyone has opinions on which route I should take any and all input is very much appreciated. Please let me know if I’ve left out any pertinent information, thanks!

Goals

  • One machine offsite with family, serving double duty as an offsite backup for me and a local storage target for them.

  • The other will run in my home as a general storage and backup box.

  • I’d like to configure both servers identically so that in future if I want to change something about the setup I can test locally before deploying the change to the remote box.

  • Each host will use zfs for data storage and a separate drive for the host os

  • The two hosts will be connected via VPN for regular replication, probably WireGuard or Tailscale

At this point I’ve landed on four options for base OS:

Options

TrueNAS:

Pros:

  • “Just works” NAS functionality, easy to set up basic shares and basic disk maintenance tasks.

  • Improved discoverability. TN provides all the knobs and buttons I might need to tweak.

Cons:

  • Difficult to automate configuration / easy for me to miss things when replicating config from one box to another

  • Any learning is specific to how TrueNAS does things, likley not very useful outside this context.

  • In the brief, rushed, single experience I had trying to get two TrueNAS boxes to replicate between each other, getting the connection established was a bit of a pain.

  • Package availability seems limited. I would need some research to figure out how to install Tailscale and Sanoid.

NixOS:

config built/tested locally and the SCP’d to the remote

Pros:

  • Easy to replicate configuration from one machine to the next

Cons:

  • I don’t actually understand what’s going on under the hood. Could be a pro for learning, but things like updating installed packages is something I’d have to look up.

  • Software availability is a bit of a question. Many packages have been built for NixOS, but I won’t be surprised if something I want hasn’t.

Ubuntu:

configured via Ansible

Pros:

  • Easy to replicate configuration between machines and back up that config in case OS drive fails.

  • Mainstream OS with mainstream support / community, easily Googleable if I run into problems.

  • I think of this as a “regular” OS, so anything I have to learn to accomplish this goal on Ubuntu would be a much more transferable skill than TrueNAS or NixOS.

  • I cannot imagine a package being available for the other options and not also Ubuntu

Cons:

  • Discoverability, from the OS, is nil. It’s entirely on me to learn important details of configurations that matter to me.

Proxmox:

Store data on the host’s pool and provide it as a virtual drive to a guest Vm which makes shares available, e.g. Ubuntu or NixOS detailed above

Pros:

  • Can use any OS I like for the NAS functionality, and change that decision much more easily than if it were installed on bare metal. Especially helpful

  • Config backup and restore would be trivially simple.

  • Because of ^, testing different configurations is much easier

Cons:

  • One more layer of complicaiton/abstraction. Possibility for unexpected funkyness as a result.
2

There are far more experienced people on this forum than I. But following is my experience and opinion.

TruNAS: No personal experience. Personally, I have little interest in the fancy UI and clicky things, nor the potential performance implications related to that. I know some people (online) would wager their own mother that it had no performance impacts and other people would wager their first born child that it does.

NixOS: Minimal personal experience, only as a desktop install. We don’t get on very well. Particularly installing apps and installing updates. I don’t get the hype, personally.

Ubuntu: Been using this as a server OS for work and as a personal NAS OS at home since 2010. Easy to setup and replicate, highly performant, very reliable, basically any package is available, no excess bloat installed. Also very easy to get help if or when you run into issues.

Edit: I should note that I do run Ubuntu with Qemu/KVM and quite a few VMs. Half a dozen are in production and I have 3 or 4 various experiments going. When I setup a quick experimental VM, I often use virtual machine manager. For production, I default to CLI because that’s how I first learned how to setup VMs.

Proxmox: I only started playing around with it around 3 or 4 years ago. My general feeling about it is, meh. For me, it’s far too bloated. Additionally, I didn’t like (or need) the UI. Anecdotally (I have no concrete data to back this up), every time I tested it, I’d always come up against something that was not intuitive. Not to mention, it always feels slow.

All that to say, I use Ubuntu and have for quite awhile. I’ve used Fedora a few times, but always end up coming back to Ubuntu. I don’t agree with every decision that Canonical makes as a business, but because it’s open source, I can easily get around it. I also prefer using the CLI and my habits are keyboard driven versus mouse driven, thus adding to my lack of interest in a UI for this sort of thing.

3

My experience:

TrueNAS:
I’ve used TrueNAS a bit in the past. It’s pretty nice having a web interface to configure things, and they’ve done a good job on allowing you to configure all the important things through the web interface. If you just want to provide storage, this is a good option.

NixOS
I have no experience with NixOS.

Ubuntu
I ran vanilla Ubuntu server for a long time on my main NAS. It’s a very good learning experience as you have to learn how to configure all the services the way you want using the config files. You can also run VM’s using libvirt, and can use Virtual Machine Manager from another computer to get a nice GUI for configuring them. A lot of manual configs though.

Proxmox
This is what I currently use. I really like having the web GUI for configuring VM’s, containers, and backups and the like. It’s similar to Ubuntu as it’s Debian-based, but VM management is easier. You can either provide file shares from the host, or from a VM/LXC Container to hide that config away, then map storage into the VM/Container.

Overall you’ve summed up the Pros/Cons pretty well yourself! :smiley:

4

If your goal is to use a configuration management tool like Ansible to manage these servers, I’d recommend a more general purpose OS like Ubuntu over something like TrueNAS.

Most of what you get with TrueNAS is the fancy GUI, and some extra bits they’ve tacked on to make SMB work more like it does with a Windows file server. (Handling setting permissions from windows, etc, getting shadow copy to work properly, etc.)

Personally, I’d say that as long as your comfortable with the command line - go with Ubuntu.

Set up ZFS - you can even do ZFS boot menu to get it on the root, if you’re feeling froggy - and automated snapshot and replication tools like Sanoid / Syncoid / Zrepl to do all of your backups.

I would recommend at least 2 root datasets, one for your local data, and one for the remote backup.

1 Like
5

I’d recommend at least considering XigmaNAS.

6

For me, TrueNAS is the preferred option because of the appliance approach and the webGUI. I like my NAS as just a NAS, nothing else. I also like a webUI. I do not work in tech by trade and while I’m pretty familiar with using the CLI to tinker, I prefer a GUI when configuring a system such as a NAS. This is simply what works best for me.

7

I ran Ubuntu for 10+ years and it was pretty great. But, I kept running into issues with snap which is now the only way to get lxc. I rebuilt with Proxmox and couldn’t be happier. It doesn’t seem slow at all. It did increase my memory use by about 1GB. I then run services inside of lxc and manage them with ansible. Nearly all of them are alpine with docker compose containers inside. That way I get logical groups of services that I can easily manage in the UI.

Zfs is managed manually from the host and works wonderfully.

The best part of proxmox is their bootloader, proxmox-boot-tool. It’s way simpler than grub for booting zfs and is rock solid.