I’ve been running mini-pcs with ZFS on USB drives for about 8 or 9 years now (but I was using ZFS before that for several years, so I wasn’t new to the concepts or to configuration). There are a few pitfalls, but in general it is a good (cheap-ish) way to create a reliable, solid filesystem with easy, over-the-net backups (zfs send/receive) built in.
If you’re starting from scratch, I’d recommend that you go for simple mirrors (rather than raidz). Before you even start with real disks, you can easily use files on an existing machine to create different configurations and get comfortable with the ZFS tools (use ‘truncate’ to create files on an existing filesystem and then simply use zpool/zfs to create pools and zfs-filesystems using those files instead of actual disks). Get comfortable with the command set (“zpool list -v” is your best friend and you almost always want to use “attach” and “detach”, -not- “add” and “remove”).
Always use USB-3 ports, not USB-2.
Don’t assume that cables or built-in controllers on new USB external drives are good (cables don’t seem to be tested at all by many manufacturers and the USB<->SATA bridge controllers in external drives have a nasty habit of failing after just a couple of months of operation). Label your external drives, so you know which is which and keep the labels updated. Always leave space for UEFI boot on every physical drive, so that you’ve got multiple alternative boot devices.
Don’t use USB hubs if you can possibly avoid it (although a good USB-3 hub is better than resorting to a USB-2 port).
Drives - Don’t use SMR drives in any ZFS array (they may work for a while in normal operation, but a resilver will probably end up killing the disk and possibly the whole array …SMR is the kiss of death).
Some USB disks just want to go into sleep mode, no matter what your system configuration. As a last resort, create a file somewhere on the offending disk which is overwritten (changed, not just touched) by cron every couple of minutes (make sure that it is excluded from snapshots and backups).
Do run “zpool list -v” regularly and mail yourself the output (FreeBSD’s daily status emails are great for this). The same applies to “dmesg”.
Do find out how to use “apply” to add ordinary user permissions to ZFS filesystems so that you can “push” a filesystem backup from a remote server without having to log in as root (this is send/receive again …use syncoid to make this whole process easier).
Do make frequent back-ups of your ZFS filesystem to another system (as has been said many, many times, RAID is not a back-up). I still use a Z8350 Atom (the cheapest, smallest X86 machine with a USB-3 port) to back-up some ZFS filesystems to a single, external USB disk (your back-up target doesn’t have to be another mirror).
Don’t despair! The learning curve for ZFS is fairly steep, but ZFS on USB drives doesn’t actually add to that.
