I’d really like to try OpenZFS, but I’d like to try it without spending too much money.
I’ve thought about using a Raspberry Pi 4, but building something with SATA (or really building with a Pi at all) feels like it’ll be too hacky. While I think a Pi 5 with some add-ons could work well, I don’t really want to invest in this path and I don’t own a Pi 5.
Hoping to build a simple experiment that works decently, and enough to learn, I’m considering using USB for storage. My understanding is that USB 3.1 Gen 2 offers speeds up to 10Gbps.
So, is this a good idea, bad idea, or terrible idea?
It has an enterprise HDD in an enclosure for mass storage (4 TB), built-in SSD (250 GB) for my container data files / documents (plus a small ext4 partition for system files) and another external SSD (120 GB) for less used docker containers. This helps stream by rebuild risk should a drive fail. That way, I can spend more pennies on my backup server drives.
However, I hear creating mirrors or raids from USB drives causes problems. I am running each drive as a single vdev. Then, once or twice a week, I replicate datasets to a backup server with a raidz1 for security.
So far no problems with ZFS running on a USB drive.
I run a scrub on the SSDs every night, which only takes 5-7 minutes. They are 60% full. And for just 20 mins a night, I run a scrub on the mass storage then after 20 minutes I pause the scrub (done via cron script), so I have a continuous scrub going to spot any errors quickly; the scrub takes about 4 days to complete.
Notes:
This server is on a backup battery
ZFS datasets are encrypted
My setup is a small laptop with two WD Passport HDDs. Small footprint, USB 3.0, low power and built in battery. About six months ago I upgraded to a mirrored setup with two more WD Passport HDDs. Unfortunately I need to use the dock for additional USB ports, which is essentially a powered USB hub. In the six months I’ve been running the mirrored setup, I’ve lost power once and the mirror dropped off of the pool. Once power came back, ZFS resilvered the mirror. (Probably just metadata since it only resilvered less than 2MB) No issues so far. I do keep several separate backups just in case though.
Using USB drives as 24/7/365 connected storage is a bad idea, with or without ZFS. If I had a dollar for ever time I heard somebody’s anguished tale of woe because “my USB connected drives worked fine for a whole year, but then I lost the pool one day” I could stop asking y’all for patreon subs, cause the server would be paid for!
RE: ZFS on Pi - I have a PiBox, which is a small unit that takes two SATA SSDs and makes use of a RaspberryPi compute module. With RaspberryPi OS, or Debian, this runs ZFS great and has been very reliable. In the process of moving I spent 1/2 my time in hotels for several months and it was great to have a little device that I could throw in my bag that would allow me a very small ‘hotellab’ if you will. Bit of media storage, DNS, Tailscale, etc.,
All that to say, there are some viable Pi based ZFS options if that ends up being a route you want to travel.
I’ve been using zepl for about 5 years to backup an encrypted zfs mirror to a WD Element external USB without any issues.
I configured the pool on usb to store data with copies=2 as it’s not mirrored (so testing with one disk is an option with some redundancy)
The WD Element drives can also be “shucked” with the first 2-3 pins covered with heat resistant tape (see DataHoarder on Reddit - “rayon” tape from memory ?) - the 8tb drives are white label helium drives (pretty much silent) - if you will be using these drives for a mirror - just space the purchases a week apart to ensure you get disks from different manufacturing batches. This is quite a bit cheaper than buying new disks.
If putting your own disks in an enclosure - you cannot go wrong with Toshiba Enterprise hard disks - buying these second hand is fine.
The only downside to usb external drives is making smart checks work can be a bit of trial & error - this however works:
# /etc/smartd.conf
# USB external
/dev/by-id/usb-WD_Elements_25A3_37484B3337485548-0:0 -d removable -o on -S on -n standby,q -s (S/../.././02|L/../../6/03) -W 4,40,44 -m me@email.com -M exec /path/to/smartdnotify
