Now, after my little adventure described here, I ended up with four SSDs of different brands but the same size in a Terramaster D4 DAS.
So, now I’m not sure how to make better use of them.
My goal is to maximize size for data, so there won’t be mirrors, maybe RAID 0 or RAID 5.
The DAS will be accessed by three different machines, though not at the same time. All machines have Linux and FreeBSD. Two of those have Thunderbolt 4 capability, so the DAS will show them as /dev/nvme devices. One machine can’t do Thunderbolt, so the DAS will show them (through USB 3.2) as /dev/sd devices.
Also, the SSDs have different sector sizes, two 512e and two 4Kn.
Is it worth creating a zfs raidz1 pool or 4 separate single disk pools?
Or simply use another filesystem?
Also, if I go with a raidz1 pool, what will be the size for data: 4 x SSD size or 3 x SSD size (1 SSD worth of capacity used for parity)?
ZFS doesn’t present any more challenge for your mismatched drives than any other RAID system would.
By your description, what you want is a four wide Z1 with ashift=12. Your 512b devices won’t mind being addressed 4K at a time, whereas trying to address your 4kn devices 512B at a time would produce MASSIVE amplification.
A 4-wide Z1, much like conventional RAID5, will offer you roughly 3n usable capacity.
---- (op-ed postscript) ----
I don’t want to discourage you, exactly, but I will note that NVMe plus small striped array is a sort of self conflicted setup. NVMe is generally more expensive, and definitely much harder to deal with at scale–but using striped RAID (any kind, not just ZFS) negates much of that speed advantage vs easier to work with SATA SSDs that otherwise made the NVMe look attractive in the first place.
But if you’re fine with that, I’m fine with you being fine with that, if you follow me.
Storage vendors choose a single arbitrary workload that makes the biggest possible number on the particular thing they’re trying to sell you, then declare that is why they’re “the fastest.”
Striped RAID can be faster than a single drive when it’s being treated like a tape drive: one utterly uninterrupted long contiguous read or write operation, with neither random access nor competing I/O processes.
When you try to use it in the real world in the way people actually use random access multi process (let alone multi user!) storage, though, what you see is performance that frequently falls WELL below that of a single drive.
You might find this helpful, if this is the first time you’re hearing this (note: this is still me, just me as published at Ars Technica):
