Okay, great. With that out of the way, I’d recommend either six wide or ten wide Z2 instead of eight wide, depending on whether you value maximum storage efficiency for large files (and no database workloads!) or a mix of storage efficiency and better performance.
RAIDz spreads each block among all data members of each stripe. So, say you’re saving a 1MiB block to an eight wide Z2, you will end up with 1MiB/6 written to each disk. That doesn’t divide evenly, so you end up needing padding, which both reduces performance and storage efficiency from what you’d naively expect.
This isn’t the end of the world, and if you’re building a bulk file server with no DB workload on a simple 1GiB LAN, the performance issues will almost certainly be hidden behind the network bottleneck. BUT while it isn’t the end of the world, it DOES make a difference, so since you’re re-architecting in the first place, I’d suggest taking this into account.
Also note that with six or ten wide Z2, you really should be using a large recordsize. 1M or larger for six wide, 4M for ten wide. The other part of splitting each block between all data members of a stripe (unlike conventional raid, which writes to each disk in fixed chunk sizes, 512KiB by default on eg mdraid) is that if those individual writes per drive are too small, each drive individually will perform poorly.
With the default 128KiB recordsize and a ten wide Z2, you’d only be writing 128/8==16KiB per drive, which means most of your I/O will be 16K random–a punishing workload for rust and SSDs alike. With recordsize=4M, you’d instead be writing 4,096KiB/8==512KiB per drive, with massively better performance.
Remember when I said “assuming no database workloads” up above? If you run MySQL InnoDB on that same ten wide Z2, it doesn’t matter what recordsize you set, because InnoDB writes in 16KiB pages–each of which will be stored 4KiB per drive on only four data and two parity drives per stripe. This leaves you with 4KiB random I/O on each drive (the most punishing workload possible), much more space per write burned on parity than you would naively expect (67% storage efficiency instead of 80%), AND nearly half of your drives idle during any given operation, because the four drives not involved in the current write aren’t enough to issue a second read or write in parallel, even if you have something in the queue.
If you need to do much in the way of database or VM workloads, I STRONGLY recommend narrower vdevs for performance reasons–4-wide Z2, 3-wide Z1, or mirrors.
You’re also going to have enough drives in this pool that you might want to consider having a spare or two, to cut down on the vulnerability window after a drive failure occurs. I would ESPECIALLY recommend allocating a spare or two if you’re planning to do 3-wide Z1 or 2-wide mirrors.
