Hi everyone, I know that this questions returns over and over again, but still - what are the latest best practice considering pool size for two vdevs each with 3x 24TB drive in RAIZ1 ? Small storage server in small company running openzfs on debian linux.
Currently running mirrored pool with 4x two way mirrors = performance is great, overall experience with resilvering is nothing but superb.
Is that additional 16% storage efficiency with raidz1 meaningless with such a large drives considering problems with resilvering and slight degradation in general performance?
And is two way mirror best way to go forward with such a large drives?
I would not consider raidz5 for large scale up costs to buy at least 5 new drives in bulk when expanding the pool.
Pool is used for general storage of rather large files, regularly backed up to LTO tapes.
If you’re happy enough with the performance of your four 2-way mirrors to call it “superb,” and you’re willing to downgrade to merely “quite good,” a pair of 3-wide Z1 vdevs will probably suit you pretty well.
There isn’t a whole lot of difference between the performance of a single 3-wide Z1 and a single 2-wide mirror. The mirror will outperform the 3-wide Z1, but not by a whole lot. Resilvering will similarly be “a bit slower” but not, you know, catastrophic.
Assuming you’re talking of replacing 4 2-wide mirrors with 2 3-wide Z1 of the same type of disk, you will have two vdevs instead of three (up to 33% hit in throughput) and each vdev will be slightly slower than the mirror vdevs were. Realistically, this means you’ll take a performance hit somewhere between 15%-25% on most common workloads, with some outliers that either don’t appear affected at all (because they bottleneck somewhere else) or are significantly more impacted (a relatively small increase in latency may be multiplied by many sequential operations necessary, eg database workloads with complex queries).
For the most part, like I said… if you were happy with a pool of mirrors, you’ll probably be happy with a pool of 3-wide Z1. It will feel like you traded some performance away, but IMO it will feel like “an honest trade” for the extra space, if you follow my drift.
Where Z1 really falls over is in wide arrays or vdevs. I frankly wouldn’t even go four wide with RAID5/Z1–but three wide is a pretty solid compromise for “I want similar performance to RAID10, but I want to trade a little of the performance for noticeably higher storage efficiency.”
To me it is now easy to understand the tradeoff with these setups (3 raidz1 vs 2way mirror).
When searching around this topic there is either very old information favouring raid5 setups but recently very often one can find horrific stories about some probability of read errors during resilvering or that resilvering on that big drives is something to just dream about, to straight not recommending multiple 3xraidz1 vdevs..
So for lighter tasks, sequential reads etc it’s good to hear it this option can be still valid when storage efficient is important (especially with today’s elevated hdd prices - at least in Europe).