I recently acquired an old Dell PowerEdge R790XD server with 16 3.5in bays (12 hotswappable), my first “real” server grade hardware. Currently I use off-the-shelf components in a rack-mounted Rosewill case with 12 bays (all 8tb drives as mirrored vdevs) and it works just fine, but the drives are getting to be close to a decade old. However I want to start fresh, put brand new drives into this Dell system, clone everything over and use the old box as a backup mirror instead.
Given I have so many (to me) bays to work with, I’m leaning towards getting 12-20tb drives and fill them up. With that kind of ZFS layout would you guys suggest with those kinds of sizes? I definitely care about resiliency, but since this is a home build I’m not super concerned about balls to the wall write performance but I do care about reads.
I’m looking at possibly doing a cluster of mirrors again, but since I have an opportunity here for more drives/more capacity, is it worth doing this as a couple of Z1/2 vdevs instead?
I salute you sir! I can barely fill a pool a fraction of that size. Neither do I have anything on that scale. But I won’t let that stop me from making recommendations. It’s generally recommended not to put that many dries in one ZVOL. Given that, the sensible (?) thing would be to construct a pool consisting of two RAIDZ2 ZVOLS with 6 drives each. You can then lose any two drives and up to 4 drives (2 in each ZVOL) w/out losing data. If you have solid backups, you might consider a RAIDZ1 for the extra capacity.
Something that is a personal practice… I plan to meet capacity needs several years out. The main reason is that bigger drives should be less expensive in the future. I’m too cheap to spend $$$ to meet needs I won’t encounter for 5 years. I’ve also had pretty good luck using “refurbished” HDDs. One of the benefits of ZFS is that you can replace drives one at a time with larger drives and when all have been upsized, you can resize the array in place. I’ve started with 4TB HDDs and I’ve replaced several with 6TB HDDs. I do one about every 6 months (to a year.) This provides two benefits for me:
I have extra cold spares on the shelf that I could swap in if needed.
The drives are not all the same age so less susceptible to all malfunctioning at the same time.
It’s easier on the budget, particularly if I can catch some on sale. (OK, three benefits.)
I don’t know any others who do this, but it has worked well for me. And refurbished datacenter HDDs have been surprisingly reliable albeit very lightly loaded in my home lab.
Appreciate the response! That’s exactly what my structure has been the past while. Started with 4TB, then 8TB, and now I’m in that weird zone where anything >=12TB seems like a gamble
Yeah I am definitely looking for the 6x2 approach with RAIDZ2 with some hot spares, or 8x2 with cold spares. You might be right in going a bit slower in the ‘upgrade’ cycle and getting ~10TB drives might be better for my wallet, assuming the ‘replacement’ process doesn’t stress the other drives too much. That said I do have 3x18TB drives laying around that I could just get 3 more and make a single RAIDZ2 to start…
Depending on where you are shopping, I just saw 14TB drives for the same price as 10TB. Also consider cost per TB vs overall cost (if your wallet and/or partner/spouse/significant other can handle it). While shopping around, I saw that 10TB disks cost more per TB than 14 or 16TB.
Thanks for the suggestions everyone. I think I’ll be aiming for 8x2 mirrors or 4x4 Z2 once I figure out how much of a budget I want to spend on new drives. Coming from 8x2 mirrors of 8TB drives maybe I don’t need to jump up to 18-20TB right away and instead go 12-18, though it would certainly make the futureproof bug in me happy.
I can barely fill a pool a fraction of that size. Neither do I have anything on that scale. But I won’t let that stop me from making recommendations. 
It’s generally recommended not to put that many dries in one ZVOL. Given that, the sensible (?) thing would be to construct a pool consisting of two RAIDZ2 ZVOLS with 6 drives each. You can then lose any two drives and up to 4 drives (2 in each ZVOL) w/out losing data. If you have solid backups, you might consider a RAIDZ1 for the extra capacity.
