Expanding zpool with lower-spec drives (180 vs 270 MB/s) - will it bottleneck the pool?

OpenZFS
1

Hi,

I need to increase space in my main zpool. I’ve been trying to wait out the current hdd market shortage but I am nearly out of space.

I’m looking at adding a couple of 28 TB Seagate drives from serverpartsdeals to my existing zpool. However, I was checking the data sheet (ST28000NM000C) and noticed that they have considerably lower transfer rates and IOPS. i.e. transfer rate is approx. 270 on my current drives and only 180 on the ST28000NM000C; write IOPS similarly is lower, 550->320. From what I understand, this will slow my whole pool down to the new drives’ level. Interestingly, the data sheet mentions re-certified in its name.

I’m completely guessing here, but could Seagate have reduced their speed as part of re-certifying them, with the pretence that this makes them last longer?

Based on this, should I skip these drives and try and find something closer in performance to what I already have? e.g. serverpartsdeals have a couple of 18 TB drives with similar performance as my current drives but they cost $19.88/TB as opposed to 15.64. Alternatively I could create a new mirror zpool with just these drives and split my data between the faster 4 drive pool and this new 2 drive pool.

Details below:

NAME                                    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
titan                                  32.7T  30.4T  2.38T        -         -    32%    92%  1.00x    ONLINE  -
  mirror-0                             14.5T  13.5T  1.03T        -         -    34%  92.9%      -    ONLINE
    ata-ST16000NM000J-2TW103_ZR5DGNBE  14.6T      -      -        -         -      -      -      -    ONLINE
    ata-ST16000NM000J-2TW103_ZR5DGMV5  14.6T      -      -        -         -      -      -      -    ONLINE
  mirror-1                             18.2T  16.8T  1.35T        -         -    32%  92.6%      -    ONLINE
    ata-ST20000NM007D-3DJ103_ZVT16508  18.2T      -      -        -         -      -      -      -    ONLINE
    ata-ST20000NM007D-3DJ103_ZVT4MRYD  18.2T      -      -        -         -      -      -      -    ONLINE

Summary of hard drives, I own the first two and am considering buying either the 18TB or 28TB drives.

Capacity Name Model Max. Sustained Transfer Rate OD (MB/s,MiB/s) Random Read/Write 4K QD16 WCD (IOPS) Price
16 TB Seagate Exos X18 ST16000NM000J 270 / 258 170 / 550 NA
20 TB Seagate Exos X20 ST20000NM007D 285 / 272 168 / 550 NA
28 TB Seagate Exos ST28000NM000C 190 / 181 170 / 320 $437.99
18 TB Seagate Exos X20 ST18000NM003D 285 / 272 168 / 550 $357.95
2

Yes, the rule of thumb is that a vdev generally performs as though it is entirely composed of its slowest member drive, and a pool generally performs as though all vdevs were the same as its slowest vdev.

That said, you’re going from a two vdev pool to a three vdev pool. Which means higher overall throughput, for most workloads. Assume your original vdevs offer performance n, and the new vdev offers 0.5n.

The rule of thumb calculation–and this is a useful but very gross rule of thumb, there are spherical chickens galore in this–you’d be going from 2n to 1.5n performance. Which is only likely to matter for local workloads, because that pool of mirrors should saturate a network link pretty easily either way.

It’s also unlikely to make much difference for IOPS-heavy workloads that really shouldn’t be on rust in the first place.

One last warning: I haven’t looked up the specs on those drives and I have no idea if there are larger problems than the spec sheet hints at. See my series of pieces on Western Digital “NAS” drives using wildly inappropriate SMR technology from a few years ago, as an example: Series: SMR where you least expect it - Ars Technica

Basically, it sounds like you’ve got your head on reasonably straight about this: you know you’re looking at trading away some potential performance in return for lower cost per TiB. Accurate.

What I can’t tell you is whether it’s a good trade for you. I don’t know enough about your workload or exactly how much you value thrift vs performance, and I’m pretty much not going to. So my goal here is to help you figure it out, not to give you “the answer.” :upside_down_face:

I think if you’re interested in the savings but nervous about the performance, you test the water: make sure you’ve got a thirty day return window. Buy two drives. DON’T add them to your pool yet: make a new pool, and test the Jesus out of it. That’s what will give you a better real world idea of the relationship between performance of your existing drives vs the new drives–that elusive n performance I’m glib about in these rules of thumb.

You may discover that they’re essentially unfit for your purposes, and you want to send them back. Or they’re okay for backup (or whatever), but you don’t want them in your main pool. Or–IMO, the most likely outcome–you test them, sure they’re a bit slower, but you realize that a three vdev pool including those drives performs roughly 3x as fast as the test pool with just the one, and decide that’s good enough for you.

But you’re going to have to do the rest of the figuring out for yourself. I’d love to hear your results, though!

3

Thank you for the thorough reply. I’d not considered that adding another mirror vdev to an existing pool would partly speed the overall pool up. I’ll thoroughly compare the data sheets too just in case.

As for buying and testing. I’m a huge fan of buying stuff to see if it’s fit for purpose and have no qualms returning stuff. However, I’ve been trying to think how to test hard drives such that the results reflect real life. Benchmarks are fine but somewhat theoretical IMO.
One thought I had for testing them was to migrate some of the data to the new vdev with the slower drives and host the data from there and see if anybody (the fam) notices. I could also rsync some data to both existing and new zpools to see if there’s a difference there.

I think the above covers day to day use. Is there any obvious test case I’m missing?

1 Like
4

Sounds like sufficient testing from here. Me, I’d whip up a test suite using fio… but “me” is a person who spent a lot of time professionally mastering the underlying skillset. If you aren’t intensely interested in putting a lot of time and effort into not just being able to run a few commands but knowing what you’re doing with a flexible I/O tester, I think your idea is a better idea because it models your real world workload well and it’s something you’re comfortable with.