SATA SSD-Based Special Device (Special VDEV?) for all-HDD 3xMirror Pool: Worth it for Backup/Mass Storage NAS?

OpenZFS TrueNAS
1

Hello,

A bit of background:

I’ve got an 8 SATA bay TrueNAS Scale install that currently contains 3 mirrors (2x14TB disks), for a total usable space ~36 TB. Its primary purpose is to be the master local backup destination in my home/home office, and also local mass storage to the extent I need it (Plex, etc.). The HDDs are a combination of WD Golds and HGST Ultrastar HC530s.

They’re connected to my network via a 2x10 GbE LACP connection, on an MTU 9000 storage network.

I’m really enjoying how it’s all working so far, and I really don’t think I’m going to come up against the storage limit of 36 TB anytime soon (or even 28 TB, if I’m reserving 20 percent of the pool to give ZFS the space it needs not to get grumpy).

Question: In this context, to prevent disk thrashing/avoidable slowdowns when working with small files (e.g., when doing snapshots or other backup operations where a lot of very small files are being compared to a lot of other very small files and read/written accordingly), would it be worth it to add a SATA SSD mirror special device?

I know not to expect a massive boost in speed, but as I realistically don’t need more than 36 TB at this time, I think it could potentially help avoid certain slowdowns (and keep the whole array a bit quieter overall the more it can lean on the special vdevs, which is a bonus in my small office).

Or would the difference be so minimal that I wouldn’t really even notice?

I know it wouldn’t hurt anything to try, but I’d like to try to better understand how to predict the impact of making these sorts of changes. I’ve watched some really good YouTube videos, but they were aimed at enterprise storage, and I’m at the “overcomplicated home office” level, at best.

These are the SSDs: https://www.intel.com/content/dam/www/public/us/en/documents/product-specifications/ssd-dc-s3710-spec.pdf

Read and Write IOPS
(Full LBA Range, IOMeter* Queue Depth 32)
Random 4KB3 Reads: Up to 85,000 IOPS
Random 4KB Writes: Up to 45,000 IOPS
Random 8KB3 Reads: Up to 52,000 IOPS
Random 8KB Writes: Up to 21,000 IOPS

Bandwidth Performance
Sustained Sequential Read: Up to 550 MB/s
Sustained Sequential Write: Up to 520 MB/s

Endurance: 10 drive writes per day for 5 years
200GB: 3.6PB 400GB: 8.3PB
800GB: 16.9PB 1.2TB: 24.3PB

Latency (average sequential)
Read: 55 μs (TYP)
Write: 66 μs (TYP)

2

I don’t think it’s worth it. I’ve tested the SPECIAL fairly thoroughly, if artificially, and saw little or no real benefit in practice, even when trying to do things like run a find against tens of thousands of files (which means lots and lots of metadata).

This was on a system with eight 12TB Ironwolf in mirrors. Basically, four mirror vdevs of those drives just had enough oomph to handle most any artificial benchmark I threw at them–and the benchmarks that they struggled with, they struggled as or nearly as hard with or without the SPECIAL.

I think the most likely benefit you’d see would be the possibly less platter-chatter than you’re accustomed to, but can’t actually promise that either.

Honestly, if you’ve got a pair of fast SSDs burning a hole in your pocket I’d say spin up a separate, blazing-fast pool for any separate, blazing-fast storage needs you might have. You’ll get better use out of them that way.

1 Like
3

Thank you! (And thank you also for taking the time to reply during the holiday. :slight_smile: )

This is exactly the context I needed. I’ve got a separate, smaller all-SSD array set up for performance-sensitive storage. My original plan for those SATA SSDs was to keep them on hand in case one of the boot mirror disks in my Proxmox node failed, so I’ll keep reserving them for that.

Honestly, I kind of like the platter chatter. After spending so much time and effort and resources to build the server, it’s nice to know it’s doing something over there. :wink:

Meanwhile, I’ll keep looking for ways to become the more interesting sort of person who actually needs the storage pool that I accidentally overbuilt for myself. I might have to become a Z-list YouTuber.

(I’m about 40, and got my first computer in about 1996. Even modern loud HDDs, like multi-TB enterprise disks, still seem quieter than those SCSI and IDE drives were. It’s also a nice medium/low tone, so it’s a lot easier to mentally tune out than something like an aggravating PSU fan. )