Hello,
I’ve got 4x 2TB enterprise SSDs that I want to put into a 2 vdev mirror pool. The pool will be given to Proxmox Backup Server as a storage target.
One pair is a set of Micron 5300 Pros, and the other is a read-intensive Samsung pair of Samsung PM883s. On paper, they’re both rated for the same (or nearly the same) sequential read/write, though that’s certainly not a reliable metric.
However, the IOPS are quite different.
So, the Samsungs are faster, but for Proxmox Backup Server’s chunk-comparison-based I/O, I’m not sure how much that actually matters.
Question (See, I was going somewhere with this!): Is there any value in mixing the drive types for each vdev? That is, using a Micron and a Samsung in each vdev, versus having a Samsung vdev and a Micron vdev.
My instinct is that mixing them would force both vdevs to perform at the Micron’s slower speeds.
I haven’t used a pool yet where whole vdevs have entirely different drive types in them, so I’m not sure what the impact of that would be.
Either arrangement would be sufficient. I’m just curious what to expect in terms of performance.
IMHO, I’d mix them (so each mirror vdev will have 1 Samsung + 1 Micron).
For reads, typically the faster one will “win” and so you shouldn’t notice any difference.
For writes, it’s the slower write that would be the “effective” write speed (so the Microns). But either way, you’ll have it spread across 2 vdevs and so they’ll be symmetric (and anyway for non-sync writes, they shouldn’t really be that impactful).
My take on the mixing of drives is that you’ll (hopefully) spread the risk of failure across 2 vdevs especially if the drives come from similar manufacturing “batches”.
Of course this is all probability, statistics and lies. Ultimately there’s no real substitute for backups 
2 Likes
Thanks!
What made you assume most of the writes would be async (“non-sync writes”)?
I’m not really as familiar with how PBS interacts with ZFS as I’d like to be; I assumed it was using sync writes but haven’t actually checked.
I’m not sure on whether PBS writes are sync or not. Unless explicitly forced, most writes are typically async (and it is usually DBs that would be forced syncs ).
Also, in case of a heavy sync-write workload, it may make sense to look at an SLOG.
I’d say it’s worth double checking.
Irrespective, for failure redundancy, I would still be more comfortable with the mixed drives VDEV.
I’d say it’s worth double checking.
I’ll definitely check when I rebuild my local PBS.
Since I’ll be using a SATA SSD array, and the fastest NIC I have on the new PBS server is 1 GB/s, I’ll have to see if async writes are really a constraint. If it’s hitting anywhere close to 1 GB/s with sync writes enabled, I’ll probably just leave it like that since data integrity is a priority.
Ironically, while I could add a slog to that system, the TrueNAS system where I really WANT a slog is a no-go. I’m out of PCIe lanes there. 
Maybe I’ll try getting crazy and doing a USB4-based TrueNAS SLOG. What could go wrong? 
2 Likes
That is not “quite different” at all; the delta described is well below the noise floor. These drives might very well have very different performance characteristics–but if they do, these stats aren’t demonstrating it.
You should probably mix the drives in the vdevs to increase the consistency of pool behavior, as well as make it less likely that you’ll lose both sides of a mirror vdev before you can resilver.
Generally speaking, you probably have your estimate of performance backwards, btw: read intensive enterprise drives perform well only under a light (write) workload, while standard enterprise drives perform much more consistently under much heavier workloads.
Most enterprise SSDs feature power loss protection, meaning they return from sync() immediately and there is very little gain to be had from implementing a LOG vdev.
I said “very little” rather than “none” because it does still offload the “extra” ephemeral writes to the ZIL away from main storage, freeing up the IOPS writing to the ZIL would otherwise have eaten. In most cases, though, that’s not enough of a benefit to make a LOG vdev worthwhile.
1 Like