Santiy check: Low value data: 8x HDD RAIDZ1 vdev + 2x2 SSD stripe/mirror special vdev (RAID10) with ~2GB partition cut out for SLOG

OpenZFS
1

Setup:
All SAS enterprise drives
8x 1.2 TB HDD
4x 200GB SSD
1x 1TB HDD
32GB DDR4 ECC RAM

Context:
Homelab
Almost entirely low value data
Any high value data will be automatically backed up elsewhere regularly
Proxmox OS – Though I think this post isn’t really fundamentally about that
Intended to run WireGuard, media, web, and email (internal and maybe relayed SMTP) in individualized containers; and OPNsense in a VM 24/7
Intended to be available for or sometimes run any other kind of service in varying configurations

I first thought of going full stripe on the SSDs and 6x HDD with a mirror of HDD–OS and some services on the SSD, storage on the HDD stripes with backups on the mirror. Really, I’m not going to care about the data and I would have external backups for configuration files or whatever else–game server data, emails, mostly-text web pages.

I ran oodles of fio testing, and felt the HDD stripes lacking in certain ways that special vdev can compliment; and how special_small_blocks can be set per dataset. I knew about SLOG, and recognize application to sync data writes; and despite the low-value, don’t-much-care, or otherwise small-sized/easy-to-backup high-value data, I thought: I should learn Ansible to trivially recover from losing installs, but also, I’m partial to mitigating against pool failure against HDD failure.

In that way, I came up with the following configuration that I’d like you to evaluate:
Pool:
8x 1.2TB HDD RAIDZ1 vdev
2x2 200GB SSD stripe/mirror (RAID10) partially partitioned for a tiny bit SLOG vdev (~2GB) and the rest as special vdev

Honestly, I anticipate the SSDs to be vastly more durable than the HDDs, so I’m open to flat-out striping them at the risk of catastrophic loss of the entire pool; and again, I’ll have an external and scripted backup service for data that I care about. Plus, there’s that lonesome HDD.

Controlling for what does or does not go into the SSDs is a matter of structuring datasets and the special_small_blocks property. In that way, I get the benefit of special assisted HDD and the power of what could otherwise be of a separate SSD pool–combined.

2

Not a great idea. Performance will be abysmal, fault tolerance is extremely low, storage efficiency will be considerably lower than you’re probably expecting (due both to padding necessary for 7 data disks per stripe and to inability to split metadata and other small blocks across the entire stripe).

Honestly, I wouldn’t really recommend using a pile of eight rust drives this small for anything. This vdev would be outperformed by a single 8TB HDD, which would also offer more usable storage space and consume a small fraction of the power.

I’m sorry to keep being a negative Nancy, but this is a bad idea also. LOG vdevs and special vdevs are both primarily intended to improve latency, and if you’re making the same drives do multiple duty for three or four separate tasks, you’re most likely going to deteriorate experienced latency, not improve it–particularly at the peak values.

3

When you say,

inability to split metadata and other small blocks across the entire stripe

, is that despite the attached SSD vdev? To clarify, this is all one pool–not two. I thought that the special vdev would carry the metadata and small blocks–that’s what the per-dataset special_small_blocks would be for. fio results, I thought, looked pretty good with the special setup (no SLOG–that was an afterthought), but I take for granted what you say as I take you as an esteemed subject matter expert.

EDIT:
I’m learning about how RAIDZ splits recordsize across number of non-parity disks then pads with junk data to fill out differences, so odd numbers don’t work so well, it seems. I’m looking more into it. I thought that record blocks were kept intact per disk. Thanks again for that tidbit.

4

Well, small blocks will land on the special up to the value selected, that’s very true. But the larger you make that value, the more quickly you’ll fill that special and then it won’t be, well… Special.

And with an eight wide Z1 over 4Kn disks, you won’t even be able to get a full stripe width write out of 32KiB blocks. 64KiB blocks will be full stripe width, but will only be doing around 8K I/O per drive, which is still deeply into pain territory and especially on rust. 128KiB blocks will at least be able to do roughly 16K I/O per drive… Which is better, but it’s still only going to manage a tiny fraction of the throughput each drive is theoretically capable of.

You get the idea.