Many uses for SSDs in a zpool -- can they share the same drives?

Good morning folks; I have a pool comprised of spinning hard drives, and I know there are places I can put SSDs in my pool to improve performance:

  • SLOG: buffer newly written blocks to the SSD so we can acknowledge the write to the client fater, then flush that buffer to the HDDs at our leisure
  • L2ARC: utility is debated, but allows you to cache frequently used blocks on faster storage automatically based on the number of cache misses
  • Special Metadata VDEV: store filesystem metadata on separate SSD instead of alongside the files
  • bonus: separate, faster pool for cache data (e.g. an application wants a separate “cache” directory for data it wants to have available quickly, so maybe that should go on a separate, all-SSD pool?)

My problem is – if I bought a pair of SSDs to mirror for each one of those use cases it’d get very expensive and I’d start to run out of PCIe lanes. So I’m thinking about buying just two or three NVMe SSDs, and partitioning them (e.g. partition 1 on each drive is the L2ARC, partition 2 is for metadata, partition 3 is SLOG and partition 4 is the separate, faster pool.)

Are there any performance/longevity pitfalls I should consider in that approach? Are there any of those cases that really need to be their own separate drives?

if I bought a pair of SSDs to mirror for each one of those use cases it’d get very expensive and I’d start to run out of PCIe lanes.

Well, you don’t need a mirror for all those use cases. CACHE vdevs (“l2arc”) do not need any sort of redundancy. LOG vdevs (SLOG) are generally recommended to be redundant, but the only thing they store of value is dirty writes–essentially, if your system crashes and your LOG vdev dies, you just lose the last few seconds of dirty sync writes on that drive; essentially, it’s the same thing as your crash happening a few seconds before it really happened.

I am generally a big fan of using your SSDs for a separate, high performance all-flash pool where the high performance workloads go. Using SSDs as LOG, CACHE, and SPECIAL saws off some of the roughest edges of a rust pool, but it does not transform it into a pool that behaves as though it were not a rust pool.