Hello all,
Long time listener, first time caller. I don’t know of any resource like this anywhere else on the internet, so I thus turn to you lot for your sage advice.
Right now I have Proxmox running on an ancient Dell R710 with storage on ZFS mirrored NVMe (but PCIe 2.0) with 128GB DDR3 800 ECC.
I have acquired an R740 with no disks, 128 GB DDR4 2400 ECC, a 9305-16e, two 12-bay MD1400 drive shelves, 12x 8TB SAS 7200rpm 3.5" HDDs to go in one MD1400, and 12x SAS 400G MLC SSDs (10 DWPD, 6% wearout when purchased) to go in the other. I have five more 8TB disks which I plan to keep as spares to replace the others as they wear out, and no spare SSDs (but they are only fifty bucks on eBay or so, and I plan to buy more soon).
My plan is to migrate Proxmox to the R740, hang the two drive shelves off the R710 (four SAS lanes into each controller of the first MD1400 and four more into each controller of the other, using all the ports on the 9305), connect the two servers with 25G Ethernet and serve the storage via iSCSI, mostly to Proxmox but also perhaps with other boxes on the network.
As for backups, there is another R710 with a small CPU and 16 GB RAM running Proxmox Backup Server with its own set of SATA disks in RAIDZ1. My plan is to keep it fully self-contained (deliberately not using the iSCSI storage) to reduce the blast radius if shit hits the fan.
My questions:
- Am I a bit crazy?
- Does this seem to be a reasonable layout given the hardware I’ve got?
- Most importantly, how should I organize the storage ZFS-wise?
- Re organization:
- Should I keep the SSDs in their own pool for VM ZVOLs and stripe across five mirrored pairs of them, using the two remaining SSDs as a mirrored SLOG or SPECIAL vdev (or one L2ARC for the spinners and one spare for the ZVOL SSD pool) with the spinning rust? If so, how should the spinners be organized? I’d rather not lose half the bulk storage by mirroring the spinners too; where’s the sweet spot in practice?
- My understanding is that if all 24 drives are added to one pool ZFS will not prioritize the SSDs re performance; is that correct? Or should they indeed all live in one pool? (Does the hybrid pool feature of ZFS 2.4 change this?)
- How should the pool(s) be set up re initial configuration, block size for Proxmox’s ZVOLs, etc?
- ORRRRR given what I’m reading about ZVOL performance, should I scrap iSCSI as it relates to Proxmox, move the 9305-16e to the R740 to give Proxmox direct access to the disks, and then use the Proxmox server itself as an iSCSI server for other clients if that need arises? (I initially wanted to give myself flexibility to add multiple Proxmox heads, hence the shared storage.)
I’m sure I’ve forgotten some things; your thoughts are welcome.
Thanks in advance!