Preferred ZFS solution for HA with virtual machines and databases

OpenZFS Proxmox
1

Hi,

I would like to replace very old ESXi solution with Proxmox and OpenZFS.

Currently it’s three server and few shared arrays over ISCSi

It’s hosting (nearly) all software in relatively small company. There’s few VMs, but only two are heavily using resources:
1). The most offender is keeping warehouse state, orders, etc. It’s windows based and uses Microsoft SQL. It’s keeping all files in database, it’s less then 1TB, but still one big blob :wink:
2) Much less resource hungry. Uses linux, postgreSQL and svn for keeping files. It’s for managing all electronic documents.

Nearly everybody wants ‘commercial supported software’, like ESX. But I dislike this outdated setup(required Adobe Flash for vSphere) with passion and i’ts sometimes corrupting files, because arrays got over 15min delays sometimes.
he worst is this are Dell/ESX blackboxes, with limited linux command set, that’s hard to debug. It’s so old, that needs to be replaced by anything more modern anyway.

I wanted to build now something for tests with some leftovers.
I’ve got 12* SATA 12TB HDDs, 3* Radian RMS-200 and 4* Optane 480GB.
Beside VMs I would like to have also some kind for NAS (Currently employers are using SMB shares…but pool is small, so are relying more on USB HDDs and Dropbox…)
Unsure if it it’s highly disregarded to have both NAS and VMs server as one setup?

I know there are few possible implementations:

  1. multiple VM servers + one shared storage. I guess it needs SAS drives for multipath and HA. Am I right?
  2. mutiple VM servers with their own storage. As I understand there’s nearly always ZFS replication used, but DRBD can also be used(is it always better synced in realtime?)
  3. mutiple VM severs and multiple shared storage. I guess it’s a mix of both above.

I wonder if there are other solutions too?
1st. is the cheaper I guess.

Does all modern SAS drives are multi-port?

Should I avoid using Optanes in modern setups?

What’s the useful life for Radian RMS devices and longevity of supercapacitors within?

I saw some layouts with shared storage, with multipath drives, but it looked like every server got it’s own ZIL(like they were in VM servers, not in shared storage server), or maybe I’ve misunderstood something(and maybe I should sleep more, before reading anything). Does it makes sense at all?

I’m not much familiar with Proxmox and wonder if I should proxmox both on VM servers and shared storage server?

And the last question is about adding servers(and another storage for them) in other(remote) location. How should I replicate data locally and remotely then? Is automatic failover possible then? (This company has warehouses and workforce in few location around country. Unsure if HA would work predictably and with acceptable performance/latency) then?

2

I do this type of thing with ZFS replication and it works fine as long as you don’t need continual synchronization. I have a data store by the same name on each of my cluster nodes and a replication task every 15 minutes. It does the trick in my setup.

3

If you want true HA, you’ll need to layer that in above the ZFS level. Generally speaking, you’d use DRBD for that. Keep in mind just having DRBD doesn’t mean your failover issue is sorted with whatever you’re running on top of the DRBD, though; I think Proxmox supports HA but I don’t know for certain.

Also keep in mind that true HA is at least as much curse as blessing. It’s an outstanding way to make sure that once you screw something up in production, you automatically screw it up in your spare as well!