I’m been reading about Proxmox as an alternative to vSphere.
Is there a way to build a “good” HA setup on Proxmox using ZFS? I’m thinking of something like a 4-5 node cluster to be used for Gitlab runners and test environments for developers.
After watching a YouTube video about Proxmox and storage (https://www.youtube.com/watch?v=a7OMi3bw0pQ), I was left with the impression that Ceph is the default way to go for an HA setup. However, upon further studying, Ceph seems…complex and hungry for hardware resources.
Is there a “better” approach to ZFS HA than what is shown in the video starting from 12m:20s? The approach taken there is that each node has its own local ZFS storage. On node 1, there are two replication jobs regularly (minutes in between) replicating a VM on node 1 to node 2 and node 3.
The way I manage this is basically what you’ve described from that video. I have local pools that replicate on a regular schedule to other nodes. For the VMs and containers that I want on local storage that’s my go-to. Everything else is run off an NFS storage pool backed by ZFS.
PVE does have ceph controls in the UI and the ability to set up gluster or DRBD manually, but those are all quite a bit more complicated than my own use case warrants.
Yeah, it’s a single TrueNAS Core box connected with a 10Gb link on a dedicated storage network.
I mostly use it for workloads that don’t need the low latency of a local disk. That data store keeps all of my iso and template files as well as qcow2 disk images for VMs.
One thing to note: Proxmox can’t do raw snapshots on NFS, so qcow2 VMs are fine, but raw images and LXC containers won’t have that functionality (you can still do snapshots from the NAS, of course) and that can impact your backup strategy.