Hi,
I’m rebuilding my lab/production environment looking for a sounding board on the approach.
For context I run all my own services and store all my families data so integrity and recoverability of data is critical to my family. The services I operate mostly impact me but my priority is for a speedy recovery.
I currently have 3 Epyc servers but could scale down to 1. I use these servers to run KVM with a range of workloads with data being backed up daily using restic to my local TrueNAS server. For the virtual machines I use bootc so I can recover the VM’s quickly and restore data from the last backup. My critical data like family photos lives on my TrueNAS server exposed over SMB with snapshots taken hourly with daily snapshots stored with 3 years of roll back. These are replicated to 3 other TrueNAS servers around the country.
My current thinking is that I can scale down to a single Epyc server with my local TrueNAS Mini X+ over a 10GB for network based storage. I am thinking about;
-
having the operating system virtual machine qcow2 on the TrueNAS server (these are mostly be read only using bootc with the files stored in git)
-
the /var/x mount is my only persistent volume which will also be on the TrueNAS server
- would you recommend attaching a second qcow2 for /var/x or just mounting an NFS volume?
-
I currently backup all my data daily with scripts that do things like stop containers like redis, mariadb, nextcloud, quay registry and minecraft servers and backup the persistent data using restic to the TrueNAS server
- To reduce my window for data loss I am hoping that my being on the TrueNAS server I can snapshot the datasets rather than just the daily backups using restic. Other then testing, what is the recommended approach to reliably understand whether my services will tolerate snapshots without the risk of corrupting the data?
-
My last question is about performance. I really enjoy my lab so money isn’t the biggest concern. I have read the Klara posts about ZFS VDEV layout and configuration for workloads which I will align with.
- Any one able to provide experience on performance? I intend on housing the data on either mirrored SSD or NVME. Again I can use my existing TrueNAS Mini X+ with these options or re-purpose an Epyc server at which point I have waay for PCIE lanes and fast disk options to play with. My priority here is being able to enjoy using my lab, not being frustrated waiting for things to build etc.
-
The TrueNAS Mini X+ (Supermicro A2SDi-8C±HLN4): currently has 4 spinning HDD’s
-
A secondary TrueNAS server (Supermicro A2SDi-16C-HLN4F ) has 6 Crucial MX500 SSD’s
-
The Epyc Servers use the ROMED6U-2L2T with an Epyc 7302 a single NVME local storage.
Also open to feedback - I want to be as professionally responsible here whilst not being over the top.
Appreciate the advice,
Adam