3-2-1 Backup of ZFS in a Home Server Environment

OpenZFS
1

Hi, I’m Maran, and I’m new to this forum. I’ve already found some useful information here, which is why I decided to create a post of my own. I don’t work in the storage industry, but I’m tech-savvy. It’s a hobby of mine, as well as a way for me to avoid Big Tech. I like to experiment with technology. Due to time constraints, I look for solutions that require some tinkering or attention but can work for six months without daily maintenance during stressful periods when I can’t devote much time to it.

I run my own home server and I really like it. I use TrueNAS CE for this purpose because it provides the power of ZFS through a user-friendly web interface. I like my home server so much that I started adding valuable things to it, like family pictures, which brings up the question of a sound backup strategy.

I already take advantage of ZFS’s incredible replication functionality to stream my data to a friend’s home far away, in a different country. This way, I have a second off-site copy of my data. However, I have three issues with my setup:

  1. I still need a third copy.
  2. I need a medium other than ZFS.
  3. My friend’s Internet is so slow that backups work fine over time, but in a serious recovery scenario, it would most likely be cheaper to catch a flight and transfer the data locally. This is kind of okay for one backup copy, but without a third, fast copy, it’s kind of unacceptable.

In order to implement a decent 3-2-1 backup strategy, I would need an additional medium. Off-site storage is already covered. Please let me know if I am wrong or have overlooked something.

Personally, I thought it would be best to have another server in the same rack running two services.

  1. Proxmox Backup Server for my VMs
  2. Something that takes care of my ZFS data. I thought MinIO might work.

What would you recommend?

I’m looking for a relatively cost-effective solution. I don’t run a company; I just have a bunch of valuable family photos. It should also be able to be turned off so that I don’t have to pay for electricity all the time. Since my home server already has 80 TB, a multi-disk setup with MinIO would be a feasible way to achieve resiliency.

The idea is that the backup system will lock every file for three months. Additionally, the backup solution should be fast. Nothing extreme, but the backup and recovery times shouldn’t be excessive, as would be the case if I needed to use data from my friend’s ZFS replication box. However, I would be happy if I maxed out at 1 Gbps.

Initially, I planned to set up a second server with Proxmox and virtualize both Proxmox Backup Server and a VM running MinIO with attached disks. Any thoughts? I might also add SSD/NVMe disks to the Proxmox Backup Server VM to speed up VM backups. Then, I could load all the data onto my main home server and stream it to my friend’s TrueNAS system, as well as to the time-locked system next to my main system. All of this should be fully automatic. Additionally, I would like to back up my Proxmox VMs in case something breaks, so I can recover quickly.

Due to price and privacy concerns, as well as my desire for relatively fast recovery times, which cannot be achieved with cloud solutions at my current location, I am not looking for another cloud solution (I consider my friend’s replication box to be one). My Internet speed is nowhere near 1 Gbps, whereas 1 Gbps is typical for an inexpensive Ethernet cable when only local networking is involved.

I hope to take advantage of the storage expertise on this forum and get feedback about my plan. Please tell me if I’m on the right track or suggest better alternatives. Any other feedback is much appreciated!

2

I use MinIO as a backend for restic and autorestic. I target both a Backblaze B2 bucket, and a locally-hosted MinIO bucket. Works quite well, although I sometimes run into issues where the bucket is locked and restic craps out until I go run restic unlock manually. Some monitoring there is a good idea, if you go that route.

MinIO recently had some controversy and removed some nice admin UI features. Since you mentioned in your post you like TrueNAS because of a “user-friendly web interface” you would be stuck using the mc command (or buying a commercial license with MinIO to access some other web GUI they have for administration).

Since it’s photos you’re backing up, I’ll mention Immich which is self-hosted and you could setup to run on a separate system.

As an iPhone user, I personally use icloud_photos_downloader to grab photos from iCloud, store on ZFS, and shuffle off to B2 and MinIO via restic.

I run a second box with Proxmox Backup Server and I agree it’s a nice safety net for VM backup and restoring is dead simple. (Although that’s just the VM, underlying datasets that I mount as Samba shares in the VM aren’t backed up to PBS.). Highly recommend using PBS if you’re in the Proxmox ecosystem.

2 Likes
3

I’m using TrueNas Core for storing my personal data. Next to zfs replication to an offside location I use Borg Backup. The software is open source and free. The reason I use this next to zfs replication is it is cheaper to buy target storage space. If I want to replicate to a zfs host then I have to rent a VM (or rent zfs space), the Borg Backup target doesn’t have to be an vm with zfs, it requires ordinary space which is cheaper.

Borg backup doesn’t rely on zfs replication basics, it supports data deduplication and data encryption. I run it in a separate jail as non root user. I created a backup user which should be able to acces the data to be backuped. I like its speed, if data to be backuped doesn’t change then backups are fast. Next to an automated backup job, I do automated tests to assert some files contain an expected checksum. The backup an test are assured to be run by triggering healthchecks.io. If jobs fail or aren’t run at predefined frequencies then I will be informed.

1 Like
4

Thank you for your insight!

I want to back up data other than images, too. It was only the images that prompted me to look into decent backup options.

I am familiar with Restic and Borg.

I know Proxmox Backup Server only backs up VM data, not attached data.

What is your recommended method for mounting data into individual Proxmox VMs from a TrueNAS VM on the same physical machine?

Before we discuss specifics, I would like to know if it makes sense to back up ZFS data on another medium. If so, which medium would you prefer? Would it be S3 with Restic or Borg?

Can Borg or Restic be integrated directly into TrueNAS?

What about the TrueNAS “Cloud Sync Tasks” tool in the web interface? I can set up Amazon S3 as a target, which should allow me to point it to my MinIO server. Would that be a good option? Why use Borg or Restic instead of an integrated feature? Or am I misunderstanding what TrueNAS Cloud Sync Tasks are?

Thanks!

5

Does anyone here have more information or real-world experience with TrueNAS “Cloud Sync Tasks”?

6

In my opinion the most important aspect is that backup is in place. Less important is how this is implemented, either zfs replication or backup to object storage target or backup over ssh to ordinary storage space. It all dependd on your preferences. I prefer borg over restic because borg offers better privacy due to meta data encryption, borg features deduplication on block level: if a large file is changed for s small part then only the changed blocks are stored. Further I prefer zfs replication over any other mechanism because FreeBsd / TrueNas Core has build in support for it. I do not have any experience in object storage providers. My first choice is replication to a zfs host however due to increased costs I am looking to use borg backup in future. Borg works flawless. If costs didn’t matter I would use zfs replication because of out of box support.

Yes, you can set up borg or restic in a jail.

I do not have any experience with TrueNas cloud sync tasks.

1 Like
7

What do you use as the Borg repository in terms of service or hardware?

8

I’m considering Hetzner storage box, their smallest subscription is € 3.20 excl VAT for 1 TB per month. They provide an extended SSH service for Borg Backup. A lot of information is available on how to setup and configure it, e.g. here.

9

Have you looked at rsync.net as I believe they still offer direct access to zfs pools for send/receive? There is also zfs.rent which it looks like will let you even colo your own HD’s with them if you want to pre seed your replication pool.

10

Indeed, at rsync.net you can choose for an option where you have direct access to the pool. I always struggle to find that offer on their website, but Jim’s 2015 article on Ars Technica has a link to it: rsync.net Cloud Storage for Offsite Backups.

The (potential) downside: the minimum storage amount is 5TB.

I have been using this service for several years now and I love it. It works as advertised and they inform you in time if they need to do maintenance or if there’s a FreeBSD update that they can or need to apply.

1 Like
11

Hi, why not securing zfs pools via usb on external disks for backup purposes ?
I use 3 disks, one attached to an ubuntu server, the other 2 disks stored in a different building. I change the disks (in a rotating manner) after some weeks.

Every Sunday a backup ist performed using a tiny script:
/usr/sbin/zpool import ExternalDisk >> /dev/null
/usr/sbin/syncoid --compress=none --sendoptions=wp --no-sync-snap Server/Tank ExternalDisk/Tank
/sbin/zpool status |grep -v scan | grep -v repaired > /Server/Tank/Dokumente/zpool-status.txt
/usr/sbin/zpool export ExternalDisk >> /dev/null

12

I do not prefer manual interaction, it should be set and forget. I’m not that disciplined to perform a manual action each every some weeks. Furthermore the expenses are also important for me:

  • rsync.net (ZFS target) charges for 1TB 12.29 USD per month, about €10.54.
  • Hetzner (borg target, non ZFS) charges for 1TB €3.20 excl VAT per month.

Although I prefer ZFS targets the borg target is about 3x cheaper compared to zfs.

13

How efficient is the borg based backup storage solution compared to ZFS? To keep the same amount of file versions does it take more, less or a similar amount of raw storage? I am not familiar with how borg stores its backup data or how efficient it is compared to zfs and its use of block level operations. If it is less efficient and you end up having to purchase more storage than expected, the difference could end up smaller than you think.

14

I don’t know how efficient a borg based backup storage solution is compared to ZFS. However I asked AI for a prediction.

My backup use case:

  • a lot of small files, typically photo’s (< 4MB), these files do not change
  • some other media files (10 MB - 10 GB), these file do not change
  • office documents (filesize in kb range), if it’s a ‘hot’ document then some frequent changes over a week, cold documents do not change anymore.
  • no virtual machines files
  • ZFS dedup disabled
  • lz4 compression enabled on ZFS data sets

According to AI is seems for my use case both systems can achieve similar sizes, but Borg still tends to be a bit more efficient because of its finer‑grained chunking.

Note I didn’t measure this statement so I can’t substantiate the statement. AI output is pasted for reference only, take appropriate skepticism.

How Efficient Is Borg‑Based Backup Storage Compared to ZFS?

Below is an overview of how both systems work, which factors affect storage size, and a practical approach to estimate efficiency yourself.

1. How Borg Stores Data

Feature Description
Chunk‑based approach Borg splits each file into variable‑size “chunks” (default 9 KB – 10 MB). Identical chunks receive a single storage identity.
Content‑defined deduplication If two files (or versions) contain the same data, their chunks are shared—regardless of path or name.
Compression By default Borg uses lz4 (fast, lossless) or zstd (higher compression). Compression is applied per chunk, so only unique data gets compressed.
Encryption (optional) When encryption (AES‑256) is enabled, deduplication can no longer occur on the encrypted data unless a deterministic mode is used (rarely recommended).
Metadata Borg also stores small amounts of metadata (timestamps, permissions, etc.). This overhead is tiny compared with the chunk data.

Result:

  • High deduplication when many files stay unchanged between backups.
  • Compression further reduces raw size, especially for text or log files.
  • Storage overhead is mainly the chunk index (a few MB per TiB, depending on configuration).

2. How ZFS Snapshots Work

Feature Description
Copy‑on‑write (COW) A snapshot is a point‑in‑time view of the filesystem. Only blocks that change later are copied; the original blocks remain part of the snapshot.
Block‑level ZFS works with fixed block sizes (e.g., 128 KB – 1 MB). Any modification creates a new block, even if only a single byte changes.
No compression by default ZFS offers optional compression (lz4, gzip, zstd). Without it, the raw block size stays the same.
Deduplication (optional) ZFS has a built‑in deduplication engine, but it requires huge RAM (≈ 1 GB RAM per 1 TB data) and is rarely enabled in production because of performance and cost implications.
Metadata Snapshots also store metadata, but this is relatively small.

Result:

  • Efficient for small changes: only changed blocks are duplicated.
  • Less fine‑grained deduplication than Borg, because blocks are larger and any change costs a whole block.
  • Compression can dramatically shrink storage, but without it the overhead is usually higher than Borg when many small changes occur.

3. Comparison of Raw Storage Requirements

Aspect Borg (with borgmatic) ZFS Snapshots
Deduplication granularity Chunk‑based (variable, often smaller than ZFS blocks) → higher chance of sharing data. Block‑based (fixed) → less fine‑grained, more “wasted space” for tiny edits.
Compression Default lz4 (or zstd) per chunk → effective, especially for text. Optional lz4/zstd per block → comparable, but must be explicitly enabled.
Metadata/index overhead A few MB per TiB (chunk index). Very low, unless ZFS dedup is turned on (then massive RAM/CPU cost).
Storage for identical versions Stored once (thanks to deduplication). Stored once per block; if a file is unchanged, no extra space, but a tiny edit forces a whole block copy.
Scenario: Many small changes Borg typically saves 30 %–70 % more space than ZFS (without ZFS dedup). ZFS can grow faster because each altered block is fully copied.
Scenario: Large files, few changes Similar efficiency; compression dominates. Also similar, but ZFS may use slightly more due to larger block size.
Scenario: Full encryption Deduplication disappears → storage comparable to unencrypted ZFS (no compression). ZFS encryption (per dataset) retains COW behavior, but dedup remains disabled.

Bottom line:

  • Without ZFS dedup and without compression, Borg usually consumes less raw storage, especially when you have many small modifications or many identical files across backups.
  • With compression enabled (lz4 or zstd), the gap narrows; both systems can achieve similar sizes, but Borg still tends to be a bit more efficient because of its finer‑grained chunking.
  • If you enable ZFS dedup, you could theoretically match or exceed Borg’s efficiency, but the RAM and CPU requirements make this impractical for most deployments.
15

I have a ZFS dataset I backup using borg for work, so I can actually provide some numbers here :slight_smile:
This is the used/refer/compressratio for the ZFS dataset

NAME           USED  LUSED  RATIO
pool/commons  4.84T  7.81T  1.49x
pool/homes    4.75T  7.83T  1.56x

And here’s the borg info for the pool/commons repo:

                       Original size      Compressed size    Deduplicated size
All archives:              106.60 TB             70.70 TB              6.90 TB
                       Unique chunks         Total chunks
Chunk index:                 5093566             80193565

I recently cleaned up some older versions (pruned the borg repo), so there’s 18 versions, spanning a year at most. Unfortunately the homes repo is pruning at the moment, so I can’t get you the data for that one right now, but it’s 4.2TB on disk.
It should be noted that pool/commons is a pretty quiet dataset, hence the high deduplication. Creating the last archive for pool/commons took a bit over 5 minutes, so speed is not bad:

Duration: 5 minutes 24.45 seconds
Utilization of maximum supported archive size: 1%
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:                7.27 TB              4.74 TB            222.55 kB
All archives:              106.60 TB             70.70 TB              6.90 TB
                       Unique chunks         Total chunks
Chunk index:                 5093566             80193565

I create the backups using a combination of sanoid and borgmatic*. Sanoid regularly creates zfs snapshots. Borgmatic mounts the last snapshot somewhere, and backs up that folder by pushing it over SSH to another server. The second server has an authorized ssh key restricted to run borg server --append-only --restrict-to-path ... to make sure archive deletion is not possible in case of a compromised ssh key/server.
So far I’m very happy with it. Restoring data from borg is also pretty easy due to the borg mount command.

HTH!

* before_backup: ["mount -t zfs $(zfs list -t snapshot -S creation -H -o name pool/commons | head -n1) /mnt/commons-backup"]; unmount after_backup and on_error.