Is a two drive mirror better than a single drive vdev that sends snapshots to another single drive vdev?

OpenZFS Sanoid
1

Continuing the discussion from Managing snapshots on target machine:

I came here looking to validate what I thought was a good topology. The linked topic above is essentially what I was looking to do. The context is that my finances don’t stretch to many drives, the space in my house is too full of kids to have anything other than a NUC and the use case is primarily fun with some practical use for the family. Basically I already have the hardware.

I was thinking that there would be two pools on top of a fairly disposable OS. One pool with one vdev with one drive would be the primary data that all my Linux ISOs are stored on for my family to browse. There would be no redundancy but there would be the other benefits of ZFS. The second pool is also a single vdev with a single drive in it. This would be the backup drive and would receive snapshots from the first drive, making up the more comprehensive retention. Each drive would be SSD 2TB. This is the setup that sort of made sense to me as I am learning the technology. It made sense as I am used to more traditional backup strategies using tools like rsync and borg backup. Data is sent from the primary location to another drive.

Reading the response from Jim on the linked topic made me continue to think on the solution. It sounded like snapshots are really meant to reside in the same pool as the data. Snapshots seem to be a bit magic to me as they don’t appear to be taking up the same space as the data they are snapshotting (?). Would a better ™ solution be to have the two drives set up as mirror, keeping the comprehensive retention of the snapshots all together?

If I am correct, you would gain redundancy but lose some storage, the amount of storage is something I can’t figure out due to a skill issue. The amount of storage lost is less than I thought originally as it sounds like there would need to be a common snapshot on both target and source anyway, I would be forgetting about the second pool and having a good range of snapshots on the first.

Side note: Super thrilled to be here talking about ZFS.

1 Like
2

I’m no expert, but here are my 2¢.

tl;dr:
You likely won’t see a difference—worth worrying about (maybe 10’s of MB in metadata, but even if it reaches the low 100’s MB, that is only a small percentage of your 2TB available)—in storage used with either case.

full response:
Do some more reading about snapshots, and experiment using them with a “fake” pool. Jim has some articles that use them. Another source you can use to learn how to get started practicing is on the Arch Linux wiki. That will help you understand the role of snapshots. They don’t take space until a file changes. They are sort of a bookmark (or a sort of version control if familiar with git) to what a file looks like at the moment the snapshot was taken. The snapshot only starts using more disk space once a file changes, because you now sort of have two copies of the file (though not duplicating the full file size, only duplicating the parts that changed).
The snapshot is the data. When you replicate to a backup, you now have the same snapshots, i.e. the same data/files/bits, in the original pool and the replication target pool.
You don’t gain data redundancy with either case of a mirror or two, separate, single-drive pools. In either case, your data is on two drives. There is perhaps slightly higher fragility to the mirror case, but if both drives are in the same box, I suspect the risk is fairly equal.

If you are replicating the first drive to the second one, as an independent pool, I believe the only extra storage space used may be for another copy of metadata.
I know metadata is kept with copy=true be default, but not sure if it mirrors that onto both drives in a single pool, 2-disk mirror. I would assume it does, in which case both of your proposed schemes will essentially use the same amount of storage.

Having two pools makes it easy to yank one drive and throw into a new box one day, and have a true external target. Again, I’m not sure, but in this case, I suspect yanking a mirror drive and throwing into an external box as a backup target may require manually changing some sort of guid on the pool or something, besides renaming the pool. Probably have to touch hostid in either case.

Last thing I can think of is recovery. I don’t know if there is a difference in load on the surviving drive between resilvering in a mirror, vs replicating from a pool in the same box. Your case of SSDs could potentially alter the answer from someone with HDDs as well. If the load is heavier in one case than the other, you have a higher risk of the 2nd drive failing mid-resilver. This would be “no bueno”.

Personally, I would probably go with two pools, then all your effort is done up front. If/when you get another drive and/or box, you don’t have to go through the pool creation process again (unless the size of your new disk is significantly larger—look up metaslabs—you would want to create a new pool out of the new, larger drives anyway). All you would need to do is scrub, attach, resilver (might be automatically started from the attach, don’t remember), and one last scrub. That would turn one of your single disk pools into a mirror. If you get a new box, scrub, export on the old one, move the drive to the new box, import, and scrub. You would already have the workflow set up for backups, you would just have to change the location to an external box instead of a local pool.
With a separate pool, you can still heal corruption reported by a scrub with an option in send and receive. It just wouldn’t happen automagically. But then, it is probably time to replace the drive anyway, unless you simply are a magnet for cosmic rays.

Keep good notes on what you did to set up the pools (i.e. the exact commands you ran) and how you set up the replication from one pool to the other (i.e. what tools you chose to use, and again, the exact commands you use with those tools). Your future self will thank you.

3

I would always recommend a raid1 mirror with zfs. There are basically two reasons:

  1. Redundancy in the pool enables zfs to repair all chksum errors on the fly. zfs on a single drive can not do that,

  2. the read performance of a raid1 mirror is twice as fast as a single drive. That makes a significant difference in day to day work. When you read big files like a movie or raw pictures the performance difference is very obvious.

1 Like
4

He’s already on NVME, and dragging the playback position bar in a media player isn’t likely to saturate that. And even if it did saturate, how often does a person drag that bar? I would suspect not often. Further, if he has a constrained budget and only 2TB, I have the strong suspicion he isn’t keeping family photos in RAW format. Unless he’s a photographer, but then, he would probably have a separate drive, and probably much larger, for business.

From the tone of original post, it seems the priority here is cost, data safety, and time sunk into regular maintenance (with kids running around, ain’t nobody got time, haha).

But yes, as I stated, automatic healing can be convenient. It just isn’t the only option. It can be done from a backup pool.

Either way, now the OP has an alternate opinion to consider, which is always a good thing!

5

All I am saying is that redundancy in a pool has big advantages over single drives.

6

These viewpoints are really valuable to me, thanks for taking the time to write them out.

@feinedsquirrel there is a bunch to digest here which I am keen to do. I will certainly need to read some more on how snapshots work. I was familiar with the concept of the snapshot containing pointers to where the data is on disk but there is certainly some more reading which needs to be done on my part.

@mabod I didn’t feel good about sacrificing redundancy but thought it was a good option. However I think that redundancy might win out in the end if the storage sacrifice isn’t really a thing. I think a Raid1 mirror with sanoid automation appeals to my sense of order and understanding.

I still need to buy the second drive though still. So I might do that and test out snapshots and raid in the mean time. After I done some good old fashioned reading though.

Thanks again, I will be back

7

Hi ya👋 !
I would argue that you have the 2 drive inside the same box. Have a replication approach make sense if the second drive is in another box outside your house. In your case the mirror would be a better choice since you want redundancy. To quote Jim: “Snapshot are cheap, not free” snapshots will grow according to changes in your data. If you have a bunch of Linux ISOs that I suppose don’t change much :slight_smile: your snapshots will not grows as much but if you start add and remove files, then those are not removed until it’s in one of your snapshots. So depending on your space needs you should plan to rotate them. Since your plan is to have the same disk size you probably can’t have more retention if you go with plan B.
Remember: things can go wrong, the nuc can be flooded and the mirror will not save you. Plan a backup outside, it can be zfs replication or something else like restic but do it!

TL;DR go with mirror and plan a backup outside

1 Like
8

Just FYI
if you want an example how much space snapshots can consume

My home directory is a ZFS dataset. It has 68 snapshots starting in February 2024 (i keep one snapshot per month plus some more recent snapshots).

# zfs list zHome/home/matthias
NAME                  USED  AVAIL  REFER  MOUNTPOINT
zHome/home/matthias   426G   393G   150G  /home/matthias

That says, that the whole dataset incl. all snapshots consums 426 GB (USED). The actual data without snapshots consumes 150 GB (REFER). That is not bad keeping in mind that these snapshots contain a version history of my documents since 19 month.

EDIT
My backup server has even 236 snapshots for my home, consuming 1.14 TB.

# zfs list zf1/BACKUP/rakete_home/matthias
NAME                                          USED  AVAIL  REFER  MOUNTPOINT
zf1/BACKUP/rakete_home/matthias              1.14T  3.18T   140G  /mnt/zf1//BACKUP/rakete_home/matthias

size difference 140 GB vs. 150 GB between backup and original home dataset is due to compression zstd (backup) vs. lz4 (original home)

9

OK, so, short version here:

  • backup is better than redundancy
  • backup AND redundancy is MUCH better than backup alone

Essentially, I’d recommend at least a little redundancy to go along with your backup. Two machines, replication backs up your data between them, but put a mirror in one or the other of them–source or target doesn’t much matter, but one of them.

The reason for this is that recovering from on-disk corruption that actually happens (as opposed to the kind that’s prevented by redundancy in the first place) is a genuine pain in the ass. At the small scale that you’re talking about operating at, there’s really (in my opinion) no good reason not to expand your plans from two disks to three, even if you don’t want to do four (for redundancy on both source and target).

If you put the mirror on your source, you get uptime benefits (production keeps trucking if a drive dies) and some performance benefits, as well as your corruption-mitigating redundancy being right there in production (so you don’t have to go to backups to recover from any corruption that might occur). But if you lose the backup drive, you have to do a new backup from scratch.

If you put the mirror in your backup, your backup gets harder to kill… but you probably care about uptime more in production (yes, whatever machine you play on the internet on at home counts as “production”) than you do on the backup. You probably also care more about performance in production than you do on the backup, and that leaves us with “it doesn’t MATTER matter, but almost certainly if you only have redundancy in one place, that one place should be production.”

If you can possibly afford it, I’d really recommend a simple mirror on both sides.

10

@mabod If I am understanding your example on your home directory, the snapshots are taking up nearly three times as much space as the actual data ?

Well I went ahead and bought an additional 2Tb drive to go with the one that came with the box. So now I have two drives installed in the server ready to be formatted and setup. I did do a setup originally but I must have stuffed something up as the server didn’t boot. Or rather it did but it got mighty confused about where things were and couldn’t locate the bits to make the network work. I am pretty sure I know what I did wrong so hoping this next run will go better.

I am going to go with a two drive mirror in the box with the snapshots all on the same place. If this all goes well I will get back the original Pi which is currently doing server duty so there is a world in which that becomes a backup machine in some capacity. Otherwise I might try and source some non ssd larger capacity drive(s) to server as a backup location for the main server.

I am reading through this thread again and I appreciate the input and advise. It sounds like more drives is always better which makes sense. Really it comes down to the finances and when a 2Tb SSD costs 100 - 150 quid it becomes hard to approve that spend with the family. Especially as we don’t even watch the Linux ISOs in the end! I have one spare drive which is too small I can use in the mean time though as a stop gap before I can source something else.

11

yes, that can happen. My home directory is a very busy place. Lots of changes happen between two snapshots. Think about it, snapshots from home started in Feb 2024. Every file that has been deleted in my home since Feb 2024 is still present in one or the other snapshot. If I changed an office document in the meantime I still have the old version of that office document in one of the snapshots.

12

Nice. Just checking I was understanding the information correctly. It all makes sense to me!