Truly Awful, Horrendous, Very Bad Write Performance on Time Machine Backups: Or, Disabling Sync Writes for Time Machine (Data Integrity Implications?)

OpenZFS
1

I noticed some truly awful I/O performance when I ran my initial Time Machine backup. I had a live view up with zpool iostat, using the instructions here: OpenZFS: Using zpool iostat to monitor pool performance and health - Klara Systems

  1. The first shell window ran watch zpool iostat -v; and
  2. The second shell window ran zpool iostat -vy 1 1.

So, I had a pretty good look at what was going on.

The aggregate/average performance showed that the system can saturate my 10 Gbps link from the test Mac to the NAS … rarely. Otherwise, I was seeing truly awful sync_waitand similar metrics: sometimes in the millisecond range. Somtimes in the multiple seconds range. A few times, it just stalled out for ~30 seconds or so.

On the Mac side, I was watching the Mac’s NIC live stats, and it was struggling to break 1 Gbps a second most of the time. Not awesome.

So, I did a bit of googling, and found this write-up: GitHub - sebnapi/zfs-samba-timemachine-ubuntu: Step-by-step guide to hosting Apple Time Machine backups on Ubuntu using ZFS for storage and Samba for SMB sharing. Covers ZFS tuning for Time Machine workloads, full vfs_fruit configuration, Avahi mDNS auto-discovery, permissions, and startup/shutdown snapshots. · GitHub

There’s a step-by-step guide for setting up ZFS to have a happy time with Time Machine via the CLI, which I of course lack the expertise to evaluate. It also isn’t really relevant to how TrueNAS handles setting up a Time Machine storage target with its SMB share template function.

But what caught my eye was this:

sync=disabled

This is the single biggest performance improvement. Time Machine issues an fsync() call after every file write to the server. With sync=standard, each fsync flushes the ZIL to disk synchronously — effectively serialising every write. On a pool without a dedicated NVMe ZIL device (SLOG), this causes severe write stalls.

With sync=disabled, fsync calls return immediately without waiting for disk confirmation. The risk is that a sudden power loss could cause the last writes since the previous snapshot to be lost. This is mitigated by the snapshot strategy described below — and Time Machine is designed to recover gracefully from an interrupted backup by redoing it on the next run.

That would certainly explain the aggro sync write behavior I was seeing. It’s an HDD pool, as well, which also probably did me no favors on frequent syncs (and might help explain the frequent all-out stalls?).

(The snapshot strategy from the article assumes the NAS is powered off most of the time, so that’s not really helpful. I’m probably going to have to bite the bullet and figure out how to apply my own snapshot schedule an not rely on the nice, clean “snapshot when backup completes” built-in.)

Even now, my hourly backups take quite a while, and I’d to feel like I’m actually taking advantage of my 10 Gbps connection and the 4x mirror 8 disk pool I built to saturate it.

The thing is, I don’t have a SLOG to use with sync=standard. I’m out of PCIe slots or NVME slots, so I’d have to YOLO it with an external USB 4 powered NVME dock. (Something like this: OWC Express 4M2: DIY Four-Slot USB4 NVMe M.2 SSD Enclosure ).

Assuming my UPS holds, sudden power loss is not really something I have to worry about here. I lose power for a few moments maybe a ten times a year, at most. I lose power for extended time periods (hours/days) maybe once or twice a year. And my UPS where the NAS is hooked up has enough juice for the NAS to gracefully shut itself off when the power loss is detected.

With that in mind, what are the realistic risks of running with sync=disabled in this scenario?

Ref: My Adventures with TrueNAS, ZFS, and Time Machine: A Master Post

2

Essentially none, because if the time machine backup crashes halfway through, you don’t actually have a usable backup anyway. And if you’re taking snapshots after each successful replication completes, then your RTO to access the last good backup before the crash is measured as “how long does it take me to shell into the NAS? well, add about 500-1500ms to that.”

It would be interesting to see how time machine itself reacts to being asked to start a second backup process, after an earlier process crashes (or you otherwise deliberately corrupt the most recent TM backup) and without rolling back to the most recent good backup.

But I suspect time machine can probably deal with that pretty well on its own… And even if I’m overestimating Apple’s developers (which I certainly have done before), you’re always just 500-1500ms from that rollback to last known good anyway.