I’m trying to set up ZFS snapshot replication with Sanoid/Syncoid using a pull architecture. Snapshots are pruned from the backup server with Sanoid.
I’m considering a scenario where production is compromised, creating bogus snapshots to trick the backup server into pruning valuable data upon the next pull.
What are the best practices for handling this? I imagine either Sanoid would need to be sufficiently intelligent about what it prunes, beyond snapshot names, or I could have the backup server set holds on snapshots. Am I stuck developing my own service or is this a solved problem?
Thanks!
Pruning is usually date based, so it doesn’t matter how many snapshots or changes production makes, you’ll still have X number of days to find out production is hosed.
NEW data won’t be transferred if you run out of space, not old stuff.
The real danger is rollbacks. If the source destroys a bunch of old snapshots but still retains one very old common snapshot with the target, a new replication will destroy all of the snapshots newer than the common one on the target as well.
This is of limited value, since (if you’re using pull replication, and the target doesn’t trust the source, as I always recommend) the attacker has no way of knowing how far back the oldest common is–and if the oldest common is destroyed on the source, further replication attempts will simply fail, without triggering rollback on the target.
This is still the ugliest current vector available, and I keep meaning to build in configurable safeguards to intelligently limit the amount of data the target is willing to destroy. But I still haven’t done that (and it’s been a few years now since I experienced this issue personally, not thanks to an actual attacker, but thanks to a weapons-grade stupid minion of a client doing something INCREDIBLY dumb).
I would be very interested in patches to address this, though I have to warn I’ll be pretty opinionated about them given how deeply this affects core functionality.
Sanoid is pretty damn smart in this regard.
If you have daily=30 as a policy and three hundred new dailies suddenly show up, there is no effect, because sanoid only prunes dailies over 30 days old AND only if there are more than 30 existing dailies.
So you’d just have 330 dailies, until the bogus 300 are 31 days old–at which point (assuming you have in fact taken 30 new dailies since then) all 300 of the bogus ones will get pruned immediately.
Similarly, if you cut the power on a machine with thirty dailies and wait a year before powering it back on, sanoid will not immediately prune all thirty dailies–it will prune them one at a time, once per day, as and only if new dailies are taken.