Not the most descriptive title but it was the best I could do.
Is there any tooling out there that could help to pin down what folders (or files) are changing a lot size wise between snapshots (or a range of snapshots).
I’m thinking basically something like ‘ncdu’ over changed data for a snapshot range.
I can’t be the only one who has asked “what is the cause of all the data churn here?”
You can zfs diff snapshots to see what files changed. Of course, this isn’t very helpful if the snapshots are of a VM image, whether that image is a zvol or a qcow2 or raw file. Very useful on a native Linux or FreeBSD system running ZFS itself, though.
Yes, but without having drilled down every flag to ‘zfs diff’ I mostly don’t care about 2000 1KiB file changes but I do care about a single folder’s content changing 2GiB between every hourly snapshots.
The default ‘zfs diff’ output doesn’t get me much closer to tracking down the latter scenario.
Without any more context or details, I’m not sure what advice people can give. From my point of view, if this amount of churn feels expensive to keep in terms of snapshot deltas, your priority is of course to come up with a pruning policy that you can live with and can justify to your users.
What kind of workloads are you hosting? Is this a CICD workload that has large bursts of intermediate files being replaced? Is this a virtual desktop environment where browser cache is constantly changing? Where I’m going with this is: what filesystems can be split and recombined with a new mount-point articulation that provides a low-to-no snapshot policy for ephemeral files like .o files or .mozilla/cache trees?
You can pipe through awk and grep to identify only changes that measure in the GiB pretty easily.
If you want to identify changes at arbitrary sizes–for example, 50MiB rather than just “MiB”–you can still do it with only awk and grep if you’re determined enough, but the regexes get more complicated. In practice, I’d probably reach for the shell scripting language of choice (Perl, for me, Python, for most people these days) if I wanted that level of granularity.
Well, that is sort of exactly the reason I’m curious about such tooling. To discover where such churn is happening. Is it browser cache, is it some other cache? I have some self hosted services that generate some churn but is it the Unifi database or the Technitium DNS query logs that are generating the most data?
@mercenary_sysadmin When you say pipe to awk and grep here. Is it to then stat each file for size and then concat to a parent folder etc. or are there zfs diff flags I am unable to uncover?
AFAIK, yes, you need to use the generated list of file and or folder names to the compare the versions on each snapshot. You pipe the diff through grep to get only folders or only files first, because if you want to compare file sizes you stat them, but if you want to compare entire folders you need to du -s them instead.
If you make du or stat only report in a single block size, so you don’t have mixed units, you can then pipe the result through sort and head so that you get eg the top twenty largest changes.
Also, there are potential issues with du at the folder level instead of stat at the file level–a folder might show very little change in size between snapshots, but every single file in it might have been replaced with one of roughly equivalent size between those snapshots, which would not show up just from du-ing both sides.