For benchmarking purposes is there a way to fully (and cleanly) disable L2ARC without removing it from the pool entirely? I tried zpool offline <device> but something went wrong when I brought it back online – I forget what it was exactly. In the end I had to fully remove it from the pool and add it back (and it had to re-populate from scratch).
zfs set secondarycache=none <poolname> works to prevent writes into L2ARC but it won’t prevent read hits from screwing with my benchmarks.
Is there a better way?
Not that I’m aware of, but really that shouldn’t typically be necessary anyway:
zfs create poolname/newdataset
zfs set secondarycache=none poolname/newdataset
fio --directory=/poolname/newdataset
And Bob’s your uncle.
If you desperately need access to existing data for some custom benchmark thing you’re doing, I’d advise just replicating that dataset into your newly-created dataset that has secondarycache=none set on it. Even though the blocks will have the same value as the original blocks, they aren’t the same blocks, and any existing entries in the CACHE vdev will not apply to the replicated copies in your new, explicit testing dataset with secondarycache=none set on it.
Note that you can also use the same technique with primarycache, if for some reason you need to run benchmarks that explicitly exclude ARC hits entirely.