You haven’t said how much RAM you have, but I suspect it was using the RAM to the best of its ability. The L2ARC is another story entirely. And it’s a long, complicated story that boils down largely to “this doesn’t work the way you think it does, and isn’t half as cool as you think it is” unfortunately.
The short version is that the L2ARC does not hold everything that isn’t in ARC, by a long shot. Nor is it actually an ARC at all. The L2ARC is a dead-simple bone-stupid FIFO cache (not even LRU), which is fed at a heavily throttled maximum rate from blocks which might get evicted soon from ARC.
An fio test isn’t generally going to do a good job feeding the L2ARC. Typically, either the fio job size is small–in which case the ARC will retain the entire working set, and the L2ARC will have nothing to do–or the fio job size will be much larger than ARC, in which case it’ll explosively flow through it without hanging around for long enough to get much of it into the L2ARC. Either way, you’re looking at (very) low hitrate city.
The L2ARC’s job is, essentially, to play cleanup behind the ARC. There is a non-zero chance that blocks recently evicted from ARC will be in the L2ARC, in which case you can maybe pick up another few percent of total cache hitrate by leveraging L2ARC. But you’re pretty much always going to be looking at ratios like >80% for ARC, <10% for L2ARC, at best.
I strongly recommend reading through the L2ARC explainer I wrote for Klara Systems, if you’ve still got questions: OpenZFS: All about the cache vdev or L2ARC | Klara Inc
hoping to get some benefit from reads
Reads are a bit of a complicated story under ZFS. Because it’s copy-on-write, ZFS generally greatly reduces effective fragmentation in most real-world workloads: odds are pretty good you’ll want to read things in roughly similar large groups to the way you wrote them, and since it’s copy-on-write, those groups get written mostly sequentially, which means fewer seeks when reading them back in a similar order.
But fio isn’t a real-world workload; it’s an entirely randomized (assuming you’re using randrw, randread, and randwrite, which you should) workload which will tend to minimize the beneficial impact of copy-on-write data ordering significantly as opposed to real-world workloads, because there is essentially no chance the working set will be read back from disk in anything faintly like similar patterns as the way it was originally written.
So, when you’re using fio, you’re not getting the typical real-world benefit out of either ARC or L2ARC, and also getting bitten much harder by seeks than you normally would. In the L2ARC’s case, that usually doesn’t matter much, because it’s such a niche utility vdev in real life also. In the ARC’s case, it can be pretty badly misleading, especially for non-database workloads. (But even databases will frequently tend to read rows in similar patterns as the way those rows were written, making even that heavily random-access workload benefit more from both copy-on-write and the ARC itself in the real world than in a fully-randomized synthetic test).
It gets even more complicated if you want to talk about read specifically in the context of RAIDz vs conventional RAID, but this is probably enough to chew on for the moment.