I have a pool of 2x14Tb and 1x5Tb mechanical disks. I want a read cache Larc2 it and is available a 2Tb sata SSD.
I have 16bg of ram but 2T seems too much. what is the right size to allocate?
thanks
I would say that is hard to tell since we don’t even know what your disks are used for.
I they are mainly used to store 4-30GB movie files (from your 4K drone, not from hollywood of course) you probably don’t need any read cache. If they are used for VMs a cache is probably very beneficial for performance.
I myself use 2x120GB partitions on a SSD to cache 2x14TB mechanical drives. But that was less for performance reaons and more in a hope that this will avoid unnecessary spinups of the mechanical drive for unimportant directory listings etc.
You don’t have enough RAM to worry about an L2_ARC. Buy more RAM before you do anything else.
You don’t need to listen to people who say “you don’t need an l2arc”. Do your own tests. I’ve found that even a cheap 64GB USB3 thumbdrive on a USB2 port can help with ZFS caching, you don’t need ultrafast nvme drives for it.
This is on MacOS Monterey with a 4TB Toshiba NAS disk mounted over esata and a 20GB l2arc partition on a Samsung T7 USB3:
time find /Volumes/ztoshtera6/ >/dev/null
real 3m42.157s
user 0m1.801s
sys 0m44.914s
2nd run
time find /Volumes/ztoshtera6/ >/dev/null
real 0m32.377s
Linux does way better performance on the 2nd run though, especially if you have limited-RAM situations like 8 or 16GB. I’m trying to get to where I can allocate SSD partitions to cache ~10% of drive/pool capacity; have been setting up things kind of ad hoc so far. It’s not been easy since most of my partitions are already used for Linux triple-booting, separate /home and the like - but I have a spare T5 that can be repurposed.
L2arc became much more interesting when ZFS came out with cache that can survive a reboot. https://www.reddit.com/r/zfs/comments/fzj98s/persistent_l2arc_has_been_added_to_openzfs/
On my dedicated ZFS tower server I’ve gotten more methodical, it’s got a 15-bay SAS shelf with 14x4TB SAS drives in DRAID and ~47TB of “raw” disk in raidz2-equivalent protection. I gave it 2x128GB L2arc partitions (not mirrored) on 2 separate Dell 800GB Enterprise SSDs, plus another 40GB of l2arc partition on a Samsung 860 Pro. Server has 32GB RAM. Second-run “find” performance (Linux) on that zpool was under 5 seconds 
with ~20TiB of the pool allocated.
For your use case I would try caching ~5% of the 2x14TB (~70GB) and 10% of the 1x5TB by using GPT partitions on your 2TB SSD. Leave extra/unallocated space on the disk in case you need more later. Do some timing tests - find, ls -lR, updatedb, etc and then reboot and try doubling the cache allocation, see if it makes a difference. You can always ’ zpool remove ’ an extra cache device if it’s not needed.
2 Likes