Speed up your resilver with more memory

I want to share something I learned about zfs today! I watched this video: https://www.youtube.com/watch?v=SZFwv8BdBj4&pp=ygUMWmZzIHJlc2lsdmVy which is all about diving into the new (well, not new anymore) algorithm for scrubs and resilvering.

In the video he talks about a tuneable which allows you to specify the percentage of RAM to use for scanning and queueing blocks for reading. The more ahead of time scanning you do the better the performance.

I’m not going to describe the process here, but it’s a fascinating watch and I recommend the video.

Anyway, I tried out the tuneable parameter zfs_scan_mem_lim_fact (Module Parameters — OpenZFS documentation). By default it is 20 which allocates 5% of RAM to the queue. I set it to 5, which allocates 20% of RAM to the queue.

On a RAIDZ1 pool with 6.5T allocated (including parity), the default scrub took 4h11m. After tuning, scrub was 3h32m. A handy improvement.

Do note that ARC will get squeezed due to high memory usage.

I don’t actually need faster scrubs but I thought this was a fun experiment and wanted to share.