L2ARC – cache vdevs

[grahamperrin]

@JesseW

… OS runs on the internal SSD … from what I’ve read, putting ZFS on USB drives is a bad idea. …

With FreeBSD 14.0-CURRENT on an internal hard disk drive, I get huge performance boosts from low-spec USB flash drives as cache vdevs.

A visual overview, with net-mgmt/netdata:

Three USB flash drives currently at da0, da1 and da2:

% lsblk
DEVICE         MAJ:MIN SIZE TYPE                                          LABEL MOUNT
ada0             0:123 932G GPT                                               - -
  ada0p1         0:125 260M efi                                    gpt/efiboot0 -
  <FREE>         -:-   1.0M -                                                 - -
  ada0p2         0:127  16G freebsd-swap                              gpt/swap0 SWAP
  ada0p2.eli     2:51   16G freebsd-swap                                      - SWAP
  ada0p3         0:129 915G freebsd-zfs                                gpt/zfs0 <ZFS>
  ada0p3.eli     0:137 915G -                                                 - -
  <FREE>         -:-   708K -                                                 - -
da0              0:155 7.2G GPT                                               - -
  <FREE>         -:-   1.0M -                                                 - -
  da0p1          0:156 7.2G freebsd-zfs                       gpt/cache3-august <ZFS>
  <FREE>         -:-   1.0M -                                                 - -
da1              2:67   29G GPT                                               - -
  <FREE>         -:-   1.0M -                                                 - -
  da1p1          2:68   29G freebsd-zfs                       gpt/cache1-august <ZFS>
  <FREE>         -:-   490K -                                                 - -
da2              0:190  14G GPT                                               - -
  <FREE>         -:-   1.0M -                                                 - -
  da2p1          0:191  14G freebsd-zfs                       gpt/cache2-august <ZFS>
  <FREE>         -:-   1.0M -                                                 - -
da3              2:72  932G GPT                                               - -
  <FREE>         -:-   1.0M -                                                 - -
  da3p1          2:73  932G freebsd-zfs                           gpt/Transcend <ZFS>
  <FREE>         -:-   712K -                                                 - -
% 

% zpool list -v
NAME                  SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
Transcend             928G   454G   474G        -         -    10%    48%  1.00x    ONLINE  -
  gpt/Transcend       932G   454G   474G        -         -    10%  48.9%      -    ONLINE
august                912G   743G   169G        -         -    70%    81%  1.00x    ONLINE  -
  ada0p3.eli          915G   743G   169G        -         -    70%  81.5%      -    ONLINE
cache                    -      -      -        -         -      -      -      -         -
  gpt/cache2-august  14.4G  14.4G  68.7M        -         -     0%  99.5%      -    ONLINE
  gpt/cache3-august  7.22G  5.97G  1.25G        -         -     0%  82.7%      -    ONLINE
  gpt/cache1-august  28.8G  18.2G  10.6G        -         -     0%  63.3%      -    ONLINE
% 

% geom disk list
Geom name: ada0
Providers:
1. Name: ada0
   Mediasize: 1000204886016 (932G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r2w2e4
   descr: HGST HTS721010A9E630
   lunid: 5000cca8c8f669d2
   ident: JR1000D33VPSBE
   rotationrate: 7200
   fwsectors: 63
   fwheads: 16

Geom name: cd0
Providers:
1. Name: cd0
   Mediasize: 0 (0B)
   Sectorsize: 2048
   Mode: r0w0e0
   descr: hp DVDRAM GT80N
   ident: (null)
   rotationrate: unknown
   fwsectors: 0
   fwheads: 0

Geom name: da0
Providers:
1. Name: da0
   Mediasize: 7755268096 (7.2G)
   Sectorsize: 512
   Mode: r1w1e3
   descr: Kingston DataTraveler 2.0
   ident: 60A44C4253CABDA03B46026A
   rotationrate: unknown
   fwsectors: 63
   fwheads: 255

Geom name: da2
Providers:
1. Name: da2
   Mediasize: 15502147584 (14G)
   Sectorsize: 512
   Mode: r1w1e3
   descr: Kingston DataTraveler 3.0
   lunname: PHISON  USB3
   lunid: 2000acde48234567
   ident: 08606E6B6446BFB138159554
   rotationrate: unknown
   fwsectors: 63
   fwheads: 255

Geom name: da1
Providers:
1. Name: da1
   Mediasize: 30943995904 (29G)
   Sectorsize: 512
   Mode: r1w1e3
   descr: Kingston DataTraveler 3.0
   ident: E0D55EA1C84FF390A9500FDA
   rotationrate: unknown
   fwsectors: 63
   fwheads: 255

Geom name: da3
Providers:
1. Name: da3
   Mediasize: 1000204886016 (932G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e3
   descr: StoreJet Transcend
   lunid: 5000000000000001
   ident: S2S6J9FD203745
   rotationrate: unknown
   fwsectors: 63
   fwheads: 255

% 

Retrospective

Three days ago, whilst starting an app that’s very heavily used by me – note how little is read from 7,200 RPM /dev/ada0p3.eli:

% zpool iostat -v 10
                       capacity     operations     bandwidth 
pool                 alloc   free   read  write   read  write
-------------------  -----  -----  -----  -----  -----  -----
august                770G   142G      3     32  90.9K   615K
  ada0p3.eli          770G   142G      3     32  90.9K   615K
cache                    -      -      -      -      -      -
  gpt/cache2-august  13.1G  1.33G     10      0   485K  31.9K
  gpt/cache3-august  5.54G  1.68G      7      0   349K  42.8K
  gpt/cache1-august  9.81G  19.0G      9      0   466K  32.3K
-------------------  -----  -----  -----  -----  -----  -----
                       capacity     operations     bandwidth 
pool                 alloc   free   read  write   read  write
-------------------  -----  -----  -----  -----  -----  -----
august                770G   142G      0     22  34.8K   275K
  ada0p3.eli          770G   142G      0     22  34.8K   275K
cache                    -      -      -      -      -      -
  gpt/cache2-august  13.1G  1.33G    195      0  6.54M  1.85K
  gpt/cache3-august  5.54G  1.68G     98      1  2.97M  7.74K
  gpt/cache1-august  9.81G  19.0G    215      1  7.43M  27.2K
-------------------  -----  -----  -----  -----  -----  -----
                       capacity     operations     bandwidth 
pool                 alloc   free   read  write   read  write
-------------------  -----  -----  -----  -----  -----  -----
august                770G   142G      0     32  5.20K  1.16M
  ada0p3.eli          770G   142G      0     32  5.20K  1.16M
cache                    -      -      -      -      -      -
  gpt/cache2-august  13.1G  1.33G     68      0  2.87M      0
  gpt/cache3-august  5.54G  1.68G     28      0  1.10M  10.9K
  gpt/cache1-august  9.81G  19.0G     64      0  2.62M  2.40K
-------------------  -----  -----  -----  -----  -----  -----
                       capacity     operations     bandwidth 
pool                 alloc   free   read  write   read  write
-------------------  -----  -----  -----  -----  -----  -----
august                770G   142G      3     50  82.4K  1.42M
  ada0p3.eli          770G   142G      3     50  82.4K  1.42M
cache                    -      -      -      -      -      -
  gpt/cache2-august  13.1G  1.34G     29      0   745K    511
  gpt/cache3-august  5.54G  1.68G     22      0   595K  79.3K
  gpt/cache1-august  9.80G  19.0G     25      0   653K      0
-------------------  -----  -----  -----  -----  -----  -----
                       capacity     operations     bandwidth 
pool                 alloc   free   read  write   read  write
-------------------  -----  -----  -----  -----  -----  -----
august                770G   142G      0     62  8.80K  1.23M
  ada0p3.eli          770G   142G      0     62  8.80K  1.23M
cache                    -      -      -      -      -      -
  gpt/cache2-august  13.1G  1.34G     19      0   530K    204
  gpt/cache3-august  5.54G  1.68G      9      1   310K   222K
  gpt/cache1-august  9.80G  19.0G     14      0   503K      0
-------------------  -----  -----  -----  -----  -----  -----
                       capacity     operations     bandwidth 
pool                 alloc   free   read  write   read  write
-------------------  -----  -----  -----  -----  -----  -----
august                770G   142G      0     24      0   406K
  ada0p3.eli          770G   142G      0     24      0   406K
cache                    -      -      -      -      -      -
  gpt/cache2-august  13.1G  1.34G      0      0  42.3K      0
  gpt/cache3-august  5.54G  1.68G      2      0   280K  41.7K
  gpt/cache1-august  9.80G  19.0G      0      0  63.4K      0
-------------------  -----  -----  -----  -----  -----  -----
                       capacity     operations     bandwidth 
pool                 alloc   free   read  write   read  write
-------------------  -----  -----  -----  -----  -----  -----
august                770G   142G      0     42    818   982K
  ada0p3.eli          770G   142G      0     42    818   982K
cache                    -      -      -      -      -      -
  gpt/cache2-august  13.1G  1.34G      5      0   521K    460
  gpt/cache3-august  5.54G  1.68G      3      0   358K  46.0K
  gpt/cache1-august  9.80G  19.0G      3      0   349K      0
-------------------  -----  -----  -----  -----  -----  -----
                       capacity     operations     bandwidth 
pool                 alloc   free   read  write   read  write
-------------------  -----  -----  -----  -----  -----  -----
august                770G   142G      0     44  9.60K   919K
  ada0p3.eli          770G   142G      0     44  9.60K   919K
cache                    -      -      -      -      -      -
  gpt/cache2-august  13.1G  1.34G      3      0   323K     51
  gpt/cache3-august  5.53G  1.68G      1      0  66.0K  78.9K
  gpt/cache1-august  9.80G  19.0G      2      0   146K      0
-------------------  -----  -----  -----  -----  -----  -----
                       capacity     operations     bandwidth 
pool                 alloc   free   read  write   read  write
-------------------  -----  -----  -----  -----  -----  -----
august                770G   142G      1     39  8.40K   741K
  ada0p3.eli          770G   142G      1     39  8.40K   741K
cache                    -      -      -      -      -      -
  gpt/cache2-august  13.1G  1.34G     28      0   125K    972
  gpt/cache3-august  5.53G  1.68G     38      0   158K  5.80K
  gpt/cache1-august  9.80G  19.0G     22      0   104K      0
-------------------  -----  -----  -----  -----  -----  -----
                       capacity     operations     bandwidth 
pool                 alloc   free   read  write   read  write
-------------------  -----  -----  -----  -----  -----  -----
august                770G   142G      0     27      0   377K
  ada0p3.eli          770G   142G      0     27      0   377K
cache                    -      -      -      -      -      -
  gpt/cache2-august  13.1G  1.34G      0      0      0    920
  gpt/cache3-august  5.53G  1.68G      0      0      0  16.1K
  gpt/cache1-august  9.80G  19.0G      0      0  1.60K      0
-------------------  -----  -----  -----  -----  -----  -----
                       capacity     operations     bandwidth 
pool                 alloc   free   read  write   read  write
-------------------  -----  -----  -----  -----  -----  -----
august                770G   142G      0     27    818   474K
  ada0p3.eli          770G   142G      0     27    818   474K
cache                    -      -      -      -      -      -
  gpt/cache2-august  13.1G  1.34G      0      0  12.8K      0
  gpt/cache3-august  5.53G  1.68G      0      0  1.60K     51
  gpt/cache1-august  9.80G  19.0G      0      0  4.40K      0
-------------------  -----  -----  -----  -----  -----  -----
                       capacity     operations     bandwidth 
pool                 alloc   free   read  write   read  write
-------------------  -----  -----  -----  -----  -----  -----
august                770G   142G      0     28      0   472K
  ada0p3.eli          770G   142G      0     28      0   472K
cache                    -      -      -      -      -      -
  gpt/cache2-august  13.1G  1.34G      0      0      0      0
  gpt/cache3-august  5.53G  1.68G      0      0  1.20K      0
  gpt/cache1-august  9.80G  19.0G      0      0      0      0
-------------------  -----  -----  -----  -----  -----  -----
                       capacity     operations     bandwidth 
pool                 alloc   free   read  write   read  write
-------------------  -----  -----  -----  -----  -----  -----
august                770G   142G      0     35  2.40K   864K
  ada0p3.eli          770G   142G      0     35  2.40K   864K
cache                    -      -      -      -      -      -
  gpt/cache2-august  13.1G  1.34G      2      0  55.7K    920
  gpt/cache3-august  5.53G  1.68G      2      0  70.2K  12.0K
  gpt/cache1-august  9.80G  19.0G      2      0  35.1K      0
-------------------  -----  -----  -----  -----  -----  -----
                       capacity     operations     bandwidth 
pool                 alloc   free   read  write   read  write
-------------------  -----  -----  -----  -----  -----  -----
august                770G   142G      0     47  8.00K   909K
  ada0p3.eli          770G   142G      0     47  8.00K   909K
cache                    -      -      -      -      -      -
  gpt/cache2-august  13.1G  1.34G     10      0   408K    870
  gpt/cache3-august  5.53G  1.68G      4      0   149K  43.2K
  gpt/cache1-august  9.80G  19.0G      9      0   425K  4.00K
-------------------  -----  -----  -----  -----  -----  -----
                       capacity     operations     bandwidth 
pool                 alloc   free   read  write   read  write
-------------------  -----  -----  -----  -----  -----  -----
august                770G   142G      0     31  5.19K   516K
  ada0p3.eli          770G   142G      0     31  5.19K   516K
cache                    -      -      -      -      -      -
  gpt/cache2-august  13.1G  1.34G      2      0  42.1K     51
  gpt/cache3-august  5.53G  1.69G      2      0  47.1K  20.9K
  gpt/cache1-august  9.80G  19.0G      2      0  28.7K      0
-------------------  -----  -----  -----  -----  -----  -----
                       capacity     operations     bandwidth 
pool                 alloc   free   read  write   read  write
-------------------  -----  -----  -----  -----  -----  -----
august                770G   142G      0     45  1.60K   958K
  ada0p3.eli          770G   142G      0     45  1.60K   958K
cache                    -      -      -      -      -      -
  gpt/cache2-august  13.1G  1.34G     94      0  3.02M    102
  gpt/cache3-august  5.53G  1.69G     40      0  1.24M  31.9K
  gpt/cache1-august  9.80G  19.0G     94      0  3.23M  2.40K
-------------------  -----  -----  -----  -----  -----  -----
                       capacity     operations     bandwidth 
pool                 alloc   free   read  write   read  write
-------------------  -----  -----  -----  -----  -----  -----
august                770G   142G      0     35    409   538K
  ada0p3.eli          770G   142G      0     35    409   538K
cache                    -      -      -      -      -      -
  gpt/cache2-august  13.1G  1.34G    102      0  3.58M    511
  gpt/cache3-august  5.53G  1.69G     48      0  1.65M  32.9K
  gpt/cache1-august  9.80G  19.0G    109      1  4.00M  80.4K
-------------------  -----  -----  -----  -----  -----  -----
                       capacity     operations     bandwidth 
pool                 alloc   free   read  write   read  write
-------------------  -----  -----  -----  -----  -----  -----
august                770G   142G      0     37  4.40K   748K
  ada0p3.eli          770G   142G      0     37  4.40K   748K
cache                    -      -      -      -      -      -
  gpt/cache2-august  13.1G  1.34G     39      0  1.34M    511
  gpt/cache3-august  5.53G  1.69G     15      1   516K   128K
  gpt/cache1-august  9.80G  19.0G     39      0  1.33M  6.39K
-------------------  -----  -----  -----  -----  -----  -----
                       capacity     operations     bandwidth 
pool                 alloc   free   read  write   read  write
-------------------  -----  -----  -----  -----  -----  -----
august                770G   142G      2     49  13.6K   974K
  ada0p3.eli          770G   142G      2     49  13.6K   974K
cache                    -      -      -      -      -      -
  gpt/cache2-august  13.1G  1.34G      3      0  45.8K   128K
  gpt/cache3-august  5.53G  1.69G      3      0  24.4K     51
  gpt/cache1-august  9.80G  19.0G      4      0  37.2K  10.8K
-------------------  -----  -----  -----  -----  -----  -----
                       capacity     operations     bandwidth 
pool                 alloc   free   read  write   read  write
-------------------  -----  -----  -----  -----  -----  -----
august                770G   142G      1     38  9.59K   752K
  ada0p3.eli          770G   142G      1     38  9.59K   752K
cache                    -      -      -      -      -      -
  gpt/cache2-august  13.1G  1.34G      6      0  34.0K  3.25K
  gpt/cache3-august  5.53G  1.69G      4      0   322K      0
  gpt/cache1-august  9.80G  19.0G      5      0  36.0K  34.4K
-------------------  -----  -----  -----  -----  -----  -----
                       capacity     operations     bandwidth 
pool                 alloc   free   read  write   read  write
-------------------  -----  -----  -----  -----  -----  -----
august                770G   142G      2     46  18.0K   898K
  ada0p3.eli          770G   142G      2     46  18.0K   898K
cache                    -      -      -      -      -      -
  gpt/cache2-august  13.1G  1.34G      5      0   114K  8.15K
  gpt/cache3-august  5.53G  1.69G      4      0  79.2K  3.70K
  gpt/cache1-august  9.80G  19.0G      3      0  48.8K  1.20K
-------------------  -----  -----  -----  -----  -----  -----
^C
% date ; uptime
Thu 20 Jul 2023 19:20:18 BST
 7:20p.m.  up 15:28, 5 users, load averages: 3.43, 2.22, 1.78
% 

% zfs-stats -L

------------------------------------------------------------------------
ZFS Subsystem Report                            Thu Jul 20 19:21:30 2023
------------------------------------------------------------------------

L2 ARC Summary: (DEGRADED)
        Passed Headroom:                        0
        Tried Lock Failures:                    0
        IO In Progress:                         0
        Low Memory Aborts:                      1.01    k
        Free on Write:                          2.37    k
        Writes While Full:                      0
        R/W Clashes:                            5
        Bad Checksums:                          27.50   k
        IO Errors:                              0
        SPA Mismatch:                           0

L2 ARC Size: (Adaptive)                         27.90   GiB
        Decompressed Data Size:                 67.44   GiB
        Compression Factor:                     2.42
        Header Size:                    0.23%   159.70  MiB

L2 ARC Evicts:
        Lock Retries:                           12
        Upon Reading:                           0

L2 ARC Breakdown:                               1.69    m
        Hit Ratio:                      91.12%  1.54    m
        Miss Ratio:                     8.88%   150.15  k
        Feeds:                                  54.40   k

L2 ARC Buffer:
        Bytes Scanned:                          0       Bytes
        Buffer Iterations:                      0
        List Iterations:                        0
        NULL List Iterations:                   0

L2 ARC Writes:
        Writes Sent:                    100.00% 19.09   k

------------------------------------------------------------------------

% 

[neofusion]

Do you have a way of quantifying the performance boost?

I expect any prolonged usage will drain the endurance of those devices quickly.

[mercenary_sysadmin]

Agreed, particularly considering the lack of wear-leveling on thumb drives. The silver lining is that the write access patterns for an L2ARC feed should likely produce pretty even wear in the first place, but still!

[grahamperrin]

@neofusion

Do you have a way of quantifying the performance boost?

I rarely aim to quantify.

From a desktop environment use perspective: loosely speaking, anything that’s more than fifty percent better is easy to feel.

---

Re: https://wiki.freebsd.org/BootTime#tslog I’ll build a kernel with support for tslog(4) then perform boots with, and without, the cache devices.

When I last performed this type of comparison, around eighteen months ago, boot was around fourteen seconds faster.

[grahamperrin]

@grahamperrin

… loosely speaking, anything that’s more than fifty percent better is easy to feel. …

Boot times

106 seconds with L2ARC:

136 seconds without L2ARC:

% uname -aKU
FreeBSD mowa219-gjp4-8570p-freebsd 14.0-CURRENT FreeBSD 14.0-CURRENT amd64 1400093 #1 main-n264312-789df254cc9e-dirty: Tue Jul 25 09:49:23 BST 2023 grahamperrin@mowa219-gjp4-8570p-freebsd:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64 1400093 1400093
%

[mercenary_sysadmin]

Thank you very much for the quantification; it does matter.

[grahamperrin]

@neofusion

I expect any prolonged usage will drain the endurance of those devices quickly.

No doubt, however it’s like a zero-cost solution.

These are the types of device that people discard, or lose and don’t care to recover. Throwaway.

[grahamperrin]

@grahamperrin

Boot times

For anyone who wonders why these are so long: it’s a fairly old computer with a lot going on (a dock with various peripherals, including a display with a USB hub; various startup items that are not part of the base system; and so on).

The result of a recent probe:

• https://bsd-hardware.info/?probe=9f4f71236e