Synthetic Single Thread Random Read (4K) Problem

spacerunner5 · October 10, 2025, 11:24am

Background

While setting up a new machine, I did some initial benchmarking on NVMe drives. I noticed a huge degradation on 4K random reads as soon as ZFS is involved. I am interested in getting as much IOPS as possible, also on non-cached (meta-) data / files.

I went all the way down bypassing as much ZFS goodies as possible, just to see raw IOPS performance. No tuning helped. Finally, I ran benchmarks against a single-vdev-pool sitting on a ramdisk. Here it gets interesting.

System Specs:

Debian (Proxmox Kernel 6.14.11-3-pve, zfs-2.3.4-pve1)
128G RAM
Epyc 7003 processor
ECC
Machine basically idle, bare metal, no guests, fresh install

To Reproduce

(1) Create a RAM disk and benchmark for 4K rand read

mkdir /media/ramdisk

mount -t ramfs ramfs /media/ramdisk

fio --ioengine=libaio --name=io --rw=randread --bs=4k --iodepth=64 --ramp_time=2 --runtime=20 --filename=/media/ramdisk/testfile --size=512M  --time_based=1 --sync=0 --direct=0 --numjobs=1 --buffered=1

io: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.39
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=1380MiB/s][r=353k IOPS][eta 00m:00s]
io: (groupid=0, jobs=1): err= 0: pid=13610: Fri Oct 10 11:57:48 2025
  read: IOPS=352k, BW=1374MiB/s (1441MB/s)(26.8GiB/20001msec)
    slat (nsec): min=1380, max=1106.4k, avg=1739.30, stdev=694.54
    clat (usec): min=2, max=1520, avg=179.96, stdev= 7.76
     lat (usec): min=4, max=1525, avg=181.70, stdev= 7.81
    clat percentiles (usec):
     |  1.00th=[  174],  5.00th=[  176], 10.00th=[  176], 20.00th=[  178],
     | 30.00th=[  178], 40.00th=[  180], 50.00th=[  180], 60.00th=[  182],
     | 70.00th=[  182], 80.00th=[  182], 90.00th=[  184], 95.00th=[  186],
     | 99.00th=[  190], 99.50th=[  202], 99.90th=[  281], 99.95th=[  285],
     | 99.99th=[  338]
   bw (  MiB/s): min= 1346, max= 1386, per=100.00%, avg=1374.46, stdev= 8.20, samples=40
   iops        : min=344794, max=355034, avg=351861.38, stdev=2099.21, samples=40
  lat (usec)   : 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
  lat (usec)   : 250=99.86%, 500=0.14%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%
  cpu          : usr=35.40%, sys=64.56%, ctx=181, majf=0, minf=37
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=7035380,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=1374MiB/s (1441MB/s), 1374MiB/s-1374MiB/s (1441MB/s-1441MB/s), io=26.8GiB (28.8GB), run=20001-20001msec

RESULT: FIO now maxes out 1 CPU core, delivering 353k IOPS.

(2) create a ZFS pool on the RAM disk

truncate -s 10G /media/ramdisk/zfsfile

zpool create -o ashift=12 -f rampool /media/ramdisk/zfsfile

(3) create a dataset and set some Testing (!) properties

dataset=testdataset

zfs create rampool/$dataset

zfs set sync=disabled rampool/$dataset
zfs set atime=off rampool/$dataset
zfs set compression=off rampool/$dataset
zfs set recordsize=4k rampool/$dataset
zfs set relatime=off rampool/$dataset
zfs set primarycache=none rampool/$dataset
zfs set secondarycache=none rampool/$dataset
zfs set prefetch=none rampool/$dataset
zfs set direct=always rampool/$dataset
zfs set xattr=off rampool/$dataset

umount /rampool/$dataset
zfs mount rampool/$dataset

(4) benchmarking ZFS

fio --ioengine=libaio --name=io --rw=randread --bs=4k --iodepth=64 --ramp_time=2 --runtime=20 --filename=/rampool/testdataset/testfile --size=512M  --time_based=1 --sync=0 --direct=1 --numjobs=1 --buffered=0

io: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.39
Starting 1 process
io: Laying out IO file (1 file / 512MiB)
Jobs: 1 (f=1): [r(1)][100.0%][r=32.6MiB/s][r=8354 IOPS][eta 00m:00s]
io: (groupid=0, jobs=1): err= 0: pid=16610: Fri Oct 10 12:05:45 2025
  read: IOPS=7967, BW=31.1MiB/s (32.6MB/s)(623MiB/20001msec)
    slat (usec): min=4, max=35962, avg=123.87, stdev=321.42
    clat (usec): min=2, max=50697, avg=7906.08, stdev=3495.07
     lat (usec): min=125, max=50891, avg=8029.93, stdev=3529.03
    clat percentiles (usec):
     |  1.00th=[ 4817],  5.00th=[ 5080], 10.00th=[ 5342], 20.00th=[ 7046],
     | 30.00th=[ 7439], 40.00th=[ 7504], 50.00th=[ 7570], 60.00th=[ 7832],
     | 70.00th=[ 8455], 80.00th=[ 8586], 90.00th=[ 8586], 95.00th=[ 8979],
     | 99.00th=[31327], 99.50th=[37487], 99.90th=[45351], 99.95th=[46924],
     | 99.99th=[49021]
   bw (  KiB/s): min=25704, max=48072, per=99.97%, avg=31874.43, stdev=6049.68, samples=40
   iops        : min= 6426, max=12018, avg=7968.58, stdev=1512.39, samples=40
  lat (usec)   : 4=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=95.93%, 20=2.87%, 50=1.21%
  lat (msec)   : 100=0.01%
  cpu          : usr=1.50%, sys=29.70%, ctx=442497, majf=0, minf=37
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=159354,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=31.1MiB/s (32.6MB/s), 31.1MiB/s-31.1MiB/s (32.6MB/s-32.6MB/s), io=623MiB (653MB), run=20001-20001msec

RESULT: 8k IOPS, no process nearly maxes out the CPU. That’s 40 times less!

Observations while ZFS benchmark running:

zpool iostat rampool -qvy 1 30

                            capacity     operations     bandwidth    syncq_read    syncq_write   asyncq_read  asyncq_write   scrubq_read   trimq_write  rebuildq_write
pool                      alloc   free   read  write   read  write   pend  activ   pend  activ   pend  activ   pend  activ   pend  activ   pend  activ   pend  activ
------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
rampool                   2.01G  7.49G  23.6K      0   629M      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0
  /media/ramdisk/zfsfile  2.01G  7.49G  23.6K      0   629M      0      0      0      0      0      0      0      0      0      0      0      0      0      0      0
------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----

zpool iostat rampool -qvl 3 30

                            capacity     operations     bandwidth    total_wait     disk_wait    syncq_wait    asyncq_wait  scrub   trim  rebuild   syncq_read    syncq_write   asyncq_read  asyncq_write   scrubq_read   trimq_write  rebuildq_write
pool                      alloc   free   read  write   read  write   read  write   read  write   read  write   read  write   wait   wait   wait   pend  activ   pend  activ   pend  activ   pend  activ   pend  activ   pend  activ   pend  activ
------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
rampool                   2.01G  7.49G  19.5K      0   521M      0   19us      -   19us      -  642ns      -      -      -      -      -      -      0      0      0      0      0      0      0      0      0      0      0      0      0      0
  /media/ramdisk/zfsfile  2.01G  7.49G  19.5K      0   521M      0   19us      -   19us      -  642ns      -      -      -      -      -      -      0      0      0      0      0      0      0      0      0      0      0      0      0      0
------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----


zpool iostat rampool -ry 3 30

rampool       sync_read    sync_write    async_read    async_write      scrub         trim         rebuild
req_size      ind    agg    ind    agg    ind    agg    ind    agg    ind    agg    ind    agg    ind    agg
----------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
512             0      0      0      0      0      0      0      0      0      0      0      0      0      0
1K              0      0      0      0      0      0      0      0      0      0      0      0      0      0
2K              0      0      0      0      0      0      0      0      0      0      0      0      0      0
4K          6.85K      0      0      0      0      0      0      0      0      0      0      0      0      0
8K              0      0      0      0      0      0      0      0      0      0      0      0      0      0
16K         6.85K      0      0      0      0      0      0      0      0      0      0      0      0      0
32K         6.85K      0      0      0      0      0      0      0      0      0      0      0      0      0
64K             0      0      0      0      0      0      0      0      0      0      0      0      0      0
128K            0      0      0      0      0      0      0      0      0      0      0      0      0      0
256K            0      0      0      0      0      0      0      0      0      0      0      0      0      0
512K            0      0      0      0      0      0      0      0      0      0      0      0      0      0
1M              0      0      0      0      0      0      0      0      0      0      0      0      0      0
2M              0      0      0      0      0      0      0      0      0      0      0      0      0      0
4M              0      0      0      0      0      0      0      0      0      0      0      0      0      0
8M              0      0      0      0      0      0      0      0      0      0      0      0      0      0
16M             0      0      0      0      0      0      0      0      0      0      0      0      0      0

IOTOP

Total DISK READ:       472.23 M/s | Total DISK WRITE:        56.22 K/s
Current DISK READ:       0.00 B/s | Current DISK WRITE:       3.27 M/s
    TID  PRIO  USER    DISK READ>  DISK WRITE    COMMAND
  26468 be/4 root      472.23 M/s    0.00 B/s fio --ioengine=libaio --name=io --rw=randread --bs=4k --iodepth=1 --ramp_time=2 --runtime=20 --filename=/rampool/testdataset/testfile --size=4k --time_based=1 --sync=0 --direct=0 --numjobs=1 --buffered=1
      1 be/4 root        0.00 B/s    0.00 B/s init
      2 be/4 root        0.00 B/s    0.00 B/s [kthreadd]
      3 be/4 root        0.00 B/s    0.00 B/s [pool_workqueue_release]
      4 be/0 root        0.00 B/s    0.00 B/s [kworker/R-rcu_gp]
      5 be/0 root        0.00 B/s    0.00 B/s [kworker/R-sync_wq]
      6 be/0 root        0.00 B/s    0.00 B/s [kworker/R-kvfree_rcu_reclaim]
      7 be/0 root        0.00 B/s    0.00 B/s [kworker/R-slub_flushwq]
      8 be/0 root        0.00 B/s    0.00 B/s [kworker/R-netns]
      ...

Some additional notes:

The results translate when going on NVMe instead of a RAM disk. 200k raw/native, 3k on ZFS. Even worse: Same on stripped mirrors – no way to pass the 3k.
As seen in both zpool iostsat and iotop, FIO read bandwidth (actual read) is a magnitude (~500M/s) higher than the printed IOPs / bandwidth result. Whats going on here?
please note the dataset properties set for testing – especially cache and recordsize.
setting primarycache=metadata delivers ~30k IOPS but that’s not the scope of this test, just for comparison.

Question:

Where could that HUGE overhead come from? Is FIO / the query wrong? It’s a bit unlikely since it brings the expected results on the raw (RAM) volume without ZFS involved.

Anything that can be done / how to diagnose further?

denvercoder9 · October 13, 2025, 7:42pm

Admittedly, I am out of my element with this, but I will take a shot.

Comparing your fio runs, the CPU’s “ctx” values are very different. I think that is context switching.

Reading the following issue, it sounds like “primarycache=metadata” is effectively broken, and that ZFS requires “primarycache=all” to go high performance:

github.com/openzfs/zfs

ZFS utterly Slow on NVME (BTRFS vs ZFS)

opened 07:11PM - 27 Jan 25 UTC

closed 08:20PM - 20 Mar 25 UTC

Ramalama2

Type: Defect

This is maybe known to everyone, but something has to change because ZFS gets mo…re and more unusable on fast NVME Drives. ### System information Distribution Name | Proxmox 8.3.3 (Debian 12) Kernel Version | 6.8 / 6.11 Architecture | x64 OpenZFS Version | 2.2.7 CPU: Genoa 9374F Memory: 12x 64GB Dimms (768GB Total) Drives: 8x Micron 7450 Max 3,2TB / 2x Micron 7500 Max 3,2TB I made a lot of Benchmarks, FIO and complex MySQL Queries on ZFS / EXT4 & BTRFS Tested MariaDB version: 11.6 Total Database Space used: 446 GB FIO Command: `fio --name=randread --ioengine=libaio --iodepth=32 --rw=randread --bs=4k --direct=1 --size=4G --numjobs=6 --runtime=60 --time_based=1 --group_reporting --filename=/var/lib/mysql/testfile` Query Command: `Complex Select Query with Left Joints, Inner Joins, Group by, from multiple Databases.` ----- MariaDB 11.6 Results: **I tested 6 Queries, they are all different but "Query 1" and "Query 3" are hard to Cache, so i would look only to these 2.** After the Startup of the VM, each Query was run only 3x and only the third Result is measured. (The First Result on BTRFS vs ZFS, is even a lot more painfull, btrfs takes on first time after startup ~5s, while zfs takes over 15 minutes!) <html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns="http://www.w3.org/TR/REC-html40"> <body link="#0563C1" vlink="#954F72"> MySQL Queries: | BTRFS 1x nvme (default) | BTRFS (Raid0) 2x nvme | EXT4 1x nvme | ZFS 1x nvme (default) | ZFS 1x nvme (tuned) | ZFS (Raid10) 8x nvme (tuned) -- | -- | -- | -- | -- | -- | -- Query 1: | 3,823 | 3,778 | 3,749 | 629,239 | 158,180 | 162,816 Query 2: | 0,023 | 0,022 | 0,028 | 0,028 | 0,025 | 0,025 Query 3: | 0,329 | 0,323 | 0,332 | 74,994 | 41,417 | 42,032 Query 4: | 0,030 | 0,028 | 0,034 | 0,034 | 0,030 | 0,030 Query 5: | 0,022 | 0,022 | 0,028 | 0,029 | 0,025 | 0,025 Query 6: | 0,023 | 0,022 | 0,028 | 0,028 | 0,025 | 0,025 Total: | 4,25s | 4,195s | 4,199s | 704,352s | 199,702s | 204,953s </body> </html> ---- FIO Results (Paired for the MySQL-Benchmarks above): <details> <summary>ZFS-Default means:</summary> ```plaintext primarycache=metadata Everything else default ``` </details> <details> <summary>ZFS-Tuned means:</summary> ```plaintext primarycache=metadata recordsize=16k atime=off xattr=sa logbias=throughput dnodesize=auto redundant_metadata=most Everything else default ``` </details> <details> <summary>ZFS-Default (single, 1x 7450, 7,9k IOPS, 20ms latency)</summary> ```plaintext Jobs: 4 (f=4): [r(4)][100.0%][r=30.9MiB/s][r=7915 IOPS][eta 00m:00s] randread: (groupid=0, jobs=4): err= 0: pid=1284: Wed Jan 22 22:19:04 2025 read: IOPS=7914, BW=30.9MiB/s (32.4MB/s)(1855MiB/60001msec) slat (usec): min=3, max=2406, avg=504.25, stdev=234.26 clat (nsec): min=1773, max=26833k, avg=15665600.77, stdev=1402757.85 lat (usec): min=543, max=27441, avg=16169.85, stdev=1425.07 clat percentiles (usec): | 1.00th=[12387], 5.00th=[13304], 10.00th=[13960], 20.00th=[14484], | 30.00th=[15008], 40.00th=[15401], 50.00th=[15664], 60.00th=[16057], | 70.00th=[16319], 80.00th=[16909], 90.00th=[17433], 95.00th=[17695], | 99.00th=[19006], 99.50th=[20055], 99.90th=[21890], 99.95th=[22676], | 99.99th=[24773] bw ( KiB/s): min=29584, max=32920, per=100.00%, avg=31658.49, stdev=129.42, samples=476 iops : min= 7396, max= 8230, avg=7914.62, stdev=32.35, samples=476 lat (usec) : 2=0.01%, 4=0.01%, 750=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.02%, 20=99.48%, 50=0.49% cpu : usr=0.20%, sys=3.38%, ctx=394391, majf=14, minf=433 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0% issued rwts: total=474864,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=32 Run status group 0 (all jobs): READ: bw=30.9MiB/s (32.4MB/s), 30.9MiB/s-30.9MiB/s (32.4MB/s-32.4MB/s), io=1855MiB (1945MB), run=60001-60001msec ``` </details> <details> <summary>ZFS-Tuned (single, 1x 7450, 27,7k IOPS, 10ms latency)</summary> ```plaintext Jobs: 4 (f=4): [r(4)][100.0%][r=108MiB/s][r=27.7k IOPS][eta 00m:00s] randread: (groupid=0, jobs=4): err= 0: pid=1054: Wed Jan 22 23:36:47 2025 read: IOPS=27.6k, BW=108MiB/s (113MB/s)(6460MiB/60001msec) slat (usec): min=3, max=3754, avg=144.08, stdev=74.25 clat (nsec): min=1733, max=8822.9k, avg=4499481.31, stdev=449415.46 lat (usec): min=6, max=9019, avg=4643.56, stdev=457.49 clat percentiles (usec): | 1.00th=[ 3326], 5.00th=[ 3720], 10.00th=[ 3916], 20.00th=[ 4146], | 30.00th=[ 4293], 40.00th=[ 4424], 50.00th=[ 4555], 60.00th=[ 4621], | 70.00th=[ 4752], 80.00th=[ 4883], 90.00th=[ 5080], 95.00th=[ 5211], | 99.00th=[ 5407], 99.50th=[ 5538], 99.90th=[ 5800], 99.95th=[ 6259], | 99.99th=[ 7439] bw ( KiB/s): min=105912, max=129912, per=100.00%, avg=110265.88, stdev=987.47, samples=476 iops : min=26478, max=32478, avg=27566.47, stdev=246.87, samples=476 lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 250=0.01%, 500=0.01% lat (usec) : 750=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 4=13.15%, 10=86.85% cpu : usr=0.60%, sys=10.79%, ctx=1299849, majf=14, minf=491 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0% issued rwts: total=1653684,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=32 Run status group 0 (all jobs): READ: bw=108MiB/s (113MB/s), 108MiB/s-108MiB/s (113MB/s-113MB/s), io=6460MiB (6773MB), run=60001-60001msec ``` </details> <details> <summary>ZFS-Tuned (Raid-10, 8x 7450, 26,7k IOPS, 10ms latency)</summary> ```plaintext Jobs: 4 (f=4): [r(4)][100.0%][r=104MiB/s][r=26.7k IOPS][eta 00m:00s] randread: (groupid=0, jobs=4): err= 0: pid=1057: Wed Jan 22 22:26:15 2025 read: IOPS=26.5k, BW=103MiB/s (108MB/s)(6204MiB/60001msec) slat (usec): min=3, max=2108, avg=150.01, stdev=74.80 clat (nsec): min=1871, max=7816.5k, avg=4684713.57, stdev=424597.41 lat (usec): min=182, max=7998, avg=4834.73, stdev=431.56 clat percentiles (usec): | 1.00th=[ 3654], 5.00th=[ 3982], 10.00th=[ 4146], 20.00th=[ 4359], | 30.00th=[ 4490], 40.00th=[ 4621], 50.00th=[ 4686], 60.00th=[ 4817], | 70.00th=[ 4948], 80.00th=[ 5014], 90.00th=[ 5211], 95.00th=[ 5342], | 99.00th=[ 5604], 99.50th=[ 5669], 99.90th=[ 5932], 99.95th=[ 6128], | 99.99th=[ 6980] bw ( KiB/s): min=102320, max=109584, per=100.00%, avg=105884.57, stdev=374.69, samples=476 iops : min=25580, max=27396, avg=26471.14, stdev=93.67, samples=476 lat (usec) : 2=0.01%, 4=0.01%, 250=0.01%, 500=0.01%, 750=0.01% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=5.81%, 10=94.19% cpu : usr=0.59%, sys=11.38%, ctx=1259655, majf=13, minf=276 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0% issued rwts: total=1588287,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=32 Run status group 0 (all jobs): READ: bw=103MiB/s (108MB/s), 103MiB/s-103MiB/s (108MB/s-108MB/s), io=6204MiB (6506MB), run=60001-60001msec ``` </details> <details> <summary>EXT4 (single, 1x 7450, 703k IOPS, 250µs latency)</summary> ```plaintext Jobs: 4 (f=4): [r(4)][100.0%][r=2747MiB/s][r=703k IOPS][eta 00m:00s] randread: (groupid=0, jobs=4): err= 0: pid=1059: Wed Jan 22 17:23:54 2025 read: IOPS=701k, BW=2739MiB/s (2872MB/s)(160GiB/60002msec) slat (usec): min=2, max=4011, avg= 3.49, stdev= 2.43 clat (usec): min=5, max=4174, avg=178.80, stdev=62.09 lat (usec): min=8, max=4178, avg=182.29, stdev=61.89 clat percentiles (usec): | 1.00th=[ 48], 5.00th=[ 111], 10.00th=[ 125], 20.00th=[ 143], | 30.00th=[ 155], 40.00th=[ 163], 50.00th=[ 172], 60.00th=[ 180], | 70.00th=[ 192], 80.00th=[ 212], 90.00th=[ 245], 95.00th=[ 269], | 99.00th=[ 330], 99.50th=[ 363], 99.90th=[ 766], 99.95th=[ 988], | 99.99th=[ 1532] bw ( MiB/s): min= 2423, max= 3070, per=100.00%, avg=2740.79, stdev=28.37, samples=476 iops : min=620500, max=786132, avg=701642.39, stdev=7263.04, samples=476 lat (usec) : 10=0.01%, 20=0.03%, 50=1.09%, 100=2.42%, 250=87.56% lat (usec) : 500=8.69%, 750=0.10%, 1000=0.06% lat (msec) : 2=0.04%, 4=0.01%, 10=0.01% cpu : usr=10.38%, sys=71.87%, ctx=661563, majf=17, minf=275 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0% issued rwts: total=42069353,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=32 Run status group 0 (all jobs): READ: bw=2739MiB/s (2872MB/s), 2739MiB/s-2739MiB/s (2872MB/s-2872MB/s), io=160GiB (172GB), run=60002-60002msec Disk stats (read/write): loop0: ios=42008706/21, sectors=336069648/432, merge=0/33, ticks=3829799/8, in_queue=3829806, util=100.00% ``` </details> <details> <summary>BTRFS (single, 1x 7450, 673k IOPS, 250µs latency)</summary> ```plaintext Jobs: 4 (f=4): [r(4)][100.0%][r=2629MiB/s][r=673k IOPS][eta 00m:00s] randread: (groupid=0, jobs=4): err= 0: pid=1058: Wed Jan 22 20:06:34 2025 read: IOPS=676k, BW=2643MiB/s (2771MB/s)(155GiB/60002msec) slat (usec): min=2, max=6080, avg= 3.97, stdev= 4.51 clat (usec): min=9, max=6426, avg=184.95, stdev=56.29 lat (usec): min=12, max=6429, avg=188.93, stdev=56.24 clat percentiles (usec): | 1.00th=[ 105], 5.00th=[ 133], 10.00th=[ 143], 20.00th=[ 159], | 30.00th=[ 169], 40.00th=[ 176], 50.00th=[ 180], 60.00th=[ 184], | 70.00th=[ 188], 80.00th=[ 204], 90.00th=[ 235], 95.00th=[ 260], | 99.00th=[ 318], 99.50th=[ 359], 99.90th=[ 490], 99.95th=[ 660], | 99.99th=[ 2311] bw ( MiB/s): min= 2323, max= 2786, per=100.00%, avg=2645.18, stdev=20.80, samples=476 iops : min=594816, max=713440, avg=677164.82, stdev=5324.43, samples=476 lat (usec) : 10=0.01%, 20=0.01%, 50=0.08%, 100=0.84%, 250=92.45% lat (usec) : 500=6.54%, 750=0.05%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01% cpu : usr=9.21%, sys=79.07%, ctx=431313, majf=0, minf=419 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0% issued rwts: total=40591264,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=32 Run status group 0 (all jobs): READ: bw=2643MiB/s (2771MB/s), 2643MiB/s-2643MiB/s (2771MB/s-2771MB/s), io=155GiB (166GB), run=60002-60002msec Disk stats (read/write): loop1: ios=40533616/22, sectors=324268936/432, merge=0/32, ticks=2531160/3, in_queue=2531164, util=99.95% ``` </details> <details> <summary>BTRFS (Raid-0, 2x 7500, 642k IOPS, 250µs latency)</summary> ```plaintext Jobs: 3 (f=0): [f(1),_(1),f(2)][100.0%][r=2506MiB/s][r=642k IOPS][eta 00m:00s] randread: (groupid=0, jobs=4): err= 0: pid=1132: Mon Jan 27 13:42:20 2025 read: IOPS=660k, BW=2578MiB/s (2703MB/s)(151GiB/60002msec) slat (usec): min=2, max=7019, avg= 3.87, stdev= 3.76 clat (usec): min=10, max=7185, avg=189.80, stdev=65.71 lat (usec): min=13, max=7188, avg=193.67, stdev=65.52 clat percentiles (usec): | 1.00th=[ 110], 5.00th=[ 129], 10.00th=[ 145], 20.00th=[ 161], | 30.00th=[ 167], 40.00th=[ 172], 50.00th=[ 176], 60.00th=[ 180], | 70.00th=[ 186], 80.00th=[ 217], 90.00th=[ 265], 95.00th=[ 297], | 99.00th=[ 371], 99.50th=[ 412], 99.90th=[ 578], 99.95th=[ 816], | 99.99th=[ 2180] bw ( MiB/s): min= 2267, max= 2796, per=100.00%, avg=2579.29, stdev=28.73, samples=476 iops : min=580424, max=715926, avg=660299.14, stdev=7353.97, samples=476 lat (usec) : 20=0.01%, 50=0.14%, 100=0.70%, 250=85.98%, 500=12.99% lat (usec) : 750=0.13%, 1000=0.03% lat (msec) : 2=0.02%, 4=0.01%, 10=0.01% cpu : usr=9.08%, sys=75.75%, ctx=367562, majf=0, minf=1286 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0% issued rwts: total=39593164,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=32 Run status group 0 (all jobs): READ: bw=2578MiB/s (2703MB/s), 2578MiB/s-2578MiB/s (2703MB/s-2703MB/s), io=151GiB (162GB), run=60002-60002msec Disk stats (read/write): loop1: ios=39448612/25, sectors=315588896/624, merge=0/53, ticks=2908870/6, in_queue=2908877, util=99.90% ``` </details> ---- ZFS is most horrible on Databases especially, on General-Usage/Filetransfer and so on, ZFS is almost on par with BTRFS. In the end, on Mixed Environments, BTRFS is a much better Choice for NVME-Drives.

Again, if I am on the wrong track, I apologize in advance.

mercenary_sysadmin · October 13, 2025, 8:59pm

I would argue with this. Primarycache=metadata is not a good idea at this particular scale because it necessarily introduces additional compute latency while you figure out whether something is data or metadata and therefore whether you should or shouldn’t cache it.

However, I would say that the vast majority of real world ZFS use happens further away from the kind of compute and bus saturation that leaves no room for adding additional logic.