2 SAS SSD mirrored. Primary cache all/metadata, direct=1 still comes out ahead–wildly so.
I notice that the average latency is lower on direct=0, but still.
I have some other tests with rampup of 2m and runtime 10m, but I pulled these more recent ones as by this point I was prototyping what it could be.
I don’t understand how come direct=1 is so much faster. I mean, I figure writes should be nearly the same, but I thought direct=0 with primarycache=all vs direct=1 with primarycache=metadata, that direct=0 would come ahead on reads on all measures, but it doesn’t.
setup
zpool create -f -o ashift=12 -o autotrim=on -O atime=off -O recordsize=1MB \
-O xattr=sa -O special_small_blocks=0 -O primarycache="${primarycache}" \
"${POOL}" mirror sda sdb
zfs create "${POOL}"/SSD -o recordsize=1MB -o special_small_blocks=1MB
zfs create "${POOL}"/SSD/Database-16k -o recordsize=16kb -o special_small_blocks=16kb
fiojob
[global]
direct=${direct}
ioengine=io_uring
refill_buffers
end_fsync=1
time_based
runtime=2m
iodepth=120
group_reporting
[SSD-Database-16k]
new_group
description=Simulate database
# Force syncing
sync=1
directory=/tank/SSD/Database-16k
rw=randrw
rwmixread=80
# Set pareto to 80/20 rule
random_distribution=pareto:0.9517
blocksize=16k
filesize=2GB
nrfiles=1
openfiles=1
direct=1
SSD-Database-16k: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=io_uring, iodepth=120
fio-3.39
Starting 1 process
SSD-Database-16k: (groupid=0, jobs=1): err= 0: pid=26430: Sat Oct 24 19:56:45 2026
Description : [Simulate database]
read: IOPS=52.3k, BW=817MiB/s (857MB/s)(95.8GiB/120003msec)
slat (nsec): min=1021, max=934074, avg=2744.53, stdev=2309.41
clat (usec): min=85, max=72519, avg=1440.77, stdev=961.27
lat (usec): min=87, max=72522, avg=1443.51, stdev=961.28
clat percentiles (usec):
| 1.00th=[ 241], 5.00th=[ 318], 10.00th=[ 424], 20.00th=[ 676],
| 30.00th=[ 898], 40.00th=[ 1090], 50.00th=[ 1254], 60.00th=[ 1450],
| 70.00th=[ 1680], 80.00th=[ 2073], 90.00th=[ 2704], 95.00th=[ 3163],
| 99.00th=[ 4080], 99.50th=[ 4948], 99.90th=[ 8979], 99.95th=[ 9765],
| 99.99th=[12387]
bw ( KiB/s): min=119360, max=902848, per=100.00%, avg=836701.08, stdev=140788.01, samples=239
iops : min= 7460, max=56428, avg=52293.83, stdev=8799.25, samples=239
write: IOPS=13.1k, BW=204MiB/s (214MB/s)(23.9GiB/120003msec); 0 zone resets
slat (nsec): min=1120, max=891277, avg=3656.43, stdev=2826.52
clat (usec): min=536, max=147243, avg=3385.47, stdev=3847.81
lat (usec): min=539, max=147248, avg=3389.13, stdev=3847.86
clat percentiles (usec):
| 1.00th=[ 1631], 5.00th=[ 1909], 10.00th=[ 2073], 20.00th=[ 2278],
| 30.00th=[ 2474], 40.00th=[ 2704], 50.00th=[ 2933], 60.00th=[ 3195],
| 70.00th=[ 3523], 80.00th=[ 3884], 90.00th=[ 4424], 95.00th=[ 4948],
| 99.00th=[ 10028], 99.50th=[ 21627], 99.90th=[ 68682], 99.95th=[ 89654],
| 99.99th=[105382]
bw ( KiB/s): min=32544, max=224736, per=100.00%, avg=209236.67, stdev=35353.37, samples=239
iops : min= 2034, max=14046, avg=13077.30, stdev=2209.59, samples=239
lat (usec) : 100=0.01%, 250=1.14%, 500=9.29%, 750=8.15%, 1000=9.63%
lat (msec) : 2=36.10%, 4=31.29%, 10=4.16%, 20=0.12%, 50=0.07%
lat (msec) : 100=0.03%, 250=0.01%
cpu : usr=28.00%, sys=23.41%, ctx=3049925, majf=0, minf=25
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued rwts: total=6275590,1569385,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=120
Run status group 0 (all jobs):
READ: bw=817MiB/s (857MB/s), 817MiB/s-817MiB/s (857MB/s-857MB/s), io=95.8GiB (103GB), run=120003-120003msec
WRITE: bw=204MiB/s (214MB/s), 204MiB/s-204MiB/s (214MB/s-214MB/s), io=23.9GiB (25.7GB), run=120003-120003msec
direct=0
SSD-Database-16k: (g=0): rw=randrw, bs=(R) 16.0KiB-16.0KiB, (W) 16.0KiB-16.0KiB, (T) 16.0KiB-16.0KiB, ioengine=io_uring, iodepth=120
fio-3.39
Starting 1 process
SSD-Database-16k: Laying out IO file (1 file / 2048MiB)
SSD-Database-16k: (groupid=0, jobs=1): err= 0: pid=57770: Sat Oct 24 21:02:20 2026
Description : [Simulate database]
read: IOPS=11.6k, BW=182MiB/s (190MB/s)(21.3GiB/120042msec)
slat (nsec): min=940, max=973584, avg=2776.15, stdev=3031.02
clat (nsec): min=1403, max=5679.0k, avg=26932.29, stdev=50299.01
lat (usec): min=7, max=5681, avg=29.71, stdev=50.50
clat percentiles (usec):
| 1.00th=[ 14], 5.00th=[ 17], 10.00th=[ 18], 20.00th=[ 20],
| 30.00th=[ 21], 40.00th=[ 21], 50.00th=[ 22], 60.00th=[ 24],
| 70.00th=[ 25], 80.00th=[ 29], 90.00th=[ 37], 95.00th=[ 51],
| 99.00th=[ 94], 99.50th=[ 101], 99.90th=[ 125], 99.95th=[ 151],
| 99.99th=[ 2638]
bw ( KiB/s): min=120320, max=260768, per=100.00%, avg=186033.94, stdev=21642.58, samples=239
iops : min= 7520, max=16298, avg=11627.11, stdev=1352.66, samples=239
write: IOPS=2906, BW=45.4MiB/s (47.6MB/s)(5451MiB/120042msec); 0 zone resets
slat (nsec): min=1076, max=919089, avg=3044.60, stdev=2302.28
clat (usec): min=1148, max=235214, avg=41157.46, stdev=14379.19
lat (usec): min=1153, max=235216, avg=41160.50, stdev=14379.24
clat percentiles (msec):
| 1.00th=[ 32], 5.00th=[ 34], 10.00th=[ 36], 20.00th=[ 37],
| 30.00th=[ 39], 40.00th=[ 40], 50.00th=[ 41], 60.00th=[ 42],
| 70.00th=[ 42], 80.00th=[ 43], 90.00th=[ 44], 95.00th=[ 45],
| 99.00th=[ 65], 99.50th=[ 201], 99.90th=[ 211], 99.95th=[ 215],
| 99.99th=[ 234]
bw ( KiB/s): min=29568, max=59680, per=100.00%, avg=46520.50, stdev=5315.81, samples=239
iops : min= 1848, max= 3730, avg=2907.52, stdev=332.24, samples=239
lat (usec) : 2=0.01%, 4=0.01%, 10=0.04%, 20=24.14%, 50=51.73%
lat (usec) : 100=3.65%, 250=0.38%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=19.79%
lat (msec) : 100=0.04%, 250=0.18%
cpu : usr=7.99%, sys=7.16%, ctx=1676599, majf=0, minf=27
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued rwts: total=1394491,348845,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=120
Run status group 0 (all jobs):
READ: bw=182MiB/s (190MB/s), 182MiB/s-182MiB/s (190MB/s-190MB/s), io=21.3GiB (22.8GB), run=120042-120042msec
WRITE: bw=45.4MiB/s (47.6MB/s), 45.4MiB/s-45.4MiB/s (47.6MB/s-47.6MB/s), io=5451MiB (5715MB), run=120042-120042msec
As an aside, I had started testing originally with sync=always set to the dataset. I thought that it should be redundant with sync=1 set to the fio job, yet it seemed to slow it down. I don’t understand that either.
Thanks for sharing your phenomenal wealth of knowledge of ZFS btw.