Pool performance degrades over the span of a week

memnoch_proxy · October 14, 2026, 10:47pm

Q: Why does my zfs pool get slower about a after a week?

I tried looking at two different filesystems on the system: /var/tmp (LVM, ext4) and a web server directory from the zfs pool. I repeatedly checked both files so they should be in cache / ARC. If both of the files are getting served out of RAM, why is /var/tmp 275% faster?

Server
This is from a AMD EPYC 7232P 8-Core server with a 3-vdev raidz pool.

 $ grep Y /etc/os-release
PRETTY_NAME="CentOS Stream 9"

 $ uname -a
Linux ww4 5.14.0-366.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 14 23:37:14 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Download Times

 $ time scp ww4-san:/home/VHOSTS/candelatech.com/html/private/downloads/r5.5.4/LANforgeServer-5.5.4_MacOS-10.15.tar.gz /tmp
LANforgeServer-5.5.4_MacOS-10.15.tar.gz                                                                                                                                                                                                     100%  105MB  46.4MB/s   00:02

real    0m2.489s
user    0m0.079s
sys     0m0.239s

 $ time scp ww4-san:/var/tmp/LANforgeServer-5.5.4_MacOS-10.15.tar.gz /tmp
LANforgeServer-5.5.4_MacOS-10.15.tar.gz                                                                                                                                                                                                     100%  105MB 853.7MB/s   00:00    

real    0m0.334s
user    0m0.068s
sys     0m0.127s

 $ time scp ww4-san:/home/VHOSTS/candelatech.com/html/private/downloads/r5.5.4/LANforgeServer-5.5.4_MacOS-10.15.tar.gz /tmp
LANforgeServer-5.5.4_MacOS-10.15.tar.gz                                                                                                                                                                                                     100%  105MB  56.5MB/s   00:01    

real    0m2.087s
user    0m0.081s
sys     0m0.165s

 $ time scp ww4-san:/var/tmp/LANforgeServer-5.5.4_MacOS-10.15.tar.gz /tmp
LANforgeServer-5.5.4_MacOS-10.15.tar.gz                                                                                                                                                                                                     100%  105MB   1.0GB/s   00:00    

real    0m0.313s
user    0m0.072s
sys     0m0.102s

Pool Architecture

 $ zpool iostat -v
capacity operations bandwidth pool alloc   free   read  write   read  write

ww4-pool                                     21.1T  11.6T  3.67K    143  1.66G  2.17M
raidz1-0                                   7.03T  3.87T  1.22K     47   568M   740K
ata-WDC_WDS400T1R0A-68A4W0_222445800721      -      -    421     15   191M   247K
ata-WDC_WDS400T1R0A-68A4W0_22300C800097      -      -    420     16   190M   247K
ata-WDC_WDS400T1R0A-68A4W0_223667800002      -      -    413     15   187M   247K
raidz1-1                                   7.03T  3.88T  1.22K     47   566M   740K
ata-WDC_WDS400T1R0A-68A4W0_223667800284      -      -    416     16   188M   247K
ata-WDC_WDS400T1R0A-68A4W0_223667800464      -      -    421     15   192M   247K
ata-WDC_WDS400T1R0A-68A4W0_223667800466      -      -    409     15   186M   247K
raidz1-2                                   7.02T  3.88T  1.22K     47   567M   740K
ata-WDC_WDS400T1R0A-68A4W0_223667800467      -      -    416     15   189M   247K
ata-WDC_WDS400T1R0A-68A4W0_223667800468      -      -    418     15   190M   247K
ata-WDC_WDS400T1R0A-68A4W0_223667800500      -      -    416     15   189M   247K

Output of arcstat 30

    time  read  miss  miss%  dmis  dm%  pmis  pm%  mmis  mm%  size     c  avail
14:09:37   15K  1.6K     10  1.5K   10    98  100     3    0   48G   48G    18G
14:09:40  8.4K  1.5K     18  1.5K   18     4   93     2    0   48G   48G    18G
14:09:43   10K  1.6K     15  1.6K   14    13  100     0    0   48G   48G    18G
14:09:46   11K  1.6K     13  1.5K   13    12  100     1    0   47G   48G    18G
14:09:49  9.9K  1.6K     15  1.6K   15    13   19    19    0   47G   48G    18G
14:09:52  4.7K  1.7K     35  1.7K   35     3  100     0    0   47G   48G    18G
14:09:55  4.9K  1.7K     35  1.6K   34   140  100     2    0   48G   48G    18G
14:09:58  4.8K  1.7K     35  1.5K   33   169   82     0    0   47G   48G    18G
14:10:01   13K  1.6K     11  1.4K   10   179   99     2    0   48G   48G    18G
14:10:04   21K  1.6K      7  1.3K    6   281  100     0    0   48G   48G    18G
14:10:07  8.4K  1.6K     19  1.5K   17   157  100     0    0   48G   48G    18G
14:10:10   10K  1.5K     15  1.4K   14   133  100     1    0   48G   48G    18G
14:10:13   20K  1.6K      7  1.3K    6   276   99    18    0   48G   48G    18G
14:10:16   17K   886      5   647    3   239   99     1    0   48G   48G    18G
14:10:19  3.9K   909     23   891   23    18  100     4    0   48G   48G    18G
14:10:22  3.2K   898     27   868   27    29  100     1    0   48G   48G    18G
14:10:25  3.0K   966     32   963   32     3  100     1    0   48G   48G    18G

Modprobe.d/zfs.conf

options zfs zfs_arc_max=51539607552
options zfs zfs_resilver_min_time_ms=18000
options zfs zfs_arc_min=17179869184
options zfs zfs_vdev_scrub_min_active=24
options zfs zfs_vdev_scrub_max_active=64
options zfs zfs_vdev_sync_write_min_active=8
options zfs zfs_vdev_sync_write_max_active=32
options zfs zfs_vdev_sync_read_min_active=8
options zfs zfs_vdev_sync_read_max_active=32
options zfs zfs_vdev_async_read_min_active=8
options zfs zfs_vdev_async_read_max_active=32
options zfs zfs_vdev_async_write_min_active=8
options zfs zfs_vdev_async_write_max_active=32
options zfs zfs_txg_timeout=15
options zfs zfs_prefetch_disable=0
options zfs l2arc_write_max=524288000
options zfs l2arc_headroom=12
options zfs zfs_immediate_write_sz=262144

strace -ttt /var/tmp/thing.tar.gz

1792015559.847692 read(3, ""..., 32768) = 32768
1792015559.847763 read(3, ""..., 32768) = 32768
1792015559.847840 read(3, ""..., 32768) = 22646

about 0.071ms per read

strace -ttt /home/VHOSTS/candelatech.com/…/thing.tar.gz

1792015762.246942 read(3, ""..., 32768) = 32768
1792015762.249041 read(3, ""..., 32768) = 32768
1792015762.251628 read(3, ""..., 32768) = 32768
1792015762.253838 read(3, ""..., 32768) = 32768

about 2.587ms per read…
After reboot I’m reading about 2.5 - 7.1ms per read.

* How much does it matter that /var/tmp is on an NVMe, compared to just hitting fs cache?
* Is there anything I can tune?

mercenary_sysadmin · October 15, 2026, 12:14am

What’s the ashift on each vdev, and what’s the recordsize on that dataset? Keep in mind, you’re running 3-wide RAIDz1, so whatever your original recordsize is, you’re storing half that amount of data in each individual “chunk” of a block on each disk of the vdev it lands on.

So if you’re doing recordsize=128K, you’re reading and writing in 64K chunks.

There are a LOT of variables here that you haven’t eliminated. For one thing, you’re testing this with scp–copying across a network. So any network bottlenecks are going to impact what’s going on. You’re also saving that data to /tmp–which means any storage bottlenecks with writes to /tmp locally will affect what’s going on.

It also doesn’t help that I have no idea who large LANforgeServer-5.5.4_MacOS-10.15.tar.gz is, so I really don’t have a good way to understand how many blocks must be written, what level of throughput we’re actually seeing, etc etc ad infinitum.

Ideally, you want to be running fio with well-chosen arguments. I get that if you were familiar with doing that, you’d already be doing that… so for the moment, let’s start by at least doing everything on localhost on ww4-san itself, and NOT hobbling it with writes.

So let’s try a simple, but better isolated test. First:

ls -lh /var/tmp/LANforgeServer-5.5.4_MacOS-10.15.tar.gz
pv < /var/tmp/LANforgeServer-5.5.4_MacOS-10.15.tar.gz > /dev/null
pv < /var/tmp/LANforgeServer-5.5.4_MacOS-10.15.tar.gz > /dev/null

Then:

ls -lh /home/VHOSTS/candelatech.com/html/private/downloads/r5.5.4/LANforgeServer-5.5.4_MacOS-10.15.tar.gz
pv < ww4-san:/home/VHOSTS/candelatech.com/html/private/downloads/r5.5.4/LANforgeServer-5.5.4_MacOS-10.15.tar.gz > /dev/null
pv < ww4-san:/home/VHOSTS/candelatech.com/html/private/downloads/r5.5.4/LANforgeServer-5.5.4_MacOS-10.15.tar.gz > /dev/null

This is still a bit crude, but it at least lets me see the actual size of the file we’re dealing with, and gives me more assurance that I’m either looking at the difference between cached and uncached reads, or else looking at two cached reads in a row. Right now, I really have no idea what I’m looking at.

Don’t forget to let me know the recordsize and ashift values in play, also, please.

memnoch_proxy · October 15, 2026, 12:55am

ashift=12
recordsize = 1M
compression = zstd
LANforgeServer-5.5.4_MacOS-10.15.tar.gz = 106M
ww4-san is on a 10GbE interface

PV in /var/tmp:

105MiB 0:00:00 [1.18GiB/s]
105MiB 0:00:00 [4.84GiB/s]

jreynolds@ww4 /home/VHOSTS/candelatech.com/html/private/downloads/r5.5.4

pv < LANforgeServer-5.5.4_MacOS-10.15.tar.gz > /dev/null
105MiB 0:00:00 [ 135MiB/s] [================================>] 100%

jreynolds@ww4 /home/VHOSTS/candelatech.com/html/private/downloads/r5.5.4

pv < LANforgeServer-5.5.4_MacOS-10.15.tar.gz > /dev/null
105MiB 0:00:00 [ 131MiB/s] [================================>] 100%

AH – metadata –

jreynolds@ww4 /home/VHOSTS/candelatech.com/html/private/downloads/r5.5.4

Zget . all | grep -i cache
ww4-pool/www-priv-downloads/r5.5.4 primarycache metadata inherited from ww4-pool
ww4-pool/www-priv-downloads/r5.5.4 secondarycache all default

jreynolds@ww4 /home/VHOSTS/candelatech.com/html/private/downloads/r5.5.4

cd ../r5.5.3

jreynolds@ww4 /home/VHOSTS/candelatech.com/html/private/downloads/r5.5.3

Zget . all | grep -i cache
ww4-pool/www-priv-downloads/r5.5.3 primarycache all local
ww4-pool/www-priv-downloads/r5.5.3 secondarycache all default

jreynolds@ww4 /home/VHOSTS/candelatech.com/html/private/downloads/r5.5.3

pv < LANforgeServer-5.5.3_MacOS-10.15.tar.gz > /dev/null 105MiB 0:00:00 [1.26GiB/s] [================================>] 100%

jreynolds@ww4 /home/VHOSTS/candelatech.com/html/private/downloads/r5.5.3

pv < LANforgeServer-5.5.3_MacOS-10.15.tar.gz > /dev/null
105MiB 0:00:00 [5.76GiB/s] [================================>] 100%

Oh, this is better:

jreynolds@ww4 /home/VHOSTS/candelatech.com/html/private/downloads/r5.5.4

pv < LANforgeServer-5.5.4_MacOS-10.15.tar.gz > /dev/null
105MiB 0:00:00 [1.89GiB/s] [===========================================

jreynolds@ww4 /home/VHOSTS/candelatech.com/html/private/downloads/r5.5.4

pv < LANforgeServer-5.5.4_MacOS-10.15.tar.gz > /dev/null
105MiB 0:00:00 [7.81GiB/s] [===========================================

And over the network, this is likewise reflected:

 $ time scp ww4-san:/home/VHOSTS/candelatech.com/html/private/downloads/r5.5.4/LANforgeServer-5.5.4_MacOS-10.15.tar.gz /tmp
  LANforgeServer-5.5.4_MacOS-10.15.tar.gz  100%  105MB 919.7MB/s   00:00


  real    0m0.421s
  user    0m0.066s
  sys     0m0.127s

 $ time scp ww4-san:/var/tmp/LANforgeServer-5.5.4_MacOS-10.15.tar.gz /tmp
  LANforgeServer-5.5.4_MacOS-10.15.tar.gz  100%  105MB   1.0GB/s   00:00

  real    0m0.324s
  user    0m0.073s
  sys     0m0.106s

memnoch_proxy · October 15, 2026, 1:11am

Thanks @mercenary_sysadmin !

mercenary_sysadmin · October 15, 2026, 1:33am

I’m seeing a couple of things there. One is obvious–it took a few more tries spewing all those blocks through a pipe before they stuck in ARC, which implies that you have a lot of blocks already filling the ARC that it “thinks” (due to high hit rate in those blocks) are too valuable to casually displace.

Once you’d hit the same blocks in your tar.gz often enough, they looked more valuable than what was already in ARC, so they stuck, and your throughput rocketed up until they were ALL in the ARC.

The second thing, though, is that 135MiB/sec is pretty unimpressive from even a single 3-wide Z1, let alone three 3-wide Z1s. Even a rust pool with that topology would normally outperform that by a country mile… Which tells me that ZSTD is probably not a good choice for that workload.

To test my assumption, create a new dataset, make sure compression is off (or ZLE) on that dataset, copy your tarball in there, then try the pv tests again. If you start out already at more than 1GiB/sec, you’re probably reading a lot of those blocks from cache, so you’ll need to either reboot the system or export and import the pool to try again with a clean ARC.

Of course, the other possibility is that the server is heavily loaded and most of the available IOPS went to other people’s workloads, starving your test workload. I don’t really think that’s the case, though, because your runs were so consistent from one to the next until the ARC started catching (and caching, lol) blocks for you.

You could also check to make sure your free space fragmentation isn’t horrendous, but my money is on the ZSTD bottlenecking you on CPU.

memnoch_proxy · October 15, 2026, 4:03am

That is a reasonable suggestion about zstd not being ideal, and I’m open to trying with an uncompressed filesystem as a test. Each software version merits it’s own filesystem so as to limit snapshot focus. The majority of the files in the filesystem are gzip, bzip2, zip, or xz compressed. I had assumed that a majority of those blocks would be considered uncompressible off the bat–the compression was for the wee stuff that would benefit, and the possible memory savings from caching compressed records in the ARC. How much of this view is naive assumption?

memnoch_proxy · October 15, 2026, 4:10am

It appears that the overall compressratio for the last three software releases is not better than 1.02. This would make sense with the nature of the content. I don’t understand why there would be a decompression penalty for content like this however.

zpool fragmentation is currently 17%
pool snapshots at 14649

htop:

ARC: 32.0G Used:32.0G MFU:30.6G MRU:595M Anon:8.23M Hdr:100 ARC: 30.5G Uncompressed, 29.5G Compressed, 1.03:1 Ratio
Disk IO: 100.0% read: 2.08GiB/s write: 4KiB/s               Network: rx: 358KiB/s tx: 62.0MiB/s (5284/43080 pkts/s)
    PID USER       PRI  NI  VIRT   RES   SHR S  CPU%▽MEM%   TIME+  Command (merged)
   1103 root         0 -20     0     0     0 S  15.1  0.0 31:45.68 z_rd_int_0
   1104 root         0 -20     0     0     0 S  15.1  0.0 31:44.24 z_rd_int_1
   1105 root         0 -20     0     0     0 S  14.8  0.0 31:44.12 z_rd_int_2
  51544 apache      20   0 2189M 81732 15640 S   3.0  0.1  1:22.11 /usr/sbin/httpd -DFOREGROUND
 610274 apache      20   0 2193M 52692 10612 D   2.0  0.1  0:24.04 /usr/sbin/httpd -DFOREGROUND
 610288 apache      20   0 2193M 52692 10612 S   2.0  0.1  0:40.40 /usr/sbin/httpd -DFOREGROUND

mercenary_sysadmin · October 15, 2026, 12:30pm

There is always a decompression penalty on already-fully-compressed data, it’s just not as bad as it would be if OpenZFS weren’t trying to detect incompressible blocks at all.

And ZSTD is a lot more punishing when it comes to unsuitable data than LZ4 was/is. Personally, I still use LZ4 on my own pools for that exact reason. I think ZSTD is typically better considered as an improved replacement for gzip than as an improved replacement for LZ4. In my personal testing, even the lightest-weight settings for ZSTD still leave it frequently punishing unsuitable workloads noticeably more than LZ4 does.

Still, don’t take my word for this until you’ve actually tested on an uncompressed (or ZLE) dataset and seen the kind of performance improvement that I suspect you’re going to see.