Docker performance influenced by ZFS dataset

Hi,

I recently migrated CI machines to ZFS (ZFS on Root, ZFS is the only filesystem overall) and noticed that CI pipelines are taking more time to finish.
I tracked down the performance degradation to docker tasks:

  • docker daemon is using storage: overlay2
  • docker’s data-root is set to default path - /var/lib/docker
  • docker’s data-root path is in the same ZFS dataset as OS’ (Ubuntu 24.04) Root:
    root@test-vm:~# zfs list zroot/ROOT/ubuntu
    NAME                USED  AVAIL  REFER  MOUNTPOINT
    zroot/ROOT/ubuntu  15.7G   157G  14.2G  /
    

With such setup, particular CI pipeline takes ~300s to complete, but if I introduce new dataset zroot/docker-data and configure docker to use it as data-root (in /etc/docker/daemon.json) - the same pipeline completes in ~50s. I run docker system prune -af --volumes between my tests, so docker’s caching shouldn’t influence my test results, which are indeed consistent.

New dataset zroot/docker-data should have all the same properties as zroot/ROOT/ubuntu , but please let me know if there is something worth checking, here are some inherited ones, that I set while creating the pool:

root@test-vm:~# zfs get compression,acltype,xattr,relatime,recordsize zroot/docker-data
NAME               PROPERTY     VALUE           SOURCE
zroot/docker-data  compression  lz4             inherited from zroot
zroot/docker-data  acltype      posix           inherited from zroot
zroot/docker-data  xattr        sa              inherited from zroot
zroot/docker-data  relatime     on              inherited from zroot
zroot/docker-data  recordsize   128K            default

root@test-vm:~# zfs get compression,acltype,xattr,relatime,recordsize zroot/ROOT/ubuntu
NAME               PROPERTY     VALUE           SOURCE
zroot/ROOT/ubuntu  compression  lz4             inherited from zroot
zroot/ROOT/ubuntu  acltype      posix           inherited from zroot
zroot/ROOT/ubuntu  xattr        sa              inherited from zroot
zroot/ROOT/ubuntu  relatime     on              inherited from zroot
zroot/ROOT/ubuntu  recordsize   128K            local

root@test-vm:~# zpool get ashift zroot
NAME   PROPERTY  VALUE   SOURCE
zroot  ashift    12      local

root@test-vm:~# zpool list
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
zroot   228G  63.5G   165G        -         -    13%    27%  1.00x    ONLINE  -

zroot/ROOT/ubuntu usually has ~50 ZFS snapshots, but I don’t think that this could be the reason for such performance degradation?

A bit more info about the system:

Instance type: VM at Hetzner Cloud
OS: Ubuntu 24.04 (server edition)
ZFS: zfs-2.2.2-0ubuntu9.2 , zfs-kmod-2.2.2-0ubuntu9.2
Docker: Docker version 28.2.2, build e6534b4
Docker's storage: overlay2
ZFS on root, using ZFSBootMenu

To summarize:

Docker `data-root` value | ZFS dataset       | CI Pipeline duration |
---------------------------------------------------------------------
/var/lib/docker          | zroot/ROOT/ubuntu | ~300s
/docker-data             | zroot/docker-data | ~50s

Any ideas why new dataset can perform that much better, while still being in the same ZFS pool?

By the way, I already tried docker’s zfs storage driver, it performed even worst (the same pipeline took ~90s) and in general I would like to understand why new dataset is introducing such performance boost, I’m not intending to move to docker`s-zfs-driver, zvol with ext4 (~50s) or something else.

Thank you!

That is very interesting. Can you do a zfs get all on each dataset, and post results here with separate spoiler tags wrapped around each dataset’s output please?

click asterisk in post toolbar, hide details

so it shows up like this

Thanks, please find get all outputs below:

zroot/ROOT/ubuntu
root@test-vm:~# zfs get all zroot/ROOT/ubuntu
NAME               PROPERTY                   VALUE                      SOURCE
zroot/ROOT/ubuntu  type                       filesystem                 -
zroot/ROOT/ubuntu  creation                   Mon May 13 15:23 2024      -
zroot/ROOT/ubuntu  used                       12.4G                      -
zroot/ROOT/ubuntu  available                  157G                       -
zroot/ROOT/ubuntu  referenced                 10.9G                      -
zroot/ROOT/ubuntu  compressratio              1.43x                      -
zroot/ROOT/ubuntu  mounted                    yes                        -
zroot/ROOT/ubuntu  quota                      none                       default
zroot/ROOT/ubuntu  reservation                none                       default
zroot/ROOT/ubuntu  recordsize                 128K                       local
zroot/ROOT/ubuntu  mountpoint                 /                          local
zroot/ROOT/ubuntu  sharenfs                   off                        default
zroot/ROOT/ubuntu  checksum                   on                         default
zroot/ROOT/ubuntu  compression                lz4                        inherited from zroot
zroot/ROOT/ubuntu  atime                      on                         default
zroot/ROOT/ubuntu  devices                    on                         default
zroot/ROOT/ubuntu  exec                       on                         default
zroot/ROOT/ubuntu  setuid                     on                         default
zroot/ROOT/ubuntu  readonly                   off                        default
zroot/ROOT/ubuntu  zoned                      off                        default
zroot/ROOT/ubuntu  snapdir                    hidden                     default
zroot/ROOT/ubuntu  aclmode                    discard                    default
zroot/ROOT/ubuntu  aclinherit                 restricted                 default
zroot/ROOT/ubuntu  createtxg                  40                         -
zroot/ROOT/ubuntu  canmount                   noauto                     local
zroot/ROOT/ubuntu  xattr                      sa                         inherited from zroot
zroot/ROOT/ubuntu  copies                     1                          default
zroot/ROOT/ubuntu  version                    5                          -
zroot/ROOT/ubuntu  utf8only                   off                        -
zroot/ROOT/ubuntu  normalization              none                       -
zroot/ROOT/ubuntu  casesensitivity            sensitive                  -
zroot/ROOT/ubuntu  vscan                      off                        default
zroot/ROOT/ubuntu  nbmand                     off                        default
zroot/ROOT/ubuntu  sharesmb                   off                        default
zroot/ROOT/ubuntu  refquota                   none                       default
zroot/ROOT/ubuntu  refreservation             none                       default
zroot/ROOT/ubuntu  guid                       15025120476681164249       -
zroot/ROOT/ubuntu  primarycache               all                        local
zroot/ROOT/ubuntu  secondarycache             all                        local
zroot/ROOT/ubuntu  usedbysnapshots            1.56G                      -
zroot/ROOT/ubuntu  usedbydataset              10.9G                      -
zroot/ROOT/ubuntu  usedbychildren             0B                         -
zroot/ROOT/ubuntu  usedbyrefreservation       0B                         -
zroot/ROOT/ubuntu  logbias                    latency                    default
zroot/ROOT/ubuntu  objsetid                   95                         -
zroot/ROOT/ubuntu  dedup                      off                        default
zroot/ROOT/ubuntu  mlslabel                   none                       default
zroot/ROOT/ubuntu  sync                       standard                   local
zroot/ROOT/ubuntu  dnodesize                  legacy                     default
zroot/ROOT/ubuntu  refcompressratio           1.42x                      -
zroot/ROOT/ubuntu  written                    1.33G                      -
zroot/ROOT/ubuntu  logicalused                16.5G                      -
zroot/ROOT/ubuntu  logicalreferenced          14.4G                      -
zroot/ROOT/ubuntu  volmode                    default                    default
zroot/ROOT/ubuntu  filesystem_limit           none                       default
zroot/ROOT/ubuntu  snapshot_limit             none                       default
zroot/ROOT/ubuntu  filesystem_count           none                       default
zroot/ROOT/ubuntu  snapshot_count             none                       default
zroot/ROOT/ubuntu  snapdev                    hidden                     default
zroot/ROOT/ubuntu  acltype                    posix                      inherited from zroot
zroot/ROOT/ubuntu  context                    none                       default
zroot/ROOT/ubuntu  fscontext                  none                       default
zroot/ROOT/ubuntu  defcontext                 none                       default
zroot/ROOT/ubuntu  rootcontext                none                       default
zroot/ROOT/ubuntu  relatime                   on                         inherited from zroot
zroot/ROOT/ubuntu  redundant_metadata         all                        default
zroot/ROOT/ubuntu  overlay                    on                         default
zroot/ROOT/ubuntu  encryption                 aes-256-gcm                -
zroot/ROOT/ubuntu  keylocation                none                       default
zroot/ROOT/ubuntu  keyformat                  passphrase                 -
zroot/ROOT/ubuntu  pbkdf2iters                350000                     -
zroot/ROOT/ubuntu  encryptionroot             zroot                      -
zroot/ROOT/ubuntu  keystatus                  available                  -
zroot/ROOT/ubuntu  special_small_blocks       0                          default
zroot/ROOT/ubuntu  snapshots_changed          Wed Jun 11  2:00:14 2025   -
zroot/ROOT/ubuntu  org.zfsbootmenu:keysource  zroot/keys                 inherited from zroot
zroot/docker-data
root@test-vm:~# zfs get all zroot/docker-data
NAME               PROPERTY                   VALUE                      SOURCE
zroot/docker-data  type                       filesystem                 -
zroot/docker-data  creation                   Tue Jun 10 17:43 2025      -
zroot/docker-data  used                       2.80G                      -
zroot/docker-data  available                  157G                       -
zroot/docker-data  referenced                 2.80G                      -
zroot/docker-data  compressratio              1.87x                      -
zroot/docker-data  mounted                    yes                        -
zroot/docker-data  quota                      none                       default
zroot/docker-data  reservation                none                       default
zroot/docker-data  recordsize                 128K                       default
zroot/docker-data  mountpoint                 /docker-data               local
zroot/docker-data  sharenfs                   off                        default
zroot/docker-data  checksum                   on                         default
zroot/docker-data  compression                lz4                        inherited from zroot
zroot/docker-data  atime                      on                         default
zroot/docker-data  devices                    on                         default
zroot/docker-data  exec                       on                         default
zroot/docker-data  setuid                     on                         default
zroot/docker-data  readonly                   off                        default
zroot/docker-data  zoned                      off                        default
zroot/docker-data  snapdir                    hidden                     default
zroot/docker-data  aclmode                    discard                    default
zroot/docker-data  aclinherit                 restricted                 default
zroot/docker-data  createtxg                  179147                     -
zroot/docker-data  canmount                   on                         default
zroot/docker-data  xattr                      sa                         inherited from zroot
zroot/docker-data  copies                     1                          default
zroot/docker-data  version                    5                          -
zroot/docker-data  utf8only                   off                        -
zroot/docker-data  normalization              none                       -
zroot/docker-data  casesensitivity            sensitive                  -
zroot/docker-data  vscan                      off                        default
zroot/docker-data  nbmand                     off                        default
zroot/docker-data  sharesmb                   off                        default
zroot/docker-data  refquota                   none                       default
zroot/docker-data  refreservation             none                       default
zroot/docker-data  guid                       6913643019376663566        -
zroot/docker-data  primarycache               all                        default
zroot/docker-data  secondarycache             all                        default
zroot/docker-data  usedbysnapshots            0B                         -
zroot/docker-data  usedbydataset              2.80G                      -
zroot/docker-data  usedbychildren             0B                         -
zroot/docker-data  usedbyrefreservation       0B                         -
zroot/docker-data  logbias                    latency                    default
zroot/docker-data  objsetid                   115571                     -
zroot/docker-data  dedup                      off                        default
zroot/docker-data  mlslabel                   none                       default
zroot/docker-data  sync                       standard                   default
zroot/docker-data  dnodesize                  legacy                     default
zroot/docker-data  refcompressratio           1.87x                      -
zroot/docker-data  written                    2.80G                      -
zroot/docker-data  logicalused                4.85G                      -
zroot/docker-data  logicalreferenced          4.85G                      -
zroot/docker-data  volmode                    default                    default
zroot/docker-data  filesystem_limit           none                       default
zroot/docker-data  snapshot_limit             none                       default
zroot/docker-data  filesystem_count           none                       default
zroot/docker-data  snapshot_count             none                       default
zroot/docker-data  snapdev                    hidden                     default
zroot/docker-data  acltype                    posix                      inherited from zroot
zroot/docker-data  context                    none                       default
zroot/docker-data  fscontext                  none                       default
zroot/docker-data  defcontext                 none                       default
zroot/docker-data  rootcontext                none                       default
zroot/docker-data  relatime                   on                         inherited from zroot
zroot/docker-data  redundant_metadata         all                        default
zroot/docker-data  overlay                    on                         default
zroot/docker-data  encryption                 aes-256-gcm                -
zroot/docker-data  keylocation                none                       default
zroot/docker-data  keyformat                  passphrase                 -
zroot/docker-data  pbkdf2iters                350000                     -
zroot/docker-data  encryptionroot             zroot                      -
zroot/docker-data  keystatus                  available                  -
zroot/docker-data  special_small_blocks       0                          default
zroot/docker-data  org.zfsbootmenu:keysource  zroot/keys                 inherited from zroot
diff between the two datasets (without the 'name' column)
root@test-vm:~# diff -u0 ubuntu-ds docker-ds
--- ubuntu-ds   2025-06-11 11:19:09.317513753 +0200
+++ docker-ds   2025-06-11 11:19:23.884533622 +0200
@@ -3,2 +3,2 @@
-creation                   Mon May 13 15:23 2024      -
-used                       12.4G                      -
+creation                   Tue Jun 10 17:43 2025      -
+used                       2.80G                      -
@@ -6,2 +6,2 @@
-referenced                 10.9G                      -
-compressratio              1.43x                      -
+referenced                 2.80G                      -
+compressratio              1.87x                      -
@@ -11,2 +11,2 @@
-recordsize                 128K                       local
-mountpoint                 /                          local
+recordsize                 128K                       default
+mountpoint                 /docker-data               local
@@ -25,2 +25,2 @@
-createtxg                  40                         -
-canmount                   noauto                     local
+createtxg                  179147                     -
+canmount                   on                         default
@@ -38,5 +38,5 @@
-guid                       15025120476681164249       -
-primarycache               all                        local
-secondarycache             all                        local
-usedbysnapshots            1.56G                      -
-usedbydataset              10.9G                      -
+guid                       6913643019376663566        -
+primarycache               all                        default
+secondarycache             all                        default
+usedbysnapshots            0B                         -
+usedbydataset              2.80G                      -
@@ -46 +46 @@
-objsetid                   95                         -
+objsetid                   115571                     -
@@ -49 +49 @@
-sync                       standard                   local
+sync                       standard                   default
@@ -51,4 +51,4 @@
-refcompressratio           1.42x                      -
-written                    1.33G                      -
-logicalused                16.5G                      -
-logicalreferenced          14.4G                      -
+refcompressratio           1.87x                      -
+written                    2.80G                      -
+logicalused                4.85G                      -
+logicalreferenced          4.85G                      -
@@ -76 +75,0 @@
-snapshots_changed          Wed Jun 11  2:00:14 2025   -

createtxg is massively higher for zroot/docker-data than for zroot/ROOT/ubuntu. This usually indicates a lot of snapshots.

zfs list -rt snap zroot/docker-data | wc -l ; please?

Thank you for your help, @mercenary_sysadmin !

Zero snapshots at zroot/docker-data :

root@test-vm:~# zfs list -rt snap zroot/docker-data | wc -l && zfs get createtxg zroot/docker-data
no datasets available
0
NAME               PROPERTY   VALUE      SOURCE
zroot/docker-data  createtxg  179147     -

A few at zroot/ROOT/ubuntu:

root@test-vm:~# zfs list -rt snap zroot/ROOT/ubuntu | wc -l && zfs get createtxg zroot/ROOT/ubuntu
7
NAME               PROPERTY   VALUE      SOURCE
zroot/ROOT/ubuntu  createtxg  40         -

Any ideas?
Thanks

I’m stumped, honestly. This might be one for @allan .