Zfs snapshot - where are the files?

OpenZFS
1

Hi again,
tldr; zfs snapshots of datasets are empty.

Please follow along and help me understand:

# uname -a
FreeBSD zabbix 14.1-RELEASE-p3 FreeBSD 14.1-RELEASE-p3 GENERIC amd64

# zfs list
NAME                                            USED  AVAIL  REFER  MOUNTPOINT
zroot                                          13.1G   216G    96K  /zroot
zroot/ROOT                                     13.1G   216G    96K  none
zroot/ROOT/default                             13.1G   216G  6.94G  /
zroot/tmp                                       148K   216G   148K  /tmp
zroot/usr                                       484K   216G    96K  /usr
zroot/usr/home                                  196K   216G   196K  /usr/home
zroot/usr/ports                                  96K   216G    96K  /usr/ports
zroot/usr/src                                    96K   216G    96K  /usr/src
zroot/var                                      3.02M   216G    96K  /var
zroot/var/audit                                  96K   216G    96K  /var/audit
zroot/var/crash                                  96K   216G    96K  /var/crash
zroot/var/log                                  2.43M   216G  1.90M  /var/log
zroot/var/mail                                  220K   216G   220K  /var/mail
zroot/var/tmp                                   104K   216G   104K  /var/tmp

# echo "lala" > /usr/afile.txt
# ls -l /usr/
total 255
-rw-r--r--   1 root wheel   5 Aug 28 19:18 afile.txt
drwxr-xr-x   2 root wheel 488 Aug  9 10:05 bin
drwxr-xr-x   4 root wheel   3 Aug 28 19:10 home
drwxr-xr-x  56 root wheel 356 Aug 13 13:23 include
drwxr-xr-x  11 root wheel 582 Aug 13 13:23 lib
drwxr-xr-x   9 root wheel 604 Aug 13 13:23 lib32
drwxr-xr-x   4 root wheel   4 Aug 13 13:23 libdata
drwxr-xr-x  12 root wheel  71 Aug  9 10:05 libexec
drwxr-xr-x  13 root wheel  15 Aug  9 14:17 local
drwxr-xr-x   2 root wheel   2 May 12  2022 obj
drwxr-xr-x   3 root wheel   2 Sep 15  2023 ports
drwxr-xr-x   2 root wheel 291 Aug  9 10:05 sbin
drwxr-xr-x  33 root wheel  33 Oct 31  2023 share
drwxr-xr-x   3 root wheel   2 May 12  2022 src
drwxr-xr-x  18 root wheel  18 Aug  9 09:44 tests

# zfs snapshot zroot/usr@now
# zfs list -t snapshot zroot/usr@now
NAME            USED  AVAIL  REFER  MOUNTPOINT
zroot/usr@now     0B      -    96K  -

# zfs get snapdir zroot/usr
NAME       PROPERTY  VALUE    SOURCE
zroot/usr  snapdir   visible  inherited from zroot
# ls -la /usr
total 289
drwxr-xr-x  16 root wheel  17 Aug 28 19:18 .
drwxr-xr-x  20 root wheel  25 Aug 14 14:14 ..
-rw-r--r--   1 root wheel   5 Aug 28 19:18 afile.txt
drwxr-xr-x   2 root wheel 488 Aug  9 10:05 bin
drwxr-xr-x   4 root wheel   3 Aug 28 19:10 home
drwxr-xr-x  56 root wheel 356 Aug 13 13:23 include
drwxr-xr-x  11 root wheel 582 Aug 13 13:23 lib
drwxr-xr-x   9 root wheel 604 Aug 13 13:23 lib32
drwxr-xr-x   4 root wheel   4 Aug 13 13:23 libdata
drwxr-xr-x  12 root wheel  71 Aug  9 10:05 libexec
drwxr-xr-x  13 root wheel  15 Aug  9 14:17 local
drwxr-xr-x   2 root wheel   2 May 12  2022 obj
drwxr-xr-x   3 root wheel   2 Sep 15  2023 ports
drwxr-xr-x   2 root wheel 291 Aug  9 10:05 sbin
drwxr-xr-x  33 root wheel  33 Oct 31  2023 share
drwxr-xr-x   3 root wheel   2 May 12  2022 src
drwxr-xr-x  18 root wheel  18 Aug  9 09:44 tests

Where is /usr/.zfs ?

# zfs clone zroot/usr@now zroot/usr-clone
# zfs list zroot/usr-clone
NAME              USED  AVAIL  REFER  MOUNTPOINT
zroot/usr-clone     0B   216G    96K  /zroot/usr-clone
# ls -la /zroot/usr-clone/.zfs/snapshot/
total 0
dr-xr-xr-x+ 2 root wheel 2 Jan  1  1970 .
dr-xr-xr-x+ 3 root wheel 3 Sep 15  2023 ..

Where are the files?

I replicated the behavior on three FreeBSD machines.
If I do the same on zroot/var/tmp, which has no child sets, everything is ok.

Thanks in advance
Marcus

2

Looks to me like zroot/usr isn’t actually mounted properly.

NAME                          USED  AVAIL  REFER  MOUNTPOINT
zroot/usr                     484K   216G    96K  /usr

See that REFER column? Your entire operating system /usr definitely takes more than 96KiB of space, which means the dataset is not mounted, and your actual /usr is simply a folder beneath (presumably) zroot/ROOT/default.

Same goes for zroot/usr/ports, zroot/usr/src, zroot/var… from the looks of it, nearly everything in your pool is unmounted, so the actual data is all in folders stored in zroot/ROOT/default.

3

This has something to do with FreeBSD’s boot environments.
Some datasets are included in the boot environment represented by zroot/ROOT/default. Those have canmount=off.
Others, like zroot/var/log are excluded (canmount=on).