I have a curious problem, where free space is just missing on my pool.
Host OS is Fedora 40. ZFS packages are in version 2.2.6-1. There is a VM running on that host that has some directories on that pool mounted through virtiofs. I also replicate datasets to external drives and a remote box, that I might have accidentally knocked offline during syncoid run. But the same problem was also going on when I had no interrupted transfers ready to be resumed.
I think this temporarily resolves itself when I reboot the host, but I would rather not do it at least until my planned maintenance this weekend. And I would like to learn what’s the deal here.
The pool in question:
# zpool status hdd
pool: hdd
state: ONLINE
scan: scrub repaired 0B in 07:39:20 with 0 errors on Tue Oct 1 07:42:39 2024
config:
NAME STATE READ WRITE CKSUM
hdd ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
ff375b35-20f7-474b-b306-3a3bab60757c ONLINE 0 0 0
9a52cf68-eba1-4f46-9277-9909ad4ca698 ONLINE 0 0 0
f1f804c5-f3b6-49ae-ae1d-be02d0dd5dbf ONLINE 0 0 0
special
mirror-1 ONLINE 0 0 0
wwn-0x500a0751e86b3225 ONLINE 0 0 0
wwn-0x500a0751e86b3461 ONLINE 0 0 0
errors: No known data errors
One of the affected datasets:
# zfs list -o name,used,usedsnap,refer,mountpoint hdd/ephemeral
NAME USED USEDSNAP REFER MOUNTPOINT
hdd/ephemeral 3.08T 84.8M 3.08T /mnt/hdd/ephemeral
Actual space used:
# LANG=C du -sh /mnt/hdd/ephemeral/
1.7T /mnt/hdd/ephemeral/
Snapshots, note that I destroyed all of the snapshots for this dataset when troubleshooting:
# zfs list -t snap hdd/ephemeral
NAME USED AVAIL REFER MOUNTPOINT
hdd/ephemeral@autosnap_2024-10-28_11:57:09_daily 0B - 3.08T -
hdd/ephemeral@autosnap_2024-10-28_11:57:09_hourly 0B - 3.08T -
hdd/ephemeral@autosnap_2024-10-28_12:00:00_hourly 856K - 3.08T -
hdd/ephemeral@autosnap_2024-10-28_13:00:01_hourly 2.17M - 3.08T -
Where did roughly 1.3 TB of free space go? Is it caused by virtiofs? Or maybe ZFS bookmarks, which I don’t yet know how they work?