Pool does not import on boot but does import by hand

OpenZFS
1

CentOS Stream 8, kernel 4.18.0-513, zfs-2.1.12-1, matching zfs-kmod version.

When booting, the zpool (fs4-pool) is not importing. After a minute we can import the pool from the console, though. Zpool status does not show errors after importing. Could the pool import fail because HDDs are not finished enumerating?

 $ dmesg | grep -i zfs
[    1.266471] systemd[1]: /usr/lib/systemd/system/zfs-snapshot-bootfs.service:8: Unknown lvalue 'ConditionEnvironment' in section 'Unit'
[    1.269850] systemd[1]: /usr/lib/systemd/system/zfs-rollback-bootfs.service:8: Unknown lvalue 'ConditionEnvironment' in section 'Unit'
[    1.271896] systemd[1]: /usr/lib/systemd/system/zfs-nonroot-necessities.service:5: Unknown lvalue 'ConditionEnvironment' in section 'Unit'
[    1.412601] ZFS: Loaded module v2.1.12-1, ZFS pool version 5000, ZFS filesystem version 5

Those first three errors appear to be related to zfs-on-root but that doesn’t apply to me.

These are the more irritating errors:

 $ journalctl -b0 | grep -C3 -i zpool 
Apr 11 15:52:48 fs4 kernel: XFS (dm-0): Ending clean mount
Apr 11 15:52:48 fs4 systemd[1]: Mounted /sysroot.
Apr 11 15:52:48 fs4 systemd[1]: Starting zfs-nonroot-necessities.service...
Apr 11 15:52:48 fs4 zpool[1501]: cannot import 'fs4-pool': pool was previously in use from another system.
Apr 11 15:52:48 fs4 zpool[1501]: Last accessed by fs4 (hostid=96e572) at Thu Apr 11 22:48:50 2024
Apr 11 15:52:48 fs4 zpool[1501]: The pool can be imported, use 'zpool import -f' to import the pool.
Apr 11 15:52:48 fs4 systemd[1]: zfs-import-scan.service: Main process exited, code=exited, status=1/FAILURE
Apr 11 15:52:48 fs4 systemd[1]: zfs-import-scan.service: Failed with result 'exit-code'.
Apr 11 15:52:48 fs4 systemd[1]: Failed to start Import ZFS pools by device scanning.
Apr 11 15:52:52 fs4 zpool[2367]: cannot import 'fs4-pool': no such pool or dataset
Apr 11 15:52:52 fs4 zed[3044]: eid=2 class=vdev.no_replicas pool='fs4-pool'
Apr 11 15:52:52 fs4 zed[3069]: eid=3 class=vdev.no_replicas pool='fs4-pool'
Apr 11 15:52:52 fs4 zed[3024]: eid=1 class=vdev.no_replicas pool='fs4-pool'
Apr 11 15:52:52 fs4 zed[3098]: eid=5 class=zpool pool='fs4-pool'
Apr 11 15:52:52 fs4 zed[3074]: eid=4 class=vdev.no_replicas pool='fs4-pool'
Apr 11 15:52:52 fs4 zed[3074]: eid=4 class=vdev.no_replicas pool='fs4-pool'
Apr 11 15:52:52 fs4 zed[3502]: eid=6 class=data pool='fs4-pool' priority=0 err=6 flags=0x8881 bookmark=0:0:-1:0
Apr 11 15:52:52 fs4 zed[3503]: eid=7 class=zpool pool='fs4-pool'
Apr 11 15:52:52 fs4 zed[3556]: eid=8 class=zpool pool='fs4-pool'
Apr 11 15:52:52 fs4 zpool[2367]: cannot import 'fs4-pool': I/O error
Apr 11 15:52:52 fs4 zpool[2367]:         Destroy and re-create the pool from
Apr 11 15:52:52 fs4 zpool[2367]:         a backup source.
Apr 11 15:52:52 fs4 zpool[2367]: cachefile import failed, retrying
Apr 11 15:52:52 fs4 zpool[2367]:         Destroy and re-create the pool from
Apr 11 15:52:52 fs4 zpool[2367]:         a backup source.
Apr 11 15:52:52 fs4 systemd[1]: zfs-import-cache.service: Main process exited, code=exited, status=1/FAILURE
Apr 11 15:52:52 fs4 systemd[1]: zfs-import-cache.service: Failed with result 'exit-code'.
Apr 11 15:52:52 fs4 systemd[1]: Failed to start Import ZFS pools by cache file.
Apr 11 15:52:52 fs4 systemd[1]: Reached target ZFS pool import target.
Apr 11 15:52:52 fs4 systemd[1]: Reached target ZFS startup target.
Apr 11 15:52:52 fs4 zed[3559]: eid=9 class=zpool pool='fs4-pool'
Apr 11 15:52:52 fs4 systemd[1]: Starting Snapshot ZFS Pool...
Apr 11 15:52:52 fs4 systemd[1]: Starting Virtualization daemon...
Apr 11 15:52:52 fs4 systemd[1]: Starting The Apache HTTP Server...

And the pool:

 $ zpool status
  pool: fs4-pool
 state: ONLINE
  scan: scrub repaired 0B in 07:49:51 with 0 errors on Sat Apr 13 04:52:52 2024
config:

	NAME                                  STATE     READ WRITE CKSUM
	fs4-pool                              ONLINE       0     0     0
	  raidz1-0                            ONLINE       0     0     0
	    ata-ST4000NM000A-2HZ100_WS209AAM  ONLINE       0     0     0
	    ata-ST4000NM000A-2HZ100_WS22RZX2  ONLINE       0     0     0
	    ata-ST4000NM000A-2HZ100_WS20CZKS  ONLINE       0     0     0
	  raidz1-1                            ONLINE       0     0     0
	    ata-ST4000NM000A-2HZ100_WS20DB76  ONLINE       0     0     0
	    ata-ST4000NM000A-2HZ100_WS20DBGB  ONLINE       0     0     0
	    ata-ST4000NM000A-2HZ100_WS20DBHC  ONLINE       0     0     0
	  raidz1-2                            ONLINE       0     0     0
	    ata-ST4000NM000A-2HZ100_WS20DPX1  ONLINE       0     0     0
	    ata-ST4000NM000A-2HZ100_WS20DQHP  ONLINE       0     0     0
	    ata-ST4000NM000A-2HZ100_WS20DV2P  ONLINE       0     0     0
	  raidz1-3                            ONLINE       0     0     0
	    ata-ST4000NM000A-2HZ100_WS20DVNE  ONLINE       0     0     0
	    ata-ST4000NM000A-2HZ100_WS20DW8Y  ONLINE       0     0     0
	    ata-ST4000NM000A-2HZ100_WS20DWB8  ONLINE       0     0     0
	logs	
	  mirror-4                            ONLINE       0     0     0
	    nvme0n1p1                         ONLINE       0     0     0
	    nvme1n1p1                         ONLINE       0     0     0
	cache
	  nvme0n1p2                           ONLINE       0     0     0
	  nvme1n1p2                           ONLINE       0     0     0

errors: No known data errors

I have created the cache file and it the /etc/hostid file and it matches the dec/hex of the pool.

 $ systemctl list-dependencies zfs.target
zfs.target
● ├─zfs-mount.service
● ├─zfs-zed.service
● └─zfs-import.target
●   └─zfs-import-cache.service

I don’t understand why cache errors, tho. Is udev-settle not happening?

$ /usr/lib/systemd/systemd --test --debug zfs.target
        -> Unit systemd-udev-settle.service:
                Description: udev Wait for Complete Device Initialization
                Instance: n/a
                Unit Load State: loaded
                Unit Active State: inactive
                State Change Timestamp: n/a
                Inactive Exit Timestamp: n/a
                Active Enter Timestamp: n/a
                Active Exit Timestamp: n/a
                Inactive Enter Timestamp: n/a
                May GC: no
                Need Daemon Reload: no
                Transient: no
                Perpetual: no
                Garbage Collection Mode: inactive
                Slice: system.slice
                CGroup: n/a
                CGroup realized: no
                CGroup own mask: memory pids
                Name: systemd-udev-settle.service
                Documentation: man:udev(7)
                Documentation: man:systemd-udevd.service(8)
                Fragment Path: /usr/lib/systemd/system/systemd-udev-settle.service
                ConditionPathIsReadWrite: /sys untested
                Requires: system.slice (origin-file)
                Wants: systemd-udevd.service (origin-file)
                RequiredBy: zfs-import-scan.service (destination-file)
                RequiredBy: zfs-import-cache.service (destination-file)
                WantedBy: multipathd.service (destination-file)
                Before: sysinit.target (origin-file)
                Before: multipathd.service (destination-file)
                Before: zfs-mount.service (destination-file)
                Before: zfs-import-scan.service (destination-file)
                Before: zfs-import-cache.service (destination-file)
                After: systemd-udev-trigger.service (origin-file)
                After: system.slice (origin-file)
                After: systemd-journald.socket (origin-file)
                References: systemd-udevd.service (origin-file)
                References: systemd-udev-trigger.service (origin-file)
                References: sysinit.target (origin-file)
                References: system.slice (origin-file)
                References: systemd-journald.socket (origin-file)
                ReferencedBy: zfs-import-scan.service (destination-file)
                ReferencedBy: zfs-import-cache.service (destination-file)
                ReferencedBy: multipathd.service (destination-file)
                ReferencedBy: zfs-mount.service (destination-file)

Maybe I should enable import zfs-import-scan and zfs-import-cache:

 $ systemctl list-unit-files | grep z | grep -v timer | sort
zabbix-agent.service                                       enabled        
zed.service                                                enabled        
zfs-import-cache.service                                   enabled        
zfs-import-scan.service                                    disabled       
zfs-import.target                                          enabled        
zfs-load-key.service                                       masked         
zfs-mount.service                                          enabled        
zfs-scrub@.service                                         static         
zfs-share.service                                          disabled       
zfs.target                                                 enabled        
zfs-volumes.target                                         disabled       
zfs-volume-wait.service                                    disabled       
zfs-zed.service                                            enabled
1 Like
2

This is the money quote, right here. Figure out why your pool thinks that the last system that mounted it isn’t the same system attempting to mount it right now.

1 Like
3

This system had a CPU upgrade. I saw this Thurs and we re-did the /etc/hostid file using:

 $ zdb | grep host
    hostid: 9889138
    hostname: 'fs4'

$ zgenhostid -f 0x0096E572

 $ zdb | grep host
    hostid: 9889138
    hostname: 'fs4'

 $ echo "ibase=10;obase=16;998138" | bc
F3AFA

 $ echo "ibase=16;obase=A;96E570" | bc
9889136

So probably my zgenhostid command was wrong because we typoed the hex conversion, and it should be:

 $ zgenhostid -f 0x000F3AFA
4

Yep, that’ll do it. You need to make sure both that your machine has a stable hostid AND that your pool is getting exported cleanly, then it will stop refusing to import without the -f flag after a reboot.

To be clear, here, ZFS will absolutely reimport a pool on boot that had not been cleanly exported first, and without requiring a -f: but only if the hostid it thinks imported the pool last is the same hostid as the one trying to import it now.

You might be able to fix this simply by doing zpool export fs4-pool ; zpool import fs4-pool. If I remember correctly, pools aren’t always expected to be exported during shutdown at all, so you may have simply never replaced the old hostid with the new one, because you kept forcibly importing the pool but never exported the pool (to update the last-imported-by stamp).

2 Likes