I am starting my learning with OpenZFS on RHEL and I am bumping into likely incompatible kernels - I think. I am looking for guidance on the most reliable installation option and maintain update cadence. I am installing OpenZFS on a homelab server but again I am seeking reliability, I dont want to sit down to get some learning time only to have a update issues.
I have a fresh minimal install of RHEL 9.6 with all available updates applied.
Trying DKMS, the automatic install appears to have failed. Trying to install DKMS manually with;
# dkms status
zfs/2.1.16: added
# dkms build -m 2.1.16
.....
Running the pre_build script..............(bad exit status: 1)
Failed command:
cd /var/lib/dkms/zfs/2.1.16/build/ && /var/lib/dkms/zfs/2.1.16/build/configure --prefix=/usr --with-config=kernel --with-linux=/lib/modules/5.14.0-570.18.1.el9_6.x86_64/source --with-linux-obj=/lib/modules/5.14.0-570.18.1.el9_6.x86_64/build --with-qat=
Consult /var/lib/dkms/zfs/2.1.16/build/make.log for more information.
.....
and the log.
checking whether blk_queue_max_hw_sectors() is available... configure: error:
*** None of the expected "blk_queue_max_hw_sectors" interfaces were detected.
*** This may be because your kernel version is newer than what is
*** supported, or you are using a patched custom kernel with
*** incompatible modifications.
***
*** ZFS Version: zfs-2.1.16-1
*** Compatible Kernels: 3.10 - 6.7
# exit code: 1
# elapsed time: 00:00:34
Trying kABI-tracking kmod also appears to have compatibility and there is pre-compiled modules found
Looking over the OpenZFS documentation I dont see any guidance for tracking kernel and OpenZFS release compatibility concerns, the documentation infers it should just work.
Appreciate guidance on how I should be approaching this.
FYSA, I can change distributions, i.e. not RHEL 9 but RHEL seems like it should be supported and is my preferred distribution.
IME RHEL is pretty virulently anti-ZFS, and may not be the easiest choice to live with.
You might have better luck with a RHEL-compatible distro like Alma or Rocky, but I am not what I would consider expert in any of the modern RHEL-derived ecosystem, and I offer this idea as something that might be worth looking into yourself to see if it could be helpful, not as blanket advice to dump what you’re doing and “do this instead.”
If you come to the conclusion that none of the RHEL-derived distros are making life easy enough for you, there are Fedora Silverblue variants with OpenZFS very very directly baked in, as well as the 900 pound gorilla in the OpenZFS-on-Linux room, Ubuntu.
OpenZFS is also well supported in some of the neither-fish-nor-fowl distros out there, like Arch and Void. But I am not really an expert in maintaining OpenZFS on any platforms but 1. Ubuntu and 2. FreeBSD, so, again, please take this as some guidance on what to look for and where, not as a command “go do this, and all will be well.” <3
Also, sincere apologies to fans of any distro who don’t appreciate being referred to as “neither fish nor fowl,” with the implication being that Debian-derived distros are ‘fish’ and RHEL-derived are ‘fowl’. I do not mean to demean anybody’s choice of distribution, nor imply that only those two overall families are worth consideration.
Arch in particular deserves everybody’s respect, because whether you want to run an Arch system or not–and I personally very very much do not–the work that community has done in relentlessly and exhaustively documenting everything is worthy of thunderous applause.
I cannot tell you how many times I have referred to the Arch wiki to find answers to problems in entirely unrelated distros, because the Arch users actually documented everything extensively and professionally, in many cases far, far better than the original upstream sources of the software they were documenting ever have!
With the understanding, again, that I am not truly expert in maintaining OpenZFS on any distribution but Ubuntu and FreeBSD, yes, I would recommend looking intoucore and seeing if perhaps it fits your needs.
I know and greatly respect Jorge Castro, and he’s gently recommended my looking into or direct involvement with uCore several times. I sadly have not found the time, but just the fact that Jorge is 1. involved and 2. thinks I should be involved is a pretty solid hint that somebody looking for a ZFS on Linux home and unhappy with what they’ve found so far should at the very least look at uCore.
Ubuntu has done a lot of good things for OpenZFS including making it directly available in the installer. Unfortunately their efforts have not always been consistent (wrt zsys in particular.)
I’ve been running OpenZFS on Debian hosts for years and which uses DKMS for the builds. I can only recall two issues which occasionally crop up.
With root on ZFS and the bpool / rpool split, if I have too many old kernels the DKMS build will fail with no space left on the device.
When running the Testing (non-)release, it is possible for the kernel version to get ahead of what is compatible with the available ZFS releases. When this has happened, I just booted the previous kernel until compatible a ZFS module was released.
OpenZFS is nearly as easy on Debian as Ubuntu. Aside from that, Debian vs. Ubuntu is just a personal choice as they’re both very good IMO.
Thank you for the direct usage report! This sounds essentially like running DKMS OpenZFS on Ubuntu, prior to Canonical bringing it in-repo, and I agree it was quite manageable–although having it in-repo genuinely has solved a couple of the papercuts you mention remaining.
It’s also of great personal interest, because I am not always entirely happy with Canonical these days, and frequently consider going back upstream to Debian (from whence I came; Debian was the first Linux distro I actually approved of, as somebody who started out as a big FreeBSD person).
I built 2.2.7 (for kernel-5.14.0-570.16.1.el9_6.x86_64) when I ran into this, so that would be my suggestion if you want to stick with RHEL, but yeah, if this isn’t the sort of thing you want to have to deal with, you might want to go with another OS for your ZFSing. You’re going to have the same problem with Alma.
I’ve been running CS8 and CS9 for some time and the ZFS repositories that work best are the zfs-testing ones. The advancement of the stable zfs RHEL compatible seem to be frightened to advance. Please try the zfs-testing sources to get a more recent zfs release.
Kind of surprised. I’ve been running ZFS on Fedora for years and besides needing to sometimes wait until OpenZFS updates before updating to the newest minor of the kernel, I haven’t had issues.
I’d think the RHEL downstream family would be much easier.
I’ve been using OpenZFS on Oracle Linux (i.e. one of the RHELatives) for several years without problems. (The choice of Oracle came about because the organisation used Solaris before). While Redhat may be anti-ZFS as Jim Salter says, RHEL is widely enough used that it is well-supported by the zfsonlinux project. They provide a zfs-release rpm that adds repositories and kmod packages for the current Redhat kernels. I’ve not used those so much, mainly because we tend to use the newer UEK kernel from Oracle. Occasionally a kernel update causes breakage and the fact that Redhat selectively backports quite a lot of changes can cause issues where openzfs uses the wrong interface for something. That might be the case here with this blk_queue_max_hw_sectors interface? What kernel version have you got here? You could also try switching to the newer ZFS release which is provided as zfs-testing.
I initially used dkms but it can be somewhat unreliable. For a single machine it’s probably the easiest choice but with a dozen machines or a setup that already entails local dnf repository mirrors, kmod packages allow trouble-free upgrades on all but the first machine. The ZFS kmods tend to work with subsequent patch releases of kernels where the nvidia/cuda kmods cause us far more trouble. For more detail on that choice, have you read:
I’ve hit this particular issue, because our work environment is RHEL-based (currently Alma, previously CentOS, and Scientifc Linux before that), and we have a fair few systems using ZFS. Booted into the previous kernel as a temporary workaround and have a couple of other systems fixed at 9.5 for now, but that’s not a long-term solution, and switching into a testing branch isn’t ideal either. I appreciate the testing branch is likely to be fine to use in practice, but it’s not an assumption I like to make for a production system.
Hopefully they’ll add the version specific RPM repositories, but I’m also about to set up two new storage systems, so I might need to look at Ubuntu/Debian for those (it’s not a requirement to use RHEL here, just a preference to avoid the extra workload managing and supporting additional distributions).