zfs-2.1.15 don't compile with RHEL 9.5 last kernel-5.14.0-503.11.1.el9_5.x86_64

[rgessner]

System information

Distribution Name | Red Hat Enterprise Linux (not a rhel clone)
Distribution Version | 9.5
Kernel Version | kernel-5.14.0-503.11.1.el9_5.x86_64
Architecture | x86_64
OpenZFS Version | 2.1.15

Describe the problem you're observing

Working zfs on RHEL 9.4 system with kernel-5.14.0-427.42.1.el9_4.x86_64.
couple of error messages during kernel update (see below).
After booting no zfs module present

Describe how to reproduce the problem

Update from RHEL 9.4 kernel 5.14.0-427 to RHEL 9.5 kernel-5.14.0.-503

Include any warning/errors/backtraces from the system logs

checking whether blkdev_get_by_path() exists and takes 3 args... no
checking whether blkdev_get_by_path() exists and takes 4 args... no
checking whether bdev_open_by_path() exists... configure: error: 
	*** None of the expected "blkdev_get_by_path()" interfaces were detected.
	*** This may be because your kernel version is newer than what is
	*** supported, or you are using a patched custom kernel with
	*** incompatible modifications.
	***
	*** ZFS Version: zfs-2.1.15-3
	*** Compatible Kernels: 3.10 - 6.7
	

Cleaning build area...(bad exit status: 2)
Failed command:
make -C /lib/modules/5.14.0-503.11.1.el9_5.x86_64/build M=/var/lib/dkms/zfs/2.1.15/build cleanBuilding module(s)...(bad exit status: 2)
Failed command:
make -j4 KERNELRELEASE=5.14.0-503.11.1.el9_5.x86_64
Error! Bad return status for module build on kernel: 5.14.0-503.11.1.el9_5.x86_64 (x86_64)
Consult /var/lib/dkms/zfs/2.1.15/build/make.log for more information.
Autoinstall on 5.14.0-503.11.1.el9_5.x86_64 failed for module(s) zfs(10).

Error! One or more modules failed to install during autoinstall.
Refer to previous errors for more information.

[AllKind]

I don't think ZFS 2.1.x is maintained any more.
Could you try with 2.2.6?

[rgessner]

2.1.15 is still the version in the "official" RHEL repo here.
se also:
https://openzfs.github.io/openzfs-docs/Getting%20Started/RHEL-based%20distro/index.html

I think 2.2.6 will work, but i need some time to test this.
Maybe the RHEL repo should be updated if 2.1.15 ist no longer supported.

[AllKind]

You could just run configure to see if the check passes. Which could also help others.

[wanghz10]

Version 2.2.6 seems to be working. You can enable zfs-testing to install.

dnf config-manager --enable zfs-testing
dnf upgrade
dkms autoinstall

[rgessner]

I can confirm 2.2.6 from zfs-testing (dkms) repo works with RHEL9.5.

Still im shure that a couple of people will run in this issue in a few days, when all the RHEL Clones (Rocky, Alma, Oracle) also bump from 9.4 to 9.5.

Maybe this should be fixed/updated in some way. ;)

[llagendijk]

The testing repo for RHEL9 has 2.2.6 and works for me

[ItalyPaleAle]

I had the same issue when updating RHEL 9 to the latest version.

• There doesn't seem to be any package in zfs or zfs-kmod available
• Using DKMS from zfs-testing (2.2.6) worked

[Germano0]

when 2.2.x will be moved from "testing" to "stable" repository for RHEL?
From https://openzfs.github.io/openzfs-docs/Getting%20Started/RHEL-based%20distro/index.html#testing-repositories

These packages should not be used on production systems.

You could just run configure to see if the check passes. Which could also help others.

@AllKind which configure?

I don't think ZFS 2.1.x is maintained any more.

On Github I do see a 2.1.16-staging branch which is newer than the 2.1.15 shipped for RHEL, and last commit is from 7 months ago

[hawkdavis]

yeah guys my plex server was down for an entire 10 minutes... unacceptable.

[tcouey]

yeah guys my plex server was down for an entire 10 minutes... unacceptable.

lol. That's how I discovered the problem this morning too... but mine had been down for over 8 hours. I thought I was the only one running Plex on RHEL!

[tonyhutter]

Thanks all for reporting this. Looks like the new kernel has hit AlmaLinux 9. I've put together a new release PR with the commits needed to get it building again: #16781

[Germano0]

Looks like the new kernel has hit AlmaLinux 9

@tonyhutter you can use RHEL for free under certain circumstances (see terms at https://developers.redhat.com/articles/faqs-no-cost-red-hat-enterprise-linux )

[mtelka]

I assume this is a duplicate of #16196.

[donvipre]

this hit also Rocky 9

checking whether bdev_open_by_path() exists... configure: error: 
        *** None of the expected "blkdev_get_by_path()" interfaces were detected.
        *** This may be because your kernel version is newer than what is
        *** supported, or you are using a patched custom kernel with
        *** incompatible modifications.
        ***
        *** ZFS Version: zfs-2.1.15-3
        *** Compatible Kernels: 3.10 - 6.7


Cleaning build area...(bad exit status: 2)
Failed command:
make -C /lib/modules/5.14.0-503.14.1.el9_5.x86_64/build M=/var/lib/dkms/zfs/2.1.15/build cleanBuilding module(s)...(bad exit status: 2)
Failed command:
make -j2 KERNELRELEASE=5.14.0-503.14.1.el9_5.x86_64
Error! Bad return status for module build on kernel: 5.14.0-503.14.1.el9_5.x86_64 (x86_64)
Consult /var/lib/dkms/zfs/2.1.15/build/make.log for more information.

[cyqsimon]

And PSA: don't forget to dnf config-manager --disable zfs-testing after you're done. Unless you actually want to be on the rolling channel, which you most likely don't.

Also, one-liner alternative to enabling the repo, updating, then disabling it:

dnf update --enablerepo=zfs-testing zfs

[principis]

Thanks all for reporting this. Looks like the new kernel has hit AlmaLinux 9. I've put together a new release PR with the commits needed to get it building again: #16781

This does not seem to work. Is there anything else that's needed?

Edit: I got it to work. Just patching zfs-dkms wasn't enough. Applying the patches to the git repo and rebuilding all rpm's worked. Thanks!

[makhomed]

Thanks all for reporting this. Looks like the new kernel has hit AlmaLinux 9. I've put together a new release PR with the commits needed to get it building again: #16781

AlmaLinux OS is not 1:1 RHEL clone, AlmaLinux OS Foundation board Jul 13, 2023 has decided to drop the aim to be 1:1 with RHEL:

Now that we will no longer be holding ourselves to being a 1:1 Red Hat downstream rebuild, we are taking some time to consider the possibilities around what that means. We will continue to provide updates around that process and will include the members of the AlmaLinux OS Foundation in that conversation and decision-making process as well. -- https://almalinux.org/blog/future-of-almalinux/

It is more safe to use original RHEL - it is no-cost for development use - https://developers.redhat.com/articles/faqs-no-cost-red-hat-enterprise-linux

Or use Rocky Linux, because Rocky Linux is designed to be 100% bug-for-bug compatible with RHEL.

[deajan]

As far as I understand ZoL supports current and last release, so since we're not far away from OpenZFS 2.3, we safely can enable 2.2.6 on current EL9 since 2.2.x branch will become default for EL9 in a not so far distant feature.
@tonyhutter Can you confirm perhaps ?
Nevertheless, I'll be happy to stay on the safe side (2.1.x on EL9) until official releases ;)

[TopherIsSwell]

dnf update --enablerepo=zfs-testing zfs

This worked for me on both Rocky 9 and Alma 9. But what was interesting was that building 2.2.6 and 2.3.0-RC3 from source (actually just running the configure script) on Alma 9 failed with the same error, but installing from the zfs-testing repo worked fine. Is there some kind of secret sauce in the RPM that's not accessible by doing the following in the source?

./autogen.sh
./configure

It is more safe to use original RHEL - it is no-cost for development use - https://developers.redhat.com/articles/faqs-no-cost-red-hat-enterprise-linux

Using the RHEL development license is as easy as 1, 2, 3 hours of searching their website for the correct process.

[Germano0]

we can close this ticket

[dennixxNL]

The on 6 December released 2.1.16 seems to work well on RHEL9

[deajan]

Confirmed.

[npelov]

Thank you for the good work!