ZFS on Tumbleweed: How to keep a working kernel version?

Hi all,

I’ve been using OpenSUSE Tumbleweed for many years and have been quite content with the rolling release model. It just works and my software is always fresh. Some days ago, I started upgrading my private storage (to 16 TB), so I wanted to switch from XFS to ZFS as I’ve been collecting a lot of (very positive) ZFS experience on Sun/Oracle Solaris 11.4 at work, especially regarding data integrity. So I know something about Tumbleweed and ZFS, but I don’t have any experience with ZFS on Linux.

Just a little note in advance: I don’t want to boot from ZFS, my boot SSD is (and will stay for now) a plain EXT4 filesystem.

So, in order to get ZFS support on Tumbleweed, I installed it according to the steps mentioned in https://en.opensuse.org/OpenZFS:


zypper addrepo https://download.opensuse.org/repositories/filesystems/openSUSE_Tumbleweed/filesystems.repo
zypper refresh
zypper install zfs

My problem: After issuing a “zypper dup” today, Tumbleweed upgraded to kernel 5.18 and all ZFS support seems to be gone. I even cannot re-install “zfs-kmp-default”:


steffen@magpie:~> sudo zypper in zfs-kmp-default  
[sudo] password for root:  
Loading repository data... 
Reading installed packages... 
Resolving package dependencies... 

Problem: nothing provides 'kernel-uname-r = 5.17.9-1-default' needed by the to be installed zfs-kmp-default-2.1.4_k5.17.9_1-1.69.x86_64 
 Solution 1: do not install zfs-kmp-default-2.1.4_k5.17.9_1-1.69.x86_64 
 Solution 2: break zfs-kmp-default-2.1.4_k5.17.9_1-1.69.x86_64 by ignoring some of its dependencies 

**Choose from above solutions by number or cancel [1/2/c/d/?] (c):**

So, obviously, the kernel which has been installed by Tumbleweed’s “zypper dup” (5.18.2-1-default) is too fresh for the most recent version of ZFS on Linux. The older kernel on my system is 5.18.1-1-default. Even compiling zfs-2.1.4 from the sources fails due kernel 5.18.

So I try to install kernel 5.17.9-1-default, but it does not seem to be available in the version of Tumbleweed anymore:


steffen@magpie:~> sudo zypper in kernel-default-5.17.9-1-default 
Loading repository data... 
Reading installed packages... 
'kernel-default-5.17.9-1-default' not found in package names. Trying capabilities. 
No provider of 'kernel-default-5.17.9-1-default' found.
Resolving package dependencies... 
Nothing to do.

Of course, I might be able to get kernel 5.17 from somewhere (or even compile it) and get access back to my zpool. And even if this didn’t work, my data would still be on the old XFS storage, where I could get everything back, so no actual data loss here. Hence, my question is rather of a generic nature: How can I make sure that a kernel update does not break anything in Tumbleweed regarding ZFS? I just would like to have a consistent ZFS-aware rolling release system as I had before. It is not a problem for me to use a slightly older kernel, but I don’t want to have it in my mind to check it manually with each update. So I would like to know if there is any possibility to tell Tumbleweed’s package manager that it should only install kernels which have a suitable “zfs-kmp-default”?

While I can play with “multiversion.kernels = latest,latest-1,running” in /etc/zypp/zypp.conf" and put a “latest-2” there, I am not sure if it guarantees me to have always ZFS support then.

I am looking forward to reading any helpful insights and experience reports of running ZFS on OpenSUSE Tumbleweed. Thank you very much in advance!

Kind regards,
Steffen

Well, btrfs snapshots would allow easy recovery in this case.

So I try to install kernel 5.17.9-1-default, but it does not seem to be available in the version of Tumbleweed anymore

https://download.opensuse.org/history/
And you can consider using tumbleweed-cli to lock specific Tumbleweed snapshot and switch manually when driver becomes available.

But this is only Tumbleweed, it does not keep the content of third-party repositories.

So I would like to know if there is any possibility to tell Tumbleweed’s package manager that it should only install kernels which have a suitable “zfs-kmp-default”?

You can lock kernel-default package and only install manually. This is lightweight version of tumbeweed-cli (which locks the whole distribution). You can create dummy package that e.g. conflicts with newer kernel versions, but this package will need to be constantly updated anyway so locking kernel package is easier.

While I can play with “multiversion.kernels = latest,latest-1,running” in /etc/zypp/zypp.conf" and put a “latest-2” there, I am not sure if it guarantees me to have always ZFS support then.

You can add “oldest” which will preserve the oldest kernel. When you are satisfied with the currently running kernel, just unistall the oldest one.

I am looking forward to reading any helpful insights and experience reports of running ZFS on OpenSUSE Tumbleweed.

You may be interested in GitHub - ndruba/LroZ: Linux root on ZFS installer

Thank for that hint, arvidjaar, very helpful!

Nevertheless, ZFS and new kernel versions are a constant source of joy because ZFS always lags behind with a significant delay. By bad experience I have learned to pay attention with new kernel versions, and I always check whether also a new zfs-kmp-default version is going to be installed. Last time, I was tricked nevertheless, because kernel 5.18 and zfs-kmp-default for the 5.17 kernel arrived at the same time. So I’ve got the 5.18 kernel on disk without ZFS support. Bäm!

I don’t know how a ZFS version for a new kernel version finds its way into Tumbleweed. But I see that ZFS was marked as compatible with 5.18 in its Git repository already two weeks ago. So I am wondering where the new version is and when it will appear!

Hi
It’s broken at present, see the comments here: https://build.opensuse.org/package/show/filesystems/zfs

Hm, last change 3 months ago! Seems this needs an update, however and by whomever this is to be done …

This is a bad idea, in my opinion of course. Stay with default btrfs.

Host erlangen is assembled using mainline hardware components and has installed mainline software only. It’s virtually maintenance free and upgrades as smoothly as possible. Changing this policy would result in asking for serious trouble.

**erlangen:~ #** inxi -zMSCGD -y 132 
**System:    Kernel:** 5.18.2-1-default **arch:** x86_64 **bits:** 64 **Console:** pty pts/1 **Distro:** openSUSE Tumbleweed 20220613 
**Machine:   Type:** Desktop **Mobo:** Micro-Star **model:** B550-A PRO (MS-7C56) **v:** 2.0 **serial:** <filter> **UEFI:** American Megatrends LLC. 
             **v:** A.90 **date:** 03/17/2022 
**CPU:       Info:** 6-core **model:** AMD Ryzen 5 5600X **bits:** 64 **type:** MT MCP **cache:****L2:** 3 MiB 
           **Speed (MHz):****avg:** 2325 **min/max:** 2200/4650 **cores:****1:** 2200 **2:** 2200 **3:** 2200 **4:** 3700 **5:** 2200 **6:** 2200 **7:** 2200 **8:** 2200 
             **9:** 2200 **10:** 2200 **11:** 2200 **12:** 2200 
**Graphics:  Device-1:** AMD Lexa PRO [Radeon 540/540X/550/550X / RX 540X/550/550X] **driver:** amdgpu **v:** kernel 
           **Display:** x11 **server:** X.Org **v:** 21.1.3 **with:** Xwayland **v:** 22.1.2 **driver:****X:****loaded:** amdgpu 
             **unloaded:** fbdev,modesetting,vesa **gpu:** amdgpu **resolution:** 3840x2160~60Hz 
           **OpenGL:****renderer:** AMD Radeon RX 550 / 550 Series (polaris12 LLVM 14.0.4 DRM 3.46 5.18.2-1-default) **v:** 4.6 Mesa 22.1.1 
**Drives:    Local Storage:****total:** 7.28 TiB **used:** 3.89 TiB (53.5%) 
           **ID-1:** /dev/nvme0n1 **vendor:** Samsung **model:** SSD 970 EVO Plus 2TB **size:** 1.82 TiB 
           **ID-2:** /dev/sdb **vendor:** Crucial **model:** CT2000BX500SSD1 **size:** 1.82 TiB 
           **ID-3:** /dev/sdc **vendor:** Western Digital **model:** WD40EZRX-22SPEB0 **size:** 3.64 TiB 
**erlangen:~ #**

Tumbleweed recently changed defaults to duplicate metadata and is now virtually unbreakable:

**erlangen:~ #** btrfs filesystem usage -T /            
Overall: 
    Device size:                   1.72TiB 
    Device allocated:            455.07GiB 
    Device unallocated:            1.28TiB 
    Device missing:                  0.00B 
    Used:                        437.22GiB 
    Free (estimated):              1.29TiB      (min: 670.67GiB) 
    Free (statfs, df):             1.29TiB 
    Data ratio:                       1.00 
    Metadata ratio:                   2.00 
    Global reserve:              512.00MiB      (used: 0.00B) 
    Multiple profiles:                  no 

                  Data      Metadata System               
Id Path           single    DUP      DUP      Unallocated 
-- -------------- --------- -------- -------- ----------- 
 1 /dev/nvme0n1p2 449.01GiB  6.00GiB 64.00MiB     1.28TiB 
-- -------------- --------- -------- -------- ----------- 
   Total          449.01GiB  3.00GiB 32.00MiB     1.28TiB 
   Used           433.23GiB  1.99GiB 64.00KiB             
**erlangen:~ #**

Hm, difficult to stay with btrfs, because I never used it – and probably never will.

Hi
That could be you with an openSUSE Build Service account…

Personally if it were me, would be looking to pull the latest git release and build/test.

It’s easy, keep the options selected by the installer: Install Tumbleweed on 970 EVO using default partitioning

BTRFS and ZFS are not equal.

To OP: use Leap + ZFS, not TW.

Yeah, I’d do that if I knew how to and if I had the time. For now, I’d be happy to understand what the usual process is, provided there is one.

Hi
Add a comment on the OBS page, ping the maintainer via email (check change log), open a bug report (last resort).

FYI see dkms fails on kernel 5.18.0+ · Issue #13562 · openzfs/zfs · GitHub hopefully the next release won’t be far away.

Thanks! Good to learn that this is being actively pursued!

Hi all - apologies for re-starting an old thread but does anyone know if the situation has improved with ZFS on Tumbleweed?

I’m building a new server and want to import an existing ZFS pool from Ubuntu. I’m using MicroOS as this seems to be the direction openSUSE is heading and I love the idea of transactional updates. However, I am not able to get the ZFS kernel modules to load. I have added the filesystems repo and installed with:

transactional-update pkg in zfs
reboot
zpool list
The ZFS modules are not loaded.
Try running ‘/sbin/modprobe zfs’ as root to load them.
/sbin/modprobe zfs
modprobe: FATAL: Module zfs not found in directory /lib/modules/6.3.1-1-default

Is it still the case that the ZFS modules are not keeping-up with the kernel releases on Tumbleweed? If so is Leap the solution, even if it may only be around until next year when it gets replaced with ALP (which I believe is meant to be based on Tumbleweed)?

Thanks.

If you look at the status of zfs package in filesytsems project you will see that it failed to build for Tumbleweed. And if you look at build log, you will see

[  118s] configure:56931: error: 
[  118s] 	*** None of the expected "capability" interfaces were detected.
[  118s] 	*** This may be because your kernel version is newer than what is
[  118s] 	*** supported, or you are using a patched custom kernel with
[  118s] 	*** incompatible modifications.
[  118s] 	***
[  118s] 	*** ZFS Version: zfs-2.1.11-1
[  118s] 	*** Compatible Kernels: 3.10 - 6.2

If you know that upstream zfs is fixed for kernel 6.3, you could ping maintainers of this openSUSE package.

Before asking “if this a solution” you need to define a problem you want to solve. Do you need Tumbleweed in the first place?

Every (open)SUSE release is based on Tumbleweed at some point in time. It does not make it equal to Tumbleweed. Regular releases have entirely different maintenance model and life-cycle.

Thanks for the detailed reply! I guess then that running up-to-date kernels in Tumbleweed does not work well with ZFS.

I would try Leap but a) there is no transactional option which has a long supported future as far as I can tell, and b) there are no official packages for cockpit. Tumbleweed has both of these, but doesn’t work well with ZFS.

Thanks again.

I am afraid I fail to interpret it. Leap does offer “Transactional Server” role during installation.

Yes, systemsmanagement:cockpit fails to build for Leap 15.4 due to missing dependencies.

Well, if you need something less volatile than Tumbleweed with cockpit and transactional option you may look at Leap Micro.

Or you can simply delay update of kernel in Tumbleweed until zfs is fixed.

Thanks again for the suggestions - much appreciated.

According to this link zfs now available for kernel 6.3 :slight_smile:

OpenZFS 2.1.12 Released With Linux 6.3 Compatibility - Phoronix