How to preserve multiple kernels?

A kernel upgrade today left me unable to boot one of systems. That issue is described here https://forums.opensuse.org/showthread.php/564928-Cannot-boot-with-Kernel-5-15-12.

I was expecting to be able to use the previous 5.15.8 kernel to boot from while this is sorted out, but it seems that my system is not keeping any previous kernels. So I’m rather stuck.

My /etc/zypp/zypp.conf has the following:

multiversion = provides:multiversion(kernel)
multiversion.kernels = latest,latest-1,running

My /var/log/zypper.log has the following for when the new kernel was installed/deleted:

2022-01-10 13:39:18 <1> cube(786) [PurgeKernels] PurgeKernels.cc(markObsoleteKernels):487 --------------------- Starting to mark obsolete kernels --------------
-------
2022-01-10 13:39:18 <1> cube(786) [PurgeKernels] PurgeKernels.cc(parseKeepSpec):308 Parsing keep spec: latest,latest-1,running
2022-01-10 13:39:18 <1> cube(786) [PurgeKernels] PurgeKernels.cc(markObsoleteKernels):577 Searching for obsolete multiversion kernel packages.
2022-01-10 13:39:18 <1> cube(786) [PurgeKernels] PurgeKernels.cc(markObsoleteKernels):581 Found installed multiversion kernel package (584)kernel-default-5.15.8
-1.1.x86_64(@System)
2022-01-10 13:39:18 <1> cube(786) [PurgeKernels] PurgeKernels.cc(markObsoleteKernels):584 Identified as a kernel package 
2022-01-10 13:39:18 <1> cube(786) [PurgeKernels] PurgeKernels.cc(markObsoleteKernels):581 Found installed multiversion kernel package (585)kernel-default-5.15.1
2-1.2.x86_64(@System)
2022-01-10 13:39:18 <1> cube(786) [PurgeKernels] PurgeKernels.cc(markObsoleteKernels):584 Identified as a kernel package 
2022-01-10 13:39:18 <1> cube(786) [PurgeKernels] PurgeKernels.cc(markObsoleteKernels):581 Found installed multiversion kernel package (586)kernel-default-5.15.1
2-1.3.x86_64(@System)
2022-01-10 13:39:18 <1> cube(786) [PurgeKernels] PurgeKernels.cc(markObsoleteKernels):584 Identified as a kernel package 
2022-01-10 13:39:18 <1> cube(786) [PurgeKernels] PurgeKernels.cc(markObsoleteKernels):644 Grouped packages: 
2022-01-10 13:39:18 <1> cube(786) [PurgeKernels] PurgeKernels.cc(operator()):646        Group ident: default
2022-01-10 13:39:18 <1> cube(786) [PurgeKernels] PurgeKernels.cc(operator()):647         Group type: 1
2022-01-10 13:39:18 <1> cube(786) [PurgeKernels] PurgeKernels.cc(operator()):648         Group flav: default
2022-01-10 13:39:18 <1> cube(786) [PurgeKernels] PurgeKernels.cc(operator()):650                Arch: x86_64
2022-01-10 13:39:18 <1> cube(786) [PurgeKernels] PurgeKernels.cc(operator()):652                        Edition: 5.15.8-1.1
2022-01-10 13:39:18 <1> cube(786) [PurgeKernels] PurgeKernels.cc(operator()):654                                 (584)kernel-default-5.15.8-1.1.x86_64(@System)
2022-01-10 13:39:18 <1> cube(786) [PurgeKernels] PurgeKernels.cc(operator()):652                        Edition: 5.15.12-1.2
2022-01-10 13:39:18 <1> cube(786) [PurgeKernels] PurgeKernels.cc(operator()):654                                 (585)kernel-default-5.15.12-1.2.x86_64(@System)
2022-01-10 13:39:18 <1> cube(786) [PurgeKernels] PurgeKernels.cc(operator()):652                        Edition: 5.15.12-1.3
2022-01-10 13:39:18 <1> cube(786) [PurgeKernels] PurgeKernels.cc(operator()):654                                 (586)kernel-default-5.15.12-1.3.x86_64(@System)
2022-01-10 13:39:18 <1> cube(786) [PurgeKernels] PurgeKernels.cc(fillKeepList):397 Starting with group default
2022-01-10 13:39:18 <1> cube(786) [PurgeKernels] PurgeKernels.cc(fillKeepList):401 Starting with arch x86_64
2022-01-10 13:39:18 <1> cube(786) [PurgeKernels] PurgeKernels.cc(fillKeepList):413 Matching packages against running kernel default-x86_64
2022-01-10 13:39:18 <1> cube(786) [PurgeKernels] PurgeKernels.cc(fillKeepList):413 Variants:
2022-01-10 13:39:18 <1> cube(786) [PurgeKernels] PurgeKernels.cc(fillKeepList):415 5.15.12-1
2022-01-10 13:39:18 <1> cube(786) [PurgeKernels] PurgeKernels.cc(fillKeepList):416 
2022-01-10 13:39:18 <1> cube(786) [PurgeKernels] PurgeKernels.cc(fillKeepList):439 Found possible running candidate edition: 5.15.12-1.2
2022-01-10 13:39:18 <1> cube(786) [PurgeKernels] PurgeKernels.cc(fillKeepList):442 Found possible more recent running candidate edition: 5.15.12-1.3
2022-01-10 13:39:18 <1> cube(786) [PurgeKernels] PurgeKernels.cc(operator()):383 Marking package (586)kernel-default-5.15.12-1.3.x86_64(@System) as to keep.
2022-01-10 13:39:18 <1> cube(786) [PurgeKernels] PurgeKernels.cc(operator()):383 Marking package (585)kernel-default-5.15.12-1.2.x86_64(@System) as to keep.
2022-01-10 13:39:18 <1> cube(786) [PurgeKernels] PurgeKernels.cc(operator()):383 Marking package (586)kernel-default-5.15.12-1.3.x86_64(@System) as to keep.

But then later, it has:

2022-01-10 13:39:18 <1> cube(786) [PurgeKernels] PurgeKernels.cc(removePackageAndCheck):157 Request to remove package: I__s_(584)kernel-default-5.15.8-1.1.x86_6
4(@System)
....
2022-01-10 13:39:18 <1> cube(786) [PurgeKernels] PurgeKernels.cc(removePackageAndCheck):241 Successfully marked package: I_Tu_u(584)kernel-default-5.15.8-1.1.x8
6_64(@System) for removal.
...
2022-01-10 13:39:18 <1> cube(786) [zypp::solver] Resolver.cc(solverInit):297 -------------- Calling SAT Solver -------------------
2022-01-10 13:39:18 <1> cube(786) [zypp::solver] SATResolver.cc(resolvePool):761 SATResolver::resolvePool()
2022-01-10 13:39:18 <1> cube(786) [zypp::solver] SATResolver.cc(solverInit):666 SATResolver::solverInit()
2022-01-10 13:39:18 <1> cube(786) [zypp::solver] SATResolver.cc(resolvePool):782 Delete I_Tu_u(584)kernel-default-5.15.8-1.1.x86_64(@System)

So I’m a bit confused. It seems to mark three 5.15.12 kernels “to keep” (one duplicated), so it ended up deleting the prevous 5.15.8 because there were a couple of 5.15.12 versions (1.2 and 1.3) which it thought were installed, though only one of them was.

Luckily, the 5.15.12 works on this machine (called “cube”), but my other machine fails with this kernel, and has no fallback.

My /boot just has this:

-rw-r--r-- 1 root root     1725 Jan  2 15:02 boot.readme
lrwxrwxrwx 1 root root       43 Jan  3 16:09 config-5.15.12-1-default -> ../usr/lib/modules/5.15.12-1-default/config
drwxr-xr-x 3 root root     4096 Jan  1  1970 efi
drwxr-xr-x 6 root root     4096 Jan 10 13:39 grub2
lrwxrwxrwx 1 root root       24 Jan 10 13:36 initrd -> initrd-5.15.12-1-default
-rw------- 1 root root 21152875 Jan 10 13:38 initrd-5.15.12-1-default
lrwxrwxrwx 1 root root       48 Jan  3 16:09 sysctl.conf-5.15.12-1-default -> ../usr/lib/modules/5.15.12-1-default/sysctl.conf
lrwxrwxrwx 1 root root       47 Jan  3 16:09 System.map-5.15.12-1-default -> ../usr/lib/modules/5.15.12-1-default/System.map
lrwxrwxrwx 1 root root       25 Jan 10 13:36 vmlinuz -> vmlinuz-5.15.12-1-default
lrwxrwxrwx 1 root root       44 Jan  3 16:09 vmlinuz-5.15.12-1-default -> ../usr/lib/modules/5.15.12-1-default/vmlinuz

Is the solution just to add more “latest-n” options to the zypp.conf??

https://en.opensuse.org/SDB:Keep_multiple_kernel_versions

Yes, thanks, I did read that. And zypp.conf is set according to those rules.

If I grep for kernel-default in the /var/log/zypp/history file, I get this sequence of installs and removes:


2022-01-03 15:38:43|install|kernel-default|5.15.8-1.1|x86_64|4071:ruby.ruby2.7|openSUSE-20211227-0|04c701a317a44614cae09b8cd190d1dc0bb3091f1b8eeaa7264cd0e0402648a0|
2022-01-03 16:09:19|install|kernel-default|5.15.12-1.1|x86_64||repo-oss|f562ebeb0cf49b7020a0cfdcd3e436527a4b857c41d5005f7a23ada2edef1ad8|
2022-01-08 13:32:08|install|kernel-default|5.15.12-1.2|x86_64||repo-oss|1d7fa7ace94c009d60abf57c198801a0899a24f22d29afded3b1d35ca1b99d0d|
2022-01-08 13:32:08|remove |kernel-default|5.15.12-1.1|x86_64||
2022-01-10 13:36:35|install|kernel-default|5.15.12-1.3|x86_64||repo-oss|ccb1e8a0f7809e351c5479f279b748c005feff884a3182c42f1eb03b914a6672|
2022-01-10 13:39:21|remove |kernel-default|5.15.8-1.1|x86_64|root@cube|

So why didn’t the install of 12-1.2 remove 8-1.1 instead of 12-1.1?

And why don’t 12-1.2 and 12-1.3 both appear in the boot menu separately? (I suspect because on disk, there is only “5.15.12-1” available, which is actually 12-1.3).

So perhaps zypper’s counting is thrown off when there are too many updates within the same “minor number”?

Normally, a kernel install does not remove any older kernel. The older kernel is removed by the purge-kernels service on the next reboot.

In this case, however, 12-1.2 conflicts with 12-1.1, and that was the reason for the removal.

That conflict is in the package. But it is there for a reason. 12-1.2 is just a recompile of the same code, and will use the same file names as 12-1.1. You cannot have both installed, because of the file name conflict.[/QUOTE]

IMO this, roughly speaking, was your misfortune. I never run into it because I keep my TW & Leap kernels locked. I decide when I want a new kernel installed or an old removed. When a wildcard lock format is used, zypper will offer to “remove” the lock in order to perform the requested installation or removal, but what it does in fact is to simply ignore the lock for purposes of the instant transaction request if you answer (1) “remove” to its request. Some of my TW installations still carry kernels from years ago, e.g. 5.3.x was on most of them until a few weeks ago when I started culling the oldest ones. One even had some 4.x.x kernels.

To fix yours apparently is going to require a rescue boot in order to chroot into the installed system for purposes of running dracut (rebuild initrd) and/or installing a newer kernel. 5.16 is now available using this repo:

[Kernel_stable]
name=Kernel builds for branch stable (standard)
type=rpm-md
baseurl=https://download.opensuse.org/repositories/Kernel:/stable/standard/
gpgcheck=1
gpgkey=https://download.opensuse.org/repositories/Kernel:/stable/standard/repodata/repomd.xml.key
enabled=1

The question is not why 12-1.1 was removed (this is clear), but why 12-1.2 was not removed when 12-1.3 was installed. If this happened on several systems, it sounds like a bug that needs to be reported.

Nope. This machine has:

**i3-4130:~ #** grep 'install|kernel-default|5.15' /var/log/zypp/history 
2021-11-20 06:50:00|**install|kernel-default|5.15**.2-1.1|x86_64||download.opensuse.org-oss|0e2f98c44bd5e58fe9b54ae8a5d6f3df67461e2734ea9a5da5133c9fbf0f7c81| 
2021-11-24 21:28:20|**install|kernel-default|5.15**.3-1.3|x86_64||download.opensuse.org-oss|01f979b207238fbcab0987d5d352d7fd16a15fcd517a28050452cfbc014f399d| 
2021-12-01 04:33:22|**install|kernel-default|5.15**.5-1.1|x86_64||download.opensuse.org-oss|e7cd9a5eb23bdea5cf501df7857216fb6816293144fa417e8bcfe48b8bce5337| 
2021-12-07 04:37:16|**install|kernel-default|5.15**.5-1.2|x86_64||download.opensuse.org-oss|a4ff25841c2d05e33562ff45e367a0cf9edc1831b163d5d0bfb9d687d200a63a| 
2021-12-08 21:13:46|**install|kernel-default|5.15**.6-1.3|x86_64||download.opensuse.org-oss|546a99e5fec1736e61442afd97eee6c833f9595c883766bab41699b877150538| 
2021-12-14 08:02:06|**install|kernel-default|5.15**.7-1.2|x86_64||download.opensuse.org-oss|9434116d6e61865d7307e4abcf4ed3a4bdbffa31505db3b94a22da58cd171210| 
2021-12-19 19:48:22|**install|kernel-default|5.15**.8-1.1|x86_64||download.opensuse.org-oss|04c701a317a44614cae09b8cd190d1dc0bb3091f1b8eeaa7264cd0e0402648a0| 
2022-01-06 18:24:58|**install|kernel-default|5.15**.12-1.1|x86_64||download.opensuse.org-oss|f562ebeb0cf49b7020a0cfdcd3e436527a4b857c41d5005f7a23ada2edef1ad8| 
2022-01-07 17:32:16|**install|kernel-default|5.15**.12-1.2|x86_64||download.opensuse.org-oss|1d7fa7ace94c009d60abf57c198801a0899a24f22d29afded3b1d35ca1b99d0d| 
2022-01-09 23:27:02|**install|kernel-default|5.15**.12-1.3|x86_64||download.opensuse.org-oss|ccb1e8a0f7809e351c5479f279b748c005feff884a3182c42f1eb03b914a6672| 
**i3-4130:~ #**

All installs are from standard download.opensuse.org-oss repo. One of your installs (marked in bold) is murky and may have caused the problem. When running into upgrade problems always check your list of repos first:

i3-4130:~ # zypper lr -uEP
# | Alias                            | Name                            | Enabled | GPG Check | Refresh | Priority | URI
--+----------------------------------+---------------------------------+---------+-----------+---------+----------+------------------------------------------------------
7 | packman                          | packman                         | Yes     | (r ) Yes  | Yes     |   90     | http://ftp.fau.de/packman/suse/openSUSE_Tumbleweed/
3 | download.opensuse.org-non-oss    | Haupt-Repository (NON-OSS)      | Yes     | (r ) Yes  | Yes     |   99     | http://download.opensuse.org/tumbleweed/repo/non-oss/
4 | download.opensuse.org-oss        | Haupt-Repository (OSS)          | Yes     | (r ) Yes  | Yes     |   99     | http://download.opensuse.org/tumbleweed/repo/oss/
5 | download.opensuse.org-tumbleweed | Hauptaktualisierungs-Repository | Yes     | (r ) Yes  | Yes     |   99     | http://download.opensuse.org/update/tumbleweed/
i3-4130:~ # 

Given the trouble this has caused me, I think I will do the same.

If I set the “kernel-default” to “Do not modify” in YaST presumably that will keep it fixed, but then how can I install newer versions?

Rather than try to repair the machine, I have just re-installed from the USB stick that I still had from a couple of weeks ago. At some point I will have to bite the bullet and upgrade the other packages, but I would like to be absolutely sure that the 5.15.8 will not be removed before I do that.

Lock it:

zypper al kernel-de*

Install newest one:

zypper in kernel-default

I don’t use YaST much. Mainly I use YaST Software only when installing, as this morning, when installing 15.4 Alpha.

OK, I see, thanks. I’ll wait until the usual TW repos have a version greater than 5.15.12, then give this a try. I think I’ve set zypp.conf to keep all kernels for now, too. And I can see how to dry-run the “purge-kernels” to check what will happen on a reboot. Fingers crossed :slight_smile:

It looks like 5.16.x will be the next TW kernel.

Just to let everyone know, the 5.16.0-1 kernel is now out, and I successfully updated. That is, the zypper install is happy to “ignore” the lock temporarily (option 1) and leaves the package locked afterwards. The reboot purge-kernels does nothing (because of zypp.conf) and the grub advanced submenu has both kernels available as an option.

Success! PHEW!! :slight_smile:

Many thanks for the help.