How to preserve multiple kernels?

A kernel upgrade today left me unable to boot one of systems. That issue is described here

I was expecting to be able to use the previous 5.15.8 kernel to boot from while this is sorted out, but it seems that my system is not keeping any previous kernels. So I’m rather stuck.

My /etc/zypp/zypp.conf has the following:

multiversion = provides:multiversion(kernel)
multiversion.kernels = latest,latest-1,running

My /var/log/zypper.log has the following for when the new kernel was installed/deleted:

2022-01-10 13:39:18 <1> cube(786) [PurgeKernels] --------------------- Starting to mark obsolete kernels --------------
2022-01-10 13:39:18 <1> cube(786) [PurgeKernels] Parsing keep spec: latest,latest-1,running
2022-01-10 13:39:18 <1> cube(786) [PurgeKernels] Searching for obsolete multiversion kernel packages.
2022-01-10 13:39:18 <1> cube(786) [PurgeKernels] Found installed multiversion kernel package (584)kernel-default-5.15.8
2022-01-10 13:39:18 <1> cube(786) [PurgeKernels] Identified as a kernel package 
2022-01-10 13:39:18 <1> cube(786) [PurgeKernels] Found installed multiversion kernel package (585)kernel-default-5.15.1
2022-01-10 13:39:18 <1> cube(786) [PurgeKernels] Identified as a kernel package 
2022-01-10 13:39:18 <1> cube(786) [PurgeKernels] Found installed multiversion kernel package (586)kernel-default-5.15.1
2022-01-10 13:39:18 <1> cube(786) [PurgeKernels] Identified as a kernel package 
2022-01-10 13:39:18 <1> cube(786) [PurgeKernels] Grouped packages: 
2022-01-10 13:39:18 <1> cube(786) [PurgeKernels]        Group ident: default
2022-01-10 13:39:18 <1> cube(786) [PurgeKernels]         Group type: 1
2022-01-10 13:39:18 <1> cube(786) [PurgeKernels]         Group flav: default
2022-01-10 13:39:18 <1> cube(786) [PurgeKernels]                Arch: x86_64
2022-01-10 13:39:18 <1> cube(786) [PurgeKernels]                        Edition: 5.15.8-1.1
2022-01-10 13:39:18 <1> cube(786) [PurgeKernels]                                 (584)kernel-default-5.15.8-1.1.x86_64(@System)
2022-01-10 13:39:18 <1> cube(786) [PurgeKernels]                        Edition: 5.15.12-1.2
2022-01-10 13:39:18 <1> cube(786) [PurgeKernels]                                 (585)kernel-default-5.15.12-1.2.x86_64(@System)
2022-01-10 13:39:18 <1> cube(786) [PurgeKernels]                        Edition: 5.15.12-1.3
2022-01-10 13:39:18 <1> cube(786) [PurgeKernels]                                 (586)kernel-default-5.15.12-1.3.x86_64(@System)
2022-01-10 13:39:18 <1> cube(786) [PurgeKernels] Starting with group default
2022-01-10 13:39:18 <1> cube(786) [PurgeKernels] Starting with arch x86_64
2022-01-10 13:39:18 <1> cube(786) [PurgeKernels] Matching packages against running kernel default-x86_64
2022-01-10 13:39:18 <1> cube(786) [PurgeKernels] Variants:
2022-01-10 13:39:18 <1> cube(786) [PurgeKernels] 5.15.12-1
2022-01-10 13:39:18 <1> cube(786) [PurgeKernels] 
2022-01-10 13:39:18 <1> cube(786) [PurgeKernels] Found possible running candidate edition: 5.15.12-1.2
2022-01-10 13:39:18 <1> cube(786) [PurgeKernels] Found possible more recent running candidate edition: 5.15.12-1.3
2022-01-10 13:39:18 <1> cube(786) [PurgeKernels] Marking package (586)kernel-default-5.15.12-1.3.x86_64(@System) as to keep.
2022-01-10 13:39:18 <1> cube(786) [PurgeKernels] Marking package (585)kernel-default-5.15.12-1.2.x86_64(@System) as to keep.
2022-01-10 13:39:18 <1> cube(786) [PurgeKernels] Marking package (586)kernel-default-5.15.12-1.3.x86_64(@System) as to keep.

But then later, it has:

2022-01-10 13:39:18 <1> cube(786) [PurgeKernels] Request to remove package: I__s_(584)kernel-default-5.15.8-1.1.x86_6
2022-01-10 13:39:18 <1> cube(786) [PurgeKernels] Successfully marked package: I_Tu_u(584)kernel-default-5.15.8-1.1.x8
6_64(@System) for removal.
2022-01-10 13:39:18 <1> cube(786) [zypp::solver] -------------- Calling SAT Solver -------------------
2022-01-10 13:39:18 <1> cube(786) [zypp::solver] SATResolver::resolvePool()
2022-01-10 13:39:18 <1> cube(786) [zypp::solver] SATResolver::solverInit()
2022-01-10 13:39:18 <1> cube(786) [zypp::solver] Delete I_Tu_u(584)kernel-default-5.15.8-1.1.x86_64(@System)

So I’m a bit confused. It seems to mark three 5.15.12 kernels “to keep” (one duplicated), so it ended up deleting the prevous 5.15.8 because there were a couple of 5.15.12 versions (1.2 and 1.3) which it thought were installed, though only one of them was.

Luckily, the 5.15.12 works on this machine (called “cube”), but my other machine fails with this kernel, and has no fallback.

My /boot just has this:

-rw-r--r-- 1 root root     1725 Jan  2 15:02 boot.readme
lrwxrwxrwx 1 root root       43 Jan  3 16:09 config-5.15.12-1-default -> ../usr/lib/modules/5.15.12-1-default/config
drwxr-xr-x 3 root root     4096 Jan  1  1970 efi
drwxr-xr-x 6 root root     4096 Jan 10 13:39 grub2
lrwxrwxrwx 1 root root       24 Jan 10 13:36 initrd -> initrd-5.15.12-1-default
-rw------- 1 root root 21152875 Jan 10 13:38 initrd-5.15.12-1-default
lrwxrwxrwx 1 root root       48 Jan  3 16:09 sysctl.conf-5.15.12-1-default -> ../usr/lib/modules/5.15.12-1-default/sysctl.conf
lrwxrwxrwx 1 root root       47 Jan  3 16:09 -> ../usr/lib/modules/5.15.12-1-default/
lrwxrwxrwx 1 root root       25 Jan 10 13:36 vmlinuz -> vmlinuz-5.15.12-1-default
lrwxrwxrwx 1 root root       44 Jan  3 16:09 vmlinuz-5.15.12-1-default -> ../usr/lib/modules/5.15.12-1-default/vmlinuz

Is the solution just to add more “latest-n” options to the zypp.conf??

Yes, thanks, I did read that. And zypp.conf is set according to those rules.

If I grep for kernel-default in the /var/log/zypp/history file, I get this sequence of installs and removes:

2022-01-03 15:38:43|install|kernel-default|5.15.8-1.1|x86_64|4071:ruby.ruby2.7|openSUSE-20211227-0|04c701a317a44614cae09b8cd190d1dc0bb3091f1b8eeaa7264cd0e0402648a0|
2022-01-03 16:09:19|install|kernel-default|5.15.12-1.1|x86_64||repo-oss|f562ebeb0cf49b7020a0cfdcd3e436527a4b857c41d5005f7a23ada2edef1ad8|
2022-01-08 13:32:08|install|kernel-default|5.15.12-1.2|x86_64||repo-oss|1d7fa7ace94c009d60abf57c198801a0899a24f22d29afded3b1d35ca1b99d0d|
2022-01-08 13:32:08|remove |kernel-default|5.15.12-1.1|x86_64||
2022-01-10 13:36:35|install|kernel-default|5.15.12-1.3|x86_64||repo-oss|ccb1e8a0f7809e351c5479f279b748c005feff884a3182c42f1eb03b914a6672|
2022-01-10 13:39:21|remove |kernel-default|5.15.8-1.1|x86_64|root@cube|

So why didn’t the install of 12-1.2 remove 8-1.1 instead of 12-1.1?

And why don’t 12-1.2 and 12-1.3 both appear in the boot menu separately? (I suspect because on disk, there is only “5.15.12-1” available, which is actually 12-1.3).

So perhaps zypper’s counting is thrown off when there are too many updates within the same “minor number”?

Normally, a kernel install does not remove any older kernel. The older kernel is removed by the purge-kernels service on the next reboot.

In this case, however, 12-1.2 conflicts with 12-1.1, and that was the reason for the removal.

That conflict is in the package. But it is there for a reason. 12-1.2 is just a recompile of the same code, and will use the same file names as 12-1.1. You cannot have both installed, because of the file name conflict.[/QUOTE]

IMO this, roughly speaking, was your misfortune. I never run into it because I keep my TW & Leap kernels locked. I decide when I want a new kernel installed or an old removed. When a wildcard lock format is used, zypper will offer to “remove” the lock in order to perform the requested installation or removal, but what it does in fact is to simply ignore the lock for purposes of the instant transaction request if you answer (1) “remove” to its request. Some of my TW installations still carry kernels from years ago, e.g. 5.3.x was on most of them until a few weeks ago when I started culling the oldest ones. One even had some 4.x.x kernels.

To fix yours apparently is going to require a rescue boot in order to chroot into the installed system for purposes of running dracut (rebuild initrd) and/or installing a newer kernel. 5.16 is now available using this repo:

name=Kernel builds for branch stable (standard)

The question is not why 12-1.1 was removed (this is clear), but why 12-1.2 was not removed when 12-1.3 was installed. If this happened on several systems, it sounds like a bug that needs to be reported.

Nope. This machine has:

**i3-4130:~ #** grep 'install|kernel-default|5.15' /var/log/zypp/history 
2021-11-20 06:50:00|**install|kernel-default|5.15**.2-1.1|x86_64|||0e2f98c44bd5e58fe9b54ae8a5d6f3df67461e2734ea9a5da5133c9fbf0f7c81| 
2021-11-24 21:28:20|**install|kernel-default|5.15**.3-1.3|x86_64|||01f979b207238fbcab0987d5d352d7fd16a15fcd517a28050452cfbc014f399d| 
2021-12-01 04:33:22|**install|kernel-default|5.15**.5-1.1|x86_64|||e7cd9a5eb23bdea5cf501df7857216fb6816293144fa417e8bcfe48b8bce5337| 
2021-12-07 04:37:16|**install|kernel-default|5.15**.5-1.2|x86_64|||a4ff25841c2d05e33562ff45e367a0cf9edc1831b163d5d0bfb9d687d200a63a| 
2021-12-08 21:13:46|**install|kernel-default|5.15**.6-1.3|x86_64|||546a99e5fec1736e61442afd97eee6c833f9595c883766bab41699b877150538| 
2021-12-14 08:02:06|**install|kernel-default|5.15**.7-1.2|x86_64|||9434116d6e61865d7307e4abcf4ed3a4bdbffa31505db3b94a22da58cd171210| 
2021-12-19 19:48:22|**install|kernel-default|5.15**.8-1.1|x86_64|||04c701a317a44614cae09b8cd190d1dc0bb3091f1b8eeaa7264cd0e0402648a0| 
2022-01-06 18:24:58|**install|kernel-default|5.15**.12-1.1|x86_64|||f562ebeb0cf49b7020a0cfdcd3e436527a4b857c41d5005f7a23ada2edef1ad8| 
2022-01-07 17:32:16|**install|kernel-default|5.15**.12-1.2|x86_64|||1d7fa7ace94c009d60abf57c198801a0899a24f22d29afded3b1d35ca1b99d0d| 
2022-01-09 23:27:02|**install|kernel-default|5.15**.12-1.3|x86_64|||ccb1e8a0f7809e351c5479f279b748c005feff884a3182c42f1eb03b914a6672| 
**i3-4130:~ #**

All installs are from standard repo. One of your installs (marked in bold) is murky and may have caused the problem. When running into upgrade problems always check your list of repos first:

i3-4130:~ # zypper lr -uEP
# | Alias                            | Name                            | Enabled | GPG Check | Refresh | Priority | URI
7 | packman                          | packman                         | Yes     | (r ) Yes  | Yes     |   90     |
3 |    | Haupt-Repository (NON-OSS)      | Yes     | (r ) Yes  | Yes     |   99     |
4 |        | Haupt-Repository (OSS)          | Yes     | (r ) Yes  | Yes     |   99     |
5 | | Hauptaktualisierungs-Repository | Yes     | (r ) Yes  | Yes     |   99     |
i3-4130:~ # 

Given the trouble this has caused me, I think I will do the same.

If I set the “kernel-default” to “Do not modify” in YaST presumably that will keep it fixed, but then how can I install newer versions?

Rather than try to repair the machine, I have just re-installed from the USB stick that I still had from a couple of weeks ago. At some point I will have to bite the bullet and upgrade the other packages, but I would like to be absolutely sure that the 5.15.8 will not be removed before I do that.

Lock it:

zypper al kernel-de*

Install newest one:

zypper in kernel-default

I don’t use YaST much. Mainly I use YaST Software only when installing, as this morning, when installing 15.4 Alpha.

OK, I see, thanks. I’ll wait until the usual TW repos have a version greater than 5.15.12, then give this a try. I think I’ve set zypp.conf to keep all kernels for now, too. And I can see how to dry-run the “purge-kernels” to check what will happen on a reboot. Fingers crossed :slight_smile:

It looks like 5.16.x will be the next TW kernel.

Just to let everyone know, the 5.16.0-1 kernel is now out, and I successfully updated. That is, the zypper install is happy to “ignore” the lock temporarily (option 1) and leaves the package locked afterwards. The reboot purge-kernels does nothing (because of zypp.conf) and the grub advanced submenu has both kernels available as an option.

Success! PHEW!! :slight_smile:

Many thanks for the help.