Boot previous kernel with nvidia

awerlang · December 1, 2021, 6:29am

Hi!

So current 5.15 kernel broke bluetooth as for many people as well. So I’m attempting to boot from 5.14 but nvidia is complaining client api mismatch:

 1526.097007] NVRM: API mismatch: the client has the version 470.86, but
 1526.097007] NVRM: this kernel module has the version 470.82.  Please

I reinstalled nvidia-gfxG05-kmp-default, didn’t work.
Reinstalled both nvidia-gfxG05-kmp-default and x11-video-nvidiaG05, neither.
Run dracut -f --regenerate-all, nope.
Moved symlink /boot/initrd to match initrd for 5.14, not sure about this.

What I’m missing? Thanks in advance!

nrickert · December 1, 2021, 6:59am

If you installed Nvidia drivers when you were running that 5.14 kernel, then there should be a correct module for it. If you install nvidia the hard way, maybe you need to reinstall while running the 5.14 kernel.

NOTE: I am not an nvidia expert. So I could be mistaken about this.

arvidjaar · December 1, 2021, 7:05am

Show output of "rpm -qa “nvidia-*”

awerlang · December 1, 2021, 7:17am

Sorry, forgot to mention that I use the easy way i.e. the repo. Usually I waited until kernel updated past ~x.y.4 to install a major kernel but since this was a boring year I decided to install as soon as a major arrives as I thought it would be easy to boot from old kernel in case of need. Turns out not so easy. Some issues attempting to boot from a previous snapshot too. But I digress…

I guess the backslash is extraneous there as it returned no packages. Removing it yield:


nvidia-computeG05-470.86-46.1.x86_64
nvidia-glG05-470.86-46.1.x86_64
nvidia-gfxG05-kmp-default-470.86_k5.14.14_1-46.1.x86_64
nvidia-container-toolkit-0.0+git.1580519869.60f165a-1.8.x86_64

conram · December 1, 2021, 2:47pm

You can probably install the new new nvidia version while using the older kernel version.
Run nvidia-xconfig and do mkinitrd.

I am using the .run file so not much knowledge with the rpm version of nvidia driver.

awerlang · December 1, 2021, 3:31pm

This is an excerpt of zypper logs.


2021-11-10 00:46:37|install|kernel-default|5.14.14-1.3|x86_64||repo-oss|3a79a0c67053deb6cd276f62511dba29efcbe5dff3a35ed774f3aee85c23f9fd|
2021-11-10 00:49:04|install|nvidia-gfxG05-kmp-default|470.82.00_k5.14.11_2-45.1|x86_64||NVIDIA|17268ca16984a3d12cdd68b2604aacaf5824848bed7b6b8caa2bb70796a89d68|
2021-11-10 00:49:05|install|nvidia-glG05|470.82.00-45.1|x86_64||NVIDIA|d606a373803570c7de54664df62ad2eb6cc4f4b6a4269e347b8bb37a876a8bcf|
2021-11-10 00:49:06|install|nvidia-computeG05|470.82.00-45.1|x86_64||NVIDIA|cfeb0d62dcc62c9701acef1653ce21aa71276748835c51503e2eb5a0b8acf1a9|
2021-11-10 00:49:10|install|x11-video-nvidiaG05|470.82.00-45.1|x86_64||NVIDIA|4ae4004cac552d62f89cf3808862cec80bfa7fc4da8d39bb3612bb7c4b3e340c|
2021-11-23 00:59:28|install|kernel-firmware-nvidia|20211115-1.1|noarch||repo-oss|739267c436084115868d8f38358cb0311bc5cf67cf599454f91069d661f7956e|
2021-11-23 01:02:18|install|kernel-default-devel|5.15.2-1.1|x86_64||repo-oss|11e91679337c3bd03c46461e406982e01c18e2b9ee6a51e6da46e7983bd4cf50|
2021-11-23 01:02:53|install|kernel-default|5.15.2-1.1|x86_64||repo-oss|0e2f98c44bd5e58fe9b54ae8a5d6f3df67461e2734ea9a5da5133c9fbf0f7c81|
2021-11-23 01:05:37|install|nvidia-gfxG05-kmp-default|470.86_k5.14.14_1-46.1|x86_64||NVIDIA|3fa5611ec7e9bc2e372af6596fa144f51b3525214b18406797cdf30804b41d11|
2021-11-23 01:05:38|install|nvidia-glG05|470.86-46.1|x86_64||NVIDIA|976bc0430a76e5c2f1e281d170b1416a5c2263cbab1d77b8be212ccfba6f5e6e|
2021-11-23 01:05:39|install|nvidia-computeG05|470.86-46.1|x86_64||NVIDIA|2e49ae898f3825f6446b1e0bda181751060787992af7ba3108db02b3468d571a|
2021-11-23 01:05:43|install|x11-video-nvidiaG05|470.86-46.1|x86_64||NVIDIA|02103a1e73d132c41a0bc9fcd9951914511864f7586f9daad635b052cd744bc0|
2021-11-24 00:41:13|install|kernel-default-devel|5.15.3-1.3|x86_64||repo-oss|c23da68a0de7ea2117b223d9c1b52db6ab92543965f230eedf43d569d31cee85|
2021-11-24 00:41:41|install|kernel-default|5.15.3-1.3|x86_64||repo-oss|01f979b207238fbcab0987d5d352d7fd16a15fcd517a28050452cfbc014f399d|
2021-11-30 11:00:25|install|nvidia-gfxG05-kmp-default|470.86_k5.14.14_1-46.1|x86_64|root@rivendell|NVIDIA|3fa5611ec7e9bc2e372af6596fa144f51b3525214b18406797cdf30804b41d11|
2021-12-01 01:52:18|install|nvidia-gfxG05-kmp-default|470.86_k5.14.14_1-46.1|x86_64|root@rivendell|NVIDIA|3fa5611ec7e9bc2e372af6596fa144f51b3525214b18406797cdf30804b41d11|
2021-12-01 01:52:23|install|x11-video-nvidiaG05|470.86-46.1|x86_64|root@rivendell|NVIDIA|02103a1e73d132c41a0bc9fcd9951914511864f7586f9daad635b052cd744bc0|

awerlang · December 1, 2021, 3:54pm

That’s about steps #2 and #3 above. Except I didn’t run nvidia-xconfig, xorg.conf.d/50-nvidia.conf doesn’t seem to contain anything relevant.

arvidjaar · December 1, 2021, 8:21pm

awerlang:

This is an excerpt of zypper logs.


2021-11-10 00:46:37|install|kernel-default|5.14.14-1.3|x86_64||repo-oss|3a79a0c67053deb6cd276f62511dba29efcbe5dff3a35ed774f3aee85c23f9fd|
2021-11-10 00:49:04|install|nvidia-gfxG05-kmp-default|470.82.00_k5.14.11_2-45.1|x86_64||NVIDIA|17268ca16984a3d12cdd68b2604aacaf5824848bed7b6b8caa2bb70796a89d68|

It installs nvidia 470.82 into kernel 5.4.14-1. Your log does not show any package removal so presumably this package is still present. KMP will not be replaced if it is already present. Show “ls -l /lib/modules/5.4.14-1-default/{updates,weak-updates/*}”.

awerlang · December 1, 2021, 9:55pm

$ ls -l /lib/modules/5.14.14-1-default/{updates,weak-updates/*}

/lib/modules/5.14.14-1-default/updates:
total 105212
-rw-r--r-- 1 root root  4517552 Nov 10 00:48 nvidia-drm.ko
-rw-r--r-- 1 root root 52933888 Nov 10 00:48 nvidia.ko
-rw-r--r-- 1 root root  2294832 Nov 10 00:48 nvidia-modeset.ko
-rw-r--r-- 1 root root    77320 Nov 10 00:48 nvidia-peermem.ko
-rw-r--r-- 1 root root 47903528 Nov 10 00:48 nvidia-uvm.ko

$ zypper se -si 'kernel-def'
Loading repository data...
Reading installed packages...

S  | Name                 | Type    | Version     | Arch   | Repository
---+----------------------+---------+-------------+--------+------------------
i+ | kernel-default       | package | 5.15.3-1.3  | x86_64 | (System Packages)
i+ | kernel-default       | package | 5.15.2-1.1  | x86_64 | (System Packages)
i+ | kernel-default       | package | 5.14.14-1.3 | x86_64 | (System Packages)
i+ | kernel-default-devel | package | 5.15.3-1.3  | x86_64 | (System Packages)
i+ | kernel-default-devel | package | 5.15.2-1.1  | x86_64 | (System Packages)
i+ | kernel-default-devel | package | 5.14.14-1.3 | x86_64 | (System Packages)

$ zypper se -si 'nvidia'
Loading repository data...
Reading installed packages...

S  | Name                      | Type    | Version                        | Arch   | Repository
---+---------------------------+---------+--------------------------------+--------+------------------------
i+ | kernel-firmware-nvidia    | package | 20211115-1.1                   | noarch | openSUSE-Tumbleweed-Oss
i  | libnvidia-container-tools | package | 1.1.1-4.1                      | x86_64 | openSUSE-Tumbleweed-Oss
i  | libnvidia-container1      | package | 1.1.1-4.1                      | x86_64 | openSUSE-Tumbleweed-Oss
i  | libnvidia-egl-wayland1    | package | 1.1.7-3.1                      | x86_64 | openSUSE-Tumbleweed-Oss
i  | nvidia-computeG05         | package | 470.86-46.1                    | x86_64 | NVIDIA
il | nvidia-container-toolkit  | package | 0.0+git.1580519869.60f165a-1.8 | x86_64 | (System Packages)
i+ | nvidia-gfxG05-kmp-default | package | 470.86_k5.14.14_1-46.1         | x86_64 | NVIDIA
i  | nvidia-glG05              | package | 470.86-46.1                    | x86_64 | NVIDIA
i+ | x11-video-nvidiaG05       | package | 470.86-46.1                    | x86_64 | NVIDIA

Removals were omitted from previous listing, so here again with removals.

$ sudo egrep '^2021' /var/log/zypp/history | egrep 'kernel-default|nvidia' | tail -n 27
2021-11-10 00:42:52|remove |kernel-default-devel|5.14.14-1.1|x86_64||
2021-11-10 00:42:59|remove |kernel-default|5.14.14-1.1|x86_64||
2021-11-10 00:45:22|install|kernel-default-devel|5.14.14-1.3|x86_64||repo-oss|e0bd43cb648f48a0517141e944f18cbafe604b3eb4eea8cb742a13cf26874e32|
2021-11-10 00:45:28|install|kernel-firmware-nvidia|20211027-1.1|noarch||repo-oss|ad484eabf8cd1196d8cdcf5a88b8af54598be1aae32d25fb945e39ce3fd040af|
2021-11-10 00:46:37|install|kernel-default|5.14.14-1.3|x86_64||repo-oss|3a79a0c67053deb6cd276f62511dba29efcbe5dff3a35ed774f3aee85c23f9fd|
2021-11-10 00:49:04|install|nvidia-gfxG05-kmp-default|470.82.00_k5.14.11_2-45.1|x86_64||NVIDIA|17268ca16984a3d12cdd68b2604aacaf5824848bed7b6b8caa2bb70796a89d68|
2021-11-10 00:49:05|install|nvidia-glG05|470.82.00-45.1|x86_64||NVIDIA|d606a373803570c7de54664df62ad2eb6cc4f4b6a4269e347b8bb37a876a8bcf|
2021-11-10 00:49:06|install|nvidia-computeG05|470.82.00-45.1|x86_64||NVIDIA|cfeb0d62dcc62c9701acef1653ce21aa71276748835c51503e2eb5a0b8acf1a9|
2021-11-10 00:49:10|install|x11-video-nvidiaG05|470.82.00-45.1|x86_64||NVIDIA|4ae4004cac552d62f89cf3808862cec80bfa7fc4da8d39bb3612bb7c4b3e340c|
2021-11-23 00:59:28|install|kernel-firmware-nvidia|20211115-1.1|noarch||repo-oss|739267c436084115868d8f38358cb0311bc5cf67cf599454f91069d661f7956e|
2021-11-23 01:02:18|install|kernel-default-devel|5.15.2-1.1|x86_64||repo-oss|11e91679337c3bd03c46461e406982e01c18e2b9ee6a51e6da46e7983bd4cf50|
2021-11-23 01:02:53|install|kernel-default|5.15.2-1.1|x86_64||repo-oss|0e2f98c44bd5e58fe9b54ae8a5d6f3df67461e2734ea9a5da5133c9fbf0f7c81|
2021-11-23 01:05:37|install|nvidia-gfxG05-kmp-default|470.86_k5.14.14_1-46.1|x86_64||NVIDIA|3fa5611ec7e9bc2e372af6596fa144f51b3525214b18406797cdf30804b41d11|
2021-11-23 01:05:38|install|nvidia-glG05|470.86-46.1|x86_64||NVIDIA|976bc0430a76e5c2f1e281d170b1416a5c2263cbab1d77b8be212ccfba6f5e6e|
2021-11-23 01:05:39|install|nvidia-computeG05|470.86-46.1|x86_64||NVIDIA|2e49ae898f3825f6446b1e0bda181751060787992af7ba3108db02b3468d571a|
2021-11-23 01:05:43|install|x11-video-nvidiaG05|470.86-46.1|x86_64||NVIDIA|02103a1e73d132c41a0bc9fcd9951914511864f7586f9daad635b052cd744bc0|
2021-11-23 01:16:08|remove |kernel-default|5.14.11-1.2|x86_64|root@rivendell|
2021-11-23 01:16:08|remove |kernel-default-devel|5.14.11-1.2|x86_64|root@rivendell|
2021-11-24 00:41:13|install|kernel-default-devel|5.15.3-1.3|x86_64||repo-oss|c23da68a0de7ea2117b223d9c1b52db6ab92543965f230eedf43d569d31cee85|
2021-11-24 00:41:41|install|kernel-default|5.15.3-1.3|x86_64||repo-oss|01f979b207238fbcab0987d5d352d7fd16a15fcd517a28050452cfbc014f399d|
2021-11-24 00:43:35|remove |kernel-default|5.14.11-2.1|x86_64|root@rivendell|
2021-11-24 00:43:36|remove |kernel-default-devel|5.14.11-2.1|x86_64|root@rivendell|
2021-11-30 10:57:59|command|root@rivendell|'zypper' 'in' '-f' 'nvidia-gfxG05-kmp-default'|
2021-11-30 11:00:25|install|nvidia-gfxG05-kmp-default|470.86_k5.14.14_1-46.1|x86_64|root@rivendell|NVIDIA|3fa5611ec7e9bc2e372af6596fa144f51b3525214b18406797cdf30804b41d11|
2021-12-01 01:49:54|command|root@rivendell|'zypper' 'in' '-f' 'nvidia-gfxG05-kmp-default' 'x11-video-nvidiaG05'|
2021-12-01 01:52:18|install|nvidia-gfxG05-kmp-default|470.86_k5.14.14_1-46.1|x86_64|root@rivendell|NVIDIA|3fa5611ec7e9bc2e372af6596fa144f51b3525214b18406797cdf30804b41d11|
2021-12-01 01:52:23|install|x11-video-nvidiaG05|470.86-46.1|x86_64|root@rivendell|NVIDIA|02103a1e73d132c41a0bc9fcd9951914511864f7586f9daad635b052cd744bc0|

awerlang · December 1, 2021, 10:25pm

I found curious that logs for 470.82 are much shorter than usual:

$ sudo cat /var/log/zypp/history | fgrep --color=always 470.82
2021-11-10 00:49:04|install|nvidia-gfxG05-kmp-default|470.82.00_k5.14.11_2-45.1|x86_64||NVIDIA|17268ca16984a3d12cdd68b2604aacaf5824848bed7b6b8caa2bb70796a89d68|
2021-11-10 00:49:05|install|nvidia-glG05|470.82.00-45.1|x86_64||NVIDIA|d606a373803570c7de54664df62ad2eb6cc4f4b6a4269e347b8bb37a876a8bcf|
2021-11-10 00:49:06|install|nvidia-computeG05|470.82.00-45.1|x86_64||NVIDIA|cfeb0d62dcc62c9701acef1653ce21aa71276748835c51503e2eb5a0b8acf1a9|
2021-11-10 00:49:10|install|x11-video-nvidiaG05|470.82.00-45.1|x86_64||NVIDIA|4ae4004cac552d62f89cf3808862cec80bfa7fc4da8d39bb3612bb7c4b3e340c|
# /usr/src/kernel-modules/nvidia-470.82.00-default /
#   CLEAN   /usr/src/kernel-modules/nvidia-470.82.00-default
#   CLEAN   /usr/src/kernel-modules/nvidia-470.82.00-default/Module.symvers
#   MODPOST /usr/src/kernel-modules/nvidia-470.82.00-default/Module.symvers
# /usr/src/kernel-modules/nvidia-470.82.00-default /
#   CC [M]  /usr/src/kernel-modules/nvidia-470.82.00-default/nvidia/nv.o

$ ll /usr/src/kernel-modules/nvidia-470.8*-default/
/usr/src/kernel-modules/nvidia-470.82.00-default/:
total 12K
-rw-r--r-- 1 root root 1.4K Nov 10 00:47 conftest25359.c
-rw-r--r-- 1 root root 1.4K Nov 23 01:00 conftest32623.c
-rw-r--r-- 1 root root  116 Nov 10 00:47 nv_compiler.h

/usr/src/kernel-modules/nvidia-470.86-default/:
total 224K
-rw-r--r-- 1 root root 6.8K Nov 10 20:47 Kbuild
-rw-r--r-- 1 root root 4.6K Nov 10 20:47 Makefile
-rw-r--r-- 1 root root   19 Nov 10 20:47 Module.supported
drwxr-xr-x 1 root root    6 Nov 10 20:47 common/
-rwxr-xr-x 1 root root 180K Nov 10 20:47 conftest.sh*
-rw-r--r-- 1 root root 1.4K Dec  1 01:50 conftest5732.c
-rw-r--r-- 1 root root 1.4K Nov 30 10:58 conftest5797.c
-rw-r--r-- 1 root root 1.4K Nov 24 00:39 conftest6958.c
-rw-r--r-- 1 root root 1.4K Nov 23 01:03 conftest12343.c
-rw-r--r-- 1 root root  601 Nov 10 20:47 dkms.conf
drwxr-xr-x 1 root root 1.5K Dec  1 01:52 nvidia/
drwxr-xr-x 1 root root 1.6K Dec  1 01:52 nvidia-drm/
drwxr-xr-x 1 root root  264 Dec  1 01:52 nvidia-modeset/
drwxr-xr-x 1 root root  110 Dec  1 01:52 nvidia-peermem/
drwxr-xr-x 1 root root 6.3K Dec  1 01:52 nvidia-uvm/
-rw-r--r-- 1 root root  116 Nov 23 01:03 nv_compiler.h

awerlang · December 2, 2021, 4:37am

Got to boot 5.14.14 again by reinstalling the kernel packages from the history repos. Should I file a bug about this situation? In any case, thanks everybody who posted here.

Problem now is, bluetooth was working in that version and it isn’t anymore. It’s a different issue from what I see people talking about, as here it at least connects but keeps disconnecting and there’s no sound out of it. The culprit could be a bunch of packages from the upgrade in Nov 23 (firmware, bluedevil, pulseaudio, bluez). I may open a new thread.

malcolmlewis · December 2, 2021, 4:53am

Hi
Did you try the latest kernel, 5.15.5, if you configure zypper you can specify the version to keep, I usually do that when changing major releases. See the multiversion section of /etc/zypp/zypp.conf then no locking required etc.

There have been lots of updates lately… don’t want to fall too far behind…

arvidjaar · December 2, 2021, 7:37am

awerlang:

$ ls -l /lib/modules/5.14.14-1-default/{updates,weak-updates/*}

/lib/modules/5.14.14-1-default/updates:
total 105212
-rw-r--r-- 1 root root  4517552 Nov 10 00:48 nvidia-drm.ko
-rw-r--r-- 1 root root 52933888 Nov 10 00:48 nvidia.ko
-rw-r--r-- 1 root root  2294832 Nov 10 00:48 nvidia-modeset.ko
-rw-r--r-- 1 root root    77320 Nov 10 00:48 nvidia-peermem.ko
-rw-r--r-- 1 root root 47903528 Nov 10 00:48 nvidia-uvm.ko

Timestamps do correspond to the time when you installed kernel-default-devel and 470.82.

Removals were omitted from previous listing, so here again with removals.

$ sudo egrep '^2021' /var/log/zypp/history | egrep 'kernel-default|nvidia' | tail -n 27
2021-11-10 00:45:22|install|kernel-default-devel|5.14.14-1.3|x86_64||repo-oss|e0bd43cb648f48a0517141e944f18cbafe604b3eb4eea8cb742a13cf26874e32|
2021-11-10 00:45:28|install|kernel-firmware-nvidia|20211027-1.1|noarch||repo-oss|ad484eabf8cd1196d8cdcf5a88b8af54598be1aae32d25fb945e39ce3fd040af|
2021-11-10 00:46:37|install|kernel-default|5.14.14-1.3|x86_64||repo-oss|3a79a0c67053deb6cd276f62511dba29efcbe5dff3a35ed774f3aee85c23f9fd|
2021-11-10 00:49:04|install|nvidia-gfxG05-kmp-default|470.82.00_k5.14.11_2-45.1|x86_64||NVIDIA|17268ca16984a3d12cdd68b2604aacaf5824848bed7b6b8caa2bb70796a89d68|

Well … nvidia KMP package is built on installation of the kernel-default-devel package or for the kernel-default-devel package that was installed last. When you update kernel-default-devel, KMP is rebuilt for it as well. But when you reinstall nvidia KMP package it is not rebuilt for all previous kernel versions.

While of course you are free to open bug report, I am not sure how to handle it. I would say that replacing existing modules in all installed kernels is just as wrong as well.

So yes, the most straightforward way is to reinstall kernel-default-devel package version as you did. You could also have removed kernel-default-devel 5.15 which should have restored symlinks used during nvidia build back to the latest remaining version (5.14).

awerlang · December 2, 2021, 3:36pm

I haven’t tried 5.15.5 yet. Funny that on laptop bluetooth fails on 5.15.3 but works on 5.15.2, and on desktop both don’t work. Need to troubleshoot some more.

arvidjaar:

Timestamps do correspond to the time when you installed kernel-default-devel and 470.82.

Well … nvidia KMP package is built on installation of the kernel-default-devel package or for the kernel-default-devel package that was installed last. When you update kernel-default-devel, KMP is rebuilt for it as well. But when you reinstall nvidia KMP package it is not rebuilt for all previous kernel versions.

While of course you are free to open bug report, I am not sure how to handle it. I would say that replacing existing modules in all installed kernels is just as wrong as well.

So yes, the most straightforward way is to reinstall kernel-default-devel package version as you did. You could also have removed kernel-default-devel 5.15 which should have restored symlinks used during nvidia build back to the latest remaining version (5.14).

Thanks, that’s good information. I hadn’t attempted booting a previous major kernel, it’s good to know how to do this next time.

awerlang · December 3, 2021, 4:17am

And the bluetooth solution was… clearing /var/lib/bluetooth

oldcpu · December 12, 2021, 9:24am

Glad to read that worked for you.

I upgraded today from the 5.14.11 kernel to the 5.15.7 kernel and bluetooth stopped working. Clearing out the /var/lib/bluetooth did not help. Rebooting to the old 5.14.11 and bluetooth works again. Shutting down and booting to 5.15.7 and no bluetooth.

Surfing the web, and I read the 5.15.x kernel broke bluetooth for many users. A fix to the 5.15.4 worked for some but not for all users. For some users with the 5.15.x a power off shut down followed by a power on will work upon a switch ON, but not for others. There are bug reports in different distros on this issue. For example: https://lore.kernel.org/regressions/CAJvGw+AJ5dHSb50RtJHnjbhMVQa+rJgYznFV4t-iaO0qx+W-jw@mail.gmail.com/T/#u
.

oldcpu · December 12, 2021, 9:44am

I found the openSUSE bug report on this issue: https://bugzilla.opensuse.org/show_bug.cgi?id=1193124

oldcpu · December 20, 2021, 4:55am

I installed the updated to the 5.15.10-lp153.2.g85804f3-default on my Lenovo X1 Carbon gen9 (running LEAP-15.3) and bluetooth is now working again with both my bluetooth mouse, and my Bose Soundsport earbuds.