Failed to start Load Kernel Modules - Maybe netatop?

English Install/Boot/Login
1

I updated TW today, this time it was only Packman and malcomlewis’ https://download.opensuse.org/repositories/home:/malcolmlewis:/openSUSE_General/openSUSE_Tumbleweed. I have the latter for the netatop utility only, packages netatop and netatop-kmp-default. Last week’s TW updates had all been done previously.

Now TW throws

Failed to start Load Kernel Modules
See systemctl status systemd-modules-load.service

to the screen while booting.

Checking:

$ systemctl status systemd-modules-load
● systemd-modules-load.service - Load Kernel Modules
   Loaded: loaded (/usr/lib/systemd/system/systemd-modules-load.service; static; vendor preset: disabled)
   Active: active (exited) since Mon 2019-09-30 14:03:37 CEST; 17min ago
     Docs: man:systemd-modules-load.service(8)
           man:modules-load.d(5)
 Main PID: 521 (code=exited, status=0/SUCCESS)
    Tasks: 0 (limit: 4915)
   Memory: 0B
   CGroup: /system.slice/systemd-modules-load.service

$ sudo journalctl -b | grep -i module
Sep 30 14:03:34 susytmblwdke8570 kernel: ACPI: Added _OSI(Module Device)
Sep 30 14:03:34 susytmblwdke8570 systemd-modules-load[260]: Inserted module 'scsi_dh_alua'
Sep 30 14:03:34 susytmblwdke8570 systemd-modules-load[260]: Inserted module 'scsi_dh_emc'
Sep 30 14:03:34 susytmblwdke8570 systemd-modules-load[260]: Inserted module 'scsi_dh_rdac'
Sep 30 14:03:34 susytmblwdke8570 systemd-modules-load[260]: Inserted module 'dm_multipath'
Sep 30 14:03:34 susytmblwdke8570 systemd-modules-load[260]: Failed to find module 'netatop'
Sep 30 14:03:34 susytmblwdke8570 systemd-modules-load[260]: Inserted module 'sg'
Sep 30 14:03:37 susytmblwdke8570 kernel: netatop: loading out-of-tree module taints kernel.
Sep 30 14:03:37 susytmblwdke8570 kernel: netatop: module verification failed: signature and/or required key missing - tainting kernel
Sep 30 14:03:37 susytmblwdke8570 systemd-modules-load[521]: Inserted module 'netatop'
Sep 30 14:03:37 susytmblwdke8570 systemd[1]: Starting VirtualBox Linux kernel module...
Sep 30 14:03:38 susytmblwdke8570 systemd[1]: Started VirtualBox Linux kernel module.
Sep 30 14:03:38 susytmblwdke8570 systemd[1]: Starting Load extra kernel modules for sound stuff...
Sep 30 14:03:38 susytmblwdke8570 systemd[1]: Started Load extra kernel modules for sound stuff.
Sep 30 14:03:38 susytmblwdke8570 systemd[1]: Starting VirtualBox Linux autostart module...
Sep 30 14:03:38 susytmblwdke8570 systemd[1]: Started VirtualBox Linux autostart module.
Sep 30 14:03:57 susytmblwdke8570 plasmashell[2387]: invalid metadata "/usr/lib64/qt5/plugins/krossmoduleforms.so"
Sep 30 14:03:57 susytmblwdke8570 plasmashell[2387]: invalid metadata "/usr/lib64/qt5/plugins/krossmodulekdetranslation.so"
Sep 30 14:04:03 susytmblwdke8570 kwin_x11[2379]: Module 'org.kde.kwin.decoration' does not contain a module identifier directive - it cannot be protected from external registrations.

So this seems to boil down to something related to netatop:

$ sudo journalctl -b | grep -i netatop
Sep 30 14:03:34 susytmblwdke8570 systemd-modules-load[260]: Failed to find module 'netatop'
Sep 30 14:03:37 susytmblwdke8570 kernel: netatop: loading out-of-tree module taints kernel.
Sep 30 14:03:37 susytmblwdke8570 kernel: netatop: module verification failed: signature and/or required key missing - tainting kernel
Sep 30 14:03:37 susytmblwdke8570 systemd-modules-load[521]: Inserted module 'netatop'

From here, I don’t know how to proceed towards a solution.

I tried to remove all netatop stuff:

$ sudo systemctl stop netatopd
$ sudo systemctl disable netatopd
$ sudo modprobe -r netatop

All these went through. Rebooted, but no success, “Failed to start Load Kernel Modules” and “Failed to find module netatop” remain. I figure modprobe did rebuild the initramfs; or should I have done that - but how??? As a next try, I uninstalled the two packages netatop and netatop-kmp-default. Reboot, but without success either. Finally, I installed the netatop and netatop-kmp-packages again, modprobe’d netatop and started the netatopd service. Rebooting to no success continues.

I am on TW KDE/Plasma, with kernel 5.2.14-1-default. I did not try the one older kernel version present yet.

Now, can I get your help again, please? In particular malcolmlewis’. What am I missing? What has gone wrong? Anyway, I’d like to have a clean boot again, either with netatop or without it. Maybe it is not related to netatop? Thanks a lot in advance.

2

Now I also booted into the one older kernel, 5.2.11. There is no “Failed to start Load Kernel Modules” being thrown to the screen while booting, but otherwise the errors are similar:

$ sudo journalctl -b | grep -i module
Sep 30 14:44:51 susytmblwdke8570 kernel: ACPI: Added _OSI(Module Device)
Sep 30 14:44:51 susytmblwdke8570 systemd-modules-load[261]: Inserted module 'scsi_dh_alua'
Sep 30 14:44:51 susytmblwdke8570 systemd-modules-load[261]: Inserted module 'scsi_dh_emc'
Sep 30 14:44:51 susytmblwdke8570 systemd-modules-load[261]: Inserted module 'scsi_dh_rdac'
Sep 30 14:44:51 susytmblwdke8570 systemd-modules-load[261]: Inserted module 'dm_multipath'
Sep 30 14:44:51 susytmblwdke8570 systemd-modules-load[261]: Inserted module 'sg'
Sep 30 14:44:52 susytmblwdke8570 systemd[1]: systemd-modules-load.service: Succeeded.
Sep 30 14:44:52 susytmblwdke8570 systemd[1]: Stopped Load Kernel Modules.
Sep 30 14:44:54 susytmblwdke8570 systemd-modules-load[516]: Failed to find module 'netatop'
Sep 30 14:44:55 susytmblwdke8570 systemd[1]: Starting VirtualBox Linux kernel module...
Sep 30 14:44:55 susytmblwdke8570 vboxdrv.sh[1077]: Sources for building host modules are not present,
Sep 30 14:44:55 susytmblwdke8570 systemd[1]: Failed to start VirtualBox Linux kernel module.
Sep 30 14:44:55 susytmblwdke8570 systemd[1]: Dependency failed for VirtualBox Linux autostart module.
Sep 30 14:44:55 susytmblwdke8570 systemd[1]: Starting Load extra kernel modules for sound stuff...
Sep 30 14:44:55 susytmblwdke8570 systemd[1]: Started Load extra kernel modules for sound stuff.
Sep 30 14:45:16 susytmblwdke8570 plasmashell[2392]: invalid metadata "/usr/lib64/qt5/plugins/krossmoduleforms.so"
Sep 30 14:45:16 susytmblwdke8570 plasmashell[2392]: invalid metadata "/usr/lib64/qt5/plugins/krossmodulekdetranslation.so"
Sep 30 14:45:22 susytmblwdke8570 kwin_x11[2384]: Module 'org.kde.kwin.decoration' does not contain a module identifier directive - it cannot be protected from external registrations.

There is this “Failed to find module netatop”, but there is no tainting the kernel from netatop now. Now VirtualBox (of course?) is compromised.

3

Hi
Have you installed the latest KMP, sounds like it’s the old one…
https://download.opensuse.org/repositories/home:/malcolmlewis:/openSUSE_General/openSUSE_Tumbleweed/x86_64/netatop-kmp-default-2.0_k5.2.14_1-1.237.x86_64.rpm

4

Yep, guess so:

$ zypper se netatop*
Repository-Daten werden geladen...
Installierte Pakete werden gelesen...

S  | Name                | Zusammenfassung                             | Typ       
---+---------------------+---------------------------------------------+-----------
   | netatop             | Kernel module to gather statistics for atop | Quellpaket
i+ | netatop             | Kernel module to gather statistics for atop | Paket     
i  | netatop-kmp-default | Kernel module to gather statistics for atop | Paket     
[myself@susytmblwdke8570 ~]$ zypper info netatop*
Repository-Daten werden geladen...
Installierte Pakete werden gelesen...


Informationen zu Paket netatop:
-------------------------------
Repository         : malcolmlewis TW-General                    
Name               : netatop                                    
Version            : 2.0-1.237                                  
Arch               : x86_64                                     
Anbieter           : obs://build.opensuse.org/home:malcolmlewis 
Installierte Größe : 28,8 KiB                                   
Installiert        : Ja                                         
Status             : aktuell                                    
Quellpaket         : netatop-2.0-1.237.src                      
Zusammenfassung    : Kernel module to gather statistics for atop
Beschreibung       :                                            
    This optional kernel module can be loaded to gather statistics about the
    TCP and UDP packets that have been transmitted/received per process and per
    thread.


Informationen zu Paket netatop-kmp-default:
-------------------------------------------
Repository         : malcolmlewis TW-General                    
Name               : netatop-kmp-default                        
Version            : 2.0_k5.2.14_1-1.237                        
Arch               : x86_64                                     
Anbieter           : obs://build.opensuse.org/home:malcolmlewis 
Installierte Größe : 670,4 KiB                                  
Installiert        : Ja (automatisch)                           
Status             : aktuell                                    
Quellpaket         : netatop-2.0-1.237.src                      
Zusammenfassung    : Kernel module to gather statistics for atop
Beschreibung       :                                            
    This optional kernel module can be loaded to gather statistics about the
    TCP and UDP packets that have been transmitted/received per process and per
    thread.
5

Don’t know if that helps:

$ sudo modinfo netatop
filename:       /lib/modules/5.2.14-1-default/updates/netatop.ko
version:        2.0
description:    Per-task network statistics
author:         Gerlof Langeveld <gerlof.langeveld@atoptool.nl>
license:        GPL
suserelease:    openSUSE Tumbleweed
srcversion:     517E441654DE67C66EF59F1
depends:        
retpoline:      Y
name:           netatop
vermagic:       5.2.14-1-default SMP mod_unload modversions 
sig_id:         xxxxx
signer:         home:malcolmlewis OBS Project
sig_key:        xxxxx
sig_hashalgo:   sha256
signature:      xxxxx

I x-ed the signatures out. Pls let me know if I should paste them here, too.

6

You probably need to explain the problem first. As can be clearly seen from logs, module is loaded. The first warning is most likely from initrd which includes load-modules configuration but not the module itself. It is actually harmless. You may explicitly add your module to initrd to avoid these errors.

7

Agreed, the module is being loaded. And I can run atop plus hitting n at some point in time, as well as run atop -n.

However, I never saw any of the error messages described above ever since installing atop and netatop, as per recommendation by malcolmlewis. Not until today after updating from Packman (probably irrelevant) and netatop from malcolmlewis‘ ”General Tumbleweed“ repo. So my issue is the occurrence of new error messages while booting; and ”tainting the kernel“ sounds really bad.

Generally, I am asking myself and you experts what I am doing wrong:

First, how should I make the netatop kernel module load? I installed the packages from malcolmlewis, and within the session I did ”modprobe netatop“ and ”systemctl start netatop“. As far as I remember, no enabling netatopd, and no edits to any /etc/module… files. All was fine with that, until today.

Second, how should I get rid of netatop completely? I did what I wrote some posts back. But it didn‘t get rid of it all. Why? What should be correct?

Third, I‘d like to get netatop back to function properly, i.e. without the errors.

8

netatop package installs /usr/lib/modules-load.d/netatop.conf

I‘d like to get netatop back to function properly, i.e. without the errors.

You need to debug it (or wait until someone debugs it for you). Does initrd include netatop module? If no, something went wrong with initrd generation. if yes, something does not work properly when initrd runs. Next step depends on the answer.

It is quite possible that simply running mkinitrd now fixes it for you.

9

Reporting on progress:

I removed all things netatop now (modprobe -r netatop, systemctl stop netatopd, systemctl disable netatopd, removed the installed packages netatop and netatop-kmp-… via YaST), then regenerated all initramfs’s (dracut -f --regenerate-all). Reboot, now everything is fine regarding the a.m. errors thrown yesterday. Checked (via Krusader search, systemctl, journalctl) that no traces of netatop are left.

Then I installed netatop from scratch (packages netatop and netatop-kmp-… for current kernel 5.2.14 with netatop version 1.237 via YaST). Without doing anything else, in particular no modprobing, no service enabling and no dracutting, I rebooted Tumbleweed to the current kernel 5.2.14. TW throws the same errors as yesterday again:

$ lsmod | grep -i netatop
netatop                69632  0

$ sudo journalctl -b | grep -i netatop
Okt 01 11:25:47 susytmblwdke8570 kernel: netatop: loading out-of-tree module taints kernel.
Okt 01 11:25:47 susytmblwdke8570 kernel: netatop: module verification failed: signature and/or required key missing - tainting kernel
Okt 01 11:25:47 susytmblwdke8570 systemd-modules-load[523]: Inserted module 'netatop'

$ systemctl status netatopd
● netatopd.service - Gather per-process statistics about network utilization
   Loaded: loaded (/usr/lib/systemd/system/netatopd.service; disabled; vendor preset: disabled)
   Active: inactive (dead)

This shows that just installing the netatop package takes care of getting the kernel module loaded - in principle, the error messages “verification failed” and “tainting the kernel” do occur since yesterday only.

Next thing to do is building an updated initramfs with netatop.

10

Built an updated initramfs now (dracut -v -f). However, the error messages do remain the same:

$ lsmod | grep -i netatop
netatop                69632  0

$ systemctl status netatopd
● netatopd.service - Gather per-process statistics about network utilization
   Loaded: loaded (/usr/lib/systemd/system/netatopd.service; disabled; vendor preset: disabled)
   Active: inactive (dead)

$ sudo journalctl -b | grep -i netatop
[sudo] Passwort für root: 
Okt 01 12:04:07 susytmblwdke8570 kernel: netatop: loading out-of-tree module taints kernel.
Okt 01 12:04:07 susytmblwdke8570 kernel: netatop: module verification failed: signature and/or required key missing - tainting kernel
Okt 01 12:04:07 susytmblwdke8570 systemd-modules-load[262]: Inserted module 'netatop'

As observed also yesterday, running atop and hitting n or running atop -n does work.

So I am back to ask you, malcolmlewis, for your help. I figure something went wrong with the current version of netatop (1.237) ? I’ll wait and see; either leave my status as-is, or remove netatop temporarily until that issue might get solved. Anyway, I would like to have netatop back and functioning properly - it’s nice and provides good insight.

Plus, since my skills are limited but hopefully growing: Whoever reads my procedures as posted in this thread is kindly asked to comment also on them and help me getting things right. Actually, this netatop thing was my first kernel module ever; except for the “automatic” VirtualBox modules, of course.

11

Hi
Not sure, it could be a transient issue, the module is being signed from the OBS build log. It could also be a kernel change for out-of-tree modules. Did you add my repository temporarily and import the OBS key (as in add, refresh, accept key, then remove)?

12

No, I had added your repo permanently about a month ago when we got into the netatop stuff and set it to always update automatically. OBS key was imported back then.

EDIT: Regarding zypper dup / vendor-change and repo priorities, I follow the suggestions by karlmistelberger given in a forum thread titled “Zypper dup priorities”. In summary, he says that “zypper dup --allow-vendor-change” with repo priorities set is superior to “zypper dup --no-allow-vendor-change” (the TW default). So what I have is: Packman is at prio 90, the TW “standard” repos are at 99, additional repos like yours and one by trmdi for kvantum are at 100.
This setting priorities has no consequence for the netatop packages.

13

Hi
See what happens after the next kernel update, but likely to remain if the kernel has changed…

14

It is signed by your repository key; this key is not known to kernel (which includes default openSUSE keys) so kernel quite correctly complains. I do not see any issue here.

It is not clear what “imported” means. If you accepted repository key when adding it via zypper then it only added it to RPM keyring which is unrelated to kernel module signature verification.

15

Hi malcolmlewis, heartfelt thanks for your help. Sounds like we should call it a day on that issue.

For the record:

  • The kernel-default change from 5.2.11 to 5.2.14-1.1 happened when zypper dupping on September 18. On that day, netatop-kmp-default was at 2.0_k5.2.14_1-1.233. All was fine.
  • On September 23, I got kernel 5.2.14-1.2, and netatop-kmp-default went to 2.0_k5.2.14_1-1.235. All fine.
  • On September 27, netatop-kmp-default 2.0_k5.2.14_1-1.235 was replaced by 2.0_k5.2.14_1-1.236. All fine.
  • On September 30 (yesterday), netatop-kmp-default 2.0_k5.2.14_1-1.236 was replaced by 2.0_k5.2.14_1-1.237. At that point in time the issue started to show up.

Additional remark: The next kernel update from 5.2.14-1.2 to 5.3.1-1.2 is landing right now. virtualbox-kmp-default is in sync with this kernel. netatop-kmp-default still seems out of sync. I will wait before running the next zypper dup …

Thank you up to now. I’m sure we will “see” each other on this thread quite soon.

16

As far as I remember, I did that in YaST > Repositiories. Please, what does that mean in the current context? What else should I do?

Anyway, the question remains why things worked fine for almost a month, and went wrong just yesterday. Cf. the “For the record” in a post today.

17

I am just thinking about something else related to signatures and security.

My laptop is UEFI booted, without Secure Boot. It is a 2013 HP EliteBook 8570w. Starting some time ago, I get the same error message at boot as was discussed in this thread: https://forums.opensuse.org/showthread.php/535324-MODSIGN-Couldn-t-get-UEFI-db-list

For me, this is from journalctl:

Okt 01 12:39:01 susytmblwdke8570 kernel: **Couldn't get size: 0x800000000000000e**
Okt 01 12:39:01 susytmblwdke8570 kernel: **MODSIGN: Couldn't get UEFI db list**
Okt 01 12:39:01 susytmblwdke8570 kernel: **Couldn't get size: 0x800000000000000e**
Okt 01 12:39:01 susytmblwdke8570 kernel: Couldn't get UEFI MokListRT
Okt 01 12:39:01 susytmblwdke8570 kernel: **Couldn't get size: 0x800000000000000e**
Okt 01 12:39:01 susytmblwdke8570 kernel: Couldn't get UEFI dbx list

On screen during boot, it shows the four lines marked in bold here, they are marked in red in journalctl. According to karlmistelberger, this stuff started with kernel 5.0; it never disappeared to this day. MODSIGN is explained as something related to the kernel trying to verify module signatures.

Just speculating: Could this Non-Secure Boot issue be related to the sudden netatop failure we are discussing here? But then again, why was netatop fine since install a month ago and went wrong only yesterday? (No kernel change yesterday, cf. above.)

18

Hi
It could be, AFAIK you need to reset the db in the BIOS and that should disappear, next time there is a kernel update it should run MokManager to load the keys again.