Deleting (very) old kernels

Hi. After an embarrassing situation where I ran out of root disk space (/) during a zypper dup, I cleaned up my disk following the advice here .

I ran sudo zypper purge-kernels to keep the “latest,latest-1,running” versions as per the standard /etc/zypp/zypp.conf settings - multiversion.kernels = latest,latest-1,running.

What I saw however in /usr/lib/modules was a little surprising as per this listing -

chris@asus-roc:~>$ls -alh /usr/lib/modules
total 0
drwxr-xr-x 1 root root 2.0K Apr  1 13:15 .
dr-xr-xr-x 1 root root  36K Apr  1 13:45 ..
drwxr-xr-x 1 root root   14 Apr 14  2019 4.12.14-lp150.12.45-default
drwxr-xr-x 1 root root   14 Jun 12  2019 4.12.14-lp150.12.48-default
drwxr-xr-x 1 root root   14 Jun 19  2019 4.12.14-lp150.12.58-default
drwxr-xr-x 1 root root   14 Jul 31  2019 4.12.14-lp150.12.61-default
drwxr-xr-x 1 root root   14 Aug 17  2019 4.12.14-lp150.12.64-default
drwxr-xr-x 1 root root   14 Aug 17  2019 4.12.14-lp150.12.67-default
drwxr-xr-x 1 root root   14 Sep 25  2019 4.12.14-lp150.12.70-default
drwxr-xr-x 1 root root  278 Dec  7  2020 4.12.14-lp151.27-default
drwxr-xr-x 1 root root   14 Apr 12  2020 4.12.14-lp151.28.13-default
drwxr-xr-x 1 root root   14 Nov  6  2019 4.12.14-lp151.28.16-default
drwxr-xr-x 1 root root   14 Nov 15  2019 4.12.14-lp151.28.20-default
drwxr-xr-x 1 root root   14 Dec 14  2019 4.12.14-lp151.28.25-default
drwxr-xr-x 1 root root   14 Mar 14  2020 4.12.14-lp151.28.32-default
drwxr-xr-x 1 root root   14 Mar 28  2020 4.12.14-lp151.28.36-default
drwxr-xr-x 1 root root   14 Apr 24  2020 4.12.14-lp151.28.40-default
drwxr-xr-x 1 root root   14 Jul  4  2020 4.12.14-lp151.28.44-default
drwxr-xr-x 1 root root   14 Aug  7  2020 4.12.14-lp151.28.48-default
drwxr-xr-x 1 root root   14 Sep 24  2020 4.12.14-lp151.28.52-default
drwxr-xr-x 1 root root   14 Jan 19  2022 5.15.8-1-default
drwxr-xr-x 1 root root   14 Mar 17  2022 5.16.10-1-default
drwxr-xr-x 1 root root   14 Mar 24  2022 5.16.11-1-default
drwxr-xr-x 1 root root   14 Apr 21  2022 5.16.15-1-default
drwxr-xr-x 1 root root   14 Feb 22  2022 5.16.5-1-default
drwxr-xr-x 1 root root   14 Apr 28  2022 5.17.1-1-default
drwxr-xr-x 1 root root   14 May 17  2022 5.17.3-1-default
drwxr-xr-x 1 root root   14 Jun 19  2022 5.17.9-1-default
drwxr-xr-x 1 root root   14 Jul  6  2022 5.18.1-1-default
drwxr-xr-x 1 root root   14 Jul 10  2022 5.18.4-1-default
drwxr-xr-x 1 root root   14 Aug  7  2022 5.18.9-1-default
drwxr-xr-x 1 root root   14 Sep  9  2022 5.19.0-1-default
drwxr-xr-x 1 root root   14 Sep 23  2022 5.19.2-1-default
drwxr-xr-x 1 root root   14 Oct 12 15:21 5.19.7-1-default
drwxr-xr-x 1 root root   14 Oct 23 13:35 5.19.8-1-default
drwxr-xr-x 1 root root   50 Dec 20  2021 5.3.18-lp152.102-default
drwxr-xr-x 1 root root   50 Dec 20  2021 5.3.18-lp152.106-default
drwxr-xr-x 1 root root  278 Dec 20  2021 5.3.18-lp152.19-default
drwxr-xr-x 1 root root   14 Jan  5 08:56 6.0.10-1-default
drwxr-xr-x 1 root root   14 Jan  5 15:37 6.0.12-1-default
drwxr-xr-x 1 root root   14 Nov 19 07:51 6.0.2-1-default
drwxr-xr-x 1 root root   14 Dec  4 08:37 6.0.7-1-default
drwxr-xr-x 1 root root   14 Jan 14 08:39 6.1.1-1-default
drwxr-xr-x 1 root root   14 Mar 17 17:45 6.1.12-1-default
drwxr-xr-x 1 root root   14 Jan 18 09:35 6.1.2-1-default
drwxr-xr-x 1 root root   14 Feb 10 18:26 6.1.4-1-default
drwxr-xr-x 1 root root   14 Feb 26 13:21 6.1.8-1-default
drwxr-xr-x 1 root root   14 Mar  5 04:12 6.2.1-1-default
drwxr-xr-x 1 root root   14 Apr  1 13:49 6.2.4-1-default
drwxr-xr-x 1 root root  668 Apr  1 14:15 6.2.6-1-default
drwxr-xr-x 1 root root  668 Apr  1 14:15 6.2.8-1-default
chris@asus-roc:~>$

Those very old kernel folders were not wiped. Should I just remove the older folders with rm or is there a more proper way when dealing with kernels? Obviously I’d leave the 3 latest kernels untouched.

FYI - those older kernels were from when this PC was running Leap before I did an online upgrade to tumbleweed.

Thanks.

Directories in /usr/lib/modules/ not associated with installed kernels can be safely removed by whatever means you prefer.

IIRC, once upon a time, purge-kernels.service was not enabled by default, leading to what you now have. I looks like it’s still not enabled for you. Most times I’ve encountered orphan module directories, most of their content had been removed, just not all.

Never heard of that service and yes it is disabled on my system - but not for much longer :wink:

Thanks.

EDIT. Um it is enabled by default but failed for some reason. Will read on.

chris@asus-roc:/usr/lib/modules>$systemctl status purge-kernels.service 
○ purge-kernels.service - Purge old kernels
     Loaded: loaded (/usr/lib/systemd/system/purge-kernels.service; enabled; preset: enabled)
     Active: inactive (dead)
  Condition: start condition failed at Sat 2023-04-01 14:18:06 AWST; 2h 59min ago
             └─ ConditionPathExists=/boot/do_purge_kernels was not met

Apr 01 14:18:06 asus-roc systemd[1]: Purge old kernels was skipped because of an unmet condition check (ConditionPathExists=/boot/do_purge_kernels).

Maybe those are directories that are almost empty. They may have contained a locally compiled module, such an Nvidia driver.

If any of them are empty, just remove the directory. If they contain some files you will need to first find out if those are still in use.

You may run touch /boot/do_purge_kernels:

erlangen:~ # touch /boot/do_purge_kernels
erlangen:~ # systemctl start purge-kernels.service 
erlangen:~ # systemctl status purge-kernels.service 
○ purge-kernels.service - Purge old kernels
     Loaded: loaded (/usr/lib/systemd/system/purge-kernels.service; enabled; preset: enabled)
     Active: inactive (dead) since Sat 2023-04-01 11:45:56 CEST; 5s ago
   Duration: 207ms
    Process: 12981 ExecStart=/usr/bin/zypper -n purge-kernels (code=exited, status=0/SUCCESS)
    Process: 12982 ExecStartPost=/bin/rm -f /boot/do_purge_kernels (code=exited, status=0/SUCCESS)
   Main PID: 12981 (code=exited, status=0/SUCCESS)
        CPU: 218ms

Apr 01 11:45:56 erlangen systemd[1]: Starting Purge old kernels...
Apr 01 11:45:56 erlangen systemd[1]: Started Purge old kernels.
Apr 01 11:45:56 erlangen zypper[12981]: Reading installed packages...
Apr 01 11:45:56 erlangen zypper[12981]: Preparing to purge obsolete kernels...
Apr 01 11:45:56 erlangen zypper[12981]: Configuration: oldest,latest,latest-1,running
Apr 01 11:45:56 erlangen zypper[12981]: Running kernel release: 6.2.8-1-default
Apr 01 11:45:56 erlangen zypper[12981]: Running kernel arch: x86_64
Apr 01 11:45:56 erlangen zypper[12981]: Resolving package dependencies...
Apr 01 11:45:56 erlangen zypper[12981]: Nothing to do.
Apr 01 11:45:56 erlangen systemd[1]: purge-kernels.service: Deactivated successfully.
erlangen:~ # 

You may remove manually what purge-kernels.service leaves behind.

No stale entries left on infamous host erlangen:

erlangen:~ # du -hd1 /usr/lib/modules
197M    /usr/lib/modules/5.18.11-1-default
203M    /usr/lib/modules/6.2.6-1-default
203M    /usr/lib/modules/6.2.8-1-default
602M    /usr/lib/modules
erlangen:~ # 

Seems kind of odd that you have to manually add /boot/do_purge_kernels yet the service then does an rm on it when run Process: 12982 ExecStartPost=/bin/rm -f /boot/do_purge_kernels (code=exited, status=0/SUCCESS).

I wonder if /boot/do_purge_kernels gets created when a new kernel is added by zypper?

You may check older journal entries:

erlangen:~ # journalctl -q -u purge-kernels.service -gconsumed
Mar 07 18:24:17 erlangen systemd[1]: purge-kernels.service: Consumed 1.894s CPU time.
Mar 12 01:05:44 erlangen systemd[1]: purge-kernels.service: Consumed 1.975s CPU time.
Mar 14 20:50:08 erlangen systemd[1]: purge-kernels.service: Consumed 1.980s CPU time.
Mar 18 10:01:41 erlangen systemd[1]: purge-kernels.service: Consumed 2.092s CPU time.
Mar 26 06:20:10 erlangen systemd[1]: purge-kernels.service: Consumed 2.343s CPU time.
erlangen:~ # 

An installation of a new kernel creates the file. On booting the machine purge-kernels.service checks for the file, successfully removes the obsolete kernel packages if any and eventually removes the file.

erlangen:~ # journalctl -b -u purge-kernels.service 
Mar 31 13:09:40 erlangen systemd[1]: Purge old kernels was skipped because of an unmet condition check (ConditionPathExists=/boot/do_purge_kernels).
Apr 01 11:45:15 erlangen systemd[1]: Purge old kernels was skipped because of an unmet condition check (ConditionPathExists=/boot/do_purge_kernels).
Apr 01 11:45:56 erlangen systemd[1]: Starting Purge old kernels...
Apr 01 11:45:56 erlangen systemd[1]: Started Purge old kernels.
Apr 01 11:45:56 erlangen zypper[12981]: Reading installed packages...
Apr 01 11:45:56 erlangen zypper[12981]: Preparing to purge obsolete kernels...
Apr 01 11:45:56 erlangen zypper[12981]: Configuration: oldest,latest,latest-1,running
Apr 01 11:45:56 erlangen zypper[12981]: Running kernel release: 6.2.8-1-default
Apr 01 11:45:56 erlangen zypper[12981]: Running kernel arch: x86_64
Apr 01 11:45:56 erlangen zypper[12981]: Resolving package dependencies...
Apr 01 11:45:56 erlangen zypper[12981]: Nothing to do.
Apr 01 11:45:56 erlangen systemd[1]: purge-kernels.service: Deactivated successfully.
erlangen:~ # 

The service checks for installed packages only, stale entries in /usr/lib/modules are skipped.

Makes sense now. Thanks.