Slow Booting & Found Culprit?

I recently went from 42.3 to 15 as a fresh install (reformatted hard drive) and have been getting some a slow boot time on the order of 2+ minutes. I ran **systemd-analyze blame and got this output:
** 1min 36.366s purge-kernels.service
22.356s plymouth-quit-wait.service
9.164s firewalld.service
9.164s wicked.service
8.969s apparmor.service
8.708s ModemManager.service
8.126s ca-certificates.service
6.948s lvm2-monitor.service
5.582s polkit.service
5.011s btrfsmaintenance-refresh.service
4.424s initrd-switch-root.service
4.293s rsyslog.service
4.111s chronyd.service
3.041s kbdsettings.service
3.032s mcelog.service
2.423s postfix.service
1.417s display-manager.service
1.335s nscd.service
1.242s colord.service
1.192s systemd-remount-fs.service
1.079s fwupd.service
1.004s systemd-udevd.service
964ms udisks2.service
915ms wpa_supplicant.service

The 1 minute 36 seconds seems to correspond to the long pause observed on boot where there is no disk activity. The purge-kernels.service seems to be related to finding and removing old kernels but why would it be running on every boot and on a new install?

Any insights or a fix would be appreciated.

What does the following report?

sudo systemctl status  purge-kernels.service
ls -l /boot|grep vm

For reference, if I run purge-kernels manually, I get

# /sbin/purge-kernels
/sbin/purge-kernels: Nothing to do.

It runs on every boot. The first thing it does, is check for a file “/boot/purge-old-kernels”. Okay, I might have that name wrong. If the file is not found, the service exits. If the file is found, then the service does its purging and removes that file.

A kernel install creates that special file.

Looking at “systemd-analyze blame” on my system, I do not see an output line for purge-kernels. However, on the first boot after a kernel update, I would expect to see that.

Maybe try booting with “plymouth.enable=0” on the kernel command. Hit ‘e’ at the grub prompt. Scroll down until you find a line that starts “linux” or “linuxefi”. Scroll to the right on that line, and replace “splash=silent” with “plymouth.enable=0”. Hit CTRL-X to continue booting.

This is a one time action. You will be looking for two things:

(1) You will see boot messages. Make notes of any that seem to suggest a problem.
(2) Check whether the boot is significantly faster when plymouth is disabled.

Running that gives:
purge-kernels.service - Purge old kernels
Loaded: loaded (/usr/lib/systemd/system/purge-kernels.service; enabled; vendor preset: enabled)
Active: inactive (dead)
Condition: start condition failed at Mon 2018-10-15 18:05:07 HST; 5min ago

There is a similar message for every boot since Leap 15 was installed.

ls -l /boot|grep vm
-rw-r--r-- 1 root root  8026729 Oct  4 22:22 vmlinux-4.12.14-lp150.12.19-default.gz

-rw-r–r-- 1 root root 8028448 Oct 13 04:55 vmlinux-4.12.14-lp150.12.22-default.gz
lrwxrwxrwx 1 root root 35 Oct 15 09:45 vmlinuz → vmlinuz-4.12.14-lp150.12.22-default
-rw-r–r-- 1 root root 7057520 Oct 4 22:47 vmlinuz-4.12.14-lp150.12.19-default
-rw-r–r-- 1 root root 65 Oct 4 22:47 .vmlinuz-4.12.14-lp150.12.19-default.hmac
-rw-r–r-- 1 root root 7057520 Oct 13 05:24 vmlinuz-4.12.14-lp150.12.22-default
-rw-r–r-- 1 root root 65 Oct 13 05:24 .vmlinuz-4.12.14-lp150.12.22-default.hmac

For reference, if I run purge-kernels manually, I get

# /sbin/purge-kernels
/sbin/purge-kernels: Nothing to do.

I get the same result.

Thank you for the suggestion. When I did that I got a few brief messages after Grub2 and then:
A Start Job is running for DEV-DISK-BY\X2UUIDTD733AD84\X2D5F44~LordyThisIsALongString~.DEVICE 1min 30sec
followed by a count-up timer. At 90 seconds the same line gets tagged with a yellow TIMED OUT and everything continues from there very quickly.

Sorry I didn’t get the exact message but it is probably in a log file. Any idea which one?

Never mind, I found the log entry. In /var/log/boot.log:

*** ] A start job is running for dev-disk-by\x2duuid-d733ad84\x2d5fd4\x2d472b\x2d81ae\x2dc895056c6430.device (1min 30s / 1min 30s)
TIME ] Timed out waiting for device dev-disk-by\x2duuid-d733ad84\x2d5fd4\x2d472b\x2d81ae\x2dc895056c6430.device.
[DEPEND] Dependency failed for Resume from hibernation using device /dev/disk/by-uuid/d733ad84-5fd4-472b-81ae-c895056c6430.

There was a blurb in the Leap 15 install program about Hibernation and a check box to enable it. I remember NOT enabling hibernation since it has never worked correctly on this computer in the past. Is this log entry an indication that the OS thinks hibernation is enabled?

That’s your problem.

I ran into that on one of my computers. And a few other users have run into the problem. It’s an installer bug.

Check for that string in “/etc/default/grub”. You won’t find the exact string, but check for part of the UUID

grep -i 733AD84 /etc/default/grub

If the installer finds several swap partitions, it sometimes uses the wrong one in the “resume=” parameter to the kernel.

If you find that string in “/etc/default/grub”, then the best place to fix it is with Yast bootloader (kernel parameters). That way Yast will regenerate whatever related files are needed to fix it.

Maybe also check the “swap” entry in “/etc/fstab”. The “resume=” parameter and the fstab “swap” entry should both be for the same partition, and it needs to be a partition that exists.

Thank you! That was indeed the problem with the actual swap partition having a totally different UUID. Fixed it in Yast and restarted. Now boots in under a minute.

Great. I’m glad to hear that.

With respect to the Plymouth “quit wait” issue, assuming that Plymouth is enabled, these two Bug Reports are currently being worked: <; and <;.

  • With Plymouth enabled, my current “work around” is to add a “plymouth-quit-wait.service” Conflict
    to ‘/usr/lib/systemd/system/display-manager.service’ – yes, yes, the file is by default write protected but, as “root” the vi command “w!” deals with this situation …

If I read your bug report and suggestion above correctly, would the workaround be just adding “-wait” to an existing “plymouth-quit.service” conflict? Or is it added as an additional conflict? I found this in my /usr/lib/systemd/system/display-manager.service:

Description=X Display Manager
Conflicts=getty@tty7.service plymouth-quit.service
After=ypbind.service gpm.service winbind.service acpid.service dbus.socket systemd-user-sessions.service systemd-logind.service dbus.socket systemd-user-sessions.service systemd-logind.service getty@tty7.service plymouth-quit.service

AFAICS, there are two conflicts: one related to the “Plymouth quit” service and, one related to the “Plymouth quit wait” service …
[HR][/HR]What’s unsettling is, simply adding the “Plymouth quit wait” Conflict to the systemd Display Manager service doesn’t seem to be as reliable as working through all the “non-systemd” script portions and making them “systemd aware” – it seems that, Plymouth is also being used by systems which do not use systemd to perform the system initialisation …
[HR][/HR]Yes, I know, there’s an awful amount of “up-stream” script files there and, theoretically, SUSE/openSUSE should be requesting the “up-stream” maintainers to get their act with respect to systemd sorted …
[HR][/HR]UNIX® has, traditionally, the motto “do it once and, do it well” …

As far as Plymouth is concerned, the current system initialisation using systemd seems to have pulled in some components which, IMHO, cause the system initialisation to break this basic “way of doing things” …
[HR][/HR]It could be that, Fraser_Bell has suggested the best solution: “remove Plymouth” …

Or just use “plymouth.enable=0” in the boot kernel command line. You can set that in Yast bootloader (replace “splash=silent” with “plymouth.enable=0”.

Boot time and shutdown time are both faster since I made that change.

Thanks. That was easy to do. I was not a real fan of the startup bouncing square (or whatever that thing is).:slight_smile:

Plymouth can be uninstalled, to reduce initrd size, save storage space, and bandwidth and time doing updates. I taboo it during initial installation, so it doesn’t get installed in the first place. :slight_smile:

For the folks who “toe the Distro decision” line, a Plymouth “bouncing Leap” symbol issue has been found and, the repair is on the way: Bug 1110199.

It seems that, the Plymouth daemon was not responding to the “quit” command quickly enough – due to a flag associated with the framing needed for the “throbber” animation and, especially the point in time when the “throbber” changes colour from white to green …

  • The animation is currently a block of 73 frames (48 frames on Tumbleweed) which will be changed to 60 frames for Leap 15.x …
  • It seems that, it takes somewhere between 0 and 2 seconds to finish the “throbber” animation – when framing at 30 FPS …

I didn’t have a very long boot time, but after applying this change my boot time has got way faster.

Thank you.

Aside from disabling Plymouth (the first thing I jettison after any openSUSE install), my initial boot times have been around 10 to 15s. About 5 years ago, I began systematically reducing boot times while experimenting:

  • 6,235s — Leap 42.2 with wicked, IPv6 and Plymouth disabled, on a single-partition ext4 SSD-based root filesystem
  • 3.737s — replace wicked with NetworkManager
  • 3.528s — replace Postfix with Exim
  • 2.992s — ntpd off, smartd off, irqbalance off
  • 2.691s — replace sddm with no-frills minimally configured kdm
  • 2.567s — cupsd off (not doing any printing)
  • 2.326s — reduce grub2 output and kernel/console output to errors only
  • 2.127s — switch from noop scheduling to deadline 

After installing Leap 15, I discovered that I can further reduce boot times by generating an uncompressed initrd after each Kernel update:

dracut --hostonly --force --no-compress --omit "img-lib cifs fcoe fcoe-uefi rdma multipath iscsi qemu lvm mdraid dm dmraid pollcdrom plymouth btrfs wacom convertfs"

It seems to take my system longer to unpack a smaller but compressed ramdisk image than it takes the SSD to read an uncompressed one (obviously, do check the above list of services omitted, maybe your installation happens to be needing them!)

Current best boot time: 1.455s (although, my average cold-boot time is more like 1.7s): @546ms
└─display-manager.service @442ms +100ms
  └─systemd-user-sessions.service @438ms +3ms
    └─ @438ms
      └─NetworkManager.service @350ms +87ms
        └─dbus.service @312ms
          └─ @311ms
            └─ @311ms
              └─dbus.socket @311ms
                └─ @310ms
                  └─ @310ms
                    └─dev-disk-by\x2duuid-f4481…08ff.swap @305ms +5ms
                      └─dev-disk-by\x2duuid-f4481…08ff.device @303ms

openSUSE Leap 15.0 (Linux 4.12.14-lp150.12.25-default #1 SMP Thu Nov 1 06:14:23 UTC 2018 (3fcf457)) x86-64
Startup finished in 229ms (kernel) + 675ms (initrd) + 550ms (userspace) = 1.455s

My system is a Core-i5 on an ASUS mainboard, with 8GB RAM and two SATA-SSDs (one for the root fs, the other for swap and quick backups).

The following is from a default KDE install (no tinkering whatsoever) running on modern low cost hardware:

notebook:~ # inxi -zSM
System:    Host: Kernel: 4.19.7-1-default x86_64 bits: 64 Console: tty 0 
           Distro: openSUSE Tumbleweed 20181211 
Machine:   Type: Laptop System: HP product: HP Laptop 15-da0xxx v: Type1ProductConfigId serial: <filter> 
           Mobo: HP model: 84A6 v: 80.24 serial: <filter> UEFI: Insyde v: F.02 date: 05/24/2018 
notebook:~ # systemd-analyze critical-chain 
The time after the unit is active or started is printed after the "@" character.
The time the unit takes to start is printed after the "+" character. @3.066s
└─ @3.066s
  └─cron.service @3.066s
    └─postfix.service @1.745s +1.319s
      └─ @1.743s
        └─wpa_supplicant.service @2.478s +22ms
          └─dbus.service @916ms
            └─ @914ms
              └─ @914ms
                └─ca-certificates.path @895ms
                  └─ @892ms
                    └─apparmor.service @334ms +558ms
notebook:~ # 

Just a tiny belated follow-up to my old post:

After years of inactivity regarding my network configuration (having switched from wicked to NetworkManager), I decided to give systemd-networkd a try two weeks ago. I’ve just posted the details over here.

Result: systemd now manages to bring up my eth0 in 19ms (for comparison: NM was around 100ms, wicked 5s to 10s), and it works quietly and flawlessly, including those 15 meters of in-wall electric cabling to my DSL router, bridged by two devolo dLAN 1000 duo powerline-to-ethernet adapters. Less clutter thanks to a systemctl-masked wicked and uninstalled NetworkManager as well as consolidation of my trivial static DSL/networking setup within the already pre-installed systemd-networkd (preinstalled with Leap 15.0, that is).

Together with all other optimizations mentioned in my first reply above, my boot times according to systemd-analyze are around 1.5 seconds now; personal best was 1.319s five days ago.

Conclusion: give systemd-networkd a try.