Booting now taking more than 8 minutes - What is going on?

I recently had a problem besause booting took much longer than usual. I saw that the booting process was waiting for a number of processes. The words flashed past, as they do, but stuck on one line in the stream and repeated three short explanations. Word included “waiting” “chrony” “cron” and Btrfs. Too fast to read and understand. At that time I also noticed an amber warning light on my machine. I found I had to change a drive which had failed. (Hot swap in Raid Array.) After the new drive was installed I had expected the long booting times to reduce but after a few days of being OK I see they are back again. My last boot took over 9 minutes.
I looked at the boot log with systemctl -b but am well out of my depth here.

There is a suggestion that another hard drive might be in a pre-failure condition.
I thought some of the SMART data suggested a temperature higher than I expected.
I couldn’t find the reason for the delay but the waiting of 3 processes is in there somewhere.

Where should I look please? How can I get a report from the hard drives SMART in readable form?
Could there be a configuration error which is resulting in the looping through the three processes I saw.

Sorry to be so dumb again and some guidance would be much appreciated.

To look at what’s taking the boot so long, these commands can help.

systemd-analyse critical-chain
systemd-analyse blame

And to look at any errors/warnings from the current boot:

journalctl -b -p 0..4

Hi and thanks for the reply. Not familiar with the commands you suggest and they didn’t work for me.
I already looked at the journalctl -b info but it is over my head.
Of course I could send it if anybody has time to look at it.
Meanwhile is there a tool application I can install. My NAS boxes have great tools built in. Surely there are applications available?
Regards,
Budge

The commands should have been

systemd-analyze critical-chain
systemd-analyze blame

Yes, the general idea is to run the requested commands and post the output here for others to review/advise (regardless of whether you can make sense of it yourself).

Understood but embarrassed. I took time to check my machine for dust and did a spring clean. Once back together I started up and the boot process was as short as it ever had been. As a result the commands you suggested, when spelt correctly, gave me figures I think are ok. Of course the earlier boot was what too the time and I know it is there in the logs but how may I filter it? Meanwhile the results from first two commands which are brief enough to include here.

alastair@ibmserv2:~> systemd-analyze critical-chain
The time when unit became active or started is printed after the "@" character.
The time the unit took to start is printed after the "+" character.

└─display-manager.service @12.412s +1.901s
  └─systemd-logind.service @17.758s +564ms
    └─tmp.mount @12.302s +8ms
      └─dev-sda3.device @5.576s
alastair@ibmserv2:~> sudo systemd-analyze blame
[sudo] password for root: 
12.868s postfix.service                                                          
 7.221s NetworkManager-wait-online.service                                       
 4.913s firewalld.service                                                        
 4.703s smartd.service                                                           
 4.167s btrfsmaintenance-refresh.service                                         
 3.462s initrd-switch-root.service                                               
 2.965s udisks2.service                                                          
 2.263s polkit.service                                                           
 2.111s dracut-initqueue.service                                                 
 1.901s display-manager.service                                                  
 1.546s apparmor.service                                                         
 1.329s plymouth-quit-wait.service                                               
 1.185s avahi-daemon.service                                                     
 1.143s issue-generator.service                                                  
 1.131s nscd.service                                                             
 1.114s rpcbind.service                                                          
 1.113s rsyslog.service                                                          
 1.059s systemd-udevd.service                                                    
 1.009s user@1000.service                                                        
  815ms upower.service                                                           
  597ms NetworkManager.service                                                   
  564ms systemd-logind.service                                                   
  485ms systemd-journald.service                                                 
  462ms ModemManager.service                                                     
  438ms multimedia.mount                                                         
  335ms systemd-udev-trigger.service                                             
  260ms sysroot.mount                                                            
  242ms systemd-sysctl.service                                                   
  239ms initrd-parse-etc.service                                                 
  224ms systemd-tmpfiles-setup.service                                           
  214ms auditd.service                                                           
  212ms plymouth-switch-root.service                                             
  197ms mcelog.service                                                           
  195ms systemd-tmpfiles-setup-dev.service                                       
  186ms kbdsettings.service                                                      
  185ms klog.service                                                             
  147ms systemd-journal-flush.service                                            
  143ms systemd-remount-fs.service                                               
  141ms systemd-tmpfiles-clean.service                                           
  139ms systemd-random-seed.service                                              
  137ms systemd-modules-load.service                                             
  132ms home.mount                                                               
  124ms user-runtime-dir@1000.service                                            
  124ms systemd-update-utmp.service                                              
  123ms dev-disk-by\x2duuid-46f191b6\x2d7472\x2d49c0\x2dbd98\x2d75824020d947.swap
  119ms rtkit-daemon.service                                                     
  115ms systemd-user-sessions.service                                            
  100ms plymouth-read-write.service                                              
   99ms sound-extra.service                                                      
   93ms boot-efi.mount                                                           
   67ms dracut-cmdline.service                                                   
   44ms iscsi.service                                                            
   37ms var.mount                                                                
   30ms dracut-pre-udev.service                                                  
   29ms dracut-shutdown.service                                                  
   28ms dev-hugepages.mount                                                      
   28ms \x2esnapshots.mount                                                      
   27ms initrd-cleanup.service                                                   
   27ms dev-mqueue.mount                                                         
   26ms sys-kernel-debug.mount                                                   
   26ms boot-grub2-i386\x2dpc.mount                                              
   25ms systemd-fsck-root.service                                                
   25ms sys-kernel-tracing.mount                                                 
   24ms kmod-static-nodes.service                                                
   18ms usr-local.mount                                                          
   18ms systemd-vconsole-setup.service                                           
   17ms boot-grub2-x86_64\x2defi.mount                                           
   16ms plymouth-start.service                                                   
   15ms root.mount                                                               
   13ms opt.mount                                                                
   10ms srv.mount                                                                
   10ms initrd-udevadm-cleanup-db.service                                        
   10ms sys-fs-fuse-connections.mount                                            
    8ms tmp.mount                                                                

alastair@ibmserv2:~> 

Your system is extremely lame:

erlangen:~ # systemd-analyze critical-chain 
The time when unit became active or started is printed after the "@" character.
The time the unit took to start is printed after the "+" character.

graphical.target @2.421s
└─display-manager.service @1.458s +962ms
  └─apache2.service @1.278s +132ms
    └─time-sync.target @1.266s
      └─chronyd.service @1.171s +94ms
        └─network.target @1.168s
          └─systemd-networkd.service @848ms +318ms
            └─systemd-udevd.service @519ms +292ms
              └─systemd-tmpfiles-setup-dev.service @474ms +10ms
                └─kmod-static-nodes.service @425ms +32ms
                  └─systemd-journald.socket
                    └─-.mount
                      └─system.slice
                        └─-.slice
erlangen:~ # 

Show system details:

erlangen:~ # inxi -zFxx
System:    Kernel: 5.6.14-1-default x86_64 bits: 64 compiler: gcc v: 10.1.1 Console: tty 2 wm: kwin_x11 dm: SDDM 
           Distro: openSUSE Tumbleweed 20200609 
Machine:   Type: Desktop Mobo: ASRock model: Z170 Pro4S serial: <filter> UEFI: American Megatrends v: P3.50 date: 06/23/2016 
CPU:       Topology: Quad Core model: Intel Core i7-6700K bits: 64 type: MT MCP arch: Skylake-S rev: 3 L1 cache: 256 KiB 
           L2 cache: 8192 KiB L3 cache: 8192 KiB 
           flags: avx avx2 lm nx pae sse sse2 sse3 sse4_1 sse4_2 ssse3 vmx bogomips: 63999 
           Speed: 800 MHz min/max: 800/4200 MHz Core speeds (MHz): 1: 801 2: 801 3: 800 4: 800 5: 801 6: 800 7: 800 8: 800 
Graphics:  Device-1: Intel HD Graphics 530 vendor: ASRock driver: i915 v: kernel bus ID: 00:02.0 chip ID: 8086:1912 
           Display: server: X.Org 1.20.8 compositor: kwin_x11 driver: modesetting unloaded: fbdev,vesa alternate: intel 
           resolution: 1920x1200~60Hz s-dpi: 96 
           OpenGL: renderer: Mesa DRI Intel HD Graphics 530 (SKL GT2) v: 4.6 Mesa 20.0.7 compat-v: 3.0 direct render: Yes 
Audio:     Device-1: Intel 100 Series/C230 Series Family HD Audio vendor: ASRock driver: snd_hda_intel v: kernel 
           bus ID: 00:1f.3 chip ID: 8086:a170 
           Sound Server: ALSA v: k5.6.14-1-default 
Network:   Device-1: Intel Ethernet I219-V vendor: ASRock driver: e1000e v: 3.2.6-k port: f040 bus ID: 00:1f.6 
           chip ID: 8086:15b8 
           IF: enp0s31f6 state: up speed: 1000 Mbps duplex: full mac: <filter> 
           Device-2: Qualcomm Atheros AR9287 Wireless Network Adapter driver: ath9k v: kernel port: f040 bus ID: 03:00.0 
           chip ID: 168c:002e 
           IF: wlp3s0 state: up mac: <filter> 
Drives:    Local Storage: total: 6.38 TiB used: 3.02 TiB (47.4%) 
           ID-1: /dev/nvme0n1 vendor: Samsung model: SSD 950 PRO 512GB size: 476.94 GiB speed: 31.6 Gb/s lanes: 4 
           serial: <filter> temp: 35 C 
           ID-2: /dev/sda vendor: Western Digital model: WD40EZRX-22SPEB0 size: 3.64 TiB speed: 6.0 Gb/s serial: <filter> 
           ID-3: /dev/sdb vendor: Samsung model: SSD 850 EVO 500GB size: 465.76 GiB speed: 6.0 Gb/s serial: <filter> 
           ID-4: /dev/sdc vendor: Crucial model: CT2000BX500SSD1 size: 1.82 TiB speed: 6.0 Gb/s serial: <filter> 
Partition: ID-1: / size: 59.45 GiB used: 14.69 GiB (24.7%) fs: btrfs dev: /dev/sdb5 
           ID-2: /home size: 406.34 GiB used: 257.03 GiB (63.3%) fs: ext4 dev: /dev/nvme0n1p3 
           ID-3: /opt size: 59.45 GiB used: 14.69 GiB (24.7%) fs: btrfs dev: /dev/sdb5 
           ID-4: /tmp size: 59.45 GiB used: 14.69 GiB (24.7%) fs: btrfs dev: /dev/sdb5 
           ID-5: /var size: 59.45 GiB used: 14.69 GiB (24.7%) fs: btrfs dev: /dev/sdb5 
Swap:      ID-1: swap-1 type: partition size: 16.00 GiB used: 0 KiB (0.0%) priority: -2 dev: /dev/sdb8 
Sensors:   System Temperatures: cpu: 30.0 C mobo: 30.0 C 
           Fan Speeds (RPM): fan-1: 0 fan-2: 1202 fan-3: 0 fan-4: 0 fan-5: 0 fan-6: 0 
           Voltages: 12v: N/A 5v: N/A 3.3v: 3.34 vbat: 3.12 
Info:      Processes: 264 Uptime: 15h 38m Memory: 31.05 GiB used: 3.04 GiB (9.8%) Init: systemd v: 245 runlevel: 5 
           target: graphical.target Compilers: gcc: 10.1.1 alt: 10/9 Shell: bash v: 5.0.17 running in: konsole inxi: 3.1.00 
erlangen:~ #

@Budgie2 I’m guessing your only using ipv4? If so set postfix to only use it…


cat /etc/postfix/main.cf |grep inet_protocol

inet_protocols = ipv4

Reduce online wait, change


cat /etc/sysconfig/network/config | grep ONLINE

NM_ONLINE_TIMEOUT="1"

If you wait to reduce a bit more, get rid of plymouth…

[QUOTE=malcolmlewis;2937829]
NM_ONLINE_TIMEOUT=“1”



Quick question: this parameter applies if you're using wicked instead of network manager?

[quote="“brunomcl,post:9,topic:140736”]

Quick question: this parameter applies if you’re using wicked instead of network manager?[/QUOTE]
Hi
That’s for NetworkManager, I use wicked on the desktop here, have it set to 0, and WAIT_FOR_INTERFACES=“1” which should work in either case. I also disable/mask a lot of cruft services like lvm2 etc.

[quote="“brunomcl,post:9,topic:140736”]

Hi
Here are my times…


systemd-analyze

Startup finished in 2.335s (kernel) + 1.891s (initrd) + 3.014s (userspace) = 7.241s 
graphical.target reached after 2.858s in userspace

systemd-analyze blame | head -n 5

1.170s wicked.service                                                           
 804ms display-manager.service                                                  
 768ms dracut-initqueue.service                                                 
 614ms fwupd.service                                                            
 491ms initrd-switch-root.service

systemd-analyze critical-chain

The time when unit became active or started is printed after the "@" character.
The time the unit took to start is printed after the "+" character.

graphical.target @2.858s
└─display-manager.service @2.051s +804ms
  └─time-sync.target @2.048s
    └─chronyd.service @1.982s +65ms
      └─network.target @1.980s
        └─wicked.service @810ms +1.170s
          └─wickedd-nanny.service @792ms +16ms
            └─wickedd.service @712ms +79ms
              └─dbus.service @698ms
                └─basic.target @695ms
                  └─sockets.target @695ms
                    └─dbus.socket @695ms
                      └─sysinit.target @692ms
                        └─systemd-udevd.service @376ms +315ms
                          └─systemd-tmpfiles-setup-dev.service @351ms +12ms
                            └─kmod-static-nodes.service @336ms +7ms
                              └─systemd-journald.socket
                                └─-.mount
                                  └─system.slice
                                    └─-.slice

[quote="“malcolmlewis,post:11,topic:140736”]

My wicked duration is considerably larger:


:~> systemd-analyze critical-chain
The time after the unit is active or started is printed after the "@" character.
The time the unit takes to start is printed after the "+" character.

graphical.target @9.601s
└─multi-user.target @9.601s
  └─cron.service @9.009s
    └─postfix.service @8.371s +635ms
      └─time-sync.target @8.360s
        └─chronyd.service @8.242s +84ms
          └─network.target @8.233s
            └─wicked.service @2.134s +**6.098**s
              └─network-pre.target @2.132s
                └─firewalld.service @1.710s +420ms
                  └─dbus.service @1.674s
                    └─basic.target @1.657s
                      └─paths.target @1.657s
                        └─btrfsmaintenance-refresh.path @1.656s
                          └─sysinit.target @1.644s
                            └─systemd-udev-settle.service @216ms +1.427s
                              └─systemd-udev-trigger.service @152ms +62ms
                                └─systemd-udevd-kernel.socket @141ms
                                  └─-.slice
:~> 

Let me try the NM stting, post back in a minute.

Hi
Just make sure the setting is 1 for waiting for interfaces and not 0, that will add a 6 second delay which I see for your wicked…

Weird, it actually increased boot time a bit:

:~> cat /etc/sysconfig/network/config | grep ONLINE
NM_ONLINE_TIMEOUT="1"
:~>
:~> systemd-analyze critical-chain
The time after the unit is active or started is printed after the "@" character.
The time the unit takes to start is printed after the "+" character.

graphical.target @**9.749**s
└─multi-user.target @9.749s
  └─getty.target @9.054s
    └─getty@tty1.service @9.054s
      └─systemd-user-sessions.service @8.470s +31ms
        └─remote-fs.target @8.469s
          └─iscsi.service @8.405s +33ms
            └─network.target @8.399s
              └─wicked.service @2.233s +**6.165**s
                └─wickedd-nanny.service @2.204s +26ms
                  └─wickedd.service @1.861s +341ms
                    └─wickedd-dhcp4.service @1.775s +29ms
                      └─dbus.service @1.736s
                        └─basic.target @1.717s
                          └─sockets.target @1.717s
                            └─avahi-daemon.socket @1.715s
                              └─sysinit.target @1.703s
                                └─systemd-udev-settle.service @219ms +1.483s
                                  └─systemd-udev-trigger.service @155ms +62ms
                                    └─systemd-udevd-control.socket @143ms
                                      └─-.mount
                                        └─system.slice
                                          └─-.slice
:~>

:confused:

Changed NM_ONLINE_TIMEOUT to “5” just for testing and boot time increased a bit:

:~> systemd-analyze critical-chain
The time after the unit is active or started is printed after the "@" character.
The time the unit takes to start is printed after the "+" character.

graphical.target @**9.846**s
└─multi-user.target @9.846s
  └─cron.service @9.189s
    └─postfix.service @8.632s +554ms
      └─time-sync.target @8.627s
        └─chronyd.service @8.501s +92ms
          └─network.target @8.498s
            └─wicked.service @2.163s +**6.334**s
              └─wickedd-nanny.service @2.130s +26ms
                └─wickedd.service @2.094s +29ms
                  └─wickedd-auto4.service @1.755s +51ms
                    └─dbus.service @1.707s
                      └─basic.target @1.691s
                        └─paths.target @1.690s
                          └─ca-certificates.path @1.689s
                            └─sysinit.target @1.679s
                              └─systemd-udev-settle.service @213ms +1.465s
                                └─systemd-udev-trigger.service @155ms +57ms
                                  └─systemd-udevd-control.socket @146ms
                                    └─-.mount
                                      └─system.slice
                                        └─-.slice
:~>

Hi
For wicked, set the other one… WAIT_FOR_INTERFACES=“1” leave the other at 0.

Sorry if I hijacked the thread, as it is for tumbleweed. Will revert to NM_ONLINE_TIMEOUT=“30” an see what happens.

Brilliant, malcolm, that did it!

:~> systemd-analyze critical-chain
The time after the unit is active or started is printed after the "@" character.
The time the unit takes to start is printed after the "+" character.

graphical.target @8.457s
└─multi-user.target @8.457s
  └─smb.service @8.318s +138ms
    └─nmb.service @3.253s +5.064s
      └─network.target @3.245s
        └─wicked.service @2.221s +**1.023**s
          └─wickedd-nanny.service @2.195s +24ms
            └─wickedd.service @1.865s +324ms
              └─wickedd-dhcp4.service @1.783s +52ms
                └─dbus.service @1.741s
                  └─basic.target @1.724s
                    └─sockets.target @1.724s
                      └─pcscd.socket @1.724s
                        └─sysinit.target @1.711s
                          └─systemd-udev-settle.service @211ms +1.500s
                            └─systemd-udev-trigger.service @152ms +58ms
                              └─systemd-udevd-kernel.socket @138ms
                                └─-.slice
:~>

Thank you very muchhttps://forums.opensuse.org/images/icons/icon10.png

I enabled what I really needed and disabled everything else showing up at the top of “systemd-analyze blame”:

erlangen:~ # systemctl list-units --type service 
  UNIT                               LOAD   ACTIVE SUB     DESCRIPTION                                              
  apache2.service                    loaded active running The Apache Webserver                                     
  auditd.service                     loaded active running Security Auditing Service                                
  chronyd.service                    loaded active running NTP client/server                                        
  cups-browsed.service               loaded active running Make remote CUPS printers available locally              
  cups.service                       loaded active running CUPS Scheduler                                           
  dbus.service                       loaded active running D-Bus System Message Bus                                 
  display-manager.service            loaded active running X Display Manager                                        
  dracut-shutdown.service            loaded active exited  Restore /run/initramfs on shutdown                       
  fetchmail.service                  loaded active running A remote-mail retrieval utility                          
  getty@tty1.service                 loaded active running Getty on tty1                                            
  haveged.service                    loaded active running Entropy Daemon based on the HAVEGE algorithm             
  hd-idle.service                    loaded active running hd-idle disk spindown service                            
  irqbalance.service                 loaded active running irqbalance daemon                                        
  kbdsettings.service                loaded active exited  Apply settings from /etc/sysconfig/keyboard              
  kmod-static-nodes.service          loaded active exited  Create list of static device nodes for the current kernel
  lm_sensors.service                 loaded active exited  Initialize hardware monitoring sensors                   
  mcelog.service                     loaded active running Machine Check Exception Logging Daemon                   
  minidlna.service                   loaded active running MiniDLNA is a DLNA/UPnP-AV server software               
  polkit.service                     loaded active running Authorization Manager                                    
  postfix.service                    loaded active running Postfix Mail Transport Agent                             
  rtkit-daemon.service               loaded active running RealtimeKit Scheduling Policy Service                    
  systemd-journal-flush.service      loaded active exited  Flush Journal to Persistent Storage                      
  systemd-journald.service           loaded active running Journal Service                                          
  systemd-logind.service             loaded active running Login Service                                            
  systemd-modules-load.service       loaded active exited  Load Kernel Modules                                      
  systemd-networkd.service           loaded active running Network Service                                          
  systemd-random-seed.service        loaded active exited  Load/Save Random Seed                                    
  systemd-remount-fs.service         loaded active exited  Remount Root and Kernel File Systems                     
  systemd-sysctl.service             loaded active exited  Apply Kernel Variables                                   
  systemd-tmpfiles-setup-dev.service loaded active exited  Create Static Device Nodes in /dev                       
  systemd-tmpfiles-setup.service     loaded active exited  Create Volatile Files and Directories                    
  systemd-udev-trigger.service       loaded active exited  udev Coldplug all Devices                                
  systemd-udevd.service              loaded active running udev Kernel Device Manager                               
  systemd-update-utmp.service        loaded active exited  Update UTMP about System Boot/Shutdown                   
  systemd-user-sessions.service      loaded active exited  Permit User Sessions                                     
  udisks2.service                    loaded active running Disk Manager                                             
  upower.service                     loaded active running Daemon for power management                              
  user-runtime-dir@1000.service      loaded active exited  User Runtime Directory /run/user/1000                    
  user@1000.service                  loaded active running User Manager for UID 1000                                
  wpa_supplicant@wlp3s0.service      loaded active running WPA Supplicant daemon (interface wlp3s0)                 

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.

40 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.
erlangen:~ # 

Hi all and thanks for the above. That was exciting but please can I get back to my question about a boot log that is getting further away every time TW does an update with reboot required. Do I give up or can I get earlier boot logs easily.

I may need to post a different question but can you suggest any good gui tools I can use to monitor system easily?