Each time I shutdown/reboot I get these red colored messages:
Failed unmounting Runtime Directory.
...
Failed unmounting /var.
This has been happening since I installed Leap 42.1 (currently I use 42.3). Nothing seems to have been broken so far but during the last boot there was some more extensive disk check.
Is there anything I need to fix or worry about?
See if this thread helps with troubleshooting the issue, at least for /run
https://lists.opensuse.org/opensuse-factory/2016-05/msg00341.html
Thank you. It says
Boot with
systemd.log_level=debug systemd.log_target=kmsg log_buf_len=1M
How do I do that?
Also what means “Runtime Directory” (is that /run?)? Could you help to customize the debug.sh given there to match my issue, so I can post more relevant feedback after that:
#!/bin/sh
mount -o remount,rw /
lsof /run > /shutdown-log.txt
dmesg >> /shutdown-log.txt
mount -o remount,ro /
create /usr/lib/systemd/system-shutdown/debug.sh and make it
executable with the following content
#!/bin/sh
mount -o remount,rw /
lsof /run > /shutdown-log.txt
dmesg >> /shutdown-log.txt
mount -o remount,ro /
Reboot the system
Ok, here are the last lines of the generated log:
350.817459] watchdog watchdog0: watchdog did not stop!
350.820838] systemd-shutdown[1]: Sending SIGTERM to remaining processes...
351.344116] systemd-journald[455]: Received SIGTERM from PID 1 (systemd-shutdow).
351.497036] systemd-shutdown[1]: Sending SIGKILL to remaining processes...
351.503541] systemd-shutdown[1]: Hardware watchdog 'iTCO_wdt', version 0
351.503790] systemd-shutdown[1]: Unmounting file systems.
351.503999] systemd-shutdown[1]: Remounting '/var/run' read-only with options 'mode=755'.
351.504021] systemd-shutdown[1]: Unmounting /var/run.
351.504028] systemd-shutdown[1]: Remounting '/var' read-only with options 'data=ordered'.
351.699374] EXT4-fs (sdb3): re-mounted. Opts: data=ordered
351.700069] systemd-shutdown[1]: Unmounting /var.
351.700087] systemd-shutdown[1]: Remounting '/' read-only with options 'data=ordered'.
351.716036] EXT4-fs (sdd5): re-mounted. Opts: data=ordered
351.726278] systemd-shutdown[1]: Remounting '/var/run' read-only with options 'mode=755'.
351.726336] EXT4-fs (sdb3): re-mounted. Opts: data=ordered
351.726360] EXT4-fs (sdd5): re-mounted. Opts: data=ordered
351.743245] EXT4-fs (sdd5): re-mounted. Opts: acl,user_xattr
While the most interesting information should be in the beginning.
https://paste.opensuse.org/dee4affc
Do you think it is related to mounting?
Here is also the output of grep -iE “error|fail|mount” shutdown-log.txt:
This particular line just got my attention:
9.888841] FAT-fs (sdd1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.
sdd1 is /boot/efi
And you immediately have the reason for busy filesystem:
plymouthd 4858 root mem REG 0,19 217032 17175 /var/run/nscd/group
I have no idea what this means and why this happens.
What can I do in order to have proper unmounting?
Reading through all of https://lists.opensuse.org/opensuse-factory/2016-05/msg00341.html (which is the same problem) it appears there is little you can do.
I suppose you could forgo the splash screen and uninstall plymouth, but I concede that is not a very satisfactory solution.
I don’t have the situation described there, i.e. plymouth is not trying to display splash screen on shutdown. In fact my grub command line parameters have always been:
video=2560x1440 resume=/dev/sdb2 splash=verbose quiet showopts nouveau.modeset=0 elevator=deadline
and I also tried:
video=2560x1440 resume=/dev/sdb2 splash=verbose quiet showopts nouveau.modeset=0 elevator=deadline **plymouth.enable=0**
but that changed nothing.
Indeed… nonetheless from your error log it does look as if plymouthd is implicated in the failure to unmount cleanly.
and I also tried:
video=2560x1440 resume=/dev/sdb2 splash=verbose quiet showopts nouveau.modeset=0 elevator=deadline **plymouth.enable=0**
but that changed nothing.
So, as you’re not using plymouth, what happens if instead of disabling you do actually remove the plymouth packages?
If it still occurs, then boot with the debug options and post the whole (rather than just extracts) of shutdown-log.txt
Searched for “plymouth” in yast and unistalled all found packages:
Deleting plymouth-dracut
Deleting plymouth-plugin-label-ft
Deleting plymouth-branding-openSUSE
Deleting plymouth-scripts
Deleting plymouth
Additional rpm output:
warning: /etc/plymouth/plymouthd.conf saved as /etc/plymouth/plymouthd.conf.rpmsave
Deleting plymouth-plugin-script
Deleting libply-splash-graphics4
Deleting libply-splash-core4
Deleting libply-boot-client4
Deleting libply4
If it still occurs, then boot with the debug options and post the whole (rather than just extracts) of shutdown-log.txt
Right after uninstalling plymouth packages I added systemd.log_level=debug systemd.log_target=kmsg log_buf_len=1M to bootloader as explained on the link. Rebooted twice to make sure it has come into effect. The first time I didn’t see any red notice during shutdown. The second time it was there again. The 2 logs I saved as shutdown-log-1.txt and shutdown-log-2.txt. Then I removed the added bootloader options.
https://susepaste.org/4723773c
I am reluctant to pasting the full text of shutdown-log.txt in a public forum for security and privacy reasons as it seems to reveal too much info about how the data is structured on this system, local usernames etc. Please let me know what in particular I should look for, so I can paste additionally only the relevant parts. Thanks.
Hi
After you removed plymouth, did you add package locks, did you rebuild initrd?
zypper rm libply-boot-client4 libply-splash-core4 libply-splash-graphics4 libply4 plymouth plymouth-branding-openSUSE plymouth-dracut plymouth-plugin-label-ft plymouth-plugin-script plymouth-scripts
zypper al libply-boot-client4 libply-splash-core4 libply-splash-graphics4 libply4 plymouth plymouth-branding-openSUSE plymouth-dracut plymouth-plugin-label-ft plymouth-plugin-script plymouth-scripts
mkinitrd
No. And I haven’t run any updates either. Why is adding locks necessary?
did you rebuild initrd?
After reading your post - yes + rebooted twice. The red error messages still appear on shutdown. The only change is that on boot the number of text info messages which display are less. They used to be a whole screen and more, now only 1/3 of the screen.
Hi
Else plymouth will re-install the next time you update…
Are you sure? I just ran ‘zypper ref’ and ‘zypper up’ and it tells me “Nothing to do”.
In any case - the problem seems unrelated to plymouth, so I wonder what to do next. Any ideas?
Still for /var and /run ?
Look at the very start of the shutdown-log (for the lsof output), any open files? (There didn’t appear to be anything shown by https://susepaste.org/4723773c but we don’t know if it includes the start of the log). As plymouth is now eliminated, journalctl files maybe?
Then look through the remainder of the log for anything suspicious (using | grep -i mount | grep -iE “var|runtime” won’t necessarily pick out the offending line). I’m sorry I can’t tell you what to specifically look for. I also understand your reasons and hence reluctance to share the entire log.
Also, you wrote in the initial post:
This has been happening since I installed Leap 42.1 (currently I use 42.3).
Could you clarify that: What version is it happening with? Was it a clean install or a (down?)grade? Or did you mean it has been happening all the time since 42.1 through 42.2 to 42.3 ???
[QUOTE=tannington;2843568]Still for /var and /run ?
[/QUOTE]
Yes.
Look at the very start of the shutdown-log (for the lsof output), any open files? (There didn’t appear to be anything shown by https://susepaste.org/4723773c but we don’t know if it includes the start of the log).
The first 20 lines of the latest log file:
https://susepaste.org/71c124c1
As plymouth is now eliminated, journalctl files maybe?
Please explain.
Then look through the remainder of the log for anything suspicious (using | grep -i mount | grep -iE “var|runtime” won’t necessarily pick out the offending line). I’m sorry I can’t tell you what to specifically look for. I also understand your reasons and hence reluctance to share the entire log.
These are the last 40 lines of the log:
https://susepaste.org/29aa84b3
The message “Failed to unmount /var” appears during shutdown right before this message (I don’t know why it is not in the log though):
31.245825] systemd-shutdown[1]: Unmounting /var.
/var is sdb3
Does that clarify anything?
Could you clarify that: What version is it happening with? Was it a clean install or a (down?)grade? Or did you mean it has been happening all the time since 42.1 through 42.2 to 42.3 ???
I am running Leap 42.3, upgraded from 42.2, upgraded from 42.1, upgraded from 13.2. IIRC it started happening after upgrading from 13.2 to Leap 42.1.