Today, as I usually do, I logged into all the computers on my network, using ssh, each in their own instance of Konsole, to do updates.
bart@ASU-X99:~> ssh asu-aio
Enter passphrase for key '/home/bart/.ssh/id_ed25519':
Last login: Thu Nov 10 15:40:42 2022 from 10.118.10.5
Have a lot of fun...
bart@ASU-AIO:~> su -
Password:
**ASU-AIO:~ #** zypper up
As expected, all went well except for … My Wife’s Computer!
During the update of her machine, I saw an error code of
installing grub2-2.06-150400.11.12.1.x86_64 message from syslogd@msi-6400 kernel 1630932.449751 c0] watchdog:BUG:softlockup - cpu2 stuck for 249s! kworker/ug:1:19589
I wrote this error down and typed it in here so it is not a screen copy. The cpu numbers kept repeating for cpu 0, 1 and 2 with the time incrementing each time.
As the cpu stuck messages would not stop, I CTL C the process and went to her machine and tried doinf a few things. It was clear that things were not right. So, I rebooted. Her machine came back up and seemed to be ok. So I repeated the zypper up process and it seemed to go without errors. I rebooted again. It never came up!
Now, when I boot the computer, I get a grub2 prompt and I don’t know how to proceed. I really, really don’t want to do a fresh install!
Booting from a Grub prompt is possible, but you have to know what to type, and when. For this purpose, your backup of /boot/grub2/grub.cfg can be used. Find in it the line containing “/etc/grub.d/10_linux”, and enter each line that follows the immediately following “menuentry”. You can skip the if & fi lines and the lines in between, and also skip the echo lines. Likely your backup doesn’t have the kernel version last installed, so instead type simply vmlinuz and initrd if those two strings have appended version strings. For the default kernel and initrd there exist symlinks that work just as well as the originals with all their dashes, dots and numbers. When typing the linu line, omit the resume= parameter, replacing it with noresume.
If you don’t succeed, try again with a “recovery” mode entry from the advanced options section of grub.cfg.
Yes but, with Leap 15.3 and Leap 15.3, the “simple” «zypper refresh» called by «zypper patch» and «zypper update» can, occasionally, cause problems.
The reason is, the meta-data churn in the repositories due to on-going activities dealing with the (almost completed) transition to the SLE and SLE Backports repositories.
The Zypp back-end for PackageKit seems to call “zypper refresh --force” which seems to be quite reliable.
How do I get to see what is in /boot/grub2/grub.cfg? When I start the computer, it goes immediately to a grub prompt. I can press tab and see a list of commands, but none are familiar to me.
Knowing that restoring a program most often doesn’t work, I never have backed up the root partition. Guess that should change!
I removed the drive from her machine, stuck it in a USB drive container and the /boot/grub2/grub.cfg file is essentially empty:
# DO NOT EDIT THIS FILE
#
# It is automatically generated by grub2-mkconfig using templates
# from /etc/grub.d and settings from /etc/default/grub
#
### BEGIN /etc/grub.d/00_header ###
### END /etc/grub.d/00_header ###
### BEGIN /etc/grub.d/00_tuned ###
set tuned_params=""
set tuned_initrd=""
### END /etc/grub.d/00_tuned ###
### BEGIN /etc/grub.d/10_linux ###
### END /etc/grub.d/10_linux ###
### BEGIN /etc/grub.d/20_linux_xen ###
### END /etc/grub.d/20_linux_xen ###
### BEGIN /etc/grub.d/20_memtest86+ ###
### END /etc/grub.d/20_memtest86+ ###
### BEGIN /etc/grub.d/30_os-prober ###
### END /etc/grub.d/30_os-prober ###
### BEGIN /etc/grub.d/30_uefi-firmware ###
### END /etc/grub.d/30_uefi-firmware ###
### BEGIN /etc/grub.d/40_custom ###
### END /etc/grub.d/40_custom ###
### BEGIN /etc/grub.d/41_custom ###
### END /etc/grub.d/41_custom ###
### BEGIN /etc/grub.d/80_suse_btrfs_snapshot ###
### END /etc/grub.d/80_suse_btrfs_snapshot ###
### BEGIN /etc/grub.d/90_persistent ###
### END /etc/grub.d/90_persistent ###
### BEGIN /etc/grub.d/95_textmode ###
### END /etc/grub.d/95_textmode ###
No wonder if won’t boot!
Is there a way to run what would normally be grub2-mkconfg from the grub prompt? It doesn’t seem to be in the list of available commands.
So, I guess it’s time for a new drive and a fresh install.
Then we can spend the next few weeks setting up all the links, installing all the packages that are not in the default installation and, and, and.
It depends on the cause of the failure. If it’s an old drive and some sort of mechanical failure, then recovery is unlikely. If the drive is out of warranty, new drive with fresh install is probably the most sensible way forward.
So, I guess it’s time for a new drive and a fresh install.
If you are able to boot it using the installation or other removable media, mount the / filesystem to /mnt, and give us here output from:
cat /mnt/etc/fstab
blkid
fdisk -l, or
parted -l
we should be able to construct a list of commands to type at a grub prompt, or include in grub.cfg.
Depending on what went wrong to cause the problem, from rescue boot you may be able to chroot into the installed system, from which you could try text mode yast bootloader, or run grub2-mkconfig -o /boot/grub2/grub.cfg, to correct the booting problem, which could well accomplish little or less if the update process went haywire for reasons other than mechanical failure.
It’s not a mechanical failure as far as I can tell. As I said, I took the drive out of her machine and put it in a USB enclosure and can mount it and pole around. I posted the contents of /etc/boot/grub.cfg before. It has the skeleton of a file there, but was obviously not completed.
I’ll put the drive back in her system and try your suggestions. Thanks!
I put the drive back into my wife’s computer, booted it with an installation disk. From the grub menu, I selected more and then boot Linux
I was presented with several pop up boxes where I was asked which system I wanted and select a name, my wife’s copy of openSUSE came up! All seemed to be well. So, I opened an instance of Konsole and su - to root. (I can’t show any of the actual screen entries).
I ran grub2-mkconfig -o /boot/grub2/grub.cfg and got the prompt back with no errors.
I did cat /boot/grub2/grub.cfg and found that the file was seemingly untouched. It was basically a skeleton as I showed above.
I ran touch /boot/grub2/junk to be sure I could write to that directory. I could.
I ran cat > /boot/grub2/junk <enter> “This is some stuff” CTRL C
I ran cat /boot/grub2/junk and verified the text was there. It was. I removed the junk file.
Using Yast in the GUI, I reinstalled all of the grub files that were shown as installed.
I ran grub2-mkconfig -o /boot/grub2/grub.cfg again and got the prompt back with no errors.
I did cat /boot/grub2/grub.cfg again and found that the file was seemingly untouched. It was basically a skeleton as I showed above.
Using Yast in the GUI, I reinstalled all of the kernel files that were shown as installed. (quite a few!)
I ran grub2-mkconfig -o /boot/grub2/grub.cfg again and got the prompt back with no errors.
I did cat /boot/grub2/grub.cfg again and found that the file was seemingly untouched. It was basically a skeleton as I showed above.
When I do an update and the kernel is included, it seems the grub.cfg file is changed as the menu is changed. I thought by reinstalling the kernel, that process would be triggered. Either it was not, or wherever grub2-mkconfig get’s it’s information from is amongst the missing.
I got these responses while using a live version of Leap 15.4
After doing these, I realized that the responses obtained from the drives of the live disk.
I tried to figure out how to get the console to look at the actual hard drive but couldn’t.
So the cat command is accurate because I opened Dolphin, made the root partition of the hard drive active, opened the file, copied the contents to kate, saved it to the desktop on the hard drive and then used drag to move it to a USB drive, brought it to my computer and pasted the results here. So, you won’t see a prompt.
cat /etc/default/grub
# If you change this file, run 'grub2-mkconfig -o /boot/grub2/grub.cfg' afterwards to update
# /boot/grub2/grub.cfg.
# Uncomment to set your own custom distributor. If you leave it unset or empty, the default
# policy is to determine the value from /etc/os-release
GRUB_DISTRIBUTOR=
GRUB_DEFAULT=saved
GRUB_HIDDEN_TIMEOUT=0
GRUB_HIDDEN_TIMEOUT_QUIET=true
GRUB_TIMEOUT=8
GRUB_CMDLINE_LINUX_DEFAULT="resume=/dev/disk/by-label/Leap-Swap showopts"
GRUB_CMDLINE_LINUX=""
# Uncomment to automatically save last booted menu entry in GRUB2 environment
# variable `saved_entry'
# GRUB_SAVEDEFAULT="true"
#Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
# GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"
#Uncomment to disable graphical terminal (grub-pc only)
GRUB_TERMINAL="gfxterm"
# The resolution used on graphical terminal
#note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
GRUB_GFXMODE="auto"
# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
# GRUB_DISABLE_LINUX_UUID=true
#Uncomment to disable generation of recovery mode menu entries
# GRUB_DISABLE_LINUX_RECOVERY="true"
#Uncomment to get a beep at grub start
# GRUB_INIT_TUNE="480 440 1"
GRUB_BACKGROUND=
GRUB_THEME=/boot/grub2/themes/openSUSE/theme.txt
SUSE_BTRFS_SNAPSHOT_BOOTING="true"
GRUB_USE_LINUXEFI="true"
GRUB_DISABLE_OS_PROBER="false"
GRUB_ENABLE_CRYPTODISK="n"
GRUB_CMDLINE_XEN_DEFAULT="vga=gfx-1024x768x16"
I checked the contents of /etc/default/grub, /etc/grub.d on two of my desktops and they are pretty much the same. The differences are obvious. So, I think I’m going to open /boot/grub2/grub.cfg and manually recreate it based on the two working copies. For most of the files in /etc/grub.d, it looks like I can just search for the “cat <<” line and copy everything up to the EOF.
I’ll be sure to let you know how it works out.
It’s 04:00 hours here and my windows keep closing on me, I’m calling it quits for tonight!
Here is what I think happened:
During the update, something went wrong and the process stopped in the middle. (That’s obvious!)
Right at the spot where the grub configuration files were being manipulated.
All of the files in /etc/grub.d/* files had been created as empty files and *.rpmnew files had been created. (Should have been the tip off)
There must be a process that merges the original file and the .rpmnew files that didn’t run.
What I did:
I booted a copy of the installation disk and seleced More->Boot Linux and selected the real copy on the hard disk.
As root, I ran an instance of kate then:
opened /etc/grub.d/00_header
opened /etc/grub2/00_header.rpmnew , Ctrl-A, CTRL-C
moved to /etc/grub2/00_header, CTRL-V, CTRL-S
closed both files.
Repeated the above process for all files that had a .rpmnew extension.
Ran grub2-mkconfig and could see that I was successful!
Ran grub2-mkconfig -o /boot/boot2/grub.cfg
Rebooted the computer and all is well!
I should have investigated this much sooner as none of my other computers had any .rpmnew files. Only the one that was problematic.
My Wife’s Computer!
