Update failed, have a GRUB prompt

English Install/Boot/Login
1

The weekly system update using zypper failed today. On boot it defaults to the GRUB prompt.

How do I use grub to select or create a bootable kernel image?
How do I discover which kernels are available to create?
I am reluctant to just start issuing commands hoping to get lucky.

I found this article. When I look at /boot, the only item is grub2/. There are no kernel images there.

> ls (hd0,gpt1)
  --shows the root directories. sames as ls / --
> ls /boot
 grub2/
> ls /boot/grub2/
 themes/ grubenv/ x86_64-efi/ i386-pc/ locale/ device.map fonts/

It has been a long time since I looked at /boot/. I do recall there was a lot more in boot/.

2

EFI or leacy BIOS boot?

3

I am not sure. I think it is BIOS.

4

My mistake. I booted to the BIOS. It proclaims “UEFI BIOS Utility.”
So I am going with UEFI.

5

Is it normal that grub only lists directories? Not files?

6

I think the machine may be booting into Busybox and displaying an initramfs prompt. I am not certain so please do correct me if needed.

7

The prompt is:
grub>

I am certain this is grub. It responds to a variety of grub commands I have given it.

8

We know nothing about your system. Boot any live image, start with the output of

fdisk -l
blkid -o +partuuid
lsblk -f -o +partuuid
efibootmgr

Mount your root partition, show the content of /etc/fstab there.

9 
# fdisk -l
Disk /dev/sda: 931.5 GiB, 1000204886016 bytes, 1953525168 sectors
Disk model: SD Ultra 3D 1TB
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: F7E0D636-85E3-4446-9F4D-1A511F7060FD

Device         Start        End    Sectors   Size Type
/dev/sda1       2048  134219775  134217728    64G Linux swap
/dev/sda2  134219776 1953525134 1819305359 867.5G Linux filesystem


Disk /dev/sdb: 447.1 GiB, 480103981056 bytes, 937703088 sectors
Disk model: SanDisk Ultra II
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 7925EFE7-A544-4B51-A583-FA0E4CDD3C48

Device         Start       End   Sectors   Size Type
/dev/sdb1       2048 209717247 209715200   100G Linux filesystem
/dev/sdb2  209717248 314574847 104857600    50G Linux filesystem
/dev/sdb3  314574848 356517887  41943040    20G Linux filesystem
/dev/sdb4  356517888 937703054 581185167 277.1G Linux filesystem


Disk /dev/sdc: 111.8 GiB, 120034123776 bytes, 234441648 sectors
Disk model: AXIOM 120GB
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0xee2412c4

Device     Boot Start       End   Sectors   Size Id Type
/dev/sdc1        2048 234440703 234438656 111.8G 83 Linux


Disk /dev/loop0: 82 MiB, 85983232 bytes, 167936 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/loop1: 13.4 MiB, 14024704 bytes, 27392 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/loop2: 59.4 MiB, 62259200 bytes, 121600 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/loop3: 46.1 MiB, 48365568 bytes, 94464 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/loop4: 4.1 MiB, 4325376 bytes, 8448 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/loop5: 82 MiB, 85983232 bytes, 167936 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/sdd: 1.4 TiB, 1500301909504 bytes, 2930277167 sectors
Disk model: BUP Slim SL
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0x00044db3

Device     Boot Start        End    Sectors  Size Id Type
/dev/sdd1  *     2048 2930277166 2930275119  1.4T 83 Linux


Disk /dev/sde: 233 GiB, 250148290560 bytes, 488570880 sectors
Disk model:  SanDisk 3.2Gen1
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x17af5f85

Device     Boot Start       End   Sectors  Size Id Type
/dev/sde1          32 488570879 488570848  233G 83 Linux


Disk /dev/sdm: 232.9 GiB, 250047627264 bytes, 488374272 sectors
Disk model:  SanDisk 3.2Gen1
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xdfd00e4e

--[ note: "-o +partuuid " was not recognized ]--
# blkid
Device     Boot Start       End   Sectors   Size Id Type
/dev/sdm1          32 488374271 488374240 232.9G 83 Linux
/dev/sda1: UUID="932d5695-cb65-4b52-9218-09b58b7a26da" TYPE="swap" PARTUUID="2678fe5f-2bcc-40a5-8dc8-932bd782203a"
/dev/sda2: LABEL="Data-01" UUID="33f43025-eece-4ca3-94c8-84571673c6a0" TYPE="xfs" PARTUUID="34e1ff5d-712b-4598-906e-b19d0ce45372"
/dev/sdb1: LABEL="op-sys-root" UUID="073627c1-59b3-419d-80fa-9fc44bc70087" UUID_SUB="fede1b7a-dc02-45d1-b25d-ecdfc0ee56d1" TYPE="btrfs" PTTYPE="dos" PARTUUID="db5466eb-fb1a-440c-a3ed-c976d4753847"
/dev/sdb2: LABEL="op-sys-usr" UUID="c914e88f-d26e-4f41-a08c-83169cd98e1b" UUID_SUB="9d1ee31e-0ce4-4bf8-b1d8-149a6ee275f9" TYPE="btrfs" PARTUUID="7a68276b-09b9-4943-8db6-cf9ffd8340a5"
/dev/sdb3: LABEL="data-03" UUID="86a38aeb-7e2b-4d95-94ca-0669dcfd7d21" TYPE="ext4" PARTUUID="77c0c504-572e-425c-a7a1-9e64a969b457"
/dev/sdb4: LABEL="data-02" UUID="0832969d-f0b2-475a-bead-b89303123eef" UUID_SUB="e6dcc289-2aff-45ea-a149-ec3bb2e81524" TYPE="btrfs" PARTUUID="9e87fc66-7765-4537-ba8e-2e16696eb6f6"
/dev/sdc1: LABEL="system-home" UUID="be652115-4d20-4421-8e44-914d3267c996" UUID_SUB="29c6663f-a14c-4082-a186-9a9fef9d1b0b" TYPE="btrfs" PARTUUID="ee2412c4-01"
/dev/loop0: TYPE="squashfs"
/dev/loop1: TYPE="squashfs"
/dev/loop2: TYPE="squashfs"
/dev/loop3: TYPE="squashfs"
/dev/loop4: TYPE="squashfs"
/dev/loop5: TYPE="squashfs"
/dev/sdd1: LABEL="Backup-01" UUID="4aedc2c7-322c-4b6f-8a6d-2bde927d8a7b" UUID_SUB="2a39a4cc-dd74-4a5c-a10b-5a7862fc1f3e" TYPE="btrfs" PARTUUID="00044db3-01"
/dev/sde1: LABEL="bkp-pub-data" UUID="4d7984c6-41c8-49dd-8ea6-dcae8c9dc11a" TYPE="ext4" PARTUUID="17af5f85-01"
/dev/sr0: UUID="2020-06-26-06-47-10-29" LABEL="openSUSE-Leap-15.2-DVD-x86_64695" TYPE="iso9660" PTUUID="6fd2dfbf" PTTYPE="dos"
/dev/sdm1: LABEL="bkp-graphics" UUID="fa3917b2-a616-4de1-adab-62c03b73908b" TYPE="ext4" PARTUUID="dfd00e4e-01"

# lsblk -f
NAME   FSTYPE   LABEL                            UUID                                 FSAVAIL FSUSE% MOUNTPOINT
loop0  squashfs
loop1  squashfs
loop2  squashfs                                                                             0   100% /mounts/mp_0000
loop3  squashfs                                                                             0   100% /mounts/mp_0001
loop4  squashfs                                                                             0   100% /mounts/mp_0002
loop5  squashfs                                                                             0   100% /parts/mp_0000
sda
├─sda1 swap                                      932d5695-cb65-4b52-9218-09b58b7a26da
└─sda2 xfs      Data-01                          33f43025-eece-4ca3-94c8-84571673c6a0
sdb
├─sdb1 btrfs    op-sys-root                      073627c1-59b3-419d-80fa-9fc44bc70087
├─sdb2 btrfs    op-sys-usr                       c914e88f-d26e-4f41-a08c-83169cd98e1b
├─sdb3 ext4     data-03                          86a38aeb-7e2b-4d95-94ca-0669dcfd7d21
└─sdb4 btrfs    data-02                          0832969d-f0b2-475a-bead-b89303123eef
sdc
└─sdc1 btrfs    system-home                      be652115-4d20-4421-8e44-914d3267c996
sdd
└─sdd1 btrfs    Backup-01                        4aedc2c7-322c-4b6f-8a6d-2bde927d8a7b
sde
└─sde1 ext4     bkp-pub-data                     4d7984c6-41c8-49dd-8ea6-dcae8c9dc11a
sdm
└─sdm1 ext4     bkp-graphics                     fa3917b2-a616-4de1-adab-62c03b73908b
sr0    iso9660  openSUSE-Leap-15.2-DVD-x86_64695 2020-06-26-06-47-10-29

# efibootmgr
BootCurrent: 0005#

Timeout: 1 seconds
BootOrder: 0005,0001,0004,0003
Boot0001* Hard Drive
Boot0003* USB
Boot0004* CD/DVD Drive
Boot0005* UEFI: PIONEER BD-RW BDR-XS061.10

# cat /etc/fstab
UUID=073627c1-59b3-419d-80fa-9fc44bc70087  /                       btrfs  defaults                      0  0
UUID=073627c1-59b3-419d-80fa-9fc44bc70087  /boot/grub2/x86_64-efi  btrfs  subvol=/@/boot/grub2/x86_64-efi  0  0
UUID=073627c1-59b3-419d-80fa-9fc44bc70087  /boot/grub2/i386-pc     btrfs  subvol=/@/boot/grub2/i386-pc  0  0
UUID=073627c1-59b3-419d-80fa-9fc44bc70087  /var                    btrfs  subvol=/@/var                 0  0
UUID=c914e88f-d26e-4f41-a08c-83169cd98e1b  /usr                    btrfs  defaults                      0  0
UUID=073627c1-59b3-419d-80fa-9fc44bc70087  /root                   btrfs  subvol=/@/root                0  0
UUID=073627c1-59b3-419d-80fa-9fc44bc70087  /opt                    btrfs  subvol=/@/opt                 0  0
UUID=be652115-4d20-4421-8e44-914d3267c996  /home                   btrfs  autodefrag                    0  0
#
UUID=073627c1-59b3-419d-80fa-9fc44bc70087  /srv                    btrfs  subvol=/@/srv                 0  0
#
UUID=932d5695-cb65-4b52-9218-09b58b7a26da  swap                    swap   defaults                      0  0
UUID=33f43025-eece-4ca3-94c8-84571673c6a0  /d500g                  xfs    defaults                      0  0
#
UUID=86a38aeb-7e2b-4d95-94ca-0669dcfd7d21  /data03                 ext4   data=ordered                  0  2
UUID=0832969d-f0b2-475a-bead-b89303123eef  /data02                 btrfs  defaults                      0  0
UUID=4aedc2c7-322c-4b6f-8a6d-2bde927d8a7b  /backup01               btrfs  nofail                        0  0
#
UUID=073627c1-59b3-419d-80fa-9fc44bc70087  /usr0/local             btrfs  subvol=/@/usr0/local          0  0
UUID=4d7984c6-41c8-49dd-8ea6-dcae8c9dc11a  /mnt/bkp-pub-data       ext4   data=ordered                  0  2
LABEL=bkp-graphics                         /mnt/bkp-graphics       ext4   data=ordered,nofail           0  2
# sma-nas-02.sma.com:/volume1/pub-data       /t                      nfs    defaults                      0  0
# sma-nas-02.sma.com:/volume1/av-media       /v                      nfs    defaults                      0  0
# sma-nas-02.sma.com:/volume1/graphics       /w                      nfs    defaults                      0  0
#
//sma-nas-02.sma.com/pub-data              /t-smb                  cifs   credentials=/home/jmoe/.smb/.smbpw,uid=sma-user4,gid=users,file_mode=0664,dir_mode=0775,noserverino,noperm  0  0
//sma-nas-02.sma.com/graphics              /w-smb                  cifs   credentials=/home/jmoe/.smb/.smbpw,uid=sma-user4,gid=users,file_mode=0664,dir_mode=0775,noserverino,noperm  0  0
10

I poked around a bit with the Rescue system. After mounting the system disk, I looked in /boot. It is completely empty.

So I loaded /var/log/messages in an editor. Something went wrong: the last entry was erasing rsyslog followed by (paraphrasing) “Logging has stopped.” Sort of the purpose of the system log to help with diagnosis.

I created a build environment and tried mkinitd. As you may expect, it failed; there are no kernel images to use.

My next step is to download a Tumbleweed installation ISO, create a disk, and select “Upgrade” from the menu.

Any other recommendations?

11

No, your system is using legacy BIOS boot.

Did you mount /usr?

12

Okay.

Yes.

13

Showing the exact commands would be more useful.

Show

btrfs subvolume list mount-point-of-root
btrfs subvolume get-default mount-point-of-root
btrfs subvolume list mount-point-of-usr
btrfs subvolume get-default mount-point-of-usr

I cannot give you copy-paste commands because you did not show what you have done.

14

I do not know what happened during a weekly update executed using zypper. I do know that it enthusiastically erased everything in /bin and /boot, which bricked the system.

I performed a fresh installation of tumbleweed using the existing partitioning. I now have a functioning computer again, although I have to reinstall the additional programs I use.

I am glad that I had separated /usr and /home to separate disks. Originally it was for spinning disks and offered parallel operation. Now it is more of a safety feature concurrent with backups.

15

Browsing the discussion, I’d like to know whether there is some message before GRUB displays it’s prompt.
The other thing is that you should have a more detailed log on your disks (unless you mis-configured it). Personally I like persistent journal, because it captures early boot and late shutdown…
The typical sequence I do from resue is:
mount the root fs on /mnt
for x in each sys dev proc: mount /x on /mnt/x using the --bind option
chroot /mnt
mount -va (to mount as much as possible)
Then examine and try to fix the system.
Before rebooting, do the opposite commands in the opposite order.

However if your /boot is really empty, you have a problem.
Having had a similar problem with MBR boot recently, GRUB was updated on the wrong disk due to a bad grub_installdevice…

16

No.
It went from the BIOS splash to the GRUB prompt. Oh, wait… There was a message about how to use grub.

The other thing is that you should have a more detailed log on your disks (unless you mis-configured it). Personally I like persistent journal, because it captures early boot and late shutdown…

Where would disk logs be? I assume you mean they are separate from /var/log/messages.
That was a part of the problem. One of the bits being erased was logging. I had searched /var/log/messages. At the end was the last of a long list of messages about erasing a distribution file, and then a message that “System Logging has stopped” after erasing rsyslog-.... What happened after that is a mystery.

I did that. I probed various directories to judge the extent of damage. That is when I found nothing in /bin and /boot.
I then proceeded with the (somewhat) fresh installation.