Can't boot with UEFI after changing kernel parameter using YaST (Lenovo Thinkpad)

rincewindcz · November 12, 2019, 3:12pm

I’m running Tumbleweed and I updated OS on Sunday (just regular zypper dup). Everything worked fine, except there was PCIe bus error showing in dmesg again. (basically version of this: https://askubuntu.com/questions/771899/pcie-bus-error-severity-corrected). This happens all the time and adding kernel parameter pci=nomsi fixes it. Normally I just edit grub config file, run grub2-mkconfig and that’s it. For some unknown reason I decided to use YaST this time - so I updated Kernel Parameters tab, clicked Ok, waited for progress bar to finish, typed reboot as root (didn’t closed YaST, but closed the Bootloader window). Since then I cannot boot to my system no matter what. My laptop is Lenovo Thinkpad L570, with NVMe SSD disk and I’m using EFI boot with Secure Boot turned off. After powering the PC on, Lenovo logo is displayed, then EFI boot menu. Selecting any boot record (i.e. opensuse) results in black screen for half a second, then boot menu again. No grub at all.

What I have already tried:

boot Live openSUSE image in EFI mode, chroot, run grub2-install. No change
fsck all partitions including EFI boot partition. No change.
deleting boot records using *efibootmgr, *recreating them either manually or using grub2-install. No change.
Enabling Secure Boot, installing shim. No change.
removing *legacy_boot *flagfrom my root partition as suggested here: forums.opensuse.org/showthread.php/534275-How-to-reinstall-grub2-efi-bootloader. No change.
drink multiple beers. No change (except feeling little bit less frustrated)

This is basically current status:


# blkid
/dev/nvme0n1p1: LABEL="SYSTEM" UUID="2C74-562A" TYPE="vfat" PARTLABEL="EFI system partition" PARTUUID="27baa8c9-d0df-42e5-b7b2-e3da9e7c604f"
/dev/nvme0n1p3: UUID="21c4d1b6-6bb8-47dd-914e-6d86be7f66f9" UUID_SUB="1bdfdc96-a5f2-475e-aeea-0ca277de4c62" TYPE="btrfs" PTTYPE="dos" PARTUUID="57cc0005-3212-db47-a691-3c5ad860e823"
/dev/nvme0n1p4: UUID="9437484d-ac48-4daa-93cb-ae639f7f752b" TYPE="swap" PARTUUID="42d2e907-55e3-994e-967d-b1a9b2e2376b"
/dev/nvme0n1p5: UUID="f98a6b07-056b-4a72-a661-52513ba9935d" TYPE="xfs" PARTUUID="68cc7507-341a-8b47-8907-a95896736287"
/dev/nvme0n1p2: PARTLABEL="Microsoft reserved partition" PARTUUID="c92b6b18-b541-44df-8aea-6ebe287da7aa"
/dev/loop0: TYPE="squashfs"
/dev/loop1: UUID="b0c866ca-f097-41ce-87fb-e06cde55d596" TYPE="ext4"
/dev/sda1: UUID="2019-11-10-02-51-52-00" LABEL="openSUSE_Tumbleweed_XFCE_Live" TYPE="iso9660"
/dev/sda2: SEC_TYPE="msdos" LABEL_FATBOOT="BOOT" LABEL="BOOT" UUID="749C-10E5" TYPE="vfat"
/dev/sda3: LABEL="cow" UUID="719d5bea-abb5-40fc-ab21-f4db07b86a29" TYPE="ext4"

# efibootmgr -v
BootCurrent: 001D
Timeout: 0 seconds
BootOrder: 0000,0017,0018,0019,001B,001C,001D,001A,001E,0022
Boot0000* opensuse    HD(1,GPT,27baa8c9-d0df-42e5-b7b2-e3da9e7c604f,0x800,0x82000)/File(\EFI\opensuse\grubx64.efi)
Boot0010  Setup    FvFile(721c8b66-426c-4e86-8e99-3457c46ab0b9)
Boot0011  Boot Menu    FvFile(126a762d-5758-4fca-8531-201a7f57f850)
Boot0012  Diagnostic Splash Screen    FvFile(a7d8d9a6-6ab0-4aeb-ad9d-163e59a7a380)
Boot0013  Lenovo Diagnostics    FvFile(3f7e615b-0d45-4f80-88dc-26b234958560)
Boot0014  Startup Interrupt Menu    FvFile(f46ee6f4-4785-43a3-923d-7f786c3c8479)
Boot0015  Rescue and Recovery    FvFile(665d3f60-ad3e-4cad-8e26-db46eee9f1b5)
Boot0016  MEBx Hot Key    FvFile(ac6fd56a-3d41-4efd-a1b9-870293811a28)
Boot0017* USB CD    VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,86701296aa5a7848b66cd49dd3ba6a55)
Boot0018* USB FDD    VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,6ff015a28830b543a8b8641009461e49)
Boot0019* ATAPI CD0    VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,aea2090adfde214e8b3a5e471856a35401)
Boot001A* NVMe0    VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,001c199932d94c4eae9aa0b6e98eb8a400)
Boot001B* ATA HDD1    VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,91af625956449f41a7b91f4f892ab0f602)
Boot001C* ATA HDD0    VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,91af625956449f41a7b91f4f892ab0f600)
Boot001D* USB HDD    VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,33e821aaaf33bc4789bd419f88c50803)
Boot001E  PCI LAN    VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,78a84aaf2b2afc4ea79cf5cc8f3d3803)
Boot001F* IDER BOOT CDROM    PciRoot(0x0)/Pci(0x16,0x2)/Ata(0,1,0)
Boot0020* IDER BOOT Floppy    PciRoot(0x0)/Pci(0x16,0x2)/Ata(0,0,0)
Boot0021* ATA HDD    VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,91af625956449f41a7b91f4f892ab0f6)
Boot0022* PCI LAN    VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,78a84aaf2b2afc4ea79cf5cc8f3d3803)

# ls -l /boot/efi/EFI/opensuse/
total 308
-rwxr-xr-x 1 root root 307200 Nov 12 11:59 grubx64.efi

Any ideas what to try next?

malcolmlewis · November 12, 2019, 3:33pm

Hi
I suspect YaST had issues with using the nvme device as is defaults to /dev/sda, are you specifying the nvme partition when creating manually? If not boot from the live system and create the entry with the -d option and /dev/nvme0n1 (or maybe nvme0).

knurpht · November 12, 2019, 3:35pm

In the GRUB2 menu, select the boot option of your choice, hit “e”, then navigate to the line that starts with ‘linux’ or ‘linuxefi’, remove the ‘pci=nomsi’ ( and other options you added yourself ), hit F10 and it should boot like before. Now you also can remove it from YaST ( that’s needed since the previous is a one time thing)

rincewindcz · November 12, 2019, 4:16pm

I suspect YaST had issues with using the nvme device as is defaults to /dev/sda, are you specifying the nvme partition when creating manually? If not boot from the live system and create the entry with the -d option and /dev/nvme0n1 (or maybe nvme0).

Hi, well, this could be the cause of it, but my problem is different I guess. I think Yast somehow corrupted EFI boot record or partition and the grub is not even found during boot. But there are all necessary files on EFI boot partition. When I boot live system, do chroot and run grub2-install, this is the result: https://pastebin.com/cD3sRLK8
(Installation finished. No error reported.) And the dev path is correct (/dev/nvme0n1). After reboot there is still no grub. Only black screen for half a second, then EFI boot menu (this: https://imgur.com/a/0mUByqn). Seems like EFI can’t find the grub binary or something.

malcolmlewis · November 12, 2019, 4:40pm

Hi
Looks like it’s a grub issue rather than a UEFI issue. If you select the NVMe0 device what happens?

If you mount the /dev/nvme0n1p3 partiton and subvolume(s) is anything present in /boot?

nrickert · November 12, 2019, 5:20pm

Make sure secure-boot is disabled. Your grub2-install does not support secure-boot (you have to use “shim-install” for that). And a fall back to the BIOS boot menu is the way that some systems respond if secure-boot is enabled but there is no system defined that can do a secure-boot.

Another possibility is that your BIOS does not see the “nvme” device. You might want to check that.

rincewindcz · November 12, 2019, 5:53pm

Selecting NVMe0 device entry is same as opensuse entry. Black screen and then EFI menu again.

/boot seems normal to me.


chroot_system:/tmp # mount
/dev/nvme0n1p3 on / type btrfs (rw,relatime,ssd,space_cache,subvolid=257,subvol=/@)
devtmpfs on /dev type devtmpfs (rw,nosuid,size=8108736k,nr_inodes=2027184,mode=755)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
/dev/nvme0n1p5 on /home type xfs (rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota)
chroot_system:/boot # ll
total 40964
-rw-r--r-- 1 root root      512 Feb 27  2018 backup_mbr
-rw-r--r-- 1 root root     1725 Oct 21 12:40 boot.readme
-rw-r--r-- 1 root root   226374 Nov  3 11:13 config-5.3.8-1-default
-rw-r--r-- 1 root root        0 Nov 12 10:52 do_purge_kernels
drwxr-xr-x 1 root root        0 Mar 16  2018 efi
drwxr-xr-x 1 root root       98 Nov 12 12:01 grub2
lrwxrwxrwx 1 root root       22 Nov 12 10:52 initrd -> initrd-5.3.8-1-default
-rw------- 1 root root 11924360 Nov 12 10:53 initrd-5.3.8-1-default
-rw-r--r-- 1 root root  1194579 Mar 29  2019 symtypes-5.0.5-1-default.gz
-rw-r--r-- 1 root root  1244614 Nov  3 12:23 symtypes-5.3.8-1-default.gz
-rw-r--r-- 1 root root   426483 Nov  3 12:22 symvers-5.3.8-1-default.gz
-rw-r--r-- 1 root root      484 Nov  3 12:22 sysctl.conf-5.3.8-1-default
-rw-r--r-- 1 root root  4232125 Nov  3 12:18 System.map-5.3.8-1-default
-rw-r--r-- 1 root root 13748383 Nov  3 12:28 vmlinux-5.3.8-1-default.gz
lrwxrwxrwx 1 root root       23 Nov 12 10:52 vmlinuz -> vmlinuz-5.3.8-1-default
-rw-r--r-- 1 root root  8904928 Nov  3 13:06 vmlinuz-5.3.8-1-default
-rw-r--r-- 1 root root       65 Nov  3 13:06 .vmlinuz-5.3.8-1-default.hmac
chroot_system:/boot # ll grub2/
total 12
drwxr-xr-x 1 root root   22 Jun 10  2017 fonts
-rw------- 1 root root 7501 Nov 12 12:01 grub.cfg
-rw-r--r-- 1 root root 1024 Nov 11 11:45 grubenv
drwxr-xr-x 1 root root 6294 Nov 11 11:46 i386-pc
drwxr-xr-x 1 root root  356 Nov 10 16:17 locale
drwxr-xr-x 1 root root   16 Nov 10 16:29 themes
drwxr-xr-x 1 root root 6130 Nov 12 14:07 x86_64-efi

Secure boot is definitely disabled. I tried to enable secure boot and install shim yesterday to see if it helps, but no luck. Fallback to legacy mode is disabled as well. EFI bios definitely see the disk device.
The thing is that it was booting/working fine after update. Things went south after I tried to add kernel parameter in YaST and since then grub is not launched after power-on. System and EFI bios should be fine I guess.

arvidjaar · November 12, 2019, 6:23pm

Try renaming /boot/grub2/grub.cfg. Does it change anything?

rincewindcz · November 12, 2019, 7:16pm

Oh, I think I just made a small progress. But still not loading grub.

It seems that poor YaST degraded my precious computer from x86_64-efi to legacy i386-pc.
There is a link called /sbin/update-bootloader which points to /sbin/pbl

/sbin/pbl is Perl script which is updating bootloader config and YaST is using this.
There is a log file in /var/log/pbl.log


2019-11-10 16:17:23 <1> update-bootloader-2546 new.126: update-bootloader-2546 = /sbin/update-bootloader, version = 0.926, root = 0:45
2019-11-10 16:17:23 <1> update-bootloader-2546 main.269: /sbin/update-bootloader --reinit
2019-11-10 16:17:23 <1> update-bootloader-2546 main.271: bootloader = grub2
2019-11-10 16:17:23 <1> update-bootloader-2546 main.277: locale = en_US.UTF-8
2019-11-10 16:17:25 <3> update-bootloader-2546 run_command.294: '/usr/lib/bootloader/grub2/install' failed with exit code 1, output:
<<<<<<<<<<<<<<<<
target = i386-pc
/dev/disk/by-uuid/27105edf-4ff9-4497-b787-43a2de6c5400: not a block device
grub_installdevice missing or invalid
determining suitable boot device
disk device(/boot)
kernel device = nvme0n1p3
disk(nvme0n1p3) = nvme0n1
disk = nvme0n1
+ /usr/sbin/grub2-install --target=i386-pc --force --skip-fs-probe /dev/nvme0n1
Installing for i386-pc platform.
/usr/sbin/grub2-install: warning: this GPT partition label contains no BIOS Boot Partition; embedding won't be possible.
/usr/sbin/grub2-install: error: filesystem `btrfs' doesn't support blocklists.
&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;&gt;
2019-11-10 16:17:27 &lt;1&gt; update-bootloader-2546 run_command.294: '/usr/lib/bootloader/grub2/config' = 0, output:

So target is definitely wrong here (i386-pc) and should be (x86_64-efi).
It seems like options for pbl script are loaded from /etc/sysconfig/bootloader
cat /etc/sysconfig/bootloader | grep LOADER_TYPE
LOADER_TYPE=“grub2”
So I changed LOADER_TYPE to “grub2-efi”

/sbin/pbl --reinit
grub2-install

and still nothing Grub is still missing.


2019-11-12 19:02:37 <1> update-bootloader-1577 new.126: update-bootloader-1577 = /sbin/update-bootloader, version = 0.926, root = 0:56 (chroot)
2019-11-12 19:02:37 <1> update-bootloader-1577 main.269: /sbin/update-bootloader --reinit
2019-11-12 19:02:37 <1> update-bootloader-1577 main.271: bootloader = grub2-efi
2019-11-12 19:02:37 <1> update-bootloader-1577 main.277: locale = en_US.UTF-8
2019-11-12 19:02:39 <1> update-bootloader-1577 run_command.294: '/usr/lib/bootloader/grub2-efi/install' = 0, output:
<<<<<<<<<<<<<<<<
target = x86_64-efi
+ /usr/sbin/grub2-install --target=x86_64-efi --no-nvram --removable
Installing for x86_64-efi platform.
Installation finished. No error reported.
>>>>>>>>>>>>>>>>
2019-11-12 19:02:43 <1> update-bootloader-1577 run_command.294: '/usr/lib/bootloader/grub2-efi/config' = 0, output:
<<<<<<<<<<<<<<<<
+ /usr/sbin/grub2-mkconfig -o /boot/grub2/grub.cfg
Generating grub configuration file ...
Found theme: /boot/grub2/themes/openSUSE/theme.txt
Found linux image: /boot/vmlinuz-5.3.8-1-default
Found initrd image: /boot/initrd-5.3.8-1-default
Found Windows Boot Manager on /dev/nvme0n1p1@/EFI/Microsoft/Boot/bootmgfw.efi
done

nrickert · November 12, 2019, 10:38pm

Now try doing that in Yast bootloader, instead of with a text editor. And maybe change the boot timeout (increase or decrease by 1 second) so that Yast recognizes that something is changed and booting must be updated.

rincewindcz · November 13, 2019, 12:20pm

**It’s booting again!

**This was definitely some weird (Lenovo?) UEFI issue. I’m not sure what YaST did when I tried to change boot params - probably set some weird flag on EFI boot partition.
Yesterday I tried to launch rEFInd Boot Manager (http://www.rodsbooks.com/refind/). Just to see if it’s grub problem or UEFI problem. But the rEFInd was unable to launch too.
This was clear sign that there is something wrong with UEFI itself. I also must add, that when I got this machine year and half ago, it came with Windows preinstalled. I’ve never booted into Windows OS and I directly went and deleted all Windows partitions except EFI boot partition. Reason is that before this, I was blatantly ignorant and inexperienced with EFI boot and UEFI at all.
So after I reassured myself that it’s completely OK to delete EFI boot partition and recreate it again… I did that. I’ve used cgdisk to delete and create new EFI partition, important step is set the correct GUID type here (EF00).
Then *mkfs.vfat -F32 to create fresh FAT32 filesystem, then run both update-bootloader and grub2-install from chroot environment. *Just to be extra sure, I’ve also reset UEFI (bios) settings to default values and made sure Secure Boot is disabled.
Rebooted and voilà… never been happier to see grub menu in my life

nrickert · November 13, 2019, 2:19pm

We are glad that you have it working.