Understanding UEFI boot drives, and how UEFI chooses which drive to boot from

I’m doing embedded development, running openSUSE 15.5 on a Mini-ITX motherboard using UEFI (AMI Aptio). Secure Boot is disabled. I’ve used an openSUSE USB drive to install to a SATA-connected SSD, and to an internal NVMe drive, and either drive boots the system without a hitch, but only if it’s the only attached drive.

However, for my work I need to be able to flexibly work with multiple boot drives, booting off of a selected drive and copying between it and another boot drive. Problem: if both the SSD and the NVMe drive are connected, then I can only boot from the SSD. It’s as if having the SSD plugged in hides the NVMe from UEFI:

  • In the UEFI’s “Boot” screen, I’ve set the “FIXED BOOT ORDER” priorities to NVMe and then hard drive. Nope: always boots from the SSD. (Interesting thing: if only the NVME is connected then this menu shows the NVMe’s model number under the “NVMe” entry, but if I have both connected then I only see the SSD’s model number under the “Hard Disk” entry.)

  • In the same screen, there’s a “UEFI Hard Disk Drive BBS Priorities” submenu. If the SSD is connected there is only an entry for the SSD, whether or not the NVMe is connected. If only the NVMe is connected, then there is an entry for the NVMe. And no matter what I choose, if the SSD is connected then it boots from the SSD.

It’s as if the BIOS can see the NVMe alone, or the SSD alone, but if both are connected then the SSD hides the NVMe from the BIOS. Once booted I can see both devices using “lsblk”, but I just can’t get it boot from the NVMe while the SSD is connected.

I could easily work around this for now, but I’ll need to have multiple drives to function in the future, and I’m concerned that this is a fundamental problem that needs to be addressed sooner rather than later.

So: any idea why UEFI can only see a single openSUSE boot drive? Or, anything else I should try to gather more information?

Thanks,
Dan

EFI does not boot from “drives”. It tries to execute programs that are specified in the BootXXXX variables in the order specified in the BootOrder variable.

Understood, but doesn’t the UEFI configuration specify the order in which drives are searched for those executable boot programs? If not, what is the meaning of the “FIXED BOOT ORDER priorities” section of the UEFI “BOOT” screen (help text: “Sets the system boot order”)?

That is the question you need to address to the manufacturer of your motherboard. If you show screenshot and provide the output of efibootmgr matching this screenshot, someone may try to guess.

Ok, so it sounds like my problems begin before openSUSE even gets a hold of the CPU.

Might this be something about how openSUSE initializes the NVMe drive?

I figured it out. I’ve been using Clonezilla to clone from one drive to another, and although the results are bootable, since the two drives share all UUIDs the UEFI can’t discriminate between them.

I even tried reinstalling openSUSE onto one of these cloned drives, blowing away the contents, but although the partition UUIDs changed, the drive PTUUID did NOT change, and UEFI was still confused.

I finally blew away the drive’s partition table:

sudo dd if=/dev/zero of=/dev/sdb bs=1M

I ran that for a few seconds and then killed the process. (Yes, it was overkill, but I didn’t want any doubt.) I then re-installed openSUSE to the drive. Bingo: the PTUUID is now different, and UEFI can now tell the difference between the drives.

So, it’s OK to clone boot drives, as long as you never have two such drives connected to the same computer.

FYI,
Dan

1 Like

But you can do it manually?

Meaning, “do I know how to change the PTUUID of an existing disk, without starting from scratch”? Not yet, but Research Is Ongoing…

In installing most distros to be booting in UEFI mode, the program the UEFI firmware loads to boot that system gets installed in a directory on the ESP filesystem using the exact same name as every other installation coming from that root distro. To illustrate, suppose a one HDD, SSD or NVME system setup to multiboot, without any compensation during or after by the admin WRT to booting, and all sharing the same ESP filesystem, (as IMO they should, lest the UEFI firmware become hopelessly tangled in most cases):

  1. Tumbleweed
  2. Slowroll
  3. Leap 15.6
  4. Leap 16.0pre
  5. Fedora 40
  6. Fedora 41
  7. Ubuntu 20.04
  8. Ubuntu 22.04
  9. Ubuntu 24.04
  10. Debian 11
  11. Debian 12
  12. Debian 13

It wouldn’t matter in which order installed, the sorted result would be the same, the five following directories in /EFI/:

    ├── BOOT
    ├── debian
    ├── fedora
    ├── opensuse
    └── ubuntu

Each of those five directories would contain as little as one file for the UEFI firmware to load. Which it would load is determined by the firmware, which can be specified within UEFI firmware setup, or software, such as efibootmgr. Exists a BBS hotkey that varies by brand that can be used to choose which of the five brands to boot. Which of each brand’s bootloader will launch will depend on which last bootloader setup got applied. Once that file is loaded, it’s up to the bootloader of the selected brand’s last bootloader setup what will be in its boot menu, Grub, systemd-boot or other, whatever. So, indirectly at least, there be 4-5 discrete starting points, depending on whether the one in BOOT actually produces a usable result.

If wishing all 12 installed systems to be directly bootable using the BBS hotkey directly, a solution I tried lies in /etc/default/grub (using Grub2, or equivalent if using systemd-boot or other bootloader(s)), on its GRUB_DISTRIBUTOR= line. Various distros populate this, or not, variously, but whatever result follows from it, empty or not, is what dictates the name of the directory in the /ESP/ filesystem, which when booted to Leap becomes /boot/efi/EFI/opensuse. In openSUSE, GRUB_DISTRIBUTOR= always defaults to opensuse if left blank by the admin. The issue is solve for me by applying a unique string value, which for the above 12 installations could (sorted) result in the following /EFI/ tree base:

    ├── BOOT
    ├── deb11
    ├── deb12
    ├── deb13
    ├── f40
    ├── f41
    ├── s156
    ├── s160
    ├── sslo
    ├── stw
    ├── ub20
    ├── ub22
    └── ub24

Most of my installations are on PCs with more than 12 installations. So as not to forget to do so, the best time to assign a value to GRUB_DISTRIBUTOR= is during installation. I have yet to experience any distro enable this, but I did ask for it in Bugzilla. Next best is on first boot, followed by rerunning bootloader setup, or at least before anything else is done to cause a bootloader setup change anywhere on the computer, such installing or removing a kernel.

This setup is good even when not using the BBS hotkey to choose what to boot, as it results in no OS overwriting any other OS’s ESP entry, which can create a big mess to untangle if using encryption and/or secure boot, and undue frustration even if not.

I still use GRUB_DISTRIBUTOR= on every installation, but I don’t often boot using a BBS hotkey. Instead, I employ a one bootloader per PC scheme. Only one is required to get the job done for most installations. So far, I’ve only been using Tumbleweed’s, except on those few PCs I don’t have TW installed on. This forum thread I think adequately covers that subject. Keyword there: custom.cfg - well applies with multiple bootable physical storage devices.

HTH

Interesting, but this is all after the UEFI has chosen a drive to boot from. Each boot drive only has a single OS installed (openSUSE, go figure), but I have multiple boot drives, and the UEFI needs to be told which one to boot from.

How many SATA ports are on the motherboard? Is the NVME hidden regardless which port you use for SATA? Some low cost motherboards use chipsets that cannot use certain combinations of storage device ports, usually some combination of an M.2 port and one or two SATA ports, as M.2 ports commonly also support SATA devices. First M.2 I bought was SATA, before I was conscious of differences among M.2 storage devices. Last week I moved it off the shelf into an adapter that equips it to be installed as if common 2.5" form factor SATA SSD and into a PC supporting only SATA v2 3Gb/s speed.

Is a motherboard firmware (BIOS) upgrade available?

gdisk, x for expert mode, f to generate new random GUIDs.

Good suggestion, but this isn’t the problem. There are four SATA ports, but I’ve seen no dependency on which port is being used for which drive, except that when multiple drives have the same PTUUID the one that boots is (probably?) dependent on the order the SATA ports are checked.

One more note: having duplicate partition UUIDs is a different problem. If you try to mount a partition that has a UUID that duplicates an existing mounted partition, then the mount will fail because mount won’t be able to create the /dev/disk/by-uuid link.

So, cloned disks are multiple flavors of trouble.

mount never created this link.

OK, here are the symptoms. If I have two drives attached with the same partition UUIDs, and the system has booted from one, then trying to mount the a partition on the second drive produces this error:

> sudo mkdir /mnt/tmp
> sudo mount /dev/nvme0n1p2 /mnt/tmp
mount: /mnt/tmp: mount(2) system call failed: File exists.
>

So, perhaps it isn’t mount creating that link, but something in the system is complaining that a file already exists, and I’m betting it’s that /dev/disk/by-uuid/XXXXX. Whether or not, though, the system does have trouble mounting multiple partitions with the same UUIDs.

From man gdisk, the expert f option:

Randomize the disk’s GUID and all partitions’ unique GUIDs (but not their partition type code GUIDs). This function may be used after cloning a disk with another utility in order to render all GUIDs once again unique.

Thanks. That’s a start, but it isn’t sufficient, as it will render the drive unbootable, as (e.g.) /etc/fstab’s drive references will be wrong.

The topic is drifting, and my original question has been answered, so I’ll raise this in another post.

New post: How to change GUIDs on a cloned drive?

The Linux kernel cannot and does not support booting with such configuration. Results are unpredictable, with catastrophic data loss very possible.

What filesytsem is on this device?