Qemu setup to boot physical hard drive

[Konfekt]

Good afternoon,

My computer has two hard drives, one booting into Opensuse 15.5, the other one into Microsoft Windows 11. Instead of booting the latter by resetting the computer, I’d like to boot it in Qemu 7.1.0 on Opensuse 15.5.

All other VM images, be it Linux or Microsoft Windows, boot fine in Qemu, but the physical hard drive is stuck at the boot screen.

According to How to boot a physical Windows partition with qemu? - Super User parameters such as

qemu-system-x86_64 --enable-kvm -cpu host -smp 8 -m 8192 -drive format=raw,file=/dev/nvme0n1

for a suitable drive path should work.
However, this then is stuck at the inital SEA bios prompt reading

SeaBl0S (version rel-1.16.0-0-94239552c-rebuilt.opensuse.org)
iPXE (http://ipxe org) 00:03.0 CAGO PCIZ.10 PnP PMM+BFF90090+BFEFO090 CHOO
Booting from Hard Disk…

Another post mentions the necessity of the right bios binary, among those supplied in /usr/share/qemu and recommends loading the VMF_CODE one by suitable -L and --bios parameters (instead of the defaul SEA bios).
With the choice of such a bios, a short message

BdsDxe: failed to load Boot0001 "UEFI QEMU DUD-ROM QM00003 ” from PciRoot (0x0) /Pci (0x1,0x1) /Ata (Secondary. Master ‚0x0) : Not Found

shows up before

it is stuck on the Windows Boot Manager screen reading

Windoze failed to start. A recent hardware or software change might be the
cause. To fix the problem:

1. Insert your Windows installation disc and restart your computer.
2. Choose your language settings, and then click “Next.”
3. Click "Repair your computer.”

If you do not have this disc, contact your system administrator or computer
manufacturer for assistance.

File: \EFI\Microsoft\Boot\BCD
Status: 0xc000000f

Info: The Boot Configuration Data for your PC is missing or contains
errors.

However, the Windows hard drive boots fine resetting the computer and opting for it in the BCD boot menu.
Therefore, it seems that Qemu is not correctly set up.
Since I rather rely on how-tos on the internet to set up Qemu, I’d appreciate some hint how to hone the Qemu start up sequence of the physical Microsoft Windows hard disk.

Best wishes

Enno

[malcolmlewis]

@Konfekt are the storage devices on separate disk controllers? If so then you would need to use vfio-pci to passthrough the device/controller.

[arvidjaar]

@Konfekt

qemu-system-x86_64 --enable-kvm -cpu host -smp 8 -m 8192 -drive format=raw,file=/dev/nvme0n1

Default guest drive interface is IDE. You would need to do whatever preparations are necessary to move Windows from physical NVMe to physical IDE.

@Konfekt

Windoze failed to start. A recent hardware or software change might be the
cause.

See above. This is absolutely normal for a Windows which was moved to another drive type/different disk controller etc.

If you carefully read the link you yourself posted, it tells you to perform Windows boot recovery …

Passing PCI device to VM (if supported by your system) is another option.

[arvidjaar]

@malcolmlewis

are the storage devices on separate disk controllers

NVMe is a separate PCIe device in any case.

[Konfekt]

@malcolmlewis Thank you for the hint, they are indeed on different PCIe slots and have different disk controllers as udevadm info -q path -n /dev/nvme0n1 and nvme1n1 indicates.
May I ask how to enable vfio-pci to passthrough, maybe by giving the suitable command-line parameter for qemu?

If I understand @arvidjaar correctly, this is always the case if one of them is an NVMe drive.

@arvidjaar The answers are somewhat contradictory, I mainly referred to How to boot a physical Windows partition with qemu? - Super User for an NVME drive.

I am a bit wary of performing boot recovery, as it took some effort to make Microsoft Windows boot fine without messing up the boot order for Opensuse, in particular the BCD entries. I am worried that changes to Windows done in Qemu persist onto the physical hard drive.
Therefore, I’d prefer to fix the Qemu setup first.

I am not sure what is meant by moving it to another drive type/disk controller, …

Microsoft Windows was installed in absence of other hard drives and boots fine when choosing, when picking the corresponding boot menu entry.

If passing PCI device to VM means converting the hard drive to a file, I’d prefer to keep only one copy of Microsoft to ease its maintenance;
however, as said in the opening post, it would be convenient to boot it in Qemu without resetting the computer.

[malcolmlewis]

@Konfekt need to confirm with;

/sbin/lspci -nnk | grep -EA3 Non-Volatile

[Konfekt]

The output shows

3c:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 [144d:a808]
	Subsystem: Samsung Electronics Co Ltd SSD 970 EVO/PRO [144d:a801]
	Kernel driver in use: nvme
	Kernel modules: nvme
3d:00.0 Non-Volatile memory controller [0108]: Sandisk Corp PC SN520 x2 M.2 2242 NVMe SSD [15b7:5005] (rev 01)
	Subsystem: Sandisk Corp PC SN520 x2 M.2 2242 NVMe SSD [15b7:5005]
	Kernel driver in use: nvme
	Kernel modules: nvme

I thought

$ udevadm info -q path -n /dev/nvme1n1          
/devices/pci0000:00/0000:00:1d.0/0000:3d:00.0/nvme/nvme1/nvme1n1
$ udevadm info -q path -n /dev/nvme0n1
/devices/pci0000:00/0000:00:1c.4/0000:3c:00.0/nvme/nvme0/nvme0n1

these to confirm that different PCIe slots are used

[malcolmlewis]

@Konfekt what CPU in this system, AMD or Intel? Need to enable iommu and also make sure they are in their own group… For Intel you need to add the kernel boot option intel_iommu=on

I use vfio-pci for gpu’s, but in the past also used a separate controller for SSD’s to pass through.

Then need to check with journalctl -b | grep -i -e amd-vi -e dmar then find /sys/kernel/iommu_groups/ -type l and make sure 3c:00.0 and 3d:00.0 are in there own groups.

[Konfekt]

Thank you again.

@malcolmlewis

find /sys/kernel/iommu_groups/ -type l

Looking good:

/sys/kernel/iommu_groups/17/devices/0000:3d:00.0
/sys/kernel/iommu_groups/7/devices/0000:00:1c.2
/sys/kernel/iommu_groups/15/devices/0000:3b:00.0
/sys/kernel/iommu_groups/5/devices/0000:00:16.0
/sys/kernel/iommu_groups/13/devices/0000:03:01.0
/sys/kernel/iommu_groups/3/devices/0000:00:08.0
/sys/kernel/iommu_groups/11/devices/0000:02:00.0
/sys/kernel/iommu_groups/1/devices/0000:00:02.0
/sys/kernel/iommu_groups/8/devices/0000:00:1c.4
/sys/kernel/iommu_groups/16/devices/0000:3c:00.0

Where to go next from here?

[malcolmlewis]

@Konfekt OK, so based on you output the device to use is;

/devices/pci0000:00/0000:00:1c.4/0000:3c:00.0/nvme/nvme0/nvme0n1

You may want to disable MSRs, but may not be necessary, I’m only running gpu’s at present

cat /etc/modprobe.d/10-kvm.conf 

options kvm ignore_msrs=1
options kvm report_ignored_msrs=0

Run the command cat /sys/bus/pci/devices/0000:3c:00.0/modalias

Create a file called: /etc/modprobe.d/11-vfio-pci.conf containing;

alias <the output from the above cat command> vfio-pci
options vfio-pci ids=144d:a808,144d:a801

You may also need to add a dracut conf, on Tumbleweed I don’t have it…

/etc/dracut.conf.d/01-vfio.conf

add_drivers+=" vfio vfio_iommu_type1 vfio_pci vfio_virqfd"

Reboot and run the /sbin/lspci -nnk | grep -EA3 Non-Volatile command again and hopefully the driver in use is vfio-pci rather than nvme.

[Konfekt]

@malcolmlewis

the device to use is;

/devices/pci0000:00/0000:00:1c.4/0000:3c:00.0/nvme/nvme0/nvme0n1

I am not sure what use here means; the drive containing the NTFS Windows installation is rather nvme1n1?

Thank you for these precise instructions. At the end, /sbin/lspci -nnk | grep -EA3 Non-Volatile showed both drives still to use the nvme driver:

3c:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 [144d:a808]
	Subsystem: Samsung Electronics Co Ltd SSD 970 EVO/PRO [144d:a801]
	Kernel driver in use: nvme
	Kernel modules: nvme
3d:00.0 Non-Volatile memory controller [0108]: Sandisk Corp PC SN520 x2 M.2 2242 NVMe SSD [15b7:5005] (rev 01)
	Subsystem: Sandisk Corp PC SN520 x2 M.2 2242 NVMe SSD [15b7:5005]
	Kernel driver in use: nvme
	Kernel modules: nvme

Would you mind digressing a bit what was meant to be achieved to make Qemu boot the Windows drive? If vfio-pci was its driver, could it still be accessed as before from Linux? Or only from Qemu?

@arvidjaar mentioned

Default guest drive interface is IDE. You would need to do whatever preparations are necessary to move Windows from physical NVMe to physical IDE.

Is the vfio-pci driver a means to avoid having to do this? If IDE a default, it sounds like the guest drive interface could still be chosen different from IDE?

[malcolmlewis]

@Konfekt Ahh ok, so it would be the other device then… No, once using vfio-pci it won’t be available to the host.

That all depends on the hardware, I did have some success with a separate SATA controller and unbinding the SATA device and using on the host again. This however was on a Motherboard that supported hot swapping of SATA devices…

[Konfekt]

I am out of luck, the alias was adapted accordingly

alias pci:v000015B7d00005005sv000015B7sd00005005bc01sc08i02 vfio-pci
options vfio-pci ids=15b7:5005

but to no avail, nvme is still used.
Any suggestions what I could have missed?

[malcolmlewis]

@Konfekt you need both ID’s, even though same…

options vfio-pci ids=15b7:5005,15b7:5005

[arvidjaar]

@Konfekt

Is the vfio-pci driver a means to avoid having to do this?

Correct.

@Konfekt

If IDE a default, it sounds like the guest drive interface could still be chosen different from IDE?

Sure, man qemu-system-x86_64, search for -drive parameter, option id=interface. You may need to also create corresponding bus (where interface is connected).

Hmm … actually you can also emulate NVMe which may work. Look at NVMe Emulation — QEMU documentation. In you case

 -drive format=raw,if=none,id=nvm,file=/dev/nvme0n1 -device nvme,serial=SERIAL_NUMBER_OF_YOUR_nvme0n1,drive=nvm

Windows may well keep track of boot device using its serial number so you better use the real one.

[Konfekt]

@malcolmlewis

@Konfekt you need both ID’s, even though same…

options vfio-pci ids=15b7:5005,15b7:5005

I am out of luck, this still lets both drives, in particular the second one, use the nvme driver.

[Konfekt]

Thanks for the documentation hints, I am now using

qemu-system-x86_64 -L /usr/share/qemu --bios ovmf-x86_64.bin --enable-kvm -cpu host -smp 8 -m 8192 -drive file=/dev/nvme1n1,if=none,format=raw,id=nvm -device nvme,serial=4332B2798871,drive=nvm

giving a boot screen reading

BdsDxe: failed to load Boot0001 "UEFI QEMU DUD-ROM QM00003 ” from PciRoot (0x0) /Pci (0x1,0x1) /Ata (Secondary. Master ‚0x0) : Not Found

Start PXE over IPu4.

I have no idea which BIOS binary is correct; removing its parameters yields a screen stuck at saying

SeaBl0S (version rel-1.16.0-0-94239552c-rebuilt.opensuse.org)
iPXE Chttp://ipxe.org) 00:03.0 CAGO PCIZ.10 PnP PMM+BFF90040+BFEF0040 CNOO
Booting from Hard Disk…

@arvidjaar: Does this hint at having to create the corresponding bus (where interface is connected)? How would that go?

Thank you for your assistance

[arvidjaar]

Is NVMe device visible in BIOS? Start pressing ESC as soon as you see console window, you should enter BIOS setup, go to Boot Manager:

This is with

qemu-system-x86_64 -L /usr/share/ovmf/ --bios OVMF.fd --enable-kvm -cpu host  -drive file=/tmp/nvme.raw,if=none,format=raw,id=nvm -device nvme,serial=4332B2798871,drive=nvm

@Konfekt

Does this hint at having to create the corresponding bus (where interface is connected)?

Not really, VM should have PCI bus by default.

[arvidjaar]

@arvidjaar

This is with

I tried using the HD image of my UEFI VM and it booted into grub2 menu. So NVMe emulation in QEMU works in principle.

[Konfekt]

@arvidjaar

Is NVMe device visible in BIOS?

No, it’s not with

qemu-system-x86_64 -L /usr/share/qemu --bios ovmf-x86_64.bin --enable-kvm -cpu host -smp 8 -m 8192 -drive file=/dev/nvme1n1,if=none,format=raw,id=nvm -device nvme,serial=4332B2798871,drive=nvm

Any idea what went wrong?