issues running qemu/kvm on leap 15.2 usin win10 guest and gpu passthrough

English Virtualization
1

quick recap for those who’re not still up to it - first: hardware

  • cpu: amd fx-8350
  • board: asus crosshair v formula-z
  • gpu1: nvidia 1030
  • gpu2: amd r9 290x

So, after a lot of tinkering and already had my hardware considered to be not compatible cause I got confused by some message which turned out to be just an information, I finally got my qemu/kvm up and running to a point where I was able to use almost full native performance of my 290x. But, unfortunately, no happy end without some additional work:
Although I was able to passthrough my gpu using vfio I don’t seem to be so lucky for my cpu: For some reason, although I have set the config to copy cpu config from host, it falls back to the opteron g3 setting when the vm is running. Also, looking into device manager in the win10 guest, it does show how many “cores” I have set for the VM (tested 2, 4 and 6) taskmanager only shows 2 cores. Also the base desktop performance is quite bad. The initial extraction during the first setup step went pretty fast - but the 2nd step in which the system is configured for the first time took over half an hour. On real hardware the whole setup takes no more than 5-10 minutes from power up to working desktop.

As there aren’t a lot of options using virt-manager I’m sure I either made some mistakes (very likely) or my hardware indeed isn’t fully compatible with what’s required for running a qemu/kvm with pci passthrough. In fact it’s even worse than using VirtualBox on a win7 host - which, at least as far as I know, doesn’T even take full advantage of Hyper-V hardware virtualization but still does quite a lot in software emulation.
I did not yet tried to configure or start the vm via terminal but only graphical virt-manager. As there seem no differences between using pci passthrough or not I don’T think that these changes make any difference.
What bothers me the most is that no matter how many cores I assign to the vm the windows taskmanager still only shows 2 although the correct number is listed in device manager. I love tinkering with my hardware - so I don’t want to accept “it’s just not compatible” quite yet.

Oh, maybe there’s one thing I have to mention: As I still use win7 and the setup dvd I use lacks the required files for UEFI I have CSM enabled. To switch over to UEFI I would need to disable the configured RAID and also re-format my system-hdd to gpt. I only tried that once as it took me quite some time to get my raid back up working again re-use the existing array instead of re-formatting the drives which would had ended up in losing my data. Aside from that I don’T have a spare drive to format for gpt as I don’T want to trash my current win7 install. TLDR: I’m currently not prepared to test if switch from CSM to UEFI does make any difference - so it’S currently no option.

2

Hi
Start directly with a qemu script, this is what I use at present (well it’s not working on Tumbleweed as iommu groups are screwed up at the moment…);

I use UEFI booting…


#!/bin/bash

#On DISK: Device Model: OCZ-AGILITY3

/data/configuration/bin/vfio-pci-gpu-binder bind

qemu-system-x86_64 \
-m 8G \
-cpu host,kvm=off,hv_vendor_id=whatever \
-smp 4,sockets=1,cores=2,threads=2 \
-rtc clock=host,base=utc \
-serial none \
-parallel none \
-vga none \
-nographic \
-usb \
-device usb-host,vendorid=0x05af,productid=0x0808 \
-device vfio-pci,host=08:00.0 \
-device vfio-pci,host=02:00.0 \
-device vfio-pci,host=02:00.1 \
-drive if=pflash,format=raw,unit=0,file=/stuff/repositories/KVM_GPU_Passthrough/qemu_vars/windows/ovmf-x86_64-ms-4m-code.bin,readonly=on \
-drive if=pflash,format=raw,unit=1,file=/stuff/repositories/KVM_GPU_Passthrough/qemu_vars/windows/ovmf-x86_64-ms-4m-vars.bin \
-boot order=c \
-machine type=pc-q35-4.0,accel=kvm,kernel_irqchip=on \
-nic tap,ifname=tap0,script=no,downscript=no

#Lets make sure qemu is all down before unbind is run!
sleep 3
echo
/data/configuration/bin/vfio-pci-gpu-binder unbind

#Used for install...
#-drive file=/data/repositories/iso_images/Windows/Win10_2004_English_x64.iso,index=1,media=cdrom \
#-boot order=dc \

The usb host is the extra keyboard, 08 is the SATA card, 02 is the nvidia card (video and audio). I bin/unbind the gpu as want access to the cuda cores in the host when qemu isn’t running.

Likewise you should be able to go in and edit the virtual machine xml config.

PS: Have you seen this? https://heiko-sieger.info/running-windows-10-on-linux-using-kvm-with-vga-passthrough/

3

I don’t know that there is such a thing as “cpu passthrough.”
Each Guest has direct access to the CPU already by way of the CPU virtualization extensions.
Maybe you might mean CPU pinning?
In any case, it’s well known in both the physical and virtual worlds that over-riding the internal processes of SMP computing usually is inadvisable… There are special situations where it makes sense to reserve and isolate CPU resources makes sense, but I doubt that any of that applies to how you describe using your physical and virtual machines.
And, note that when talking about virtual CPUs, that has nothing to do with physical CPUs (and cores).

TSU

4

Thanks for your replies as always.
I’ve just started up the vm and grabbed the very long command via /proc:

/usr/bin/qemu-system-x86_64
-name guest=win10,debug-threads=on
-S
-object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-win10/master-key.aes
-machine pc-q35-4.2,accel=kvm,usb=off,vmport=off,dump-guest-core=off
-cpu Opteron_G5,vme=on,x2apic=on,tsc-deadline=on,hypervisor=on,arat=on,tsc-adjust=on,bmi1=on,arch-capabilities=on,mmxext=on,fxsr-opt=on,cmp-legacy=on,cr8legacy=on,osvw=on,perfctr-core=on,ibpb=on,virt-ssbd=on,skip-l1dfl-vmentry=on,hv-time,hv-relaxed,hv-vapic,hv-spinlocks=0x1fff
-m 16384
-overcommit mem-lock=off
-smp 6,sockets=6,cores=1,threads=1
-uuid 578b1b6f-882f-42ae-97d5-96a8c939928b
-no-user-config
-nodefaults
-chardev socket,id=charmonitor,fd=31,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control
-rtc base=localtime,driftfix=slew
-global kvm-pit.lost_tick_policy=delay
-no-hpet
-no-shutdown
-global ICH9-LPC.disable_s3=1
-global ICH9-LPC.disable_s4=1
-boot strict=on
-device pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2
-device pcie-root-port,port=0x11,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1
-device pcie-root-port,port=0x12,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2
-device pcie-root-port,port=0x13,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x3
-device pcie-root-port,port=0x14,chassis=5,id=pci.5,bus=pcie.0,addr=0x2.0x4
-device pcie-root-port,port=0x15,chassis=6,id=pci.6,bus=pcie.0,addr=0x2.0x5
-device qemu-xhci,p2=15,p3=15,id=usb,bus=pci.2,addr=0x0
-device virtio-serial-pci,id=virtio-serial0,bus=pci.3,addr=0x0
-blockdev {"driver":"file","filename":"/var/lib/libvirt/images/win10.qcow2","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}
-blockdev {"node-name":"libvirt-2-format","read-only":false,"driver":"qcow2","file":"libvirt-2-storage","backing":null}
-device ide-hd,bus=ide.0,drive=libvirt-2-format,id=sata0-0-0,bootindex=1
-device ide-cd,bus=ide.1,id=sata0-0-1
-netdev tap,fd=33,id=hostnet0
-device e1000e,netdev=hostnet0,id=net0,mac=52:54:00:2c:30:42,bus=pci.1,addr=0x0
-chardev pty,id=charserial0
-device isa-serial,chardev=charserial0,id=serial0
-chardev spicevmc,id=charchannel0,name=vdagent
-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0
-device usb-tablet,id=input0,bus=usb.0,port=1
-spice port=5900,addr=127.0.0.1,disable-ticketing,image-compression=off,seamless-migration=on
-device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pcie.0,addr=0x1
-device ich9-intel-hda,id=sound0,bus=pcie.0,addr=0x1b
-device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0
-chardev spicevmc,id=charredir0,name=usbredir
-device usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=2
-chardev spicevmc,id=charredir1,name=usbredir
-device usb-redir,chardev=charredir1,id=redir1,bus=usb.0,port=3
-device vfio-pci,host=0000:08:00.0,id=hostdev0,bus=pci.5,addr=0x0
-device vfio-pci,host=0000:08:00.1,id=hostdev1,bus=pci.6,addr=0x0
-device virtio-balloon-pci,id=balloon0,bus=pci.4,addr=0x0
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny
-msg timestamp=on

Although I only understand just a few of these lines yet I can see why the vm starts up the way it does and way, although set otherwise, the cpu is shown as weird as it is. I guess I first have to figure out what all these options are and what they mean. I also gave the linked tutorial a quick look - and although it seems old it does link to a more recent one right at the top. Anyway - I’ll give a good read.
As for starting the vm via terminal: Haven’T tried that yet, but I guess as I now have the command that virt-manager uses to start it I should be able to modify it.
Oh, one last thing I just overlooked up until now: Currently I only have one set of keyboard and mouse using only one logitech unify dongle. I’ve read that I would need a 2nd pair so I can pass it through pretty much like my gpu - but as I don’t see a reason to use it on the host while I’m in the vm I guess it should be possible to boot this logitech dongle via vfio - use a script to bind it to the host - and when the vm is started up unbind it from the host so it’s available to be used by the vm - and after the vm has shut down to re-bind it to the host - about the same you do with your gpu. Do you mind give me some advice how to do this?

5

Hi
Yes, I have a second keyboard USB keyboard with a built in trackpad it is needed at times for the likes of BIOS control etc, suggest you look around for something. While in the virtual machine I use barrier (fork of synergy - GitHub - debauchee/barrier: Open-source KVM software), host runs the server so use the keyboard mouse across machines. The linux package is available in the oss repo, windows client from the github site.

Your keyboard/mouse probably can be bound and unbound, this is what I use for the gpu;


#!/bin/bash

# This program is free software; you can redistribute it and/or modify it
# under the terms of the GNU Lesser General Public License as published by
# the Free Software Foundation; either version 2.1 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
# Lesser General Public License for more details.
#
# Copyright (C) 2020 Malcolm Lewis <malcolmlewis@opensuse.org>

# Description: Script to bind/unbind Nvidia gpu and hdmi audio card used by QEMU
# Version: 0.0.1
# Date: 9th February, 2020
# 02:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK208B [GeForce GT 710] [10de:128b] (rev a1)
#         Subsystem: ZOTAC International (MCO) Ltd. Device [19da:6326]
#         Kernel driver in use: nvidia or vfio-pci
#         Kernel modules: nouveau, nvidia_drm, nvidia
# 02:00.1 Audio device [0403]: NVIDIA Corporation GK208 HDMI/DP Audio Controller [10de:0e0f] (rev a1)
#         Subsystem: ZOTAC International (MCO) Ltd. Device [19da:6326]
#         Kernel driver in use: snd_hda_intel or vfio-pci
#         Kernel modules: snd_hda_intel

case "$1" in
    bind)
    if  ! -h "/sys/bus/pci/drivers/vfio-pci/0000:02:00.0" ]; then
       echo '0000:02:00.0' > /sys/bus/pci/drivers/nvidia/unbind
       sleep 2
       echo '0000:02:00.0' > /sys/bus/pci/drivers/vfio-pci/bind
       echo "NVIDIA GeForce GT 710 now bound to vfio-pci..."
    else
       echo "NVIDIA GeForce GT 710 already bound to vfio-pci..."
    fi

    if  ! -h "/sys/bus/pci/drivers/vfio-pci/0000:02:00.1" ]; then
       echo '0000:02:00.1' > /sys/bus/pci/drivers/snd_hda_intel/unbind
       sleep 2
       echo '0000:02:00.1' > /sys/bus/pci/drivers/vfio-pci/bind
       echo "NVIDIA GK208 HDMI/DP Audio Controller now bound to vfio-pci..."
    else
       echo "NVIDIA GK208 HDMI/DP Audio Controller already bound to vfio-pci..."
    fi
    ;;
     unbind)
    if  -h "/sys/bus/pci/drivers/vfio-pci/0000:02:00.0" ]; then
       echo '0000:02:00.0' > /sys/bus/pci/drivers/vfio-pci/unbind
       sleep 2
       echo '0000:02:00.0' > /sys/bus/pci/drivers/nvidia/bind
       echo "NVIDIA GeForce GT 710 now bound to nvidia..."
    else
       echo "NVIDIA GeForce GT 710 already bound to nvidia..."
    fi

    if  -h "/sys/bus/pci/drivers/vfio-pci/0000:02:00.1" ]; then
       echo '0000:02:00.1' > /sys/bus/pci/drivers/vfio-pci/unbind
       sleep 2
       echo '0000:02:00.1' > /sys/bus/pci/drivers/snd_hda_intel/bind
       echo "NVIDIA GK208 HDMI/DP Audio Controller now bound to snd_hda_intel..."
    else
       echo "NVIDIA GK208 HDMI/DP Audio Controller already bound to snd_hda_intel..."
    fi
    ;;
        *)
    echo "Usage: $0 {bind|unbind}"
    exit 1
;;
esac
6

So, although my last reply was almost a year ago I guess I can finally close this topic as solved.

How: By switching over to Ubuntu.

It happened as I finally found a solution how to access a host ZFS pool from within a windows guest: libvirt-daemon-driver-storage-zfs.
According to several sources this package is currently only available to four distributions: Debian, Ubuntu, Fedora and ALT (some weird russian distro). As my main reason is a setup for gaming I remebered that valve used ubuntu as base for their steamos and that they currently actively developing proton, a custom wine-based environment with specific changes for steam games.
So, I just used the current 21.04, which provides the mentioned package in v7.0, and gave it a try - it works flawless with very good performance.

Oh, btw, before someone comes up with “Yea, nice, but there’s another way to expose a file system path as a virtual drive!” - ok, fine, I just discovered this today. If someone had told me this about one to two years ago I would already had my setup changed accordingly.

Also: As it turns out it’s not a limitation of my hardware as I mentioned in some other thread, it’s actually a limitation resulting from an old kernel and old kvm/libvirt/vfio drivers. Using more recent versions, as provided by ubuntu 21.04, comes with quite a lot of fixes make everything work on my hardware which I already thought to be incompatible.

Anyway, I just thought it may be helpful to others. Unfortunately OpenSuSE currently lacks a bit behind with its stable leap release - making it hard to use some special stuff like this even with hardware already near 10 years old.

And … so you don’t get sad: I’ll continue to use opensuse on my servers … but at least for my desktop it doesn’t fit my needs.

So long …