Strange KVM issue with vfio passthrough

kdemello · April 30, 2016, 3:18pm

I am having a strange problem passing through PCIe devices with vfio. I have two virtual machines, both windows 7 guests. The first one uses i440fx and the standard bios (I created this with virt-install under openSUSE 13.2 a very long time ago). This first machine works with vfio passthrough for the video, audio and USB controller.

The second machine is identical, created with virt-install on openSUSE Leap 42.1, but with a Q35 chipset and UEFI via ovmf. Windows is installed correctly for uEFI and is now fully updated.

The problem is the video, usb and audio devices that I am passing through to this new uEFI/Q35 guest can’t be started. They show up in device manager, but they all have code: 10 errors.

If I shut down the Q35 guest and boot the i440fx guest, the devices work with the i440fx guest (they’re the same physical devices, so the two guests can’t run concurrently).

Is there something different about the Q35 type?

kdemello · April 30, 2016, 6:29pm

Seems like I’m running into this:

https://bugzilla.redhat.com/show_bug.cgi?id=1273196

tsu2 · April 30, 2016, 10:16pm

Are you sure you’re not seeing a simple, very ordinary problem?

When you enable any kind of “device pass through” you’re granting direct access to a Guest, which means that no other Guest or the Host can have access to that device.

Additionally, although not described anywhere I’ve seen, I’ve always strongly suspected that once you enable a device pass through, that configuration becomes fixed for the duration of that HostOS session, the configuration is either cached or implemented in a way that if the pass through is disabled or removed the configuration persists… And this might apply to many virtualization technologies.

So,
What I’m recommending is that after you configure your pass through for one Guest, if you want to configure that pass through for another Guest instead, shut down your Guests and reboot the HostOS before you try configuring for your new Guest.

TSU

arvidjaar · May 3, 2016, 7:29am

You could test QEMU from Virtualization project which currently has 2.6 (https://build.opensuse.org/package/show/Virtualization/qemu). Alternatively you could try to change VM configuration to use PCIe bus as suggested.

kdemello · May 5, 2016, 11:23pm

The host reboot suggestion was a bust. There is nothing in the hostOS session that would hold onto the guest passthrough assignments.

I have 4 devices to pass through, and I can’t seem to assign them all to bus 0x02, virsh edit refuses my changes when I try to put the sound codec on bus 0x02 (but accepts 0x01). I think my best bet would be to try a newer version of qemu.

I did have some limited success with the USB3 controller when I changed it to bus 0x02, but with everything set for passthrough, it doesn’t boot correctly.

tsu2 · May 6, 2016, 4:30pm

kdemello:

I am having a strange problem passing through PCIe devices with vfio. I have two virtual machines, both windows 7 guests. The first one uses i440fx and the standard bios (I created this with virt-install under openSUSE 13.2 a very long time ago). This first machine works with vfio passthrough for the video, audio and USB controller.

The second machine is identical, created with virt-install on openSUSE Leap 42.1, but with a Q35 chipset and UEFI via ovmf. Windows is installed correctly for uEFI and is now fully updated.

The problem is the video, usb and audio devices that I am passing through to this new uEFI/Q35 guest can’t be started. They show up in device manager, but they all have code: 10 errors.

If I shut down the Q35 guest and boot the i440fx guest, the devices work with the i440fx guest (they’re the same physical devices, so the two guests can’t run concurrently).

Is there something different about the Q35 type?

Re-reading your original post,
AFAIK what you’re trying to do, and the RHEL bug you reference is based on a real, physical motherboard architecture and yet… Your original post seems to describe switching architecture on the same physical machine (or am I reading your post incorrectly?).

If you’re going to pass through a physical device, there’s no emulation and you are stuck with whatever physically exists in your machine.

TSU

kdemello · May 6, 2016, 8:49pm

Hardware wise, here is what I am passing through. The iommu groups are listed.


### Group 22 ###
    00:1b.0 Audio device: Intel Corporation C610/X99 series chipset HD Audio Controller (rev 05)
### Group 30 ###
    01:00.0 USB controller: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller (rev 03)
### Group 32 ###
    03:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii PRO [Radeon R9 290]
    03:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii HDMI Audio

The xml looks like:


    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x00' slot='0x1b' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x02' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>

What the redhat bug is talking about are these lines:


      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x02' function='0x0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>

By default, they are placed on bus=‘0x02’ by libvirt when the machine is created. I managed to get the USB card to work, as I said before, by changing its, and only its bus to 0x00, as the bug suggests. That really has nothing to do with the physical slot the device is plugged into on the host. It’s defining where the guest sees it.

What I was trying to do was use the qemu Q35 emulation, instead of i440fx, since guests can be created with either. The reason I’m trying this is the newest AMD drivers blue screen with my current windows 7 guest, and I think one of the workarounds was only working with the Q35 qemu guest type.

tsu2 · May 8, 2016, 12:14am

So,
If you’re emulating Q35,
Then why are you doing any kind of pass-through?
Once you’ve emulated a specific hardware device, the GuestOS should recognize that emulated device natively without any pass through.

TSU

tsu2 · May 8, 2016, 12:37am

Thinking a bit more about what you may be trying to do, emulating Q35 <and> configuring VGA pass through,

I don’t think your RHEL bug reference is relevant. That bug is based on a real Q35 motherboard, not emulated.
You’re really operating on the bleeding edge, and may be possible only with the current LEAP kernel or later. Any earlier kernel like what is generally used on 13.2 is likely impossible. Take a look at the following Arch Wiki article which describes setting up what I think you are trying to do, in particular skim the first sections but concentrate more on the sections which use QEMU and OVMF commands. Note also you will likely have to handcraft a few config files.
https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF

Good Luck,
TSU

kdemello · May 21, 2016, 6:00pm

No, it isn’t. They are alternating between q35 and i440fx machine times (qemu -machine argument).
Kernel isn’t an issue. I’m running 4.5.5, which I compiled myself this morning.

Actually, with qemu-2.6.0, I can install the AMD crimson 16+ drivers, using the q35 machine type.

So I guess the problem is with how libvirt and qemu interact.

Here’s the working script:


#!/bin/bash

module load qemu

qemu-system-x86_64 \
    -machine pc-q35-2.6 \
    -serial none \
    -parallel none \
    -nodefaults \
    -nodefconfig \
    -no-user-config \
    -enable-kvm \
    -name Windows \
    -cpu host,kvm=off,hv_vapic,hv_time,hv_relaxed,hv_spinlocks=0x1fff,hv_vendor_id=sugoidesu \
    -smp sockets=1,cores=4,threads=1 \
    -m 8192 \
    -mem-prealloc \
    -soundhw hda \
    -device ich9-usb-uhci3,id=uhci \
    -device usb-ehci,id=ehci \
    -device nec-usb-xhci,id=xhci \
    -machine pc,accel=kvm,kernel_irqchip=on,mem-merge=off \
    -rtc base=localtime,clock=host,driftfix=none \
    -boot order=c \
    -net bridge,vlan=0,name=bridge0,br=br0 \
    -net nic,vlan=0,macaddr=00:16:3e:63:e8:65,model=e1000,name=net0 \
    -drive if=virtio,id=drive0,file=/dev/vgraid/win7-efi,format=raw,cache=none,aio=native \
    -device vfio-pci,host=03:00.0,addr=09.0,multifunction=on \
    -device vfio-pci,host=03:00.1,addr=09.1 \
    -device vfio-pci,host=01:00.0,addr=09.2 \
    -device vfio-pci,host=00:1b.0,addr=09.3 \
    -vga cirrus \

kdemello · May 21, 2016, 8:15pm

Scratch that. Even though I’m specifying q35 on the command line, it is still emulating a 440fx.

Even so, since the real problem was the updated AMD drivers blue screening the guest, and that’s now resolved, I don’t really care what machine type is emulated.