KVM with vga passtrough

Hi all :slight_smile: I’m about to install opensuse Leap 42.1 and I have some questions. Is it possible to setup vga passtrough with IOMMU using tutorial for Arch? It looks like it should be compatible, but it would be great if someone confirm that.
I have Intel HD4000 gpu integrated with my cpu, and Nvidia GTX680. I want to use HD4000 for X server (rendering desktop, etc.), and pass Nvidia to VM. The problem is that i want to be able to use Nvidia for CUDA rendering in Blender running on suse (when VM is not running). Afaik CUDA requires nvidia drivers, but I need blacklist nvidia modules to be able to run vga passtrough (according to Arch tutorial). I though about something like conditional loading modules (for example module is loaded with OS, but i unload it manually before starting VM), but I don’t know if it’s possible and will work. Thanks in advance and sorry for my bad english c:

That is an optimus system which is not a normal NVIDIA setup. You can not use the normal NVIDIA Driver You must use the bumblebee programs and the nvidia-bumblebee driver. Unless you can turn off the Intel GPU

In any case nothing is easy with an Optimus system so you really need to read up on it

here is basic instruction

https://en.opensuse.org/SDB:NVIDIA_Bumblebee

Thank you for your answer. It’s a desktop PC. I can turn off Intel GPU, but i don’t want to do this, because it will make me unable to launch vga passtrough. Bumblebee looks good, I’ll try it if there will be no other ideas how to accomplish this.

No if this is a desktop it is NOT an Optimus system and bumblebee will not work. DO NOT USE Bumblebee. Optimus is a special hardware set up on some hybrid GPU lap tops only

Here is the problem the NVIDIA driver has for about forever replaced parts of the X stack (mesa in particular) with it’s own version this makes things broken for any other brand GPU so the Intel can not work with the NVIDIA driver. The nouveau can work with the Intel but I guess that is not what you want.

Maybe this will help

https://forums.opensuse.org/showthread.php/511230-NVIDIA-Optimus-without-bumbleebee

but this method is to switch from one GPU to the other like Bumblebee. not run them at the same time

If you want someone’s opinion on a reference, you’ll have to provide the URL, not just refer to it… after all, your reference to “Arch” might refer to a forum thread, the wiki, official documentation and sometimes there are multiple articles about various aspects of a topic.

In general though, I rely heavily on Arch Wiki articles for things not covered by official openSUSE/SUSE documentation.

Re your topic, you need to also specify what virtualization you’re using, eg KVM, QEMU, Xen or something else (I’m guessing based on your reference to Arch) and whether you have libvirt installed and are using vm manager.

You should be aware that the whole topic of vga pass-through is still very much leading/bleeding edge with relatively little well-tested guidance, and then to suggest that you want to also run CUDA puts you in an even more rarefied scenario.

In any case, I would encourage you to try and if you run into issues or attain success, to go ahead and post again here.
And when you post, include all details like the specific guide you’re following, what if tried and if it didn’t work what were the issues you either were able to resolve or need help on.

TSU

Thanks for your answers c:
I’m basing mainly on Arch wiki page and this thread.
I’m using qemu with KVM, without any vm manager, and afaik without using libvirt.
Today I installed Leap 42.1. Grub was updating for more than 30minutes, I was close to give up, but finally grub managed to finish installing and updating. Anyway, I managed to set up working vm with vga passtrough. Here is my starting command (I need to dev it some more, but even now it do the most important work correct):

qemu-system-x86_64 \
 -enable-kvm m 8192 \ 
 -cpu host,kvm=off \ 
 -smp cores=4,threads=2,sockets=1 \ 
 -vga none \ 
 -drive if=pflash,format=raw,readonly,file="/path/to/OVMF-pure-efi.fd" \ 
 -drive if=pflash,format=raw,file="/path/to/my_vars.fd"  \ 
 -device vfio-pci,host=01:00.0 \ 
 -device virtio-scsi-pci,id=scsi \ 
 -drive file="/path/to/win10.qcow2",id=disk,format=raw,if=none \ 
 -device scsi-hd,drive=disk \ 
 -usb \ 
 -usbdevice host:e0ff:0005 \ 
 -usbdevice host:04d9:a02e

Now it’s time to try make CUDA working on suse, while vm is off.

So the question is that how to accomplish this? To clarify: I want to use integrated Intel GPU to run X, and have option to use GTX680 as a CUDA renderer in Blender. But afaik I cannot have loaded Nvidia driver when I’m starting vm with vga passtrough, so I need option to unload this driver without rebooting.
Tell me if you need any other informations, or something is not clear, I’ll try to do the best I can. Thanks in advance :slight_smile:

Thx for posting.

Your research and references to Arch articles are significant for anyone attempting to do what you describe.
I don’t know how you can unload the nVidia from your Guest (if that is what you’re asking) without shutting down the Guest because that is the only GPU/CPU it’s assigned when started.

If you’re asking how to return the nVidia to the Host without the <Host> rebooting, I would assume that should happen once the Guest is shutdown entirely. If it’s not returned automatically, I suspect then there is still a QEMU related process that’s still running. Maybe stopping virtualization services altogether might be tried as part of the troubleshooting.

HTH,
TSU

Well, it looks that you misunderstood me. I don’t want to unload Nvidia drivers on guest. I want to be able to use CUDA on host (which probably need nvidia driver loaded on host) XOR launch vm with vga passtrough. Anyway, now I’m stuck with installing Nvidia drivers, because installation fails with that error:

ERROR: Unable to load the kernel module 'nvidia.ko'.  This happens most frequently when this kernel module was built against the wrong or improperly configured kernel sources, with a version of gcc that differs from the one used to build the target kernel, or if a driver such as rivafb, nvidiafb, or nouveau is present and prevents the NVIDIA kernel module from obtaining ownership of the NVIDIA graphics device(s), or no NVIDIA GPU installed in this system is supported by this NVIDIA Linux graphics driver release.

I think its caused by boot parameters, which are pci-stub.ids=10de:1180,10de:0e0a, but without them for some reason KDE doesn’t start telling that “could not start kmserver” and throw back to SDDM login screen. Without that pci-stub parameter GTX680 is getting initialized by something (without any driver GTX fans are running quite loud, while with driver loaded they slow down noticeably), but SDDM login screen is rendered by Intel gpu. I tried blacklisting nouveau and nvidia in modprobe, but it doesn’t change anything.
What could I do?

Anything with “ko” as the filename suffix is a kernel module. This should be a Kernel Loadable Module (often referred to as a KLM) which depending on circumstances might be loadable only at boot or later on demand.
Where did this come from?
Was it compiled on the fly when you added the driver or was it pre-compiled somewhere else?

The error reference suggests the gcc version is incorrect. You can find out which version gcc you have with the following

gcc --version

I don’t know if it should make a difference if your gcc should be 4.8 or 5 to maybe fix this issue if your nVidia kernel module was compiled on the fly.

TSU

gcc --version output:

gcc --version 
gcc (SUSE Linux) 4.8.5

I’ve got this error while i was running driver installer from nvidia website (*.run). In my opinion it’s not the gcc fault, but the problem is with pci-stub’ed gpu, as error says:

or no NVIDIA GPU installed in this system is supported by this NVIDIA Linux graphics driver release.

My plan is to:

  1. Disable pci-stub, and make KDE working without pci-stub (as i said in previous post, KDE doesn’t load without pci-stub’ed GTX) (I want to have desktop rendered by Intel integrated gpu)
  2. Install nvidia driver and fix (if any) problems with X configuration, to continue using only Intel for rendering desktop.
  3. Then I should be able to pci-stub gpu using modprobe (step 4 from here).

So as for now I need help with making KDE work without pci-stub (:

The VM has captured the card so the host does not see it

Ok, I installed nvidia drivers, and manually edited xorg.conf generated by nvidia installer:

# nvidia-xconfig: X configuration file generated by nvidia-xconfig# nvidia-xconfig:  version 352.63  (buildmeister@swio-display-x64-rhel04-11)  Sat Nov  7 22:00:19 PST 2015


Section "ServerLayout"
   Identifier     "Layout0"
   Screen      0  "Screen0"
   InputDevice    "Keyboard0" "CoreKeyboard"
   InputDevice    "Mouse0" "CorePointer"
EndSection


Section "Files"
EndSection


Section "InputDevice"
    #generated from data in "/etc/sysconfig/mouse"
   Identifier     "Mouse0"
   Driver         "mouse"
   Option         "Protocol" "IMPS/2"
   Option         "Device" "/dev/input/mice"
   Option         "Emulate3Buttons" "yes"
   Option         "ZAxisMapping" "4 5"
EndSection


Section "InputDevice"
   # generated from default
   Identifier     "Keyboard0"
   Driver         "kbd"
EndSection


Section "Monitor"
   Identifier     "Monitor0"
   VendorName     "Unknown"
   ModelName      "Unknown"
   HorizSync       28.0 - 33.0
   VertRefresh     43.0 - 72.0
   Option         "DPMS"
EndSection


Section "Device"
    Identifier     "Intel Graphics"
    Driver         "intel"
    VendorName     "Intel Corporation"
EndSection


Section "Screen"
   Identifier     "Screen0"
   Device         "Intel Graphics"
   Monitor        "Monitor0"
   DefaultDepth    24
   SubSection     "Display"
       Depth       24
   EndSubSection
EndSection

Xorg start on Intel GPU, as it should, but KDE won’t start. I have black screen and after few seconds I get this error (plasmashell has been Aborted). How to fix it?

Hm, it looks that I’m not able to edit my post. Here is log from plasmashell:

kscreen: launcherDataAvailable: "org.kde.KScreen.Backend.XRandR"kscreen: Primary output changed from KScreen::Output(Id: 69 , Name: "HDMI1" ) ( "HDMI1" ) to KScreen::Output(Id: 69 , Name: "HDMI1" ) ( "HDMI1" )
unversioned plugin detected, may result in instability
unversioned plugin detected, may result in instability
unversioned plugin detected, may result in instability
unversioned plugin detected, may result in instability
unversioned plugin detected, may result in instability
unversioned plugin detected, may result in instability
unversioned plugin detected, may result in instability
unversioned plugin detected, may result in instability
unversioned plugin detected, may result in instability
unversioned plugin detected, may result in instability
unversioned plugin detected, may result in instability
unversioned plugin detected, may result in instability
unversioned plugin detected, may result in instability
unversioned plugin detected, may result in instability
unversioned plugin detected, may result in instability
unversioned plugin detected, may result in instability
unversioned plugin detected, may result in instability
unversioned plugin detected, may result in instability
unversioned plugin detected, may result in instability
unversioned plugin detected, may result in instability
unversioned plugin detected, may result in instability
unversioned plugin detected, may result in instability
unversioned plugin detected, may result in instability
kscreen: Launcher finished with exit code 1 , status 0
kscreen: Service for requested backend already running
kscreen: Primary output changed from KScreen::Output(Id: 69 , Name: "HDMI1" ) ( "HDMI1" ) to KScreen::Output(Id: 69 , Name: "HDMI1" ) ( "HDMI1" )
QXcbIntegration: Cannot create platform OpenGL context, neither GLX nor EGL are enabled
Failed to create OpenGL context for format QSurfaceFormat(version 2.0, options QFlags(), depthBufferSize 24, redBufferSize -1, greenBufferSize -1, blueBufferSize -1, alphaBufferSize 8, stencilBufferSize 8, samples -1, swapBehavior 2, swapInterval 1, profile  0) 
Application::crashHandler() called with signal 6; recent crashes: 1
KCrash: Application 'plasmashell' crashing...
KCrash: Attempting to start /usr/lib64/libexec/drkonqi from kdeinit
kscreen: launcherDataAvailable: "org.kde.KScreen.Backend.XRandR"
kscreen: Primary output changed from KScreen::Output(Id: 69 , Name: "HDMI1" ) ( "HDMI1" ) to KScreen::Output(Id: 69 , Name: "HDMI1" ) ( "HDMI1" )
unversioned plugin detected, may result in instability
unversioned plugin detected, may result in instability
unversioned plugin detected, may result in instability
unversioned plugin detected, may result in instability
unversioned plugin detected, may result in instability
unversioned plugin detected, may result in instability
unversioned plugin detected, may result in instability
unversioned plugin detected, may result in instability
unversioned plugin detected, may result in instability
unversioned plugin detected, may result in instability
unversioned plugin detected, may result in instability
unversioned plugin detected, may result in instability
unversioned plugin detected, may result in instability
unversioned plugin detected, may result in instability
unversioned plugin detected, may result in instability
unversioned plugin detected, may result in instability
unversioned plugin detected, may result in instability
unversioned plugin detected, may result in instability
unversioned plugin detected, may result in instability
unversioned plugin detected, may result in instability
unversioned plugin detected, may result in instability
unversioned plugin detected, may result in instability
unversioned plugin detected, may result in instability
kscreen: Launcher finished with exit code 1 , status 0
kscreen: Service for requested backend already running
kscreen: Primary output changed from KScreen::Output(Id: 69 , Name: "HDMI1" ) ( "HDMI1" ) to KScreen::Output(Id: 69 , Name: "HDMI1" ) ( "HDMI1" )
QXcbIntegration: Cannot create platform OpenGL context, neither GLX nor EGL are enabled
Failed to create OpenGL context for format QSurfaceFormat(version 2.0, options QFlags(), depthBufferSize 24, redBufferSize -1, greenBufferSize -1, blueBufferSize -1, alphaBufferSize 8, stencilBufferSize 8, samples -1, swapBehavior 2, swapInterval 1, profile  0) 
Application::crashHandler() called with signal 6; recent crashes: 2
KCrash: Application 'plasmashell' crashing...
KCrash: Attempting to start /usr/lib64/libexec/drkonqi from kdeinit
kscreen: launcherDataAvailable: "org.kde.KScreen.Backend.XRandR"
kscreen: Primary output changed from KScreen::Output(Id: 69 , Name: "HDMI1" ) ( "HDMI1" ) to KScreen::Output(Id: 69 , Name: "HDMI1" ) ( "HDMI1" )
unversioned plugin detected, may result in instability
unversioned plugin detected, may result in instability
unversioned plugin detected, may result in instability
unversioned plugin detected, may result in instability
unversioned plugin detected, may result in instability
unversioned plugin detected, may result in instability
unversioned plugin detected, may result in instability
unversioned plugin detected, may result in instability
unversioned plugin detected, may result in instability
unversioned plugin detected, may result in instability
unversioned plugin detected, may result in instability
unversioned plugin detected, may result in instability
unversioned plugin detected, may result in instability
unversioned plugin detected, may result in instability
unversioned plugin detected, may result in instability
unversioned plugin detected, may result in instability
unversioned plugin detected, may result in instability
unversioned plugin detected, may result in instability
unversioned plugin detected, may result in instability
unversioned plugin detected, may result in instability
unversioned plugin detected, may result in instability
unversioned plugin detected, may result in instability
unversioned plugin detected, may result in instability
kscreen: Launcher finished with exit code 1 , status 0
kscreen: Service for requested backend already running
kscreen: Primary output changed from KScreen::Output(Id: 69 , Name: "HDMI1" ) ( "HDMI1" ) to KScreen::Output(Id: 69 , Name: "HDMI1" ) ( "HDMI1" )
QXcbIntegration: Cannot create platform OpenGL context, neither GLX nor EGL are enabled
Failed to create OpenGL context for format QSurfaceFormat(version 2.0, options QFlags(), depthBufferSize 24, redBufferSize -1, greenBufferSize -1, blueBufferSize -1, alphaBufferSize 8, stencilBufferSize 8, samples -1, swapBehavior 2, swapInterval 1, profile  0) 
Application::crashHandler() called with signal 6; recent crashes: 3
Too many crashes in short order or respawning disabled, not restarting automatically.
KCrash: Application 'plasmashell' crashing...
KCrash: Attempting to start /usr/lib64/libexec/drkonqi from kdeinit



If your PCI device is made available to guest via pass through, it is not available to host. This is as simple as that. You will need to remove pass through configuration on host (i.e. vfio-pci binding), reboot and then you will be able to use device on host again.

You can’t have them both at the same time.

I think the question which I’ve never heard asked before is whether it’s possible to shut down the Guest using the pass through and then undo the pass-through binding without rebooting the Host.

I imagine it should be possible to do that but as I said I’ve never heard the question asked before, and the whole topic is relatively on the bleeding edge so maybe the people who figure out how to do something haven’t gotten around yet to describe to undo what they worked so hard on to configure.

TSU

It should be possible here’s my system (I have amd for guest passthrough). First from lspci -vv -s 03:00.0

03:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii PRO [Radeon R9 290] (prog-if 00 [VGA controller])
    Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device 0b00
    Physical Slot: 15
    Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Interrupt: pin A routed to IRQ 36
    Region 0: Memory at e0000000 (64-bit, prefetchable) [size=256]
    Region 2: Memory at f0000000 (64-bit, prefetchable) [size=8]
    Region 4: I/O ports at d000 [size=256]
    Region 5: Memory at fb600000 (32-bit, non-prefetchable) [size=256]
    Expansion ROM at fb640000 [disabled] [size=128]
    Capabilities: [48] Vendor Specific Information: Len=08 <?>
    Capabilities: [50] Power Management version 3
        Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1+,D2+,D3hot+,D3cold-)
        Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
    Capabilities: [58] Express (v2) Legacy Endpoint, MSI 00
        DevCap:    MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited
            ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
        DevCtl:    Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
            RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
            MaxPayload 256 bytes, MaxReadReq 512 bytes
        DevSta:    CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
        LnkCap:    Port #2, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
            ClockPM- Surprise- LLActRep- BwNot-
        LnkCtl:    ASPM L1 Enabled; RCB 64 bytes Disabled- CommClk+
            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
        LnkSta:    Speed 8GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
        DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
        DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
        LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
             Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
             Compliance De-emphasis: -6dB
        LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
             EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
    Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
        Address: 0000000000000000  Data: 0000
    Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
    Capabilities: [150 v2] Advanced Error Reporting
        UESta:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UEMsk:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UESvrt:    DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
        CESta:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
        CEMsk:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
        AERCap:    First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
    Capabilities: [270 v1] #19
    Capabilities: [2b0 v1] Address Translation Service (ATS)
        ATSCap:    Invalidate Queue Depth: 00
        ATSCtl:    Enable-, Smallest Translation Unit: 00
    Capabilities: [2c0 v1] #13
    Capabilities: [2d0 v1] #1b
    Kernel driver in use: pci-stub
    Kernel modules: radeon

We see the kernel driver associated with the card is pci-stub. We can unbind it by going to that node in the device tree:


alcyone:/sys/devices/pci0000:00/0000:00:02.2/0000:03:00.0/driver # ll
total 0
lrwxrwxrwx 1 root root    0 Dec 13 08:25 0000:00:1b.0 -> ../../../../devices/pci0000:00/0000:00:1b.0
lrwxrwxrwx 1 root root    0 Dec 13 08:27 0000:03:00.0 -> ../../../../devices/pci0000:00/0000:00:02.2/0000:03:00.0
lrwxrwxrwx 1 root root    0 Dec 13 08:25 0000:03:00.1 -> ../../../../devices/pci0000:00/0000:00:02.2/0000:03:00.1
--w------- 1 root root 4096 Dec 13 08:27 bind
--w------- 1 root root 4096 Dec 13 08:25 new_id
--w------- 1 root root 4096 Dec 13 08:25 remove_id
--w------- 1 root root 4096 Dec 13 08:25 uevent
--w------- 1 root root 4096 Dec 13 08:26 unbind

And echo the device path into the unbind file to disassociate the pci-stub driver:


alcyone:/sys/devices/pci0000:00/0000:00:02.2/0000:03:00.0/driver # echo 0000:03:00.0 > unbind


alcyone:/sys/devices/pci0000:00/0000:00:02.2/0000:03:00.0/driver # lspci -vv -s 03:00.0
03:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii PRO [Radeon R9 290] (prog-if 00 [VGA controller])
    Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device 0b00
    Physical Slot: 15
    Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Interrupt: pin A routed to IRQ 0
    Region 0: Memory at e0000000 (64-bit, prefetchable) [size=256]
    Region 2: Memory at f0000000 (64-bit, prefetchable) [size=8]
    Region 4: I/O ports at d000 [size=256]
    Region 5: Memory at fb600000 (32-bit, non-prefetchable) [size=256]
    Expansion ROM at fb640000 [disabled] [size=128]
    Capabilities: [48] Vendor Specific Information: Len=08 <?>
    Capabilities: [50] Power Management version 3
        Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1+,D2+,D3hot+,D3cold-)
        Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
    Capabilities: [58] Express (v2) Legacy Endpoint, MSI 00
        DevCap:    MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited
            ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
        DevCtl:    Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
            RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
            MaxPayload 256 bytes, MaxReadReq 512 bytes
        DevSta:    CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
        LnkCap:    Port #2, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
            ClockPM- Surprise- LLActRep- BwNot-
        LnkCtl:    ASPM L1 Enabled; RCB 64 bytes Disabled- CommClk+
            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
        LnkSta:    Speed 8GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
        DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
        DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
        LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
             Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
             Compliance De-emphasis: -6dB
        LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
             EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
    Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
        Address: 0000000000000000  Data: 0000
    Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
    Capabilities: [150 v2] Advanced Error Reporting
        UESta:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UEMsk:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UESvrt:    DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
        CESta:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
        CEMsk:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
        AERCap:    First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
    Capabilities: [270 v1] #19
    Capabilities: [2b0 v1] Address Translation Service (ATS)
        ATSCap:    Invalidate Queue Depth: 00
        ATSCtl:    Enable-, Smallest Translation Unit: 00
    Capabilities: [2c0 v1] #13
    Capabilities: [2d0 v1] #1b
    Kernel modules: radeon


So we can indeed unbind the device from the pci-stub driver without rebooting. Now there is another matter of the nvidia driver and whether or not it will be able to initialize the card without going through some sort of vga bios. The open source radeon driver seems to require KMS, so even though I can take the card back from the guest, it’s useless to me.[/size][/size][/size][/size][/size][/size][/size][/size][/size][/size]