As you can see, it doesn’t have the line “kernel driver in use”. That’s because I deleted both the nvidia drivers and the opensource nouveau drivers to stop them from loading.
Still I cannot get VFIO to load on the card or however that’s worded, and it’s not listed in the kernel modules but I do think it’s in my kernel modules as seen by this output:
tim@linux-lgui:~> sudo modinfo vfio-pci
filename: /lib/modules/4.12.14-lp151.28.4-default/kernel/drivers/vfio/pci/vfio-pci.ko
description: VFIO PCI - User Level meta-driver
author: Alex Williamson <alex.williamson@redhat.com>
license: GPL v2
version: 0.2
suserelease: openSUSE Leap 15.1
srcversion: 43445CE6A64818F93EEACB2
depends: vfio,irqbypass,vfio_virqfd
retpoline: Y
intree: Y
vermagic: 4.12.14-lp151.28.4-default SMP mod_unload modversions
parm: ids:Initial PCI IDs to add to the vfio driver, format is "vendor:device:subvendor:subdevice:class:class_mask]]]]" and multiple comma separated entries can be specified (string)
parm: nointxmask:Disable support for PCI 2.3 style INTx masking. If this resolves problems for specific devices, report lspci -vvvxxx to linux-pci@vger.kernel.org so the device can be fixed automatically via the broken_intx_masking flag. (bool)
parm: disable_vga:Disable VGA resource access through vfio-pci (bool)
parm: disable_idle_d3:Disable using the PCI D3 low power state for idle, unused devices (bool)
tim@linux-lgui:~>
as well as:
Well I can’t find the command again but I have VFIO.conf if you all want to view it.
What do I need to do for VFIO to take over my nvidia card?
I have two GPUs, AMD 560 and Nvidia 960. I’m using the AMD card on the host, and without the VFIO working for the 960, the following error happens when I try to add the hardware and “begin installation” in virt-manager:
Unable to complete install: 'unsupported configuration: host doesn't support passthrough of host PCI devices'
Traceback (most recent call last):
File "/usr/share/virt-manager/virtManager/asyncjob.py", line 75, in cb_wrapper
callback(asyncjob, *args, **kwargs)
File "/usr/share/virt-manager/virtManager/create.py", line 2205, in _do_async_install
guest.installer_instance.start_install(guest, meter=meter)
File "/usr/share/virt-manager/virtinst/installer.py", line 444, in start_install
doboot, transient)
File "/usr/share/virt-manager/virtinst/installer.py", line 387, in _create_guest
domain = self.conn.createXML(install_xml or final_xml, 0)
File "/usr/lib64/python3.6/site-packages/libvirt.py", line 3743, in createXML
if ret is None:raise libvirtError('virDomainCreateXML() failed', conn=self)
libvirt.libvirtError: unsupported configuration: host doesn't support passthrough of host PCI devices
Somehow I need to find out how to enable VFIO for the Nvidia card, but I’m completely clueless to how all the configuration files work, and what to modify. For the work that I have done, I’ve posted on the Level1Techs forum, but it seems I need specific bits for OpenSUSE to get this to work.
Awhile ago I posted the following which even today should be “the” reference for guides to configure GPU pass-through.
It’ continues to be valid over time because it points to “live” documentation that is continuously updated.
Identify which is most appropriate for what you’re trying to do, and if you run into any obstacles, then post including the guide you’re following and provide errors and output for whatever you’re seeing
Since I had a few brief moments, I took a look at what you posted.
You never identified your GPU by name, I’m assuming that the device id you posted means your GPU is a GeForce GTX 1660
Looking up that GPU on IOMMU support lists, I don’t see it anywhere on any list. That might be a show stopper but I also located one (The only one!)Forum discussion that describes what one guy did that supposedly configured IOMMU… I didn’t inspect it closely, but it may contain useful info
If that doesn’t help, then I’d think that you’re going to have to ask nVidia directly about IOMMU support for your GPU and now to configure if possible.
Only after you get past that, then you can proceed following the ArchWiki OVMF guide.
I actually acquired a GTX 1660 Ti but I haven’t done much with it. I may sell it as I received it as a promotional item of sorts. I have a GTX 960 and an AMD 560, for the host, and Nvidia is for the VM.
To malcolmlewis: Here is my grub and other config files:
GNU nano 2.9.6 /etc/default/grub
# If you change this file, run 'grub2-mkconfig -o /boot/grub2/grub.cfg' afterwards to update
# /boot/grub2/grub.cfg.
# Uncomment to set your own custom distributor. If you leave it unset or empty, the default
# policy is to determine the value from /etc/os-release
GRUB_DISTRIBUTOR=
GRUB_DEFAULT=saved
GRUB_HIDDEN_TIMEOUT=0
GRUB_HIDDEN_TIMEOUT_QUIET=true
GRUB_TIMEOUT=12
GRUB_CMDLINE_LINUX_DEFAULT="splash=silent
resume=/dev/disk/by-id/ata-Hitachi_HDT721032SLA380_STA207MC26MATB-part6 quiet intel_iommu=on
vfio-pci.ids=10de:2182,10de:1aeb rd.driver.blacklist=nouveau modprobe.blacklist=nouveau
mitigations=auto"
GRUB_CMDLINE_LINUX=""
and I don’t have this exact path: /etc/modules-load.d/vfio.conf but I do have the following:
Oh, and before I do that, here’s the VFIO module stuff I wanted in my first post:
I think this file was mistyped. I’m not sure what difference it makes, but there was an extra space after the "virqf " so the last quotation was one space after the f. I wonder if this makes a difference. Testing now.
Before I reboot, I ran and read
dracut -f
and this is the output–wish I paid attention to this sooner:
linux-lgui:/home/tim # dracut -f
dracut: Executing: /usr/bin/dracut -f
dracut: *** Including module: bash ***
dracut: *** Including module: systemd ***
dracut: libkmod: kmod_config_parse: /etc/modprobe.d/vfio.conf line 1: ignoring bad line starting with 'add_drivers="vfio_pci'
and it even tells me the exact file! What’s the correct way to write this?
I probably JUST posted this file, but just in case: here it is again:
Write what? What effect these two lines were supposed to have? Each of them individually? And no, “I want to pass through GPU” is too high level answer.
You apparently miss the fact that vfio driver is never auto-loaded, so having it as part of kernel is irrelevant (but of course mandatory prerequisite to be able to use it). You were hinted multiple times at /etc/modules-load.d which is there to manually load drivers during boot unconditionally.
That text file was blank and I checked the path and I don’t remember having that directory. For the vfio.conf I’m trying different commands but the first try was from the arch wiki:
etc/modprobe.d/vfio.conf
options vfio-pci ids=10de:13c2,10de:0fbb
“This, however, does not guarantee that vfio-pci will be loaded before other graphics drivers. To ensure that, we need to statically bind it in the kernel image alongside with its dependencies. That means adding, in this order, vfio_pci, vfio, vfio_iommu_type1, and vfio_virqfd to mkinitcpio:”
In case you didn’t fully understand the significance of what I posted
When you follow a guide, typically in the beginning there will be a statement and list of prerequisites, and if you can’t satisfy that section it’s not worth proceeding further.
Hi
Small steps… small steps As indicated I think you are getting to far ahead without checking before moving on…
Can you rollback all your changes, just so you boot with the intel_iommu=on and check it’s working, eg;
For me;
cat /etc/default/grub | grep GRUB_CMDLINE_LINUX_DEFAULT
GRUB_CMDLINE_LINUX_DEFAULT="splash=silent scsi_mod.use_blk_mq=1 intel_iommu=on quiet"
dmesg | grep iommu
0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.1.7-1-default root=UUID=01c3dc8a-0bae-468a-8acf-08bf697ce6df splash=silent scsi_mod.use_blk_mq=1 intel_iommu=on quiet
0.154137] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.1.7-1-default root=UUID=01c3dc8a-0bae-468a-8acf-08bf697ce6df splash=silent scsi_mod.use_blk_mq=1 intel_iommu=on quiet
1.401110] pci 0000:00:00.0: Adding to iommu group 0
1.401123] pci 0000:00:01.0: Adding to iommu group 1
1.401132] pci 0000:00:14.0: Adding to iommu group 2
1.401144] pci 0000:00:16.0: Adding to iommu group 3
1.401151] pci 0000:00:16.3: Adding to iommu group 3
1.401158] pci 0000:00:19.0: Adding to iommu group 4
1.401165] pci 0000:00:1a.0: Adding to iommu group 5
1.401174] pci 0000:00:1b.0: Adding to iommu group 6
1.401182] pci 0000:00:1c.0: Adding to iommu group 7
1.401190] pci 0000:00:1c.6: Adding to iommu group 8
1.401197] pci 0000:00:1d.0: Adding to iommu group 9
1.401207] pci 0000:00:1e.0: Adding to iommu group 10
1.401221] pci 0000:00:1f.0: Adding to iommu group 11
1.401228] pci 0000:00:1f.2: Adding to iommu group 11
1.401235] pci 0000:00:1f.3: Adding to iommu group 11
1.401239] pci 0000:01:00.0: Adding to iommu group 1 <== this is my Nividia GT8800 (not capable of pass through)
1.401248] pci 0000:03:00.0: Adding to iommu group 12
The next step would be to check your Nvidia card capabilities via lspci -vv output to check for “Access Control Services”.
So do you get this far? If so I can swap out the GPU in my system for another Nvidia card and use the IGD on the mother board to see if get further…
Now, when reading articles for other distributions, they need to be may need to be adapted for openSUSE. Don’t blacklist anything at the moment either, as per the forum link, just add your vfio modules the same in a file. Or you can load manually one by one first to test.
No I haven’t made it even this far. It’s a real shame–Linux makes this difficult by having different directories for different versions of the same software of different distributions. So here’s what I’ve got for you:
GNU nano 2.9.6 /etc/mkinitcpio.conf
MODULES=(vfio_pci vfio vfio_iommu_type1 vfio_virqfd)
HOOKS=(modconf)
Your command above is VERY helpful–nano formatting is weird I guess, and so it wouldn’t stay on one line, and I had to backspace and add a space in a certain manner to get that to change to this:
****, thank goodness for Klipper, or I’d have lost all the text:
No I haven’t made it even this far. It’s a real shame–Linux makes this difficult by having different directories for different versions of the same software of different distributions. So here’s what I’ve got for you:
GNU nano 2.9.6 /etc/mkinitcpio.conf
MODULES=(vfio_pci vfio vfio_iommu_type1 vfio_virqfd)
HOOKS=(modconf)
Your command above is VERY helpful–nano formatting is weird I guess, and so it wouldn’t stay on one line, and I had to backspace and add a space in a certain manner to get that to change to this:
You’re command of checking the default grub line was extremely helpful–I wouldn’t have been able to see the issue. So now, the following command works!
tim@linux-lgui:~> dmesg | grep iommu
0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.12.14-lp151.28.4-default root=UUID=0347972f-4cd8-4ae3-93d8-3072095afb3d splash=silent resume=/dev/disk/by-id/ata-Hitachi_HDT721032SLA380_STA207MC26MATB-part6 quiet intel_iommu=on vfio-pci.ids=10de:2182,10de:1aeb mitigations=auto
0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-4.12.14-lp151.28.4-default root=UUID=0347972f-4cd8-4ae3-93d8-3072095afb3d splash=silent resume=/dev/disk/by-id/ata-Hitachi_HDT721032SLA380_STA207MC26MATB-part6 quiet intel_iommu=on vfio-pci.ids=10de:2182,10de:1aeb mitigations=auto
2.042750] iommu: Adding device 0000:00:00.0 to group 0
2.042759] iommu: Adding device 0000:00:01.0 to group 1
2.042765] iommu: Adding device 0000:00:14.0 to group 2
2.042772] iommu: Adding device 0000:00:16.0 to group 3
2.042777] iommu: Adding device 0000:00:19.0 to group 4
2.042782] iommu: Adding device 0000:00:1a.0 to group 5
2.042787] iommu: Adding device 0000:00:1b.0 to group 6
2.042793] iommu: Adding device 0000:00:1c.0 to group 7
2.042798] iommu: Adding device 0000:00:1c.3 to group 8
2.042804] iommu: Adding device 0000:00:1c.4 to group 9
2.042809] iommu: Adding device 0000:00:1d.0 to group 10
2.042820] iommu: Adding device 0000:00:1f.0 to group 11
2.042825] iommu: Adding device 0000:00:1f.2 to group 11
2.042829] iommu: Adding device 0000:00:1f.3 to group 11
2.042833] iommu: Adding device 0000:01:00.0 to group 1
2.042836] iommu: Adding device 0000:01:00.1 to group 1
2.042841] iommu: Adding device 0000:03:00.0 to group 12
** 2.042866] iommu: Adding device 0000:05:00.0 to group 13
2.042882] iommu: Adding device 0000:05:00.1 to group 13
2.042897] iommu: Adding device 0000:05:00.2 to group 13
2.042913] iommu: Adding device 0000:05:00.3 to group 13**
tim@linux-lgui:~>
tim@linux-lgui:~> sudo mkinitrd
[sudo] password for root:
Creating initrd: /boot/initrd-4.12.14-lp151.28.4-default
dracut: Executing: /usr/bin/dracut --logfile /var/log/YaST2/mkinitrd.log --force /boot/initrd-4.12.14-lp151.28.4-default 4.12.14-lp151.28.4-default
dracut: *** Including module: bash ***
dracut: *** Including module: systemd ***
dracut: *** Including module: warpclock ***
dracut: *** Including module: systemd-initrd ***
dracut: *** Including module: i18n ***
dracut: *** Including module: drm ***
dracut: Possible missing firmware “amdgpu/vega20_asd.bin” for kernel module “amdgpu.ko”
dracut: Possible missing firmware “amdgpu/vega20_sos.bin” for kernel module “amdgpu.ko”
dracut: Possible missing firmware “amdgpu/vega20_rlc.bin” for kernel module “amdgpu.ko”
dracut: Possible missing firmware “amdgpu/vega20_mec2.bin” for kernel module “amdgpu.ko”
dracut: Possible missing firmware “amdgpu/vega20_mec.bin” for kernel module “amdgpu.ko”
dracut: Possible missing firmware “amdgpu/vega20_me.bin” for kernel module “amdgpu.ko”
dracut: Possible missing firmware “amdgpu/vega20_pfp.bin” for kernel module “amdgpu.ko”
dracut: Possible missing firmware “amdgpu/vega20_ce.bin” for kernel module “amdgpu.ko”
dracut: Possible missing firmware “amdgpu/vega20_sdma1.bin” for kernel module “amdgpu.ko”
dracut: Possible missing firmware “amdgpu/vega20_sdma.bin” for kernel module “amdgpu.ko”
dracut: Possible missing firmware “amdgpu/vega20_uvd.bin” for kernel module “amdgpu.ko”
dracut: Possible missing firmware “amdgpu/vega20_vce.bin” for kernel module “amdgpu.ko”
dracut: Possible missing firmware “amdgpu[size=1]/vega20_smc.bin” for kernel module “amdgpu.ko”[/size]
dracut: *** Including module: plymouth ***
dracut: *** Including module: btrfs ***
dracut: *** Including module: kernel-modules ***
dracut: *** Including module: resume ***
dracut: *** Including module: rootfs-block ***
dracut: *** Including module: suse-btrfs ***
dracut: *** Including module: suse-xfs ***
dracut: *** Including module: terminfo ***
dracut: *** Including module: udev-rules ***
dracut: Skipping udev rule: 40-redhat.rules
dracut: Skipping udev rule: 50-firmware.rules
dracut: Skipping udev rule: 50-udev.rules
dracut: Skipping udev rule: 91-permissions.rules
dracut: Skipping udev rule: 80-drivers-modprobe.rules
dracut: *** Including module: dracut-systemd ***
dracut: *** Including module: haveged ***
dracut: *** Including module: ostree ***
dracut: *** Including module: usrmount ***
dracut: *** Including module: base ***
dracut: *** Including module: fs-lib ***
dracut: *** Including module: shutdown ***
dracut: *** Including module: suse ***
dracut: *** Including modules done ***
dracut: *** Installing kernel module dependencies and firmware ***
dracut: *** Installing kernel module dependencies and firmware done ***
dracut: *** Resolving executable dependencies ***
dracut: *** Resolving executable dependencies done***
dracut: *** Hardlinking files ***
dracut: *** Hardlinking files done ***
dracut: *** Stripping files ***
dracut: *** Stripping files done ***
dracut: *** Generating early-microcode cpio image ***
dracut: *** Constructing GenuineIntel.bin ****
dracut: *** Store current command line parameters ***
dracut: Stored kernel commandline:
dracut: resume=UUID=7d8685e3-4c8e-44c4-98f3-1af5c3a99a7a
dracut: root=UUID=0347972f-4cd8-4ae3-93d8-3072095afb3d rootfstype=btrfs rootflags=rw,relatime,space_cache,subvolid=267,subvol=/@/.snapshots/1/snapshot,subvol=@/.snapshots/1/snapshot
dracut: *** Creating image file ‘/boot/initrd-4.12.14-lp151.28.4-default’ ***
dracut: *** Creating initramfs image file ‘/boot/initrd-4.12.14-lp151.28.4-default’ done ***
tim@linux-lgui:~>
This doesn't include the VFIO module, and it's not listed as a kernel module in the
My 960 also doesn't show "Access Control Services" so i'm not sure what to make of it, but I have read posts about the card being able to use VFIO and the like for gpu passthrough so maybe that's not necessary (I hope not), this being Nvidia, and the Error 43 issue, I don't feel like the hardware will advertise this feature.
Hi
Can you remove this part of the grub command line;
vfio-pci.ids=10de:2182,10de:1aeb
Did you remove the /etc/mkinitcpio.conf file (it’s not needed)
Yes the ACS is related to the iommu, which by the looks of your output should be fine, can you run (after you fix grub and reboot) as your user;
for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU Group %s ' "$n"; /sbin/lspci -nns "${d##*/}"; done;
As long as both cards are in separate iommu groups you should be fine.
Once you have the above information, can sort out the vfio stuff…
IOMMU Group 13 05:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:2182] (rev a1)
IOMMU Group 13 05:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:1aeb] (rev a1)
IOMMU Group 13 05:00.2 USB controller [0c03]: NVIDIA Corporation Device [10de:1aec] (rev a1)
IOMMU Group 13 05:00.3 Serial bus controller [0c80]: NVIDIA Corporation Device [10de:1aed] (rev a1)
IOMMU Group 1 00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller [8086:0c01] (rev 06)
IOMMU Group 1 01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Baffin [Radeon RX 560] [1002:67ff] (rev cf)
IOMMU Group 1 01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:aae0]
Yep, they’re in separate iommu groups! I’ve deleted the mkinitcpio file (and a misspelled version of it, and deleted the vfio bit from grub. I used YaST to remove that section so I’m assuming it’s already run a grub update, and I also ran
dracut -f
as well and will reboot. So what do you think it required next to get the VFIO drivers to work? Thank you for all of your assistance so far, I’m glad we’re working on this in the order it needs to be. You’re correct I jumped ahead and started worrying about UEFI in virt-manager before I got the grub line correct.
Hi
Just a follow on a quick test here with an Nvidia GT8800 (Not sure it will work further from a UEFI perspective in KVM, but can try), I just use the GPU cores here, no nvidia GL libs, no monitor attached, just use the intel GPU…
That did it. Now I have horrendously slow storage performance so I haven’t gotten to a windows desktop again yet. Thank you for every command and patience. I couldn’t have found the exact commands on my own in a reasonable amount of time.
As indicated I think you are getting to far ahead without checking before moving on…