Page 1 of 3 123 LastLast
Results 1 to 10 of 21

Thread: Nvidia GPU not using VFIO and it's not listed as a kernel module

  1. #1

    Question Nvidia GPU not using VFIO and it's not listed as a kernel module

    05:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:2182] (rev a1)
    Subsystem: eVga.com. Corp. Device [3842:1261]
    Kernel modules: nouveau, nvidia_drm, nvidia
    05:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:1aeb] (rev a1)
    Subsystem: eVga.com. Corp. Device [3842:1261]
    Kernel driver in use: snd_hda_intel
    Kernel modules: snd_hda_intel
    05:00.2 USB controller [0c03]: NVIDIA Corporation Device [10de:1aec] (rev a1)
    Subsystem: eVga.com. Corp. Device [3842:1261]
    Kernel driver in use: xhci_hcd
    Kernel modules: xhci_pci
    05:00.3 Serial bus controller [0c80]: NVIDIA Corporation Device [10de:1aed] (rev a1)
    Subsystem: eVga.com. Corp. Device [3842:1261]
    tim@linux-lgui:~>

    As you can see, it doesn't have the line "kernel driver in use". That's because I deleted both the nvidia drivers and the opensource nouveau drivers to stop them from loading.

    Still I cannot get VFIO to load on the card or however that's worded, and it's not listed in the kernel modules but I do think it's in my kernel modules as seen by this output:

    Code:
    tim@linux-lgui:~> sudo modinfo vfio-pci
    filename:       /lib/modules/4.12.14-lp151.28.4-default/kernel/drivers/vfio/pci/vfio-pci.ko
    description:    VFIO PCI - User Level meta-driver
    author:         Alex Williamson <alex.williamson@redhat.com>
    license:        GPL v2
    version:        0.2
    suserelease:    openSUSE Leap 15.1
    srcversion:     43445CE6A64818F93EEACB2
    depends:        vfio,irqbypass,vfio_virqfd
    retpoline:      Y
    intree:         Y
    vermagic:       4.12.14-lp151.28.4-default SMP mod_unload modversions 
    parm:           ids:Initial PCI IDs to add to the vfio driver, format is "vendor:device[:subvendor[:subdevice[:class[:class_mask]]]]" and multiple comma separated entries can be specified (string)
    parm:           nointxmask:Disable support for PCI 2.3 style INTx masking.  If this resolves problems for specific devices, report lspci -vvvxxx to linux-pci@vger.kernel.org so the device can be fixed automatically via the broken_intx_masking flag. (bool)
    parm:           disable_vga:Disable VGA resource access through vfio-pci (bool)
    parm:           disable_idle_d3:Disable using the PCI D3 low power state for idle, unused devices (bool)
    tim@linux-lgui:~>
    as well as:

    Well I can't find the command again but I have VFIO.conf if you all want to view it.

    What do I need to do for VFIO to take over my nvidia card?

  2. #2
    Join Date
    Jun 2008
    Location
    Podunk
    Posts
    27,108
    Blog Entries
    15

    Default Re: Nvidia GPU not using VFIO and it's not listed as a kernel module

    Hi
    You don't have dual gpu's?

    There is some info in this tread;
    https://forums.opensuse.org/showthre...th-libvirt-KVM

    But I have dual gpu's, AFAIK you need that, two keyboard mice etc when things go wrong.....
    Cheers Malcolm °¿° SUSE Knowledge Partner (Linux Counter #276890)
    SUSE SLE, openSUSE Leap/Tumbleweed (x86_64) | GNOME DE
    If you find this post helpful and are logged into the web interface,
    please show your appreciation and click on the star below... Thanks!

  3. #3

    Default Re: Nvidia GPU not using VFIO and it's not listed as a kernel module

    I have two GPUs, AMD 560 and Nvidia 960. I'm using the AMD card on the host, and without the VFIO working for the 960, the following error happens when I try to add the hardware and "begin installation" in virt-manager:

    Code:
    Unable to complete install: 'unsupported configuration: host doesn't support passthrough of host PCI devices'
    
    Traceback (most recent call last):
      File "/usr/share/virt-manager/virtManager/asyncjob.py", line 75, in cb_wrapper
        callback(asyncjob, *args, **kwargs)
      File "/usr/share/virt-manager/virtManager/create.py", line 2205, in _do_async_install
        guest.installer_instance.start_install(guest, meter=meter)
      File "/usr/share/virt-manager/virtinst/installer.py", line 444, in start_install
        doboot, transient)
      File "/usr/share/virt-manager/virtinst/installer.py", line 387, in _create_guest
        domain = self.conn.createXML(install_xml or final_xml, 0)
      File "/usr/lib64/python3.6/site-packages/libvirt.py", line 3743, in createXML
        if ret is None:raise libvirtError('virDomainCreateXML() failed', conn=self)
    libvirt.libvirtError: unsupported configuration: host doesn't support passthrough of host PCI devices
    Somehow I need to find out how to enable VFIO for the Nvidia card, but I'm completely clueless to how all the configuration files work, and what to modify. For the work that I have done, I've posted on the Level1Techs forum, but it seems I need specific bits for OpenSUSE to get this to work.

    https://forum.level1techs.com/t/gpu-...iver/143736/13

    So I haven't had any progress on this, for the last few days of configuring files.

  4. #4
    Join Date
    Jun 2008
    Location
    Podunk
    Posts
    27,108
    Blog Entries
    15

    Default Re: Nvidia GPU not using VFIO and it's not listed as a kernel module

    Quote Originally Posted by timpster View Post
    I have two GPUs, AMD 560 and Nvidia 960. I'm using the AMD card on the host, and without the VFIO working for the 960, the following error happens when I try to add the hardware and "begin installation" in virt-manager:

    Code:
    Unable to complete install: 'unsupported configuration: host doesn't support passthrough of host PCI devices'
    
    Traceback (most recent call last):
      File "/usr/share/virt-manager/virtManager/asyncjob.py", line 75, in cb_wrapper
        callback(asyncjob, *args, **kwargs)
      File "/usr/share/virt-manager/virtManager/create.py", line 2205, in _do_async_install
        guest.installer_instance.start_install(guest, meter=meter)
      File "/usr/share/virt-manager/virtinst/installer.py", line 444, in start_install
        doboot, transient)
      File "/usr/share/virt-manager/virtinst/installer.py", line 387, in _create_guest
        domain = self.conn.createXML(install_xml or final_xml, 0)
      File "/usr/lib64/python3.6/site-packages/libvirt.py", line 3743, in createXML
        if ret is None:raise libvirtError('virDomainCreateXML() failed', conn=self)
    libvirt.libvirtError: unsupported configuration: host doesn't support passthrough of host PCI devices
    Somehow I need to find out how to enable VFIO for the Nvidia card, but I'm completely clueless to how all the configuration files work, and what to modify. For the work that I have done, I've posted on the Level1Techs forum, but it seems I need specific bits for OpenSUSE to get this to work.

    https://forum.level1techs.com/t/gpu-...iver/143736/13

    So I haven't had any progress on this, for the last few days of configuring files.
    Hi
    So did you configure grub and a /etc/modules-load.d/vfio.conf file?
    Cheers Malcolm °¿° SUSE Knowledge Partner (Linux Counter #276890)
    SUSE SLE, openSUSE Leap/Tumbleweed (x86_64) | GNOME DE
    If you find this post helpful and are logged into the web interface,
    please show your appreciation and click on the star below... Thanks!

  5. #5
    Join Date
    Jun 2008
    Location
    San Diego, Ca, USA
    Posts
    11,393
    Blog Entries
    2

    Default Re: Nvidia GPU not using VFIO and it's not listed as a kernel module

    Awhile ago I posted the following which even today should be "the" reference for guides to configure GPU pass-through.
    It' continues to be valid over time because it points to "live" documentation that is continuously updated.
    Identify which is most appropriate for what you're trying to do, and if you run into any obstacles, then post including the guide you're following and provide errors and output for whatever you're seeing

    https://forums.opensuse.org/showthre...hlight=gpu+pas

    Applies to all virtualization technologies.

    TSU
    Beginner Wiki Quickstart - https://en.opensuse.org/User:Tsu2/Quickstart_Wiki
    Solved a problem recently? Create a wiki page for future personal reference!
    Learn something new?
    Attended a computing event?
    Post and Share!

  6. #6
    Join Date
    Jun 2008
    Location
    San Diego, Ca, USA
    Posts
    11,393
    Blog Entries
    2

    Default Re: Nvidia GPU not using VFIO and it's not listed as a kernel module

    Since I had a few brief moments, I took a look at what you posted.

    You never identified your GPU by name, I'm assuming that the device id you posted means your GPU is a GeForce GTX 1660
    Looking up that GPU on IOMMU support lists, I don't see it anywhere on any list. That might be a show stopper but I also located one (The only one!)Forum discussion that describes what one guy did that supposedly configured IOMMU... I didn't inspect it closely, but it may contain useful info

    https://forums.unraid.net/topic/7961...reen-gtx-1660/

    If that doesn't help, then I'd think that you're going to have to ask nVidia directly about IOMMU support for your GPU and now to configure if possible.
    Only after you get past that, then you can proceed following the ArchWiki OVMF guide.

    TSU
    Beginner Wiki Quickstart - https://en.opensuse.org/User:Tsu2/Quickstart_Wiki
    Solved a problem recently? Create a wiki page for future personal reference!
    Learn something new?
    Attended a computing event?
    Post and Share!

  7. #7

    Default Re: Nvidia GPU not using VFIO and it's not listed as a kernel module

    Quote Originally Posted by tsu2 View Post
    Since I had a few brief moments, I took a look at what you posted.

    You never identified your GPU by name, I'm assuming that the device id you posted means your GPU is a GeForce GTX 1660
    Looking up that GPU on IOMMU support lists, I don't see it anywhere on any list. TSU
    I actually acquired a GTX 1660 Ti but I haven't done much with it. I may sell it as I received it as a promotional item of sorts. I have a GTX 960 and an AMD 560, for the host, and Nvidia is for the VM.

    To malcolmlewis: Here is my grub and other config files:

    Code:
      GNU nano 2.9.6                                                   /etc/default/grub                                                             
    
    # If you change this file, run 'grub2-mkconfig -o /boot/grub2/grub.cfg' afterwards to update
    # /boot/grub2/grub.cfg.
    
    # Uncomment to set your own custom distributor. If you leave it unset or empty, the default
    # policy is to determine the value from /etc/os-release
    GRUB_DISTRIBUTOR=
    GRUB_DEFAULT=saved
    GRUB_HIDDEN_TIMEOUT=0
    GRUB_HIDDEN_TIMEOUT_QUIET=true
    GRUB_TIMEOUT=12
    GRUB_CMDLINE_LINUX_DEFAULT="splash=silent
    resume=/dev/disk/by-id/ata-Hitachi_HDT721032SLA380_STA207MC26MATB-part6 quiet intel_iommu=on
    vfio-pci.ids=10de:2182,10de:1aeb rd.driver.blacklist=nouveau modprobe.blacklist=nouveau
    mitigations=auto"
    GRUB_CMDLINE_LINUX=""
    and I don't have this exact path: /etc/modules-load.d/vfio.conf but I do have the following:

    Oh, and before I do that, here's the VFIO module stuff I wanted in my first post:

    Code:
     linux-lgui:/etc # find /lib/modules/4.12.14-lp151.28.4-default/ | grep vfio
    /lib/modules/4.12.14-lp151.28.4-default/kernel/drivers/vfio
    /lib/modules/4.12.14-lp151.28.4-default/kernel/drivers/vfio/mdev
    /lib/modules/4.12.14-lp151.28.4-default/kernel/drivers/vfio/mdev/mdev.ko
    /lib/modules/4.12.14-lp151.28.4-default/kernel/drivers/vfio/mdev/vfio_mdev.ko
    /lib/modules/4.12.14-lp151.28.4-default/kernel/drivers/vfio/pci
    /lib/modules/4.12.14-lp151.28.4-default/kernel/drivers/vfio/pci/vfio-pci.ko
    /lib/modules/4.12.14-lp151.28.4-default/kernel/drivers/vfio/vfio.ko
    /lib/modules/4.12.14-lp151.28.4-default/kernel/drivers/vfio/vfio_iommu_type1.ko
    /lib/modules/4.12.14-lp151.28.4-default/kernel/drivers/vfio/vfio_virqfd.ko
    linux-lgui:/etc #
    And here's my VFIO.conf file:
    Code:
      GNU nano 2.9.6          /etc/modprobe.d/vfio.conf                                        
    
    add_drivers="vfio_pci vfio vfio_iommu_type1 vfio_virqf"
    options vfio-pci ids=10de:2182,10de:1aeb
    I think this file was mistyped. I'm not sure what difference it makes, but there was an extra space after the "virqf " so the last quotation was one space after the f. I wonder if this makes a difference. Testing now.

    Before I reboot, I ran and read
    Code:
    dracut -f
    and this is the output--wish I paid attention to this sooner:

    Code:
    linux-lgui:/home/tim # dracut -f
    dracut: Executing: /usr/bin/dracut -f
    dracut: *** Including module: bash ***
    dracut: *** Including module: systemd ***
    dracut: libkmod: kmod_config_parse: /etc/modprobe.d/vfio.conf line 1: ignoring bad line starting with 'add_drivers="vfio_pci'
    and it even tells me the exact file! What's the correct way to write this?

    I probably JUST posted this file, but just in case: here it is again:

    Code:
    add_drivers="vfio_pci vfio vfio_iommu_type1 vfio_virqf"
    options vfio-pci ids=10de:2182,10de:1aeb

  8. #8
    Join Date
    Sep 2012
    Posts
    5,202

    Default Re: Nvidia GPU not using VFIO and it's not listed as a kernel module

    Quote Originally Posted by timpster View Post
    What's the correct way to write this?
    Write what? What effect these two lines were supposed to have? Each of them individually? And no, "I want to pass through GPU" is too high level answer.

    You apparently miss the fact that vfio driver is never auto-loaded, so having it as part of kernel is irrelevant (but of course mandatory prerequisite to be able to use it). You were hinted multiple times at /etc/modules-load.d which is there to manually load drivers during boot unconditionally.

  9. #9

    Default Re: Nvidia GPU not using VFIO and it's not listed as a kernel module

    Quote Originally Posted by arvidjaar View Post
    Write what? What effect these two lines were supposed to have? Each of them individually? And no, "I want to pass through GPU" is too high level answer.

    You apparently miss the fact that vfio driver is never auto-loaded, so having it as part of kernel is irrelevant (but of course mandatory prerequisite to be able to use it). You were hinted multiple times at /etc/modules-load.d which is there to manually load drivers during boot unconditionally.
    That text file was blank and I checked the path and I don't remember having that directory. For the vfio.conf I'm trying different commands but the first try was from the arch wiki:

    etc/modprobe.d/vfio.conf

    options vfio-pci ids=10de:13c2,10de:0fbb

    "This, however, does not guarantee that vfio-pci will be loaded before other graphics drivers. To ensure that, we need to statically bind it in the kernel image alongside with its dependencies. That means adding, in this order, vfio_pci, vfio, vfio_iommu_type1, and vfio_virqfd to mkinitcpio:"

    /etc/mkinitcpio.conf

    MODULES=(... vfio_pci vfio vfio_iommu_type1 vfio_virqfd ...)

    The above text and code was gathered from: https://wiki.archlinux.org/index.php...rough_via_OVMF

    When I do
    Code:
    dmesg | grep -i vfio
    there is no response and it just lets me enter a new command.

  10. #10
    Join Date
    Jun 2008
    Location
    San Diego, Ca, USA
    Posts
    11,393
    Blog Entries
    2

    Default Re: Nvidia GPU not using VFIO and it's not listed as a kernel module

    Quote Originally Posted by timpster View Post
    I actually acquired a GTX 1660 Ti but I haven't done much with it. I may sell it as I received it as a promotional item of sorts. I have a GTX 960 and an AMD 560, for the host, and Nvidia is for the VM.
    In case you didn't fully understand the significance of what I posted
    When you follow a guide, typically in the beginning there will be a statement and list of prerequisites, and if you can't satisfy that section it's not worth proceeding further.

    Tead the docs.
    Entirely.
    From the beginning.

    TSU
    Beginner Wiki Quickstart - https://en.opensuse.org/User:Tsu2/Quickstart_Wiki
    Solved a problem recently? Create a wiki page for future personal reference!
    Learn something new?
    Attended a computing event?
    Post and Share!

Page 1 of 3 123 LastLast

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •