Page 2 of 3 FirstFirst 123 LastLast
Results 11 to 20 of 21

Thread: Nvidia GPU not using VFIO and it's not listed as a kernel module

  1. #11
    Join Date
    Jun 2008
    Location
    Podunk
    Posts
    26,513
    Blog Entries
    15

    Default Re: Nvidia GPU not using VFIO and it's not listed as a kernel module

    Quote Originally Posted by timpster View Post
    That text file was blank and I checked the path and I don't remember having that directory. For the vfio.conf I'm trying different commands but the first try was from the arch wiki:

    etc/modprobe.d/vfio.conf

    options vfio-pci ids=10de:13c2,10de:0fbb

    "This, however, does not guarantee that vfio-pci will be loaded before other graphics drivers. To ensure that, we need to statically bind it in the kernel image alongside with its dependencies. That means adding, in this order, vfio_pci, vfio, vfio_iommu_type1, and vfio_virqfd to mkinitcpio:"

    /etc/mkinitcpio.conf

    MODULES=(... vfio_pci vfio vfio_iommu_type1 vfio_virqfd ...)

    The above text and code was gathered from: https://wiki.archlinux.org/index.php...rough_via_OVMF

    When I do
    Code:
    dmesg | grep -i vfio
    there is no response and it just lets me enter a new command.
    Hi
    Small steps... small steps As indicated I think you are getting to far ahead without checking before moving on...

    Can you rollback all your changes, just so you boot with the intel_iommu=on and check it's working, eg;

    For me;
    Code:
    cat /etc/default/grub | grep GRUB_CMDLINE_LINUX_DEFAULT
    
    GRUB_CMDLINE_LINUX_DEFAULT="splash=silent scsi_mod.use_blk_mq=1 intel_iommu=on quiet"
    
    dmesg | grep iommu
    
    [    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.1.7-1-default root=UUID=01c3dc8a-0bae-468a-8acf-08bf697ce6df splash=silent scsi_mod.use_blk_mq=1 intel_iommu=on quiet
    [    0.154137] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.1.7-1-default root=UUID=01c3dc8a-0bae-468a-8acf-08bf697ce6df splash=silent scsi_mod.use_blk_mq=1 intel_iommu=on quiet
    [    1.401110] pci 0000:00:00.0: Adding to iommu group 0
    [    1.401123] pci 0000:00:01.0: Adding to iommu group 1
    [    1.401132] pci 0000:00:14.0: Adding to iommu group 2
    [    1.401144] pci 0000:00:16.0: Adding to iommu group 3
    [    1.401151] pci 0000:00:16.3: Adding to iommu group 3
    [    1.401158] pci 0000:00:19.0: Adding to iommu group 4
    [    1.401165] pci 0000:00:1a.0: Adding to iommu group 5
    [    1.401174] pci 0000:00:1b.0: Adding to iommu group 6
    [    1.401182] pci 0000:00:1c.0: Adding to iommu group 7
    [    1.401190] pci 0000:00:1c.6: Adding to iommu group 8
    [    1.401197] pci 0000:00:1d.0: Adding to iommu group 9
    [    1.401207] pci 0000:00:1e.0: Adding to iommu group 10
    [    1.401221] pci 0000:00:1f.0: Adding to iommu group 11
    [    1.401228] pci 0000:00:1f.2: Adding to iommu group 11
    [    1.401235] pci 0000:00:1f.3: Adding to iommu group 11
    [    1.401239] pci 0000:01:00.0: Adding to iommu group 1 <== this is my Nividia GT8800 (not capable of pass through)
    [    1.401248] pci 0000:03:00.0: Adding to iommu group 12
    The next step would be to check your Nvidia card capabilities via lspci -vv output to check for "Access Control Services".

    So do you get this far? If so I can swap out the GPU in my system for another Nvidia card and use the IGD on the mother board to see if get further...

    Now, when reading articles for other distributions, they need to be may need to be adapted for openSUSE. Don't blacklist anything at the moment either, as per the forum link, just add your vfio modules the same in a file. Or you can load manually one by one first to test.
    Cheers Malcolm °¿° SUSE Knowledge Partner (Linux Counter #276890)
    SUSE SLE, openSUSE Leap/Tumbleweed (x86_64) | GNOME DE
    If you find this post helpful and are logged into the web interface,
    please show your appreciation and click on the star below... Thanks!

  2. #12

    Default Re: Nvidia GPU not using VFIO and it's not listed as a kernel module

    Quote Originally Posted by malcolmlewis View Post
    Hi
    Small steps... small steps As indicated I think you are getting to far ahead without checking before moving on...

    Can you rollback all your changes, just so you boot with the intel_iommu=on and check it's working, eg;

    For me;
    [CODE]
    cat /etc/default/grub | grep GRUB_CMDLINE_LINUX_DEFAULT

    GRUB_CMDLINE_LINUX_DEFAULT="splash=silent scsi_mod.use_blk_mq=1 intel_iommu=on quiet"

    dmesg | grep iommu

    The next step would be to check your Nvidia card capabilities via lspci -vv output to check for "Access Control Services".

    So do you get this far? If so I can swap out the GPU in my system for another Nvidia card and use the IGD on the mother board to see if get further...

    Now, when reading articles for other distributions, they need to be may need to be adapted for openSUSE. Don't blacklist anything at the moment either, as per the forum link, just add your vfio modules the same in a file. Or you can load manually one by one first to test.
    Do you think snapper will be ed both to reset my config?

  3. #13
    Join Date
    Jun 2008
    Location
    Podunk
    Posts
    26,513
    Blog Entries
    15

    Default Re: Nvidia GPU not using VFIO and it's not listed as a kernel module

    Quote Originally Posted by timpster View Post
    Do you think snapper will be ed both to reset my config?
    Hi
    Just remove the blacklist bit in grub and that mkinitcpio.conf, rebuild initrd with the mkinitrd command and you should be fine.

    I just checked a GT640 and a GT710, neither show "Access Control Services"......
    Cheers Malcolm °¿° SUSE Knowledge Partner (Linux Counter #276890)
    SUSE SLE, openSUSE Leap/Tumbleweed (x86_64) | GNOME DE
    If you find this post helpful and are logged into the web interface,
    please show your appreciation and click on the star below... Thanks!

  4. #14

    Default Re: Nvidia GPU not using VFIO and it's not listed as a kernel module

    ****, thank goodness for Klipper, or I'd have lost all the text:

    Quote Originally Posted by malcolmlewis View Post
    Hi
    Small steps... small steps As indicated I think you are getting to far ahead without checking before moving on...

    Can you rollback all your changes, just so you boot with the intel_iommu=on and check it's working, eg;

    For me;
    Code:
    cat /etc/default/grub | grep GRUB_CMDLINE_LINUX_DEFAULT
    
    [    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.1.7-1-default root=UUID=01c3dc8a-0bae-468a-8acf-08bf697ce6df splash=silent scsi_mod.use_blk_mq=1 intel_iommu=on quiet
    [    0.154137] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.1.7-1-default root=UUID=01c3dc8a-0bae-468a-8acf-08bf697ce6df splash=silent scsi_mod.use_blk_mq=1 intel_iommu=on quiet
    
    [    1.401239] pci 0000:01:00.0: Adding to iommu group 1 <== this is my Nividia GT8800 (not capable of pass through)
    The next step would be to check your Nvidia card capabilities via lspci -vv output to check for "Access Control Services".

    So do you get this far? If so I can swap out the GPU in my system for another Nvidia card and use the IGD on the mother board to see if get further...

    Now, when reading articles for other distributions, they need to be may need to be adapted for openSUSE. Don't blacklist anything at the moment either, as per the forum link, just add your vfio modules the same in a file. Or you can load manually one by one first to test.
    No I haven't made it even this far. It's a real shame--Linux makes this difficult by having different directories for different versions of the same software of different distributions. So here's what I've got for you:

    Code:
      GNU nano 2.9.6           /etc/mkinitcpio.conf                      
    
    MODULES=(vfio_pci vfio vfio_iommu_type1 vfio_virqfd)
    HOOKS=(modconf)
    GRUB_CMDLINE_LINUX_DEFAULT="splash=silent scsi_mod.use_blk_mq=1 intel_iommu=on quiet"

    Your command above is VERY helpful--nano formatting is weird I guess, and so it wouldn't stay on one line, and I had to backspace and add a space in a certain manner to get that to change to this:

    Code:
    GRUB_CMDLINE_LINUX_DEFAULT="splash=silent resume=/dev/disk/by-id/ata-Hitachi_HDT721032SLA380_STA207MC26MATB-part6 quiet intel_iommu=on vfio-pci.ids=10de:2182,10de:1aeb mitigations=auto"
    tim@linux-lgui:~> reboot
    I'll test the rest and edit in a bit.

    You're command of checking the default grub line was extremely helpful--I wouldn't have been able to see the issue. So now, the following command works!

    [CODE]tim@linux-lgui:~> dmesg | grep iommu
    [ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.12.14-lp151.28.4-default root=UUID=0347972f-4cd8-4ae3-93d8-3072095afb3d splash=silent resume=/dev/disk/by-id/ata-Hitachi_HDT721032SLA380_STA207MC26MATB-part6 quiet intel_iommu=on vfio-pci.ids=10de:2182,10de:1aeb mitigations=auto
    [ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-4.12.14-lp151.28.4-default root=UUID=0347972f-4cd8-4ae3-93d8-3072095afb3d splash=silent resume=/dev/disk/by-id/ata-Hitachi_HDT721032SLA380_STA207MC26MATB-part6 quiet intel_iommu=on vfio-pci.ids=10de:2182,10de:1aeb mitigations=auto
    [ 2.042750] iommu: Adding device 0000:00:00.0 to group 0
    [ 2.042759] iommu: Adding device 0000:00:01.0 to group 1
    [ 2.042765] iommu: Adding device 0000:00:14.0 to group 2
    [ 2.042772] iommu: Adding device 0000:00:16.0 to group 3
    [ 2.042777] iommu: Adding device 0000:00:19.0 to group 4
    [ 2.042782] iommu: Adding device 0000:00:1a.0 to group 5
    [ 2.042787] iommu: Adding device 0000:00:1b.0 to group 6
    [ 2.042793] iommu: Adding device 0000:00:1c.0 to group 7
    [ 2.042798] iommu: Adding device 0000:00:1c.3 to group 8
    [ 2.042804] iommu: Adding device 0000:00:1c.4 to group 9
    [ 2.042809] iommu: Adding device 0000:00:1d.0 to group 10
    [ 2.042820] iommu: Adding device 0000:00:1f.0 to group 11
    [ 2.042825] iommu: Adding device 0000:00:1f.2 to group 11
    [ 2.042829] iommu: Adding device 0000:00:1f.3 to group 11
    [ 2.042833] iommu: Adding device 0000:01:00.0 to group 1
    [ 2.042836] iommu: Adding device 0000:01:00.1 to group 1
    [ 2.042841] iommu: Adding device 0000:03:00.0 to group 12
    [ 2.042866] iommu: Adding device 0000:05:00.0 to group 13
    [ 2.042882] iommu: Adding device 0000:05:00.1 to group 13
    [ 2.042897] iommu: Adding device 0000:05:00.2 to group 13
    [ 2.042913] iommu: Adding device 0000:05:00.3 to group 13

    tim@linux-lgui:~>

    Code:
     tim@linux-lgui:~> sudo mkinitrd
    [sudo] password for root:
    Creating initrd: /boot/initrd-4.12.14-lp151.28.4-default
    dracut: Executing: /usr/bin/dracut --logfile /var/log/YaST2/mkinitrd.log --force /boot/initrd-4.12.14-lp151.28.4-default 4.12.14-lp151.28.4-default
    dracut: *** Including module: bash ***
    dracut: *** Including module: systemd ***
    dracut: *** Including module: warpclock ***
    dracut: *** Including module: systemd-initrd ***
    dracut: *** Including module: i18n ***
    dracut: *** Including module: drm ***
    dracut: Possible missing firmware "amdgpu/vega20_asd.bin" for kernel module "amdgpu.ko"
    dracut: Possible missing firmware "amdgpu/vega20_sos.bin" for kernel module "amdgpu.ko"
    dracut: Possible missing firmware "amdgpu/vega20_rlc.bin" for kernel module "amdgpu.ko"
    dracut: Possible missing firmware "amdgpu/vega20_mec2.bin" for kernel module "amdgpu.ko"
    dracut: Possible missing firmware "amdgpu/vega20_mec.bin" for kernel module "amdgpu.ko"
    dracut: Possible missing firmware "amdgpu/vega20_me.bin" for kernel module "amdgpu.ko"
    dracut: Possible missing firmware "amdgpu/vega20_pfp.bin" for kernel module "amdgpu.ko"
    dracut: Possible missing firmware "amdgpu/vega20_ce.bin" for kernel module "amdgpu.ko"
    dracut: Possible missing firmware "amdgpu/vega20_sdma1.bin" for kernel module "amdgpu.ko"
    dracut: Possible missing firmware "amdgpu/vega20_sdma.bin" for kernel module "amdgpu.ko"
    dracut: Possible missing firmware "amdgpu/vega20_uvd.bin" for kernel module "amdgpu.ko"
    dracut: Possible missing firmware "amdgpu/vega20_vce.bin" for kernel module "amdgpu.ko"
    dracut: Possible missing firmware "amdgpu/vega20_smc.bin" for kernel module "amdgpu.ko"
    dracut: *** Including module: plymouth ***
    dracut: *** Including module: btrfs ***
    dracut: *** Including module: kernel-modules ***
    dracut: *** Including module: resume ***
    dracut: *** Including module: rootfs-block ***
    dracut: *** Including module: suse-btrfs ***
    dracut: *** Including module: suse-xfs ***
    dracut: *** Including module: terminfo ***
    dracut: *** Including module: udev-rules ***
    dracut: Skipping udev rule: 40-redhat.rules
    dracut: Skipping udev rule: 50-firmware.rules
    dracut: Skipping udev rule: 50-udev.rules
    dracut: Skipping udev rule: 91-permissions.rules
    dracut: Skipping udev rule: 80-drivers-modprobe.rules
    dracut: *** Including module: dracut-systemd ***
    dracut: *** Including module: haveged ***
    dracut: *** Including module: ostree ***
    dracut: *** Including module: usrmount ***
    dracut: *** Including module: base ***
    dracut: *** Including module: fs-lib ***
    dracut: *** Including module: shutdown ***
    dracut: *** Including module: suse ***
    dracut: *** Including modules done ***
    dracut: *** Installing kernel module dependencies and firmware ***
    dracut: *** Installing kernel module dependencies and firmware done ***
    dracut: *** Resolving executable dependencies ***
    dracut: *** Resolving executable dependencies done***
    dracut: *** Hardlinking files ***
    dracut: *** Hardlinking files done ***
    dracut: *** Stripping files ***
    dracut: *** Stripping files done ***
    dracut: *** Generating early-microcode cpio image ***
    dracut: *** Constructing GenuineIntel.bin ****
    dracut: *** Store current command line parameters ***
    dracut: Stored kernel commandline:
    dracut:  resume=UUID=7d8685e3-4c8e-44c4-98f3-1af5c3a99a7a
    dracut:  root=UUID=0347972f-4cd8-4ae3-93d8-3072095afb3d rootfstype=btrfs rootflags=rw,relatime,space_cache,subvolid=267,subvol=/@/.snapshots/1/snapshot,subvol=@/.snapshots/1/snapshot
    dracut: *** Creating image file '/boot/initrd-4.12.14-lp151.28.4-default' ***
    dracut: *** Creating initramfs image file '/boot/initrd-4.12.14-lp151.28.4-default' done ***
    tim@linux-lgui:~>
    This doesn't include the VFIO module, and it's not listed as a kernel module in the
    Code:
    lspci -nnk
    Code:
    05:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:2182] (rev a1)
            Subsystem: eVga.com. Corp. Device [3842:1261]
            Kernel modules: nouveau, nvidia_drm, nvidia
    05:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:1aeb] (rev a1)
            Subsystem: eVga.com. Corp. Device [3842:1261]
            Kernel driver in use: snd_hda_intel
            Kernel modules: snd_hda_intel
    05:00.2 USB controller [0c03]: NVIDIA Corporation Device [10de:1aec] (rev a1)
            Subsystem: eVga.com. Corp. Device [3842:1261]
            Kernel driver in use: xhci_hcd
            Kernel modules: xhci_pci
    05:00.3 Serial bus controller [0c80]: NVIDIA Corporation Device [10de:1aed] (rev a1)
            Subsystem: eVga.com. Corp. Device [3842:1261]
    tim@linux-lgui:~>
    My 960 also doesn't show "Access Control Services" so i'm not sure what to make of it, but I have read posts about the card being able to use VFIO and the like for gpu passthrough so maybe that's not necessary (I hope not), this being Nvidia, and the Error 43 issue, I don't feel like the hardware will advertise this feature.
    Last edited by timpster; 09-Jun-2019 at 18:30. Reason: added info

  5. #15
    Join Date
    Jun 2008
    Location
    Podunk
    Posts
    26,513
    Blog Entries
    15

    Default Re: Nvidia GPU not using VFIO and it's not listed as a kernel module

    Quote Originally Posted by timpster View Post
    ****, thank goodness for Klipper, or I'd have lost all the text:



    No I haven't made it even this far. It's a real shame--Linux makes this difficult by having different directories for different versions of the same software of different distributions. So here's what I've got for you:

    Code:
      GNU nano 2.9.6           /etc/mkinitcpio.conf                      
    
    MODULES=(vfio_pci vfio vfio_iommu_type1 vfio_virqfd)
    HOOKS=(modconf)
    GRUB_CMDLINE_LINUX_DEFAULT="splash=silent scsi_mod.use_blk_mq=1 intel_iommu=on quiet"

    Your command above is VERY helpful--nano formatting is weird I guess, and so it wouldn't stay on one line, and I had to backspace and add a space in a certain manner to get that to change to this:

    Code:
    GRUB_CMDLINE_LINUX_DEFAULT="splash=silent resume=/dev/disk/by-id/ata-Hitachi_HDT721032SLA380_STA207MC26MATB-part6 quiet intel_iommu=on vfio-pci.ids=10de:2182,10de:1aeb mitigations=auto"
    tim@linux-lgui:~> reboot
    I'll test the rest and edit in a bit.

    You're command of checking the default grub line was extremely helpful--I wouldn't have been able to see the issue. So now, the following command works!

    [CODE]tim@linux-lgui:~> dmesg | grep iommu
    [ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.12.14-lp151.28.4-default root=UUID=0347972f-4cd8-4ae3-93d8-3072095afb3d splash=silent resume=/dev/disk/by-id/ata-Hitachi_HDT721032SLA380_STA207MC26MATB-part6 quiet intel_iommu=on vfio-pci.ids=10de:2182,10de:1aeb mitigations=auto
    [ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-4.12.14-lp151.28.4-default root=UUID=0347972f-4cd8-4ae3-93d8-3072095afb3d splash=silent resume=/dev/disk/by-id/ata-Hitachi_HDT721032SLA380_STA207MC26MATB-part6 quiet intel_iommu=on vfio-pci.ids=10de:2182,10de:1aeb mitigations=auto
    [ 2.042750] iommu: Adding device 0000:00:00.0 to group 0
    [ 2.042759] iommu: Adding device 0000:00:01.0 to group 1
    [ 2.042765] iommu: Adding device 0000:00:14.0 to group 2
    [ 2.042772] iommu: Adding device 0000:00:16.0 to group 3
    [ 2.042777] iommu: Adding device 0000:00:19.0 to group 4
    [ 2.042782] iommu: Adding device 0000:00:1a.0 to group 5
    [ 2.042787] iommu: Adding device 0000:00:1b.0 to group 6
    [ 2.042793] iommu: Adding device 0000:00:1c.0 to group 7
    [ 2.042798] iommu: Adding device 0000:00:1c.3 to group 8
    [ 2.042804] iommu: Adding device 0000:00:1c.4 to group 9
    [ 2.042809] iommu: Adding device 0000:00:1d.0 to group 10
    [ 2.042820] iommu: Adding device 0000:00:1f.0 to group 11
    [ 2.042825] iommu: Adding device 0000:00:1f.2 to group 11
    [ 2.042829] iommu: Adding device 0000:00:1f.3 to group 11
    [ 2.042833] iommu: Adding device 0000:01:00.0 to group 1
    [ 2.042836] iommu: Adding device 0000:01:00.1 to group 1
    [ 2.042841] iommu: Adding device 0000:03:00.0 to group 12
    [ 2.042866] iommu: Adding device 0000:05:00.0 to group 13
    [ 2.042882] iommu: Adding device 0000:05:00.1 to group 13
    [ 2.042897] iommu: Adding device 0000:05:00.2 to group 13
    [ 2.042913] iommu: Adding device 0000:05:00.3 to group 13

    tim@linux-lgui:~>

    Code:
     tim@linux-lgui:~> sudo mkinitrd
    [sudo] password for root:
    Creating initrd: /boot/initrd-4.12.14-lp151.28.4-default
    dracut: Executing: /usr/bin/dracut --logfile /var/log/YaST2/mkinitrd.log --force /boot/initrd-4.12.14-lp151.28.4-default 4.12.14-lp151.28.4-default
    dracut: *** Including module: bash ***
    dracut: *** Including module: systemd ***
    dracut: *** Including module: warpclock ***
    dracut: *** Including module: systemd-initrd ***
    dracut: *** Including module: i18n ***
    dracut: *** Including module: drm ***
    dracut: Possible missing firmware "amdgpu/vega20_asd.bin" for kernel module "amdgpu.ko"
    dracut: Possible missing firmware "amdgpu/vega20_sos.bin" for kernel module "amdgpu.ko"
    dracut: Possible missing firmware "amdgpu/vega20_rlc.bin" for kernel module "amdgpu.ko"
    dracut: Possible missing firmware "amdgpu/vega20_mec2.bin" for kernel module "amdgpu.ko"
    dracut: Possible missing firmware "amdgpu/vega20_mec.bin" for kernel module "amdgpu.ko"
    dracut: Possible missing firmware "amdgpu/vega20_me.bin" for kernel module "amdgpu.ko"
    dracut: Possible missing firmware "amdgpu/vega20_pfp.bin" for kernel module "amdgpu.ko"
    dracut: Possible missing firmware "amdgpu/vega20_ce.bin" for kernel module "amdgpu.ko"
    dracut: Possible missing firmware "amdgpu/vega20_sdma1.bin" for kernel module "amdgpu.ko"
    dracut: Possible missing firmware "amdgpu/vega20_sdma.bin" for kernel module "amdgpu.ko"
    dracut: Possible missing firmware "amdgpu/vega20_uvd.bin" for kernel module "amdgpu.ko"
    dracut: Possible missing firmware "amdgpu/vega20_vce.bin" for kernel module "amdgpu.ko"
    dracut: Possible missing firmware "amdgpu/vega20_smc.bin" for kernel module "amdgpu.ko"
    dracut: *** Including module: plymouth ***
    dracut: *** Including module: btrfs ***
    dracut: *** Including module: kernel-modules ***
    dracut: *** Including module: resume ***
    dracut: *** Including module: rootfs-block ***
    dracut: *** Including module: suse-btrfs ***
    dracut: *** Including module: suse-xfs ***
    dracut: *** Including module: terminfo ***
    dracut: *** Including module: udev-rules ***
    dracut: Skipping udev rule: 40-redhat.rules
    dracut: Skipping udev rule: 50-firmware.rules
    dracut: Skipping udev rule: 50-udev.rules
    dracut: Skipping udev rule: 91-permissions.rules
    dracut: Skipping udev rule: 80-drivers-modprobe.rules
    dracut: *** Including module: dracut-systemd ***
    dracut: *** Including module: haveged ***
    dracut: *** Including module: ostree ***
    dracut: *** Including module: usrmount ***
    dracut: *** Including module: base ***
    dracut: *** Including module: fs-lib ***
    dracut: *** Including module: shutdown ***
    dracut: *** Including module: suse ***
    dracut: *** Including modules done ***
    dracut: *** Installing kernel module dependencies and firmware ***
    dracut: *** Installing kernel module dependencies and firmware done ***
    dracut: *** Resolving executable dependencies ***
    dracut: *** Resolving executable dependencies done***
    dracut: *** Hardlinking files ***
    dracut: *** Hardlinking files done ***
    dracut: *** Stripping files ***
    dracut: *** Stripping files done ***
    dracut: *** Generating early-microcode cpio image ***
    dracut: *** Constructing GenuineIntel.bin ****
    dracut: *** Store current command line parameters ***
    dracut: Stored kernel commandline:
    dracut:  resume=UUID=7d8685e3-4c8e-44c4-98f3-1af5c3a99a7a
    dracut:  root=UUID=0347972f-4cd8-4ae3-93d8-3072095afb3d rootfstype=btrfs rootflags=rw,relatime,space_cache,subvolid=267,subvol=/@/.snapshots/1/snapshot,subvol=@/.snapshots/1/snapshot
    dracut: *** Creating image file '/boot/initrd-4.12.14-lp151.28.4-default' ***
    dracut: *** Creating initramfs image file '/boot/initrd-4.12.14-lp151.28.4-default' done ***
    tim@linux-lgui:~>
    This doesn't include the VFIO module, and it's not listed as a kernel module in the
    Code:
    lspci -nnk
    Code:
    05:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:2182] (rev a1)
            Subsystem: eVga.com. Corp. Device [3842:1261]
            Kernel modules: nouveau, nvidia_drm, nvidia
    05:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:1aeb] (rev a1)
            Subsystem: eVga.com. Corp. Device [3842:1261]
            Kernel driver in use: snd_hda_intel
            Kernel modules: snd_hda_intel
    05:00.2 USB controller [0c03]: NVIDIA Corporation Device [10de:1aec] (rev a1)
            Subsystem: eVga.com. Corp. Device [3842:1261]
            Kernel driver in use: xhci_hcd
            Kernel modules: xhci_pci
    05:00.3 Serial bus controller [0c80]: NVIDIA Corporation Device [10de:1aed] (rev a1)
            Subsystem: eVga.com. Corp. Device [3842:1261]
    tim@linux-lgui:~>
    My 960 also doesn't show "Access Control Services" so i'm not sure what to make of it, but I have read posts about the card being able to use VFIO and the like for gpu passthrough so maybe that's not necessary (I hope not), this being Nvidia, and the Error 43 issue, I don't feel like the hardware will advertise this feature.
    Hi
    Can you remove this part of the grub command line;

    Code:
     vfio-pci.ids=10de:2182,10de:1aeb
    Did you remove the /etc/mkinitcpio.conf file (it's not needed)

    Yes the ACS is related to the iommu, which by the looks of your output should be fine, can you run (after you fix grub and reboot) as your user;

    Code:
    for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU Group %s ' "$n"; /sbin/lspci -nns "${d##*/}"; done;
    As long as both cards are in separate iommu groups you should be fine.

    Once you have the above information, can sort out the vfio stuff....
    Cheers Malcolm °¿° SUSE Knowledge Partner (Linux Counter #276890)
    SUSE SLE, openSUSE Leap/Tumbleweed (x86_64) | GNOME DE
    If you find this post helpful and are logged into the web interface,
    please show your appreciation and click on the star below... Thanks!

  6. #16

    Default Re: Nvidia GPU not using VFIO and it's not listed as a kernel module

    Code:
    IOMMU Group 13 05:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:2182] (rev a1)
    IOMMU Group 13 05:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:1aeb] (rev a1)
    IOMMU Group 13 05:00.2 USB controller [0c03]: NVIDIA Corporation Device [10de:1aec] (rev a1)
    IOMMU Group 13 05:00.3 Serial bus controller [0c80]: NVIDIA Corporation Device [10de:1aed] (rev a1)
    IOMMU Group 1 00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller [8086:0c01] (rev 06)
    IOMMU Group 1 01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Baffin [Radeon RX 560] [1002:67ff] (rev cf)
    IOMMU Group 1 01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:aae0]
    Yep, they're in separate iommu groups! I've deleted the mkinitcpio file (and a misspelled version of it, and deleted the vfio bit from grub. I used YaST to remove that section so I'm assuming it's already run a grub update, and I also ran
    Code:
    dracut -f
    as well and will reboot. So what do you think it required next to get the VFIO drivers to work? Thank you for all of your assistance so far, I'm glad we're working on this in the order it needs to be. You're correct I jumped ahead and started worrying about UEFI in virt-manager before I got the grub line correct.

  7. #17
    Join Date
    Jun 2008
    Location
    Podunk
    Posts
    26,513
    Blog Entries
    15

    Default Re: Nvidia GPU not using VFIO and it's not listed as a kernel module

    Quote Originally Posted by timpster View Post
    Code:
    IOMMU Group 13 05:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:2182] (rev a1)
    IOMMU Group 13 05:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:1aeb] (rev a1)
    IOMMU Group 13 05:00.2 USB controller [0c03]: NVIDIA Corporation Device [10de:1aec] (rev a1)
    IOMMU Group 13 05:00.3 Serial bus controller [0c80]: NVIDIA Corporation Device [10de:1aed] (rev a1)
    IOMMU Group 1 00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller [8086:0c01] (rev 06)
    IOMMU Group 1 01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Baffin [Radeon RX 560] [1002:67ff] (rev cf)
    IOMMU Group 1 01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:aae0]
    Yep, they're in separate iommu groups! I've deleted the mkinitcpio file (and a misspelled version of it, and deleted the vfio bit from grub. I used YaST to remove that section so I'm assuming it's already run a grub update, and I also ran
    Code:
    dracut -f
    as well and will reboot. So what do you think it required next to get the VFIO drivers to work? Thank you for all of your assistance so far, I'm glad we're working on this in the order it needs to be. You're correct I jumped ahead and started worrying about UEFI in virt-manager before I got the grub line correct.
    Hi
    All looks good sofar

    So create a /etc/modules-load.d/vfio.conf file containing;

    Code:
    vfio_pci
    vfio
    vfio_iommu_type1
    vfio_virqfd
    Based on your output;
    Code:
    IOMMU Group 13 05:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:2182] (rev a1)
    Create a /etc/modprobe.d/11-vfio.conf file containing;

    Code:
    options vfio-pci ids=10de:2182,10de:1aeb
    Reboot and see the modules are loaded, nvidia card using vfio-pci etc

    Code:
    dmesg | grep VFIO
    
    lsmod | grep kvm
    
    lsmod | grep vfio
    
    lspci -kn | grep -A 3 05:00.0
    lspci -kn | grep -A 3 05:00.1
    Last edited by malcolmlewis; 10-Jun-2019 at 17:37.
    Cheers Malcolm °¿° SUSE Knowledge Partner (Linux Counter #276890)
    SUSE SLE, openSUSE Leap/Tumbleweed (x86_64) | GNOME DE
    If you find this post helpful and are logged into the web interface,
    please show your appreciation and click on the star below... Thanks!

  8. #18
    Join Date
    Jun 2008
    Location
    Podunk
    Posts
    26,513
    Blog Entries
    15

    Default Re: Nvidia GPU not using VFIO and it's not listed as a kernel module

    Quote Originally Posted by malcolmlewis View Post
    Hi
    All looks good sofar

    So create a /etc/modules-load.d/vfio.conf file containing;

    Code:
    vfio_pci
    vfio
    vfio_iommu_type1
    vfio_virqfd
    Based on your output;
    Code:
    IOMMU Group 13 05:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:2182] (rev a1)
    Create a /etc/modprobe.d/11-vfio.conf file containing;

    Code:
    options vfio-pci ids=10de:2182,10de:1aeb
    Reboot and see the modules are loaded, nvidia card using vfio-pci etc

    Code:
    dmesg | grep VFIO
    
    lsmod | grep kvm
    
    lsmod | grep vfio
    
    lspci -kn | grep -A 3 05:00.0
    lspci -kn | grep -A 3 05:00.1
    Hi
    Just a follow on a quick test here with an Nvidia GT8800 (Not sure it will work further from a UEFI perspective in KVM, but can try), I just use the GPU cores here, no nvidia GL libs, no monitor attached, just use the intel GPU...

    Code:
    dmesg | grep VFIO
    
    [    3.805703] VFIO - User Level meta-driver version: 0.3
    
    /sbin/lsmod | grep kvm
    kvm_intel             307200  0
    kvm                   733184  1 kvm_intel
    irqbypass              16384  2 vfio_pci,kvm
    
    /sbin/lsmod | grep vfio
    vfio_pci               61440  0
    vfio_virqfd            16384  1 vfio_pci
    vfio_iommu_type1       32768  0
    vfio                   36864  2 vfio_iommu_type1,vfio_pci
    irqbypass              16384  2 vfio_pci,kvm
    
    /sbin/lspci -kn | grep -A 3 01:00.0
    
    01:00.0 0300: 10de:0611 (rev a2)
        Subsystem: 3842:c801
        Kernel driver in use: vfio-pci
        Kernel modules: nouveau, nvidia
    Cheers Malcolm °¿° SUSE Knowledge Partner (Linux Counter #276890)
    SUSE SLE, openSUSE Leap/Tumbleweed (x86_64) | GNOME DE
    If you find this post helpful and are logged into the web interface,
    please show your appreciation and click on the star below... Thanks!

  9. #19

    Default Re: Nvidia GPU not using VFIO and it's not listed as a kernel module

    That did it. Now I have horrendously slow storage performance so I haven't gotten to a windows desktop again yet. Thank you for every command and patience. I couldn't have found the exact commands on my own in a reasonable amount of time.

  10. #20
    Join Date
    Jun 2008
    Location
    Podunk
    Posts
    26,513
    Blog Entries
    15

    Default Re: Nvidia GPU not using VFIO and it's not listed as a kernel module

    Quote Originally Posted by timpster View Post
    That did it. Now I have horrendously slow storage performance so I haven't gotten to a windows desktop again yet. Thank you for every command and patience. I couldn't have found the exact commands on my own in a reasonable amount of time.
    Hi
    Excellent

    One last thing to add The aliases and vga arbitration....

    Code:
    cat /sys/bus/pci/devices/0000\:05\:00.0/modalias
    cat /sys/bus/pci/devices/0000\:05\:00.1/modalias
    Add the output to /etc/modprobe.d/11-vfio.conf as (example is my pci id);

    Code:
    alias pci:v00008086d00000151sv00008086sd00002035bc06sc04i00 vfio-pci
    alias pci:<nvidia audio> vfio-pci
    options vfio-pci ids=10de:2182,10de:1aeb
    options vfio-pci disable_vga=1

    Then you can move onto the vm and uefi boot etc
    Cheers Malcolm °¿° SUSE Knowledge Partner (Linux Counter #276890)
    SUSE SLE, openSUSE Leap/Tumbleweed (x86_64) | GNOME DE
    If you find this post helpful and are logged into the web interface,
    please show your appreciation and click on the star below... Thanks!

Page 2 of 3 FirstFirst 123 LastLast

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •