Nvidia Multi GPU support possible?

Hi All
This is just a nice to have, but would like to use the third GPU’s cuda cores when I don’t need gpu passthrough.

With just the one Nvidia GPU using the driver and the other using vfio-pci all is good, can access the cuda cores as required. When using both cards with the nvidia driver it all heads off into lah-lah-land, I can boot to multi-user and all is fine, driver loading etc, just not graphical target… Note: The nvidia driver is installed with the no-opengl-files and cuda with no-opengl-libs.

The intel GPU is driving primary displays via Display Port and DVI-D. Nvidia GPU 1 is in a PCIe x16 slot, Nvidia GPU 3 is in a PCIe x1 slot.


/sbin/lspci -nnk | egrep -A3 "VGA|Display|3D"

00:02.0 VGA compatible controller [0300]: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller [8086:0152] (rev 09)
    DeviceName:  CPU
    Subsystem: Intel Corporation Device [8086:2035]
    Kernel driver in use: i915
--
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK208B [GeForce GT 710] [10de:128b] (rev a1)
    Subsystem: ZOTAC International (MCO) Ltd. Device [19da:6326]
    Kernel driver in use: nvidia
    Kernel modules: nouveau, nvidia_drm, nvidia
--
03:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK208B [GeForce GT 710] [10de:128b] (rev a1)
    Subsystem: ZOTAC International (MCO) Ltd. Device [19da:5360]
    Kernel driver in use: vfio-pci
    Kernel modules: nouveau, nvidia_drm, nvidia

inxi -Gxx

Graphics:  Device-1: Intel Xeon E3-1200 v2/3rd Gen Core processor Graphics driver: i915 v: kernel bus ID: 00:02.0 
           chip ID: 8086:0152 
           Device-2: NVIDIA GK208B [GeForce GT 710] vendor: ZOTAC driver: nvidia v: 430.26 bus ID: 01:00.0 chip ID: 10de:128b 
           Device-3: NVIDIA GK208B [GeForce GT 710] vendor: ZOTAC driver: vfio-pci v: 0.2 bus ID: 03:00.0 chip ID: 10de:128b 
           Display: x11 server: X.Org 1.20.5 driver: fbdev,modesetting,nvidia,vesa alternate: nouveau,nv 
           compositor: gnome-shell resolution: 1920x1080~60Hz, 1600x900~60Hz 
           OpenGL: renderer: Mesa DRI Intel Ivybridge Desktop v: 4.2 Mesa 19.0.5 compat-v: 3.0 direct render: Yes

I’m guessing I need some Xorg device and screen tweaking for the additional card, or just live with what I have and get a card with more cuda cores…

Hi
Solved… was the power to the 1x slot, moved the card over to the 4x slot and all is good now, can switch one card to vfio-pci when required for gpu passthough and use both gpu’s for cuda when required…


 /sbin/lspci -nnk | egrep -A3 VGA

00:02.0 VGA compatible controller [0300]: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller [8086:0152] (rev 09)
    DeviceName:  CPU
    Subsystem: Intel Corporation Device [8086:2035]
    Kernel driver in use: i915
--
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK208B [GeForce GT 710] [10de:128b] (rev a1)
    Subsystem: ZOTAC International (MCO) Ltd. Device [19da:6326]
    Kernel driver in use: nvidia
    Kernel modules: nouveau, nvidia_drm, nvidia
--
02:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK208B [GeForce GT 710] [10de:128b] (rev a1)
    Subsystem: ZOTAC International (MCO) Ltd. Device [19da:5360]
    Kernel driver in use: nvidia
    Kernel modules: nouveau, nvidia_drm, nvidia

25-Jun-2019 12:32:00 ---] CUDA: NVIDIA GPU 0: GeForce GT 710 (driver version 430.26, CUDA version 10.2, compute capability 3.5, 981MB, 959MB available, 366 GFLOPS peak)
25-Jun-2019 12:32:00 ---] CUDA: NVIDIA GPU 1: GeForce GT 710 (driver version 430.26, CUDA version 10.2, compute capability 3.5, 981MB, 959MB available, 366 GFLOPS peak)
25-Jun-2019 12:32:00 ---] OpenCL: NVIDIA GPU 0: GeForce GT 710 (driver version 430.26, device version OpenCL 1.2 CUDA, 981MB, 959MB available, 366 GFLOPS peak)
25-Jun-2019 12:32:00 ---] OpenCL: NVIDIA GPU 1: GeForce GT 710 (driver version 430.26, device version OpenCL 1.2 CUDA, 981MB, 959MB available, 366 GFLOPS peak)

I seem to remember many, many years ago in these Forums there was a thread about multiple nVidia GPUs,
Of course don’t know if any of that is still relevant today, but I remember that when there are multiple nVidia GPUs there is/was a special communications channel that could be used for communications between the GPUs. Without remembering the specifics, I’d guess today that it would have been a hardware/wired connection bypassing board hardware.

TSU

Hi
Probably SLI? I’m not using the graphics output on the host (jus passthrough) since I have not installed the opengl files with the nvidia driver, I guess if I did can set the multigpu option in the Xorg device config file…

If I had more CPU cores could run multiple systems… I’m contemplating getting a RX570 to see how that goes else a GTX1650 both around the same price…

Yeah,
That sounds very familiar, I’m pretty certain SLI was what was discussed.

TSU