Kernel bug after rmmod

I have a PCI-E USB 3.0 card and a USB 3.0 hub which is connected to it.
Everything is working out of box but sometimes it stops working so i’m performing a kernel module reload

sudo rmmod xhci_pci
sudo modprobe xhci_pci

That usually fixes all problems.

But lately i started to have a problem with this method.
rmmod causes a kernel bug pastebin, kernel module xhci_pci cannot be removed “device or resource is busy” after modprobe,
devices stops working until reboot is performed.

I’m using openSUSE Tumbleweed x86_64 with linux 4.15.2-1-default.