WOW what the heck, it was blacklisted in 50-blacklist.conf. I do not know if it was done by broacom-wl or something before that, that that was the problem. As a side not broadcom-wl also creates file called 50-brcmfmac.conf in modprobe.d directory that does the same thing and I thought I got rid of blacklist, but apparently no.

So you can close this thread.