The problem is that Virtual Manager isn't able to ask for the password.
I am having the same issue.
But I found this on Red Hat forum that might be useful.
I am able to connect, but only if I do a virt-manager --ssh from the cli, type in my password. I never get a pop up window from the gui.