The problem is that Virtual Manager isn't able to ask for the password.

I am having the same issue.

But I found this on Red Hat forum that might be useful.

https://bugzilla.redhat.com/show_bug.cgi?id=579579

I am able to connect, but only if I do a virt-manager --ssh from the cli, type in my password. I never get a pop up window from the gui.

Pup