I have had many problems with VirtualBox either freezing when I log in to Windows XP or freezing when switching from seamless mode to another mode.
I can tell that it is not 100% frozen because I can still see XP loaded even the screen saver would come up but there is a total disconnection from the software to the keyboard and mouse.
I run openSUSE 11.1.
Possibly try turning off desktop effects if you have it enabled, see if that helps.
V.Box runs sweet for me either way though.