Typically the behaviour you describe is because of a bad mixer setting. If you are using gnome, then it is likely a bad setting in alsamixer, and if you are using KDE it is likely a bad setting in kmix.
Turn down your speaker volume, and then go examine your mixer setting.