KDE nearly unresponsive when RAM full and swap is in use

lazlojamf · January 26, 2012, 4:37pm

I have been running KDE 4.6 and openSuSE 11.4 for several months, on a lenovo W510 laptop. I’ve ironed out every hardware/software issue except for the following problem, which crops up frequently.

When running an application that uses a lot of RAM (e.g. mathematica or a program called visual molecular dynamics or VMD), the desktop locks up almost completely when the RAM gets close to full and the swap file starts filling up. I have 4GB 1333mHz ram and an 8GB swap file (located on my extended partition along with /home and root). For comparison, I can perform identical tasks in windows7 (some sluggishness when the RAM fills up, but I can still interact with the windows). I understand that the swap file is much slower than RAM, but sometimes it takes me several minutes just to close the offending program. Given that windows shows much better performance with the same task and nearly full RAM, I suspect something is wrong.

I have a desktop (i7-2600, NVidia 460) with the exact same software setup, and it has the same exact problem. I assume that even a quad-core laptop shouldn’t stall like this when the memory gets full. Please let me know if anybody has any suggestions for diagnosing this problem.

martin_helm · January 26, 2012, 5:05pm

With my old PC which had 4GB of RAM I ran several time into that issue
when doing calculations with very large datasets. As far as I know this
is a known issue of the linux kernel.
http://www.phoronix.com/scan.php?page=news_item&px=ODQ3OQ
http://www.phoronix.com/scan.php?page=news_item&px=ODQ3Mw

Since I moved a few months ago to a new machine with 16GB I did not
investigate further.
A blind shot would be to try a newer 3.x kernel and see if it improves
the situation (you can do that with 11.4 keeping the old kernel in case
it does not work well and you want to switch back).
It amy also be worth to try kernel-default vs the kernel-desktop in your
situation because they have different settings for latency.

Beside that I have no further idea. That is just what I would try.

lazlojamf · January 26, 2012, 5:34pm

Thank you for your advice. I’ll give the 3.x kernel a try later this afternoon. Another thought occurred to me, as well.

I’ve noticed some weird sporadic file transfer issues that may be related. For example, when moving large files over a network, CPU usage seems unnaturally high (sometimes up to 25% on an 8-thread machine) and the laptop tends to heat up more than expected (implying that the i7 is speed-boosting for some reason). Perhaps I’m being paranoid, but I find it hard to believe that a transfer could be so expensive. Additionally, network transfers of large files (i.e. those that are 4-10GB) sometimes stalls for several minutes with no apparent activity. Lastly, I get absolutely abysmal transfer rates on some USB 2.0 flash drives (and it’s not a file quantity problem).

I can’t help but wonder if these issues are related, or if the position of my swap drive in the overall partition table is an issue, too (it’s currently in-between the root and home directories).

DenverD · January 26, 2012, 6:17pm

On 01/26/2012 05:05 PM, Martin Helm wrote:
> I have no further idea

he could minimize other factors which may be taking resources and/or
causing slow downs in getting the window’s attention…(i mean it seems
s/he is happy with the speed of the calc speed, and is only stiffed by
the time it takes to access the window, so)

-use the lightest possible desktop environment/window manager which
supports the visual output (my guess from most to least needy: KDE,
Gnome, LXDE, Xfce and then all the others are really low [enlightment,
blackbox, openbox, fluxbox, iceWM and others]

-minimize the needs of the desktop selected: e.g.
–turn off desktop effects
–pick a simple theme (stay away from rounded corners, 3d look buttons
and panels, drop shadows, etc etc etc [all of that has to be calculated,
sent to the screen, etc etc–taking up space, time and power]

-you wrote “sometimes it takes me several minutes just to close the
offending program” which causes me to wonder: What is that offending
program? is it mathematica or VMD? or (my WAG: Flash in a browser)…you
may need to shut down other application when running the RAM hog: close
the browser, stop the torrent, don’t send or receive dense email files,
no movie watching, or music listening, etc etc etc…in fact when
running the heavy duty stuff turn off the wi-fi or wired net (it takes a
good deal of cycles to do everything network)

-increase the size of the swap partition–and do NOT place swap in RAM
(which will slow the system), nor on a network or USB drive (which will
really gum up the system; put the swap partition on the fastest drive
available…]

i probably forgot several things…Oh!: if happy with the way Windows
does it (“but I can still interact with the windows”), then use what works!

–
DD http://tinyurl.com/DD-Caveat
have you TODAY checked what they say about us at
http://tinyurl.com/SUSEonDW

lazlojamf · January 26, 2012, 7:35pm

This is a good point. While I understand that desktop effects consume some resources, I see no reason why a modern quad-core workstation laptop with a nice video card can’t handle moving, closing, and switching windows while mathematica plugs away with 6 or 7 threads. I’m fairly certain that this is either a deficiency in opensuse or a bug of some kind.

As a point of reference, I’ve done the same number-crunching on a friend’s desktop. He has a very similar i7-2600k system, with 16GB RAM (to my 8GB desktop), and a solid state hard drive. Running Ubuntu 11.10, we can load large datasets into the same programs almost endlessly with no lag in the window manager (unity with the 3D effects enabled). This implies that the problem lies with opensuse or KDE. For the record, I like opensuse more than ubuntu. If I switched, it would take a lot of time to iron out driver issues on the laptop. Likewise, windows works fine with these calculations, but I really prefer suse over windows for a number of reasons.

I’ll try disabling all 3D effects and reproducing the error.

lazlojamf · January 26, 2012, 7:47pm

I’ve tried the following solutions, but they haven’t worked.

(1) I switched to a separate install of opensuse 12.1 and loaded the datasets. The windows still froze beyond any kind of useability. This installation was running the 3.1 kernel and was otherwise identical to my 11.4 setup.

(2) I disabled 3D effects by turning off compiz. This produced a barely noticeable improvement in my ability to move windows around. I don’t think it’s solved the problem, but it helped a little bit. Still, I maintain that even my substantially weaker laptop should be able to handle window movements while the RAM is full and the swap is in use.

Please let me know if you have any other suggestions, and thank you for the input so far.

martin_helm · January 26, 2012, 7:50pm

Am 26.01.2012 19:36, schrieb lazlojamf:
> This implies that the problem lies with opensuse or KDE.
I cannot subscribe to that assessment, compare the kernel versions in
this two distributions!
You think too much here in terms of what you see on the surface.

Ubuntu 11.10 has the 3.0 kernel while openSUSE 11.4 has 2.6.37!
One reason more to try upgrading the kernel on openSUSE 11.4, you may
not get the same performance you see on your friends machine if you do
not also add a solid state drive for faster paging.
http://download.opensuse.org/repositories/Kernel:/stable/standard
contains kernel 3.2 for openSUSE.
Or switch directly to openSUSE 12.1 which is roughly as young as Ubuntu
11.10.

martin_helm · January 26, 2012, 8:03pm

Am 26.01.2012 19:56, schrieb lazlojamf:
> (2) I disabled 3D effects by turning off compiz.
You use compiz? With which desktop environment?

KDE does not use compiz and Gnome 3 in 12.1 also does not use compiz, so
what did you disable really?

martin_helm · January 26, 2012, 9:14pm

If you are willing to invest a bit of time to check, I would install
Ubuntu 11.10 parallel to your openSUSE install on the same machine and
have a look if it runs more efficient on exactly the same hardware,
things like that can happen (and also vice versa I have seen notebooks
where openSUSE outperforms any debian based distro, without knowing the
reason and also the opposite).

lazlojamf · January 26, 2012, 10:10pm

My mistake. I just disabled compositing. I’m using KDE 4.6, which doesn’t use compiz as far as I know. I assume the compositing is native, but I’m not very knowledgeable when it comes to the details of the window manager. Basically, alt+shift+F12 turns off the 3D, and that’s what I did.

lazlojamf · January 26, 2012, 10:17pm

I apologize for being unclear. Here are the tests I’ve run (this involves loading a large dataset in VMD, but the same thing happens in mathematica).

(1) suse 11.4 on lenovo W510 — stalls
(2) suse 11.4 on desktop — stalls
(3) suse 12.1 (kernel 3.1) on lenovo W510 — stalls
(4) ubuntu 11.10 (kernel 3.0) on desktop with SSD — totally fine
(5) ubuntu 11.10 (kernel 3.0) on coworker’s W520 (no SSD) — stalls
(6) windows 7 on my W510 — much more useable. loads the dataset without inhibiting my ability to use chrome, for example. everything is as sluggish as you would expect when the RAM is 88% full and the swap file is in use. Eventually VMD refuses to load more (out of memory)

It looks like I was wrong when I said this was a suse/KDE problem. However, I still think this is a linux problem, since windows performs much better on the same hardware. This occurs on both ubuntu and suse (on comparable machines: W520 vs W510) and with both old and new kernels (suse 11.4 with 2.6 kernel vs suse 12.1 with 3.1 kernel).

Therefore, I’m starting to think that maybe this is a general linux memory problem (possibly with the applications I’m using: mathematica and VMD, though it’s very coincidental that both have this problem). Perhaps linux handles the swap differently than windows handles the paging file. If it turns out to be a case where windows beats linux in memory management, I’ll be a bit disappointed, but I can still live with it. I’ll just buy some more RAM.

Please let me know if you have any other insights into this problem, or if you think I’ve failed to consider all possible culprits.

martin_helm · January 26, 2012, 10:33pm

Am 26.01.2012 22:16, schrieb lazlojamf:
> My mistake. I just disabled compositing. I’m using KDE 4.6, which
> doesn’t use compiz as far as I know. I assume the compositing is
> native, but I’m not very knowledgeable when it comes to the details
> of the window manager. Basically, alt+shift+F12 turns off the 3D, and
> that’s what I did.
>
I just wanted to understand. KDE uses the native compositing of kwin and
you did of course the correct thing with the key combination you used.

martin_helm · January 26, 2012, 10:51pm

Am 26.01.2012 22:26, schrieb lazlojamf:
> I apologize for being unclear. Here are the tests I’ve run (this
> involves loading a large dataset in VMD, but the same thing happens
> in mathematica).
>
> (1) suse 11.4 on lenovo W510 — stalls (2) suse 11.4 on desktop —
> stalls (3) suse 12.1 (kernel 3.1) on lenovo W510 — stalls (4)
> ubuntu 11.10 (kernel 3.0) on desktop with SSD — totally fine (5)
> ubuntu 11.10 (kernel 3.0) on coworker’s W520 (no SSD) — stalls (6)
> windows 7 on my W510 — much more useable. loads the dataset without
> inhibiting my ability to use chrome, for example. everything is as
> sluggish as you would expect when the RAM is 88% full and the swap
> file is in use. Eventually VMD refuses to load more (out of memory)
>
> It looks like I was wrong when I said this was a suse/KDE problem.
That was what I wanted to point to. The desktop environment has nothing
to do with paging and I/O, so the kernel is the usual suspect here. I
just hoped that the kernel versions 3.x are better here, which is
obviously not the case.

> However, I still think this is a linux problem, since windows
> performs much better on the same hardware.
Sad about that and I am also disappointed that windows is better here
but I agree, it seems simply the SSD which makes the difference for your
linux tests and this is not surprising, due to the tremendous
performance boost which comes from the fast I/O of the SSD.

> This occurs on both ubuntu and suse (on comparable machines: W520 vs
> W510) and with both old and new kernels (suse 11.4 with 2.6 kernel vs
> suse 12.1 with 3.1 kernel).
>
> Therefore, I’m starting to think that maybe this is a general linux
> memory problem (possibly with the applications I’m using:
> mathematica and VMD, though it’s very coincidental that both have
> this problem).
I also doubt it is a problem of that applications, they just rely on
what the operating system does.

> Perhaps linux handles the swap differently than
> windows handles the paging file. If it turns out to be a case where
> windows beats linux in memory management, I’ll be a bit disappointed,
> but I can still live with it. I’ll just buy some more RAM.
>
Seems to be your only option or to add a SSD and use that for swap (I do
that, other people are concerned about doing it and that it shortens the
live of the SSD, honestly I do not care about that, the reason I have a
SSD is to use it even if it wears out, you have to judge yourself).

> Please let me know if you have any other insights into this problem,
> or if you think I’ve failed to consider all possible culprits.
>
I am afraid I have no further insight. One possibility is of course that
you contact a kernel mailing list and seek for further advise. Maybe
there is an option you can pass to the kernel which improves that and is
unknown to me or a special configuration you can use to recompile your
kernel for better performance with swap. You can also open a new thread
for that in the forums (not sure which is the best maybe “hardware”
there are some really clever people around here, who know more about
such things than I will ever learn.

robin_listas · January 27, 2012, 12:03am

On 2012-01-26 19:56, lazlojamf wrote:

> Still, I
> maintain that even my substantially weaker laptop should be able to
> handle window movements while the RAM is full and the swap is in use.

Not if there is a lot of memory movements, and the dataset movements happen
to hit swap. In that situation the power of the CPU is useless, because it
has to sit waiting for disk.

Those two programs were initially designed for Windows. It is possible they
plan things as to need swaping less often. I mean that the amount of swap
used be the same but the number of times a data is needed and it happens to
be in swap is less in Windows than in Linux.

Such type of programs benefit by handling disk memory themselves, instead
of leaving the task to the OS: they can predict their movements and
optimize operations for it. For example, The Gimp does just that.

> Please let me know if you have any other suggestions, and thank you for
> the input so far.

Find out if those programs define a local temporary disk space.

–
Cheers / Saludos,

Carlos E. R.
(from 11.4 x86_64 “Celadon” at Telcontar)

robin_listas · January 27, 2012, 12:08am

On 2012-01-26 17:36, lazlojamf wrote:

> I’ve noticed some weird sporadic file transfer issues that may be
> related. For example, when moving large files over a network, CPU usage
> seems unnaturally high (sometimes up to 25% on an 8-thread machine) and
> the laptop tends to heat up more than expected (implying that the i7 is
> speed-boosting for some reason).

This can happen if the network chip is bad, I mean, it wants more cpu
attention than others. Or how good the driver is.

–
Cheers / Saludos,

Carlos E. R.
(from 11.4 x86_64 “Celadon” at Telcontar)

martin_helm · January 27, 2012, 12:08am

Am 26.01.2012 22:51, schrieb Martin Helm:
> I am afraid I have no further insight.
One last idea, I forgot completely to mention swapiness. The default
value on my system is 60, you may play with the value to see if it
improves the situation


cat /proc/sys/vm/swappiness

shows you the current value, you can set it temporarily with


sudo /sbin/sysctl vm.swappiness=10

or any other value between 0 and 100 for experiments.

lazlojamf · January 27, 2012, 3:58pm

Thank you for the swapiness idea. I tried different low values, but the problem was the same.

lazlojamf · January 27, 2012, 4:00pm

This problem is fairly rare, and part of it is likely due to the painfully mediocre (yet expensive) internet connection I have at home. Perhaps the driver isn’t optimal, but I’m just going to let this one go. It could be worse - when I installed ubuntu, I couldn’t get the ethernet port to run above about 10 Mbps for some reason.

lazlojamf · January 27, 2012, 4:03pm

I’m going to look into this. I doubt the VMD developers have the manpower to have this kind of feature, but surely mathematica has some knobs to play with. That said, mathematica is a crutch anyway, and I should probably be writing this code myself.

lazlojamf · January 27, 2012, 4:12pm

I was originally going to upgrade my RAM, but getting an SSD might be more sensible, since I can get a 60GB drive for $90 on amazon and it will likely fix most of the problems with RAM/swap that I have.

Given that I only have a budget for 1 SSD, I would like to know whether it would be more useful to put it in my laptop (the W510 has SATA 1.5 Gbps in the bay and 3Gbps in the primary slot) or my desktop (which has SATA 6Gbps). For the record I use both the laptop and desktop evenly, and it’s no trouble to process the large datasets on the desktop when I get a chance. Is there a significant benefit to using SATA 6Gbps?