kernel 2.6.37 running much slower than 2.6.34 ?

rems0 · February 2, 2011, 7:10pm

Hi list,

we run a bunch of nodes here (880 cores, 1368 GB memory) all of them with openSUSE 11.3.
We were testing to update to a newer kernel, and are now trying to do some simple performance tests.
We downloaded kernel-default-2.6.37-6.1 from Index of /repositories/Kernel:/stable/openSUSE_11.3 and ran our production calculations comparing to 2.6.34.7-0.5 .
2.6.37-6.1 was in all cases much slower! Up to 50% and more.

A simple testcase:

time perl -e ‘foreach (1…9e7) { sqrt(1234)/sin(1234) }’

runs on 2.6.34.7-0.5 on 3.5 sec average.
On 2.6.37-6.1 it runs between 3.3 sec and 5.1 sec intermittently.

time perl -e ‘foreach (1…9e8) { sqrt(1234)/sin(1234) }’

takes on average 35 sec on 2.6.34.7-0.5 and 33 sec to 53 sec on 2.6.37-6.1 .

I see much more interrupts and context switches on 2.6.34.7-0.5 .
I don’t know what is happening here.

Any clues?

Thanks,
Richard

jdmcdaniel3 · February 3, 2011, 1:05am

So, I guess I am not sure what this test does, but here are the times I got on my fastest system:

real 0m2.286s
user 0m2.277s
sys 0m0.003s

Also, when any version of openSUSE is released, it is surely optimized with its existing kernel. The question I have is did you base the compile on the existing configuration? I have a script file that can do this and compile the kernel for you here:

S.A.K.C. - SUSE Automated Kernel Compiler - Version 2.00

I request that you take a look at message #17 & 22, even if you are a seasoned kernel compiler.

Thank You,

rems0 · February 3, 2011, 12:53pm

It’s not important what the test does, just simple basic calculations, since there is where I have seen the performance drop from 2.6.34 to 2.6.37.

I just wanted other people to try running this simple perl command I posted (or anything similar) on one computer, with both 2.6.34 AND 2.6.37 and post their results. Do others also see 2.6.34 being that much faster than 2.6.37 ?
Could you do that? Could you run several times on 2.6.34 AND 2.6.37 on one computer and post your results?

As I wrote before, I downloaded the kernel from Index of /repositories/Kernel:/stable/openSUSE_11.3 , so I did not compile it myself.
I was just comparing two openSUSE kernels with their default configuration and was surprised to see this big performance drop in calculation time.

Anyone else who could try this on 2.6.34 AND 2.6.37 on the same hardware? The kernel from the above repository is pretty easy to install.

Thanks,
Richard

pistazienfresser · February 3, 2011, 2:27pm

My installed openSUSE 11.4 factory-tested with kernel Linux 2.6.37 seems to be calculating in a much faster way than my openSUSE 11.3 with Kernel Linux 2.6.34.7-0.7. But I doubt that this would be ether representative or reproductive (without installing both freshly).

11.4 factory-tested GNOME

~> uname -a
...] 2.6.37-desktop #1 SMP PREEMPT 2011-01-07 00:20:37 +0100 i686 i686 i386 GNU/Linux

~> time perl -e 'foreach (1..9e7) { sqrt(1234)/sin(1234) }'

real	0m8.366s
user	0m8.206s
sys	0m0.007s

real	0m9.004s
user	0m8.186s
sys	0m0.009s

real	0m8.618s
user	0m8.220s
sys	0m0.005s

real	0m8.536s
user	0m8.183s
sys	0m0.008s

real	0m8.480s
user	0m8.210s
sys	0m0.006s

with compiz off:
> time perl -e 'foreach (1..9e7) { sqrt(1234)/sin(1234) }'

real	0m8.375s
user	0m8.183s
sys	0m0.002s

real	0m8.270s
user	0m8.168s
sys	0m0.007s

real	0m8.302s
user	0m8.181s
sys	0m0.007s

real	0m8.400s
user	0m8.184s
sys	0m0.010s

real	0m8.300s
user	0m8.176s
sys	0m0.009s


11.3 GNOME
~> uname -a
Linux ...] 2.6.34.7-0.7-desktop #1 SMP PREEMPT 2010-12-13 11:13:53 +0100 i686 i686 i386 GNU/Linux

With compiz on:

real	1m34.764s
user	0m29.426s
sys	0m52.432s

real	1m34.562s
user	0m29.423s
sys	0m52.415s

real	1m33.946s
user	0m29.433s
sys	0m52.407s

real	1m34.696s
user	0m29.531s
sys	0m52.370s

real	1m34.772s
user	0m29.469s
sys	0m52.400s


with compiz off:

real	1m28.276s
user	0m29.394s
sys	0m52.514s

real	1m23.269s
user	0m29.335s
sys	0m52.446s

real	1m23.302s
user	0m29.328s
sys	0m52.459s

real	1m22.748s
user	0m29.320s
sys	0m52.461s

real	1m22.767s
user	0m29.348s
sys	0m52.454s

real	1m23.498s
user	0m29.417s
sys	0m52.406s

Have a lot of fun
Martin
(pistazienfresser)

rems0 · February 3, 2011, 3:46pm

Hi Martin,

thank you for testing! In your case 2.6.37 is so much faster than 2.6.34! I wish I could see those factors here!

On what hardware are you running? What processor?

I am running here on a dual Xeon X5650 @ 2.67 GHz, runlevel 3 so no X running, and both tests on equal nodes with equal openSUSE 11.3 64bit system installed via PXE, only on the 2nd one I updated the kernel to 2.6.37 from the repo listed above.

Could you test on your 11.3 also with 2.6.37 from the above repo?

Thank you,
Richard

rems0 · February 3, 2011, 5:43pm

Martin,

could you try another command:

time perl -e ‘use threads; $T=12 ; foreach (1…$T) { $thr$i++] = threads->create(sub { printf "I am thread %s
", threads->tid(); foreach (1…9e6) { push(@a, sqrt(1234)/sin(1234)) } ; printf "thread %s finished.
", threads->tid(); }); } foreach (0…$T-1) { $thr$_]->join(); }’

Set T to your number of cores and u could have to reduce the 9e6 in the inner loop to a number suitable to your amount of memory.
Is this still faster on 2.6.37 than on 2.6.34 for you? Similar values?
On my hardware, 2.6.37 needs about 70% more time for this, 17 secs to 10 secs on 2.6.34.

Thanks,
Richard

jdmcdaniel3 · February 3, 2011, 11:37pm

Here is the result on openSUSE 11.4 and kernel 2.6.37 and an Intel i7 2600K 3.4 GHz CPU T=4 and no other changes.

james@linux-9w3h:~> ptest
I am thread 1
I am thread 2
I am thread 3
I am thread 4
I am thread 5
I am thread 6
I am thread 7
I am thread 8
I am thread 9
I am thread 10
I am thread 11
I am thread 12
thread 6 finished.
thread 1 finished.
thread 5 finished.
thread 7 finished.
thread 8 finished.
thread 4 finished.
thread 11 finished.
thread 3 finished.
thread 2 finished.
thread 9 finished.
thread 12 finished.
thread 10 finished.

real    0m8.022s
user    0m26.863s
sys     0m3.092s

Thank You,

rems0 · February 4, 2011, 1:24pm

Thanks James, but what I would need is always at least one run with each kernel version 2.6.34 AND 2.6.37.
I don’t want to compare run times on different hardware, but to compare on the same hardware.
Is 2.6.37 faster/same/slower than 2.6.34 on your hardware using the perl test run posted above?
Could you try the same test on the same hardware but booting 2.6.34 ?

Many thanks,
Richard