A very hot second: june 30 UTC 23:59:60

All openSUSE 12.1 machines - without exception - got very hot today here. I had to turn off or reboot all the servers. It didn’t seem to have affected the 11.4 systems nor our openBSD firewalls. I guess the culprit was the leap second. If you wonder why mysqld or something else eats 100% of CPU resources, simply reboot and it should be OK.

On 2012-07-01 03:16, please try again wrote:
>
> All openSUSE 12.1 machines - without exception - got very hot today
> here. I had to turn off or reboot all the servers. It didn’t seem to
> have affected the 11.4 systems nor our openBSD firewalls. I guess the
> culprit was the leap second. If you wonder why mysqld or something else
> eats 100% of CPU resources, simply reboot and it should be OK.

What!?

My 11.4 indeed had thunderbird and firefox both consume lots of cpu
precisely after 00:00 UTC.

<0.5> 2012-07-01 01:59:59 Telcontar kernel - - - [11174.799580] Clock:
inserting leap second 23:59:60 UTC

It disappeared after I hibernated and restored.


Cheers / Saludos,

Carlos E. R.
(from 11.4 x86_64 “Celadon” at Telcontar)

On 2012-07-01 03:38, Carlos E. R. wrote:
> On 2012-07-01 03:16, please try again wrote:

> What!?
>
> My 11.4 indeed had thunderbird and firefox both consume lots of cpu
> precisely after 00:00 UTC.
>
> <0.5> 2012-07-01 01:59:59 Telcontar kernel - - - [11174.799580] Clock:
> inserting leap second 23:59:60 UTC

My laptop, which is running the same kernel version did not do the leap
second adjustment (at least no message in the log), and runs quietly.
Differences: it is not running ntpd, and in the desktop I’m running the
same kernel but compiled locally with some changes.

In the desktop machine I see this in the xntpd log:

30 Jun 23:32:38 ntpd[4194]: 130.206.3.166 9689 89 leap_armed
1 Jul 01:59:59 ntpd[4194]: 0.0.0.0 061b 0b leap_event

I see a report from 2009, where redhat machines crashed at the leap second
inserted on Dec 2008

here

But how come you can relate the busy cpu to the leap second? I believe you,
but why?


Cheers / Saludos,

Carlos E. R.
(from 11.4 x86_64 “Celadon” at Telcontar)

I don’t know, but mysql went crazy. I started to hear fans spinning from every corner and even a processor overheating alarm on one server (took me a while to find out which one). Shutting down the daemon didn’t help. It freed the CPU resources but as soon as I restarded the daemon, it jumped to 100% again and temperature climbed to 89C (now it is back to 42C). Of course, such things always happen when you’re busy with something else. I didn’t have time to tweak around. I rebooted everything, except the file server (11.4) which doesn’t seem to have problems. ntpd server is an openBSD machine, which was fine. I rebooted it too though.

> I started to hear fans spinning from every corner

bug report?

ah, google finds lots of
chatter…<https://www.google.com/search?q=leap+second+linux+problem>

i guess it is a kernel (not openSUSE) problem, still . . .


dd

Our organization did a number of precautions wrt the leap second where I work (in the field of space satellite control), with meetings starting a few months ago in preparation for the event. I was not involved directly, so I don’t have any stories to pass on, other than to note there was a lot of preparation to ensure the transition went smooth.

On 2012-07-01 10:16, oldcpu wrote:

> Our organization did a number of precautions wrt the leap second where
> I work (in the field of space satellite control), with meetings starting
> a few months ago in preparation for the event. I was not involved
> directly, so I don’t have any stories to pass on, other than to note
> there was a lot of preparation to ensure the transition went smooth.

Ah, but that is surely related to timing ans synchronizing things. But CPUs
going berseck? Surely you did not prepare for that :slight_smile:


Cheers / Saludos,

Carlos E. R.
(from 11.4 x86_64 “Celadon” at Telcontar)

On 2012-07-01 08:38, dd@home.dk wrote:
>> I started to hear fans spinning from every corner
>
> bug report?
>
> ah, google finds lots of
> chatter…<https://www.google.com/search?q=leap+second+linux+problem>

wired

ycombinator

serverfault

Google did not insert a leap second: instead it distributes milliseconds
all over the day and adjusts slowly. They did this on their internal ntp
server, so all machines adjusted to this automatically.

> i guess it is a kernel (not openSUSE) problem, still . . .

Indeed, but what could cause it?

Java is also blamed. In my case, both FF and Thunderbid went over the roof
in cpu cycles. I did not reboot, just hibernated, and it got cured. The
process in my machine restarts ntpd too.


Cheers / Saludos,

Carlos E. R.
(from 11.4 x86_64 “Celadon” at Telcontar)

Carlos E. R. wrote:
> Indeed, but what could cause it?
>
> Java is also blamed. In my case, both FF and Thunderbid went over the roof
> in cpu cycles. I did not reboot, just hibernated, and it got cured. The
> process in my machine restarts ntpd too.

I blame a secret international conspiracy myself.

From http://en.wikipedia.org/wiki/Leap_second

“January 2012: The ITU decided to postpone a decision on leap seconds to
the World Radio Conference in 2015. France, Italy, Japan, Mexico and the
US were reported to be in favor while Canada, China, Germany and the UK
were reportedly against.”

It’s obviously the American imperialists and their lackeys that have
planted these bugs in order to strengthen their case :slight_smile:

On 07/02/2012 04:53 AM, Dave Howorth wrote:
> Carlos E. R. wrote:
>> Indeed, but what could cause it?
>>
>> Java is also blamed. In my case, both FF and Thunderbid went over the roof
>> in cpu cycles. I did not reboot, just hibernated, and it got cured. The
>> process in my machine restarts ntpd too.
>
> I blame a secret international conspiracy myself.
>
> From http://en.wikipedia.org/wiki/Leap_second
>
> “January 2012: The ITU decided to postpone a decision on leap seconds to
> the World Radio Conference in 2015. France, Italy, Japan, Mexico and the
> US were reported to be in favor while Canada, China, Germany and the UK
> were reportedly against.”
>
> It’s obviously the American imperialists and their lackeys that have
> planted these bugs in order to strengthen their case :slight_smile:

The only problem with that theory is that the bug was introduced into the kernel
by Thomas Gleixner <xxxxxxxxxx.de>. Clearly, it is a German conspiracy to
discredit the countries that voted for postponing the decision. What is even
more amazing is that the trap was planted in the Linux kernel in July 2007 - 4.5
years BEFORE the ITU decision. Wow, time travel must have been used. :slight_smile:

I think the time servers should have waited a second instead of adding a second. When you switch to DST, you skip an hour, you don’t add an hour. Could you imagine what computers would do during 25th hour?

Yup, but I’m not allowed to speak about that in public :smiley: