Leap second: Linux can freeze

[jdmcdaniel3]

The leap second that was inserted on Saturday night can cause permanent high CPU loads on Linux computers. Among other examples, the behaviour is documented in the blog of the Mozilla Foundation, where strange peak loads on one Mozilla server were observed from the time the leap second was added. The developers said that the problem is easily solved by resetting the date or rebooting the system.

Read More: Leap second: Linux can freeze - The H Open: News and Features

Thank You,

[malcolmlewis]

[image:
http://www.h-online.com/imgs/43/8/8/...a547b6332.png]

The leap second that was inserted on Saturday night can cause
'permanent high CPU loads' (https://lkml.org/lkml/2012/6/30/122) on
Linux computers. Among other examples, the behaviour is 'documented'
(http://tinyurl.com/7k7asbb) in the blog of the Mozilla Foundation,
where strange peak loads on one Mozilla server were observed from the
time the leap second was added. The developers said that the problem is
easily solved by resetting the date or rebooting the system.

'Read More: Leap second: Linux can freeze - The H Open: News and
Features' (http://tinyurl.com/7z2jjvu)

Thank You,

Hi
There is a simple fix... see here
https://lkml.org/lkml/2012/7/1/27

--
Cheers Malcolm °¿° (Linux Counter #276890)
SUSE Linux Enterprise Desktop 11 (x86_64) Kernel 3.0.34-0.7-default
up 21:10, 2 users, load average: 0.63, 0.50, 0.43
CPU Intel i5 CPU M520@2.40GHz | Intel Arrandale GPU

[jdmcdaniel3]

Here is a script you can use. Save it as the file $HOME/bin/leap-second.sh

#!/bin/bash

#: Title       : leap-second.sh
#: Date Created: Sun Jul 1 17:35:00 CDT 2012
#: Last Edit   : Mon Jul 2 11:38:56 CDT 2012
#: Author      : Sheeri K. Cabral
#: Version     : 1.00
#: Description : 
#: Options     : 

#
# Check to see if we are root ************************************************
#
if [[ $UID -ne 0 ]]; then
  clear
  id -Gn | grep -q wheel || echo "Root User Permissions are required, Please Enter the ..."
  echo
  sudo $0 $@
  Exit_Code=$?
  exit $Exit_Code
else
  clear
fi

# this is a quick-fix to the 6/30/12 leap second bug

if [ ! -f /tmp/leapsecond_2012_06_30 ] ; then
  /etc/init.d/ntpd stop; date -s "`date`" && /bin/touch /tmp/leapsecond_2012_06_30
fi

exit 0

# End Of Script

Then use the this terminal command to mark it executable: chmod +x $HOME/bin/leap-second.sh

When run, it will ask for the root user password if you are not root. According to the LKML article, you could just run the the terminal command: date -s "`date`", but then proceeds to post a patch file as follows:

@@ -942,7 +942,7 @@ static void timekeeping_adjust(s64 offset)
  *
  * Returns the unconsumed cycles.
  */
-static cycle_t logarithmic_accumulation(cycle_t offset, int shift)
+static cycle_t logarithmic_accumulation(cycle_t offset, int shift, int* clockset)
 {
     u64 nsecps = (u64)NSEC_PER_SEC << timekeeper.shift;
     u64 raw_nsecs;
@@ -963,6 +963,8 @@ static cycle_t logarithmic_accumulation(cycle_t offset, int shift)
         leap = second_overflow(timekeeper.xtime.tv_sec);
         timekeeper.xtime.tv_sec += leap;
         timekeeper.wall_to_monotonic.tv_sec -= leap;
+        if (leap)
+            *clockset = 1;
     }
 
     /* Accumulate raw time */
@@ -994,6 +996,7 @@ static void update_wall_time(void)
     struct clocksource *clock;
     cycle_t offset;
     int shift = 0, maxshift;
+    int clockset = 0;
     unsigned long flags;
 
     write_seqlock_irqsave(&timekeeper.lock, flags);
@@ -1026,7 +1029,7 @@ static void update_wall_time(void)
     maxshift = (64 - (ilog2(ntp_tick_length())+1)) - 1;
     shift = min(shift, maxshift);
     while (offset >= timekeeper.cycle_interval) {
-        offset = logarithmic_accumulation(offset, shift);
+        offset = logarithmic_accumulation(offset, shift, &clockset);
         if(offset < timekeeper.cycle_interval<<shift)
             shift--;
     }
@@ -1079,6 +1082,8 @@ static void update_wall_time(void)
         leap = second_overflow(timekeeper.xtime.tv_sec);
         timekeeper.xtime.tv_sec += leap;
         timekeeper.wall_to_monotonic.tv_sec -= leap;
+        if (leap)
+            clockset = 1;
     }
 
     timekeeping_update(false);
@@ -1086,6 +1091,8 @@ static void update_wall_time(void)
 out:
     write_sequnlock_irqrestore(&timekeeper.lock, flags);
 
+    if (clockset)
+        clock_was_set();
 }
 
 /**
-- 
1.7.9.5

I presume this patches the kernel source file though I was unable to gleam the intended kernel version or if it matters. As always, I come up short on the exact method to apply such a kernel patch. I wonder when the next time a leap second will even be needed again? I think I would just patch the kernel source files for active kernel versions and see what happens next time around.

Thank You,

[jdmcdaniel3]

Rise of power consumption at Hetzner Online's data centresSource: Hetzner Online

Power consumption at Hetzner Online's data centres jumped by "about a megawatt" in the night from 30 June to 1 July due to a kernel bug which resulted in many Linux systems reaching 100% processor usage in dealing with the leap second added at midnight. The problem is described in an email sent by the web hosting company to customers, asking them to check their systems' CPU usage and, where necessary, to restart their systems in order to restore processor usage to normal levels.

Read More: Leap second bug in Linux wastes electricity - The H Open: News and Features

Thank You,

[Knurpht]

Personal experience after reading this: found my server running all 6 cores at the max, but not due to kernel trouble, but java instead. The machine always runs a couple of instances of java since it's being used by the kids as minecraftserver(s). Son reported that the issue is spoken of in the minecraft forums as well.
Since the machine had been up since I built it in October, I rebooted it. Which went nice. Minecraftservers back up again, all back to normal.

[jdmcdaniel3]

Personal experience after reading this: found my server running all 6 cores at the max, but not due to kernel trouble, but java instead. The machine always runs a couple of instances of java since it's being used by the kids as minecraftserver(s). Son reported that the issue is spoken of in the minecraft forums as well.
Since the machine had been up since I built it in October, I rebooted it. Which went nice. Minecraftservers back up again, all back to normal.

So I have a couple of computers running Linux all of the time, but they are not servers. I guess I did not notice anything odd the morning after and have updated the kernel to 3.5-rc5 since. I would say I would like to hear the fix is in the kernel before the next leap second gets added. Also, its not that good to hear that only Linux servers are increasing their power consumption in the megawatts over a leap second being added. Sounds like someone did not consider that issue in their testing of the Linux kernel. Of course, how might one go about adding in a leap second to test such a thing I wonder?

Thank You,

[hendersj]

On Tue, 03 Jul 2012 20:16:03 +0000, Knurpht wrote:

> Personal experience after reading this: found my server running all 6
> cores at the max, but not due to kernel trouble, but java instead. The
> machine always runs a couple of instances of java since it's being used
> by the kids as minecraftserver(s). Son reported that the issue is spoken
> of in the minecraft forums as well.
> Since the machine had been up since I built it in October, I rebooted
> it. Which went nice. Minecraftservers back up again, all back to normal.

Yep, saw the same thing here with the PS3 Media Server. But just my 12.1
system that runs it; the 11.4 system didn't have a problem at all.

Jim



--
Jim Henderson
openSUSE Forums Administrator
Forum Use Terms & Conditions at http://tinyurl.com/openSUSE-T-C

[robin_listas]

On 2012-07-03 22:26, jdmcdaniel3 wrote:

> So I have a couple of computers running Linux all of the time, but they
> are not servers. I guess I did not notice anything odd the morning
> after and have updated the kernel to 3.5-rc5 since.

I had two computers running at the time, and only one was affected. At first, I blamed that I was
running a customized kernel, till I read the post from please_try_again. Also, the non affected
computer had no entry in the log about the leap second. I now blame the ntp daemon: it was this app
who told the kernel to jump.


> Of course, how might one go about adding
> in a leap second to test such a thing I wonder?

Two ways: you change the time, or create a controlled network and you change the time references
appropriately. Google did test this, they thought something was wrong, and did not apply the leap
second at the legal point, but instead spread it over all the day, millisecond by millisecond.

Now the rest of world knows they did it right. Why was not their testing known and the errors
corrected in time?

--
Cheers / Saludos,

Carlos E. R.
(from 11.4 x86_64 "Celadon" (Minas Tirith))