In the past is in the future - boot failure due to HW clock in the past

openSUSE 11.3 32bit, fully updated, ASUSTeK P7H55-M PRO, i3 CPU 530 @ 2.93GHz, BIOS American Megatrends Inc. 1604 07/22/2010

I had to reboot a server to launch the new kernel. Before this the server was running for several months with no downtime. The following happened:

Some boot message lines reported that the file system was corrupted, because some blocks or tables were in the future (I can’t remember the exact message). Then it put me into rescue mode.

I could verify that the timestamps in the file system were correct, but the HW system clock was about 2 days in the past. I booted again with ^D and entered the BIOS, set the system time to approximately $NOW, then F10 and it booted ok.

The server is running ntp and showing the correct time, but - as it appears - this is not written to the hardware clock. I think this should happen during shutdown.

Any ideas how to fix this permanently?

vodoo wrote:
> The server is running ntp and showing the correct time, but - as it
> appears - this is not written to the hardware clock. I think this should
> happen during shutdown.
>
> Any ideas how to fix this permanently?

It should happen by default, as I guess you know. I would go into YaST
and set up Date/Time & NTP again, looking especially for options
connected with the hardware clock. Sorry, I don’t remember more detail.

Setting the hardware clock at shutdown should allways happen. The only thing to configure there is if it is set to UTC (normal for Linux systems) or to local time (when crossboot with non-Linux (read: MS Windows) systems is intended. But it is set.

When it then on next boot is not even approximately correct, the CMOS battery may be lousy. When that is the case, the coincidence with the new kernel is…, well a coincidence.

Thus when you know how to check the hardware clock in the BIOS, check it after shutdown and before reboot to see if it work as intended.

hcvv wrote:
> Setting the hardware clock at shutdown should allways happen. The only
> thing to configure there is if it is set to UTC (normal for Linux
> systems) or to local time (when crossboot with non-Linux (read: MS
> Windows) systems is intended. But it is set.

Hmm, ‘the only thing to configure’ sounded like a challenge, so I
checked. Actually, there’s also

/etc/sysconfig/clock:SYSTOHC=“yes”

There’s also USE_HWCLOCK but that only seems to affect a particular type
of IBM mainframe :confused:

Well, you are correct of course. But when we allways tell of all possibilities most people will go crazy. :frowning:

I had in mind the configuration options YaST is offering the user as the ones most normaly used. There are myriads of others in many areas where YaST helps to make a choice (think alone of the options the NTP deamon has).

On 2011-08-04 11:45, Dave Howorth wrote:

> Hmm, ‘the only thing to configure’ sounded like a challenge, so I
> checked. Actually, there’s also
>
> /etc/sysconfig/clock:SYSTOHC=“yes”

Yep.

The manual procedure involves setting the system clock somehow (ntp, for
instance), then copying that value to the hardware clock (man hwclock),
then erasing the /etc/adjtime file. That is quite important. It will be
correctly recreated later - if not erased, it will record that the hw clock
has to be shifted by two days… and repeat it.


Cheers / Saludos,

Carlos E. R.
(from 11.4 x86_64 “Celadon” at Telcontar)

On 2011-08-04 10:16, vodoo wrote:

> Some boot message lines reported that the file system was corrupted,
> because some blocks or tables were in the future (I can’t remember the
> exact message). Then it put me into rescue mode.

Ah, yes, a known “feature” of ext3.


Cheers / Saludos,

Carlos E. R.
(from 11.4 x86_64 “Celadon” at Telcontar)

The more I look into this problem the more it is a mystery. Configuration looks normal:

# sed -e "/^#/d" /etc/sysconfig/clock
HWCLOCK="-u"
SYSTOHC="yes"

TIMEZONE="Europe/Zurich"
DEFAULT_TIMEZONE="US/Eastern"

# cat /etc/adjtime 
-867.914027 1312464001 0.000000
1312464001
UTC

# dmesg | grep rtc
    1.030401] rtc_cmos 00:03: RTC can wake from S4
    1.030426] rtc_cmos 00:03: rtc core: registered rtc_cmos as rtc0
    1.030449] rtc0: alarms up to one month, y3k, 114 bytes nvram, hpet irqs
    1.165459] Using IPI No-Shortcut mode
    4.205298] rtc_cmos 00:03: setting system clock to 2011-08-03 18:23:44 UTC (1312395824)
# rcntp status
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 LOCAL(0)        .LOCL.          10 l   34   64  377    0.000    0.000   0.001
+magma.woody.ch  192.33.96.102    2 u  904 1024  377   29.625    1.994   0.893
*metasweb01.admi .HBGi.           1 u  111 1024  377   29.071    1.377   0.993
+swisstime.ee.et 129.132.2.22     2 u  628 1024  377   22.628    1.731   1.763

Checking for network time protocol daemon (NTPD):                    running
# hwclock -r -u
Thu 04 Aug 2011 03:47:54 PM CEST  -0.844296 seconds

# echo $(($(adjtimex --print | sed -e '/status:/!d; s/.*: //') & 64))
0

According to man hwclock this means:

Automatic Hardware Clock Synchronization By the Kernel
You should be aware of another way that the Hardware Clock is kept syn-
chronized in some systems. The Linux kernel has a mode wherein it
copies the System Time to the Hardware Clock every 11 minutes. This is
a good mode to use when you are using something sophisticated like ntp
to keep your System Time synchronized. (ntp is a way to keep your Sys-
tem Time synchronized either to a time server somewhere on the network
or to a radio clock hooked up to your system. See RFC 1305).

   This mode (we'll call it "11 minute mode") is off until something turns
   it on.  The ntp daemon xntpd is one thing that turns it  on.   You  can
   turn it off by running anything, including hwclock --hctosys, that sets
   the System Time the old fashioned way.

   To see if it is on or off, use the command adjtimex --print and look at
   the  value  of  "status".  If the "64" bit of this number (expressed in
   binary) equal to 0, 11 minute mode is on.  Otherwise, it is off.

   If your system runs with 11 minute mode on, don't use hwclock  --adjust
   or  hwclock  --hctosys.   You'll just make a mess.  It is acceptable to
   use a hwclock --hctosys at startup time to get a reasonable System Time
   until  your  system  is  able  to set the System Time from the external
   source and start 11 minute mode.

that 11 minute mode is on and the HW clock should be continually updated by the kernel.

However - and that’s really odd - when I try to re-configure ntp with yast (section network services) and I click on accept I get:

Error
Cannot update the dynamic configuration policy.

What’s going on here? Never seen this before.

Are you sure you are not suffering from CMOS battery failure? When I had a similar problem, I set the BIOS clock every time I switched on until I was able to replace the battery.

@vodoo
In the **pastis **in the future, we might be still looking for ice cubes. lol!

On 2011-08-04 16:26, vodoo wrote:
>
> The more I look into this problem the more it is a mystery.
> Configuration looks normal:

This problem is a non issue, I would not employ much time on it, unless it
happens on every boot.

If it happens, my educated guess is that the adjtime file is off.

> According to man hwclock this means:
>
>> AUTOMATIC HARDWARE CLOCK SYNCHRONIZATION BY THE KERNEL

What it does not say is if the adjtime file is adjusted as well. I don’t
have clear if in this mode SYSTOHC should be off.


Cheers / Saludos,

Carlos E. R.
(from 11.4 x86_64 “Celadon” at Telcontar)