Problem with guest additions services in openSUSE 13.1

English Virtualization
21

I think I have this running as well as I can reasonably expect now and I can run commands on the guest from a shell script on the host.

I’ve used the “Before=” although it doesn’t really seem necessary; systemd seems to start the service as early as it can, which is before syslog.service in any case (I only see the vboxservice startup messages in the system log if I change the “Before=” to “After=syslog.service”).

Unfortunately, the actual time change happens a bit later, so I still need to find out why the initial time is set wrongly. Possibly it’s because I’ve built the VM from a script instead of from the GUI (so maybe there’s an extra flag or two I need to set).

Anyway, that’s a separate issue. Time to get on with the rest of the project…

Thanks again for your help.

22

On 2014-07-01 15:36, xanti wrote:

> Unfortunately, the actual time change happens a bit later, so I still
> need to find out why the initial time is set wrongly.

Find out if the emulated bios clock (ie, the bios config screen, before
booting the guest) is correct. If it is correct, then, once booted, do,
after the clock is correct:


rm /etc/adjtime
hwclock --systohc --utc
hwclock --debug
date

The last two commands should show the correct time and date.

Notice that I’m unsure whether you need “–utc” or “–localtime”. On
vmware, it is the later. It depends on what the emulated bios shows on boot.

Maybe you should also edit “/etc/sysconfig/clock” so that “SYSTOHC” and
USE_HWCLOCK are set to “no”.

–
Cheers / Saludos,

Carlos E. R.
(from 13.1 x86_64 “Bottle” at Telcontar)

23

VirtualBox doesn’t have a BIOS config screen.

But there’s a checkbox in the VM’s settings (“System”) that is labelled “Hardware Clock in UTC Time”:
http://wstaw.org/m/2014/07/01/vbox.png

This should match the hardware clock settings in YaST->System->Date and Time (in the guest), otherwise the clock will be of some hours depending on your time zone (i.e. the offset to UTC).
Preferably both are set to UTC.

I don’t think SYSTOHC and USE_HWCLOCK matter in any way (in the guest), as I don’t think VirtualBox writes anything back to any (real or pseudo) hardware clock. And I don’t think “hwclock --systohc” would have any effect for the same reasons.

24

On 2014-07-01 18:06, wolfi323 wrote:
>
> robin_listas;2651904 Wrote:
>> On 2014-07-01 15:36, xanti wrote:
>>
>>> Unfortunately, the actual time change happens a bit later, so I still
>>> need to find out why the initial time is set wrongly.
>>
>> Find out if the emulated bios clock (ie, the bios config screen, before
>> booting the guest) is correct.
> VirtualBox doesn’t have a BIOS config screen.

Ah! Curious.

> I don’t think SYSTOHC and USE_HWCLOCK matter in any way (in the guest),
> as I don’t think VirtualBox writes anything back to any (real or pseudo)
> hardware clock. And I don’t think “hwclock --systohc” would have any
> effect for the same reasons.

It does, in a way.

The guest operating system thinks that it is sitting on a real
machine, and tries to write to the emulated bios cmos clock (ok, there
is is no bios config screen at boot, but nevertheless, it’s got to have
a bios and emulated hardware). VB may actually not write anything, but
the guest /thinks/ it did.

Writing this data there is of course useless. On next “emulated power
up” sequence, the virtual machine (virtual box) will read the time from
the host, and give it to the guest when it asks for the cmos clock, and
hwclock will try to adjust for drift - which does not exist and can
cause weird effects, like the machine waking up with the wrong time,
till the guest tools kick in and correct the time, but not inm,immediately.

–
Cheers / Saludos,

Carlos E. R.
(from 13.1 x86_64 “Bottle” at Telcontar)

25

Ah, ok.
I didn’t think of this.

Might be the case, yes.

26

On 2014-07-02 12:16, wolfi323 wrote:

>> It does, in a way.
>> …
>> and hwclock will try to adjust for drift
>> …
>
> Ah, ok.
> I didn’t think of this.
>
> Might be the case, yes.

As a matter of fact, I though about it just yesterday :slight_smile:

And now I have to consider review all of my virtual machines for those
settings.

–
Cheers / Saludos,

Carlos E. R.
(from 13.1 x86_64 “Bottle” at Telcontar)

27

OTOH, if the time is off by hours, this can’t really be the reason I think.
From “man adjtime”:

      adjtime()  is intended to be used to make small adjustments to the sys-       tem time.  Most systems impose a limit on the adjustment  that  can  be

       specified  in  delta.   In the glibc implementation, delta must be less

       than or equal to (INT_MAX / 1000000 - 2) and greater than or  equal  to

       (INT_MIN / 1000000 + 2) (respectively 2145 and -2145 seconds on i386).

If I understand this correctly, the maximum adjustment value is +/- 2145 seconds, which is of course less than an hour (or ~71.5 minutes if you take the whole interval).

And before you point out that the man page speaks only of i386: :wink:
I just checked, (INT_MAX / 1000000 - 2) is indeed 2145 on my 64bit system as well.

No idea if that applies to the automatic drift adjustment by the system as well, but I would think so.

Worth to check anyway, but I think a wrong (mismatching) VirtualBox UTC setting is the more likely reason in this case, especially because xanti wrote this:

28

Sorry, the formatting was completely off. The forum editor messed up my copy/paste from Konsole.
Here it is again for better readability:

       adjtime()  is intended to be used to make small adjustments to the sys-
       tem time.  Most systems impose a limit on the adjustment  that  can  be
       specified  in  delta.   In the glibc implementation, delta must be less
       than or equal to (INT_MAX / 1000000 - 2) and greater than or  equal  to
       (INT_MIN / 1000000 + 2) (respectively 2145 and -2145 seconds on i386).

But (as I read the complete man page now myself), apparently that drift adjustment would be done continuously over time, not by setting an explicit time.
So I don’t think this could cause the clock to be off by hours at system start at all.

       The adjustment that adjtime() makes to the clock is carried out in such
       a manner that the clock is always monotonically increasing.  Using adj-
       time()  to adjust the time prevents the problems that can be caused for
       certain applications (e.g., make(1)) by  abrupt  positive  or  negative
       jumps in the system time.

Although, as I said, I don’t know whether this applies to the automatic drift adjustment as well.

29

On 2014-07-02 14:16, wolfi323 wrote:
>
> wolfi323;2652059 Wrote:
>> From “man adjtime”:
> Sorry, the formatting was completely off. The forum editor messed up my
> copy/paste from Konsole.
> Here it is again for better readability:
>
> Code:
> --------------------
> adjtime() is intended to be used to make small adjustments to the sys-
> tem time. Most systems impose a limit on the adjustment that can be
> specified in delta. In the glibc implementation, delta must be less
> than or equal to (INT_MAX / 1000000 - 2) and greater than or equal to
> (INT_MIN / 1000000 + 2) (respectively 2145 and -2145 seconds on i386).
>
> --------------------
>
>
> But (as I read the complete man page now myself), apparently that drift
> adjustment would be done continuously over time, not by setting an
> explicit time.
> So I don’t think this could cause the clock to be off by hours at system
> start at all.

I’d have to read the document myself again, as last time I studied this
in some detail was years ago. :slight_smile:

So, from memory, there are two types of adjustment.

One, is a small correction to the cpu clock speed. It is to discipline
the cpu clock in a similar manner to what ntp does, but applied from a
static calculation, not by comparing continuously to external sources.

Another, is a one time correction, applied only at boot, to correct the
accumulated and estimated drift on the CMOS clock, while the computer is
powered off. It does not adjust the clock speed at all.

It is the second type of adjustment that concerns us now, and this one
in particular I know for a fact that has caused machines to be off by
hours before, unless they now decided to apply a limit to the
adjustment, which would be sensible.

This issue is the typical cause for machines to have the time off by
hours on boot, specially after they changed the clock manually. The cure
has been, for years, to remove and recreate the /etc/adjtime file.

–
Cheers / Saludos,

Carlos E. R.
(from 13.1 x86_64 “Bottle” at Telcontar)

30

I agree that if you don’t set UTC for both the VirtualBox VM and the guest OS clock, then you’ll get errors. However, that’s one thing I definitely did correctly - so that’s not the problem.

Since getting the VBoxService time sync running, I haven’t seen the large offsets into the future I was originally getting. So I suspect this may have been an initial error offset that never got corrected for some reason (maybe because I kept rolling the VM back or because something wasn’t shutting down cleanly and writing the required data back). The guest openSUSE was installed pretty much with the default settings. That includes New York as a time zone, so perhaps that offset arose at some point because the host (and the guest desktop) is on London time. I don’t know how that could happen, but it’s the right sort of offset.

TBH, I’ve yet to see any ill effects from not having the time initially set correctly, but I guess it’s worth trying to get it working properly. The VBoxService daemon usually (although not always) seems to sync the time before syslog starts, so to reliably see any time offset at startup I need to disable that service.

With it disabled, and the guest time set correctly, what I’ve now done is suspend the VM (save its state) for 5 minutes and then re-start it. At this point, the guest time becomes 5 minutes slow.

So it looks as if either VirtualBox isn’t transferring the host time to the guest clock when the VM starts, or the guest is ignoring the clock time, or correcting it wrongly. To decide which, I set the guest clock to the guest time using “hwclock --systohc --utc”, suspended the VM for 5 minutes, re-started it and checked the guest clock with “hwclock --debug”.

After this, the guest clock was 5 minutes slow. So it appears to be recording execution time of the VM and not host time (and therefore it never gets set to host time, so any initial error never gets corrected). There is an option for it to do this, but it shouldn’t be the default. So it looks like I need to explore a bit more to see what’s set wrongly.

31

So that is your problem?

Of course, while the VM is suspended, the (virtual) clock does not continue to run.
If you restore the VM, the clock continues at the point where it has been suspended, so has a time in the past.

Linux will only synchronize to the hardware clock at boot anyway, so it doesn’t matter whether the virtual hardware clock is set correctly afterwards or not. So even if VirtualBox would set the hw clock correctly when resuming (I don’t know if it does), it would have no effect for the running guest system (which doesn’t even know/recognize that it has been suspended).

There’s no way around that I suppose, unless using ntp or time synchronization.
Actually that’s the only reason why that synchronization would be needed I think, as otherwise the clocks should not get out of sync anyway.

32

As I mentioned, it’s not actually that much of a problem, so long as the VBoxService daemon is running. However, it doesn’t seem to be quite the behaviour you’d expect from reading the VirtualBox documentation:

http://www.virtualbox.org/manual/ch09.html#changetscmode

which seems to say that by default the guest clock should reflect monotonic host time.

In fact, the guest clock is behaving like the “special TSC handling mode” instead of the default mode.

33

The TSC counts the processor clock cycles since the last reset.
It’s for getting CPU timing information, not the system time.

I have no idea whether the TSC is used at all in regards to the system time, but it definitely is not the only source.
It seems to be unreliable especially on multi-cpu systems and with powermanagement features and is not supported by all processors anyway.

At least that’s what I understand after looking at the Wikipedia article:

And according to this: http://www.tldp.org/HOWTO/Clock-2.html

The other is the “system clock” (sometimes called the “kernel clock” or “software clock”) which is a software counter based on the timer interrupt.

That doesn’t sound like the TSC is influencing the system time either.

34

You’re right.
Apparently the clock is adjusted explicitely and immediately on boot-up:

Since the RTC is only used when the system is not running, the correction factor is applied when the clock is read at boot time, using clock(8) or hwclock(8).

When the system is running, only the “speed” of the system clock is adjusted:

The system clock is corrected by adjusting the rate at which the system time is advanced with each timer interrupt, using adjtimex(8).

Of course VBoxService does not have to use that adjtime() function, but can set the system time directly to any value.
A maybe interesting note from that text as well:

A crude alternative to adjtimex(8) is to have chron run clock(8) or hwclock(8) periodically to sync the system time to the (corrected) RTC. This was recommended in the clock(8)man page, and it works if you do it often enough that you don’t cause large “jumps” in the system time, but adjtimex(8) is a more elegant solution. Some applications may complain if the time jumps backwards.

Btw, you can print a lot of information about the kernel’s clock (like precision, raw time, current clock rate) with:

adjtimex --print

See also “man adjtimex”.

35

Hmm, there’s some interesting stuff there that I never knew existed, especially the bits on calibrating time sources.

Certainly, if the guest system time isn’t derived directly from the TSC and it only checks the RTC at boot time, then it’s going to fall behind host time whenever you suspend the VM. In that case it doesn’t really matter whether the guest RTC is correct or not, except at boot time. Having just done a quick check, the guest RTC does appear to get synced to the host at boot time, but not subsequently. So I guess it’s doing as much as it needs to support a Linux guest.

Possibly I was confused by the first sentence in that VirtuaBox documentation, where it says:

By default, VirtualBox keeps all sources of time visible to the guest synchronized to a single time source, the monotonic host time.

because I took that to include the guest RTC. Perhaps that statement is just wrong, because the guest RTC doesn’t seem to track host time at all.

I must say I’m a little surprised that this isn’t a more common issue. I sort of assumed that people suspended their VMs from time to time. Isn’t that what they’re for? If you do that and your system time gets a long way off then things stop working (internet things especially). It looks like that will happen unless you have VBoxService running, but it doesn’t run by default. So I’d expect a lot of problems, but Google turns up very little on this, so obviously I’m in the minority.

36

On 2014-07-02 22:06, wolfi323 wrote:
>
> robin_listas;2652110 Wrote:

…

> When the system is running, only the “speed” of the system clock is
> adjusted:
>> The system clock is corrected by adjusting the rate at which the system
>> time is advanced with each timer interrupt, using adjtimex(8).

Ah, yes, adjtimex. I had forgotten that one.
You see, now we have internet, we do not need those tricks :wink:

> A maybe interesting note from that text as well:
>> A crude alternative to adjtimex(8) is to have chron run clock(8) or
>> hwclock(8) periodically to sync the system time to the (corrected) RTC.

Ah, yes! I actually used this on ocassion. IIRC, some kernel had a bad
clock.

And this could be again of use in a virtual machine, by the way: if you
don’t run guest tools, this should work (the RTC is emulated by the
virtualization software, reading from the host, I understand)).

>> This was recommended in the clock(8)man page, and it works if you do it
>> often enough that you don’t cause large “jumps” in the system time, but
>> adjtimex(8) is a more elegant solution. Some applications may complain
>> if the time jumps backwards.

Indeed they complain. The experiment is instructive :slight_smile:

> Btw, you can print a lot of information about the kernel’s clock (like
> precision, raw time, current clock rate) with:
>
> Code:
> --------------------
> adjtimex --print
> --------------------

Right…

–
Cheers / Saludos,

Carlos E. R.
(from 13.1 x86_64 “Bottle” at Telcontar)

37

On 2014-07-02 23:16, xanti wrote:

> Possibly I was confused by the first sentence in that VirtuaBox
> documentation, where it says:
>
>> By default, VirtualBox keeps all sources of time visible to the
>> guest synchronized to a single time source, the monotonic host time.
>
> because I took that to include the guest RTC. Perhaps that statement is
> just wrong, because the guest RTC doesn’t seem to track host time at
> all.

No? I’m surprised. :-o

I’ll have to try.

> I must say I’m a little surprised that this isn’t a more common issue. I
> sort of assumed that people suspended their VMs from time to time.

I do, a lot, but I use vmware. Often the clock maybe off for some long
seconds, even a minute, after thawing (and not all my virtual machines
exhibit it). It is not an issue for me, though. Or not big enough :wink:

–
Cheers / Saludos,

Carlos E. R.
(from 13.1 x86_64 “Bottle” at Telcontar)

38

On 2014-07-02 20:06, wolfi323 wrote:

> Of course, while the VM is suspended, the (virtual) clock does not
> continue to run.
> If you restore the VM, the clock continues at the point where it has
> been suspended, so has a time in the past.

Right.

> Linux will only synchronize to the hardware clock at boot anyway,

Er… no, it also does on restore from hibernation.

However, I see an issue: if you request the guest operating system to
hibernate, it will know on restore that it has to jump the clock.
However, if you tell virtualbox to hibernate the machine, the guest
knows nothing. At least it is so with vmware.

(In the first method, the image is stored in the swap partition of the
emulated hard disk. In the second method, the image is a memory map file
in the vmware image directory, outside. And by the way, the first method
crashes).

> so it
> doesn’t matter whether the virtual hardware clock is set correctly
> afterwards or not. So even if VirtualBox would set the hw clock
> correctly when resuming (I don’t know if it does), it would have no
> effect for the running guest system (which doesn’t even know/recognize
> that it has been suspended).

Maybe… with the traditional cpu time keeping method, you are
absolutely correct. With some of the new timers, I’m not sure, they
could be rewritten. It could crash the guest, though.

> There’s no way around that I suppose, unless using ntp or time
> synchronization.

Guest tools do that, but they take some time to kick in. Ntp is no no,
but you can call a one-shot time adjustment instead.

> Actually that’s the only reason why that synchronization would be needed
> I think, as otherwise the clocks should not get out of sync anyway.

But not attempting to adjust the timer speed on the guest. The older
method, via cron calls instead.

–
Cheers / Saludos,

Carlos E. R.
(from 13.1 x86_64 “Bottle” at Telcontar)

39

Yes, f.e. there’s /usr/lib/pm-utils/sleep.d/90clock for that, but it says in the first line:

# Synchronize system time with hardware time.
# Modern kernels handle this correctly so we skip this hook by default.

So this should be done by the kernel itself nowadays I suppose, when resuming.

But that’s not what we are talking about now.
It’s about VirtualBox’s “Save the machine state” feature.
The guest operating system isn’t suspended at all in this case.

VirtualBox just saves the current state and aborts the VM.
When you “resume”, the saved state is loaded and the VM continues as if nothing happened.

However, I see an issue: if you request the guest operating system to
hibernate, it will know on restore that it has to jump the clock.
However, if you tell virtualbox to hibernate the machine, the guest
knows nothing. At least it is so with vmware.

Right, that’s the point.
It’s the same with VirtualBox of course.

(In the first method, the image is stored in the swap partition of the
emulated hard disk. In the second method, the image is a memory map file
in the vmware image directory, outside. And by the way, the first method
crashes).

Yeah. It crashes here with VirtualBox as well (both hibernate and suspend).

> Actually that’s the only reason why that synchronization would be needed
> I think, as otherwise the clocks should not get out of sync anyway.

But not attempting to adjust the timer speed on the guest. The older
method, via cron calls instead.

Or the guest tools synchronization. That was what I meant here.

40

On 2014-07-03 13:06, wolfi323 wrote:

> So this should be done by the kernel itself nowadays I suppose, when
> resuming.

Right.

> VirtualBox just saves the current state and aborts the VM.
> When you “resume”, the saved state is loaded and the VM continues as if
> nothing happened.

Exactly.
The guest doesn’t know that it has to re-adjust the clock.

>> (In the first method, the image is stored in the swap partition of the
>> emulated hard disk. In the second method, the image is a memory map file
>> in the vmware image directory, outside. And by the way, the first method
>> crashes).
> Yeah. It crashes here with VirtualBox as well (both hibernate and
> suspend).

Ah, good to know.
I do have bugzillas on the issue, using vmware.

>>> Actually that’s the only reason why that synchronization would be needed
>>> I think, as otherwise the clocks should not get out of sync anyway.
>>
>> But not attempting to adjust the timer speed on the guest. The older
>> method, via cron calls instead.
>>
> Or the guest tools synchronization. That was what I meant here.

Yes, of course.
If you have them, some guests may not have them.

Virtualbox, I know, does not have guest tools for all operating systems.
Vmware has more versions.

–
Cheers / Saludos,

Carlos E. R.
(from 13.1 x86_64 “Bottle” at Telcontar)