PDA

View Full Version : Fedora KVM, clock drift, time and older x86_64 kernels



RobertThiem
19th January 2009, 11:18 PM
If I wasn't already bald I'd have lost a lot of hair over timekeeping in the last week.
For anyone else out there as clueless as me about time:

I'm new to KVM (having skipped FC7, 8, 9) and in my old Xen environment the Xen hypervisor handled all the stuff for me.

In a KVM environment the guest OS is trying to talk directly to hardware to work out how time is going in the same was as a physical guest.

Older kernels (< 2.6.18 on 32bit, and < 2.6.21 on x86_64) use old fashioned ticking timers, and if ticks or timer interrupts are lost (as they sometimes are in a virtual environment) the kernel starts compensating for it and may or may not be able to get the right time. Also, the more ticks the guest is trying to use for timekeeping the more it's context switching to handle them.

My particular grief was with CentOS 5.2, which uses kernel 2.6.18 (and being on x86_64 uses the old timer models). It runs (I believe) on a default 1000Hz timer. I was getting clock drift of somewhere between 7-120 seconds per hour (or more depending on which wrong kernel options I was giving). Worse, it was positive drift, so resetting it to the right time or using NTPD would cause the clock to more backward in time every now and again, causing dovecot to kill itself and "bad stuff".

Newer kernels can use a clocksource= option at boot that tell them what timer to use. Most of the new timers are "tickless" , and my Fedora 10 x86_64 guest seems to keep time flawlessly (within 100 micro seconds of real time) straight out of the box from what I can see.

The kernel options you use to control how the timers are used are different between x86_64 and 32bit, so (my problem) the clock= and clocksource= kernel options I was finding on random posts did no good for me. For older kernels clock=pmtmr (32 bit) or divider=10 (x86_64) may give you reasonable results.

I ended up adding on "divider=10 notsc" (cuts down old fashioned interrupt rate to 100 Hz instead of 100 Hz on CentOS + don't use TSC) per the VMWare article below. This cut my clock drift down to 165 PPM, which NTPD was quite happy to compensate for when I pointed it at the host's NTP server. Cutting back the interrupt rate also seems to have decreased the idle CPU usage down to 0.5% (it was about 1.5% before, I think)

Good starting points are probably:
http://oss.intel.com/pdf/maximum_tickless.pdf - a discussion of the tickless kernel
http://www.vmware.com/pdf/vmware_timekeeping.pdf - VMWare discussion of timekeeping in general with a section on Linux
VMWare Knowledge Base article 1006427 (kb.vmware.com) - lists recommended kernel options for a bunch of Linuxes