For this answer, I'll assume that there may be several elements working hard to set your time straight. Since I don't really want to wild-guess about which one is working against you, I'll try and give you an answer which should help you find it yourself instead.
On a UNIX system, the clock can typically be set using the stime
system call. As things evolved, it also became possible to set the clocks more accurately using the clock_settime
call instead. You might also come accross settimeofday
. When running date --set
on a CentOS machine, strace
revealed that it used clock_settime
.
Knowing this, a solution would be to trace these system calls. Good thing is, Linux has a mechanism for that: debugfs. On my system, calling mount
, I can see that this is available at /sys/kernel/debug
:
$ mount
none on /sys/kernel/debug type debugfs (rw)
...
However, on some systems (including RedHat and probably CentOS), it isn't mounted at boot time. You'll therefore need to run...
# mount -t debugfs nodev /sys/kernel/debug
Also note that if you were in that directory before mounting, you might have to go out and back in before files start to appear in it.
Now we're ready to go. Let's enable the trace for our systems calls. I'm tracing all of them because I don't really want to check which one is really being used. Tracing for system calls can be set in /sys/kernel/debug/tracing/events/syscalls
. In this directory, you should find...
sys_enter_stime
sys_enter_clock_settime
sys_enter_settimeofday
... depending on what's available on your system.
These correspond to the events of entering our system calls, which is what we want to trace (you'll also find sys_exit_*
directories). Within each directory, you'll find a file named enable
, the contents of which should appear to be 0. To trace these calls, set that to 1 instead:
# echo 1 > /sys/kernel/debug/tracing/events/syscalls/sys_enter_stime/enable
# echo 1 > /sys/kernel/debug/tracing/events/syscalls/sys_enter_clock_settime/enable
# echo 1 > /sys/kernel/debug/tracing/events/syscalls/sys_enter_settimeofday/enable
Now that we've set up our trap, just wait until something sets your time to its correct value. Once it has happened, run for the trace logs at...
# cat /sys/kernel/debug/tracing/trace
Now, unless something wrong occured, you should see one of the following lines:
stime-xxxxx [xxx] .... x.x: sys_stime(...)
clock_settime-xxxxx [xxx] .... x.x: sys_clock_settime(...)
settimeofday-xxxxx [xxx] .... x.x: sys_settimeofday(...)
The number right after stime-
(or another call's name) is the PID of the process which made the system call. Now go get it:
# ps -fp xxxxx
UID PID PPID C STIME TTY TIME CMD
root XXXXX XXXXX 0 hh:mm ? hh:mm:ss time_warrior
You should now have everything you need to make sure your system stops getting the time right. The simplest thing would probably be to kill the process, and make sure it isn't spawned at boot time ; of course, you'll have to make sure it doesn't serve a more important purpose before doing so : you don't want to completely crash your system...
Also remember to disable the trace when you're done by writing 0 to the files we edited earlier. A shortcut could be:
# echo 0 > /sys/kernel/debug/tracing/events/syscalls/enable
(this file acts as a master switch for all others ; it allows you to switch all system calls tracing off)
Note: as Mark Plotnick said in a comment systemtap
could be a slightly easier way to achieve similar results. I'll let him write a stap
answer if he feels like it, since I'm not fluent with stap
scripts at all.
Best Answer
The relevant aspects are the kernel versions and these lines from early in the boot process:
YMMV and you may not be using TSC or PIT
AFAIK this is a bug that's caused by the clock of at least one of your CPUs being out of sync, in your case probably running too fast.
It should be easy to confirm by running this:
which will run
date
against each cpu (assuming you have up to 8 cores/threads). If my guess is correct then one of your CPUs will consistently have the wrong time.If that's the case then you should first try upgrading the kernel and if that doesn't work, fiddle with the clocksource boot parameter (assuming it's
x86-64
):See also the output of this: