I'm experiencing an issue with the boot up timestamps of cRIO 9047s. They are incorrect by approximately 8 hours. This causes the system log to be real confusing as entries have bogus timestamps until the time sync daemon initializes about 15 seconds after boot.
Here is a snippit from the log:
2019-01-08T07:39:31.177+00:00 daq2a kernel: [ 0.000000] ACPI: LPIT 0x0000000079ECC000 00005C (v01 INTEL EDK2 00000003 BRXT 0100000D)
.... lots of lines
2019-01-08T15:37:35.027+00:00 daq2a kernel: [ 15.168186] random: crng init done
I also see these lines in the log, I'm not sure if this is related:
2019-01-08T07:39:34.882+00:00 daq2a nitimestampd: Initializing the timestamp daemon.
2019-01-08T07:39:34.882+00:00 daq2a nitimestampd: Ironclad clock interface loaded successfully
2019-01-08T07:39:34.882+00:00 daq2a nitimestampd: Starting the timestamp daemon.
... lots of lines
2019-01-08T15:36:51.936+00:00 daq2a nitimestampd: Stopping the timestamp daemon.
2019-01-08T15:36:51.937+00:00 daq2a nitimestampd: Cleaning up the timestamp daemon.
When we reboot the 9047s, they are actively being time slaved to our PTP master, so I am confused why the time is not being saved to their on board clock and persisting upon reboot.
This is causing us data analysis issues so any way to resolve this would be greatly appreciated.
Solved! Go to Solution.
I took nitimestampd out of init.d and verified it wasnt running at boot up, and the 8 hour correction still happens. So it doesn't seem to have anything to do with nitimestampd.
It looks like its booting with the local timestamp (PST) and then being corrected by NI-TimeSync to UTC, as that is 8 hours different. I want UTC all the time.
Logging in and checking the time zone (well after correction has happened) its in UTC.
admin@daq2a:~# readlink /etc/localtime
admin@daq2a:~# readlink /etc/natinst/share/localtime
Creating a script to check the timezone at boot, before correction, shows the same thing. So that didnt change.
I also just noticed this line in the system log. Perhaps that has something to do with it?
2019-01-08T18:03:14.760+00:00 daq2a nisds: detected two local timescales, creating NOP correlator: localhost/nisdwebs/timescales/timeSync/IEEE-1588-2008_2/20:67:7C:FF:FE:E2:8E:B4, /localhost/nisds/timescales/system
This behavior is expected, and is due to the RT clock's role in the bootup process. The RT clock is the only clock on the target that's not affected by the time keeping processes, and is also the clock used by the OS to set the time at boot-up when time keeping processes are unavailable. You should be able to manually set the RT clock so it stays close to the correct time in your system using the System Configuration API:
If you execute this VI in your closing operations, it will set the RT clock to the specified time and should retain a value close to that time (with some drift) next time the system boots up. The RT clock is battery-powered so it should keep the time even if the target is off.
We run ptp4l and phc2sys, similar to you guys do on cRIOs, on some HP Gen10 servers and do not see this behavior. When we reboot them, they come up with boot timestamps that are still in UTC. As far as I know, we are not doing anything special to set the RT clock on those servers. Why do we need to take an extra step on the cRIOs? I'll talk to someone on our kernel team to see if I'm missing something.
Also note we are not using LabVIEW so if you have any references for that suggestion that are not LabVIEW, that would be preferable.
Our kernel team got back to me and says the issue is you're storing your hardware time in local time rather than UTC, and it can be corrected with a call to hwclock. Reference: https://www.tldp.org/HOWTO/Clock-2.html
admin@daq2a:~# hwclock; date
Wed Jan 9 01:28:31 UTC 2019
admin@daq2a:~# hwclock --systohc --utc
admin@daq2a:~# hwclock; date
Wed Jan 9 01:30:51 UTC 2019
That seemed to prove the problem and correct it.
I then rebooted and indeed the problem was resolved. All bootup timestamps were consistent before/after boot and didnt make an 8 hour jump when sync happened.
I can setup our scripts to make this call on all our cRIOs during first-time setup, but I'm concerned there is an NI process running somewhere that is saving the time to the hwclock in local time, and thus my efforts will be moot. Is that the case?
Our software shouldn't be setting the real-time clock so your solution shouldn't have any issues. I want emphasize that when you set the RT clock, it's not continuously synchronized with anything and may drift from the system time until it's set again. You should call the systohc command during your shut down operations to keep the clock close to the desired system time when the system reboots.
I dug into this a bit more to verify, considering my system still seems synchronized quite well between hwclock and os clock.
You're running phc2sys:
admin@daq1a:~# ps aux | grep phc
2037 admin 0:04 phc2sys -O 0 -l 4 -c CLOCK_REALTIME -F 0.000001 -s eth0
Which is part of the linux ptp project http://linuxptp.sourceforge.net/
Which made me wonder if you're turning on the "11 minute mode" mentioned at the bottom of the hwclock man page: https://linux.die.net/man/8/hwclock
Automatic Hardware Clock Synchronization By the Kernel
You should be aware of another way that the Hardware Clock is kept synchronized in some systems. The Linux kernel has a mode wherein it copies the System Time to the Hardware Clock every 11 minutes. This is a good mode to use when you are using something sophisticated like ntp to keep your System Time synchronized. (ntp is a way to keep your System Time synchronized either to a time server somewhere on the network or to a radio clock hooked up to your system. See RFC 1305).
This mode (we'll call it "11 minute mode") is off until something turns it on. The ntp daemon xntpd is one thing that turns it on. You can turn it off by running anything, including hwclock --hctosys, that sets the System Time the old fashioned way.
To see if it is on or off, use the command adjtimex --print and look at the value of "status". If the "64" bit of this number (expressed in binary) equal to 0, 11 minute mode is on. Otherwise, it is off.
If your system runs with 11 minute mode on, don't use hwclock --adjust or hwclock --hctosys. You'll just make a mess. It is acceptable to use a hwclock --hctosys at startup time to get a reasonable System Time until your system is able to set the System Time from the external source and start 11 minute mode.
Which I verified with:
-o offset: 0 us
-f freq.adjust: 1313569 (65536 = 1ppm)
status: 8256 (UNSYNC)
-p timeconstant: 2
precision: 1 us
-t tick: 10000 us
return value: 5 (clock not synchronized)
(note your kernel does not come with adjtimex installed so I had to grab that)
Bit 64 in status is 1, so the 11 minute mode is not on.
So I'm still curious why 11 minute mode isn't on. Seems like phc2sys should be turning it on, and my RTC after nearly 2 days is still well synchronized. Did you fork phc2sys and the sources I'm linking are not what I should be looking at? Is there something else going on?
I don't want to just set this in a shutdown script like /etc/rc0.d/K07hwclock.sh does on ubuntu systems, because this is a RTOS on an embedded controller. We should be able to power it off whenever we want via power supply.
I can make some init script to call hwclock in the background every so often, but I'd like to know whats going on here.
You should be able to set the RT clock periodically without issues. You should keep a wide cadence (11 minutes is a good amount of time) between calls to set the clock. It's worth noting that the RT clock is used to set the system time when a device boots. If that device is the master to other devices, it will propagate the RT time to other targets. If the system time is somehow set wrong, it becomes possible for the RT clock time to be set wrong which would persist after reboots.
Regarding phc2sys, I'm looking into how we've configured it. I'll post again when I have more information.
To get things happy, I went ahead and updated our provisioning scripts to drop a script named set_hwclock in /etc/cron.hourly/ which simply contains
hwclock --systohc --utc
and update /etc/crontab to uncomment the hourly task.
Yeah I'd be interested to know why your phc2sys doesnt set the 11 minute mode flag. We also run phc2sys on many things, and it always sets that flag.
We have made a few modifications to phc2sys for use on our targets, but the code isn't open source and I can't share details. I hope this isn't an inconvenience for you.