What could cause my date to change to the year 2028?

[WillNorris]

Hello, I currently have 24 identical Fieldpoint banks running the same application. Each bank consists of a FP2015, a 422 relay module, a AI-110 analog input module, and two TC-120 TC modules. These modules have been installed for about two years now. These banks use the NI Time Service provided with the software.

Just in the past week however, I have had two banks that started incorrectly reporting the date stamp on a regular basis. Data is logged to file every 5-10 minutes (depending on the setup). Everything will be working fine at one scan, and then by the next scan the date will have changed to the year 2028 in the module. If I reset the FP2015 module (either through Fieldpoint explorer or by powering cycling), the next scan will again have the correct time and date. However, the errored scans with the 2028 date remain in the file, which messes up the software app when it is trying to calculate elapsed time. This has been happening now since last Friday (six consecutive days), and sometimes happens more than once per day. It also appears to happen at random times, that is, the two modules don't error at the same time as the other, and they error at different times from day to day.

One of these same modules did this once about two years ago when we were first bringing everything online. We just thought it was some install glitch because we rebooted the module and it never happened again (until now). We have upgraded from the A series firmware, which had a time service problem. I believe we currently have an E series firmware installed.

Any idea what could be causing it to jump to a date of 2028 (it is always the same date in 2028, starts with July 1 2028 I believe)? Why are only these two modules effected? Any help would be appreciated.

Thanks

[Matthew_Williams]

Has anything else in the environment changed, i.e. new timeserver pc or OS updates,
new network routing, etc.? Does the problem persist if the timeserver pc is
rebooted?

We saw some incorrect timestamps with some of the (now) older timeserver/fieldpoint
software so we built the time setting into our application and disabled the timeserver.
Don't know whether that is an option for you but it worked for us. Maybe try newer
software on the timeserver side?

See also:
http://forums.ni.com/ni/board/message?board.id=110&message.id=2603

Matt

[WillNorris]

We actually have two time server PC's (half the banks on each). Of the two error modules, one is on each PC. There have been no changes to the PC other than anti-virus updates passed down from IT. AS far as network routing, to the best of my knowledge it hasn't changed. The IP's, gateway, DNS are all the same settings. It's always possible for IT to change out a switch or server somewhere in the infrastructure. I asked and they had not done so since the time this problem started occurring.

If we reboot the timserver PC, the FP2015 still maintains the 2028 (at least for the several hours we let it run after the reboot). The only way we can find to get rid of the 2028 date is a reboot of the 2015 module. Within 24 hours, the problem returns.

I didn't know there was an upgrade for just the time server software. I can check into that. Thanks. However, I'm not sure how much that will help. After all, 22 modules are working perfectly with only two experiencing problems. I'm tending to lean towards the belief that there is just something wrong with the modules themselves, even though I'm not sure what that would be.

Tonight I think I will try letting one module run with the net disconnected. Both modules have errored again today and I need to go fix them. After doing so, I think I will pull the plug on one for 24 hours to see if it has a 2028 date on it when it is reconnected. This might give me more insight into whether the problem is on the PC side (the time server or interface app running there), on the real time app (running on the FP2015), or perhaps some kind of fault with the hardware.

[Aaron_G]

Will,

One thing that can improperly effect the timestamp, causing jumps in years, is extremely heavy/slow/failing network traffic. The time synchronization works somewhat like a PID loop and if the packet round-trip times between the FP and computer get too long it can cause the time jumps that you are seeing. Is there anything special/different about the network connection to the two modules that are having problems. If you can, try changing the ports in the router that they are wired to, and make sure there is no EM source that may be affecting the ethernet cable that may be corrupting data packets and resulting in retries.

Regards,
Aaron

[WillNorris]

I actually think you are on to something there. I climbed up in the cable trays yesterday and found that one of the ethernet cables had recently been cut (there has been construction in the factory for the past three months) and the piece of metal that cut it was still lodged in the wire. With the piece of metal still in there, it was shorting out 2-3 of the lines of ethernet, but the connection still appeared to function (data and control signals still pass over the cable correctly). However, I removed the piece of metal and the module lost communication with the PC. So perhaps the time signal was being corrupted by the short or taking too long because of dropped packets.

This leads me to believe there must be something wrong with the other cable, since it has the exact same problems. I looked the length of it over and can't find anything obvious like a cut though. I've put in a request for IT to check the termination and port on the switch (even though we purchased new Cisco switches to upgrade to 100bT last November) for something that could be wrong there. In the meantime, I'm going to try to swap cables with another module to see if the problem follows the cable.