Ethernet timer blocks serial activity in real-time PXIe controller

[glenw]

Something to do with Ethernet appears to be blocking serial activity ~10 minutes after startup and again exactly 10 minutes later, then no more. A thread named "ETS TCP/IP Timer Thread" precedes a thread named "StartAddr=0x01CA645E", during which timed loops containing "VISA Write" and "VISA Bytes at Serial Port" don't execute. Other code appears to execute normally.

This is a follow-on from an earlier post but I have ended that thread and referred to this.

My current theory is that something to do with Ethernet is doing something at those times, which may share a resource with the serial writes/reads.

The effect on our system is that the device receiving the serial messages from the application gets errors indicating the messages are the wrong length, presumably because they're broken into two parts, and the application gets errors receiving messages from the outside though I haven't looked into those details.

A project containing the serial write part of the application is attached. debug_app.vi is the top level. write_loop.vi transmits an 80 byte message every 2ms. watch_loop.vi saves the trace when the write does not complete on time.

The attached real-time trace log show an instance of the serial write getting interrupted. An instance of a normal serial write runs from time 106.58ms to time 108.01ms, consisting of the following threads:
"0: LV TL[95]...", the timed loop in write_loop.vi
"10: LabVIEW [Other1: Time Critical #0]", SerialWrite.vi
"3: Ni-Serial Deferred Interrupt..."
time passes
"3: Ni-Serial Deferred Interrupt..."
"10: LabVIEW [Other1: Time Critical #0]", SerialWrite.vi
"0: LV TL[95]...", the timed loop in write_loop.vi

An interrupted write follows, starting at 108.58ms, but at 109.98ms after the final interrupt, it does not continue with thread 10 and thread 0, presumably because it's being blocked by thread "18: StartAddr=0x01CA645E".

write_loop.vi also saves a trace but occurs later and and that trace is not useful. Before I added watch_loop.vi, the serial write was being interrupted by a trace named "Terminated (Name Unknown)", but with watch_loop.vi I can see the name is "StartAddr=0x01CA645E". If I had a map file like is produced by c compilers/linkers I might be able to interpret that. I've tried: changing priority; changing execution system; eliminating the final VISA Flush I/O Buffer. One of my constraints is that the serial receivers look for gaps to delimit messages instead of frame words. It seems that it would still drop messages even if delimited by frame words.

The system is a PXIe-1071 chassis with a PXIe-8115 controller, a PXI-8433/4 serial module and a DAQ module (which is not operated in the attached project. LabVIEW is 2018 SP1 with the real-time module. No special action is required to reproduce the problem except maybe to tweak the default value of "Write Time Limit" on the front panel of watch_loop.vi.

[glenw]

If I delay all my initialization code by 2 minutes with (real-time) Wait.vi or Wait(ms).vi, the problem still occurs 10 minutes and about 10 seconds after the program starts. Thus it seems like my code is not starting something which comes back and causes a problem 10 minutes later.