FieldPoint Family

[MarkFreeman]

I am supporting a LabVIEW real-time system with DSC and Compact FieldPoint (cFP-2200 plus I/O),
Having repeated connectivity problems:  

- data logging stops and cannot be restarted from HMI
- Can ping the cFP fine
- MAX can see the cFP, System State shows "Connected - Running"
- When navigating around in Max, occasionally get pop-up "Unable to communicate with the FieldPoint network module."
- when try the MAX <Restart> button, responds with "Unable to send command to remote system."  Then the System State changes to "Disconnected" (but can still can ping the cFP ok).

Requires restarting the cFP-2200 (RESET button on front)

What else should I check?

Related question:
If network communications between the HMI server and the cFP is interrupted momentarily, will communications normally recover without intervention?
This is an isolated network (only the server and cFP on the link), but there is a pair of MOXA wireless bridge deveices in between.

Thanks!

[MarkFreeman]

Additional info:

Happened again today twice, cFP-2200 status LED is off, but user LEDs indicated program not running (they normally blink under program control).

So this may actually be a problem with the program hanging, rather than just a communications issue.

Observed today and in the past occurrences:
Ping response time in this "hung" state is a consistent at 2 to 3 ms
(normally the cFP ping response varies considerably from 2 to 100ms)

I am looking for the best steps to troubleshoot when encounter this problem, identify if a hardware issue, etc.

 Thanks.

[Patricia B]

Hello,

Thanks for all the details posted thus far. Just a few questions to for further troubleshooting. Could you please provide the following additional information:

Revision of NI-Fieldpoint driver currently on PC and cFP:
Host O/S:
Version of LVRT:
Do you see the same communcation failure over time when you connect directly to the cFP with just an ethernet cable?:
Are you using a startup VI?:

Regards, 

Patricia B.

[MarkFreeman]

Thanks for  your reply -- I realized over the weekend that I left out all the important details:

Revision of NI-Fieldpoint driver currently on PC and cFP:
  6.0.8
Host O/S:
  Windows Server 2008 R2 Standard
Version of LVRT:
  2010 SP1
Do you see the same communcation failure over time when you connect directly to the cFP with just an ethernet cable?:
  Don't know, this system is at a customer's location and it is not practical in the short term to run an ethernet cable.

  We have another system running FieldPoint 6.0.7 that is direct connected with ethernet cable (and a switch), our technician has had to occasionally cycle power on the cFP to restore communications but we don't have the almost daily problems that occur with this system.
Are you using a startup VI?:
  Yes, the primary HMI screen is setup as the Startup VI in the Build Specification properties, Source Files category, Startup VIs.

Thanks again,

[unclebump]

What does wireshark show is going on with the network packets?

[Patricia B]

Hello, 

From the information provided thus far, what might be happening is that the code being run in the time critical loop is large and consuming massive amounts of memory to the point where the processor is only able to handle that code and not allow/handle network communications. Perhaps something in the time critical loop is being done inefficiently such as dynamically buidling arrays or passing lots of data using shared variables instead of network streams. My suggestion to investigate this further would be to use the Distributed Systems Manager to monitor the CPU usage during execution of the code and see where there are spikes in the usage. Creating a debuggable exectuble may also aid in tracking the correlation between CPU usage and processor intense sections of your code. 

Best,

Patricia B.

[MarkFreeman]

Thanks for your reply.  So far the CPU usage looks very consistent at a little over 50% (see attached).  No loss of communications since I started monitoring CPU on 7/28.

Wireshark looks like a good tool, will try it next time the comm fails.

[clenk]

hallo,

i encountered the same problem. did you found a solution for your issue?

i logged the memory and cpu usage until the breakdown. cpu usage is constantly at 95% .. memory usage is constantly at ~90mb. the failure occurs without a preceding change at the usages.

any hints?

with kind regards

christoph

[MarkFreeman]

Hi Christoph,

No solution yet.  We have recently added an antenna to the second MOXA wireless bridge so we have good line-of-sight comm, hoping this will reduce/eliminate any comm drops.  But the failure we have seen looks like the cFP processor OS hanging -- we can ping the cFP and it responds, but can get no other response with DSM or MAX.  Our application flashes a pattern on the A,B and C LEDs on the front of the cFP-2200 and these lites stop flashing when we have the problem.

However I have not seen any evidence of high CPU usage.

regards,

Mark 

[clenk]

Hi,

yeah it totally sounds like the same issue. the processor is hanging and therefore the communication breaks.

is there anyone from the ni-team who can help?

kind regards

christoph