Problem with connection to PXI RT using WIndows 7 and labVIEW 2009

[BenSparkes]

Hi,

Until recently I have been running LabVIEW 8.6 on a FPGA 7852R card on a PXI-1042Q chassis with a Windows 7 computer and was having issues with the PXI connection whenever the computer screen saver came on with the following message appearing:

"Waiting for Real-Time target (rt PXI target) to respond"

With the only way of fixing it is to re-deploy the code (and even that would not always fix the problem).

I have now moved to LabVIEW 2009 and found that the problem has become much worse, with the above message coming up approximately every 5 minutes even without the screen saver being active.

I was wondering if anyone else has experienced this problem or knew a way or resolving it?

Cheers, Ben

[NAKeel]

Hi Ben,

Can you tell me a little more about your application?  Are you deploying a VI or a real time executable?  Do you get this message in the project explorer,  MAX, or both? Are you using large volume RT FIFO's (>50,000 elements)?  Did you upgrade this computer from XP/Vista to Windows Seven, or did this come with Windows Seven already installed?  Are you using shared variables?  If you are, do you see anything in Distributed System Manager?

I have found a couple instances of this problem occuring.  I have posted links to the forums below.  Take a look and see if either is useful to you.  

http://forums.ni.com/ni/board/message?board.id=280&message.id=5344&requireLogin=False

http://forums.ni.com/ni/board/message?board.id=280&message.id=2530&requireLogin=False

Regards,

Nick Keel 

Applications Engineering 

National Instruments

[BenSparkes]

I am deploying a VI on my FPGA target on my PXI and am getting the message in the project explorer (I havn't checked MAX for this, but have found in the past that MAX stays connected to the PXI even when the VI won't delpoy due to "lost connection"). I am using 2 32,767 FIFOs to transfer data from the FPGA to the RT, both of which are updating continuously from a part of the code running at 747 kHz. The computer was built in house and initially we loaded a beta version of 7 but have recently done a blank install with a bought version of 7 Ultimate and installed LabVIEW 2009 (32 bit version) at the same time. I also am not using shared variables.

I found out today while trying to run the VI that even after the connection is lost the system will keep working, but obviously will not update the data on the screen.

I have had a look at the two posts but don't think they are the same thing.

Thanks for looking though and any further thoughts would be greatly appreciated.

Cheers, Ben

[Caseyw]

Hi Ben,

One issue that can cause the type of behavior that you are seeing is starvation of the network communication stack on the real-time target due to high CPU usage. Even if this is not what is happening in your system, it would be worth ruling out.

What does the CPU usage look like on your real-time target when you see the errors indicating that the connection to the real-time system has been lost? You should be able to see this CPU usage via a monitor connected to your real-time PXI system, or using the Distributed System Manager (while the connection is still active).

If this starvation is not causing the issue, then we'll have to keep digging to find the root cause of the issue. I'm sure that we can work together to figure out what is happening and either find a solution, or report the issue to our R&D team and attempt to find a suitable workaround for the time being.

Best Regards,

Casey Weltzin

Product Manager, LabVIEW Real-Time

National Instruments 

[BenSparkes]

Hi Casey,

Thank you for you advice, I'll look into it, however I am off on holidays from now until mid-January. When I get back I'll let you know how it goes.

Cheers (and Merry Christmas/Happy New Year!)

Ben

[BenSparkes]

Hi Casey,

I hope you had a good holiday!

I have just measured the CPU usage on my Real Time and found that normally it runs at 50/85ish on the two CPUs, but when it crashes the usage goes to 20/95-100% and stays there until the code is redeployed.

I'm guessing this means that it is the CPU usage which is a problem. Is there any way to reduce the usage without reducing the size of the code? (as ideally I want to increase the size of the code rather than decrease in the not-too-distant future).

Cheers, Ben

[DiscoBall]

Hi Ben,

Can you isolate the point/time at which it jumps to 95-100%? Had you modified the code since upgrading to LabVIEW 2009? How are you handling priorities between different loops in  your application? Do you have one time critical loop with other loops running at normal priorities?

In general it is not the size of the code that is the issue, you can have a large amount. What causes an issue is that you might not be yielding time to other threads/loops in your application and that background processes never have the time to execute. The best tool for looking at timing issues of this nature is the Real Time Execution Trace Toolkit. Do you have access to this tool?

More infomation here: NI Developer Zone Tutorial: Exercise 1: Create and Debug a Real-Time Project and Application

Message Edited by DiscoBall on 01-29-2010 08:02 AM

Message Edited by DiscoBall on 01-29-2010 08:03 AM

[BenSparkes]

Hi Joshua,

The code seems to lose connection every 5-10 minutes and doesn't seem to be related to any process (as the code is normally always performing the same task). As for modifications, I don't think any have been made since moving to 8.6 (we only upgraded due to driver compatibility issues for DAQmx with Windows7). As for loops, the whole code runs from one loop (which may cause the problem?). I've attached the code if you're interested.

I do have the Toolkit, but not entirely how it works.

Cheers, Ben

[DiscoBall]

Hi Ben,

You have quite a bit going on in that code. My main recommendation is to use multiple while loops and implement a Producer/Consumer Design Pattern so that you can move your file I/O into a consumer loop. This will increase the modularity of your code and make it easier to debug. There is so much going on in the while loop it is difficult to tell where your hangup may be. It could be on how you are using the FGEN VIs, it could be on how you are using your FPGA.

I would begin by moving the file I/O into a consumer loop. Then modularize your code further by turning as much code as possible into subVIs. You can then disable portions using the Diagram Disable Structure to see if you can stop the loss of connection issue.

The Producer/Consumer loop will be huge in improving performance. The subVI step will be huge in helping debug your issue. For example this portion of code:

can easily be turned into a subVI. If you replace the local variables with terminals you can then improve the dataflow of your VI which will also increase the performance and eliminate race conditions.

Message Edited by DiscoBall on 02-01-2010 06:15 PM