FieldPoint Family

cancel
Showing results for 
Search instead for 
Did you mean: 

cFP slows over time

I have an interesting problem, that I cna't seem to figure out.  I have a cFP running LabVIEW real-time, which communicates with a panel PC using TCP/IP.  Basically, the application runs fine for anywhere from thirty minutes to an hour, then it starts to slow down, and lose messages.  There are a few different loops in the RT application that control IO, communication, and datalogging.  I send the execution time of the main loop to the GUI, as well as keeping track of the number of messages sent and received.  At first, and for at least twenty minutes, the loop speed is exactly right, and I never have any messages lost.  Then messages start to dissappear, and the loop execution time starts to jitter, this increase until the loop execution time is three to four times what it is supposed to be, and almost all the TCP messages are lost.  Seemed like a wierd memory leak, so I took out all the code that was hadling the datalogging (although I calculated that I have at least a few months worth of space for datalogging in free memory), but the problem persisted.  I then thought that maybe TCP messages were stacking up, so I reset the cFP and disconnected the GUI for an hour, which makes the cFP just listen on a given port, but as soon as I plugged it in again, the cFP was already slowed down.  I slowed the main loop execution from 200 to 800 ms, to see if maybe I was pushing the processor to fast, but the problem was still there.  I tried installing the code on another cFP, but it had the same problem.  I reloaded the firmware, same thing.  I installed another applcation I wrote that is almost exactly the same, and has the same overall architecture, and identical TCP communication, and datalogging, and it ran fine, so it is definitely something in my application.

The really strange thing is that at first the cFP seems to be able to keep up fine, and then after about a half hour, without changing anything, over the course of ten minutes or so, the whole thing starts slowing dramatically, and communication becomes almost completely unusable.  I have used cFPs and LV real-time many times in the past, and I have never encountered this type of problem, also I'm a certified developer, so I feel confident that I know how to write LV code properly, so I don't think this is due to poor programming.  The only thing I'm doing, that I don't normally do, is using globals in my RT application instead of functional globals, but I use globals in the similar application I mentioned, and it ran without incident for days.  Also, because I havve to keep track of data accross power cycles, I occasionally update a config file, which I don't normally do, but, again, the alternateapplcation did the same thing, and it worked fine.

I am attaching the top level VI, but I'm afraid it doesn't reveal much.  I have almost everything compatmentalized into the sub-VIs, as per NI recommendation.

Can anyone think of any reason why an RT application run on a cFP would seem to run fine at a given rate without any sign of thread starvation, or the like, and without writing anything to memory that I can see, would then, seemingly out of no where, and very consistently, suddenly just not work anymore?

Thanks,
Benjamin
0 Kudos
Message 1 of 16
(9,084 Views)
Hi Benjamin,

Have you tried monitoring your Memory and CPU usage?  You can do this by going to your project explorer window while the code is running and select Tools>>Real-Time Module>>System Manager.  Enable both Memory and CPU tracking and click on the Start Monitoring button.

From your top level VI, I did see two coercion dots where you are subtracting the time stamps.  This could be causing the problem because LabVIEW will make a copy of the data each time it is coerced because it does not parse your code to see whether or not the wire is branched off prior to the coercion point.  Your memory is probably slowly climbing over time.

I cannot see a lot of your subVIs, but as a general recommendation I recommend monitoring memory, and if it is climbing, start disabling portions of code to determine what exactly is causing that.  Be careful with arrays and shift registers.  For example, do not use a build array with a shift register because LabVIEW will allocate another chunk of memory without freeing the memory used for the slightly smaller array.

Trey B
Applications Engineering
National Instruments

Message Edited by Trey B on 07-03-2007 10:45 AM

0 Kudos
Message 2 of 16
(9,055 Views)
So I took a look at the system manager, and it said that my CPU usage is at 100%.  I started removing more and more code, but no matter how much I took out, it still read 100%.  I eventually removed all of it except a single while loop with a wait set to 1000ms, and nothing else, and it was still 100%.  I disable the vi using dip switch 6, rebooted the cFP, and it was still at 100%.

Can anyone explain this to me?


0 Kudos
Message 3 of 16
(9,048 Views)
Benjamin,

The reason why it is always 100% CPU usage is because there is a pause setting that determines how long the controller sleeps after polling I/O modules.  The default is 0 ms, but it is a low priority task and does not affect performance.  We do not recommend changing this value.

http://digital.ni.com/public.nsf/allkb/8CBE08A24018A8F086256F79006CEB12?OpenDocument

What are you seeing in terms of memory usage?

Trey B
Applications Engineering
National Instruments
0 Kudos
Message 4 of 16
(9,040 Views)
After about two hours, my memory is

Used: 21870 (intial: 20279)
Available: 43270

I expect small amounts of memory to be consumed as time passes as a result of some datalogging I am doing, which is reflected in slight increase in memory used.  The loop execution speed, which I am monitoring in the program (it can be seen in the VI I attached earlier) is now bouncing between 400 and 800 ms.  It should be at about 400 ms, which it was stable at for about the first hour and a half.  Also, the TCP communication is very slow (I keep track of how long it takes the cFP to respond to query commands), which makes sense, since the main loop is slowing down, and it has the highest priority, I assume all the other loops are slowing also.  Almost all of the CPU usage is yellow (high priority), but this is what I expect, and it hasn't really changed since I started the VI a couple hours ago.

Any ideas?
0 Kudos
Message 5 of 16
(9,036 Views)
OK, so I have narrowed my problem to the following VI.  In the previously posted code, this is the one all the way on the end.  If I take all the code excpet this VI out, it slows the loop over time, and if I leave everything in except this VI it works fine, so I'm fairly certain that this VI is causeing the problem.  Can anyone tell me what about this VI would cause the slow down that I described earlier in this thread?  I am operating with simulation mode set to false, of course.  Also, sorry this thing is so big, I had a lot of stuff to unbundle.
0 Kudos
Message 6 of 16
(8,990 Views)
Benjamin,

I would try building arrays once outside of the while loop and then use replace array subset and shift registers.  This is because every time you call build array, LabVIEW is going off and allocating memory. If you allocate memory once and the replace the values, then you'll never allocate any more memory.

I also noticed a couple of coercion dots in the True case of that case structure on the bottom right.

Trey B
Applications Engineering
National Instruments
0 Kudos
Message 7 of 16
(8,983 Views)
OK, so I have finally really narrowed it down.  The writes out to the cFP are what is causing the slow down.  If I pull them out, and leave the reads in, then it works fine.  The symptoms I specified earlier don't seem to indicate that the cause is due to slow memory allocation because of the build array functions, but just to be sure, I pulled out all the build arrays, and replaced them with a number of cFP writes so that each number to be written has its owns write function, and it behaved the same way, slowing down over time.  So I am nearly certain that the problem has something to do with the cFP write functions, or the channel configurations in MAX.  I feel, unfortunately, that I am about to hit a dead end, as I have narrowed it down as fine as I can, and determined that, it seems to me anyway, it is not a problem with how I wrote the code.  It just seems that either the cFP is not behaving properly, or the write functions do not work as expected.  Although I previously ran the code on a known good cFP and it still didn't work, so I assume the cFP is working right. 

Where do I go from here?  I'm just about out of ideas.
0 Kudos
Message 8 of 16
(8,955 Views)

Your slow writes could be caused by a couple of things:

One is if the read of the data values you wish to write is slow.  It looks like you are using a global variable to bring in those values.  If your system allows, you could replace the global with a constant and test to see if things still slow down.  Not sure where that data is coming from but you should probably look for an alternative to the global. 

The second reason would be if there is some sort of problem with the FP Write.  You are executing the FP Write a boatload.  As a test you might set up your vi so it writes an array of data to an entire module with a single FP Write.  That way you only do the FP Write once for each output module. 

0 Kudos
Message 9 of 16
(8,940 Views)
Benjamin,

What versions of the following do you have:

1.  Fieldpoint driver
2.  LabVIEW Real-Time
3.  Fieldpoint hardware's RT version.  Find this by going to remote systems in Measurement and Automation Explorer (MAX), right click on the target's software folder, and select "add/remove" software.

What Fieldpoint hardware are you deploying to?

Trey B
Applications Engineering
National Instruments
0 Kudos
Message 10 of 16
(8,929 Views)