PXI-8196 Controller locks up after 12 hours

[ssto]

I have a PXI-8196 RT controller running with 2 FPGA modules doing DAQ and instrument control.  The RT system runs for about 12 hours before it locks up.  Using the distributed system manager tool, I found that the memory usage is fine (only about 5% of total memory is used), but the CPU usage from the Timed Structures continually grows, starting at about 10% and going up toward 80% over time.  I'm not sure what this represents, but I do have a timed loop in my RT code that has a 200ms period.  Inside this timed loop are two DMA FIFO reads with 10 ms timeouts.  Each DMA FIFO read is inside its own while loop which reads until either an error occurs (-10400) or the FIFO is empty. 

Does anyone have any ideas on what could be causing the run-away CPU usage and causing the RT system to lock up?

Timed Loop:

[Caseyw]

Hello ssto,

While it is not immediately obvious what is causing your CPU usage to grow over time and the memory usage to stay bounded at 5%, based on the code  screenshot that you posted I do have a suggestion.

Instead of allocating a blank array inside of your timed loop and then using build array within your while loops, I would strongly recommend pre-allocating fixed size arrays before the timed loop runs. Then, inside of your while loops, use Replace Array Subset to write data to your arrays and keep track of your current index (this is basically a circular buffer). Allocating memory inside of a real-time application not only leads to increased jitter, but using Build Array wastes a lot of CPU cycles, as each time you add elements to the array you force LabVIEW to allocate a larger block of memory and copy the existing array contents over to the new array.

Implementing this change should help work towards solving the behavior that you are seeing. If you are still seeing the increasing CPU usage after making this change, then please post again and I (and others) am glad to help further in finding the root of the problem.

Have a great day!

Regards,

Casey Weltzin

Product Manager, LabVIEW Real-Time

National  Instruments 

[ssto]

Thanks Casey,

I agree with your analysis that the arrays should be allocated up front.  I did see a message in the knowledge base that states something similar.  If I understand correctly, the RT system has to find continuous memory locations when it allocates a new array, and it does behave as you mentioned.  I am also wondering if the timed loop is causing issues if the DMA FIFOs time-out for some reason.

I'll make your suggested changes and test it out.  Thanks for your comments.