Ideas needed for debugging excessive system CPU usage

[MarkMoss]

I could use some suggestions regarding how to debug intermittent excessive CPU usage in a large LabVIEW application (CAN, DAQ, and IVI for 16 stations).

Normally, my application consumes about 25% of the CPU time on each of the host computer's four CPU cores.  Occasionally, after running anywhere from a few minutes to several hours, the application will consume 100% of the CPU time on one of the processor cores for a few seconds.  When this happens, it results in a CAN read queue overflow for any of the CAN handling loops (one timed loop per station) running on that processor core.

This has been going on for a couple of weeks and I've had very little luck figuring out what part(s) of my application are responsible for the excessive CPU usage.  Here is what I've tried so far:

1. Verified that the CAN frames are being received at the correct rate.  When the CAN read queue overflows, the data in the queue was always received at the correct rate (100 frames per second).

2. Logged the CPU usage of each process running on the computer.  There is no abnormally high CPU usage at the time the error occurs.  However, I'm only able to log the CPU time from each process once every 20 seconds or so.  Since the problem only lasts for a few seconds, it may not be showing up in the log.

3. Watched memory, CPU time, and I/O rate in Process Explorer as the problem occurred.  Purely by luck, I was watching the application's process in process explorer when the error occurred.  For 3 or 4 seconds, one processor was at 100% system CPU time usage.  At the same time, system CPU usage for my process spiked to about 25% (corresponding to 100% of one core on a four-core machine).  The I/O rate and memory usage remained constant while this happened.

I haven't tried using the profiler as the problem only occurs for a few seconds out of hours of operation.  Does anybody know of a way to automatically log the profiler information every few seconds?

Is there anything else I can do to narrow down what part of my program may be causing the spike in CPU usage?  Is there anyway to determine which VI's and/or threads are running on a particular processor core?

Mark Moss

Electrical Validation Engineer

GHSP

[JorgeNakan]

Hi Mark, have you tried using the performance and memory utility in LabVIEW? (Tools-->Profile-->Performance and memory).  This will give you a better idea where your program is using excessive memory.  Best Regards,

Jorge N.

[MarkMoss]

---

@JorgeNakan wrote:

Hi Mark, have you tried using the performance and memory utility in LabVIEW? (Tools-->Profile-->Performance and memory).  This will give you a better idea where your program is using excessive memory.  Best Regards,

Jorge N.

---

I've used that utility, but I don't think it's going to help me here.  The problem is that my program operates normally for a few hours than uses excessive CPU time for a few seconds.  I haven't found a way to capture that event with the profiler.

Mark Moss

[PhillipBrooks]

I would look at two things, how and when your application writes data to disk and secondary applications like antivirus.

I had problems once when trying to capture a high speed UDP stream to disk where the system would 'thash' the disk every now and then and at the same time drop UDP data. When writing to disk with the basic I/O functions, the data is passed off to the OS. If the OS creates a large buffer, it might be consuming enough processor and RAM that it interferes with your application. My solution was to use the Flush File primative every Nth write to disk. I had to tweek the interval a bit, but my system load was flat after this addition.

If you have antivirus software running and you create new files on a regular basis, the AV software may be evaluating your data files before allowing them to be written to disk. Set your AV application to exclude your app's data directory.

[James_W]

I had an issue like that recently on a project. What turned out to be happening was that over time the Windows OS was moving the application onto one of the Processors (CPU2 as Windows runs on CPU by default).

I would look at the whole architecture and take a modular approach - ensure you have properly setup the program to use the different execution systems available in LabVIEW (VI Properties>Execution system) to maximise the chances of this not being an issue.

Look at all of your queues and memory allocations - this might not seem intuitive, but when a Queue is allocated, until it is destroyed totally it keeps all memory assigned to it. the CPU overhead can be the memory management tool trying to assimilate the memory blocks.

Look at any DLL calls (and any FFT functions & higher level maths functions that take more CPU time)
Look to optimise any array manipulations - you can optimise for memory (higher CPU) or speed (higher memory)

James