CPU load unbalanced and too low in multi-processor computers

[Dave_B]

I have a rather large test system 5-6000 VIs and we have noted some peculiar behaviour.

On a single processor system the application can take up to 100% of the available resources.

On a two processor system the application can take up to 50% of the available resources.

On a four processor system the application can take up to 25% of the available resources.

On an eight processor system the application can take up to 12-13% of the available resources.

I think you guys can get the gist of what I stated above (I didn't get pictures for 1 and 2 CPUs but I have observed the behaviour, personally)

We haven't done any specific multithreading code (we do lauch many parrallel processing loops dynamically = should be a good thing)

Why is the application behaving this way? Does anyone have a similar problem? Any tips?

Thanks for any help you can offer.

//David

[Ben]

That sounds like a single threaded app.

If you are using a lot of property nodes in your VIs, you could be spending most of your time in the UI thread (which is sinlge threaded).

Ben

[Dave_B]

I don't use many property nodes in my process VIs but I do in my GUI processes. I have realized that the app behaves like a single threaded app, what can I do to get it to behave like a multi-threaded app? I didn't think that there was any specific coding principles needed and that LabVIEW would handle it all on it's own (one of the strengths of LabVIEW, at least from a sales pitch perspective).

I can give a little more detail as to the inner workings of the app.  We have "variables" in our app and these variables are queue-based data so that an I/O class places data in the queue and a process/GUI class can subscribe to the queue and get/set the data.  Can this architecture be the reason behind the app only executing on one thread?

//David

[Yamaeda]

The queue structure sounds solid and good, i wouldn't be surprised if there's a bottleneck with ui or property nodes as mentioned. If you do some VI performance analysis, is it 1-2 taking up most of the time? 

Is the data acquisistion a greedy loop?

/Y 

[altenbach]

You really don't provide enough information.

Is there really always something to do to keep a core busy at all times? Unless there is serious computations involved for solving a math problem, a typical VI should not be using a lot of CPU. (For example if it needs a full core today, it would not have been possible to even run the test a couple of years ago. That would be hard to believe). Don't overestimate how much a modern computer can do using only 5% of the CPU at any given time. 😄 I still run an acquisition system on a 120MHz Pentium 1 and LabVIEW 4. There are no performance problems even though the hardware is orders of magnitude slower than even a modern Atom processor on a netbook.

How many things can really occur at the same time? If you make your code overly sequential by lining everything up along error wires or by overuse of sequence structures, the code cannot be efficiently parallelized, no matter how hard the compiler tries.

How many of the 6000 VIs are called concurrently? How many require full CPU? If this is a test system, I assume that there is interaction with a device under test? Are the tests sequential or parallel? How many devices are tested at the same time? How much is post-processing compared to waiting for results? Are the tests really CPU limited??? Really???

Have you done any profiling? Did you identify the part that is most demanding? Could it be that a single greedy loop is consuming most of the CPU doing nothing?

Except for lengthy simulations or complex data processing, a typical application should never be CPU limited. The timings should be fully software controlled such that the code runs the same, independent of the power of the computer.

All that said, LabVIEW can easily keep all CPU cores busy if really needed and if programmed correctly and with parallel processing in mind. I have a complicated lengthy fitting code where I was able to keep 64 cores at near 100% (4 AMD 6274 CPUs with 16 cores each). As a more typical example, and compared to sequential execution, the parallelized code is 4.5x faster on a quad core I7 (if I disable hyperthreading in the bios, I get a 3.8x speed increase), so not only can it keep all four cores at 100%, it can even get a measurable boost from hyperthreading. On a six core I7, I get a 6.9x speedup, also keeping all cores busy.

I am not going to look at a project with 6000 VIs, but please show us some profiling data. Find the VI that carries most of the load and attach it if you want.