Serious Performance hit on RT (cFP 2120) using LV 8.5 compared to LV 8.2

[geirove]

Hello,

We have just finished a large project using LabView 8.2 and Compact FieldPoint (cFP) 2120 on a Windows XP Pro computer.

The system developed controlled equipment that forms critical parts of a Concrete Plant here in Norway.
2 Identical units whre installed controlling two almost (but not entirely) identical pieces of equipment.
The project has been a great success, and used technologies such as: StateDiagrams (StateDiagram Toolkit) to describe the System Processes, Network Shared Variables to transfer approx. 50 different  pieces of non-critical information over WLAN to a LabView based Surveillance Program in a Control Room from both "machines".

The CPU Load on the cFP was approx. 65 % which gives the ETS Realtime OS (RTOS) plenty of "room" to run its "maintenance threads".

After the last installation, we decided to try to migrate the system to LabView 8.5 on another Computer running Windows Vista. The system ported nicely and run on our test cFP 2120 as well.

We had done extensive performance / cPU Load tests on our 8.2 system, and new that in a particular setting the CPU Load was:
  *** 65% *** measured using the System Manager (disabling VI and Memory Monitioring).  (Absolutely max measured is 71 % when the system is running)

When we did the same test on the cFP running the LV 8.5 compiled code and with the LV 8.5 software installed on the cFP, we where in for a chock:
The CPU Load was *** 87% *** in the particular setting, and could run over 90%.

This renders LV 8.5 useless for our controll task. A setback, as we had plan to port our current code using the Statechart Module that we had bought for LV 8.5.

We also installed a VI to "Monitoring the CPU Usage of a LabVIEW Real-Time Target" found at this link:  http://zone.ni.com/devzone/cda/epd/p/id/4713

It turns oot that this VI gave us the same result as using the "System Manager" (if we summed the load for each priority), but also gave more info about the load at each priority.

We of course did not run the System Manager at the same time, as this would cause a conflict: We either run the System Manager or the VI above. When the System Manager was run, the Vi above was

disabled !

We are running updating of the 50 Network Shared Variables in a Normal Priority Thread. The 9 StateDiagrams are each running in a Separate Thread started by including the VI for the StateDiagram in a  Timed Sequence with one frame: This gives all these Processes "Very High" Priorty and you can see that most of the work is done at this priorty level. The Very High Priorty level uses Single Process  Shared Variables with FIFO to send the variables to the Network Shared variables in the "Normal Priorty" Thread.

The Shared Variabel Server is run on the PC in the Control Room and not on the cFP.

Below you can see the Installed Components and the Performance Result for the cFP when Running configured for LV 8.2 and 8.5.
For LV 8.5, the performance of each Priority Level is degraded with the same percentage (approx 30%) which is really serious.

National Instruments Engineers: Please help identify / fix this problem and give us a Time frame for when the problem can be fixed!

LabView 8.2 Installed Components on cFP 2120:
http://objective.no/gostemp/cFP_Components_LV_8-2.jpg

LabView 8.2 cFP 2120 cPU Load:
http://objective.no/gostemp/cFP_LV_8-2_CPU_Load.jpg
--------

LabView 8.5 Installed Components on cFP 2120:
http://objective.no/gostemp/cFP_Components_LV_8-5.jpg

LabView 8.5 cFP 2120 cPU Load:
http://objective.no/gostemp/cFP_LV_8-5_CPU_Load.jpg

regards

Geir Ove Skjærvik
Objective Technology
Norway

[geirove]

Hello,

I just found this:
http://zone.ni.com/devzone/cda/tut/p/id/6449

4CUCQEAR — Slower Execution Time When Manipulating Arrays Than In Previous LabVIEW Versions
4DOC349I — Performance Decrease When Calling XControl Properties and Methods

I do not use XControls, but I do read all of my 4 16 channel DI-301 Input Cards in a While Loop by reading "all" inputs and then wiring them to Shift Register at the Edge of the Loop. Upon request, a cached channel is read by indexing the 16 Byte large Array using the "Index Array" VI.

I do not know if this qualifies as "Manipulating Arrays".  This is done at "Very High" Priority and may explain why I get a 30 % performance hit on LV 8.5 there. It is still very bad for us though: There is nothing we can do about it !

However, it does NOT explain why the Normal Priority Thread also get a 30% Performance Hit: There is no "Array Manipulation" there whatsoever !

Geir Ove

[geirove]

Hello,

Yet another comment: On LV 8.5 the CPU Load is the same whether I run my Top Level VI from the Project Explorer or create an Executable and download to the cFP.

This is also strange, since on LV 8.2, an .rtexe is always a bit faster (debugging turned off) than running the Top Level VI !

What is causing all this ?

[Jimmie_A.]

Hello,

I haven't seen it reported before but one thing that has changed between 8.2 and 8.5 is the underlying architecture of shared variables. If you use 8.5 on your host and 8.2 on your target it will exchange data between host and target using the old architecture based on UDP while if both the target and the host are 8.5 they would make use of the new architecture based on TCP/IP. Not sure if that could explain the performance hit you are currently seeing.

[geirove]

Hello,

Thanks for answerring.  I do not think what you suggest is the problem. On LV 8.2, the receiver of the Shared Variables (the PC Program) do not even need to be running: I still get the same low 65 % CPU Load.

Also, when testing with LV  8.5, I ran the Program that received the data over Shared Variables on LV 8.5 as well.

Also, the "Very High" Proirity Threads are only writing data to  Single Process Shared variables that are read in "Normal" Priority thread Thus it is a communication internal to cFP Program.

However, if the efficieny of the Shared Variables has detoriated in LV 8.5, this will certainly give a performance Hit in both the "Normal" Priority and the "Very High" Proirity Threads !

This is a Show Stopper for us, and I guess we all do not want to see that performance goes down, so I hope NI can take a look at this !

Is there a way to configure the the Shared Variables to use UDP again on 8.5? If so, I could try this !

[geirove]

Hello,

I have reproduced the problem in a minimalistic test application !

Download the test from this link: http://objective.no/gostemp/lv_8_5_probleml.zip

This is how the test looks: http://objective.no/gostemp/LV-8-2_8_5_Test_01_vi.jpg

The ZIP file contains the following files:
a) LV-8-2_8_5_Test_01.vi    -  The test VI itself
b) SK1_Host_Remote_Panel_Shared_Variables.lvlib   -   The Network Shared Variables that I use only one of in this test (SK1_Error_Code)
c) Interprosess_Data.lvlib  -  The Single Process Shared variables that I use only one of in this test (Reported_Error_Code)

I have not included a Project File and you need a Compact FieldPoint to test this. I have a cFP 2120.

Now to the intersting figures:

Compiled under LabView 8.2 this application runs with 21 % CPU Load as measured by the System Manager on the cFP 2120.

Compiled under LabView 8.5 this application runs with 42 % CPU Load as measured by the System Manager on the cFP 2120.

This is a dramatic degradation of Performance !

I have not yet investigated if removing the Shared Variables will change this.

NI: Please Help locating this problem !

The only thought I have is this: Does LV 8.5 take care of the change of the Shared variables when the two .vlib libraries are moved to a 8,5 project ?

[geirove]

Hello,

Test update:

I have run the test application above with and without the Shared Variables, and LV 8.5 is much slower even without the Shared Variables. Here are the figures:

Compiled under LabView 8.2 test application without Shared Variables runs with 15 % CPU Load as measured by the System Manager on the cFP 2120.

Compiled under LabView 8.5 test application without Shared Variables runs with 25 % CPU Load as measured by the System Manager on the cFP 2120.

I also tried to use the In Place Structure around the Array Indexing under LV 8.5, but it did not make any difference.

Seems like there is a fundamental performance problem with RT (ETS) running under LabView 8.5.

[Jimmie_A.]

Hello,

I will see if I can reproduce it on another ETS based controller (removing FieldPoint calls) since I don't have access to a cFP 2120 controller. If I see a performance hit when running the code using 8.5 compared to 8.2 then we know that it is due to the new RT version. If not it might be a combination of the FieldPoint driver and the new version of LV RT. I will get back with results later on but I am currently teaching a course so I might not be able to do it until tomorrow.

[geirove]

Thanks Jimmie ,

If the problems cannot be reproduced on another ETS based system, I assume you forward the problem to someone that does have a cFP system.

Geir Ove

[Jimmie_A.]

Ok,

I finally had some time to test it myself on a less powerful target (cFP-2020):

I had one timed loop reading data from an analog input module where the timed loop ran with an iteration time set to 500 ms.

I had another loop (while loop) set to run with 750 ms loop rate.

Between these loops I used RT-FIFO’s in order to perform my inter-process communication and then I used network published shared variables to exchange data to the host back and forth.

The results when using the System Manager or the VI method to get an estimate of the CPU usage was exactly the same on both targets. Around 75% on the normal loops and the rest spent in high priority loops. Blue color was barely visible.

I will need someone test it on a more powerful target and see if they can reproduce the issue you see where the CPU workload differs in different versions of LabVIEW RT: