Serious Performance hit on RT (cFP 2120) using LV 8.5 compared to LV 8.2

[Ben]

Hi Michael,

Thank you very much for this update.

I am sure there are many questions you will be looking into but here are some ideas that spring to mind when reading your post.

1) Does reading "all" channels help?

2) Are we opening and closing references to the I/O?

3) Do any of the setting in MAX help?

again thanks fo rthe update. I'm sure y'all can figure this out.

Ben

[Mellobuck]

First of all, thanks to Ben and everybody for continuing on this discussion while I (the hands-on engineer) have been AWOL.

I wanted to quickly answer a few of the questions asked.

1) We had moved from 8.2.1 to 8.5 and saw the CPU performance problem (Geir had some question on this as he had moved from 8.2.0)

2) We remained in the XP operating system for both deployments.

3)We use both 'read all' (for data collection) and single FP reads in the code.

Regards,

 Mellobuck

[geirove]

My answer in Blue within the quote:

---

@Riconquistiamola wrote:

Geir,

Thank you for your post. To immediately answer your simple question, Yes, we have reproduced the problem. Your questions seems to suggest we have not made any attempt to reproduce.

I remember Ching said that, but if my memory serves me right Ching also said that they did not see the same dramatic (30%) that we saw. That is why I thought NI had not reproduced the problem.

In the same response from Ching, she asked if you could run an Execution Trace on your code. I have not seen you respond to this request, so I ran an Execution Trace myself. It appears the increased CPU usage is caused by decreased execution speed of the FieldPoint Read/Write VIs. This is the source of the problem.

I answered Ching that I would try to produce an Execution Trace in a week or so: I am extremely busy on this project making my current project a success. We have had to prioritize the customer, and this has not left us any time looking for errors in NI's system.

However, the larger issue with your post has been the inability to release your application to your client. It does not seem possible for you to test any troubleshooting steps I have recommended nor downgrade to LVRT 8.2. At this point we have done everything requested of you. How can we help you release your application?

As I have stated before, the system is up and running, we just wanted to do the final release on 8.5 rather than 8.2 because LV 8.2 uses 3.5 minutes to start our application (under the Project Explorer, which we have to use) each time we want to test. This is unbearable and wastes a lot of time. Also, editing VI's under 8.2 it takes 20 - 30 seconds each time we do any minor change like deleting / adding a wire.

P.S. -- You mention, "I am quite a bit dissapointed [sic] with the progress of this issue. It is now 14 days since my first post." First, it takes 24 hours for you to post and us to respond. Second, in several posts (and an email) Ching and I have asked you to contact your local office to gain the assistance of the Applications Engineering Department. This is the most efficient way to expedite resolving your issue. I cannot emphasize this enough.

Ching asked us to talk to our local representative, and we have done this. They have issued a "service request" to NI in the US, but we never received any more feedback from them. It's not in our hands!

Let us now concentrate on joining forces and pin down the problem: It is important for both NI , us and all other realtime / cFP users.

---

[oasisjo]

i seem to have the same problem ever since i upgraded to 8.5 the program i made is starting to run slow after a few days

it is that bad that i have to shut it down at least 1 time a week to keep it running

when i shut it down and restart it everything is normal again

and i mean by shutting down that i have to close also the project manager otherwise it is no use

so it is not enough to stop running

i have to restart labview completely

i did not have this problem with 8.2.1

[Riconquistiamola]

Hello all,

I spent most of yesterday running tests, so this will be a fairly long post. I ran three tests comparing 8.2.1 and 8.5 execution in multiple situations.

Test #1: Geir's code on cFP-2120.
CPU Usage
In LV 8.2.1, the CPU ran at 10% in the idle state. When the VI ran, the CPU rose to 35%.
In LV 8.5, the CPU ran at 10% in the idle state. When the VI ran, the CPU rose to 40%.
Execution Time
In LV 8.2.1, a sample of 10 timed loops took an average of 0.5197976 ms to execute. The FieldPoint Read VI took an average of 0.36105743 ms to execute.
In LV 8.5, a sample of 10 timed loops took an average of 0.972205 ms to execute. The FieldPoint Read VI took an average of 0.7654437 ms to execute.
Investigation
CPU usage is directly related to execution time. LV 8.5 takes roughly 0.45 ms longer to execute the while/for loop. At least 0.40 ms can be attributed to the FieldPoint Read VI. I am not immediately aware of the cause of the other 0.05 ms.
Test #2: Ching's code on cFP-2120.
CPU Usage
n LV 8.2.1, the CPU ran at 10% in the idle state. When the VI ran, the CPU rose to 33%.
In LV 8.5, the CPU ran at 10% in the idle state. When the VI ran, the CPU rose to 35%.
Execution Time
In LV 8.2.1, a sample of 10 timed loops took an average of 0.74459845 ms to execute. The FieldPoint Write VI took an average of 1.28436 ms to execute.
In LV 8.5, a sample of 10 timed loops took an average of 0.40673969 ms to execute. The FieldPoint Write VI took an average of 0.92243598 ms to execute.
Investigation
LV 8.5 takes roughly 0.54 ms longer to execute the timed loop. 0.52 ms can be attributed to the FieldPoint Write VI. Again, I am not aware of the cause of the other 0.02 ms.
P.S. Ben, I moved the IO Point out of the loop. There should be no problems with opening/closing every iteration.
Test #3: Benchmarking FP Write (Boolean -IO).vi using Ching's code.
Benchmark Code

Investigation
I don't have formal number crunching, but the times between Timestamp 1 & 2, between Timestamp 3 & 4, and between Timestamp 5 & 6 showed the greatest increase. Still, there was an overall increase in the time it took to execute (as expected, roughly 0.54 ms).
Future investigation
This raised the question, what is it about the DLL call and/or the ErrorConvert.vi that causes 8.5 to run less efficiently than 8.2.1. I benchmarked the ErrorConvert.vi code (see below), and recieved wildly inconsistent results. I'm refining my testing as we speak.
ErrorConvert Benchmark Code


Conclusions to follow in secondary post ...

Message Edited by Riconquistiamola on 10-25-2007 11:45 AM

[Jimmie_A.]

Hello,

Just because there was a lack of updates in the thread that doesn’t mean that we are not working on the issue. Small local branches (Norway and Sweden) tried to reproduce the issue as good as we possibly could with the resources we had available (read cFP controllers and modules) and when we discovered that we couldn’t see the huge difference in performance like Ben and Geir have experienced we involved Ching and Michael so they could help out with the resources they had available. The code I used and the code my colleague in Norway used were pretty much the same as Geir’s but we used other I/O modules. I used a 2020 controller and could see a performance decrease that was about 2-4 percent and my colleague that used a 2120 controller could see a performance decrease that was about 5-7 percent.

I still believe that my comments in earlier posts in this very thread are valid and I am in particular referring to these and I hope you are agreeing on those:

"If I see a performance hit when running the code using 8.5 compared to 8.2 then we know that it is due to the new RT version"

"What I wanted to achieve with the test though was to see if I could see similar behavior and a reasonable assumption is that this very issue you have encountered should be present on all targets running 8.5 and not just on cFP-2120."

Finally I would also like to apologize that have taken so long to provide an explanation to the issues you all were experiencing. 

Message Edited by Jimmie A. on 10-25-2007 12:05 PM

[Riconquistiamola]

So I've battled through all these tests and showed pieces of code are taking longer to execute. Now what?!?
Conclusion
Well, we have no other choice but to continue the investigation. I have several tests I'd like to run, but I'm unsure how many I'll get done by the end of the week. Here's my next set of steps:

1. Consistently benchmark ErrorConvert.vi.
2. Consistently benchmark FieldPoint DLL call.
3. Benchmark simple DLL call to determine whether or not the FieldPoint DLL is specifically the problem.
4. Benchmark simple subVI (similar enough to ErrorConvert.vi) to determine if there is any increased overhead in calling subVIs.
5. Benchmark multiple FieldPoint Read/Write calls.(I ran some off-hand tests during lunch to determine the behaviour of multiple FieldPoint calls. It appears fairly linear, but it's still worth investigating.)

We're making progress, but for the time being my best advice is to refrain from upgrading to 8.5 if you need something out the door immediately. If you're interested in sticking with 8.5, Ching and I have made several recommendations for possible workarounds. If you are having any trouble implementing workarounds or would like further assistance, please contact the Applications Engineering Department.

P.S. Geir, your extremely long edit time in 8.2 is not expected behaviour. 8.2 should act as responsive in edit mode as 8.5. More than likely there has been a corruption in LV, your project, or your VIs.

Cheers.

[Ben]

Hi Michael,

These results are interesting.

Re: the Error convert

The "Source String" should be on the root of the diagram since it is on the icon connector (See Greg McKaskle's postings here).

Other thoughts:

The bunch of checks for a variety of error codes would probably (not sure) be better handled with a case structure since only one of them can be true at a time.

Please keep us posted!

Ben

[geirove]

Riconquistiamola wrote:
..............

We're making progress, but for the time being my best advice is to refrain from upgrading to 8.5 if you need something out the door immediately. If you're interested in sticking with 8.5, Ching and I have made several recommendations for possible workarounds. If you are having any trouble implementing workarounds or would like further assistance, please contact the Applications Engineering Department.

P.S. Geir, your extremely long edit time in 8.2 is not expected behaviour. 8.2 should act as responsive in edit mode as 8.5. More than likely there has been a corruption in LV, your project, or your VIs.

Cheers.

Hello,

What is the status on this subject?

On the current project I am sticking with 8..2.

Regarding long edit times on 8.2: This is a much discussed issue other places on the LV forum: It happens when you have a lot of Shared Variables in a VI.

Start up times is even worse (~ 3 minutes). Creating new projects does not solve the problem. This also happens when there is a lot of Shared Vars in the project. Solved in 8.5-

However, we bought 8.5 because NI only supports the new Statechart Module on 8.5. We have indeed planned to use the Statechart Module on upcoming projects, so this performance issue is still a problem.

We need to know what NI plans to do about this performance problem, so we can plan accordingly.

Geir Ove

[Roy_F]

Geir,

I apologize for the lack of response and am looking into it. Support for this issue will improve moving forward.

I do not know what the technical resolution will be, but I am highly confident we will be able to resolve the technical issues to your satisfaction.

Roy Faltesek
Senior Group Manager
LabVIEW R&D, Sustaining