02-10-2011 11:08 AM
Hi all,
I am the product manager for NI TestStand and I wanted to get your feedback on TestStand performance.
We know that performance is an important concern for a lot of our users and in order to improve TestStand performance, we need to truly understand what areas of performance are most important to our users.
I would love to hear your thoughts on:
Thank you very much for your feedback!
02-12-2011 12:14 AM
Hey Justin,
Some of my thoughts:
02-14-2011 01:32 AM - edited 02-14-2011 01:34 AM
Hi Justin,
1.) I am of the same opinion as Jigg. The Load/(Unload) behaviour of TS.
My major focus is on the Load behaviour. There was always a bad impact when changing an old PC into
of state-of-art in productionfield or development and after pressing Start(F5) it tooks
the same time till the execution was starting (jumping into PreUut). Sometimes it took 3-5 minutes!
And you wonder why did i changed the maschine ?
It seems that loading modules on TS4.5 is even faster than the old ones.
Another topic of performance is editing the steps itself. In the TS-Idea exchange
there are a lot of entrys dealing with this. At the moment no one is under consideration !
2.) TS has some features about performance for example the "Profie Resource Usage"
but i am honest i have never used it. But what should i answere here ?! Lets better go to 3
3.) There should be must a callback that shows the current state of loading modules. With this
you will show the user that Teststand is still alive after F5 and it would be make sense to get some cup of coffee now.
- Hm I should place this in Idea-Exchange.
That in the Idea-Exchange the "Under Consideration" counter is beginning to start counting!
4.) No, just feelings and it seems it is geeting better.
Regards
Juergen
02-14-2011 08:21 AM
For module loading speed issues, please provide more details such as the following:
1) What adapter(s) do you primarily use?
2) If LabVIEW is one of you primary adapters, are your VIs typically already mass compiled (i.e. up to date with the version of LabVIEW you are using)?
3) Approximately how many code modules are you loading and how much time is it taking?
The above information will help us better diagnose the issue and target the area with the greatest problem.
Thanks for any details you are able to provide us.
-Doug
02-14-2011 08:24 AM
For issues on the idea forum that you think are related to this topic and should be addressed please let us know specifically which ones you think are most important.
Thanks,
-Doug
02-15-2011 12:04 AM - edited 02-15-2011 12:05 AM
1) What adapter(s) do you primarily use? LabVIEW. We have some .NET, ActiveX, CVI and DLL but they are few and far between.
2) If LabVIEW is one of you primary adapters, are your VIs typically already mass compiled (i.e. up to date with the version of LabVIEW you are using)? Isn't it bad to mass compile all the native VIs in LabVIEW... not to mention TestStand installs older VIs. I doubt everything is mass compiled.
3) Approximately how many code modules are you loading and how much time is it taking? Depends on the sequence file. We probably average about 15 per sequence with about 20-30 sequences per file/build. A lot of them are the same VIs (i.e. Instrument Control VIs, UUT Control VIs, etc..) just different parameters. If we do a snapshot view in ClearCase it loads a ton faster than a dynamic view. However, this could be chalked up to the server communication.
I did notice the other day, and this is only with dynamic views, that when there are two VIs named the same put pointing to different libraries (e.g. Initialize.vi in different instrument libraries) that the build on deployment takes about 1.5-2 hours. It finally builds but WOW! Whereas the same build in my snapshot view will take 5-10 minutes. In the build log it had a bunch of colision errors pointing out those VIs with the same names. I have gotten the dynamic view to build in 10-20 minutes on different sequences with about the same amount of calls though. ODD??
02-15-2011 10:04 AM
Hi Jiggawax,
Thanks for the additional details. I have some follow up questions.
Are you talking about module loading time being slow or deployment building time or both?
Are you loading VIs from TestStand using the LabVIEW development environment or the runtime engine? What version? If using the development environment, are the VIs saved with the same version of LabVIEW.
Are you using source code control integration with TestStand (i.e. using teststand workspaces with source code control integration enabled). Are you saying that TestStand is loading things faster or slower based on your view in ClearCase? or am I misunderstanding what you meant?
I'm not that familiar with ClearCase, what exactly is the difference between a snapshot view and a dynamic view? How would this affect which VIs are referenced?
For your 15 files with 20-30 sequences, about how many VIs would you say are loaded (doesn't need to be exact, just a rough estimate)? And how long does it typically take for them to load? I just want to make sure that when we look at the performance of the LabVIEW adapter that we make sure we are seeing the same thing as you.
Thanks for any additional details you are able to provide,
-Doug
@~jiggawax~ wrote:
1) What adapter(s) do you primarily use? LabVIEW. We have some .NET, ActiveX, CVI and DLL but they are few and far between.
2) If LabVIEW is one of you primary adapters, are your VIs typically already mass compiled (i.e. up to date with the version of LabVIEW you are using)? Isn't it bad to mass compile all the native VIs in LabVIEW... not to mention TestStand installs older VIs. I doubt everything is mass compiled.
3) Approximately how many code modules are you loading and how much time is it taking? Depends on the sequence file. We probably average about 15 per sequence with about 20-30 sequences per file/build. A lot of them are the same VIs (i.e. Instrument Control VIs, UUT Control VIs, etc..) just different parameters. If we do a snapshot view in ClearCase it loads a ton faster than a dynamic view. However, this could be chalked up to the server communication.
I did notice the other day, and this is only with dynamic views, that when there are two VIs named the same put pointing to different libraries (e.g. Initialize.vi in different instrument libraries) that the build on deployment takes about 1.5-2 hours. It finally builds but WOW! Whereas the same build in my snapshot view will take 5-10 minutes. In the build log it had a bunch of colision errors pointing out those VIs with the same names. I have gotten the dynamic view to build in 10-20 minutes on different sequences with about the same amount of calls though. ODD??
02-15-2011 01:14 PM
Are you talking about module loading time being slow or deployment building time or both? Just deployment for the really long times. Loading is slower with dynamic view.
Are you loading VIs from TestStand using the LabVIEW development environment or the runtime engine? What version? If using the development environment, are the VIs saved with the same version of LabVIEW. We use the dev env when loading VIs. We have standardized to Dev Suite Feb 2009 which is LV 8.6.1 and TS 4.1.1. All of our VIs (meaning the VIs we created) are saved in the same version of LV. We will upgrade in a few months to Feb or May of 2010.
Are you using source code control integration with TestStand (i.e. using teststand workspaces with source code control integration enabled). Are you saying that TestStand is loading things faster or slower based on your view in ClearCase? or am I misunderstanding what you meant? We do have the source code control integration turned on. It makes it nice for working in workspaces (quickly check things in and out). Refer to the next question to understand the ClearCase part of this question.
I'm not that familiar with ClearCase, what exactly is the difference between a snapshot view and a dynamic view? How would this affect which VIs are referenced? Think of the dynamic view the same way you use Perforce currently. All of the files reside on the server and nothing is local. With a snapshot view copies are put on the local PC so it's as if you are working out of a local directory. We switch up our search directories depending on which view we are using. With a snapshot view you can select only which folders you want to copy over locally. However, we have our root folder for our team and our search directories for dynamic and snapshot cover the same amount of folder depth and space. Everything we do is in ClearCase (including our instrument libraries) except for native VIs.
For your 15 files with 20-30 sequences, about how many VIs would you say are loaded (doesn't need to be exact, just a rough estimate)? And how long does it typically take for them to load? I just want to make sure that when we look at the performance of the LabVIEW adapter that we make sure we are seeing the same thing as you. I would guess maybe 120 VIs per file. Unless you take into account all the SubVIs (which I'm not sure when those get loaded). With a snapshot view the loading is negligible. I'd say less than 30 seconds. With a dynamic view we are looking at a couple of minutes. The deployment is the bigger issue. However, this may not be due to loading but rather linking.
02-16-2011 10:30 AM
Hi Doug,
I did a small Start(F5 and entering ProcessSetup) benchmark in our UI
over some machines and sequenceFiles:
MaschineA: P3 WIN2K TS4.0 MaschineB: QuadCore XP TS4.2
Seq 1: 8300 Steps LV 5 - 10 VI modules 40s not possible Wrong LV RTE!
Seq 2: 1275 Steps DLL 10-15 differnt modules 2min42s 2min8s
Seq 3: 545 Steps DLL 2-3 modules 8s 5s
Seq 4: 1050 Step DLL and DLL Steptype --- 7s
If i have some more time, i will send some data for TS4.5
Regards
Juergen
03-07-2011 04:47 PM
Hi jiggawax & Juergen,
Wow, sorry for the (very) late reply, I somehow had a weird rule on my email subscriptions from back when I was an AE and didn't see the replies. My applogies...
Anyway, thanks for the feedback, let me see if I can summarize your points, and please correct me if I am wrong:
Main Concerns:
Module Load/Unload Overhead:
Deployment:
Additional Feedback:
Questions:
Again, sorry for posting a question, and then dropping off the thread! Thanks for your feedback!