NI TestStand

cancel
Showing results for 
Search instead for 
Did you mean: 

Feedback on TestStand Performance

Hi all,

I am the product manager for NI TestStand and I wanted to get your feedback on TestStand performance.
We know that performance is an important concern for a lot of our users and in order to improve TestStand performance, we need to truly understand what areas of performance are most important to our users.

I would love to hear your thoughts on:

  1. What aspects of performance are most important to you?
    Example: Total test execution time, report generation time, parallel executions, TestStand launch time, code module load/unload overhead, deployment time, edit-time responsiveness, etc
  2. In what aspects of performance does TestStand meet/exceed your needs?
  3. In what aspects of performance does TestStand need to improve?
  4. Do you currently benchmark TestStand or your test system currently in some way? If so, can you please elaborate?


Thank you very much for your feedback!

Jervin Justin
NI TestStand Product Manager
0 Kudos
Message 1 of 22
(3,317 Views)

Hey Justin,

 

Some of my thoughts:

 

  1. What aspects of performance are most important to you?  Load/Unload Overhead.  We use ClearCase as our repository and the Search Directories have to encompass all of our instrument libraries as well as our own libraries.  They tend to get rather large and deep.  This also affects deployment and edit-time responsiveness... especially when we use a dynamic view (that's due to the server communication though).  We are using a lot of shared VIs.
    Example: Total test execution time, report generation time, parallel executions, TestStand launch time, code module load/unload overhead, deployment time, edit-time responsiveness, etc
  2. In what aspects of performance does TestStand meet/exceed your needs? Our deployed systems seem to execute exactly how we want them.  That is if we get them set up correctly.. 🙂
  3. In what aspects of performance does TestStand need to improve? DEPLOYMENT!!! 'nuf said.  Ok maybe I should say a few things-  Why do we have to analyze the source and then when we go to build does it have to re-link all the VIs?  Doesn't the analyze source find all of the VIs already for us?  It would be nice if it could remember all that when we go to build.  I do understand that there is more going on when the build happens but it does search for the VIs for quite some time.  And if we ever have 2 VIs named the same...which we do because ALL of the NI/Agilent instrument driver libraries have an Initialize.vi, then it seems that TS goes back and forth and it's not able to make up it's mind for quite a while.  Then it finally settles on one.
  4. Do you currently benchmark TestStand or your test system currently in some way? If so, can you please elaborate? Nope.
jigg
CTA, CLA
testeract.com
~Will work for kudos and/or BBQ~
Message 2 of 22
(3,278 Views)

Hi Justin,

 

1.) I am of the same opinion as Jigg. The Load/(Unload) behaviour of TS.

My major focus is on the Load behaviour. There was always a bad impact when changing an old PC into

of state-of-art in productionfield or development and after pressing Start(F5) it tooks

the same time till the execution was starting (jumping into PreUut). Sometimes it took 3-5 minutes! 

And you wonder why did i changed the maschine ? 

It seems that loading modules on TS4.5 is even faster than the old ones.

 

Another topic of performance is editing the steps itself. In the TS-Idea exchange

there are a lot of entrys dealing with this. At the moment no one is under consideration !

 

2.) TS has some features about performance for example the "Profie Resource Usage"

but i am honest i have never used it. But what should i answere here ?! Lets better go to 3

 

3.) There should be must a callback that shows the current state of loading modules. With this

you will show the user that Teststand is still alive after F5 and it would be make sense to get some cup of coffee now.

- Hm I should place this in Idea-Exchange.

That in the Idea-Exchange the "Under Consideration" counter is beginning to start counting!

 

4.) No, just feelings and it seems it is geeting better.

 

Regards

 

Juergen 

 

 

 

 

 

--Signature--
NIWeek 2017: Standardized HIL Test Automation With TestStand
NIWeek 2016: Capturing ATE Requirements With Requirements Gateway

Feedback or kudos are welcome
Message 3 of 22
(3,256 Views)

For module loading speed issues, please provide more details such as the following:

 

1) What adapter(s) do you primarily use?

2) If LabVIEW is one of you primary adapters, are your VIs typically already mass compiled (i.e. up to date with the version of LabVIEW you are using)?

3) Approximately how many code modules are you loading and how much time is it taking?

 

The above information will help us better diagnose the issue and target the area with the greatest problem.

Thanks for any details you are able to provide us.

-Doug

0 Kudos
Message 4 of 22
(3,246 Views)

For issues on the idea forum that you think are related to this topic and should be addressed please let us know specifically which ones you think are most important.

 

Thanks,

-Doug

0 Kudos
Message 5 of 22
(3,244 Views)

1) What adapter(s) do you primarily use? LabVIEW.  We have some .NET, ActiveX, CVI and DLL but they are few and far between.

2) If LabVIEW is one of you primary adapters, are your VIs typically already mass compiled (i.e. up to date with the version of LabVIEW you are using)?  Isn't it bad to mass compile all the native VIs in LabVIEW... not to mention TestStand installs older VIs.  I doubt everything is mass compiled.

3) Approximately how many code modules are you loading and how much time is it taking? Depends on the sequence file.  We probably average about 15 per sequence with about 20-30 sequences per file/build.  A lot of them are the same VIs (i.e. Instrument Control VIs, UUT Control VIs, etc..) just different parameters.  If we do a snapshot view in ClearCase it loads a ton faster than a dynamic view.  However, this could be chalked up to the server communication.

 

I did notice the other day, and this is only with dynamic views, that when there are two VIs named the same put pointing to different libraries (e.g. Initialize.vi in different instrument libraries) that the build on deployment takes about 1.5-2 hours.  It finally builds but WOW!  Whereas the same build in my snapshot view will take 5-10 minutes.  In the build log it had a bunch of colision errors pointing out those VIs with the same names.  I have gotten the dynamic view to build in 10-20 minutes on different sequences with about the same amount of calls though.  ODD??

jigg
CTA, CLA
testeract.com
~Will work for kudos and/or BBQ~
0 Kudos
Message 6 of 22
(3,208 Views)

 

Hi Jiggawax,
Thanks for the additional details. I have some follow up questions.
Are you talking about module loading time being slow or deployment building time or both?
Are you loading VIs from TestStand using the LabVIEW development environment or the runtime engine? What version? If using the development environment, are the VIs saved with the same version of LabVIEW.
Are you using source code control integration with TestStand (i.e. using teststand workspaces with source code control integration enabled). Are you saying that TestStand is loading things faster or slower based on your view in ClearCase? or am I misunderstanding what you meant?
I'm not that familiar with ClearCase, what exactly is the difference between a snapshot view and a dynamic view? How would this affect which VIs are referenced?
For your 15 files with 20-30 sequences, about how many VIs would you say are loaded (doesn't need to be exact, just a rough estimate)? And how long does it typically take for them to load? I just want to make sure that when we look at the performance of the LabVIEW adapter that we make sure we are seeing the same thing as you.
Thanks for any additional details you are able to provide,
-Doug

@~jiggawax~ wrote:

1) What adapter(s) do you primarily use? LabVIEW.  We have some .NET, ActiveX, CVI and DLL but they are few and far between.

2) If LabVIEW is one of you primary adapters, are your VIs typically already mass compiled (i.e. up to date with the version of LabVIEW you are using)?  Isn't it bad to mass compile all the native VIs in LabVIEW... not to mention TestStand installs older VIs.  I doubt everything is mass compiled.

3) Approximately how many code modules are you loading and how much time is it taking? Depends on the sequence file.  We probably average about 15 per sequence with about 20-30 sequences per file/build.  A lot of them are the same VIs (i.e. Instrument Control VIs, UUT Control VIs, etc..) just different parameters.  If we do a snapshot view in ClearCase it loads a ton faster than a dynamic view.  However, this could be chalked up to the server communication.

 

I did notice the other day, and this is only with dynamic views, that when there are two VIs named the same put pointing to different libraries (e.g. Initialize.vi in different instrument libraries) that the build on deployment takes about 1.5-2 hours.  It finally builds but WOW!  Whereas the same build in my snapshot view will take 5-10 minutes.  In the build log it had a bunch of colision errors pointing out those VIs with the same names.  I have gotten the dynamic view to build in 10-20 minutes on different sequences with about the same amount of calls though.  ODD??


 

0 Kudos
Message 7 of 22
(3,196 Views)
Are you talking about module loading time being slow or deployment building time or both? Just deployment for the really long times.  Loading is slower with dynamic view.
Are you loading VIs from TestStand using the LabVIEW development environment or the runtime engine? What version? If using the development environment, are the VIs saved with the same version of LabVIEW.  We use the dev env when loading VIs.  We have standardized to Dev Suite Feb 2009 which is LV 8.6.1 and TS 4.1.1.  All of our VIs (meaning the VIs we created) are saved in the same version of LV.  We will upgrade in a few months to Feb or May of 2010. 
Are you using source code control integration with TestStand (i.e. using teststand workspaces with source code control integration enabled). Are you saying that TestStand is loading things faster or slower based on your view in ClearCase? or am I misunderstanding what you meant?  We do have the source code control integration turned on.  It makes it nice for working in workspaces (quickly check things in and out).  Refer to the next question to understand the ClearCase part of this question.
I'm not that familiar with ClearCase, what exactly is the difference between a snapshot view and a dynamic view? How would this affect which VIs are referenced? Think of the dynamic view the same way you use Perforce currently.  All of the files reside on the server and nothing is local.  With a snapshot view copies are put on the local PC so it's as if you are working out of a local directory.  We switch up our search directories depending on which view we are using.  With a snapshot view you can select only which folders you want to copy over locally.  However, we have our root folder for our team and our search directories for dynamic and snapshot cover the same amount of folder depth and space.  Everything we do is in ClearCase (including our instrument libraries) except for native VIs. 
For your 15 files with 20-30 sequences, about how many VIs would you say are loaded (doesn't need to be exact, just a rough estimate)? And how long does it typically take for them to load? I just want to make sure that when we look at the performance of the LabVIEW adapter that we make sure we are seeing the same thing as you.  I would guess maybe 120 VIs per file.  Unless you take into account all the SubVIs (which I'm not sure when those get loaded).  With a snapshot view the loading is negligible.  I'd say less than 30 seconds.  With a dynamic view we are looking at a couple of minutes.  The deployment is the bigger issue.  However, this may not be due to loading but rather linking.
jigg
CTA, CLA
testeract.com
~Will work for kudos and/or BBQ~
0 Kudos
Message 8 of 22
(3,181 Views)

Hi Doug,

 

I did a small Start(F5 and entering ProcessSetup) benchmark in our UI

over some machines and sequenceFiles:

 

                                                                          MaschineA: P3 WIN2K TS4.0   MaschineB: QuadCore XP TS4.2

Seq 1:  8300 Steps LV 5 - 10 VI modules                              40s                            not possible Wrong LV RTE!

Seq 2: 1275 Steps DLL  10-15 differnt modules                   2min42s                                   2min8s

Seq 3: 545 Steps DLL  2-3 modules                                     8s                                            5s

Seq 4: 1050 Step DLL and DLL Steptype                               ---                                           7s

 

If i have some more time, i will send some data for TS4.5

 

Regards

 

Juergen

 

 

 

--Signature--
NIWeek 2017: Standardized HIL Test Automation With TestStand
NIWeek 2016: Capturing ATE Requirements With Requirements Gateway

Feedback or kudos are welcome
0 Kudos
Message 9 of 22
(3,155 Views)

Hi jiggawax & Juergen,

 

Wow, sorry for the (very) late reply, I somehow had a weird rule on my email subscriptions from back when I was an AE and didn't see the replies. My applogies...

 

Anyway, thanks for the feedback, let me see if I can summarize your points, and please correct me if I am wrong:

 

Main Concerns:

  • Module Load / Unload overhead
  • Time it takes to create a deployment

 

Module Load/Unload Overhead:

  • It takes a really long time for TestStand to load modules into memory for execution
  • In your cases, this is particularly refering to LabVIEW VIs
  • It can sometimes take 3-5 minutes to load modules before the test sequence begins execution, during which test operator simply stares at the screen
  • Sidenote: There should be some visual indicator (or callback) to the operator that modules are loading, so they don't think that TestStand crashed
  • Load time problem is exasberated when using ClearCase - because there are a lot of Search Directories that are very deep.
  • LabVIEW Versions - Jiggawax is using LV 8.6.1, possible that VIs are not mass compiled. Using the Dev Environment.
  • Upgrading the computer did not seem to help with load times

Deployment:

  • Creating a deployment can take a very long time
  • Again in your cases, this is particularly referring to deployments that include LabVIEW VIs
  • It would be nice if we cached some of the information when we did the build analysis, and used it during the actual build
  • Possible corner case: ClearCase with 2 VIs with same names in different libraries with dynamic view took ~2 hours to build

Additional Feedback:

  • Editing step settings can be improved. See idea exchange ideas
  • Overall, performance seems to be getting better

 

Questions:

  1. Jiggawax: If the VIs are mass compiled, do you see an improvement in module load time?
  2. Juergen: What version lf LabVIEW are your VIs written in? Are they mass compiled? Do you use the RTE or the dev environment?
  3. Both: What kind of module load/unload settings do you typically use?
  4. Juergen: Can you specify which ideas on the idea exchange particularly you think would help with editting steps from a performance point of view?
  5. Both: Any other performance related feedback, particularly on the execution (run-time) side?

Again, sorry for posting a question, and then dropping off the thread! Thanks for your feedback!

Jervin Justin
NI TestStand Product Manager
0 Kudos
Message 10 of 22
(2,987 Views)