NI TestStand Idea Exchange

Community Browser
cancel
Showing results for 
Search instead for 
Did you mean: 
Post an idea

Hi there.

 

This may seem like a radical idea, and probably my most blunt idea I've ever posted but please bear with me:

 

NI folks working on TestStand as software, I don't mean to be offensive. I understand being in the position of having to make incremental changes, eventually outgrowing the original design of the software. I'm just saying what I think needs to be said and is long overdue.

 

Anyone who has had to customize the NI Report Generator will know that it is a complete hodgepodge of incremental changes from over the years.

 

As a high level summary:

  • Within NI_ReportGenerator.seq, all of the report types (ATML, XML, HTML, text, etc.) are handled in the same sequences
  • Depending on the report type, different steps are executed via preconditions, Post Actions (essentially GOTOs ☠️) and somewhat rare if-then flow logic.
  • This tight coupling of the report types makes it really difficult to follow the flow of steps that execute, and it's also just bad software design.
  • Further, there are sequence calls set by expression depending on the report type, making it even harder to follow. I'm guessing that this was supposed to be extensible by adding more ReportGen_<type>.seq files, but we're long, long past that being realistic.
  • Nomenclature is often inconsistent
  • Naming of steps often doesn't match what's really happening.
  • Preconditions and post action conditions are very often implicit.

 

AllOf( This, that, AnyOf( Sometimes the other, or another still), ! The other options that might apply)

&& !( Other implicit conditions because post actions in prior steps precluded execution of this step )

 


Highly technical details follow:

First off, in the TestReport callback, if report generation is disabled, the message to display is stored in Locals.ReportSection.AsReportSection.Body. Tricky, tricky, you didn't fool me! 😄 Yet, why not use an appropriately named variable?

 

Also, in the configuration dialog, the report style is referred to as the "Report Format", but the report format in the code is independent of the report style. The nomenclature isn't consistent. Along this theme, in the c code, there is a struct for ReportOptions, but it's not the report options from the TestStand sequence, and it's gradually populated over several consecutive function calls.

 

Taking ATML, for example, depending on which ATML standard is selected, the execution flow varies substantially, is inconsistent, counterintuitive, and seemingly arbitrary:


ATML 2.02

  • In the TestReport callback, if and only if the ATML standard is 2.02, the step "Get Report based on ReportStyle" is called.
    • ...but it's only for one style, and not even a style, which would be "xml"
    • This Get<report type>Report (GetATMLReport) sequence only exists for ATML (no other formats) and is only called for ATML 2.02, thereby completely removing any justification for a sequence call by expression.
  • GetAtmlReport then calls one of two functions in ATMLSupport.dll, depending on whether the caller has initialized a report section 
  • The report section is always initialized two calls up in "Model Plugin - UUT Done", so there is one case with a DLL call that never executes and is dead code.
  • Ultimately, the Get_Atml_ReportSection function is called in ATMLSupport.dll.
  • In ATML_Report.c, this calls Get_Atml_Report_Impl, which is where most of the business of appending report strings takes place

ATML 5.00

 

In the TestReport callback:

  • Locals.ReportGenSequenceFileName is determined by looking for the first four characters of FileGlobals.ATML202StandardReportStyle. (i.e. ATML) This is regardless of the ATML version, even though there are respective constants for each ATML version. You might incorrectly assume that it only applies to ATML 2.02
  • This time, by precondition we skip execution of "Get Report based on ReportStyle", which is a misnomer and exclusively for ATML 2.02
  • Counterintuitively, we also skip "Get Report Body (DLL)", even though we've configured body generation to be via DLL. By precondition, this step is implicitly for any format other than "xml", which I guess means ASCII and HTML? But they're not explicitly named.
  • No, we execute "Get Report Body (Sequence)" instead, even thought we're configured to use a DLL. Via various preconditions and post actions, this step is implicitly: Not for ATML 2.02, not for ASCII, and not for HTML. So I guess that means ATML 5.00, ATML 6.01 and XML? 
  • "Get Report Body (Sequence)" is yet another call by expression to an external reportgen_<format>.seq
  • "Get Report Body (Sequence)", when called for these XML based report styles, does not use a sequence. In all of those cases, the AddReportBody sequence it calls calls a DLL and only has half a dozen steps, so it's not a sequence, really.
  • Within AddReportBody, there are two possible DLL calls based on whether Parameters.ReportSection is null, but it's never null because the calling sequence always initializes Parameters.ReportSection. (More dead code)

I thought about talking about ATML 6.01, and I could go on, and on... and on. I think you all get the idea.

This is an absolute nightmare to navigate and very messy to customize without an extreme makeover crying edition.

Surely we can do better.

 

Thank you for reading my missive.

 

Mr. Jim

With the removal of the GAC from TestStand, it's now the user's responsibility to include common .NET assemblies in their source to be leveraged in TestStand.  It would be great if I could go the menu, launch the Nuget Package Manager, download packages from nuget.org (and my local Artifactory repo), and unpack these assemblies into a specified target folder. 

 

While I'm primarily focused on .NET and nuget, the same thing could and should be provided for VI Package Manager and lvlibs, lls, vis, etc.  

As someone whom regularly maintains custom process models for my team, there are times when I wish to harvest data from 'any MainSequence run' without requiring my individual developers to remember to set specific custom-parameters or 'runstate.root.* or Locals.ResultList[] values within their test programs correctly every time. 

It would be great if there were hooks, beyond simple parameter passing, that could allow me to dynamically 'reach in' to active execution context of MainSequence ideally 1x at start and end of MainSequence execution (i.e. just after context fabrication, and just before context garbage collection).

In this way, I could use API to monitor data that usually gets garbage collected before the model can 'see' it.

i.e. if I see they have a local variable in their test, Locals.Debugging_Active == true, perhaps I could catch that at the framework level, and update the report header to say 'possibly not production data!',   even if developer forgot to correctly set the 'is Validation Data' flag that is formally available.

In an ideal world we would have strict process & TS style rules to enforce that all developers must set key variables as part of their executions, but also having a scrappy way to catch test-internal context issues within process model (even if this does break the paradigm of immediately garbage collecting sequence specific internal data) would be a great fall-back strategy for teams where they have 1 or 2 test experts and a much larger pool of non-expert test developers using TestStand. 

I upgraded from TestStand 2019 SP1 to TestStand 2024 Q4 and the UI looks really bad, with black and white theme with icons not recognizable and hard to look at for longer periods.

1) Please add function to have themes for TestStand like Visual Studio, some examples:
Dark theme
Classic theme (Older version of TestStand)

2) Make it easier for users to change the UI and function to zoom in/out in TestStand (like it´s possible in newer versions of LabView).

Iam thinking about expressions field should have zoom in/out function.

 

3) Change it back how it was for showing Sequences and Variables (TestStand).
It´s not better to show only one or the other. I would like both to be shown at the same time as it was in previous TestStand versions.

Step type: Label

Tab: Label

Field: Label Description

 

Most text field can zoom in, but not the Label Description.

It would a lot easier to read if one can ZOOM IN in the field.

 

When building test system, when I have analog measurement, I'm always using multiple measurement values for my test. Test is often based on mean and standard deviation.

 

I suggest to have some function in TestStand to calculate some standard statistics calculation like mean, standard devation, variance... on array (at least 1D array)

 

I don't count how many times I wrote this function or implement it in TestStand.

I would like to request a change in the behavior of the Sequence Editor when dragging and dropping multiple subsequences into, for example, the Main Sequence as Sequence Call steps.

 

Currently, after dropping, the view automatically jumps to one of the dragged subsequences. A much better UX/DX would be if the view remained at the drop location instead of shifting focus to another sequence.

I would like to suggest adding a public API to the Sequence Editor in TestStand.

 

Recently, I watched a recording featuring members of the TestStand R&D team, where they explained that users often request various features in the Sequence Editor, even though they already have access to the TS API and TS UI Controls, which allow them to create their own custom user interfaces. Meanwhile, building a custom operating system just to play Heroes of Might and Magic is not standard practice — we buy a fully functional system and run the game on it. Similarly, when writing sequences in TestStand, we purchase a framework along with a ready-made GUI that should provide a comprehensive set of features for creating test sequences. The Sequence Editor is a paid tool, and since users pay for it, it should offer value out-of-the-box without requiring additional development of functionalities that users believe should already be standard.

 

For this reason, I believe that a paid GUI like the Sequence Editor should be enhanced with an API that allows programmatic control of its functionality.

 

Examples of functionalities that interest me:

  1. Managing the Output Messages Pane

    • Programmatically clearing messages.
    • Exporting messages to a file.
    • Opening or hiding the pane.
  2. Managing UI Settings

    • Programmatically loading a saved layout.
    • Dynamically controlling the layout during runtime.
  3. Changing Pane Colors

    • Many users currently complain about the lack of a dark mode in TestStand. A partial dark mode can be achieved by changing the background colors of panes. Most (if not all) panes have a Color property that allows color changes, but for some reason, this doesn’t work in the Sequence Editor (I attempted to achieve this by retrieving references to panes through the Application Manager returned by Engine Internal Options, but unfortunately, without success).

 

These are just examples of features that would be useful to me today. However, long-time TestStand users could likely identify many more ideas for such API. A public API would open the door to much more advanced and personalized use of TestStand, while simultaneously increasing its value for a wide range of users.

 

 

It would be great to be able to generate a FileGlobal or a local variable by right clicking a detected field in a Sequence step and choosing the relevant option as shown in the image below:

TomerKigel_0-1732697800669.png

That is instead of manually having to do it.

Currently, you have to choose between these options on each file (and the only indication of the choice is in the sequence file properties dialog).  Both options can be quite useful in the Parallel and Batch Model (of course there isn't much of a difference in the Sequential Model).  Clunky workarounds also exist like using Shared File Globals and having arrays for anything per-execution or using Separate File Globals while putting anything shared in Station Globals or in LabVIEW Functional Globals, but I think native support for both in the same file would be much cleaner and easier.

This could be just having two "FileGlobals" categories in the Variables pane (I prefer this), or it could be a right-click option on each individual File Global.

 

TomOrr0W_1-1729015156875.png

TomOrr0W_0-1729014466032.png

 

Provide a UserManagementCallbacks.seq to replace the currently hard-coded built in TestStand User management so that we can implement our own site-specific user management.

 

Sequences should allow for

 

1) Retrieving and returning of a user container from a database or file of our choosing (login)

2) Storage of new user credentials to a database or file of our choosing

3) Adding a New User with the option to reject (i.e. we don't like the name "!#%ThisGuysName!!!")

3) Deleting a User with option to reject deletion (i.e. we don't want to allow deleting user "Administrator" or "ThisReallyImportantUser"

4) Updating User Information

5) Logout, return a default user or shutdown on logout.

6) Retrieving of All Users available for login.

7) Check if a UserExists

 

This would be in addition to the FrontEndCallbacks that handle the Login/LogOut process.

 

 

When building any type of distribution it would be nice to have the following actions

 

1) Pre-Image Build Script -> Execute windows command or batch file

2) Post-Image Build -> Execute windows command or batch file 

3) Post-Distribution Build -> Execute windows command or batch file 

 

The purpose of the three entry points is to allow the user to preform actions of their choosing.

 

Action #1 is to be run just prior to Starting Analysis to allow the user to preform any automated actions prior to analyzing the build project.

Action #2 is to be run after the deployment image structure and all files are created on disk but before the distribution package is create. The purpose of this action is to allow the user to inject any files, update files, preform verification or hashing and so on. In my specific use case, I need to create a file that has relative paths and a hash value for each file being deployed. This process is automated but currently run manually prior to a package build.

Action #3 is to be run after the Distribution package is made. Automate any file steps such as uploading the file to a network location, or generating an automated email.

 

Each action should call the script with parameters such as path info, deployment name or provide a list of variables that users can choose to have added to the command line for each action.

 

If the action script or command returns a non-zero value the Build process should terminate.

 

Re-Posted from (https://forums.ni.com/t5/NI-TestStand/TestStand-Deployment-Utility-Scripts/td-p/4373139)

 

NI_Follower_0-1710508258374.png

I have created a custom type variable. Now to update the contents in this variable, without disturbing the original definiton - I need to create a container and then copy past all these variables inside. 

 

If there is a right click option to remove the custom type and make it as a container which can be then editable it will be helpful.

Modifying names in interface are never good idea, but some times it is must.

No matter is the change made in LabVIEW Front Panel or in TestStand type cluster mapping. Or the cluster type is changed for some other reason. The changes needs to be applied in LabVIEW module call parameters.

When you have multiple sequence files and multiple calls to same VI or with same cluster type in each of them, you would not really want to go trough the steps one by one.

 

Ideas:

- Cluster mapping changes apllied with LabVIEW Protorype reload

- Cluster mapping changes applied to hole sequence file or/and to all same VI calls in the sequence file

- Need of cluster mapping change applying notified with Analyzer

When test cycles run and fail for some reason, sometimes LabVIEW will either crash or get hung and wont close.

 

Even when exiting or restarting Teststand, LabVIEW does not close properly...or re-open properly, locking any HW resources. Non-technical savvy operators are unsure / do not know what to do.

 

If the operators were notified with a pop-up informing them of a windows restart is needed to clear LabVIEW PRIOR to even the login prompt, that saves a lot of false start/false fails and headaches.

 

We have implemented this in the "FrontEndCallbacks.seq" and works great for false-fail runs.

 

Teststand should have this implemented built-in.

Hi NI Teststand developers

Don’t you find it hard setting up Teststand with all the different settings possible, scattered in al the different locations?

 

Why is it not possible to put all this settings in one form with tabs. (like in MS VisualStudio)

Especially the Cfg directory is  packet with possible stumbling rocks.
This improvement includes the configuration menu.

 

I usually spend a lot of time just setting up TS. When finally arriving in the deployment state. I stumble again trough this setting forgetting important things because they are scattered throughout Teststand.

 

It would be an immense improvement of the TS usability

 

 

If you start improving this, don’t forget to adjust the editable field sizes of forms etc., so they can be read and edited in a practical way.

 

Another thing I find not really user friendly, is the form settings lock. When I try to load a workspace and the lock is set (worked in TS2019), I get an error. Why this?

 

It would be nice to hear your statement to this,  and when you find time to improve this.
If you need more improvement hints  (there are some more) , don’t hesitate to contact me.

 

When user opens the Offline Processing Utility and at the same time starts to type on the keyboard the user can accidently rename the profile.

(Attached a screen-recording to visualize)

It would be great if there was no selected row or column when starting/opening ORPU. The renaming of the profile disturbs the production since the database logging will not work as expected.

 

When we start ORPU we already have the '/tray' enabled but somehow its still possible to accidently rename the Profile.

My team uses Analyzer Rules to enforce best practices for developers across sequence files and workspaces with great results.  However we would love to be able to also provide rule checking and warnings for developers on some basic *.tsd file content  such as Installer Name field, and build-path & installation-path/image-path, etc...  New (or rusty!) developers often miss steps in their configuration and needlessly struggle with badly behaving deployments.

As TSD files do not appear to have an API available from TestStand at this time nor conform to  a json/xml syntax internally, for easy parsing via 3rd party tools, some method(s) for 'reading' file (or converting to readable syntax) may also be required in order to build rules effectively?

Hi all,

 

wouldn't be really nice if one could copy and paste step properties?

  1. copy each property idividually...OR
  2. copy selected properties....OR
  3. copy ALL properties (well, like #2 but select all...haha...oops...yes)

T999_0-1694113001941.png

 

 

Hi folks,

 

When writing TestStand UI components using LabVIEW, there are a lot of scenarios where using a callback VI to handle a TestStand event is the right move. As of writing this, that's the only way.

 

There have also been plenty of times when I would have preferred to handle a TestStand event directly within a LabVIEW event structure. In these kinds of scenarios my code would be less complicated if we had the ability to directly register TestStand engine events with LabVIEW's "Register for Events" node.

 

Years ago when learning TestStand, I tried to do this out of intuition and I was kind of surprised that it wasn't supported.

 

Below is an illustration of what I'm proposing.

 

TestStandDynamicEvents.png