NI Package Builder hangs on Initializing Teststand Engine

[Phil.Lundeen]

We have a Continuous Integration (CI) pipeline for our codebase in which we automate unit tests, build executables and packages, etc. In the case of building packages, we have situations where we build packages from LabVIEW Project (*.lvproj) context, and others where we leverage the standalone NI Package Builder; in both cases, we try to automate the task via command-line calls.

The NI Package Builder command line interface (nipbcli.exe) is helpful to us....when it works. But it seems to randomly hang indefinitely on initailization, showing the following behavior:

System summary
  Date 04/13/2022 01:10:05
  NIPB version 20.6.0.1227
  NIPB bitness 64-bit
  NIPM version 21.3.0.49255
  Operating System Windows 10 64-bit
Initialization starting...
Initializing TestStand engine...

 and then it never gets any further.

Does anyone know what could be happening to cause it to hang there, or (more importantly) how to prevent it from hanging there? I have tried programmatically restarting the TestStand background service just prior to initiating a package build, but that didn't seem to help. 

Even more frustratingly, the package it is hanging on for me right now has no Teststand dependencies whatsoever. It is just a package containing an exectuable and some pre/post install script batch files. I would love it if I could just convince NI Package Builder not to try to invoke the TestStand Engine at all, but I don't know if that is possible. Here is how I am invoking the CLI (via Powershell script in our CI pipeline):

& "C:\Program Files\National Instruments\Package Builder\NipbCli.exe" --open="$pbsFilePath" --build=packages --teststand-env-do-not-use-for-discovery --teststand-env-do-not-use-for-staging

Anyone have any tricks/suggestions here? Thanks!

[Scott_Richardson]

Phil -

This is not a known issue. I would agree that the loading of the TestStand runtime appears to be hanging.

Some initial questions:

1. After you exit the hung invocation, can you look at the below directory and share the more detailed log file associated with a hung invocation. It may have more information.
   %LocalAppData%\National Instruments\NI Package Builder\Logs\*.txt
2. What versions and bitnesses of TestStand do you have installed and what is the active version in TestStand Version Selector?
3. It might be helpful to get a minidump file with heap. Do you have a recent version of Visual Studio? If yes, you can use Debug>>Attach To Process (to hung NIPBEditor.exe), Debug>>Break All, Debug>>Save Dump As, and share.

[Phil.Lundeen]

1. After you exit the hung invocation, can you look at the below directory and share the more detailed log file associated with a hung invocation. It may have more information.
   %LocalAppData%\National Instruments\NI Package Builder\Logs\*.txt
   • There are only a handful of logs in this directory, and the latest one was from 2 days ago, even though I have performed several builds (both successful and unsuccessful) in the last 24 hours. So apparently it doesn't log consistently. 😞
2. What versions and bitnesses of TestStand do you have installed and what is the active version in TestStand Version Selector?
   • 
3. It might be helpful to get a minidump file with heap. Do you have a recent version of Visual Studio? If yes, you can use Debug>>Attach To Process (to hung NIPBEditor.exe), Debug>>Break All, Debug>>Save Dump As, and share.
   • I don't have Visual Studio on this build machine.

It might help if I understood:

• Why does Package Builder think it needs to talk to the TestStand Engine at all?
• What is happening when it is "Initializing Teststand Engine"? Is it making DLL calls? Opening a TCP socket (like with VI Server)? Some other interprocess communication?

As of now, the behavior is completely nondeterministic. I can see no rhyme or reason as to why it works sometimes and not others.

[Scott_Richardson]

Phil -

You asked "Why does Package Builder think it needs to talk to the TestStand Engine at all?":

All of NIPB's knowledge about TestStand is in the plug-in, so when the installed plug-in is found, it initializes and part of the initialization is to determine what version of TestStand is active, and loads that engine. That is just the current design.

You asked "What is happening when it is 'Initializing Teststand Engine?'":

The TestStand plug-in is loading the TestStand runtime engine via ActiveX, similar to how a TestStand operator interface would load the engine. There are calls into the TestStand API that are likely not returning.

So, looking back at some earlier development notes we did hit an issue where our internal CI did intermittently hang and I believe it was only with TestStand 2019. The following TestStand KB discusses a case where an error can occur in one of the adapters on some systems:

Error -17501 Unable to Load Test Environment CVI Adapter When Opening TestStand

I believe that when we followed the suggested change in the DCOM Config settings as indicated, the issue was resolved. I believe that TestStand 2020 and later have the corrected DCOM Config settings. I would try this first.

If you want to disable the TestStand plug-in you can remove or just rename the file extension on the following file which will prevent NIPB from supporting TestStand:

"C:\Program Files\National Instruments\Package Builder\NationalInstruments.PackageBuilder.Plugins.Teststand.dll"

Note that the NIPB installation is fairly siloed, so for now you can close the directory to maintain the original installation and in the cloned directory remove the DLL. Then launch the version you need.

If we need to look at this further in some way, we will need to get a dump file for the hung process:

1. download dumpproc from Microsoft
2. unzip to a location accessible from a command prompt
3. execute "dumpproc -ma NIPBEditor.exe" to capture the process while hung
4. share the output dmp file output to the working directory of the command prompt

Let us know how it goes.

[Phil.Lundeen]

Thanks, Scott. I tried updating the DCOM config settings for now, and I'll see if it starts working more consistently.

If it continues to hang, I might try removing/renaming the teststand DLL, but at some point we might try automatic packaging of Teststand sequences (I've dabbled in this a bit already), so I don't necessarily want to preclude that possibility unless I absolutely have to.