TestStand command line options

[A.E.M]

Hi All.

I'm trying to understand how to interface with TestStand from the command line, so I can call it my from CI server (Jenkins).

I have so far found these article:

Forum: Using TestStand And Jenkins

Tutorial: Getting Started with TestStand and Jenkins

TestStand command line options

Sequence Analyzer command line options

For now I just working on the example project 'Computer Motherboard Test Sequence' that ships with TestStand.

I am able to open TestStand and run the sequence with this command:

SeqEdit.exe /run MainSequence "C:\your-path\Computer Motherboard Test Sequence.seq"

I have one issue though: The command returns immediately, so how will Jenkins know when the sequence is completed running?

I would also like to run the Analyser, which I can do manually by pressing the "Analyzer" button in the sequence. But if I want to do this operation from the command line, it seems like I need an Analyzer project? I have tried this command:

AnalyzerApp.exe "C:\CIM\CIM-Test\HardwareTest\Computer Motherboard Test Sequence.seq" /analyze

But it just opens the standalone Analyzer app with an error because I do not have a Analyzer project. Do I really need to have that?

Do you have any comment as to how to import the TestStand XML file in Jenkins. I found this suggestion:

https://forums.ni.com/t5/Example-Programs/Converting-TestStand-XML-Reports-to-JUnit/ta-p/3689131

But just want to hear if you have any other solution?

Thanks for reading.

[WireWeaver]

I responded to your other TestStand thread about the command line options.

The other option for publishing results to Jenkins would be to write a custom model plugin that publishes them, which is probably more work than the stylesheet to JUnit route. I found that the Jenkins JUnit results wasn't a good fit for what i wanted and abandoned it in favor of SystemLink Test Module (which uses the model plug-in approach).

Hope this helps!

Trent

[A.E.M]

Thanks for your reply.

I don't really see SystemLink Test Module as an option, as I need Jenkins integrations.

Regarding the command line commands:
I'm using the 'Computer Motherboard Test' example from TestStand. I have followed the tutorial to disable all pop-up messages, and slightly modified the example so it can run without any user interactions. 

When I run this command:

SeqEdit.exe /run MainSequence "C:\your-path\Computer Motherboard Test Sequence.seq"

here is what happens:

1) As soon as I hit enter, the cmd prompt returns without any output.

2) TestStand opens

3) The sequence file is loaded automatically

4) The sequence file is running automatically

5) In this case, all tests are passed and a report is generated

6) TestStand remains open

So the command is "working" as it is running the sequence file as intended. But how do I know when the command is really finished with executed my sequence file...

I then found this forum-topic which linked to this documentation:

http://zone.ni.com/reference/en-XX/help/370052N-01/tsuiref/reftopics/applicationmgr_exitcode_p/

So it seems like it should be possible to get an exit code.

By using the following command, the command prompt are now waiting for the sequence to finish execution:

start /wait SeqEdit.exe -Quit -Run MainSequence "C:\Users\Virtual PC\Desktop\Computer Motherboard Test\Computer Motherboard Test Sequence.seq"

When it returns, it is possible to call 

ECHO %ERRORLEVEL%

which will then return either 0 or 2 for passed or failed. Yah!

* However, cmd -OutputToStdIO does not add any information. Why is that?

* Is this the best / expected way to call a TestStand sequence from Jenkins?

[WireWeaver]

I've never had to use start /wait to get the Sequence Editor or any of the operator interfaces to wait. Here's an example i wrote:

https://forums.ni.com/t5/Example-Programs/Passing-Custom-Command-Line-Arguments-to-a-TestStand-Opera...

Here's the snippet from the Jenkins pipeline that i used to run it:

        stage('Run Single Execution') {
            steps {
				script{
					try{
						bat "\"${WORKSPACE}\\UserInterface_DotNet\\bin\\Release\\TestExec.exe\" /runEntryPoint \"Single Pass\" \"${WORKSPACE}\\Image\\target\\Computer Motherboard Test Sequence.seq\" /SimulateFailure ${params.SimulatedFailures} /OutputToStdIO /quit"
					}
					catch(Exception err){
						echo 'Exception caught '+err.getMessage()
						switch (err.getMessage()){
							case 'script returned exit code 1':
								echo 'One or more command line errors reported'
								throw err //rethrow so we stop the pipeline
								break
							case 'script returned exit code 2':
								echo 'Sequence Failed'
								break
							case 'script returned exit code 3':
								currentBuild.result = 'FAILURE'
								echo 'Sequence Terminated'
								break
							case 'script returned exit code 4':
								currentBuild.result = 'FAILURE'
								echo 'Sequence Aborted'
								break
							case 'script returned exit code 5':
								currentBuild.result = 'FAILURE'
								echo 'Killed Threads'
								break								
							default :
								throw err //rethrow so we stop the pipeline
								break
						}
					}
					
					//convert XML report to HTML and archive
					bat "${MSXSL} \"${WORKSPACE}\\Image\\target\\Report.xml\" \"${TS}\\Components\\Models\\TestStandModels\\ATML\\StyleSheets\\TR5_Horizontal.xsl\" -o \"${WORKSPACE}\\Image\\target\\TestReport${BUILD_NUMBER}.html\""
					archiveArtifacts "Image\\target\\TestReport${BUILD_NUMBER}.html"
					
					//publish results as JUnit
					bat "${MSXSL} \"${WORKSPACE}\\image\\target\\Report.xml\" \"${WORKSPACE}\\Stylesheets\\ATML_JUnit_Simple.xslt\" -o \"${WORKSPACE}\\testcases.xml\""
					junit allowEmptyResults: true, testResults: 'testcases.xml'
				}
            }
        }

[A.E.M]

I really appreciate your input!

I spend most of the day trying to call TestStand from Jenkins. My suggestion above works fine when called from the windows CMD, but when being called from Jenkins I do not get the expected output from ERRORLEVEL. This is because I use 'start', which means that changes of environment variables are lost when the <prog> ends. However, I need to use 'start /wait' because the CMD does not wait for TestStand to finish execution. 

Since I cannot get TestStand to wait, this is the command I have used for now:

START /WAIT /MIN CMD /C CALL "C:\Program Files (x86)\National Instruments\TestStand 2019\UserInterfaces\Full-Featured\CSharp\Source Code\Bin\x86\release\TestExec.exe" -Quit -RunEntryPoint "Single Pass" "C:\Jenkins\workspace\PC-TestStand\HardwareTest\Computer Motherboard Test Sequence.seq" & CALL ECHO %%^ERRORLEVEL%%

Background info (just as FYI, not needed to read):

The 'start' command actually executes the command in a new cmd windows. This means you normally do not get the exit code returned. However it is possible, and is described here:

https://stackoverflow.com/questions/28318643/getting-the-exit-code-of-an-application-started-with-th...

But; in order to have spaces in both, the command (in this case the path to TestExec) and of the parameters and try to handle them with quotes, it turns out there is a bug, so the work-around is explained here:

https://stackoverflow.com/questions/17674255/why-does-windows-start-command-not-work-with-spaces-in-...

Finally, when calling TestStand from Jenkins with the above mentioned command, I managed to get the call to wait for TestStand and return a value of 0 if all passes, and a value of 2 if a test fails. Output from a Jenkins build:

[Pipeline] bat

C:\Jenkins\workspace\CIM-Test-TestStand>START /WAIT /MIN CMD /C CALL "C:\Program Files (x86)\National Instruments\TestStand 2019\UserInterfaces\Full-Featured\CSharp\Source Code\Bin\x86\release\TestExec.exe" -Quit -RunEntryPoint "Single Pass" "C:\Jenkins\workspace\CIM-Test-TestStand\HardwareTest\Computer Motherboard Test Sequence.seq"   & CALL ECHO %ERRORLEVEL% 
2

So far so good! A return value of 2, because a test failed.

But.... For some reason, Jenkins does not mark the step as 'passed' or 'failed' based on this value. It is always passed. I do not understand why it does not fail the stage when the last output is different from 0... Do you?

I have tried your solution, but I do not have any exceptions to catch - properly due to the fact I use the 'start' command.

Arg! I would really appreciate if you can verify that TestStand 2019 are waiting for the execution to finish? This really seems like a bug with TestStand 2019?

Passing the XML report

I then tried to pass the XML report using junit and your guide:

https://forums.ni.com/t5/Example-Programs/Converting-TestStand-XML-Reports-to-JUnit/ta-p/3689131

I'm just using the report from the Computer Motherboard Test Sequence demo in TestStand. The cmd I run is:

msxsl testReport.xml XML_JUnit_Simple.xsl -o test_output.xml

which generates the test_output.xml attached. But there is not really anything in it. Can you figure out what is wrong?

I appreciate your help in this matter. Thanks!

[A.E.M]

Hi @WireWeaver.

Any change you have had the time to look into the issues mentioned in my previous post? It would be highly appreciated and I would of course by you a beer if we ever meet up in an airport 😄

[DGuest]

Please Note: 

The statement made in the forum is incorrect:

"I've never had to use start /wait to get the Sequence Editor or any of the operator interfaces to wait."

As stated here, the start /wait command is required to use a command line to run the user interface.

Regards,

David Guest