TestStand Event Log

[ChrisHawk]

Overview

For a long time working as a TestStand architect I was asked “how can an event log be created to allow additional information about the execution of a solution to be stored?”  I could see a real need to log data independent of the test results as this information would clutter the result data for factory systems with information only really required to sort out test problems.

On thinking about this for a while I found the current mechanism for OutputMessages in TestStand to be very powerful.  This allows messages to be sent from TestStand sequences using the built in function to send a message as well as from code modules where a simple TestStand Engine API call can be made.  Messages are automatically recorded with date/time information, and they can also be categorised.  The messages are also linked to the current execution and therefore can be tied to a particular UUT.  Finally for the sequence editor at least there is a nice way to present them to the user.

The bit that is missing is the ability to log the data to one or more files.  So I embarked on a project to create such a feature.

Code Module Description

The first part of the challenge was to allow the capture of output messages.

I tend to use C# for executive code module implementation so snippets will be from the full example Visual Studio C# project. 

As TestStand supports multiple executions I implemented the module as a singleton to only allow one instance to be running even if multiple executions start.  This is more a C# technical challenge and my implementation my not be the best, but it works!

The simplest way is to use the Application Manager from the current TestStand Engine instance and add an event handler for the UI messages.  Fortunately when you create a TestStand Engine object from a function called within TestStand it returns the current Engine rather than creating a new one.  And the Engine then offers a method to get the ApplicationMgr object.

// Will get the current engine as will be executed in the TestStand process
Engine m_EngineObj = new Engine();
// Get the application manager from the engine
m_ApplicationMgr = (ApplicationMgr)m_EngineObj.GetInternalOption(InternalOptions.InternalOption_ApplicationManager);

// Setup the event handler
m_ApplicationMgr.UIMessageEvent += new NationalInstruments.TestStand.Interop.UI._ApplicationMgrEvents_UIMessageEventEventHandler(uiMessageEvent);

So we can now add code to our module the event handler to take any UI messages tagged as an OutputMessage log a file.  The module has a component to allow the setting of a default log path and the file format to be stored and used within the module.

Note it is also a good idea to capture the ShutDownComplete message to ensure any log files are flushed and closed.

To add flexibility a custom UI message can be used to provide the log file path for the current execution in the string of the message. Flags can also be sent via the numeric part of the UI message to control log file format (for example if RFT or plain test format). 

The logger instance will need to maintain a store of log file paths against execution ID’s.  Now when an OutputMessage is captured we can see if there is a custom execution log file defined, and if so log to this file, and if not log to the default log. 

To clear the custom logging for a given execution an empty string can be provided to clear the custom log file for the execution.  In this case the log file will be flushed and closed and any future UI messages will go to the default file.

The detail on the formatting for the log file is up down to what is wanted.  In the example the messages are stored as tab delimited and with the option of plain text or rich text format (RTF) files.

Sequence Description

To make this feature separated from a test solution implementation the Plugin feature of TestStand can be used.  It is an ideal way to allow for the logging of messages without the need to add calls into every solution to enable and configure it.  This allows enhancements to the logging options to be added without the need to constantly go back and touch all the solutions.  It also means that solution developers need not worry about how it is done, just that they can get their OutputMessages captured or not by station configuration.

I will assume knowledge of the Plugin architecture as I see no point in going over the documentation on this topic. 

The plugin provides the ability to specify an expression in the plugin configuration settings for both a default log path and a path for UUT log path.  These expressions have the option for macros to be used for UUT information (serial number, etc) and system paths (e.g. PublicDocuments).  There are also options to give the log format and if the UUT log for a unit to be added as an additional result in the final result file.

• During the Model Plugin – Begin the plugin obtains the settings from the plugin configuration and updates the path for code module.  It then starts the event logger to start the capture of messages to the default file.
• During Model Plugin – Pre UUT the UUT log path expression is evaluated.  A UI message is then sent to configure the current execution to log to that file.  Note the plugin also has an option to write the log data for an execution to the result file.  If this option is set, however no UUT log path is specified then the plugin will use a temporary file.
• During Model Plugin – UUT Done if the option is set to copy the log data to the test result we read the current log file and then store this to the UUT.AdditonalData with a flag set to include in results.  Note that the sequence can be run in a new thread, therefore we need to copy this LogData variable back to the calling thread before the other result processor plugins.  There is an expression to move to the calling thread and the unblocking of the UUT Done is move till after we are done.
• During Model Plugin – Post UUT sends a message to clear the UUT specific log file.  Then we delete the temporary file if it was created.

There is a configuration sequence in the plugin to set the options.  This uses another project of mine to allow the edit the TestStand variable holding the options (https://forums.ni.com/t5/Example-Program-Drafts/Edit-Variable-Control-and-Popup-for-TestStand/ta-p/3...).  This was a quick solution using a specially created DLL from another of my projects.

Additional Features

Another feature that I had requested was to add any TestStand errors to be automatically logged.  To do this I added a StationPostStepRuntimeError callback to the StationCallbacks.seq file. 

Here I simply added a step to send an output message with the error details.

Then things got complicated as I was asked to also have the option to email any errors to a group of support staff.  To do this I needed configuration data, so I added the configuration options to the EventLog plugin I created.  I then had the station callback sequence find the EventLog plugin configuration data and processed this to build an email message if configured to mail errors.  Then I used .Net to send an message using SMTP.

Future Updates

Items that could be added to this project in the future would be:

• Embed a configuration dialog into the code module to remove the dependence on another project.
• Add options to filter the messages logged based on the category and or severity.
• Add similar filter options to the error log email messaging.
• Find a way to trick TestStand to report the actual sequence and step information for the step that errored in the OutputMessage rather than it reporting the information on the StationCallback step.

Software Requirements

Test 2012 or later.

Steps to Implement or Execute Code

1. Download the attached zip. Note the files in the TestStand folder in the zip should match the structure that is found in your <TestStand Public Directory>.
2. Make a rename (or make a backup copy) of your <TestStand Public Directory>\Components\Callbacks\Station\StationCallbacks.seq
3. Copy the content of the TestStand folder to your <TestStand Public Directory>.  
4. Launch TestStand
5. Goto Configure->Result Processing
6. Tick Show More Options
7. Select the Inset new, and select EventLog Plugin from the popup.
8. Now the plugin is added you can configure and use it!

Note the Documentation and Source files are also contained within the zip.