Generating a very, very customized XML report

[Mr._Jim]

Hi there,

It might sound like this is an easy one from the title - maybe it is but I suspect not.

We've been experimenting with NI's offline results, and some time ago I'd gotten all of our customizations (additional columns in our tables, etc) working. To be perfectly frank, the stock offline results and ORPU leave much to be desired from our perspective.

Collaborating with my database wizard colleague, we're trying to keep things really simple. I'd like to generate a very direct XML dump of our results so that we can have SQL Server parse the XML and insert it. (Sample attached) He already has some code put together to do that.

Our schema is customized, but it's based off the stock TestStand schema with some extra columns. We use many of the common types for PROP_RESULT but nothing too exotic. (No waveform types)

I'm already familiar with the plugin architecture - I've written all our many model customizations into a plugin and customized two others. It seems to me that parsing ResultList objects into UUT_RESULT and STEP_RESULT records is for the most part pretty straightforward. Where I get hung up is how exactly to handle all the variations in additional results, step looping, PROP_RESULT records, etc. I've stared at some of NI's code for plugins and they've unsurprisingly put a lot of this functionality into libraries that may or may not be publicly documented? I thought maybe I could "borrow" some of their ResultList traversal code but it looks pretty challenging.

I thought about customizing an XML stylesheet and using the stock report generator, but I'm a little overwhelmed by the stylesheets that TestStand ships with. I'm a novice with XML stylesheets, and while I'm dangerous in JavaScript, wow that is a lot to take in.

Any ideas or experience relating to how to get this end result?

Thank you very much in advance,

Mr. Jim

[Dane.S]

Hi Mr_Jim,

The Stylesheets are large and can be quite daunting. We have a overview and splash page here, that can act as a starting place. 

This also points to how to add a column and row to an XML report, that provides a step by step guide how to add a column for StepID. You can use this and change the XPath value in the stylesheet from 

<xsl:value-of select=”./Prop[@Name="TS"]/Prop[@Name='StepId']/Value”/> to whatever XPath data value you want. 

To provide a little more insight on the hierarchy here is some info:

XML:

The Report XML that NI’s Report Generator creates for XML reports contains all the reported data for each step organized by step ID. All steps are recorded in separate nodes within the Prop nodes named TEResult > TS > SequenceCall > ResultList. Each step is reported in order in the form <Value ID=’[N]’>

Different pieces of data for the step are recorded in various nodes within the step. For example, the name of the step is within the Prop Node named StepName that exists in the relative path Value ID=’[N]’ > TEResult > TS > StepName

Stylesheet:

The Stylesheets that are used to modify and format the XML reports are organized by sections adding different pieces of info to the report. Within these sections there are templates used to select a node from the XML document to operate on. There will be a template match for your XML nodes such as <xsl:template match=”Value[@ID]/Prop[@Type=’TEResult’]”’>, which will apply to all the steps results. Within this template for step results, different properties are checked to determine what type of step it is and add the appropriate objects to the report.

Hope this helps a little!

[Mr._Jim]

Thanks for the information, Dane. I will look through these in more detail.

Hopefully the following isn't a dumb question:

So, in your opinion, is it doable / practical to make the kinds of changes we're thinking of using a stylesheet? My intuition is that I'll pretty much have to throw everything in the shipped stylesheet out and start from scratch, because per the XML example I attached, what we're looking for is very different from the stylesheets that ship with TestStand. Is this something I should attempt?

Thanks again,

Mr. Jim

[Dane.S]

I think that depends on exactly what the end result of the changes you are looking for is. I've got a vague idea of the end goal but it would be easier to provide possible paths if we get a better more concrete idea of what you want in the end.

If parsing through the XML for results to plug into a SQL database is the end goal the stylesheet won't be worth re-writing. 

[WireWeaver]

---

@Mr._Jim wrote:

the stock offline results and ORPU leave much to be desired from our perspective

---

I would be curious to know what more you expected from ORPU. If you can provide any more details i would appreciate it.

My 2 cents on the stylesheet vs plugin scenario is mainly a matter of preference. Creating a basic stylesheet to do what you described should be possible, but it's somewhat miserable if you don't know XSLT. Here's a more basic example for Junit (another simple XML format):

https://forums.ni.com/t5/Example-Programs/Converting-TestStand-XML-Reports-to-JUnit/ta-p/3689131

You'll have to either use the MSXSL utility, or write a small .NET app to apply the translation before you ingest it.

If you already know how to write plug-ins, then you could start with the simple text report example and modify it to write your basic XML schema instead of CSV. You could also go a step further if you wanted and create a plug-in that has it's own store & forward mechanism to log directly to the database... that's how the new SystemLink Test Module Plugin works.

Hope this helps,

Trent

[Mr._Jim]

Hi Trent,

(Thank you both, by the way)

I think at this point I'm leaning more toward using text to generate XML. (Your latter suggestion)

So, about the ORPU and offline results in general, since you asked:

Before I get into it, I should note that offline results are a great concept - I don't want to be overly harsh with it.

Here are the things we had a hard time with (please forgive my candor - I'm only answering your question)

• Unless there's something I'm ignorant of, the offline results format and API are a black box. There were times when I would have loved to have tweaked certain things, but there was no access to the guts of it. (I wish I had a tangible example - sorry)
• The reason we're presently stalled and not able to use it: The biggest pain point we had with it is when writing a very large result set, which is something we frequently do and is unavoidable.  (~14,000 results per UUT) We are using TS 2017 32 bit due to some things (drivers, etc) that are only supported in 32 bit versions. The ORPU would predictably run out of memory every time it was processing one of these result sets. Contacting support, I was advised to move to the 64 bit version of TestStand, which seems like a workaround rather than a solution. Does the ORPU leverage the same instance of the TS engine as the one executing the tests? I may be very ignorant here, but we couldn't get around the memory issue. The thing is, a CSV containing all 14,000 results with all columns is around 10.5 MB. Surely there's a way to make this work in under 3-ish GB of memory?
• Offline results seem unforgiving of schema customization in the form of extra columns being populated. It took a ton of trial and error to figure out how to "smuggle" the values of extra step result columns through the data. For instance, we have columns for temperature, frequency, etc. Eventually I figured out that you can create a new container under result->common (for each step result), and it makes its way through to the ORPU. On the database plugin side I had to parse those results out again. I'd tried attaching the data to each result in all sorts of ways, but the offline result generator plugin would drop the data and it wouldn't show up in the final results.
• I had a similar experience with extra UUT_RESULT columns (e.g. fixture ID, firmware versions, etc). I finally settled on creating a special dummy step that carried those values, and my customized database plugin would look for that result and remove it, parsing the values within.
• If we'd get a partial write on the database due to a technical issue and had to try again, we'd get primary key collisions. I freely admit that this could amount to "you're not doing it right", though.  What we're trying to do with the XML effort is have SQL server create a temporary table and only commit the data to the database after a verified complete data set. That way it can back out if the data set is incomplete or somehow interrupted.
• Minor annoyance: To get the splash screen to go away on the ORPU you have to jump through some hoops - in my humble opinion this should just be an option somewhere in the API. This might have already been changed by now, though. I haven't tried recently.

That being said, there are people way, way more knowledgeable of TestStand than myself on here and I don't want to claim these were the only ways to do things, but those were the methods I eventually settled on. If the offline result and ORPU developers can make it a bit "lighter" (on memory for instance) and a little more customizable I would absolutely love it. We had really high hopes for it and I'm getting a lot of pressure within my job to come up with an offline results solution that fits all of our requirements.

A humble thanks for your soliciting feedback based on my early comment, and thanks for reading this ridiculously long post.

(P.S. I will be in Austin next month - maybe this warrants further discussion?)

Mr. Jim

[Mr._Jim]

One other thing I forgot to mention... I tried moving one of the offline result files (with 14,000 results) to another PC and a freshly opened instance of the ORPU, no other TestStand activity. I was stoked about the prospect of success, but alas, it ran out of memory in that scenario, too. (I'd love to reduce the data set, by the way.  Clearly that would be the best workaround. )

[WireWeaver · Accepted solution]

Thanks for sharing those details.

So we actually do ship the source code for ORPU if you're familiar with C# and want to modify it:

%TestStand%\Components\Models\TestStandModels\Offline Results Processing Utility

It's worth noting there's a bug with the source shipped in 2016 that prevents you from recompiling if you make any changes... That was resolved in 2016 SP1. TestStand 2017 also fixed a few other small issues with ORPU.

ORPU loads its own instance of the Engine and essentially just deserializes the offline results and runs them through the result processors like a normal TS execution would. If you can run a normal execution and generate a CSV report that large without running out of memory, but ORPU crashes... you might have found a bug. The only thing that comes to mind is that I'm not sure if ORPU is Large Address Aware (i don't have a 32-bit build handy to check), but the Sequence Editor is. We actually ship a utility you could use to check and change it if it's not. If that doesn't fix it, I'd appreciate you sharing some example files that could help reproduce it with myself and Dane.

Thanks!

Trent

[Mr._Jim]

A heartfelt thanks to both of you again - I will take a look at the C# code!

I'll also check the large-address-awareness in the hopes that we can buy a little more headroom.

Absolutely - regarding a proof of concept for what we encountered I should be able to reproduce it and make some files available at the next opportune time. (I'm sensing deja vu... I may have attached such an example to a support ticket at one point? It was a while ago. Regardless I'll get that going again.)

Kind regards,

Mr. Jim

[James_Grey]

I wouldn't consider 64-bit mode as a workaround for dealing with large data sets because that is pretty much what it is good for.  

TSR files are not completely black boxes in that, in addition to being accessed in the public ORPU source code, they can be directly written with the ResultLogger class and read with the ResultLog class in the TestStand Engine API. Internally the files are mainly just serialized PropertyObjects. Specifically, serializations of most of the arguments that the model passed to the model plugin entry points when the UUT was tested, including the result list tree.