Entry point execution time

[pwaszkewitz]

I have done some tests on turnaround time (i.e. time from starting a test to a finish message) and would like to know if the results seem plausible or if I overlooked something.

I use two notifications, one to start the test, one to signal that the test has ended - basically simulating a handshake with a control system, and I have tested the following four variations with 1000 runs each and a basically empty MainSequence:

1. Starting MainSequence through the SinglePass execution entry point : 1.133 s/run
2. Starting MainSequence through the UUT execution entry point with one report file/UUT: 0.141 s/run
3. Starting MainSequence through the UUT execution entry point with a single, accumulated report file: 0.142 s/run
   (which is obviously not a statistically significant different)
4. Starting MainSequence directly from a while loop without an execution entry point: 0.001 s/run

Obviously, the last one will not give me a test report as the resport is created through a callback from the execution entry point. And I am well aware and do appreciate all the setup and cleanup work the execution entry points are doing.

So, my first question is: do the results seem plausible (i.e., the time ratios between the variants; the absolute time of course depending on the machine etc.)?

And my second question is: is there a faster way to carry out tests with a report on every run than variations 2&3?

or, put in another way: if in variation 4 I were to call all the sequences necessary for setting up a report, outputting it and cleaning up by myself, could that be significantly faster than letting the UUT execution entry point do that?

Regards

Peter

[dug9000]

A few ideas:

1) If you don't need the report right away, try the offline report generation option. The intermediate binary format used by that option should be very quick to serialize. Then you can generate a report or log to database as needed, even on a completely different machine.

2) You might also try asynchronous mode for the report generation, that mode attempts to generate the report for the previous uut asynchronously while you are testing the next UUT. That should give you more parallelism, depending on how fast your test sequence is running and how busy the CPU cores and disk drive are on your machine.

3) report generation can be disk IO intensive, try on a machine with a fast SSD drive to see if you get better performance.

Hope this helps,

-Doug

[~jiggawax~]

What version of TestStand are you using.  In the newer versions reporting is irrelevant to timing because they can be spawned in new threads/executions. 

Also, you have to take into account tracing.  If tracing is off your times will be much much lower.

What is your goal here?  If you are trying to reduce test time per UUT then I'd recommend doing the reporting in a new thread with the newer versions of TestStand.  I really don't think it would buy you much to move the steps from the model to your sequence file.

Regards,

[pwaszkewitz]

Thank both of you for your replies. I am aware of the possibility to generate the report in parallel and the actual time for generating the report is not really the point.

Perhaps I stressed the report topic too strongly. The reason is that being able to use TestStand standard reporting is the most important feature that we are missing using our current approach.

We are currently using the scheme of a non-ending execution because of reaction time which can be crucial in the image processing systems we do. In this setup, it takes approximately 1-2ms from receiving a start signal notification to the start of the main testing sequence.

My tests (using TS2014) show that using the UUT execution entry point it takes 50-60ms from receiving the start signal (in the PreUUT callback) to actually starting the MainSequence. After the MainSequence, it takes another 80ms for the system to re-enter the PreUUT callback to check for the start signal again (with the report being generated in a different hread).

So it boils down to the question: it is possible to use standard TestStand reporting and reach reaction times < 10ms?

Mind: we have a lot of applications where we do not need this reaction time and use UUT reporting; it would just be interesting to extend this to the "fast applications".

Best regards

Peter