Alternative LabVIEW Test Executive/Sequence Tools Available

[apg_air]

---

@Yamaeda wrote:

---

@FiloP wrote:

@ Yamaeda

I think this is the problem NI set out to solve with TestStand and with a high degree of success.

Does it makes sense that in a single company for whatever reasons 3 different test sequencers / executives are built ? Why the 1st 2nd and 3rd one wasn't "good enough" ? 

---

1. We can discuss the success, but yes, that was the problem they tried to solve.

2. Have you heard the expression "every system grows until it's a development platform"? That happened. 1st version was a simple ini-based runner, then we added the possibility to link files and sections and have parameters in them, then we changed to xml and added ... basically programming features.

---

I totally agree with your second comment, about "every system grows until it's a development platform".

During one of my previous contracts, where the requirement was to use TestStand, I wrote 95% of my code in LabVIEW, proved it all worked, and then I put together a simple TestStand sequence to call the individual tests.

Another contractor at the time almost did the opposite. He wrote some low level instrument drivers in LabVIEW, then wrote all of the complicated test code, adjustment routines, data processing etc. all using TestStand.

Two completely different techniques. At the time there was no one person in the company who was overseeing the project, and there were no stipulated guidelines to foloow, so they ended up with two test sequences written in completely different ways.

[Hooovahh]

---

@FiloP wrote:

@ Yamaeda

" At my company we've built 3(?) sequencers and where i've worked as consultant i've seen at least as many others. " 

I think this is the problem NI set out to solve with TestStand and with a high degree of success.

Does it makes sense that in a single company for whatever reasons 3 different test sequencers / executives are built ? Why the 1st 2nd and 3rd one wasn't "good enough" ? 

---

This is a pretty hot take that I don't agree with, and makes me question your intentions.  If the first iPhone was so great why did they ever make any new ones?  Why make a LabVIEW 2020 if LabVIEW 2019 was so great?  Software is never done and evolving to new things, refactoring, and improving is a sign of continued investment and not stagnation.

I personally don't mind waving the AI flag, if there is some evidence to point to, or some results worth reviewing.  But the 3 sites you linked to have a lot of AI slop images, on AI slop websites.  I don't have much faith that the sequencer they wrote with AI is any better.  The blog post also reeks of AI. Lots of words, but not much substance.  Showing examples, giving code, and talking about specifics would change my opinion.

Another point that I think could be made is that TestStand has a big company behind it wanting it to succeed and gives support.  Open Source is great and I get the benefit from it.  But a business wants a number to call when things don't work right.  It's the same reason we will buy NI cards, instead of an Arduinos.  The Arduino is great and cheap but when it needs to have a certified calibration, what are the options?  The company wants to just write a check and know it is taken care of.  And with standard, adopted hardware they can get that.  There is a lot of inertia and changing takes time at companies.  Agile software only companies, may change quickly.  But real tests using industry stands are going to stick with their stuff until the dust settles on a new standard to adopt.  27 years of continued TestStand development is a big deal and I suspect PyTestFlow won't be around that long.

I also don't agree with the conclusion in the Buy Me a Coffee post.  It shows Python-native as one check for TestStand. Why? It has been able to call native Python since 2019 how it it lacking?  And Traditional Manufacturing is 2 checks for OpenTAP and one for PyTestFlow, while showing OpenTAP as the evolution of Test in the 2010s. How? PyTestFlow is a new products how are they part of the traditional manufacturing? OpenTAP seems to be a somewhat new product with the oldest video I could find on Youtube being 6 years ago.  Yet that was the evolution of test automation in the 2010s?  And they get two checks for Traditional Manufacturing?  Because of that post and the ones here, I'm lead to believe they are biased, and possibly disingenuous.  If you have actual case studies, or big name companies stating something like how Ford moved away from TestStand and used PyTestFlow instead, I'd like to hear about it.  If this is an opinion piece, with some hidden agenda, then I'd advice others to be aware of that and form their own opinion.

---

@FiloP wrote:

Now AI is changing this landscape.

---

It certainly is, but I'm not sure we both agree on the direction.

[FiloP]

Fair comments, and let me remove any possible ambiguity.

Yes, I know Alberto personally. He is a friend, and I think the project deserves a fair chance.

So my position is not neutral in the sense that I am curious and supportive.

But there is no hidden agenda here: I am not asking anyone to replace TestStand, and I am not claiming PyTestFlow is a drop-in replacement for a mature manufacturing TestStand deployment. On the contrary I think TestStand is one of the best product NI ever made and is still incredibly relevant.

My point is much simpler.

The question of “alternative test executives” is worth reopening because the environment has changed. Python is now everywhere in engineering, AI-assisted coding is becoming normal, wiht good and bad aspects, and many teams are already building internal sequencers anyway.

So I think we have among many possible futures also these two:

1. Every company keeps building its own small sequencer, that then slowly grows into another internal development platform.

2. Some lightweight open frameworks emerge and solve the common 80% once: sequence execution, operator prompts, result collection, reporting, pass/fail logic, and basic process structure.

These possibilities will remain alongside companies who invest into TestStand standardization, OpenTAP learning curve, OpenHTF and several others.

But I think PyTestFlow is at least worth evaluating because the cost of evaluation is extremely low. It is basically:

pip install pytestflow

https://pytestflow.com/?page_id=56

and a few minutes later you can run a sequence and generate a report.

You don't have to take my word.
Examples matter more than claims. A useful third-party example is James Costello’s walkthrough, where he uses PyTestFlow with an RS-3005P PSU and an NI USB-6009 DAQ:

https://costellotechnical.com/blog/pytestflow/

To all of us working in test: the next few years will be very interesting. Have a nice weekend!