Producer Consumer Loop for my Project?

[bockdoug]

I have been asked to write a ( I guess) fairly large program to control and save several things. Controlling the parameters of:

• an optical spectrum analyser (OSA)
• a temperature sensor 
• a pressure sensor 
• a humidity sensor

(all from different manufacturers, with different acquisition rates). Besides these controls, I want to save all this data onto some spreadsheets (I have not decided if a single one or one each). Additionally,  all data points need to get some sort of time stamp so that I know which wavelength in the OSA corresponded to which temperature (again, sampling rate will be different).
Here come my questions: What is the best architecture for this? I was thinking of producer/consumer loops. The producer would be an event structure inside a while loop to handle any value changes to the parameters by the user. The consumer loop would save all the data points.
How would I best implement a time stamp? At this stage I am thinking of creating a time stamp whenever the driver of one of the instruments acquires a data point. This would then also handle different sampling rates, no?
Cheers in advance
Doug

[crossrulz]

1. For logging, I highly recommend TDMS.  It handles the different acquistions using different groups.

2. I recommend a loop for each of your sensors.  Look up the Queued Message Handler.  I think it will work a little better for you.  Same principles as a Producer/Consumer, but slightly different.

3. Have a loop just for logging.  Again, the QMH will be useful here.  But the idea here is that any loop can send data to be logged.  As part of the message, include the timestamp, the data, and what the data is.

This sounds like a slightly complicated system, but not difficult.

[Blokk]

I like to follow the KISS principle 🙂

I had many similar projects as you are dealing now with. Most often temperature, pressure and humidity do not change with a fast rate, so you do not need high acquisition rate neither tight sync with the other DAQ tasks! So since they do not change fast, you could just get their values into your logging loop (TDMS file is the way) via a lossy Notifier (like a tag, you are only interested in the most recent values and do not care about tight sync).

If the above applies for your project, I would do the following:

• I would use a Producer/consumer design, where your producer loop (Event structure) is the GUI loop (sending commands via the "Commands Queue").
• The first While loop is just measuring the temperature/pressure/humidity values with a slow rate (1 Hz I guess is enough?) and inserting the values into a Notifier (Send Notification function, lossy data transfer).
• The next While loop is the OSA loop where you do DAQ with the required (I guess higher) rate, and Enqueue data into your Data Queue plus an absolute time stamp coming from the OSA or you generate it here (lossless data transfer). This While loop could hold a state machine too, with the following states: Idle, Start Acq, Acq, Stop Acq. (...and so on as you require, you could also have states for the file logging loop, and command the file logging loop from the OSA loop)...

When you Dequeue the OSA data in your TDMS logging loop, you just need to "pull" the most recent pressure/temperature/humidity values from the Notifier line (Get Notifier Status) and together with the OSA result(s) and time stamp you save the values into your TDMS file.

Moreover, use Dynamic User Events for communication tasks from any consumer loop toward the Producer GUI loop (like error handling purpose, etc.).

[udka]

I would suggest to look into continuous measurement and logging example which is based on Queued Message Handler.
-You can check here: http://www.ni.com/video/2741/en/
-it handles five loops each for event handling, UI messaging, acquisition*, logging, and display

[bockdoug]

I have played around a bit with the TDMS filetype. In the attached vi, I am simulating four different sources (waveforms) with different amounts of channels and sampling rates. Is this an acceptable way of implementing what I need for my project? Eventually, the signal generation vi's will be replaced with my four acquisition instruments. The resulting TDMS file, when viewed in EXCEL, consists of 5 worksheets, one for each waveform and each sheet with a time column and one column for each source. Exactly what i wanted, really!
Also, am I right in thinking that the queued message handler cretaes strings in an event structure, which get passed to a state machine that runs the state appropriate for the message?
Cheers in advance
Doug

[drjdpowell]

Whenever someone asks about "Producer-Consumer" I recommend they instead look into the JKI "State Machine" Template, a single-loop event handler.  Use one JKI loop for each piece of hardware, get them to work independantly at first, and then have a "Main" loop that communicates to/from all of them via User Events.  Do NOT use "Continuous Measurement and Logging" as a guide.  

[bockdoug]

Ok so I have played around with the TDMS stuf and am becoming quite comfortable with that. The QMH not so much. I am modifying a copy of the "Queued Message Handler.lvproj" to understand whats going on, without success...
Here's my understanding so far: The EHL enqueues strings, which trigger the corresponding event structure in the MHL. Easy. But what if there is more than one MHL? I believe I will need five MHL's: one each for the four instruments and one logging MHL that writes to a TDMS file. Does that mean I need five queues?

I have also tried to consult the Documentation provided in the sample project, which I find very confusing. Does each queue need its own type def in which all the possible messages are stored?

In addition, contrary to my initial post, I now do not require to change the settings while a measuremnt is running! Could I now just use one MHL with cases "open Settings", "run Continuous", "run Single", etc., or even simpler, just a state machine?
I don't think I am mentally ready for large QMH allpications...
Any advice will be appreciated.
Doug

[Blokk]

What I described above, is a simple easy solution...I would go this way.

[billko]

---

@drjdpowell wrote:

Whenever someone asks about "Producer-Consumer" I recommend they instead look into the JKI "State Machine" Template, a single-loop event handler.  Use one JKI loop for each piece of hardware, get them to work independantly at first, and then have a "Main" loop that communicates to/from all of them via User Events.  Do NOT use "Continuous Measurement and Logging" as a guide.  

---

I would actually encourage the use of LabVIEW best practices first - i.e., the Producer/Consumer design pattern - for two reasons:

1. Standard design patterns are recognized and understood by every trained LabVIEW developer.  That's why it's a "standard".  Introducing a non-standard way to do things means that you will own that method and be responsible for teaching everyone how to use it.
2. Standard design patterns teach you the underpinnings of the more advanced architectures - i.e., the JKI state machine.

[drjdpowell]

---

@billko wrote:

---

@drjdpowell wrote:

Whenever someone asks about "Producer-Consumer" I recommend they instead look into the JKI "State Machine" Template, a single-loop event handler.  Use one JKI loop for each piece of hardware, get them to work independantly at first, and then have a "Main" loop that communicates to/from all of them via User Events.  Do NOT use "Continuous Measurement and Logging" as a guide.  

---

I would actually encourage the use of LabVIEW best practices first - i.e., the Producer/Consumer design pattern - for two reasons:

---

That's just it: the "Producer Consumer" NI template is very poor practice.  It takes a lot of experience to build an application based on that without introducing race conditions.  Its apparent simplicity is a weakness, since it is inherently more complex than a single-loop design, such as the JKI.  It should be eliminated as a standard design pattern.

Here's a talk I gave on this last year.