Best Way to Handle Datalogging between Parallel Loops?

[Glibby1234]

Midway through a project using multiple parallel QSMs to log data on different units. I'm a labview rookie so I'd like to learn the best way to do things rather than crash through what happens to work. 

Each QSM is a parallel loop for a different unit. They may start and stop at different times, and all I'd like to do is log data for each unit in the same file. Obviously the data-logging has to happen separate from any one QSM loop, otherwise it won't log at all. 

As a rough prototype I came up with the attached, which seems to work.... is there a better way to do this? 

[Gregory]

If those two loops you have are the same, you could (should) put it in a SubVI and launch two of them. You can save the data inside the loop if you want. Perhaps more common would be to send the data to a separate loop where it is logged to a file so that accessing the hard drive doesn't slow down your main acquisition loop.

[Bob_Schor]

Some comments:

• Your two loops (running at different speeds, potentially) are accumulating the data in Shift Registers, which means that the data being saved gets larger and larger (without bounds, potentially).  If the program should crash before the data are saved (when you exit the loop), all the data are lost.
• You combine the two Data Arrays into a 2D array before writing the data to disk.  I assume you know that in a 2D array, all the Rows and all the Columns need to be the same length.  You are combining (as columns!) two independent Arrays which may well be of different lengths, while will cause the shorter dimension to be padded with zeros.  Do you want this?
• A more "natural" way to combine two 1D arrays into a 2D array is to use Build Array.  This is a "row" operation, but if you want the arrays treated as column vectors, you can follow Build Array with an Array Transpose.
• What Gregory was talking about is the Producer/Consumer Design Pattern.  If you open LabVIEW, go to the File Menu, click the second choice ("New ..."), and choose VI/From Template/Frameworks/Design Patterns/Producer/Consumer Design Pattern (Data) , LabVIEW will build you a test routine showing how to simultaneously (almost) generate and save your data.
• If you have two data channels coming in at two different rates, I'd suggest writing two Data Files ("Data Ch 0" and "Data Ch 1").  If you want, you can combine them later into a single concatenated file, or keep them separate so you can easily access them in parallel.

Bob Schor

[Glibby1234]

Yes they are the same (will be) I just hadn't gotten around to building them into a Sub-VI yet. I may be misunderstanding your advice.... but if they are saving the data within the QSM (Sub-VI) won't they be incapable of writing to the same file? Same with sending the data to a separate loop to be logged.... if that loop is outside of the Sub-VI (which it would have to be to be accessible by everything) then it will only run once the Sub-VI is complete, yes? 

[johntrich1971]

You could, for example, use queues to pass the data from the producer loops (subVIs) to the consumer loop. Create the queue and pass it to the subVIs.

[Gregory]

---

@Glibby1234 wrote:

but if they are saving the data within the QSM (Sub-VI) won't they be incapable of writing to the same file? 

---

They could write to the same file, but I don't know what your data is supposed to look like. If you want 1 column for loop 1 and 1 column for loop 2, that would be very inefficient to do writing from both loops. It may be easier to write 2 separate files and then combine at the end if need be.

---

@Glibby1234 wrote:

Same with sending the data to a separate loop to be logged.... if that loop is outside of the Sub-VI (which it would have to be to be accessible by everything) then it will only run once the Sub-VI is complete, yes? 

---

This would be in a parallel loop to your data acquisition, but you would still have the trouble of organizing your data that I mentioned above. I would either make 2 separate files, or acquire the data from both devices in a single loop if it belongs together.

[Glibby1234]

---

They could write to the same file, but I don't know what your data is supposed to look like. If you want 1 column for loop 1 and 1 column for loop 2, that would be very inefficient to do writing from both loops. It may be easier to write 2 separate files and then combine at the end if need be.

---

---

Yes that was the intent. Each loop would have 2 channels, and each channel would be a column. If loop 1 is started (logging data) it will start record in it's column(s). If loop 2 get started after loop 1, it's data could either be offset (vertically) to correspond with the delta between when the file was created and when it started logging or it could simply start in the first blank row corresponding with its column(s) I dont really care. 

Sounds like I would be best logging independently and then combining to a single file, the only thing I don't like about that is in the event of a crash, I would only have the non-combined files instead of a single master. 

[RavensFan]

---

@Glibby1234 wrote:

---

Sounds like I would be best logging independently and then combining to a single file, the only thing I don't like about that is in the event of a crash, I would only have the non-combined files instead of a single master. 

---

So.  Just create a VI that handles the combining part and run it separately on those files if you do have a crash!  That part of your code should be a subVI within your main VI anyway.

[Bob_Schor]

Please read my reply again.  I point you directly to the Producer/Consumer Design Pattern that lets two parallel loops run simultaneously, one (the Producer) "producing data" and the other (the Consumer) almost-simultaneously "consuming it" (in this case by writing each array as it comes to disk).  One way to transfer data asychronously between parallel loops (shown clearly in the Template you didn't open) is via a Queue, shown in detail for you to study.

I also mentioned that you can (should?) create a pair of files for a pair of (independent) data streams.  If you are creating them synchronously, then do them in the same loop and create (and export) a 2D array.  Don't make things more difficult for yourself.

Bob Schor

[Glibby1234]

Whoa Bob, I was responding to Greg because I didn't understand the concept and hadn't seen your response yet. Please don't accuse me of things that are un-true, I did look at the producer consumer loop but got side-tracked by the discussion around saving in separate vs. different files. 

So the producer consumer loop would be within the sub-vi loop to produce (read the data) and consume (save the data) to an independent file? Then, each sub-vi loop would have an independent data-log file which would be combined to a single file but a completely separate loop from all of what was just described? Do I have the framework of that correct?