report generation write to file at interval

[kmarcella]

Thank you for all your suggestions! However, I am sampling frequently as I need my data to my updated continuously for graphs. I only want to record the data every minute though. What did you mean by 'exporting' the data to another loop? Any help is appreciated!

[kmarcella]

Upon more research, I vaguely understand the Producer/Consumer structure but have never worked with queues before so an example (or specific explanations towards my VI) would be very helpful. Thanks!

[Bob_Schor]

---

@kmarcella wrote:

Thank you for all your suggestions! However, I am sampling frequently as I need my data to my updated continuously for graphs. I only want to record the data every minute though. What did you mean by 'exporting' the data to another loop? Any help is appreciated!

---

So here is a question -- what can you tell me about data rates and data amounts?  Describe the data (i.e. "Two channels of analog data sampled at 100 Hz" or "images from 24 cameras arriving asynchronously at 30 frames/second"), being sure to say "how much" data (i.e. # of channels), what type of data (16-bit samples, text, images) and at what rate.  What rate do you want to plot?  Are you plotting every point, or plotting averages (and if so, over how many points)?  Note that if your data is coming in more rapidly than, say, 20Hz, you probably do not want to plot every point (hard for displays to keep up, hard for our eyes to see things changing too rapidly).

How much of the data do you want to save?  You mention "recording the data every minute" -- do you mean to record data at the rate of one point per minute, to record all of the data, but create a new data file every minute, or to record the data in "bursts", with the bursts coming every minute?  

Note that it is certainly possible to have high data rates, save everything, and have meaningful displays.  I'm sampling up to 24 channels at the rate of 1KHz (meaning I get 24K points/second), streaming all of it to disk as it is collected (so I have all of the data "all of the time"), and creating a display of every 50th point (so my display updates at 50 Hz).  The trick to doing this is to use LabVIEW's Secret Weapon, parallel While loops connected by Queues (the Producer/Consumer pattern).

My first loop gathers the data, typically getting it in "chunks" of 50 samples at a time.  This loop basically puts each point on a Queue bound for Loop 2, the loop that simply writes the data to disk.  I don't recall whether Loop 1 or Loop 2 takes every 50th point and sends it to Loop 3 (it doesn't much matter), but Loop 3 is responsible for display each point that it gets.  And it all happens simultaneously.

So tell me about the data flow in your application ...

Bob Schor

[kmarcella]

No problem. I have 12 channels coming into my DAQmx that are recieving analog inputs from temperature, pH, conductivity, and pressure sensors. The DAQmx is sampling at 41 Hz. I believe I am currently plotting every point and they are arriving into the spreadsheet as arrays. 

So by recording the data every minute, I mean that when a minute is up, I want to record the data that is currently coming through in that exact moment so it would it would be the data coming in at 1:00, 2:00 etc and ignoring all the data in between. Ideally, I think I would probably want to average all the data that was colleted over that minute and record that as one number but I feel like that might be more difficult with my limited experience. I hope that made sense.

Currently I just have the one While loop where the data is coming in and being displayed from. But from your example, maybe I have to change that? All of the writing to excel is outside the loop and not contained within another loop but I'm assuming that might need to change also. 

I figured I would have to use Queues but have never worked with them before so I wasn't sure how to properly set it up. First, I'm not sure how to name the data going into the queue and then, in general how to connect both the loops. Any examples/VIs of that would be great.

[kmarcella]

Okay so I think I have figured out how to use queue but my data is all in arrays and when I connected the queue elements to be written to excel, only one data point one line of data was written. I have  not yet established the data to be taken every minute so it should be giving me rows of data but it is not. Why would this be? Does the data somehow have to be indexed or initialized before/after queuing?

Also, if I want the data every minute, would that be taken before or after it is queued?

I really appreciate all your help!

[Bob_Schor]

Your Producer loop is the loop with the DAQmx Read Analog function.  I'm assuming that the DAQ accurately samples at 41Hz, so 41 samples = 1 second and 41*60 samples = 1 minute.  The Sample I'll consider as an array of I16 (I'm assuming a 16-bit A/D converter, but its simple enough to deal with floats.

Since you want to "see" all of the data, you could take the array representing a single sample of 12 channels and wire it (as appropriate) to a Chart.  Since (for simplicity) I"m assuming that we can use the A/D as an accurate "clock", we need to count every 41*60 samples and send these samples to the Writing (Consumer) loop.  One way to do this is to wire the index of the While loop that holds the Producer to a Remainder function whose divisor is 41*60 -- when the remainder = 0, we have a point to send.

It takes almost no time to put an entry into a Queue, and then it is out of the Producer loop, and present in the Consumer loop.  You now have a whole minute to do something with this point before the next one arrives -- plenty of time to write it as a row in Excel.

I'll try to put together a model of this, and will get you started with a simple Queue.  You should look up Queues and see how the various elements fit together.

BS

[Bob_Schor]

OK, here's a demo.  I changed the sampling rate to 40 Hz to simulate it with a 25msec loop.

One nice things about Queues is they simplify stopping multiple loops cleanly, without needing Local Variables.  When the Stop button is pushed, the Producer stops, and Data Flow causes the Queue to be released.  The Consumer loop, below, doesn't "know" that there is no more Queue, and keeps trying to Dequeue elements.  Well, without a Queue, this will generate an error, which is detected by wiring the Error line to the While loop's Stop indicator.  Note that this is an "expected" (indeed, a "hoped-for") Error.  When we exit the Consumer, we can clear the error and do any final post-processing, such as closing any data files we've opened to save the data.

There are four Queue functions here -- Obtain Queue, which creates the Queue as a Queue of 1D arrays of Dbl (wired into the "Queue Type" terminal), Enqueue Element (in the True case of the Producer), Dequeue Element, and Release Queue at the exit of the Producer loop.  Note that the Dequeue Element function will "block" if the Queue is empty -- the Consumer loop basically sits idle, consuming no computer resources, for 99.99% of the minute it is waiting.  We put a 100msec timeout on it (and do nothing if the Queue times out) mainly so we won't have to wait a whole minute to discover that the Producer has released the Queue and the Consumer can now exit.

Lots of little tricks here, but the basic idea is, I hope, understandable.  Remember, this is just a "model" of the Producer/Consumer pattern -- adapt to your own needs (be creative).

Bob Schor

[kmarcella]

Okay I think I understand your code. Thanks for putting it all together. I am currently adapting it to my code and will let you know when I have it all working.

However, I was wondering what is in the "True" statement in the case structures? Are they just empty?

Thank you!

[Bob_Schor]

It's a Snippet.  If you copy it onto a LabVIEW Block Diagram, it will become code, and you can see what's in all the Case options.  Try it (or look up "Snippet" in the LabVIEW Help).

BS

[kmarcella]

So I thought I set up my program similar to your example but it immediately gets an unidentified error when I try to run it. Is there something I did wrong? Or do you see where the error could be coming from?

Thanks so much!