Producer Consumer Data Acquisition Multiple Channels

[JonnyK]

Hello all,

I am a student at university working on my senior project and am having some troubles using LabView.

I am currently trying to implement a producer consumer format, tdms writing program for multiple analog input channels on my CDaq 9205. Producer loop iterates very quickly and consumer follows writing values to file from queue at slower speed.

I have passed values from loops using local variables in the past, many say this is a no-no and I agree. Doing this I have found data dropped when writing to file. This result has led to my attempt at producer consumer data acquisition and logging.

The goal is to acquire the analog inputs from my transducers in the producer loop and then send the values to my consumer loop for data writing. This is concurrent with using LabView GUI to visualize transducer outputs through experiment.

The biggest issue is the following: I want to scale my data before writing to file. I do not want to log raw voltage. I would like to do as little post processing of data as possible.

I would like to acquire raw data from transducers very quickly, 25-50kHz. I would also like to prompt the user to specify how many samples per second they want to write. Max samples per second writing to file ~25ks/s. 

I am currently getting the following error at continuous data acquisition speeds larger than 1kHz:

"The application is not able to keep up with the hardware acquisition.

Increasing the buffer size, reading the data more frequently, or specifying a fixed number of samples to read instead of reading all available samples might correct the problem."  

I think my use of arrays in the producer loop is filling buffers but I am unaware of how to fix this.

Any help and or advice would be greatly appreciated. I have over 40 hours writing this code and searching these forms for like minded programs without any luck.

I have attached a vi, front panel is large and messy, I'm sorry. I have cleaned up under the hood to make schematic more readable. 

[GerdW]

Hi Jonny,

The biggest issue is the following: I want to scale my data before writing to file. I do not want to log raw voltage. I would like to do as little post processing of data as possible.

You can apply scales on your DAQmx channels, so you would read scaled values instead of raw voltage. See the example VIs coming with LabVIEW and DAQmx for reference…

I would like to acquire raw data from transducers very quickly, 25-50kHz. I would also like to prompt the user to specify how many samples per second they want to write. Max samples per second writing to file ~25ks/s. 

The main problem here is: you set a rather high samplerate (10kHz by default), but you read those samples one by one. That is a NO-GO!

Read blocks of samples using the "N chan N samples" mode. Usual recommendation is to read 1/10 of sample rate, so with 10kHz sample rate you should read 1000 samples…

Next problem: why do you keep your samples in separate arrays in the producer? Why do you keep them in arrays of size 1???

Next problem: why do you need 5 queues to send data to your consumer? Why not bundle all data into just one queue?

- Why do you round floats just for display? You may simply set the formatting of the indicator as needed…

- use shift registers instead of all those local variables in the producer to keep the offset/tare values…

- why do you control the "write to file" in the producer? That should be done in your consumer where the writing actually is done…

- NEVER use the STOP function in your VIs to stop your VI. NEVER!

- why do you read the queue status as fast as possible? What's the point of reading it a million times per second?

- why is this while loop set to run forever?

- why do you use so many DAQAssistents in your motor control loop? Why not use plain DAQmx functions here?

[JonnyK]

GerdW,

Thank you, Thank you, This is juicy stuff 

I will do as you suggested and try N chan N samples mode. I also knew of the scaling feature in Daqmx, I have just done it a different way for so long. Stubborn I suppose. I will try the Daqmx feature. 

Next problem: why do you keep your samples in separate arrays in the producer? Why do you keep them in arrays of size 1???

To be honest I was just trying to pass the most recent value read by the transducer into an array then replace that value with the next most recent value using the shift register. I was trying to find a way to pass values between loops using queues and the queues like arrays. 

Next problem: why do you need 5 queues to send data to your consumer? Why not bundle all data into just one queue?

This is due largely to my ignorance, I did not fully understand how to bundle all into one queue. From some readings I recall they work as FIFO, but I was unable to fully grasp the concept. Can you point me to an example of how to bundle data into one queue then read it in the consumer loop?

- Why do you round floats just for display? You may simply set the formatting of the indicator as needed…

In previous versions of this program I rounded the values for display and also those same rounded values I am writing to file. It made a much more attractive toms file with rounded values.

- use shift registers instead of all those local variables in the producer to keep the offset/tare values…

This is very interesting. I would have never considered this. Can you point me to an example please?

- why do you control the "write to file" in the producer? That should be done in your consumer where the writing actually is done…

To be honest my previous version of this vi did all data writing in the consumer loop but I could not get it to function correctly. I came across a example vi where the the writing was controlled in the producer by releasing the enqueue element. I thought this was interesting and ran with it.

- NEVER use the STOP function in your VIs to stop your VI. NEVER!

I will delete. 

- why do you read the queue status as fast as possible? What's the point of reading it a million times per second?

I was reading the status out of frustration. I can delete that portion of the code. It was for trouble shooting purposes.

- why is this while loop set to run forever?

Typically I just quit the execution of the vi with the stop button in the upper left hand corner. Never had a need to do otherwise. Is this bad?

- why do you use so many DAQAssistents in your motor control loop? Why not use plain DAQmx functions here?

I am sorry I forgot to change this. It was a copy and paste from a previous vi. Are these DAQAssistents really bad news bears?

Again thank you

[GerdW]

Hi Jonny,

It made a much more attractive toms file with rounded values.

I don't know your definition of "attractive", but my collegues would kill me for rounding values before writing to a file…

I would have never considered this. Can you point me to an example please?

Please:

Typically I just quit the execution of the vi with the stop button in the upper left hand corner. Never had a need to do otherwise. Is this bad?

Well, do you stop your car by driving against a wall?

It's the same like using the STOP function or the STOP button in the menu…

Are these DAQAssistents really bad news bears?

They usually bring some (internal) overhead into your VI.

Especially when all they do is writing a scalar value to an AO in several places in that loop…

Can you point me to an example of how to bundle data into one queue then read it in the consumer loop?

In the current version of your VI I would have used BuildArray to put the 4 AI values into an array and send that array to your consumer. In the proposed suggestion using an 1D array per channel I would still use BuildArray to create a 2D array from your 4 channels and send it to the consumer. You just need to change your queue definition…

(There is a huge example VI library with LabVIEW, and there are examples for using queues…)

[JonnyK]

I have taken advantage of very good suggestions and I very much appreciate the comments.

Attached is a bare bones version of my Producer Consumer Analog Voltage Data Writing vi.

Currently vi is running well however I still would like to be able to write only a certain number of data points to file per second. 

Current vi lets me write to file at data acquisition rate. I would like to keep this data acquisition rate constant, say 10kHz, and then write 20, 100, 200, 700(arbitrary numbers) data points to file. I am having trouble understanding how to accomplish this without writing the whole 2D array to file. I would also like to include a dt timestamp to tdms file. That is to say write "5 Samples Per Second" and the time column reads " .2, .4, .6, .8, 1.0....etc". I am having trouble implementing timestamp to 2D array of numbers. Is there a better way?

Any suggestions are greatly appreciated.

Thank you,

-JK

[cbutcher]

Some thoughts on your most recently uploaded VI:

• Nice work with the initialization of the DAQmx channels. Although it takes up more space than might be nice, the way they all have the same layout makes it easy to scan them and get a feeling of what you're doing. The comments above them are also helpful.
• In your producer loop, you don't need the Queue reference to be wired to a shift register - a simple tunnel is sufficient for Queue references (as you use in your Consumer loop)
• Why are you using Flush Queue in your consumer? If the consumer falls behind the producer loop, then you'll start discarding data. Is this intentional?
• You have a couple of State Machines using a selection of different methods to specify state. You have strings controlling the motor, and integers for consumer and an "auto-record" loop. Have you considered Enums? These allow you to associate a label/string with a numeric value, allowing both "Add Case for Every Value" and prevention of typing errors or similar, whilst retaining the information held in a string.
• If you do try using Enums, create TypeDefs. This allows you to more easily change the values in the enum without making a mess of your block diagram constants.
• If "auto-record" is false, that loop will run at an unconstrained rate. Although I don't desperately like that loop at all (the use of division to specify time durations isn't the simplest thing to read), just adding a Wait (ms) function to case 0 will help (and for a short wait, ~50-100ms, probably won't be significantly noticeable).

Coming to the point of your question, if you can specify a fractional amount of data (like, 1 point in 5) then you can use the Decimate function in the Signal Operations subpalette of the Signal Processing palette to select a fraction of your data. This operates on 1D arrays, so you could wire your 2D array to a parallel For loop if the fraction and the lengths are the same in each case (I think they should be).

To implement a time axis, you can just create a 1D array of doubles by taking "dt" and multiplying it by "i" in a For loop of the appropriate length. Adding a constant each time you go around the loop would allow you to create the array once and increment it by the (acquisition) loop time, 0.1s in the code given. Take care if sample rate isn't divisible by 10.

If you instead need to specify a number of points, then I'd guess it's easier to first transform to a fractional decimation and dt value, then go as above. 

Note that you can't place dissimilar elements in an array, so you can't have a timestamp and double values. Either use timestamps separately (with two calls to TDMS Write) or use doubles and add it to the array.

[JonnyK]

Thank you very much for the reply. 

I will look into these fancy things called Enum's and will also implement delay into auto record loop.

If you have a moment?

"To implement a time axis, you can just create a 1D array of doubles by taking "dt" and multiplying it by "i" in a For loop of the appropriate length. Adding a constant each time you go around the loop would allow you to create the array once and increment it by the (acquisition) loop time, 0.1s in the code given. Take care if sample rate isn't divisible by 10."

Could you expand a little on "For loop of the appropriate length" and "each time you go around the loop"?

I am assuming a For loop of appropriate length is an array of doubles equal in length to the number of channels I am acquiring data from? Each time I go around the loop, do you mean a shift register? And would I specify a 'dt' to multiply by, for instance wire a constant?

Thanks again,

-JK

[cbutcher]

For loop of the appropriate length:

Here, I just meant if you have a 2D array already with Y values for some series of measurements, then they all have a number of samples - your Time array will need the same number of samples. Array Size will get you the value but be careful to index the correct value since you have a 2D array (I think that DAQmx returns N rows M columns for M results in each of N channels, so you'd want the second index value, i.e. Index Array and a "1" constant. I think... I have to check this constantly...)

Adding a constant each time you go around the loop:

Yes, my suggestion would be to put the "Time" array in a shift register and add a value to it each time which matches the change in time between producer loop iterations. The goal is to have the Time array be continuous in the TDMS file or wherever. If you don't add to the array, you'll have a Time array that reads something like

0, 0.2, 0.4, 0.6, 0.8, 0, 0.2, 0.4, 0.6...

when you'd presumably want

0, 0.2, 0.4, 0.6, 0.8, 1, 1.2, 1.4, 1.6...

In the example there, I'd do something like this:

If you run that snippet, the array should list the "current" time values for your data set.