How do I stream continous Digital Waveform data to a TDMS File

[Calvus]

Hi,

I have a question regarding how best to continuosly acquire digital waveform data and save it to a TDMS file using change detection as the clock mechanism. I have developed a framework to do this which "mostly" works but am hung up on a couple of issues that I don't quite understand.

I am using an event based producer/consumer design pattern with two additional loops, one for error logging and one for charting/saving the waveform data. The consumer loop only acquires the data and then queues it for the chart loop to dequeue, compress and save the data to TDMS. (I am acquiring up to16 lines of data from a PCI-6229)

Everything works fine if I do not try to write the data to file and the dequeue mechanism is able to keep up with the data acquisition. However, when I enable writing the data to a file the dequeue function can't keep up and the queue quickly grows to contain thousands of elements waiting to be written to file - which makes sense but from what I have read.

Also - how do I determine what values are best for the Number of Samples and the Sample Rate using change detection?

Attached are 3 screen shots of the LV code.

The first depicts the Acquire Data state of the consumer (DAQ) loop.

The second depicts the chart loop with TDMS disabled

The third depicts the chart loop with TDMS enabled

Many thanks in advance.

[andre.buurman@carya]

Where do you do your change detection?

You could consider to use a boolean array for your sample data?

How often do the lines change?

Do the states of all lines be recorded if only one line changes?

Your sample rate should depend on the fasted change you want to capture and whether or not you need info on the steepness of a change.

I must say the producer/consumer loop is the way to go with this type of application and hoera for the neat wiring :).

Message Edited by andre.buurman@carya on 06-11-2009 08:49 PM

[Calvus]

Where do you do your change detection?

A - Not sure what you mean by this - am looking for changes on 1-16 digital input lines - Screen shot attached

You could consider to use a boolean array for your sample data?

A - Sure - how is that better than a compressed digital waveform in terms of efficiency or file size?

How often do the lines change?

A - From 0hz when idle to 50Khz when active. There could be minutes of idle separated by minutes of activity depending upon the state of the device being monitored.

I want to be able to record for at least an hour - the data file be HUGE but HD space is not an issue.

Do the states of all lines be recorded if only one line changes?

A -  Yes.

Your sample rate should depend on the fasted change you want to capture and whether or not you need info on the steepness of a change.

A - That would be 50Khz- would I need to consider sampling at Nyquist (100Khz) when using change detection?

Not sure what you meant by "Steepness of change"? Are you refering to the line edges/slew rate or the number of changes detected in a given period?

I must say the producer/consumer loop is the way to go with this type of application and hoera for the neat wiring .

Thanks, it will be even more neat and better commented when the desired functionality is established.

What are your thoughts on a solution for the queue filling up while waiting to be written to file - continuosly streaming data to file must be a fairly common requirement?

Thanks for your help!

[Herbert Engels]

Boolean arrays are not going to be much faster than digital tables/waveforms. In case of compressed waveforms, they might actually be slower.

I tried a simple VI (see attachment) on my machine (average office PC) that stores digital waveforms (50000 samples, 16 lines each) to TDMS. I can continuously write 18 of these waveforms per second. Is that about what you're seeing on your machine?

Herbert 

[andre.buurman@carya]

---

Calvus wrote:

Where do you do your change detection?

A - Not sure what you mean by this - am looking for changes on 1-16 digital input lines - Screen shot attached

OK

You could consider to use a boolean array for your sample data?

A - Sure - how is that better than a compressed digital waveform in terms of efficiency or file size?

See Herberts reply.

How often do the lines change?

A - From 0hz when idle to 50Khz when active. There could be minutes of idle separated by minutes of activity depending upon the state of the device being monitored.

I want to be able to record for at least an hour - the data file be HUGE but HD space is not an issue.

OK, you definitely want to use change detection.

Do the states of all lines be recorded if only one line changes?

A -  Yes.

OK

Your sample rate should depend on the fasted change you want to capture and whether or not you need info on the steepness of a change.

A - That would be 50Khz- would I need to consider sampling at Nyquist (100Khz) when using change detection?

I would choose a sample rate of at least 100kHz, since you only save data at change it should not affect your total amount of data acquired.

Not sure what you meant by "Steepness of change"? Are you refering to the line edges/slew rate or the number of changes detected in a given period?

Line edge characteristics.

I must say the producer/consumer loop is the way to go with this type of application and hoera for the neat wiring .

Thanks, it will be even more neat and better commented when the desired functionality is established.

What are your thoughts on a solution for the queue filling up while waiting to be written to file - continuosly streaming data to file must be a fairly common requirement?

Place to graph out in a different loop, displaying graphs can take to much time when acquiring at high speeds.

Thanks for your help!

---