Best strategies for reducing effective data collection rate?

[CFJ]

Hi,

I'm writing a VI to collect data from cDAQ modules that have a minimum sampling rate of 1612Hz due to the master timebase (specifically 9237 and 9215 modules). I would like to log data (1D waveform with timestamps) at 100Hz for 12 hours and save this to file. THe data also needs to be displayed during this time, and data logging should be robust so there is minimal risk of data loss.

I've made one attempt at doing this previously - I used the Align and resample expressVI but this was fairly messy, and the problems I ran into were that I had errors about the maximum array size being reached, and also constant buffer overwrite errors. 

I would like to basically start this VI again from scratch, and I'm wondering if there are any suggestions for an overall strategy for this?  I'm not asking for any code to be written for me, just concrete shoves in the right direction regarding things like data file type, how to resample or downsample data, how to clear arrays, etc.

Thanks,
Claire.

[lopez]

Clair,

I often need to sample data at high rates.  Many problems in the past have come from using text based write and read.  I find TDMS format is much better for high speed data acquisition, or large file sizes.

Also, I have been advised by VT92 that a producer/consumer architecture can be used for this kind of application. Which I am currently trying to navigate. In this scheme you have one loop that is producing data, while another loop consumes it.  Hence, you have a buffer that is always being filled and them emptied and you don't bog down the processor.

good luck.

AL

[johnsold]

Claire,

You mentioned a minimum  sample rate of 1612 Hz.  Can you sample at a rate like 2000 Hz, which is an integer multiple of the desired 100 Hz?  Then just extract every 20th sample and construct your waveform from those.

Lynn 

[CFJ]

Hi Lynn, thanks for your reply. Yes, that's what I figured out to do, thanks for the suggestion. I've been using the expressVI "align and resample" to do the downsampling. I've posted what I've done so far on the thread below, as you can see I'm still having difficulties with the writing to file. If you have any ideas on that I'd be grateful.

Cheers, 

Claire.

http://forums.ni.com/ni/board/message?board.id=170&message.id=401265#M401265

[johnsold]

Claire,

It appears that you are appending the resampled waveform to an empty array of waveforms and then writing the appended array to the file.   What does appending to an empty array accomplish?  Just write the decimated array directly to the file.

If you want to have an array of all the waveforms, then you need to use shift registers to pass the appended array to the next iteration of the consumer loop. 

You do not need the sequence structure.  Dataflow takes care of making things happen in order.

I do not see how the feedback node will ever change the file name as it only executes once.  I am not sure what you intended, but you may want to move the file path creation inside the loop and put it into a case structure which executes when it is time to create a new file.

The attached image is an example of the way I was thinking of decimating your data.

In both my image and your program the use of build array in a loop is not a good practice it the array can get large due to memory allocation issues.  Better is to initialize an array of the maximum size and use Replace Array Element to put the data into the array. 

Lynn 

[CFJ]

Hi Lynn,

I've attached my attempt at incorporating your decimating method.  I have put a "replace array subset" block in the loop, and initiated an array of 100 rows (i'll increase this later).

Three main problems:

1) the queue waveform comes in as an array - how can I reference just the next element of the queue? is the next element added to the top or the bottom of the queue array?

2) what should happen inside the "1" case? (so the that the terminal has connections in both cases).

3) with this approach it seems that the timestamp is lost as it always remains a waveform type which can't be decimated - is this correct?

Thanks for your help, 

Claire.

[DFGray]

Instead of throwing away your data, consider averaging it or taking a max/min.  A simple decimation may be good enough, but really does not describe what is going on unless you know your input will not change very fast.