Smart way to save great amounts of data using circular buffer

[michaelboiger91]

Hello everybody,

i'm currently stepping into LabView as i'm developing a five-channel measurement system. Each 'channel' will provide up to two digital counter inputs, up to three analog RSE inputs (sample rate will be around 4k to 10k each channel) and up to five analog thermocouple inputs (sample rate will be less than 100 S/s). Depending on user specified events (such as sudden rotational speed drops) the system should save a TDMS file containing a row for each data channel, storing the values n seconds before the impact happened, and with an user specified length (e.g. 10 seconds before the rotational speed drop and then with a length of 10 minutes).

My question is how to handle those rather huge amounts of data in a smart way, and how to get the error cases on the harddisk without loss of samples and without dumping huge amounts of data on the disk while recording signals when there is no impact. I thought about the following:

- use one single producer loop to only acquire data at constant and high rate and write data into queues

- use consumer loop to process the packages of signals when they become available and to determine impacts and save data when impact is triggered

- use third loop with event structure to give the ability to control the VI while not having to poll control elements from the front panel every time

- use some kind of circular buffer in the consumer loop to store a specific number of data that can be written to the harddisk.

I hope that this is the right way to do it so far.

Now i thought about three ways to design the circular data buffer:

- use RAM as buffer (queues or arrays with limited number of entries) which is written to harddisk in one single step when finished while the rest of the program and DAQ should still be active

- directly stream to harddisk using the advanced TDMS functions, and re-setting the TDMS Write Position markers back to the first entry when a specific amount of data entries was written.

- directly stream all data to harddisk using TDMS streaming, splitting files at a certain time and deleting TDMS files that do not contain any abnormalities later on at runtime.

Concerning the first possibility, i'm afraid that there will be problems with quickly growing arrays/queues, and especially when it comes to save the RAM data to disk, my program would be stuck for a time writing only data to disk and thus losing samples in the DAQ loop which i want to continue without interruption.

Concerning the second possibility, i'm encountering trouble with TDMS, as data gets corrupted easily and i'm not certainly sure if the TDMS Set Next Write Position is suitable for my needs (i have to set positions for (2ctr+3analog+5thermo)*5channels=50 data rows plus timestamp row in worst-case!). Also i'm afraid the harddisk will not be able to write fast enough to stream all data simultaneously in worst-case scenario..?

Concerning the third possibility, i'm afraid that closing TDMS file and opening a new TDMS file to continue recording will not be fast enough to not lose any data packets.

What are your thoughts here? Is there anybody that has already dealt with similar tasks? Does anyone know some raw criterions on how much data can be attempted to stream to an average speed hard disk at the same time?

Thanks a lot

[Bob_Schor]

I use fixed length Queues filled by Lossy Enqueue as my Circular Buffers.  I've used them to hold "pre-Event" data (so when the "trigger" appears, they can be simply Dequeued to get the data preceding the trigger in the correct order) and also as elements of a "moving average" buffer (where I simply use Get Queue Status to dump the 10-20 points into an Array and return the mean as the Moving Average).

Bob Schor

[altenbach]

What does your attached picture have to do with this discussion? (All I see is this :D)

[michaelboiger91]

Hello,

thank you Bob_Schor for the idea of using lossy queues. Do you dequeue data in a second loop where the data is then transferred to harddisk, while the acquisition loop continues to read values? If my queue contains a maximum of 1500 elements, i could start dequeuing in a second loop - after, let's say, 1000 elements post-trigger. This would mean that i'm dumping 1500 elements to disk, where element 0-499 is the time domain before the trigger event happened, and element 500-1499 the time domain after the trigger event. I'll try that out.

@ altenbach

This picture was just intended to give an overview about my planned VI structure. I think that's quite important as it's difficult to change something here that after the program was written. As a beginner, so many faults can be made, so i wanted to ensure nobody here has objections against this structure design.

Greetings

[Bob_Schor · Accepted solution]

OK, I'm reaching back four years when I implemented this scheme, so bear with me.

Let's consider having a Trigger and wanting to capture N samples before the Trigger and M samples after the Trigger.  The scheme is a little complicated, as the goal is to not "miss" any samples.  We came up with this several years ago, and it seems to work -- there may be an easier way to do this, but never mind.

We created two Queues -- a fixed length "Pre-Event" Queue of N samples and a Per-Event Queue of unlimited size.  We use a Producer/Consumer design, with State Machines running each loop.  Without worrying about naming the States, let me describe how each works.

The Producer starts in its "Pre-Trigger" State, using Lossy Enqueue to put data on the Pre-Event Queue.  If the Trigger does not occur during this State, we stay here for the next Sample.  There are some details I'm forgetting about how we ensure that the Pre-Event Queue is full, but skip that for now.  At some point, the Trigger switches us to the Per-Event State.  Here we enqueue into the Per-Event Queue, counting the number of elements we enqueue.  When we reach M, we switch states back to the Pre-Event State.

On the Consumer side, we start in an "Idle" state, where we simply ignore the two Queues.  At some point, the Trigger occurs, and we switch the Consumer to the Pre-Event State.  This is responsible for dequeuing (and dealing with) the N Elements in the Pre-Event Queue, then handling the subsequent M Elements in the Per-Event Queue.  [Hmm -- I don't recall how we knew when the Per-Event Queue had finished -- did we count to M, or did we dequeue until the Queue was empty and the Producer was in the Pre-Event State again?].

There are some "holes" in this simple explanation, some of which I think we filled.  For example, what happens when triggers come too close together?  One way to handle this is to not allow a trigger to be processed until the Pre-Event Queue is filled.

Bob Schor

[michaelboiger91]

Hi,

sorry for the late answer.

This is quite a nice idea, and i have already tested it with some small modifications. Lossy Queue 1 is filled with elements as long as no trigger event occurs, then after trigger appeared, through a case structure, the data is saved into another Queue 2 until the internal condition for stopping the recording process is triggered (this can be a time limit, or a logics that recognizes when the error has vanished, for example if rotation speed stabilizes again). A notification is sent to the second consumer loop which reads all elements from Queue 1 (if size of queue is not constant due to short run-time, it's possible to get the number of elements by using the Get Queue Status block), and then read as many elements from Queue 2 as defined (again, this can be a fixed number or a "stop read Queue 2" trigger upon error vanished).

Concerning greater amounts of data, i will try to write cluster data into that queue, containing the low-res values like rotation speed, temperature etc as DBL and the high-res sampled analog data like current waveform with 8kHz in an array. As dequeuing and streaming to harddisk runs parallel to the producer loop, i hope there will be enough calculation time to not exceed the deadline of those tasks.

Greetings