data streaming

[Cowboy]

I would like to be able to create a circular buffer to acquire approximately 3 minutes worth of data sampled at 12800 S/s and up to 16 channels. If an event occurs, I would like to store the pre-event data to disk and capture three minutes worth of post data. I have done some experimentation with this and so far the acquisition buffer of the 8176 with 512MB of RAM is unable to keep up for more than a few seconds. I am storing the data into a binary array but that still doesn't get the jobe done. Would someone have any suggestions?

[Ben]

Hi Cowboy,

You are really pushing the limits here.

Three minutes of data will use about 1/4 of your memory. This is assuming that tohe data never gets duplicated.

You are going to have to work "in-place".

I can not give away my circular buffer code but I am allowed to advise.

Could you post some sample code so I do not have to start from page one?

Quick note:
Use action engine
Init buffer ahead of time
Replace array sub-sets
Use seperate pointer for updates and reading

Watch your memory usage! Once you go virtual, the game is over.

Ben

[Cowboy]

Ben,

I appreciate your quick response. I have attached the rather crude code that I am using to initially test the process.

I am new to LabVIEW development so forgive my ignorance, but I do not know what "action engine" is. I am also unsure of what you mean by use seperate pointers for updates and reading. The other things you mentioned I have done although in the sample code I am sending you, I am using Array Subset instead of Replace Array sub-set.

Thanks again for your response,

Mark Blankenship

[Ben]

Hi Mark,

A complete answer will take longer than I have available this AM.

1) Add another AI Read before your existing call and wire a constant of "0" into the # of scans to read. This should take no time and will return the # of samples waiting in your buffer. Wire the backlog from this first AI Read into the # of scans to read of your existing AI Read. Reason: AI Read will wait until the # you spec is available. It will chew up CPU until that # is ready. The above change will get your data as fast as it comes in and will not waste time doing it.

2) The shift register that you are saving info in is being RE-SIZED every iteration of your loop becauwse you are doing a build array! When an array has to be sized up, LV will check to see if the new data will fit into the memory block that is already allocated. If not, it will allocat a larger block. This requires a call to the OS to allocate memory. This call is slow and will kill your app! To avoid this re-allocation of memory, initialize you shift-register to a size that will hold all of the data you will ever store BEFORE the loop starts. This way the memory is just sitting there waiting to be filled.

3) Second part of #2. Get rid of your "build Array" functions. Use a replace array subset to put the data where you want it. A replace array subset will operate in-place. THis means that the same data buffer that is passed to the node will be used as its output. No need to copy etc.

4) Add another shift register to be used to track where the next update should be stored. This will be the index that you pass to your replace array subset.

5) Get rid of your "Array 3" indicator. The presense of the wire going to that indicator tell LV you want it to copy all of your data into that indicator on every iteration. Not onyl that waste your memory (with the extra copy for the indocator) but it also wastes alot of time. Any other controls and indicators that you do not need inside the loop should be tossed.

I believe these suggestions will get you somewhere.

Trying to help,

Ben

[Cowboy]

Thanks again Ben! I will try your suggestions and see where that leads. I will let you know.

The reason for the Array3 was that it was temporary while I tried to ensure that I was inputting data correctly. I just forgot to take it out. I also initialized the array at one point during playing with what to do and forgot to add it back in. I kind of rushed sending you an example without checking fully what I was sending.

Thanks again for your response and assistance.

Mark E. Blankenship ("Cowboy")

[Ben]

No problem Mark!

I ask only the following;

1) Keep us (others are reading along) informed as to how things go.

2) You don't give on LV without trying what the Enthusiast suggest.

3) Be patiant with us responding (I, like most Enthusiats,are volunteers and can only answer in our spare time).

4) Rate my answers if they help. This will let others quickly find the answers to their Q in the future.

Keep us informed,

Ben

[Cowboy]

Ben,

I am working on trying your suggestions but I have a question.

By using the Replace Array Subset.vi, how do you maintain a circular buffer? What I wind up with is a buffer that contains all of the data, but I have to search the array to find where it left off. Then I guess you would have to parse the array to put it back together in the right order. This isn't necessarily a problem if it gets you your end result, but I thought maybe I was missing something.

I have attached an example of the circular buffer to illustrate what I am talking about.

I am working on this code from home and don't have the equipment here to test my results with the other modifications you mentioned. That will h
ave to wait until Monday. I also attached the code so you could look at it to make sure I understood you correctly.

Thanks again,
Mark

[Ben]

Hi Mark,

Sorry about the delay.

First;
Is the code you have keeping up with your data stream?

Second;
The deatils of circular buffer are dependent on how you plan to use it.

Suggestions;
You could use a couple of SR's to do the following;
1) Buffer you already have this
2) Write pointer - Use to store the index where new data will be stored. After use, it should be updated by adding the size of the current update OR set back to zero if you close to or at the end of the buffer.
3) Last value - Stores the location of the last valid value. This is used when the Read Pointer (see below) is greater than the write pointer. This will be the condition when the buffer has filled up and writing is starting back at teh begining. Should start being equal to the write pointer. This will be the case until the buffer has filled and the write pointer gets set back to zero. I this case the Last value will stop following the write pointer and stay fixed until the next time the buffer loops back on itself.
4) Read pointer - is updated any time you do a read. Normal reads will fetch data between read pointer and write pointer unless the Read pointer is greater than the write pointer. The will be the case when the oldest data is between the Read pointer and the end of the buffer and the writes are occuring at the start of the buffer.

Like I said in my first post, I can not give away code. I am allowed to answer to my hearts content though.

The whole thing operates sorta like a snake chasing its tail around a ring. It keeps growing at the head end and its tail stays fixed. When ever you read, you grab the whole snake from tail to head. The next write operation starts the snake growing again.

There are all kind of variations on the above that I have used. Some of these are;
a) Dual read pointers - allows screens to be updated pseudo-Real-Time while mainataining a pre-trigger buffer for logging.
b) Auto-loop around - Handles condition where reading from the end of the buffer but new data is available at the start of the buffer.
c) Re-Settable Read Ponters - Allow clients of the buffer to specify how far back in time it want to read.

That is all I can say for now! I have a meeitng coming up to talk about our backlog!

It may be a good idea to re-submit this Q as a new Q on the LabVIEW-General list. You will get more opinions (and help) there and it looks like I will have to be a little busy elsewhere over the next couple of days (weeks).

Ben

[Cowboy]

Thank you very much for your time Ben! I do appreciate it and will try your suggestions. You given me some points to ponder and quite a challenge to figure out how to implement them. I look forward to it.

I may also repost the question as you suggested to get some other opinions on the issue. It never hurts to ask questions. My best learning I think will come from it.

Thanks again and I hope your backlog improves. I am working under quite a backlog myself. No fun!

Mark