Storing and analyzing large Arrays

[jereger]

Hello NI Community!

I'm working with LabVIEW 18.0 and a NI PCI 6503 DAQ card to collect digital input signals from a hall effect sensor on a flight mill. Currently, the signal is collected at a adjustable frequency in a while loop. When the program completes, the data is passed to a for loop to be written to a text file and the number of falling edges is recorded on the front panel. The text file contains a boolean (0 or 1) value representing if the sensor was being triggered and a corresponding timestamp. 

Issue:

1. File size: At the current sampling frequency (40 msec) about a million data points are recorded to the text file during a run. This includes the times when the sensor was triggered (minority) and was not triggered (majority). I'm going to have to increase the sampling rate to 1 msec in order to collect data more accurately but this would create a file too large from me to work with. Is there way I can write into my program to only write the events where the sensor is triggered to the text file? 

Thanks 

Joshua 

[crossrulz · Accepted solution]

1. Use a Producer/Consumer so that your analysis happens in parallel with your data acquisition.  This will eliminate the need for that large array.  Part of your queue data should be the timestamp of when you read that sample.

2. Since you are only getting a single sample of a single line from the DAQ, you really should set your DAQmx Read to output a single Boolean, not an array.

3. You should use Format Into String for creating your line of data.  All of your timestamp formatting, tabs, and Boolean to Number conversion can all be done with that single function call.

4. There is a VI in the palettes called Boolean Crossing PtByPt.  Use that to do your transition checking.

5. You should have some logic for when to log the data.  Use a case structure around your file write so that you can conditionally write to the file.

[jereger]

Thank you for the advance! I was able to locate all the vi's you mentioned and read up on producer and consumer loops. I wasn't sure how to include both the timestamps and my data into the queue, so I used a digital waveform. I believe the waveform has the time included in the stream. I'm still very new to labVIEW and was wondering if this looks like I'm started on the right path.

[crossrulz]

---

@jereger wrote:

I wasn't sure how to include both the timestamps and my data into the queue, so I used a digital waveform. I believe the waveform has the time included in the stream.

---

That should work.  You just have to use Get Waveform Components and get the T0 value.

[Bob_Schor]

I'm going to assume (!!) that your transitions happen "fairly infrequently", since while sampling at 40 ms = 25 Hz, the "majority" of your points are not "in transition".  So for purposes of exposition, I'm going to assume that the transitions are on the order of 10 Hz = 100 msec apart.  I'm also going to assume that you are really interested in only one "edge" of the transition, say the Rising Edge.

So here's a suggestion based on these assumptions:

• Acquire Digital In data using Continuous Sampling at 1 kHz, saving the data as a Waveform.  For this example, collect 1000 points/sample.
• Use a Producer/Consumer Loop to send the data every second to a Consumer loop.
• The Consumer Loop looks for rising edges (it should find, on average, 10, by assumption).  For each one, it computes the Time of Occurrence by adding the number of milliseconds corresponding to the index of the Sample to the Waveform's t0, and saves only this "time of transition".  This gets written to disk, gets sent to a Display Routine, or whatever else you need to do with it.

This scheme saves data only when an Event of interest occurs, and saves precisely the information you need, namely the time of the transition.  Thus if you are saving 10 Channels being sampled at 1 kHz for 1 hour, saving all the binary data would require 10 (channels) * 1000 (samples/sec) * 3600 (seconds/hour) * 1000 bytes (approximate size of Waveform record) = 36 Gb, whereas saving just positive-transition Time Stamps would require 10 (channels) * 10 (transitions/second) * 3600 (seconds/hour) * 128 bytes (size of TimeStamp) = 46.08 Mb, three orders of magnitude less space.  Note that you can save even more space (but might not need to bother!) by saving a t0, when the recording starts, and a simple millisecond "elapsed time" -- a U32 (4 bytes) can hold about 7 weeks of "milliseconds".

Bob Schor

[mcduff]

---

@Bob_Schor wrote:

I'm going to assume (!!) that your transitions happen "fairly infrequently", since while sampling at 40 ms = 25 Hz, the "majority" of your points are not "in transition".  So for purposes of exposition, I'm going to assume that the transitions are on the order of 10 Hz = 100 msec apart.  I'm also going to assume that you are really interested in only one "edge" of the transition, say the Rising Edge.

So here's a suggestion based on these assumptions:

• Acquire Digital In data using Continuous Sampling at 1 kHz, saving the data as a Waveform.  For this example, collect 1000 points/sample.
• Use a Producer/Consumer Loop to send the data every second to a Consumer loop.
• The Consumer Loop looks for rising edges (it should find, on average, 10, by assumption).  For each one, it computes the Time of Occurrence by adding the number of milliseconds corresponding to the index of the Sample to the Waveform's t0, and saves only this "time of transition".  This gets written to disk, gets sent to a Display Routine, or whatever else you need to do with it.

This scheme saves data only when an Event of interest occurs, and saves precisely the information you need, namely the time of the transition.  Thus if you are saving 10 Channels being sampled at 1 kHz for 1 hour, saving all the binary data would require 10 (channels) * 1000 (samples/sec) * 3600 (seconds/hour) * 1000 bytes (approximate size of Waveform record) = 36 Gb, whereas saving just positive-transition Time Stamps would require 10 (channels) * 10 (transitions/second) * 3600 (seconds/hour) * 128 bytes (size of TimeStamp) = 46.08 Mb, three orders of magnitude less space.  Note that you can save even more space (but might not need to bother!) by saving a t0, when the recording starts, and a simple millisecond "elapsed time" -- a U32 (4 bytes) can hold about 7 weeks of "milliseconds".

Bob Schor

---

Too much work. If Bob's assumptions are correct, and you have a digital line, you can create an event case to trigger when the line changes states; then do what you need. You need to know how to use an event structure.

mcduff

[Bob_Schor]

@mcduff,

     A much neater way to "get the data".  I presume you'd still advocate storing only "Rising Edge" data using a format similar to what I suggested?  [Unless, of course, you want "Falling Edges", or "both Edges" ...].

Bob Schor

[jereger]

Thank you for your help! It took a few days to piece all of the information together, but I got it to work.