Arrays versus waveforms for large data sets?

[Magma]

Is there a significant difference in functionality, memory use or speed of execution between using 2D arrays vs. 1D waveforms for large (>2M measurements) multichannel (16 channels) data sets? I've noticed different VIs default to one or the other format, and am wondering what the performance penalty of dynamically converting between types might be.

Regards,

Mark

[smercurio_fc]

Any time you have to "dynamically" change it costs. The performance penalty is, obviously, memory and CPU time. How much depends on how much memory reshuffling needs to take place. You also need to keep in mind that with arrays you must have a contiguous amount of memory available if a new array needs to be created. Thus, if you're dealing with large data sets you should always try to do this "in place". At a minimum you want to stick with the same way of storing data so you don't need to keep changing formats. That said, if you can provide some specifics as to what you are encountering, then specific suggestions can be made.

[Magma]

smercurio_fc,

I'm sampling a very weak repetitive signal in a noisy background using a multichannel DSA board. Through trial and error and some theoretical considerations I've found that oversampling, running a low-pass filter, and then compressing the data using a simple average works well (for this data, anyway). I then record the data in ASCII format and do post-processing (background removal, stacking and robust signal processing) in MATLAB.

I've found what works OK for 4 channels and 1.2 M readings/channel doesn't scale well to 16 channels and 2.4 M readings/channel (have had a few out of memory incidents), so I've started working with streaming TDMS VIs, and have also realized that much of what I was doing in MATLAB could easily be done in LabVIEW. These operations tend to be for the most part simple resampling, splitting a 1 x mn array into an m x n subarray, column sorts, etc. that are best suited for arrays. (My sampling time and rate are fixed, so I can pre-allocate the arrays at the start, if that helps)

I'm coming to LabVIEW the slow way via SignalExpress and express VIs and couldn't find a simple answer for the benefits of using waveforms vs. arrays. Obviously I don't want to be converting multiple times within a VI, so what I'm looking for is what advantage(s) there is(are) to using waveform data - besides the obvious one that many express VIs in signal acquisiton and processing are designed for them.

My gut feeling was that I could use waveform data type during the data acquisition and preprocessing stage, write it to disk in TDMS format, and then import the TDMS data as arrays for postprocessing after the acquisition stage when execution time is not so critical... but I wanted some feedback from experienced users.

[smercurio_fc]

---

Magma wrote:

I'm coming to LabVIEW the slow way via SignalExpress and express VIs

---

Well, there's the problem right there.   Just kidding. The "experienced" users around here hate Express VIs.

---

what I'm looking for is what advantage(s) there is(are) to using waveform data - besides the obvious one that many express VIs in signal acquisiton and processing are designed for them.

---

Actually, most Express VIs are designed to work with dynamic data (which is another abomination). The main advantage of waveform data over pure arrays is that the waveform datatype contains time information. The main disadvantage is that you can't create 2D waveform datatypes. In other words, the "Y" element of the waveform datatype is always a 1D array. Is either of these important to you?

[Magma]

Thanks. In this case the absolute time is not important - dt is, but is known and constant so the array index should work fine instead. I don't need 2D arrays during acquisition, but later on, yes.

I have picked up very subtle vibes against Express VIs on other forums...

So I'll try rewriting the logging part using arrays and see how the performance compares.

[Carlz]

Hi

Im having problems with really large data sets also and I wonder the best format to store them in.

I want to read 11 lines from a port on my daq. The port has 32 lines. What is the best way to read this in terms of performance.

My problem is when reading the data as 32bit the files become huge! (32bit = 4 byte. Samle rate 100kHz --> 400kb/sec. 11 lines --> 4.4mb/sec!)

I have tried to read the data on single channel but then it seems to just be storing the data for 1 line.

Is there any advice on this?

Regards

Carl Zandén

[Coq_Rouge]

Just to ask since I have never used digital input and Labview. But is every digital line represented with a U32? You should at least use U8. That will cut your memory needs to 1.1 Mb/sec. You could also compress more. I made a quick and dirty example showing how in 7.0. It takes about 37 ms to "compress" 1 second of measurement. And the data is reduced to 200Kb/sec

Message Edited by Coq rouge on 05-08-2009 01:41 PM

[Carlz]

Hi.

Actually I think I was mistaken. I was confused with the concept of Channels and lines in DAQ. 

I think it is possible to take readings of the entire port with one u32. Thus each reading 32bit = 4 byte , 4*100k = 400kb/s. 

What I'm a bit uncertain about is how the physical channels setting affects the reading.

eg. I wanna read line 16 to 26 from my 32 line port.

So I select the physical channels to be dev1/port0/16:26,

choose the property one channel for all lines, and daq mx read single channel with 32 bit.

My question is: How are the bits in the 32 bit reading assigned?

Is it 1st bit = line 16, 2nd bit = line 17....11th bit = line 26?

Or is it:

1st bit = empty....16th bit = line 16, 17th bit = line 17... etc?

If it is the first of these cases. Would it be enough for me to read 16 bit since the rest of the bits not interesting/ empty?

Another question since you seem to know how to work with bits =).

How do I invert the bit reading for each line?

eg. reading is 32bit number =  0 1 1 1 ....0

I want the inverted: 1 0 0 0 ....1

Thank you very much for taking time.

Regards

Carl Zandén 

[Coq_Rouge]

You could use the "Split Number" function. Owning Palette: Data Manipulation. The boolean "not" functions do also work on integers. Use this to flip the bits

[smercurio_fc]

---

Carlz wrote:

Hi.

Actually I think I was mistaken. I was confused with the concept of Channels and lines in DAQ. 

I think it is possible to take readings of the entire port with one u32. Thus each reading 32bit = 4 byte , 4*100k = 400kb/s. 

What I'm a bit uncertain about is how the physical channels setting affects the reading.

eg. I wanna read line 16 to 26 from my 32 line port.

So I select the physical channels to be dev1/port0/16:26,

choose the property one channel for all lines, and daq mx read single channel with 32 bit.

My question is: How are the bits in the 32 bit reading assigned?

Is it 1st bit = line 16, 2nd bit = line 17....11th bit = line 26?

Or is it:

1st bit = empty....16th bit = line 16, 17th bit = line 17... etc?

If it is the first of these cases. Would it be enough for me to read 16 bit since the rest of the bits not interesting/ empty?

---

This is all explained in the LabVIEW Help, and is dependent on how you set up the DAQmx Create Channel. Right-click on each function and select Help. Especially on the DAQmx Read. Since this function is polymorphic, select the specific instance of it to jump down to the help for that specific version. For the data output it says:

data returns a 2D array of 32-bit unsigned integer samples. Each element in the first dimension of the array corresponds to a channel in the task. Each element in the second dimension of the array corresponds to a sample from each channel

The underlined portion is a link to another page that has more details.

If you are still not clear after you've read the help, please post back.