Read TDMS in loop to make trend plot

[LabViewer35242]

I'm trying to create a trend plot from waveform peak values on all 32 channels of my data.  The only way I came up with was to put my TDMS read inside a case structure.  If you want to build a trend you select the "Build Trend Boolean" and this case has the TDMS read inside a for-loop that steps through every sample.  For just scrolling through the data in timewaveform and spectrum plots you would be in the false case which is not in a loop.  I just have a scrollbar going to the TDMS Read offset input.

I get an error with the TDMS Read inside the for loop.  I'm guessing it has to do with the TDMS file from the TDMS Open coming into the for loop.  I get this error:

Possible reason(s):

LabVIEW:  End of file encountered.
=========================
NI-488:  Invalid argument or arguments to function call.

[DFGray]

The error is correct.  You are reading all the data in the first iteration of your loop, ending up at the end of the file.  You then attempt to do the same thing again, which gives you and EOF error.  You have two options:

1. Read a subset of the data in the loop.
2. Read all the data outside the loop, then parse a subset of it in the loop.

It would probably be easiest to implement option 2, but option 1 would give you better performance.

[LabViewer35242]

So the TDMS read when in a loop can only read the block of data that you give it with the initial offset and count? I was trying to look at one block at a time, and still don't understand why it can't go to the next block of the waveform. Each block is 4096 data points per waveform.  I have 4814 blocks to read. In the first iteration I'm looking at the 1st block of data (0-4096) and it sends out the correct array of peak values. Then as you said I'm at the end of the block and I get my error when trying to read the next block.

If I understand correctly in the next iteration I'm looking at the second block of data (offset = 4097, count=4096; so samples 4097-8192), but you're saying I can't look at this block of data b/c when the loop initializes (offset = 0, count=4096).  Is this data (offset=0, count=4096) the only data available once the loop starts?

To get started in the right direction per your suggestion:

" 1. Read a subset of the data in the loop" - To do this do I need to initially read all the data (offset=0, count = 19,718,144) and pass this into the loop and then use the waveform subset VI to do what I'm trying to do in my loop?  Is there a better sub VI for this?  Can I use a TDMS read outside the loop to read (0-19,718,144) and then another TDMS read inside the loop to read (0-4096)?

" 2. Read all the data outside the loop, then parse a subset of it in the loop." - To do this do I need to initially read all the data (offset=0, count = 19,718,144) then.... I'm not really sure what to do.

[LabViewer35242]

I've attempted to read all of the data at once outside the loop, but this gives me a memory full error.

[DFGray]

Given almost 20 million points, I would strongly recommend you only read from file what you are attempting to analyze at the time (option 2 above).  If you have not already done so, take the time to read the information in the LabVIEW help about dealing with large data set.  Or read Managing Large Data Sets in LabVIEW.

My original post was incorrect (my apologies).  You were correctly trying to read a subset of your data using the count and offset.  However, you were not specifying a group or channel, which you should do.  Use your waveform selection to select a channel from a block diagram array constant containing the names of the channels.  It appears you only have one group, so code that in (or put a control in so you can easily change it in the future). This will leave you with a single waveform instead of an array of waveforms for the output, so initialize the data type input accordingly.

Note that you may wish to constrain your offset slider so it cannot read more than the total number of blocks in the file.  You can do this programatically by determining the number of blocks in the file and setting the maximum value of the slider (or just its scale maximum) using property nodes.  You will get an EOF if you try to read a block past the end of the file.

Let us know if you continue to have problems.

[LabViewer35242]

I'm still struggling with reading through this data.  Before I upgraded to TDMS there was a VI that worked great.  It wasn't an issue changing what part of the file you wanted in a loop like I'm trying to do.

In response to only reading from the file what I need at the time:

Let me give you a little more background on what I'm trying to do.  I'm trying to create a vibration viewer.  I have written these big files (20GB) of data from 32 channels.  Now I'm working on the viewer .exe to look at this data.  I'm trying to replicate what most of the vibration software on the market allows you to do.  Right now I'm just trying to replicate a trending feature.  All their trending does is take the max or min value from a particular block of data (if analysis frequency=1000Hz and # of lines=3200 then one block is 3.2s).  So for every block of data I want to get the one max/min and trend it.

So ideally you could look at this trend plot (perhaps push a boolean to build it) and see 32 lines going across the screen.  Each line would be a channel of data composed of  some "#-of-samples=NI_channellength/waveform_samples.  I'll also add a cluster of elements so you can select only certain channels to trend.

When I did this with regular binary files I created the trend as soon as the user opened the file.  It took about 5 seconds to go through one channel. I made just a real quick display of how I did this before with the binary file.  I guess the difference is that with the "Read from Binary" VI before my datasets were smaller so I could read all of the data into memory and pass it into that top for loop.

I've attached a quick view of how I did it before with the binary file.

[YongqingYe · Accepted solution]

When using TDMS Read in a loop, you need to be a little bit careful when you not wiring the group name. 

In TDMS Read documentation:

If you do not wire data to this input, LabVIEW reads data from the first group when you run this function for the first time. If you run this function continuously for multiple cycles without wiring the group name in input, LabVIEW reads data from the nth group at the nth cycle.

[LabViewer35242]

Thanks YongqingYe this was my main problem. I'm now able to step through all of the data as I was needing to.  I have a few more things to fix: build arrays of the data and send timestamp for x-axis.  I should be good to go from here.  Once I get through all of the differences between my old binary write/read this TDMS write/read will be much better.