Write Data storage VI performance issue

[Seaman008]

I have a program that logs data from mulitple channels.  It waits for 500 points from each channel, then saves the data (as a raw 2D DBL array) to file, wait for another 500 points, saves and so on.  At the end, I have a raw file.  I created a utility VI that would take the raw file and converts it to a tdm file so I can read it in Diadem.  The utility VI has a for loop.  In each iteration, the loop reads one data segment (the 500-point 2D array), converts the channel data to waveforms and then writes/appends to a tdm file using the Write Data (Channel) storage VI.  The VI works fine for small files.  But as the file gets larger, the conversion time seems to increase exponentially.  Eventually grinding to a halt.

I noticed that each iteration of the for loop seems to take longer and longer.  So I suspect the Write Data VI may not be simply appending to the existing channel data.  But instead, each time it's probably reading back the waveform data already saved, recombine them with the new data and then rewrite the entire file.  This is just a guess.

Anyway, I really need some help resolving the performance issue.  Any response will be appreciated.  The conversion VI is attached.

S.
LabView 8.0

Message Edited by Seaman008 on 06-01-2007 01:50 PM

[mikeporter]

Where exactly does the slow-down seem to be taking place?

Mike...

[Seaman008]

It is in the inner-most for loop where the "Write Data [Channel]" VI is called repeatedly to save the waveforms.  Within each loop iteration, a progress bar is updated.  With large files, the progress bar update slows down noticeably as the loop iteration count increases.

S.

[mikeporter]

Inside the loop, if you turn on execution highlighting, what function(s) are taking up the time? Where does the execution profiler say the time is being used?

Mike...

[Seaman008]

Here are two profile captures.  Each capture start at the first entry into the inner-most for loop and stop when it exited the loop.  Profile2.txt was captured when processing a RAW input file twice the size as in Profile.txt.  I'm not familiar with the VIs listed, but it shows that lot of time was spent in _saveDdtAttributes.vi and _saveDDT.vi. 

[mikeporter]

This, unfortunately is what I was afraid of. The problem VIs are subVIs inside NI's code which is locked so we can't even begin to even figure out what the problem is - let alone how to fix it. Is there any other way to get this data into Diadem? Could you perhaps segment the dataset into a number of smaller, manageable files?

Have you looked into the TDM Streaming VIs? I haven't used them personally, but they might also be of some assistance -- at least the name sounds promising...

Finally, to attract the attention of the NI folks that monitor the forum it might be worthwhile posting a new question asking specifically about performance problems with the data storage VIs.

Mike...

[mikeporter]

Ummmm.... Just noticed the name of this thread... Never mind the last comment...

Mike...

[Seaman008]

Thanks for trying to help, Mike.  I appreciate it.

While waiting for other answers, I want to mention that I only have LV 8.0.  TDMS is only available in LV 8.2 so I can't use it.

S.

[Elizabeth_S]

Hello,

I would like to have a little more information so that I can help you troubleshoot the issue.  Could you please post the raw file that you are converting?  I would like to reproduce the behavior on my end. 

How long does the application run?

What type of computer are you using?

Elizabeth S.

Applications Engineer

National Instruments

[Seaman008]

Hi Elizabeth,

Here's a RAW file.  It took 10s to process it on my PC.  In comparison, it took 145s to process another file 10 times larger.  I'll be collecting files many times bigger so the processing time is becoming a problem for me.  The slow down really shows with much bigger files, but the system won't let me attach them.

My PC has a Pentium 4, 1G RAM, Windows XP.

Thank you!

S.