Adding large arrays to a text file as new columns

[rgehmlich]

I am trying to merge several large text data files into a single file in Labview 8.0.  The files are too large to read in all at once (9-15 million lines each), so decided I need to read them in as smaller chunks, combine the arrays, and write them to a new file.

The reason there are three separate data files was for speed and streaming purposes in the project, and the users wanted the raw, unadulterated data written to file before any kind of manipulation took place. 

My VI:

1.  Takes a header generated from another VI and writes it to the output file.

2.  Creates a time column based on sample rate and the total number of data points

3.  Reads in 3 files that each have text data (each data point is 9 bytes wide, there are up to 15 million data points per file.

4.  Each iteration of the for loop writes a chunk of 10 to 100 thousand points (Somewhere in there seems to be the fastest it will do), formatted with the time column on the left, then the three data columns, until it's done.  I haven't quite figured out how to write the last iteration if there are fewer data points than the chunk size.

Anyways, the main thing I was looking for was suggestions on how to do this faster.  It takes about a minute per million points on my laptop to do this operation, and though I recognize it is a lot of data to be moving around, this speed is painfully slow.  Any ideas?

[Efosa O.]

Hello.

After looking at your code, the first thing that jumped at me was the fact that you initialize different constants everytime the loop iterates. You can easily initialize the constants outside the loops and wire them to their respective inputs. This should save some time.

Another thing you can do is use the Binary File VIs as opposed to the Text File VIs. These definitely operate much faster as the computer does not have to convert the binary data to ASCII.

Efosa O.

[rgehmlich]

Thanks for the tip.  I put the constants outside the array and noticed a little improvement in the speed.  I know I could improve the speed by using the binary file VI's but I need the files as tab delimited text files to import them into MATLAB for another group to do analysis.  I have not had any luck converting binary files into text files.  Is there an easy way to do that?  I don't know enough about binary file systems to use them.  I looked at the high speed data logger examples but they seemed complicated and hard to adapt to what I need to do.  Creating the binary header file seemed like a chore. 

I am up for more advice on the VI I posted, or suggestions on different ways to convert a binary file to a MATLAB readable text file.

Thanks!

[Efosa O.]

Thanks for the post.

You are dealing with a lot of data hence the slowdown.

You can read the Binary Files back in LabVIEW and write them to a text file, but that might also take a while (depending on how big the file is.)  At least, this will just write the data from one file to another without doing much other processing.

Efosa O.
NIAE