big waveform to text

[engwei]

Hi,

I have to append waveform arrays to a single waveform then convert them to text file. 

There are 8M lines (rows) X 60 column of character (maximum).

Then, I m facing "not enough memory" problem, and I think I have to save the waveform to binary files. But for viewing purpose, I still have to convert them to text file. 

Any idea to view big waveform ? 

Here I attached the VI and waveform which saved in binary format. Please advise. Thanks. 

[MikeS81]

Hi engwei,

i think you can't view the entire data at ones, so you should store your data diretly into a file. Don't load all into memory if you don't need. You can build a big file little by little. If you need some data to view it, then you can load it from your file, but only the part you need.

Hope it helps.

Mike

[Ben]

8M X 60 collumns X 6 characters per number = 2.9 Gig of memory.

1) Fist your OS must be able to handle 3 Gig of memory (? 3 Gig aware ?)

2) What app is going to open a 3 Gig file?

3) Divide and conquer. Right each collumn to a seperate file, or break-up the output file into parts 1-10 etc.

Ben

[engwei]

Hi,

LoL. Thanks for your reply.

It's actually to append all the waveform and convert to text file. The text file will then be opened and added some label "*" according to the column and row number.

By append all the waveform, I no need to reindex the column and row numbers.

Seems like I have to append a part of the big waveform, read the text, add the label "*", and save in files. It would caused multiple txt file, and it seems the only way to solve it.

Thanks. 

[DFGray]

Is there a reason to use text vs binary?  I usually switch to binary at about 10,000 points.  That said, the advice you were given above is good.  You can't do it all in one step.  You have to break it up. I have one more tip for you.  If you save your data so each text field is always exactly the same width (it can be padded with spaces), it is trivial to seek to a specific line by calculating the offset from the beginning of the file.  With variable length fields, you need some sort of indexing.  At that point, you should probably use binary (try TDMS or NI-HWS).

You may also want to read the tutorial on handling large data sets in LabVIEW.

[engwei]

Hi DFGray,

All my data is always same width. will take a try on the file offset. 

Thanks for your advice.