TDMS to ASCII 'not enough memory'

[usuario]

Hi,

I have several EA data .tdms files (few seconds sampled at 800kHz). I want to generate ASCII (.txt) files. I actually do it, with these and with some other multichannel (but low fs, 3kHz). The problem is that if the EA files are too big (say, 15 seconds), when running the .vi an error appears saying 'Not enough memory to perform the operation'.

So, any ideas?

I guess there is some way to write the file by blocks, open---- write some data --- write some more data --- again and again --- close file.

But my LV skills are not too good yet, managing to .tdms save data and to convert it has been a big battle.

So, any ideas for dummies?

Thanks in advance.

[YongqingYe]

Which LV node returns this error and how big of your TDMS files, how many channels in it?

[usuario · Accepted solution]

Hello,

I've seeked for the error-giving block, and found that none does. I think it is a computer limitation, not LV.

In the .vi, I perform a for loop for file format transformation. The 1rst .tdms (68Mb in size, one channel) is done correctly. The 2nd one (380Mb, also one channel, just longer acquisiton time) is the problematic one. When reading it, it appears the 'Not enough memory to perform the operation' message window, but not LV style, nor error code, nor window title, just the message and a Ok button. And running the program step by step (light bulb on), no error message appears on the error wires.

Even more, it creates a 2nd .txt file, with the proper name (002_xxx), BUT, just same size as the first one. I would say that the computer is not able to read that much data (32bit, 1Gb RAM) and keeps working with previous data (first file's).

I'm just gonna try in another computer (64bit, 3,93Gb RAM) and will post the results.

Thanks for your interest and questions, you make me check for new issues.

[usuario]

Hello,

yes it is a computer problem and not a LV or .vi issue.

I can run the .vi without any problems in a 64bit 3.93Gb RAM without any problmes (and much faster too).

Thank you all for your time.

[DFGray]

Running out of memory while trying to do this type of operation usually means the coder did it the easy way instead of the efficient way.  The easy way is to read all your data out of a channel, convert to ASCII, then write to disk.  This can run any computer out of memory if the channel has sufficient data.  It also can be very inefficient.  The efficient way is to read a block of data (part of your data) from disk, convert it to ASCII and write, then repeat until all data in the channel is finished.  You can accommodate almost any memory size with this method.  Your block size will be dictated by available memory and efficiency.  Disk operations on Windows machines are usually most efficient with block sizes about 65,000 bytes.

You can get a more complete explanation in the tutorial Managing Large Data Sets in LabVIEW.

[labmaster]

Dear usuario:

I found you had a same problem in reading large TDMS file.

Can you share your experence with me about the large file in 64bit?

Can you estimate the maximum size of TDMS which you can read? 

What is your system (CPU, Memory, PXI?, Window version) before and after this problem?

I have never experienced with 64bit Windows. if you found some limitation, please let me know.

labmaster.

[usuario]

Dear LabMaster,

When I changed to a more capable computer the problems ended. I would say it is not a matter of 32 or 64bit., but a matter of more characteristics, mainly RAM and GHz.

Another issue is how to read the file for conversion. I read the whole file, and wrote the whole ASCII file at once. Probably, a stepped read and write (open file -  read one block  - create ascii file - write block on ascii file  - read 2nd block  - write 2nd block -- read 3rd block -- write 3rd block ---....-- end of file - close files) is both more efficient and requires less from the computer. But my experience in LabView is short, and since I only need to do it this time and maybe a second one, not more, it is faster for me to change the computer. My files are around 40Mb in TDMS format (7 seconds at 800kHz, single channel).

Hope this helps you, please write back for any further issues.

usuario

By the way, my 'better' computer is:

Microsoft Windows XP

Professional x64 Edition

Version 2003 Service Pack 2

Dell Precision T3400

Intel(R) Core(TM)2 Duo CPU

E8200 @ 2.66GHz, 3.93GB RAM