Convert many large TDMS files to CSV files

[Crucis]

Hi, I'd like some help with my problem.

I have 200ish TDMS files of 200MB each, and would like to convert them to CSV or another file system which I would be able to import it on python/R/Lua.

I managed to find a solution using Write to Spreadsheet File on a for loop, but the csv files don't have header, are transposed (2xN instead of Nx2) and give me no time column.

So I am using now Export Waveforms to Spreadsheet File, which gives me the output I am expecting (with header, Nx2 and time information), the problem is that it is giving me the error "Memory is full". I have 20GB of memory RAM and 60GB of virtual memory on Windows 7 64bits, so I don't actually understand how it happens.

Does someone know how to copy large TDMS files to CSV files (or anything that Python/R/Lua can read)? (I need files for those languages because this data is going to be used on machine learning algorithms by people that does not have NI licenses).

[JÞB · Accepted solution]

A few thoughts hit my frontal lobe.

Like, "Why would I choose to expand the data to text just to read it into a language that has types?" and "I bet someone has done that much more efficiently!" which I quickly Googled!

[Crucis]

>Why would I choose to expand the data to text just to read it into a language that has types?

Because it is important to save my data on a database to be used by people using diferent languages and enviroments, and it would be usually saved on a txt file, or hdf5.

>"I bet someone has done that much more efficiently!
It certainly helps me! Thanks for that!

[Hooovahh]

It sounds like you have a solution.  But in case someone else is looking to convert a batch of TDMS files, I'd recommend you look into my TDMS to XLSX conversion tool.  It has an example of converting a batch of files.  Before converting a bunch I'd perform a defrag on the files first just to make sure they aren't fragmented, as that will make the TDMS open function take much longer.  This should include TDMS properties on separate worksheets, and extra features like freezing the first column and row.

If you are interested in indexing and searching/querying data you might also want to look into the DataFinder toolkit, or DIAdem.  Both of which can sit and index new data as it is found, in TDMS or other file formats.  DataFinder then provides an API for finding data that matches some criteria, and DIAdem has a bunch of automated report generation features.  This all might sound over kill for you and it might be but I've seen them used successfully for large amounts of data.

And lastly I think what Jeff was trying to say is instead of going about converting your data into something other programs can process, why not write code (or find code) that can read the data as it stands without needing conversions?  TDMS is a well documented and semi-open standard and parsers can be written in any language, if one hasn't been written already.