08-11-2008 12:08 PM - edited 08-11-2008 12:09 PM
08-11-2008 01:52 PM
08-12-2008 03:33 PM
Hi SS,
My recommendation is to immediately convert that 3.5 GB ASCII file into a binary file. You could do this in LabVIEW by reading 1000 lines at a time in a loop and appending the parsed block of values to the end of a TDMS file with the TDMS Write.vi. You probably won't be able to load all the values from that file into RAM, so that's why I suggest you load the data buffer by buffer fromt he ASCII file and convert each data block from ASCII into binary. A binary file should be smaller and will load MUCH faster into all possible environments (LV, DIAdem, Excel), but still you probably won't be able to load the whole data file into RAM. So you'll need to either process/graph the data in batches, or you'll need to condense each set of N values into one representative value (average) or perhaps two representative values (min and max). This is what DIAdem calls loading with data reduction, and it effectively results in a downsampling of your data set to reduce its size. If you acquired your data 5 times faster than you strictly speaking needed to, say, then condensing N values into 1 would result in no loss of information but would enable LV or DIAdem to load the roughly 700 MB of resulting downsampled data into RAM. You could even do this in your ASCII to binary conversion loop to create a much smaller binary file right away.
DIAdem offers serveral methods for dealing with large data sets, but no application is going to deal well with 3.5 GB of ASCII data. Let us know if you're interested in exploring any fo the DIAdem options.
Brad Turpin
DIAdem Product Support Engineer
National Instruemnts
08-12-2008 03:56 PM
Brad,
Thanks for the reply, sorry I accidentally posted to the DIAdem forum instead of LabVIEW. Turns out that LabVIEW can create a 3.5GB file just fine. Also I was able to break it down into 1MB chunks using code from this reposted thread. So the short answer is it's probably not good practice but LabVIEW can do it depending on your system resources and which file handlers you use.
Regards,
-SS