Questions on Saving and mining data with Labview and DIAdem

[pen]

Hi,

I am sampling two signals at 200k sampling rate. I am trying to save the data on harddisk and analyzing the data using DIAdem.

If I save the data using mesurement file format .tdm and .lvm, the file size will be about 4 Gigabytes for only 10 mins' acquisition. It is very slow to process it.

I used the software Clampex and pCLAMP(Axon Instruments) before. At the same 200k sampling rate and also acquiring two signals, these programs save the data as .atf format and the size is only 400 MB for 10 mins' acquisition.

I wonder if there is also a good way to handle this situation using Labview and DIAdem, and how to do it?

[DFGray]

Do not use LVM.  It is a text file format, so not suitable for data set over about 5000 points in length.  While it is readable by just about anything (it is a tab separated text format), it is also much larger and takes much longer to parse on input (and output) that an equivalent binary file.  If you want to use DIAdem, your best choice is TDM, but make sure you are saving your data as binary, not text, or you will have worse parsing and size problems than you will with LVM.  If you can do your analysis in LabVIEW, your best bet is NI-HWS, a binary format whose performance is better than either LVM or TDM.  It is available on the driver CD under the computer based instruments tab (it is the native file format of the soft front panels and waveform editors).  HWS is not currently supported by DIAdem.

As a side note, you may want to check out Managing Large Data Sets in LabVIEW.

[Brad_Turpin]

Hi pen,

With 2 channels at 200 KHz for 10 minutes I calculate: 2 * 200,000 * 60 * 10 = 240,000,000 samples.  If each sample were stored as an unscaled I16, that would result in 480 MBytes on disk, which is roughly what you are used to.  Certainly with large amounts of data like this saving unscaled integers makes sense.  If you are using DAQmx, you have the option in your buffer read to request the raw integer values and stream those straight to disk, and you can get the scaling information from your DAQmx task and write those into the TDM header.  An alternative approach is to request SGL data from DAQmx and stream the scaled data to disk-- this would result in a 960 MByte file for the same 10 minute run.  You might also want to consider streaming each channel to a separate binary file.

But I hear from the AEs that you actually want to stream at these rates for several hours and you want to segment your binary files into separate time slice files, so that no one file gets too large.  All of this is possible, but the fancier your requirements the more complicated the code will be which accomplishes that.

Please clarify exactly what you need to do, and we can help you with the LabVIEW code which will create dense binary files that load quickly in DIAdem.  It would also be very useful to know what you want to do with the data in DIAdem afterwards-- always graph the whole run?  Only graph a small time slice of the run?  Apply data reduction to graph every 50th point for the whole run?

Regards,
Brad Turpin
DIAdem Product Support Engineer
National Instruments

[pen]

Hi, Brad:

Thanks a lot!

What I want to do is to  stream at 200k for several hours and  save the data in a group of files. Each file is for about 10 mins to prevent large files. As you suggest, the data stored in each file should be unscaled I16, so each file will be about 480 MB.

After that, I will load each files to DIAdem, graph all of them, and try to locate the signal (each signal could be in the length of miliseconds) that I needed. Then I want to export only these signals into ASCII format so that I can draw pictures to present to other people.

Sometimes there will be serveral thousands interesting signals in 3 hours' data. So I want to use Labview to read all the files and pick up these signals out and save only these signals into one file and then perform ananlysing.  

I am a graduate in Physics, really not an expert in software. Currently, I use "save to measurement file" function in the Labview, select TDM binary format, and get 800 MB file size in only 1 min. And It will take about the same time to open it in DIAdem. If I swith windows and come back to the graphs, it will take that time again to refresh it. It is not efficient.

Thanks again!

[pen]

Hi, Brad:

I am using DAQmx. Could you explain step by step how to request the raw integer values and stream those straight to disk?

[pen]

I figured out how to save the unscaled I32 into binary file (1.8 GB/ 10 mins). The attachment is the code. I also have the function to segment the binary files into separate time slice files.

I still have serveral questions to ask:

(1) How to get the scaling information from the DAQmx task ?

(2) How to open this binary file with DIAdem? I tried it, but it did not work.

(3) When I use unscaled I16, there is an error message: " Read can not be performed beacause this version of QADmx REad uses a datatype that is too small for the channels in this task". I am using Labview 8, NIDAQ800.

[Brad_Turpin]

Hi pen,

Here's the code I have.  In it you'll see how to read out the scaling information from the DAQmx task and also how to create a TDM header file for that binary file so that DIAdem can read it in.  This application was designed to stream all the binary data to 1 interleaved file, so it is not exactly what you're looking for, but if you're willing to wade in and make edits you could morph it into what you want.  I'd like to make the edits myself, as what you're doing will undoubtedly be something others are interested in also.  However, I definitely can't get to it in the next few days and also can't give you a definite timeline as my in-box is very full now, so I thought I'd just shoot you the code I would have started with in case you want to use it in your efforts.

Note that DIAdem will load the resulting TDM file much faster if you can keep track of the maximum and minimum values for each channel and write the "minimum", "maximum", and "monotony" channel properties in the TDM header file (set "monotony" = "not monotone").  Even more important if you are using DIAdem 10 is to try using the "Register data..." loading option instead of loading all the data values into RAM.  That way for huge data sets DIAdem will simply use the existing data file in-place as DIAdem-managed virtual memory.

Check out the TDM Header Writer VI application note at:

http://zone.ni.com/devzone/conceptd.nsf/webmain/117CB3D512ED96928625707B004DC4E9

And download the example code I would have started with at:

http://sine.ni.com/apps/we/niepd_web_display.display_epd4?p_guid=00AD0A77D3D5318CE0440003BA7CCD71&p_...

(look at the "\Examples\DAQmx Streaming.llb" file)

Regards,
Brad Turpin
DIAdem Product Support Engineer
National Instruments

[pen]

Hi, Brad:

Thans a lot!

I worked out the code to stream I16 to TDM files based on the code you gave. The binary file size is about 470 MB for 10 mins, 2 channel, 200k rate. And this binary file can be opened with DIAdem.

For 1 mins data (47 MB), it is pretty quick to open the file. But for 10 mins data ( 470 MB ), the loading is very slow. I tried the "register" function, but it do not work. ( I am using DIAdem 9.1)

I also saw you reply post for Nick on loading big files to DIAdem. I downloaded you script "TimeSlice Load", but I found that it only works for DAT file. How to adapte it to apply to TDM file?

I also have one question on the TimeSlice Load script. Bascially you load only on point for a small section of the file based on the minimum and maxmum. My question is that if I zoom in to that section later on, can DIAdem actually load all the details of that section?

[Brad_Turpin]

Hi pen,

Yes, the TimeSlice Load application will let you do just that-- quickly load a high-level (data-reduced) graph of the whole data set, then interatively zoom into a particular region in ever increasing granularity until at some point way zoomed in you are looking at every value acquired over a tiny window of time.

File Registration works just fine in DIAdem 9.1, but plotting file-registered data channels is WAY faster in DIAdem 10.

The library posted on the web just includes the TDM header writing VIs, but there are also DAT header writing VIs.  There's no reason you can't have both.  You will need to download the "LabVIEW-DIAdem Connectivity VIs" in order to run the DAT header file creating VIs, so I didn't bother you with that part initially.  Now that you're interested in the TimeSlice Load application, it's time to look at creating DAT header files (too):

http://digital.ni.com/softlib.nsf/websearch/5121C7F322C9392886256EDF005BB4FF?opendocument&node=13206...

Let me know if you have further questions,

Brad Turpin

DIAdem Product support Engineer

National Instruments

[mlut]

Hi Brad,

So there's no plan of offering the TimeSlice feature also for TDM header files? We're creating these headers outside of DIAdem/LabVIEW and would much prefer to use the new TDM file format rather than the older DAT.

Regards,

Matthias