Maximum data size in DIAdem

[Yashasvi]

Hi,

I am trying to load a 2 GB text file in DIAdem. This causes DIAdem to hang and I am forced to restart the software. Can it support this file size? If yes, then what is an approximate loading time?

My system config is:

Windows XP OS

1 GB RAM

3 GHz, P4 processor

Thanks,

Yash

[Brad_Turpin]

Hi Yash,

Yes, DIAdem can load up to 2 billion values (2^32) per channel, and it can load up to 65535 channels (2^16).  GigaByte ASCII files will always load slowly because that's simply a lot of ASCII parsing. How were you loading the ASCII file-- with the ASCII Import Wizard or the ASCII DataPlugin Wizard or a posted DataPlugin from www.ni.com/dataplugins?

Unless you were using the ASCII Import Wizard, my guess is that DIAdem was not hung, it was just very busy.  When DIAdem "white screens" it usually means it's deep into loading a huge data set and doesn't get around to redrawing its screen until the next time it comes up for air-- say after it finished loading the current channel and before it starts loading the next channel.

Can you post a small data file of the same format to this Discussion Forum?

With ASCII data files in the GigaByte range, task #1 is to convert them to a binary file such as TDM or TDMS.  We can usually help with this. 

Brad Turpin
DIAdem Product Suppor Engineer
National Instruments

[Otmar]

Hello Yashasvi,

Please note that loading large files will be significantly faster the more memory (2GB or better 3-4 GB) and the faster harddrive you have.  A fast (and defragmented) hard drive helps tremendously, especially on Laptop computers (make sure to have a 7,200 RPM drive vs. a 5,400 PRM drive if you have a choice).

Your file is about 2x larger than your available memory, and once you add Windows and DIAdem to the memory needs, you have very little available memory for data. Memory upgrades are relatively cheap and will have a big  influence on your import performance.

I can only reinforce what Brad mentioned, DIAdem probably doesn't crash, it's just very busy importing your large data file - we are talking hours here due to having to swap memory on the harddrive. I recently tested something similar on my Laptop (2GB RAM dual core 1.83 Mhz) and on my home PC (Quad core with 4 MB and fast HDD). My home desktop PC was about 7x faster with the same file.

Have a great weekend,

[Yashasvi]

Hello Brad,

I am using the ASCII DataPlugin to import data. I am unable to post the data sample here but I am open to suggestions on converting data to TDM format. Is it just a question of creating the right plugin?

Regards,

Yash 

[Brad_Turpin]

Hi Yash,

There are several methods you can use to convert your huge ASCII files to binary, such as TDM or TDMS, only some of which require a DataPlugin.  A popular alternative to using a DataPlugin to convert the file is to create a LabVIEW program to read several thousand ASCII rows at a time and append those to the TDM(S) file, buffer by buffer, until the whole ASCII file has been read and converted to binary.  Once the data files have been converted to TDM(S), then you will not need a special DataPlugin to read those converted files-- LabVIEW and DIAdem both natively read these files very efficiently.

Can you send us a dummy file with no real data, just preserving the ASCII file structure?  Or is the ASCII layout itself a secret?  It is usually sufficient to send over just the first 100 or so lines from the ASCII file.

Brad Turpin
DIAdem Product Support Engineer
National Instruments

[Yashasvi]

Hi,

I am enclosing a sample data set having a 1000 rows of data. The number of columns may go upto 150 and the number of rows to around a billion. I am open to ideas for converting the ASCII data sets to TDM but I wouldn't want the conversion itself to run into hours as that would again defeat the purpose. If we can come up with a method to reduce the loading/conversion time for a 2 GB file to less than say 15 minutes, then that would be really useful. I am also ready to boost up the hardware on my PC if that helps.

Regards,

Yash

[Brad_Turpin]

Hi Yash,

Here is a LabVIEW 8.6 program that converts your ASCII file to a binary TDMS file, which will load efficiently into both DIAdem and LabVIEW.  I'm concerned, though, that your time column has repated time values (why don't you have the milliseconds?) and has no date values.  Will your test runs ever span midnight?  If so, the time channel will jump backwards from 23:59 to 00:00, and graphing the data vs. this time channel will look funny.

If you don't have LabVIEW, I could build this into an executable, but we should hammer out the rough edges mentioned above first.

Brad Turpin

DIAdem Product Support Engineer
National Instruments

[Yashasvi · Accepted solution]

Hello Brad,

Thanks a lot for the code. It is working fine. Please don't worry about the time column issue as I had populated the test data set from LV and had not included millisecond values for it. On the actual data set the code works great.

Regards,

Yash

[Greg@Delphi]

Hello, I received an error message while trying to load a 2 GB file into Diadem.  The error read something like "Diadem was unable to load all of the data"  that was it...  Any ideas as to why or where to look for clues... ? 

[Brad_Turpin]

Hi Greg,

Have you tried register-loading the data file instead?  Right-click on the file in the NAVIGATOR and choose the "Register Data" context menu.

Do you know how many channels are in this 2 GB data file?  Are they all the same length?  Do you know the binary data type the channel values are stored in?  What is the file extension of this data file, and what DataPlugin are you using to load it?

Brad Turpin

DIAdem Product Support Engineer

National Instruments