04-25-2007 05:32 PM
Benchmark Tests Using National Instruments (Brad Turpin) Scripts
M90 Laptop 1– 592 seconds (9.86 minutes)
Workstation 2 – 4 GB, hyperthreading – 672 seconds (11.2 minutes)
Workstation 2 – 4 GB, no hyperthreading – 699 seconds (11.65 minutes)
Workstation 2 – 2 GB, no hyperthreading – (original configuration) - 681 seconds (11.35 minutes)
Fuel System Lab Workstation – 678 seconds - trial 1 (11.3 minutes)
Fuel System Lab Workstation – 648 seconds - trial 2 (10.8 minutes)
Benchmark Tests Using DataPlugIn
M90 Laptop 1 – 2312 seconds (38.5 minutes)
M90 Laptop 2 – 2433 seconds (40.5 minutes)
M70 Laptop - 4914 seconds (81.9 minutes)
Fuel System Lab Workstation - 1710 seconds (28.5 minutes)
Workstation 1 - 815 seconds - trial 1 (13.6 minutes)
Workstation 1 - 812 seconds - trial 2 (13.5 minutes)
Workstation 1 - 820 seconds - trial 3 (13.7 minutes)
Workstation 2 - 1741 seconds - trial 1 (with other applications running) (29.0 minutes)
Workstation 2 - 1149 seconds - trial 2 (with no other applications running) (19.1 minutes)
04-26-2007 08:38 AM
Hi Ken,
That's a nice set of benchmarks. It's been a while since we last talked about this topic, though I do remember the VBScripts that you used for those tests. Could you remind me what DIAdem version you are using?
Brad Turpin
DIAdem Product Support Engineer
National Instruments
04-26-2007 08:46 AM
05-03-2007 02:26 PM
Benchmark Tests Using National Instruments (Brad Turpin) Scripts
M90 Laptop 1– 592 seconds (9.86 minutes)
Workstation 2 – 4 GB, hyperthreading – 672 seconds (11.2 minutes)
Workstation 2 – 4 GB, no hyperthreading – 699 seconds (11.65 minutes)
Workstation 2 – 2 GB, no hyperthreading – (original configuration) - 681 seconds (11.35 minutes)
Fuel System Lab Workstation – 678 seconds - trial 1 (11.3 minutes)
Fuel System Lab Workstation – 648 seconds - trial 2 (10.8 minutes)
Benchmark Tests Using DataPlugIn
M90 Laptop 1 – 2312 seconds (38.5 minutes)
M90 Laptop 2 – 2433 seconds (40.5 minutes)
M70 Laptop - 4914 seconds (81.9 minutes)
Fuel System Lab Workstation - 1710 seconds (28.5 minutes)
Workstation 1 - 815 seconds - trial 1 (13.6 minutes)
Workstation 1 - 812 seconds - trial 2 (13.5 minutes)
Workstation 1 - 820 seconds - trial 3 (13.7 minutes)
Workstation 2 - 1741 seconds - trial 1 (with other applications running) (29.0 minutes)
Workstation 2 - 1149 seconds - trial 2 (with no other applications running) (19.1 minutes)
Workstation 3 - 2021 seconds - trial 1 (33.7 minutes)
Workstation 3 - 2052 seconds - trial 1 (34.2 minutes)
Workstation 3 - 2037 seconds - trial 1 (33.9 minutes)
Workstation 3 - 2013 seconds - trial 1 (33.6 minutes)
05-04-2007 01:57 PM
Hi Ken,
In answer to your latest questions, I do not expect this process to run any faster on a 64 bit operating system. DIAdem is a 32 bit application, so the operating system would have the extra work of thunking back and forth between 32 bit and 64 bit environments. I would expect that to slow down the process, if anything.
DIAdem 10.2 Beta 3 (available on the Beta web site) does run on Vista, but here again I would not expect it to improve your execution speed. Each new Microsoft operating system takes something like twice the memory of the previous operating system, so in general on Vista you would be dealing with less RAM available to the DIAdem application. Note, though, that a 32 operating system will only grant up to 2 GB of RAM to any one application (such as DIAdem), so if you have 4 GB of RAM, say, and if Vista and all your other processes take no more than 2 GB, then DIAdem would have all the RAM it's going to get from the OS. Now if you had 8 GB of RAM, then it MIGHT help to be on a 64 bit operating system, because then the OS MIGHT give DIAdem more than 2 GB of RAM to work with, and DIAdem MIGHT be able to use more than 2 GB. I don't really know, and I don't think it's been tested.
I would concentrate on making sure that you give DIAdem the full 2 GB of high speed RAM to work with and also invest in very high speed hard disk drives. It sounds like you already have plenty fast CPUs going there.
Brad Turpin
DIAdem Product Support Engineer
National Instruments
05-10-2007 10:19 AM
05-11-2007 09:15 AM
Hi Ken,
Just to confuse the issue further, I actually sent you 2 different scripts, one that used the DataPlugin to load individual channels one at a time, and the other which used the older DAT header approach to load individual channels one at a time. My guess is that you are using this second script with the DAT header code, which turned out to be faster.
Are you asking why there is a difference between dragging and dropping a file from the NAVIGATOR into the Data Portal and running my script, or are you asking why there is a difference between using the 1st script that loads channels with the DataPlugin and using the 2nd script that loads channels with the DAT header approach?
Let me go ahead and give you some answers independent of your answer back to me on my question above. Both of my scripts load only 1 channel at a time from the ASCII file into the Data Portal, then immediately save that one loaded channel to a new and separate binary file. The benefits you get from this approach are as follows:
1) Only one channel has to be in DIAdem memory at any time. DIAdem channels in the Data Portal are always stored as DBLs in RAM. The hope with this approach is that each channel (as DBLs) will fit into your available RAM without having to resort to virtual memory, which could easily occur if you try to load all channels into DIAdem memory at the same time.
2) Storing each channel in its own separate binary file gives you maximum random access speed for selective loading and interval/reduced loading of the binary file afterwards.
3) Loading one channel at a time gives you an easy way to show a "progress bar" on the overall process, which as you've noted tends to take a while.
By contrast, dragging and dropping an ASCII file from the NAVIGATOR into the Data Portal (thereby using the DataPlugin directly) causes all channel values to be loaded into DIAdem memory (as DBLs) at the same time, most likely necessitating virtual memory. If, however, you right-click on the ASCII file and choose "Open with", then use selective loading to load only 1 channel, you are effectively reproducing the first part of the 1st script. If you then manually output that one channel to a new binary file with the DAT export functionality, you are effectively completing the steps of the 1st script.
If, on the other hand, you are comparing the performance of the 1st script against the 2nd script, the only difference there is the way in which each channel is loaded individually and sequentially. The older DAT header approach is less flexible and involves fewer layers of software, and often can load ASCII data about twice as fast as a DataPlugin can. This is just a difference in the underlying C code which parses the ASCII file and passes the data to the Data Portal.
Hope that helps,
Brad Turpin
DIAdem Product Support Engineer
National Instruments
05-14-2007 09:32 AM
05-15-2007 09:49 AM
Hi Ken,
Both the 1st script I sent you (which uses a DataPlugin, either mine or yours, to load the data), and also the 2nd script I sent you (which uses the DAT header approach to load the data), can ouput the data to binary files in a variety of data types. I made I16 the default data type because it has the smallest hard drive footprint while preserving most of the data precision. You can change one of the parameters at the top of either script so that the different ouputted binary files matched with the DAT header file are REAL64 or REAL32. So this would be an easy change.
If, on the other hand, what you mean is that you want the converted binary file to have ONE binary R64 file to go along with the DAT header, then that would require additional effort and processing time. As I explained in my last post, the script you are using to load the data channels never has more than one channel's data in DIAdem at any one time, so saving all those channels to a single binary file would require an extra process or a fundamental change in the script-- in either case the processing would take more time.
If you do want only one R64 file, then I would suggest that you add a few lines of code at the end of the script which loads the newly converted DAT header (matched with the N binary files), then saves that data out to a DAT or TDM or TDMS file. My recollection is that loading the converted DAT file took less than half a minute, so this would probably be the fastest way to arrive at only 1 consolidated binary file. You could even use the new DataFileHeader object in DIAdem 10.1 to read out the header properties with your DataPlugin from the original ASCII file (without loading any of the data) after you've loaded the converted DAT file into the Data Portal, so that when you save the data out to TDM or TDMS file you get all right File, Group, and Channel properties, as laid out in your DataPlugin. Note that if you use the TDMS file format, you only have one binary file to keep track of. Also note that both the TDM and TDMS file formats can be loaded directly into Excel with the TDM Excel Add-in available for free download on our web site.
Brad Turpin
DIAdem Product Support Engineer
National Instruments