Looking for DIAdem advice before it is too late!

[Ben]

Hi All,

I am currently in the design phase of an application that could greatly benefit from all of the cool stuff it appears DIAdem can do. Unfortunately I do not have personal experience using DIAdem so I am looking for advice from those that have already walked the "DIAdem path" and are willing to share any warinings and hints.

As is usally  the case with applications of this type I am restricted in what I can share while asking this question, but the following should give you an idea what I am putting together.

I think the application can be understood as being a 20,000 channel 1KHz storage oscilloscope with high-speed video play-back.

Each time the O'scope will be used, it may be "cabled up" to different devices. Many tests will be conducted using the same or similar configurations. The end-user will need to be able to compare multiple "runs" to analyze how the changes to this system impact the over-all performance.

So now my questions;

1) Do you have any recomendations regarding the support information that is saved along with the raw data to be able to excersize the full capabilities of DIAdem?

2) With data sets that can easily exceed 1Gig of raw data per run, is seems the memory limitiations of computers can be an issue, is it?

3) What would you have liked to have known before you set up yur first DIAdem driven application?

4) How far can I go with using DIAdem before I have to start coding in VB?

5) Please share any other warnig that could save headaches.

Thank you,

Ben

[Brad_Turpin]

Hi Ben,

This sounds like a very interesting application.  I'm going to focus on your question 4) in this answer to make it under the 5000 character response limit.  I will send a second answer with my thoughts on your question 2).  We will also be contacting you directly on your question 1), since that's going to need to be a two-way conversation to make much progress, though the short answer there is that you save all the information you might later on want to build queries with.

4) How far can I go with using DIAdem before I have to start coding in VB?
This depends on how you use DIAdem.  If all you ever want to do is operate interactively in the DIAdem environment to load, analyze, inspect, and report your data, then you never need to create any scripts at all.  There are a few exceptions such as making calls out to another application's ActiveX server or running specific SQL queries on an external data base.  What usually happens, though, is that you use the DIAdem evironment interactively to work out a series of steps that results in the particular inspection/analysis/reporting that you're after, but then you find yourself robotically repeating this sequence of interactive steps over and over again for multiple data sets.  This is the point where it makes sense to contemplate scripting.  The good news is that DIAdem has a VBScript Recorder, like the Macro Recorder in Excel, which provides an excellent bridge between interactively using the DIAdem environment and programmatically automating steps in DIAdem.

Welcome aboard!
Brad Turpin
DIAdem Product Support Engineer
National Instruments

[Brad_Turpin]

Hi Ben,

This is the second part of my answer, in which I will focus on your question 2).

2) With data sets that can easily exceed 1Gig of raw data per run, it seems the memory limitiations of computers can be an issue, is it?
It certainly can be.  If you run 20,000 channels at 1 KHz for 1 second, you'll have 20 million data points.  If you save them as I16s you'll have 40 MBytes of data for that one run.  If you run it for 100 seconds you'll have 4 GBytes of data!  You didn't mention your typical run durations, but any time your data sets approach the GByte range, it can get dicy to process that data efficiently.  DIAdem really does help in this area.  Current 32 bit versions of Windows operating systems allow up to a maximum of 2 GBytes of memory for each application, by definition, and in most cases the amount of memory the OS chooses to allocate per application is much smaller than that.  This hard 2 GByte limit includes the sum of a given application's RAM and virtual memory (page file usage) and also includes the memory the application itself needs to run.  With the advent of NTFS file allocation tables in Win2000 and WinXP, which enabled OS management of files that can be >> 4 GBytes, this introduced a serious discrepancy that most applications can't manage.

But actually DIAdem has been handling this issue for years without bothering with the OS.  Of course, if your data sets are reasonably sized, by which I mean they fit into the amount of RAM your OS chooses to give to your application, then DIAdem happily will use OS-managed RAM or OS-managed virtual memory to store the data sets you have loaded into the Data Portal or created in the Data Portal with DIAdem analysis routines.  But for unreasonable data set sizes, by which I mean data sets your OS is incapable of allocating enough memory to a single application to load, DIAdem employs one of two separate methods to manage the data internally, without troubling the OS about it.  The first of these methods DIAdem calls file registration.  You'll see this "Register data" option, in addition to the normal "Load data" option if you right-click on a data file in the NAVIGATOR tree view.  When DIAdem register-loads a data file, it loads all the information ABOUT that data file, without loading the data values themselves into memory.  So you see a ghost of the data file in the Data Portal-- all the Groups and Channels show up correctly named, and with all their properties loaded, but what you see in the Data Portal is a wispy light grey, indicating that the values behind each of those channels are not currently resident in DIAdem memory.  Having registered channels in DIAdem means that DIAdem will go get those values on-demand from the data file whenever it needs them for graphing or analysis routines.  Essentially, file registration is DIAdem using the existing data file as in-place virtual memory, without the operatig system's involvement.  As such, the virtual memory footprint is always instantly there and you always have enough hard disk space for it, because it already exists.  Note that DIAdem 10 graphs registered data channels MUCH faster than earlier versions.

DIAdem can handle 65000 (2^16) channnels, and each channel can have up to 2 billion (2^31) values.

The other way that DIAdem handles enormous data sets is by managing its own virtual memory, independent of the operating system.  If you use a 6 GByte registered channel to create a new 6 GByte channel with an analysis routine, DIAdem will automatically allocate and manage the required 6 GBytes of hard disk space itself, using an internal HDF5 layer.  DIAdem can also use this same HDF5 virtual memory if you try to load data sets for which the OS can not allocate enough memory to DIAdem.  In fact, DIAdem 9.1 and later will automatically use whatever memory the OS will give it in such a loading scenario, then switch over to its own HDF5 virtual memory to load the rest of the data set.  This means that some channels (in RAM) will graph faster than other channels (in HDF5 virtual memory), but you automatically and cleanly navigate through the 2 GByte OS-imposed memory boundary without ever having to think about it.

Welcome aboard!
Brad Turpin
DIAdem Product Support Engineer
National Instruments

[Ben]

Hi Brad,

Thanks for the detailed reply. It still sounds like DIAdem is going to be very useful to me.

Please contact me off-list where I can talk under NDA.

Ben

Message Edited by Ben on 02-23-2006 06:02 PM