DATAFILESAVE nitty gritty

[aartjan]

hello Diadem gurus

I have automated quite a bit of interaction between LV and Diadem, which is great. One command puzzles me, though. I want to use DATAFILESAVE to save the changes made to the tdms file currently in the dataportal. I know the call has two parameters; path and file type (eg. Call DataFileSave("C:\Example.tdm","TDM"))

The issue is that I am saving to the same (only one) file I loaded into the portal, which should be the default values for the call. Indeed simply calling "DataFileSave" (in script) seems to do the job. But I also remember getting errors on this. Other times "Call DataFileSave(,"TDMS")" turned out legal aswell. Other times it turnes out illegal. Sometimes Diadem seems to decide I am working in the example directory!! How do I make sure I make this call legal, regardless of the settings in my DDD. Or maybe, What setting do I need to change, such that I get consistent behaviour?

Thanks for any help!

[Twigeater]

Hello Aart-Jan!

If you want to know wich values DIAdem uses for the DataFileSave you can inspect the variables DataFilename and FileExportFilter. These variables will be changed on every DataFileSave or DataFileLoad (DataFilename). If no path is definied the variable DataDrvUser is used, preset from DDD. You can set these variable on runtime or define the full qualified filename as a parameter to get a defined behaviour. I recommend this way.

Matthias

[aartjan]

Thanks Matthias!
One more question concerning this issue: I could really use the script lines needed to load the file path into a variable such that I can have my script save the file changes. I am really no good at vbs yet..
Also, it appears to me that saving a file with this command will not so much save the changes made to the data, but simply overwrite the file. Is that the case and if so, is there an alternative?
Thanks!

Okay, I am going to answer the first question myself. The following script lines will load the last loaded file into a user variable and prompt it:
dim CurrentFile
CurrentFile=DataFilename
MsgBox(CurrentFile)

Message Edited by aartjan on 05-01-2007 10:23 PM

[Twigeater]

Hello Aart-Jan!

At the moment I don't understand what your problem is or what you want to know. Can you please explain it with other words ore more details?

Matthias

[aartjan]

Hi Matthias,
Simple, This way of saving (using Datafilesave) overwrites the complete file, meaning it will delete the old one and write a complete new one from its cache. Some of my files, say 200 MB and larger even encountered an error (probably due to some memory issue) and were lost. (see another recent post from me).
I hoped there would be a way to ADD the changes to the file. It would be ridicoulous to changes something simple such as a property value and have to completely rewrite a huge file.
Thanks for your help.

[Twigeater]

Hello Aart-Jan!

Now I understand! DataFileSave will always save the whole file. If this already exist it will be completly overwritten. This is by design.
There are different aproaches for your kind of use case but nothing what is simple as a DataFileSave call.

Matthias

[Brad_Turpin]

Hello Aart-Jan!

I agree that this is an inefficient and undesireable limitation for TDMS file editing.  DIAdem 10.1 shipped with a new function, DataFileHeaderAccess(), which creates a TDM header object that you can use to read off the hierarchy and property values of all data files for which there is a DataPlugin (without loading the file into DIAdem memory).  For TDM files only, this object can also change any of the property values WITHOUT touching the binary file.  This is particularly easy for TDM files because they store all the descriptive information (structure plus properties) in a separate XML file with the file extension TDM, while all the bulk (numeric and date/time) data are stored in a separate binary file, usually with the file extension TDX.  So it's an easy matter to rewrite the TDM file to change the property values, because it's a small ASCII file completely separate from the huge binary data.

TDMS, though, is built for streaming speed and uses only 1 binary file which is designed to be written to buffer by buffer.  Each buffer has a binary header and binary data section.  The Data Portal structure and the property values are stored in the same file as the binary data values in one or more of these binary buffers.  Since some properties are strings, so that you never know how many bytes they will take up, there's no guaranteed way to replace the string property value in the existing binary file without potentially truncating the property string or overwriting neighboring binary data.  Now that you mention it, though, it should be possible with the DataFileHeaderAccess(), to add a new binary header at the very end of the TDMS file which clobbers the previously declared values of one or more properties, but to my knowledge this is not implemented.  I will check with R&D if this is something they can add to their list.

For the time being, you either convert to TDM so that you can make all the property changes you want, or you stick with TDMS and live with the file re-write.  I will get back to you on your other thread when I have the exact details of the large TDMS file export error.  I believe this is fixed now in a newer version of the TDMS DataPlugin, and I am just checking which older versions of DIAdem it will work with and also how to deliver it to you.

What version of DIAdem do you have?
Brad Turpin
DIAdem Product Support Engineer
National Instruments

[Herbert Engels]

Writing the whole file is something that DIAdem does by design. Overwriting a property in TDMS can simply be done by adding it to the end of the file. That however is not directly supported in DIAdem. From what I understand though, DIAdem can directly call LabVIEW VIs, so there is a rather trivial solution to this. The architecture would for sure not win any beauty contests, but given the limitations of using a TDM based approach, it might still be the better choice.

Overwriting channel data of course is a different ballgame. That would require a complete rewrite of the file.

Hope that helps,
Herbert

[aartjan]

I have Diadem as an option with my LV dev suite, so I have the professional version 10.1 including the recent service pack.
Thanks for all the help!

[Twigeater]

Hello Brad! Hello Herbert!

Hopefully we will get a state-of-the-art, easy to use, object-oriented Data/Metadata access inspired by ODS in the near future. We are working with really large files and/or a lot of channels and metadata at the moment. With the 'standard' data access features in DIAdem the customers requirements could not be fullfilled! ODS gives the necessary functions but is not easy to use in the DIAdem build in way and could not be recommended for 'normal' users. This kind of access is still verry, verry useful!

What do you think will there be a object-oriented data/metadata interface wich will not load the complete data in a future DIAdem version?

Have a nice day in Austin

Matthias