What do I need to do now to ensure that datalog files will not become obsolete with changes in the future?

[dj143]

I'm developing a LabVIEW program to test and record calibration data for manufactured products. In addition to acquiring data, the user inputs a variety of information as part of the record. (i.e. Customer Name, Test Date, Operator Performing the test, etc.) Because changes are a way of life, I know I'll be in trouble if I later change the datalog type. Changes are likely to be the addition of data, but it may also involve deletions. What can I do to ensure that previously saved files will not become unreadable by new revisions in the program? I've considered saving "spare" data fields, but this seems archaic and there are still uncertainties. I'm using La
bVIEW v.7.1 with Windows 2000.

[Greg_McKaskle2]

> What do I need to do now to ensure that datalog files will not become
> obsolete with changes in the future?
>

This is a problem with all binary formats and most textual ones too.
There is no simple solution, but the key is to place versioning or
descriptive information about the contents at a known location in the
file, then never change that. As an example, just one integer at the
front of your data header, can be used to switch on to carry out the
appropriate reads or unpacking to get back the data that was saved.

Of course five years from now, you will likely have in the same switch,
code to compute or store default values for other fields that the rest
of the application expects, but that weren't saved.

So, like the binary storage fo
rmat of VIs, it is possible to add many
features and be able to load and save all the formats. It is easier to
load the old and only save a few of the newest. Coming up with a
universal storage format is very difficult.

Greg McKaskle

[Dennis_Knutson]

The way that I've avoided obsolesence is to not use binary storage at all. All of my test and measurement results get stored in a SQL Server database (actually, they're first written as XML and imported). Oracle, MySQL will work just as well and even Access in a pinch. I have had the same issue you face and long term support has always been a problem. With the standard database approach, users will often already have the tools necessary to view and analyze the data(i.e. MS Query) and that relieves you of providing a program that does that. Adding or removing columns, modifying the entire schema, is relatively simple and importing old data to a new database format is something I've done a couple of times recently with lttle impact. If the time it takes
to write to a database is a concern, then maybe you could move the datalog files as a post-processing step.

[unclebump]

I write my cluster data as binary strings to tags that get stored in the Citadel database. I had some issues with database file sizes being too large to store months of data. I used Citadel because it handles all of the file storage stuff internally. I would think NI will always make future versions read/import an older Citadel database.

[Dr._Imad]

I have been saving my data to text-based files,of course, keeping track of how the data are formatted, thru flags. And I usually create extra fields for future expansion. It is tedious, but the file can be read by any spread sheet and/or text editing software.
I use datalog files to handle data temporarily, but not for storage. I have seen LabView converting datalog files to new format, and I did not like that. If I am the end user, I will live with it, but customers usually will not like it and wonder what's happening.

This is not the ultimate good solution, but in my case it saves me the headache if my binary files format becomes outdated or for some reason their format (the software) is lost.

[dj143]

It was recommended to use the XML format because it's easy to use and can be imported to other software (i.e. Excel). Your suggestion to set up a leading header to describe the file is a good one, and that I've done. I've set up the first data field as a simple integer that describes the version. Thank you for your response.

[dj143]

I wish I had the time to consider and look into everything. I have chosen to use the XML format because of it�s universality. (Or at least its perceived universality.) The Citadel system is not something I�m familiar with and I haven�t the time to look into it. It�s maddening to think there may be something out there that is better than what I�m using now. Thank you for your response.

[dj143]

This is what�s cool about LabVIEW. I don�t have to be an expert in everything. Up until now I haven�t even heard about XML and really didn�t care. However, looking into it, I found that using the XML format addresses two issues. 1) It is easily extensible. 2) It�s a very convenient format to export data to other software. The extensibility issue is solved by adding additional CASE structures in the file retrieval program. The applicable CASE is activated by the specific file version, which I have specified in the first data field within the file. The bottom line here is that I can read old data files with newly revised *.vi programs. I found the Write Data to XML File.vi and the Read Data from XML File.vi examples very useful to get started using X
ML. (For LabVIEW v.7.1, go to the NI Example Finder and search for XML.) Dennis, I wrote this reply for the sake of other readers. Thank your for your response.