What is the format of Datalog files?

[wpd]

I would like to write an application to post process data collected in a Datalog file.  Can somebody direct me to the documentation that specifies the format of these files.  I can't seem to find it anywhere.  I have looked at http://digital.ni.com/public.nsf/3efedde4322fef19862567740067f3cc/747b83b6c8e4a6218625669100563b61?OpenDocument, but that does not match the format of the data I've got.  (The Datalog files I have all start with "DTLG" as the first 4 bytes and doesn't seem to have I32 bytes giving lengths of character strings and such.)

Thanks for your help.

--wpd

[Dennis_Knutson]

"Datalog files" is a generic term. If you are talking about files created and written with the functions on the File I/O?Advanced File Functions>Datalog palette, then the format of the data is whatever is specified with the Open/Create/Replace Datalog function. In other words, anything you want. Do you have access to the code that created the files?

[altenbach]

If the application is also LabVIEW, things are easy. You simply need to know the record type and use tools from the datalog palette.

If the application is not LabVIEW, you probably should not be using datalog.

Quote from the LabVIEW help:

"Use datalog files to access and manipulate data only in LabVIEW and to store complex data structures quickly and easily"

(I have not tried to reverse-engineer the format ;).)

Message Edited by altenbach on 07-03-2008 11:51 AM

[wpd]

Hi,
Thanks for the replies.
The data was created within the File I/O>Advanced File Functions>Datalog palette, and I do have access to the code (in the form of walking over to the developer who wrote the code, tapping him on the sholder, and asking him if I can have access to the machine on which it resides).  But I don't have access to a LabView license on every machine on which I might want to view this data.  Nor do I have the patience to run through a GUI to extract and analyze the data.  (That's what automated scripts are good at).  Hence my desire to be able to post-process the data with custom code that I write myself.

I don't want to reverse engineer the data format, I want to read and interpret the documented specifications for the format that is used.  Thus far, I have not been able to find any such specifications, though I have read the same line you quoted about "Use datalog files to access and manipulate data only in LabVIEW and to store complex data structures quickly and easily".

It will be quite disappointing to learn that such specifications are not available.  I'll keep looking, and keep hoping that somebody will say "It's clearly documented in the LabView Data Formats manual", which I have yet to find.


--wpd

[Dennis_Knutson]

You don't understand. The specifications for the file is whatever the original programmer decided it should be. Since you have access to the source code, look at it if he did not document the file format somewhere. Creating a program to read the files is simply a matter of specifying the same data format that is in the write VI. If you are not familiar with LabVIEW, then post the VI that created the files.

[altenbach]

Maybe the original developer can make a DLL for you that can read that exact flavor of datalog files and output the desired data in some other format.

[wpd]

You are entirely correct.  I don't understand.
The datalog file was created with the Open/Create/Replace Datalog vi.
I know exactly what the Record Type was that was used to open the file.
I know how he collected the data that was written to the file.

I have the file on a separate PC.  I know that the first 4 bytes of the file are the letters 'D', 'T', 'L', 'G', followed by binary data interspersed with some ASCII data like "Date/Time", "Waveform Name", "Average Count", etc... in other words, the names of the data paths that he bundled together.

That header is followed by a bunch of binary data, presumably the timestamp at which he created the data, the name of the waveform, the average count, etc...

What I don't know is the documented format of that file itself.  From past experience, I would expect that there are records with some field (8, 16, 32 bits?) indicating a record type, record size, etc... followed by the data in big or possibly little endian format.

From the research I've done so far, I'm not the first person to have asked this question, and I have yet to find the answer.  I truly don't understand what NI gains by keeping it secret.  So I'm hoping that the format of the file isn't secret, but just tremendously obscure.  (Because I can figure out "tremendously obscure").

Thanks again for the reply.  It helps to talk/email about this.

--wpd

[wpd]

Now that's a clever idea.  I hadn't realized that was a possibility.  Could he (in the sense of "does the technology exist" not in the sense of "does he have the time and capability to") create a DLL which, when given a Datalog file, would produce a binary output file formatted according to my own specifications?  Could I then link that DLL into whatever (and however many) applications I wish?

I'll explore that option with him.  In the mean time (for the next 15 minutes or so), I'll continue to see if I can find any documentation on the format of the file itself.

Thanks for the idea.

--wpd

[Dennis_Knutson]

There is no secret that NI is keeping because they did not create the datalog files that you are trying to read. The format of the file is the record type that you mention in the Write File. Look at the shipping examples called Write Datalog File Example and Read Datalog File Example. In the Write example, the programmer decided to use a record type of a cluster with a string and a 1D SGL array. When the VI runs, that is the binary format of the file. In the read example, the record type from the writer was copied and specified. Then when the Read Datalog function is called, it returns a record that consists of cluster with a string and a 1D SGL array.

The programmer of the example could have just as easily chosen a different record type. He/she could have decided to save DBL data instead of SGL and that would change the file format and would require a change in the reader. And your original programmer is almost certainly using a different format. That is why I keep saying that the format of a datalog file cannot be documented by anyone other than the programmer. You have an infinite number of choices when using binary files because you not only have the various data types (I32, DBL, SGL, etc.), but you can define the data consists of clusters/arrays, etc. The original programmer could have also chosen to use the Write Binary function. It works like the Write Datalog but you also have the option to use big or little endian byte order.

I hope this helps. Take the record type from the write program and copy it into the shipping example. See if you get data that is meaningful. Otherwise, post the write program and a small sample file.

[wpd]

Hello Dennis,
I've been out of the office for a couple of days and have now returned to this issue.  Thanks for pointing me at Write Datalog File Example.vi.  I have been studying it and have learned a lot.  Unfortunately, I'm still at a loss to understand the format of the file produced by the example.  I'll walk through what I've learned, and perhaps light will dawn on marble head (mine) as I do so.  Or perhaps a light will shine for you that says, "Oh, I see what he's missing.  If he just knew this blatently obvious piece of information, then he never would have had any questions to begin with.  Let me tell him this and his confusion will vanish!"

So... off to the races.

I see that the example collects 10 records of 100 single precision random numbers.  (The "10" and the "100" can be changed at runtime, but I'll go with these numbers for now).

For each 100 sample record, a date and timestamp string is bundled with the sampled data.

A datalog is opened (replacing an existing one with the same name, or creating a new one as needed) with the "Open/Create/Replace Datalog" Palette object.  This object is provided a sample record type consisting of a bundle with an empty string and an empty single precision array.

After opening the datalog, the example application enters a "while" loop in which it generates 100 (nominally) single precision random values (between 0 and 1) and a date/time string, bundles the string and the array of single precision values together, and hands that bundle to the "Write Datalog" Pallette object to scribble onto disk.  It executes this while loop 10 times (nominally) and the application exits.

That all makes sense to me.  Now, I want to look at the file that was produced by running this example...

The first 4 bytes of the file are:

0x44, 0x54, 0x4c, 0x47, which just happen to be the ASCII characters 'D', 'T', 'L', G'.

Those 4 bytes are followed by 41 various and miscellaneous binary valued bytes until the ASCII string "empty array" shows up in the binary data stream.

Following that is a whole bunch more, mostly 0 bytes until finally, 582 bytes into the file, I see a 4 byte number (in big endian format) 0x00000013 (19 in decimal), an ASCII string reading "7/8/2008:3:29:44 PM", a 4 byte number (in big endian format) 0x00000064 (which coincidentally is equal to 100 in decimal), followed by 100 4 byte quantities that look suspiciously like they are single precision floating point values between 0 and 1.  (They are probably in big endian order -- I haven't (yet) gone so far as to extract the binary values out of the file and attempt to convert them to floating point using big or little endian order.)  BTW, the ASCII string "7/8/2008:3:29:44 PM" just happens to be 19 bytes long.

This pattern: 0x00000013, a 19 character timestamp string, 0x00000064, 100 single precision floating point values, repeats 9 more times in the file.


What I have done is to reverse-engineer enough of the file to be able to guess at its format.  I knew going into the exercise that I was writing timestamp strings followed by 100 sample single precision arrays.

What I am missing is the specification from NI that says:
"This is the documented format of a datalog file.  Every datalog file shall start with the 4 byte header 'D','T', 'L', 'G' followed by a 4 byte 'machine type' field indicating the type of processor used to create the data, followed by a 2 byte 'format' field indicating the format of the data, etc..." (I'm making that bit up).

Right now, I have no expectation that data I write using another LabView application will start precisely 582 bytes into the file.
Nor do I have any expectation that the data will always consist of a 4 byte length word followed by the data itself.
Nor do I have any expectation that the data will always be in big endian format.

I have a whole lot of intuition, which I can put to good use, and very likely, extract the data I want out of the raw data file, but a written specification beats the best intuition any day.

If anybody can point me at that specification, I would be very grateful.  I have found a few similar "what is the format of the datalog file" posts, and, thus far no "the format is given in this piece of documentation" posts.  I am hoping that I have just missed that answer.  I am expecting that the answer is "NI does not divulge the raw format of such files".  If my expectation is correct, I am (more) confused and at a loss to understand what NI gains by not publishing that format.

Does my original question make more sense now?  Or have I totally lost everybody who had even the slightest interest in this long boring thread?

--wpd