How does Labview denote the end of a datalog file?

[AllisonSCJ]

Hello,

I'm encountering a strange error where one of my datalog files becomes corrupted, always at the first and second records. 

When setting the offset of the datalog file at offset 0 from start, I receive error code 4. And I receive error code 116 when setting offset to 1 from start. If I set offset to 2, Labview reads the record with no issue.

It appears one of my records was truncated, resulting in some kind of indication to Labview that the end of file had been reached, even though more data was in the file. How does Labview mark a datalog file as ended? Is a character placed in the file? I had assumed that Labview would just check the number of bytes in the file compared to its current position, but the existence of the error above would suggest otherwise. If I can figure out what situation is causing error code 4 at the start of the file, I might be able to figure out what is causing the file corruption.

[AllisonSCJ]

According the the reverse-engineered Datalog file as recorder here, after an initial datalog header specifying file type, record count, and the record type descriptor (the last of which would of course vary by record type - I do not know how to parse mine or how long it may be) there is a section of 512 bytes specifying the indexes of the next 128 records.

Presumably, when the first record is written, the 512 byte index block is initiated, with a new block created at record 129. 

My thoughts on how the corruption occured:

1. Writing data to record 0 failed; the read record expected, say, 30 bytes of data, but no data was recorded at that chunk, so when it's read, an EOF error is recorded. If the chunk is still initialized in the record, then following records would still be in their appropriate location, which is why I'm able to read at offset 3.

Issue with this theory: I'm not sure how writing to the record would fail at index 0 without throwing an error at the write vi.

2. The index chunk was garbled

Issue with theory: pretty sure all of the indexes would be buggered, then.

I've seen issues before where data packets have been sent out of order, without triggering any application errors, over a network connection in the domain that I reside. This may explain why data is being garbled.

If the reverse engineering of the file format from above is correct, I may be able to avoid this corruption if I read and write 128 records at a time when writing a large number of records. But if this record chunk count is variable based on the data type, I may make even worse of a mess.

Still looking for further verification of header format.

[AllisonSCJ]

Wrote a vi snippet to explore the format of a datalog file. You still need the data type of the record to fully parse it, as I have no idea how to parse the data type chunk (probably for the best). I used the vi to check my theory. It appears that yes, the first chunk of record offsets was garbled. I can confirm though that each chunk is a set of 128 records, so perhaps a read/write of 128 records at a time and a validation check of the written data would be the best way to catch these errors and avoid corruption.

I'm not sure why the corruption is happening, but I believe it's a network-related error.

I'll update this thread if I learn anything else.

[rolfk]

The datatype chunk is in essence the typedef descriptor of the cluster/data that you setup the log file for. This type descriptor is what the Variant to Flattened String also returns and its format is pretty much documented, although there are some LabVIEW private datatypes for refnums and such that are not detailed in the documentation but logging such information doesn't make any sense as a refnum is only valid within the session and context in which it was created. 

[AllisonSCJ]

Thanks Rolfk.

I know that with a little work, parsing the typedef chunk of the header wouldn't be impossible. However, for my purposes the snippet I wrote gives me the information I needed. Someone else may decide to take it a step further.

Regardless, I think reading/writing in chunks of 128 records may be the best way to edit a single record in the file to avoid corruption over a network connection.

Reading in chunks has the benefit of being faster as well.

[AllisonSCJ · Accepted solution]

Based on what I've learned from parsing an existing file, there is no special character denoting the end of a file (I didn't expect there to be one, as this would cause possible issues when parsing binary to other data types)

Instead, the existence of a chunk of bytes specifying the expected locations of each record would suggest that the Labview datalog read vi expected a certain amount of data at record #1 and, finding less than the data expected (or possibly no data) at that location, error 4 was returned.