How to move to specific locations within text file?

[Tarnarmour]

I'm making my first forays into file IO tools with a project where I record data from a device, analyze the data, save a result of a test and the raw data, then change the settings on the device and repeat the process. Currently I have my VI working where I can conduct multiple tests in a row and write all this data into a text file by just opening the file, moving to the end of the file, and appending the next test result.

However, I'd like to be able to write the test results near the beginning of the file and have all the raw data at the end, for readability. The text file would look something like: 

header section

test 1 result

test 2 result

test 1 raw data

test 2 raw data

end

where as now it looks like:

header section

test 1 result

test 1 raw data

test 2 result

test2 raw data

end

I know which VI does this, the scan to file location VI. But I don't know how to find a specific location in a file, say the first character after the keyword "test 1 result". I would also like to know a good way to read specific data out of a file like this without needing to pull the entire text file into a string and matching string sub-sections (though maybe that's the only way to do it?).

I'd appreciate any help, I'm very new to file IO stuff in labVIEW or in file IO stuff for programming in general, so any corrections and advice is welcome.

[crossrulz · Accepted solution]

Since you are trying to insert data in the middle of a file, you will need to read the entire file.  Then with the string, insert the data.  Then overwrite the file.  But with the entire string, you can use Match Regular Expression or Match Pattern to find the locations you want.

[altenbach]

With lots of discipline, you could make some of the header fields (or the entire header) a generous fixed size, keep track of all their offset positions, then overwrite with new (correctly padded) data (or new header) as needed. Probably not recommended.

How big are these files?

[tst]

Note that this means you would either need to know what goes into the file before putting it in (i.e. keep all the data in memory until saving), modify the file to move stuff around as more data comes in or leave enough space in the file for the relevant data and preallocate the space, none of which are ideal.

I would suggest rethinking the file structure, if at all possible. For example, you could simply place the data in separate files in the same folder or you could save the data in a database (some databases, like SQLite, are fairly simple to install and use).

If you do want to stay with this structure, then I would suggest keeping the beginning of the file for an allocation table, where you write the exact offsets of the relevant sections, as altenbach suggested, but I would also recommend against that, as this is a programmatic access feature and it seems like you also want this file to be human readable (and hopefully not editable). You could also store the offsets in an external file, but that complicates things.

[JÞB]

Only because three other Knights have not mentioned it yet....   I will state bluntly that you are using a text file to store Test Data.

Thats analogous to a braile color scheme.   

A Test Data Management Streaming format, TDMS, has all the features that you want by simply using file, group and channel properties to add the interesting data.

EXAMPLES: 

• File properties; Computer name, location, etc...
• Group properties; Test version,  hardware configuration, calibration, etc....
• Channel properties; start time, RESULT,  first failure,  user ID, UUT data [a, b, c...], etc...

Assuming 1 file per station 1 group per test type and 1 channel per test execution <Process Control model>.   Another common use with TDMS and advanced concurrent access is 1 file per uut, 1 group per Test type,  and 1 channel per Test execution < Factory Flow management model >

[crossrulz]

---

@JÞB wrote:

A Test Data Management Streaming format, TDMS, has all the features that you want by simply using file, group and channel properties to add the interesting data.

---

TDMS is actually not a bad idea.  I have used it to store test data before.  I am not a fan of it as it felt kind of forced.  A good database design would be the more natural route.

[JÞB]

---

@crossrulz wrote:

---

@JÞB wrote:

A Test Data Management Streaming format, TDMS, has all the features that you want by simply using file, group and channel properties to add the interesting data.

---

TDMS is actually not a bad idea.  I have used it to store test data before.  I am not a fan of it as it felt kind of forced.  A good database design would be the more natural route.

---

Getting a bit OT for a new user.  I am not knocking a well designed database.  The trouble there is that when you think of new data you want you need to update the database, the test code and anything that uses the data.  The TDMS import addon for Excel makes the presentation on the backend "Cave-man simple," while providing a single edit point to add or remove measurements, groups, channels and properties.  

<Grumble mode>

Unless you are using DAQmx Logging.   Whatever happened to that Idea for the DAQmx Set Log Attribute?

</Grumble mode>

[crossrulz]

---

@JÞB wrote:

<Grumble mode>

Unless you are using DAQmx Logging.   Whatever happened to that Idea for the DAQmx Set Log Attribute?

</Grumble mode>

---

It is still listed as "New": Additional TDMS Metadata 

[altenbach]

It really depends on what that file is later used for.

Are there users on the production floor that want to read it in Notepad later?

Does it need to be processed by other software later (matlab, excel, word, sheets, openoffice calc, etc.)?

TDMS is a special advanced format designed by NI to describe measurement and related metadata. Yes, some third party software can read it directly and some others offer plugins, but there is really no way to conclude that this is what the OP was looking for, especially since there was a description for a very defined file structure. 

A file is just a linear sequence of bytes represented in hardware (Flash cells, magnetic sectors, etc.). There is no concept of lines, columns, headers, etc. For example a "line" is defined by the arbitrary presence of line break delimiters between sections of bytes that represent readable characters. Any byte can be overwritten, but "Inserting" data is not possible without reading all later bytes and rewriting them with a higher offset.

[Tarnarmour]

First of all thanks for all the helpful comments and explanations. I guess I was naively hoping there was some magical way to just find locations in a file but it makes sense that to find a specific substring in a text file you would need to read out the file. The data here is not crazy large but it's probably big enough to make it not worth doing that. I understand a lot more now the use of things like database structures, this has been informative.

I'm curious about the TDMS format. I thought to use text files mostly for ease of access; the raw test data being stored is more of a redundancy. The main use of these files is for other engineers to be able to verify that a product validation test was carried out and that the product passed the test when they were manufactured. I'll do some reading about the TDMS format, but do you think it's easy enough to access for people who may not have LabVIEW?