Reading from an index in a continuously written to text file

[twescott]

Hi Everyone, 

I have a text file which is continuously being written to by another program with data.

I am trying to read that data from the file in real time or as close as I can get. However, as this file gets written to it gets quite large, and the file read takes longer and longer as LabVIEW has to read more and more. I am only interested by the new data appended by the program to the text file each write. Is there a way I can read only the newly added text each time, and thus keep the read time manageable? I've searched both online and in LabVIEW's help but i have not seen a VI that offers any sort of "Read text from index" functionality.

[RavensFan]

What program is writing to this text file?  Usually if more than one program has a file open, one or the other will throw an error about the file being locked.

There is a Set File Position function.  You can set that to beginning, end, or any byte location in between.  But what does the data look like?  If it the data you are writing isn't a consistent number of bytes, you might have a hard time finding where to set the position to.  But you could store in a shift register where in the file you are based on the last position and the last number of bytes you read.

I suggest giving a little more detail and the big picture of why you need to read from a file continuously being written to.  Even attach a VI and show a small example of the data file.

[twescott]

Its a proprietary program, which has write privileges open on the file in question. The file is a log file with a simplified structure as follows:

Data1

Entry seperator

Data2

Entry seperator

etc etc.

the data size are not reliably the same, so I cant read in set amounts, instead I've been reading with this function http://zone.ni.com/reference/en-XX/help/371361P-01/glang/read_characters_from_file/ and keeping a count of the characters read.

The data in question contains safety operating metrics that I am displaying to a graph in real time. I sadly cant post the actual code or data due to the proprietary nature. Ideally, I would have a function that lets me read characters from a referenced index in a file to the end of the file. I could then count those characters and update the reference for the next read through.

[RavensFan · Accepted solution]

Does the proprietary program ever modify the data you would have already read in your LabVIEW program?  (Does LabVIEW open it read only mode so that it doesn't throw and error when the proprietary program has it open first with the write privileges?)

What you said in your last paragraph is exactly what I'm telling you to do.

First read, set file position to 0, read X  # of bytes.   Get file position and put  in shift register.  Call it Y

Next read, set file position to  Y, read X  # of bytes.   Get file position and put  in shift register. 

Continue.....

[Alexander_Sobolev · Accepted solution]

 RavensFan, 

Do you need to store old position if you do not close file? It continues read from previous position on next iteration

I would add a check if file has enough data to read

.

You can edit  file in notepad in parallel, when you save it, this vi reads extra portions (5 bytes).

[RavensFan]

Good question.  I was thinking of storing the position because I wasn't sure what the file position would be considering a different application is manipulating the file in the background.  If the position is a property stored for the file within that LabVIEW file reference, then I guess you don't need to.  If the file position is a property that gets modified when the other application adds to the file, and what if that proprietary application happens to be another LabVIEW application, then can you trust the file position remained the same?

I'm guessing you are probably right, but I'd have to experiment to prove it.  And even if that is true, it doesn't do any harm to reset the file position after saving the current position value in the shift register.

Good point about checking the file to see if there is enough data.  One thing the OP hasn't told us and we can't see without the code is exactly how he is reading it and then processing the results.  You have your read has a regular read of a given number of bytes.  I based mine on a given number of lines since the example of the data file seemed to be line based.  In that case, what if 2 1/2 lines were written since the last read?  I'd only want to read the latest 2.

One thing about your code is that you seem to be accumulating all the data file in the shift register as it comes in, and the OP said that he only cares about the latest data and is willing to discard the oldest.

The whole program will wind up being more complicated than the simplest version based on the minimal information provided, because there really needs to be protection in the reads to know that valid data is available to be read and that it is complete.  Not just something to read but it only happens to be about 1 1/2 lines and the important remaining 1/2 line hasn't been written yet.

[RavensFan]

Actually,  I may have not understood the OP's example data file correctly.  I saw pairs of lines.

But the line breaks they had may have been arbitrary as the pairs say

Data 1

Entry Separator   (notice I've corrected the spelling)

Data 2

Entry Separator.

I wonder what is the Entry Separator?  Is it a comma or tab  or something else?  Or is it a line separator like a carriage return, line feed, or both?

[twescott]

So with regards to the structure of the file:

the other program is not a LabVIEW program. I'm not entirely sure of the language it is written in actually.

The program has the write privileges locked down on the file and only appends data to the file, it does not change any pre-existing data.

The data lines I showed include multiple lines of text with text based tags marking out different types of data, followed by said data. each data label and value is separated by a carriage return. The line separator is just a bunch of dashes.

so the file takes the form:

timestamp1

DataLabel_1_a   Data_1_a

DataLabel_1_b   Data_1_b

DataLabel_1_b   Data_1_b

------------------------------ (entry separator)

timestamp2

DataLabel_2_a   Data_2_a

DataLabel_2_b   Data_2_b

DataLabel_2_c   Data_2_c

------------------------------ (entry separator)

Note: there are plenty of lines of data I dont parse out due to them being unrelated to what I want to plot, but this is the general structure.

I already have built the necessary parsing VI's to break out a string into the various types of data. Effectively what I've been doing to parse is use a mixture of search/split and regular expression/pattern find vi's to break that data down and extract the data according to what it is.

---

---

The issue that caused me to post is that over any long time of running this program, as the file that's being written to by the other program gets larger and larger, the initial read of such a large string starts to bog down the VI.

I already parse and test whether or not a entry is complete or not in the file by testing if the entry contains a timestamp and entry separator, and basically ignore that entry, push back the counter i have for characters, so that during the next cycle when I read I will start parsing from there. 

I'll have to play around with those VI's you guys showed and see if they work, it looks promising from what I'm seeing though.

[twescott]

Just played around with that structure for file reading and it seems to work great, I didn't see any loss of performance as the file got larger. I still need to implement it in my main Vi, but I'm confident it will work based on how this appears to perform.

Thanks for the help folks!