Dr. Damien's Development - Object Based File I/O VIII - Text File I

[DFGray]

Building on the previous post, I have created a text File I/O object and implemented the read functionality. Included is a small read test which exercises just about everything. This is definitely a work–in–progress, so any and all feedback would be appreciated.

The UML diagram has changed a bit, since I had some interdependencies on the Location object I had to reconcile. I also split files into pointer–based (e.g. binary or text) and hierarchy–based (e.g. TDMS, XML, or HDF5). Pointer–based files will use the Location object as a convenience, while hierarchy–based files will use it as the primary means of access. Implementing it in both will be critical to getting them to inter–operate. However, I have not yet implemented it for text. I will implement it, since it will be highly useful when I take text to the next level — spreadsheet files and LVM.

The file read is buffered, in hopes of speeding multiple accesses to the same file. This is based on benchmarks I did many years ago. I have not yet benchmarked it in a current version of LabVIEW. The primary use case will be repetitively fetching successive lines from a file, such as one would do accessing a spreadsheet type text file.

Searching for tokens and getting lines is supported. I have not tested this code very much, so there could be all sorts of bugs. Let me know what you think. I have already received the following feedback:

1. The Data object needs to be renamed, since it is not data, but data access. Good candidates for this object’s new name gladly accepted.
2. The system needs to be by–reference, not by–value so it will avoid issues when the object wires are forked (e.g. simultaneous use of a file reference, multiple setups for the same file).
3. Some of the wire colors ended up looking like error wires, so need to be recolored. This needs to be done, in any case, when going to a by–reference architecture. The wires should look like references.
4. Some of the icons have poor glyph choices.

I realized as I was writing this that the basic algorithms also work quite well for binary files, since that is essentially what a text file is. So, implementation of a generic binary reader would not be too tough. The architecture would also work for almost any sort of streaming data access, such as networks or loop–to–loop communication.

Previous Posts In This Series:

1. Representative File Types with conversation on design
2. Requirements
3. Top Level Design
4. Design II
5. Design III
6. Implementation I
7. Implementation II — Base Classes

[Neil.Pate]

Hi AQ,

Did you get any further with this side-project? (Is this the latest update?)

The zip you have attached to this post does not have the text File I/O object you talk about in this post.

I am in the process of rolling something similar (but significantly cut down feature set), I want to have a file class that can handle two (for now) different binary data formats.

[DFGray]

You can find the latest update here, but it is still not complete, and it is currently undergoing a major rearchitecture, all while we are trying to get betas of our software out the door.

In short, due to the possibility of simultaneous access to a file, the state information needed to be in some sort of global, protected database.  I could have used named single element queues (name being the filename) but after many conversations with a lot of folks here at National Instruments, I opted for a more elegant and expandable architecture, which I am in the midst of implementing.

I changed to a daemon-based model.  There will be a master file broker to handle file open requests, and a specific file broker to handle I/O for each open file.  The use of the file brokers ensures serialization of commands to any particular file (from any one process) and allows dynamic loading and unloading of file I/O code as needed.  You can think of the file broker as a free-running action engine.  I considered using the Actor Framework, but its use would have resulted in an extra copy of data at each save and read, and effeciency is paramount in file I/O.

If you subscribe to this thread, you will get notification when I post new code.  Things are quieting to a dull roar, so it should not be too far off.  Thank you for your interest.