Dr. Damien's Development - Object Based File I/O I - Representative File Types

[DFGray]

File I/O is a classic area where object oriented programming can greatly ease your day to day development. I have been kicking around the idea of wrapping LabVIEW objects around file I/O for quite some time, and got inspired by the fact that Ben has already done it. I like to plan things up front, so first will look at what file types to support and their logical groupings. Here is a first pass.

ASCII

• Spreadsheet — file containing columns and rows of data. Columns are separated by a specified field, usually a tab or comma. Rows are separated by a lines.
• LVM — specified spreadsheet format. Contains a specified metadata header and file format information.
• Configuration — Windows configuration file (a.k.a INI file).
• XML — hierarchical text format. Somewhat human readable. Useful for documents and configuration data.
• TDM — DIAdem native format. An XML variant with structure.

Binary

General Purpose

• Flat Binary — simple binary file. Could contain multiple elements of different data types.
• HDF5 — the ultimate hierarchical binary file. Has more features than any file format I have ever encountered. Very stable.

Sound

• WAV — Microsoft Windows sound file. Exists in many variants both with and without compression.
• MP3 — most common format for lossy compressed audio. Patent encumbrance causes distribution issues.
• Ogg Vorbis — open source answer to MP3. Has no patent encumbrance, so is extensively used by open source software.
• AAC — designed as successor to MP3. Also has patent encumbrance.
• FLAC — cross–platform open source lossless compressed audio file.
• Monkey Audio — Windows only, free, closed–source lossless compressed audio format. Slightly better compression than FLAC, but not as well supported.
• UFF58 — standard file format for sound and vibration analysis.

Logging

• TDMS — excellent logging format.

2D Image

• PNG — standard format designed for web use. Lossless compression. Very flexible file format.
• JPG — probably the most common image format. Compression is lossy.
• GIF — older format with lossless compression. Patent encumbrance prior to 2004 caused the creation of PNG, which is a more capable format.
• TIFF — one of the oldest, but still one of the best, image file formats. Has a plethora of options, including multiple images per file and numerous compression types.
• FITS — image type created for scientific use. It was created to replace TIFF due to patent encumbrance issues prior to 2004.
• SVG — vector and XML based image format created for web use.

3D Image

• DICOM — standard volume imaging representation for medical data.

Office Suite

• Microsoft Office — very commonly used word processor, spreadsheet, &c. file formats. The Excel file is probably the most commonly needed from LabVIEW.
• OpenOffice — open source word processor, spreadsheet, &c. file formats. The OpenOffice Calc file is probably the most commonly needed from LabVIEW.

But what about EMF, Ogg Theora, or AVI formats? There are hundreds of file formats that LabVIEW could potentially read and write. If the object structure will support the above list, it can reasonably be expected to support just about anything. However, if there is a file type you think would help develop the object structure, please let me know. Also let me know if you disagree with my grouping (does SVG belong in image or text?).

Next time, we will coalesce the file formats into a list of requirements and start a specification to support the above list.

[DFGray]

The link to the FITS file format above has a cut and paste error.  It should be this.

[GregSands]

I look forward to seeing where this goes.

For binary files, Matlab formats are very commonly used, and are a fairly simple binary format with a header.  I wonder whether they are worth including explicitly, or whether HDF5 encompasses all of these features (such as Sparse Data storage).

For 3D Images, there are possibly a range of other common formats.  Three come to mind as providing various modes of data storage:

• 3D TIFF - due to the ability to handle multiple images in one file
• VTK - current versions are XML-based.  A format like this would add the ability for mesh-based (and unstructured) data rather than just structured image data
• Image sequence - often a sequence of 2D images is used as an easy substitute for a true 3D image format.  It would be great to handle multiple files using the same interface as a single file

[pauldavey]

C3D is a common file format for motion capture systems.  There is already a DataPlugin for reading them.

[StefanR]

You can use DataPlugins (available at ni.com/dataplugins for free download) to read (and in some cases write) over one hundred (100++) different file formats using LabVIEW 2010 Storage VIs.

You can even program those DataPlugins on your own by using the LabVIEW DataPlugin SDK or referring to the DataPlugins VBS Programmer's Reference Guide. DataPlugins can be used throughout several products of the NI software platform such as LabVIEW, DIAdem and DataFinder.

[Ben]

Although it seems like over-kill it has already started to pay-off in my applications. i was turning my attention to the writting of the final report when I remebered "I already have that type of file in my Library!" since the hardware I/O classes used the File functions to get their configuration (I use methods like "Go configure yourself") which reduces coupling between the hardware and the classes that use those services. The big pay-off comes when I get an error message returned by one of my functions reminding me that I was mis-using the Class by not declaring a section name etc. THe non-wrapped function would (in the case of config file functions) return a default for a missing value and I would have to chase back the wires to figure out where I screwed up.

Again to wax philosophical:

LVOOP has opened a new tool box for me. Prior to LVOOP I would be seraching the bottom of my hiearachies for functions that where autonomous and could be re-used. Since I generally had a collection of Action Engines that in turn had app specific functions, the "code at the bottom" was tightly "Coupled" to the app for which they had been devloped.

LVOOP turns things inside out compared to Action Engines (AE data is on inside, in SR, In LVOOP data is outside and we just point at it using teh VLOOP wire.)

Since LVOOP support dynamic dispatching I can now re-use the code that used to be INSIDE AE's in other app on different data sets (Private data) many of the "Actions" I used to code into AE have become my Methods in my Libraries. And as part of dynamic dispatching, there is no need to recompile and re-test your code when re-used as would be the case with AEs.

Done with the personal witnessing, can I get a big AMEN ? *

Ben

* Well maybe not now. I'll let you try it for yourself.

[F._Schubert]

Ben, you brought me to new insights with this posts.

When I started using AEs it was feeling strange of not having 'the' wire. I even wrote some code based on the old Config VIs just to get a pseudo refnum. Then I got used to this error in the design.

All the way since then I was just missing inheritance in AEs and thus embracing the idea of LVOOP, while on the other hand just treating it as 'glorified cluster'.

How wrong I was, it's the AE with the wire! + inheritance.

On the approach of Damien (DFGrey): I think it could be time to start a new big-big project like OpenG, not 2.0 but OOP; a project which aims for getting the native palettes powered by LVOOP. Maybe we as community should use this series of posts to discuss if we want to put the necessary effort into an adventure like this.

Felix

[Ben]

---

@f. Schubert wrote:

Ben, you brought me to new insights with this posts.

When I started using AEs it was feeling strange of not having 'the' wire. I even wrote some code based on the old Config VIs just to get a pseudo refnum. Then I got used to this error in the design.

All the way since then I was just missing inheritance in AEs and thus embracing the idea of LVOOP, while on the other hand just treating it as 'glorified cluster'.

How wrong I was, it's the AE with the wire! + inheritance.

On the approach of Damien (DFGrey): I think it could be time to start a new big-big project like OpenG, not 2.0 but OOP; a project which aims for getting the native palettes powered by LVOOP. Maybe we as community should use this series of posts to discuss if we want to put the necessary effort into an adventure like this.

Felix

---

Thanks for that reply Felix. My studies of LVOOP in the context of what I already used to devlop pre-LVOOP showed me that somehow (thank God!) I had managed to guess right at coding (after all it should make sense regardless of the fancy words used).

The big gap I have now is keeping track of where my data is since it not just in the SR and sharing a sigle copy (Singleton) requires I put it where everyone can get at it without resorting to a single mega class at the top end.

For the challenge above, I would think that a "worse Case" Domain diagram and static Class diagrams since the inheritance if implemented optimally (is there optimal LVOOP?) will be rather large and consist of many thin layers. From where I sit I don't see any recursion in the project so it should develop nicely and incrementally with little or no complications until we get to the higher level Classes like TDMS. My library for TDMS functions is already pretty heavy and I have only started to scratch the surface of its capabilities.

Still thinking out-loud...

My Hardware functions use Config files to configure themselves using the Sections of the config to set the parameters for the Board and also for each channel. But in theory I should be able to mod the file stuff i have already such that it points at a previously saved TDMS file to pull all of the same data. But due to the differences and (having what resembles a life) I have never goten aroun dto that switch.

if you don't mind I'll keep throwing ideas at this thread until someone says stop.

Ben

[F._Schubert]

Ben, don't stop! Not because you're famouse 'Ben', but because the most couragous thing man/woman can do is to publicly state a prediction and accept if proven wrong.

The AE design was good before LVOOP because it did facilitate an important concept: encapsulation. It still has a niche which isn't touched by LVOOP.

This would be my big champion challange: xmi for uml with support for profiles (SysML) and 'of course' it's defined by xsd. Gives you a way to deal with dependencies to other documents including recursion. Sure, I'd accept anything that won't target this for this series.

Felix

[Ben]

OK this mornings dump.

"Low Coupling" is one of those ideas put forth by the Gang of Four (who invented OOP in the first place). As i understand the term it speaks of the relationship between code elements and to what extent they are dependent on each other. This rule/pattern is used to evaluate different design possiblilities and to help choose between them.

Yes a pile of high powered words but in terms of LabVIEW I understand this in terms of the "dirty dot", the asterisk telling us something has changed. When we see the dirty dot, it is a sign that something in the VI changed and requires saving AND for the anal-retentive types (like myself) it is a sign that the code needs to be re-tested to ensure it still operates as we expect.

After coding a couple of LVOOP apps and looking back at what I had done, I found that my later projects exihibted all or most of the patterns described by the Gang of Four (opps, I accidently did something right) and it is these later projects that have been yielding fruit (re-use code).

So the goal I would like to present today is "Low Coupling".

If the implementation of the file classes (question to other readers "What kind of inheritance do you see for this challenge?" What names can we assign to the elements in the Domain that will support "High Cohesion"* while allow expansion for future file types?") exposes a robust interface that also imposes little coupling to the code that uses those functions (methods in LVOOP speak), then the resulting classesa and libraries can be contually enhanced without impacing the callers!

Sound far fetched?

Not really. Stephan has already pulled that stunt on us without many of us realizing it. The LV Config file functions were converted from GOOP to LVOOP in about LV 8.2 > 8.6 I believe.

Done dumping for this AM.

I will watch for others.

Ben

* High Cohesion is another Gang of Four pattern that ask the questions "Can someone familiar with the real world system capable of understanding the software model used to implement same?" Or a real-world example "Can someone that understands using the third party driver understand the LVOOP implementation?"