How to save data in Labview for easy loading

[jonty_roads]

Hello,

I generate 6500 patterns using labview (each of size 500x500) and save them as .mat files for later use with another vi.

However loading these patterns from .mat is not happening for me due to plugin issues. 

When i try to convert these mat files into text (stacking in rows each patern; 6500x250000), it again is a problem because of huge size and loading problem.

Please suggest how i can save my data from labview to a fast loading file for loading in labview.

[jonty_roads]

Hello All,

I am finding it difficult to read .mat files (1 to 6000 files) each having binary patterns (500*500).

I am trying to save these patterns from matlab in a text file so that i can read in LV using spread sheet and further use them.

Matlab is creating a huge file if i save them in an array of size (600,250000). 

Please tell me an efficient way to read these .mat files and make them available in labview. The plugin option is not working for me.

Regards

[cbutcher]

You mention .mat files - is Matlab a required tool at some point in your process?

If you don't need to use .mat files and don't need to make using Matlab easy, then a simple binary file will be your best bet.

You could create one file per pattern for small files but many, or group several for a larger file, smaller number, or group all for 1 large file.

The total size will be around 6500*500*500*value size, where value size is 8B for double and 4B for single precision floating point, or smaller for e.g. U8, U16.

With doubles, you'd be using ~13GB if I calculate correctly...

[jonty_roads]

No Matlab is not important,not it is used in the process.

When using .txt file with GB data, it gets difficult for LV to load data and crashes with memory issue or takes long to load.

Therefore, I am saving each pattern in a seperate txt file : 1.txt, 2.txt, ....

and then its easy to load in LV.

Any better alternative is welcome.

[cbutcher]

So when you write a text file, it uses one U8 value per character. Depending on the number of decimal places you save, that can make a text file much much larger than storing numeric values. 

The advantage of a text file is easily reading with a text editor, but binary files are almost always much more efficient. 

[Yamaeda]

As mentioned, for fast loading and saving, binary files is the goto solution. As for the memory issue when reading it back, you'll have to handle it in parts, e.g. 1 file at a time.

/Y

[Bob_Schor]

The structure of a .mat file is fully documented (by MathWorks) here.  It's a fairly complex format, but you could write a parser.  

It might be simpler to save the data in Matlab as a straight binary file.  I haven't used Matlab in years, but recall reading some .mat files and writing the back in binary format, easily read and parsed in LabVIEW.

Bob Schor

[cbutcher]

Is this question the opposite of your other (seemingly similar-ish) question here:  How to save data in Labview for easy loading?

It might be easier to keep discussion in one thread if the questions are very similar and being asked essentially concurrently.

[Bob_Schor]

---

@cbutcher wrote:

Is this question the opposite of your other (seemingly similar-ish) question here:  How to save data in Labview for easy loading?

It might be easier to keep discussion in one thread if the questions are very similar and being asked essentially concurrently.

---

Yes it was.  I've combined them.

Bob Schor