Opening a large .mat file on Labview

[Mikael__Z]

Hi everyone,

I'm very new to using LabView so I hope my explainations will be clair enough.

Here is my problem :

I'm working on very large .mat files (about 400-500Mo). These are 3D maps, so on each file contains a 1D array for X, a 1D array for Y and a 2D array for Z.

I'm trying to collect the data from the 2D array Z.

I'm currently using the matio library to extract the data but the heavy files cause memory issues.

To solve that, I would need to know how to locate where is the Z array in the file to treat a smaller amount of data.

So I guess I'm looking for informations about how to use the matio library, or about how is a .mat file with multiple arrays built.

Thanks a lot for your answers.

[Yamaeda]

As it's a Matlab file i'd use a Matlab node and use matfile to extract part of the data, according to this:

https://stackoverflow.com/questions/3947549/what-is-the-difference-between-m-and-mat-files-in-matlab

[Mikael__Z]

Thank you for your answer.

The thing is I'm trying to develop a software which will be used by other people.

I've already managed to extract a txt file of the 2D array thanks to scilab.

But it has to be an automaed process and it woulld not be convenient to ask the user to extract the file himself before using the software.

[mcduff]

Posting mobile here, so cannot show any examples.

the Matio library is a great starting point, but extremely memory inefficient. There are multiple string concatenations among other issues. I rewrote parts that I needed to handle large files. Lastly the Matio library uses the old version of Matlab files, the new one is based on HDF.

mcduff

[Mikael__Z]

Ok, thanks a lot, I would be very interesed to see how you rewrote these parts if you agree. I think I'm too new at this to really undertsand how the matio VIs really work.

Mikael

[mcduff]

I rewrote specific parts for writing Matlab files. My users wanted to save data in .MAT format and analyze it in Matlab. Why don't you post what you are trying to do, and I will see if I can help.

mcduff

[Mikael__Z]

Here is my file on Scilab. As you can see, there are 3 arrays.

I'm trying to extract the data of the Z axis on Labview in a 2D array.

Right now, I'm using Scilab to have a txt file for the Z axis and then extract the data.

But I would like to work directly on the mat file in Labview and to read only the 2D array of the Z axis.

Mikael 

 

[mcduff]

---

@Mikael__Z wrote:

Here is my file on Scilab. As you can see, there are 3 arrays.

I'm trying to extract the data of the Z axis on Labview in a 2D array.

Right now, I'm using Scilab to have a txt file for the Z axis and then extract the data.

But I would like to work directly on the mat file in Labview and to read only the 2D array of the Z axis.

Mikael 

 

---

It would help if you post a small example file(mat) and any attempts at reading that file you have tried. I am willing to help, but not willing to write something from scratch based on pictures.

mcduff

[BertMcMahan]

A 500 MB file shouldn't be a problem to handle. Either your toolbox or your other code is duplicating your data somehow. If you post your code we can help you figure that out.

Reading only the Z array won't help your memory issue. For easier math, say you have a square grid m by m. Your X array and Y array are m units long each, but your Z array is m^2.

Let's pretend each data point is a single byte, then you have 5 million points. Total size is 2*m + m^2 = 5 million. m is therefore 22360, so your X data is 22 MB, your Y data is 22 MB, and your Z data is 450 MB.

In other words, reading in the whole file isn't your issue. It's your handling code. Look for buffer allocations in your handling code. Each time you allocate a new buffer you use an additional 500 MB. If it's not in your handling code then it's in your library's code, which will be trickier to fix.