How do you read a large (>100Mb) 3-D matrix from a file and make sure that you keep only one copy of it within the LV program for future use?

[inquisative]

I have to deal with large 3-D, I16 matrices that are stored in files. This data represents the density of an object at a sample point in 3-D space. I need to be able to display a 2-D section through the cube in an Intensity Chart. Since I want to update images in the intensity chart in real time to show different sections on demand I cannot adopt the approach of reading a section from the file when required - the I/O time assocaited with the disk read is way to long.

The solution, is to keep EXACTLY one copy of the entire data set in memory at all times and then just get the required section on demand.
This changes the disk I/O time to a memory access time, which is fast enough for my puposes.

Here is the problem. Since the matrices are so large, I want to be very sure that I only keep one in copy of the data in memory. After a couple of weeks discussioin with Tech Support, and the development of several sample programs, they have agreed that it is not possible to do it with less than two copies of the data in memory.

This additional memory requirement is a real problem for me and I am hoping someone is ingenious enough to come up with a way of doing it with only one.

I can send a sample program illustrating the problem if necessary.

Many thanks.

[JeanPierre]

The solution is to strore the array in the shift register of a so-called LV2 global. See the attached VI. You'll see that the array is never copied from the shift register, only portion as required. Adapt this VI to include a case to read the data from file and put it in the SR. Create other cases as needed to extract protions of the array. All array operations has to be done inside a case of the VI.

[Joe_Guo]

About the "real time", do you mean select a section and it displays immediately? If this is the case, re-organize the data file may improve the speed. For example, keep each section as individual block so it can be read in one operation, the start offset and size is computed before eading. Also remeber to open and close the file only once.

if the 3D plot does not need hi-resolution, downsample the data to be displayed because the scree size (pixles) of the 3D plot is limited, the section display can be used to see the details (it can also be downsampled). 1,000,000 data points shows no visual differences to 5,000 data points in a plot.

-Joe

[inquisative]

Jean Pierre,

Thanks for your reply - most kind of you to take the time. But.....

This is exactly what I thought, and is similar in concept to the way I wrote my code. But it doesn't work. I modified you llb slightly by setting the variables to be I16's and setting the default array size to 54,000,000 and the default action to be initialize. ( I'll enclose a copy of my modified llb for you.)

Here's what happened. I start the windows task manager, loaded LV and the program and looked at the memory shown in the task manager. It shows about 28Mb before I start to run the program. I then single step through the program. When it gets to the output of the Initial Array the memory usage jumps to about 134Mb, indicating the creation of the first copy of the array. As you continue single stepping it jumps to 239Mb as you leave the Case loop and head for the Shift Register. This memory does not drop down again.

After talking with Platinum Tech Support, they agree and tell me that they cannot see a way to have just one copy of the array. That's why I posted it on the message board to see if there was anyone out there who had come up with an ingenious solution. It seems to be a serious limitation to labview.

Let me know what you think.

Thanks once again.

[inquisative]

Joe,

Thanks. I some what over simplified the problem for the sake of brevity. In fact there is quite a lot of other processing of the section that may also be done after reading the section, it depends on prior choices made by the user. I think it s probably not practical to prestore the other views as there are just too many possibilities, and the time to calculate and pre-store all of them would become a major headache. But it does bear thinking about and I'll ponder it some more.

Thank you.

[JeanPierre]

In the palette Advanced>Data Manipulation there is the primitive "Request Deallocation". Drop it on your VI with flag=TRUE. When the VI finished, the unused extra copy is deallocated.

[DFGray]

If you change your intialization to a reshape array, you will only have one copy, not two. See the attached screenshot. The key to making this work is to only use in-place VIs on the shift register and to never break the wire, as your current initialization does. The array functions are mostly in-place. When in doubt, check it with the Task Manager or the buffer viewer. If you don't know what the buffer viewer is, read this tutorial - Managing Large Data Sets in LabVIEW. You probably want to, anyway, since it has a lot of tips you can probably use. Good luck. Dealing with huge data in LabVIEW is not easy, but it is possible.

[inquisative]

Brilliant. I tried it and it really seems to work. I'm now in the process of trying to implement it on my software which has a lot more distracting details that get in the way. I also looked up the note about Managing Large Data Sets. Combining your technique with the idea from the note of using VI server not only allows me to keep only one copy of the data, it also allows me to achieve the functionality of passing a large array by reference instead of by value.
Thank you.