64-bit LabVIEW - still major problems with large data sets

[DonRoth]

Here is the screenshot.

Not 100% sure what you meant regarding use of the IMAQ reference.  Now I dispose of it on each iteration, a strategy I have used in the past but that I initially did not use here, then went back to upon Christian's suggestion (but it did not affect the memory allocations).

[DFGray]

Looking at your image, I don't see anything obviously wrong.  The intensity graph plot will cost you a copy or so, as will the record and convert to cluster of array, but these are single images and the memory should be reused loop-to-loop. At exactly what part of this code does the memory blow up on you?  I would expect a gradual increase at each loop, but no sudden balloon.  Is it possible to see more of the code?

The IMAQ reference wire is exactly that.  IMAQ realized early that memory allocation could be a problem with LabVIEW and images.  As such, the IMAQ reference is a pointer to a memory location which holds the image.  It is not a full copy of the image.  It is essentially a single-purpose, early form of the Data Value Reference.  As such, using an array of IMAQ references should not eat much memory.  In fact, it should eat a lot less, since you images appear to be I8s.

[DonRoth]

Hi Damien -

I am attaching the code (LabVIEW 2010).  The memory usage is at 4.9 gbyte at the end of the 'write' case, when opening up an ~ 1.2 gbyte file. 

(By the way, to recap, that is pretty great compared to where we were when we started this thread). 

The rest of the code is just the 'read' case to just test whether the images read properly and a 'stop' case.  The 'read' cases is the same as the 'read' case image I have previously posted, except I moved the intensity graph inside the interior in-place structure.  You can ignore the portion of the code asking to resave the images to another format (I have put the 'save' portion of the code in a diagram disable structure as it is not pertinent to this discussion).

As far as the IMAQ Vision discussion goes, it is only a reference to one image.  There should not be an array of IMAQ references as I am disposing the reference after exiting the for loop each time.

I use other software as well for visualizations of these image series and they are using much less memory.  I know we are going in circles with this, but this is actually a major issue for me now.  The data sets I will be working with will only be getting larger and larger, perhaps up to 4 - 5 gbyte.  So it could be very useful for me and probably many others to be able to figure out 1) why are we creating these extra copies and 2) a way to deallocate memory after function calls where it is suspected memory is being chewed up (perhaps unnecessarily). 

I really appreciate your input to the discussion and bringing this possibly to the forefront at LabVIEW R&D.

Sincerely,

Don

[Ben]

---

@DonRoth wrote:

...

I really appreciate your input to the discussion and bringing this possibly to the forefront at LabVIEW R&D.

Sincerely,

Don

---

And I really appreciate that you are having this discusion in public where I can learn as well.

My schedule has prvented from getting involved but in the event you have any question about the buffers for the IMAQ stuff, the attached VI may help.

One of my associates got it from NI support when trying to chase an issue with IMAQ buffers.

Again, thank you!

Ben

[DonRoth]

I wanted to take a step back and compare the data value references approach to the queue-enqueue strategy for this simple piece of code.

My results are that the equivalent amount of memory is used during the write portion, with one less in-place structure on the read portion, using the queue-enqueue strategy.

So it probably just makes sense for me to replace my 'Preview queue' fxs with the Queue-Enqueue strategy in my main application as opposed to replacement with data value references.

Still do not understand why, with the data set in the 1.2 gbyte range, RAM memory usage is at 5 gbyte. This is one of LabVIEW's mysteries, I guess, but it has major negative ramifications for me.

Sincerely,

Don

[Yamaeda]

You'll get one buffer allocation in each of the cases.

It looks to me like you're basically doing the same in all of them. Cant you somehow skip the case altogether? Since the biggest output format seems to be U32, the memory footprint would be smaller if you converted to U32's instead of SGL (though there might be issues with IMAQ preventing that, i dont have it nor have used it).

Would the memory/buffer change if you dont update through the Local Variable in Write?

Ranting/Brainstorming

/Yamaeda

[DonRoth]

Actually - you have probably hit the nail on the head.  The data set I am loading is ~ 1.2 gbyte of 8-bit BMP.    My required conversion to SGL is the problem as it requires 4x the memory vs. using 8-bit BMP (SGL is 4 bytes (32-bits).    We mentioned this in the thread early on but it somehow got lost as we moved into the data value references focus.

So at this point, I need to really store the image series as 8-bit, and do the conversion to SGL as needed outside of storage.  This requires a major re-architecting of the application.

Thanks for all of your inputs.

Sincerely,

Don

[DonRoth]

...And as a corollary to this, my real application opens all of the common image types (tiff, png, bmp, etc.) as well as images in raw binary data (SGL) format and text format, and image series stored as AVI.  So if the data sets are so large that I can't live with the RAM hit, for some of these formats, I actually will be downconverting and losing dynamic range in the images going from as much as 4 bytes down to 1 byte for the image storage if I was to want to store the image series as 8-bit (1 byte) in all cases.   My other option is to bump my RAM to 32 gbytes, and more as computers continue to evolve.

Don

[Yamaeda]

A rather drastic thought is ofcourse: Do you really need to keep all images in memory? 🙂

Often it's the analysis and results that's the important part, not the raw data ...

Another drastic idea is to store downsampled versions in memory (downsample be 2 or 4) for quick access and if you need to see all details you'll reload the full image. The analysis should ofc be made from the full data.

/Y

[DonRoth]

With the advent of 64-bit OSes and 64-bit LabVIEW, the thought was to move to RAM and away from disk access for these large data sets to allow speedier access to all of the data.

I thought about data compression / decimation as one other possibility as alternate to downconversion.  I have to give it more thought.

Thanks,

Don