memory limits

[lebecker]

I have devloped a Windows XP-based DAQ using 2 PXI-1045 chassis populated with PXI-6120 digitizers using DAQmx drivers. It runs into 'not enough memory' messages when attepting to acquire 1Msamples for 128 channels. According to Perfromance and Memory monitors, the main VI is using 286M, and according to perfmon, there is 2G of available memory for processes. The machine has 4G RAM. I have incorporated all posted suggestions for improving memory effeciency, all to no avail. The data is being read in 250M chunks with call by reference VIs, all references are closed after execution, the data structure is an I16 2D array (128x1000000), and no large arrays are being displayed. The VI generates 2 tasks, one for each chassis, and the data reads are serial (flat sequence) using the same subVI with different task IDs. (I noticed buffer allocations for each call to the data reader subVI.) The 3G switch has been added to the boot.ini file. Knowing that Windows systems have about 800M limit on processes, if the Performance monitor is accurate, there should be plenty of data space available. If this is a memory frag problem, how is it fixed? Do you have any suggestions? Thanks,

lb

[Norbert_B]

lb,

you should post the VIs you are using.

Nevertheless some basics on array handling:
a) Each array has to be in RAM in one piece, so if you have an array of 1,000,000 elements I16 1D, you need 2MB RAM in one piece without "interrupts"
b) Conversion of data creates copies. So converting 1,000,000 dbl 1D array to i16 needs (temporarily) 10MB, 8MB in one piece and 2MB in another piece
c) Resizing arrays, if not preallocated properly, needs more than twice (up to three times) the arraysize of the bigger array to be put together.
d) Memory fragmentation prevents the OS to allocate enough memory for one piece. So the out of memory message will be always created by the OS. So other way round: If you get such an OS message, you have most propably an issue with reallocation of or newly allocating arrays.

hope this helps,
Norbert

[lebecker]

Attached are the two DAQ subVIs. Thanks for the help

lb 

[TCPlomp]

One interesting option for you might be TDMS.

You can write from several places into the same TDMS file. This means you just stream the data to disc. In the way you have built it currently you will need one consequitive space of 128*1M*2 bytes=256 MBytes. That might be hard to find in your system memory.

Most likely you never need to have all the data in memory. You can't display it on screen or paper without smoothing or filtering.

Ton

[Norbert_B]

lb,

from taking a look into your VIs, you should definetly change some things:
a) Remove all sequence structures. This can easily be done by using proper errorhandling
b) You use "insert into array" to resize an (initial empty) array. This is not good because it has to reallocate the array each time. You should allocate enough memory from the beginning (see my previous post)!
c) "Request Deallocation" is nonsense in this context. I believe, that this function is one of the most misunderstood functions in LV ever. It is not the LV-counterpart of free()! LV does not provide the user with dedicated memory management functions like alloc(), malloc(), calloc() and of course free().

If the sum of all acquired tasks fill up RAM, you have no chance to put the data together for a single display. If this is the case, please write the data to disk and look at them after the whole acquisition is finished.

hope this helps,
Norbert

[RT4CY]

I have always found the following links to be vey helpful with memory management.  The In Place Element function is a particularly useful function which was added in LabVIEW 8.5.

Managing Large Data Sets in LabVIEW
VI Memory Usage
In Place Element Structure
Improving Performance with Efficient Memory Management in LabVIEW