Building array exceeds available memory

[AdamEdinburgh]

Hi,

The programme that I have been working on for some time now is nearly complete.  I have one problem remaining which I can't find the solution to in the forums.

My programme has to create a very large array (around 20000x2500) and then write it to a file so that another programme can use it.  It actually processes all the data for the array (which is great!) but can't write the array to file because doing so exceeds the available memory.

Can anyone offer any advice about managing large arrays in LabView that might help with this problem?

Many thanks

[Ben]

---

@AdamEdinburgh wrote:

Hi,

The programme that I have been working on for some time now is nearly complete.  I have one problem remaining which I can't find the solution to in the forums.

My programme has to create a very large array (around 20000x2500) and then write it to a file so that another programme can use it.  It actually processes all the data for the array (which is great!) but can't write the array to file because doing so exceeds the available memory.

Can anyone offer any advice about managing large arrays in LabView that might help with this problem?

Many thanks

---

This tag cloud contains links to thread on that topic or related.

Are you using "Write to Spreadsheet file"?

Those type of chanllenge generally have me thinking Action Engine since it lets me work "in-place".

Another approach that works well is to use multiple queues rather than a multi-dimensional array so that a single buffer is not required.

have fun!

Ben 

[crossrulz]

How are you writing the file?  What format are you saving the data?  I would look into possibly writing in chunks.  Can you write to the file as the data is acquired?

[AdamEdinburgh]

Hi,

So originally I had the programme building this aray line by line inside a big for loop, then passing the entire aray out to a 'write to spreadsheet file'.  This gave a 'not enough memory' error.  I then switched this to reading the spreadsheet inside the for loop and inserting each line into the spreadsheet inside the array.  This seemed like a fix, but I realise now that the programme has to read in the array inside the for loop anyway and it is still too big.

What would be good would be to be able to write lines to the spreadsheet without opening up the whole thing.  Is that possible?

Many thanks

[P@Anand]

I am not getting a clear picture on how you have done here is a method you could use to effectively write tha value and save it to array. The 'out of memory error' will occur whenever you try to hold a large amount of data into your dynamic memory so try to avoid that. Let me make it simple

• create an array contant before the for loop and wire it to the for loop with a shift register
• build the data into the array in one case structure and write the data into the file into the other case of the same case structure
• write to the file in regular intervals and empty the elements once they are written to the file
• once you are about to complete the for loop execution make sure that you are writing the rest values to the file
• 

Good luck

[crossrulz]

Write the array in chunks.  Something like this.

[AdamEdinburgh]

Hi everyone,

Thanks for the advice.  This is really helpful!  🙂

Adam

[mikeporter]

There is one problem with the write to file in chnks examples that you have been given, and that is that the write to spreadsheet file is inside the loop. This will work but it will be much sloweer than it needs to be. The probelm is that the VI opens the file and then recloses it with each iteration. Better to open a file outside the loop, wirite all the data to it inside the loop and then close the file when you are done.

MIke...

[altenbach]

What is "another program"? Does it require a specific format?

What is the data type of the large 2D array (U8, DBL, CDB, etc. ) Are you using 32 bit or 64 bit LabVIEW?

With large data, it is most efficient to use binary files. The reading and writing will be orders of mangitude faster and the file will be significantly smaller.