spreadsheet to array memory problems

[johnsold]

If you open the Read from Spreadsheet File.vi and look at the block diagram, you can see that it does make copies. First it reads the entire file as a string and then converts it to a 2D array of DBL.

Your memory use seems a bit large, but not out of the question.  I ran into the same problem when needing to read files of 140 MB and larger. I was seeing 600-800 MB of memory allocation.  What I did was to read the file in modest segments, convert those strings, and use Replace Array Subset on an array that had been Initialized to the size of the final array.  With this method my memory use was about 20-25% higher than the size of the final array.

Lynn

[belgron]

Thanks for the response, Lynn.

That's roughly the path I'm taking and usage is lower. I'm left with the memory deallocation quandary, however. Even though usage is smaller, it's still higher than I'd like, and I can't find a method to reclaim memory that has been allocated to string arrays. It seems to work fine with doubles. If anyone has any thoughts on what might be causing this problem, I would really appreciate hearing them,

thanks!

[BTC_Admin]

If you have some liberty in design, and if the top data row you're trying to replace is equal or smaller in size compared to existing top data row, there is one more thing you could try that may work much better:

1. open file for read/write, set pointer at the beginning

2. Write your 1st data row as a string (insert delimiters when forming one string from your Headers array, end with EOL character). If your new data is smaller, you could append spaces AFTER EOL character to ensure that it still works. This step will replace your top data row without using much/any memory (since you're doing disk read/write when you flush)

3. set file pointer at beginning of last line in the file (will require little bit logic using file functions)

5. Write EOF, flush/save file and close file ref.

Once done with above steps, simply read file & parse it into your array of strings. And if needed, you could do such parsing in chunks as previous poster mentioned.

In above method you will not use so much memory as you're taking care of data replacement before reading the entire file into memory for creating string array.

-DP

BatchTest Corp.

NI Alliance Partner