memory issues when saving Excel

[labview_beginner]

All

I recently wrote a small piece of software which is collecting data at a very high frequency and for a long time. This is the first time I am attempting to do something like this and have hit an issue with memory not being enough as the size of the Excel file increased.

Excel has a limit of about 65000 rows, but I have written only ~10000 rows when the following error showed up:

I can briefly tell you how the data acquisition loop works:

1) timed loop has a period of 1 minute.

2) there is a sequence structure in the loop.

3) first - a sub vi goes to agilent equipment and collects all the data

4) second - a sub vi processes all the data into individual arrays in preparation for writing them to the excel file

5) third - a sub vi writes all the arrays into a single Excel file (which is already open) and saves the file

6) then the timed loop checks if the condition is met for termination of the loop. If not, then it repeats steps 3, 4, 5 above.

Possible issues that may be causing the problem:

1. The period of 1 minute may not be enough. I checked the size of the excel file, and it is right now 23 MB at ~10000 rows. I intend to collect about 20000 rows of data over 250 hours. Now what will happen if 1 minute is not enough to save the Excel file of size 23 MB or above? I am not familiar with timed loops, but can somebody explain will I get an error similar to above if that happens?
2. How can I create a functionality where the timed loop "waits"  till the Excel file is saved? I basically want the timed loop to keep counting (not stop time) but wait till file is completely saved. In that case the period of timed loop cannot be fixed to 1 minute.  Because as the size of the file increases that period has to increase too.

Appreciate all the help.

Thanks in advance.

[richjoh]

Instead of writing to Excel try a regular txt file. Excel does have its limits for scientific engineering...2^16 rows and 2^8 columns, see Excel specs and sometimes due to memory limitations of your hardware....

[labview_beginner]

Like I said in my email, I am only at about 10000 rows and the limit per worksheet is 65000. That is still way off.

And file size is only 23MB.

Can somebody help me understand the scenario which I outlined in my post? What happens if Excel is not done saving (which means save_EXcel vi is still working) and the next write to Excel command is invoked? Basically the 1 minute period of the timed loop is not enough for "collecting date, processing date, adding the newly collected data to 5 strip charts, writing to excel and then saving 23 MB file", then repeat all of it. That means even before saving operation is complete the next data may be in queue to be dunmped into Excel.  Am not able to understand will I get an exception like the one I have mentioned in mypost.

Thanks.

[xseadog]

Hi looking at your first graph , you appear to have been running for 52 hrs.How often do you save to disk?

If you are not utilising a buffer then you memory usage will increase with time!

What version of software are you running? Is it Execl 2003 or 2007?

If you look at you vi properties you can see how much memoy is being used on you program.

We can postulate ideas theories etc but you can help yourself by showing your source code. More help will then be forthcoming.

xseadog

[labview_beginner]

Thanks. 

I am attaching the main vi of the application where the main timed loop is located. Hopefully this will help.

Could you please elaborate about buffers?

About the data acquisition freqency : I am acquiring data every minute  - and the intentiion is to do it for 250 hours at least.

Thanks. 

Message Edited by labview_beginner on 10-31-2008 09:22 PM

Message Edited by labview_beginner on 10-31-2008 09:24 PM

[smercurio_fc]

Error 2147024882 is a generic error that has many causes. One of the more common causes had to do with ADO, and you would get it when accessing databases. It was also caused by MDAC. It's also possible it's caused by a memory leak, and given the mess that your code is, I wouldn't be surprised in the least if that's the problem. You did not include any of the subVIs, though I don't really want to look at them if they're anything like your main code. The first thing I would check is to make sure you're closing all the references. I think, though that simply writing to a text file, as suggested earlier, will be a lot easier.

[labview_beginner]

Thanks for your inputs. I agree that the code is not very clean. But this the best I could do in whatever time I had to write the code.

Will try to clean it up and will also check whether I am closing all the references. Any advice on checking on memory leaks?

Thanks again.

[labview_beginner]

Seems like there are some seriour memory issues in the application.

I was just checking the task manager and when the error occured Labview was taking up about 255 MB of memory. So I closed Labview, restarted my application to see how much memory usage Labview showed at start - it is only about 27 MB. And I was tracking each loop of data acquisition, excel write and save and update strip chart adds about 400 kb to the memory usage.

Can somebody give me pointers to checking memory leaks?

Thanks in advance.

[smercurio_fc]

I would suggest starting with the built-in memory profiler under Tools->Profile->Performance and Memory. Run this VI before you run your main VI and it will show you memory statistics for all your VIs.