Showing results for 
Search instead for 
Did you mean: 

How to handle ever expanding array?


I'm looking for help with a temperature control experiment...

I have a program to aquire data from 24 channels through a while loop. I am displaying the data in a waveform graph. Currently, I am using shift registers and build array to add new data to the graph every time through. This worked fine while I was prototyping the hardware, but now I'd like it to run for days on end; as everyone can imagine, the array keeps getting bigger and the build array function keeps hogging memory until the whole thing crashes.

I'm guessing that I should be writing the data to a file every so often, which would keep the size manageable, and would allow me to know the size of the array and could use other opperations rather than build array. The array would need to stay pretty small so the write operation didn't slow down the loop too much. The program is pretty slow, since I'm limited in how quickly I can change temperatures. Currently, I go through all 24 probes about every 2.5 seconds (this is set just by how long it takes the routine to read each temperature and calculate the control parameters). If it takes a little bit longer each time through, that's not a huge problem, but a big pause every so often would make the control routine behave oddly.

What the save-to-file option takes away, however, is the ability to walk up to the machine in the morning and see at a glance how well it regulated the temperature during the night. I've thought about keeping some reduced set of the data (every minute or so) to give me a low-frequency glance at the data, but it seems like this will simply postpone the inevitable memory loading. I don't know if I could have a separate program that opened up the files that the control routine wrote, but this would probably be too much for the poor old PII machine that is running this.

Does anyone have a simple solution to this probelm? Failing that, a complex solution might do, although it might get beyond my programming abilities.

I only have LV 6.0, so I can't see examples saved for versions later than this.

Thanks for any suggestions.

0 Kudos
Message 1 of 3
You could preallocate the array for some large number of datapoints, and replace each array element as a new datapoint is collected.  Depending on how you set it up, this might fail if the data collection went longer than expected, and you went past the end of the array.

Alternately, write each data point to the file as it is collected, and display the progress on a chart instead of a graph.  You eliminate the need for arrays altogether that way.  (Except for the chart, of course, which is taken care of for you behind the scenes somehow.)

0 Kudos
Message 2 of 3

Writing data to an array every so often is definitely what you want to do. Then if power fails or the program shuts down for some reason, the data to the last save is still in the file. At the 2.5 second sample rate you only get ~35000 points per channel per day, so data files will not get too large. Decide how often to write by how valuable the data is: What is the penalty for losing data for the last minute, hour, day...?

Another point is that the panel display only occupies a few hundred pixels in each direction. Plotting more points than that is meaningless. If you plot one point per minute you will have 1440 points per day (per channel). You could always show the most recent 24 hours using a circular buffer. With more programming effort, you could keep previous days plots available for selection by the user or read them back from the file.

0 Kudos
Message 3 of 3