I am collecting a large amount of data from a PXI system, and I am stuck on the final, seemingly most simple step of organizing and displaying the data. I use a RT FIFO variable (20 x 10000 array in this case, possibly up to 20-100 x 100000) to store data that is generated at 20 Hz from a DAQ card. I grab this data at 1 Hz using a script on my computer, then store it in a binary file. Up to here, everything works well. I compiled the script that grabs data separately to check, and there is no memory leakage as it runs.
Periodically, I need to actually look at this data, so I wrote a simple script to parse the binary file and display it in a graph. However, this script is a memory management disaster. It quickly blows up to gigabytes. I compiled it separately as well. Running a ~200 MB binary file through it causes it to balloon to almost 3 GB and then, most worryingly, settle at around 900 MB even with no graph display (I removed the graph to make sure that this wasn't the sole reason for the poor performance of the script). This may just be one copy of the dataset--I'm not sure exactly how much compression to expect.
Where is data being stored here? I thought that using for loops, array functions, and shift registers like this was the correct way to handle large datasets in Labview. Shouldn't this memory all be released once the VI finishes running, given that I do not display it anywhere? I have to kill Labview to remove the VI from memory, or kill the compiled program.
My next step will be to try to write a version that goes through a very large file piece-by-piece to create a summed graph like the one I make here, but I am not sure how to go about this if the structures I am using now are storing data.
I'm attaching the VI, the EXE, and a sample data file as examples. The program and the sample file are in the zip.
