LabVIEW
PNRF and LabVIEW memory limitations
Solved!
I've written a VI that reads a PNRF data file (using LabVIEW's PNRF data plugin) and derives the channel data, channel names, and sampling rate. However, I'm limited to reading files that have no more than 8 channels with 2,370,000 readings in each channel. Anything more and LabVIEW gives me an out-of-memory error on a Windows 7 Pro 32 bit computer that has 4 GB of RAM. Using task manager, LabVIEW errors out with a memory usage at 1,063,216 KB.
Is there a better way to read in larger PNRF files?
The VI looks like this:
Thanks,
Ron
Well 1GB of memory use is pretty large for a 32Bit system. LabVIEW by default only gets 2GB of your 4GB at most since the rest is used by Windows. So it will have to error out at some point. Depending on how large your data chunks are this can sometimes go up to 1.3 GB or so but then it definitely stops. Since you want to read in 8 channels and each channel waveform requires a contingious chunk of memory, you acquire in fact 8 chunks of 2'370'000 * 8 Bytes. As soon as you start to put this in your "converted data" array you tell LabVIEW to allocate another chunk of 8 * 8 * 2'370'000 Bytes. And once you do something with that data LabVIEW will repeatedly have to create copies of that data.
The right way to process such large files is not to read them in all at once but instead in smaller chunks. I have no idea if the PRNF Toolkit has an advanced Function palette that supports reading in single channels or a repeatedly smaller timespans of the data though.
Hello Mavis,
I created a read-by-chunk version. Hopefully, I wrote it correctly. The VM memory and Mem usage (as reported by Windows task manager) is about the same as when the VI that doesn't read each data channel individually. In both VIs, I have added the decimation function to reduce that amount of data processed, which has helped in reading larger files. I will be testing the limits with both VIs in the next few days.
Thanks,
Ron
Update: I improved the time (to 1/4) and memory ( to 1/2) parameters by changing the "output data channel" property of "Read Data [Channel]" function to "array of waveforms".
@Mavis wrote:
Hi Ron,
Glad to hear that your time and memory usage were improved a lot with array of waveforms!
The index/count inputs of Read Data mean the start index and number of values to read from all channels. You can refer to Help for detail.
But it might not help in your case, for all values will be concatenated later.
Best Regards,
Mavis
Well it would help if you do the decimation in the loop that reads the chunks for each channel. As it is now, the VI reads in all the data as a block for each channel and then decimates them afterwards. Doing the same in two loops, one for the channels and the other for individual smaller blocks per channel would certainly reduce the maximum required memory considerably.
