24 bit histogramming (memory issues)

[ds_1]

Hi All,

          I'm attempting to create a 24 bit histogram of code for a 24 bit ADC. For later processing of this data I require all the bins to be stored, i.e. I have found that the Histogram VI in the functions pallette will not work for 24 bit data, I'm on LV 2010 but on 32bit WinXP, (with ~4G of RAM)

    The histogram vi runs into memory issues with my data.

  Even at a more basic level if I try init a blank array with 2^24 elements of U32 (and so 16M elements in the array) each index of the array represents an adc code and is incremented each time it is aquired. when I initialise and begin using such an array I get "out of Memory" errors. I would have thought that 16M elements was not a huge dataset and it would not be problematic to find 16M of contiguous memory with the application's 2G allocation by the OS.

    I was wondering if anyone had a suitable explaination of this or could suggest a way around this issue.

    Thanks,

     dsone1

[GerdW]

Hi DSone,

well 16M entries of U32 need 64MB of RAM - but that should be a problem on a standard PC.

It seems your histogram VI is coded rather ineficient in terms of memory usage! You could:

- attach the VI for getting helpful comments

- try to minimize data copies on your own

- read the knowledge base entriy on handling large datasets to get information on the topic

[GerdW]

Just for reference: Handling large data sets

[altenbach]

What is the highest expected count? How many bits do you need for each bin?

64MB of array data should not be a problem, but make sure to avoid to graph it all or trying to display it in an indicator. (If you want to graph the histogram, resample the data for display to a few hundred bins) None of this should be needed for "processing". Keep the array in a shift register and avoid all extra data copies (local variables, indicators, wire branches, subVIs, etc.). Use the "in place element" structure to increment bins using the value as index.

Show us some code....

[ds_1]

Hi Guys,

             Thanks for responses. I'm aware of the large datasets issues but I'm not sure why it impacts at 64M for my histogram. It does have to run in a much larger application which can consume 210M - 500M of RAM so I guess that is a big factor. I'm just wondering why it imediately struggles with this array. I

       I'm trying to gather samples, add to the histogram, aquire more and add again. Later processing can also be quiet intensive but I can worry about that later!. Dropping bin from the indicator would be less than ideal, an empty code bin on ADC suggests a missing code, its nice to be able to visually inspect for these. I could alternately search for array indices containing 0. But a visual inspection can also be a better indicator of the cause.

    I constructed my vi as re-enterant (to use as a global functional variable) perhaps this also adds memory duplication? I could try streaming to memory instead but I'll still need to open the saved array and save another copy.

       All suggestions welcome!

          Thanks

          dsone1

[GerdW]

Hi ds_1,

I removed the unneeded indicators from your VI. Now it constantly takes ~192MB (after several runs) using 3 buffers of 64MB (shiftregister, "Output" indicator, and one additional buffer shown at the InitArray function). Everything else seems fine for me...

[ds_1]

Hi GerdW,

               Thanks for your time on this, this works well for gathering my data. I've added code that will save off to TDMS after I run (n) loops building up samples. However the tdms file fails to open (out of memory) if I try to open it to process.

            I'm rewiting an INL adc linearlity vi, which never worked with 24bit data and struggled with 18bit data. It uses a sinewave histogramming method, so this futher added to the headache, as sampled data is compared with an ideal sinewave histogram (another 24bit code array) to determine the linearity.

   I may need to farm this processing out to another machine as 2 x 192MB (at best) along with the application may not be feasable.

[altenbach · Accepted solution]

I would change a couple of things:

1. Since the shift register is uninitialized, memory use will be low until the first time the VI is run in init mode, at which time the memory needs to be allocated at runtime. It is better to allocate at compile time as in my modification.
2. I would recommend a globally initialized feedback node, eliminating the while loop. It also ensures an initialized array if the user forgets to run init first. (implementing the same with a shift register would also need the "first run?" primitive).
3. I would move the init code before the histogram code, allowing to init and compute a new histogram concurrently.

Here is a rough draft

[ds_1]

Thanks guys,

                     This has been a great help. I think I can efficiently gather my histogram now. I've implemted it saving to .tdms, which should work so I can process offline or on another machine.

   I have an issue reading back this file however, perhaps I would be best opening another post as the histogramming aspect is resolved. My next 2 steps are:

1. Retrieve data efficiently from TDMS, ( I'd imagine I should be able view the histogram in "chunks" for any visual inspection) I'll start another post for this.

2. Process the data. This will be the real fun. An INL measurement using the method I require is "Sinewave Histogramming". This compares sampled data (histogram gathered here) with ideal data (a 2nd 24bit histogram) and compute linearity.

The ideal method would be to perform all this on an FPGA, which would be much quicker, although the hardware/development/testing could take a while yet!

Thanks again,

  ds_1

[SIVA_M]

Hi,

Could you share your final code to do histogram. I am also  doing 24 bit sigma delta linearity.

Thanks,

Siva