Need to improve speed when graphing large arrays

[BBend]

My app generates several arrays of 500.000 elements which I want to display in a graph. Sending this array directly to a graph makes everything go very slow, I guess because of large memory allocations (LV is obviously not very good at handling large data unless having very good insight in how LV handles memory etc). It has been suggested in other posts and in App notes to decimate the data before displaying, since the graph can't display more points than the pixel-width anyway. The question is how to implement a decimation algorithm in order to get the same graph as if no decimation was made, and to preserve resolution when zooming into the graph. which require doing a new decimation with a new subset of the data. I think this graph decimation algo should have been implemented as a VI in LV, to put between the data and the graph. Since this is inherent in Labview when trying to graph more data points than pixelwidth, it should be hard to implement such a feature in LV. I would think this is  quite common problem and would like to get some comments , especially from NI people, about this issue. Any work-arounds are also appreciated.

[shoneill]

It is correct that this CAN be a common problem.

It seems also that you already know how to solve the problem, more or less.

You should keep ONE copy of the entire array in memory and via event cases re-allocate data to the chart as required.  If zooming in to the full data set is not neccessary, then it becomes a much easier thing to do.

Is the zooming on to see the "original" data really neccessary, or just a "neat" feature?

Hoping to help

Shane

[eweltmer]

I'm having the same problem.

Using LV8.01, I initialize a 1600 X 2400 X 8 bit, 2 D array. Then I replace one "line" (1 X 2400) at a time using the "replace array subset" vi.

The application backlogs as I am using an enque/de-enque function.

I was considering using an array handling sub routine written in "C" as I am informed that "C" handles arrays more efficiently.

Welcome any comments especially from NI or anyone experienced in LV and "C".

Earl

[shoneill]

Are you sending the entire aray per Queue every time you want to replace a line?  That's very inefficient and won't run any faster in C than in LV.

Make sure you don't have any unneccessary data copies floating around as that will kill performance.

Turn on the buffer allocation tool (I don't have LV 8 myself, so I can't say exactly where the option is) and you'll see if you're working on a single array, or several copies (Which will of course slow things down considerably).

Hope this helps

Shane.

[eweltmer]

I'm only sending a 1D array (1 one) in the Que at a time.

Once the line is received it is assembled in the consumer loop using the replace array vi.

More overhead is saved by updating the IMAQ image every six lines instead of every line.

I have checked with NI about this & they tell me that LV does not handle arrays very efficiently.

Earl

[shoneill]

I find the comment that someone at NI should have said that LV handles arrays inefficiently rather surprising....

Are you sure it was someone working for NI who told you that?  Of course efficiency is one part compiler, two parts coding.  If you know how to avoid data copies, LV can be surprisingly efficient.

Shane.

[eweltmer]

Yes. I was taking a Users class (Intermediate II). I asked the instructor about the problem with array handling I had.

I showed him my VI & approach, by initializing the array first then replacing the array subset.

He said that "Labview does not handle arrays very efficiently" & the R&D group was working on the problem.

He went further to state that "C" is a more efficient language to handle arrays.

I am not touting "C" by any means but each language has it's advantages (hence Labview CVI).

You notice that no one from NI has responded to this thread.

Earl

[shoneill]

In case I haven't said it before, post some example code and we'll see what we can do.

Shane.

[eweltmer]

Here's the code but NI has looked at it & determined that it is efficient.

The NI Instructors who have personally inspected this code are:

Mark Walters
Brooke Waite
Somendra Sreedhar

The array is initialized which allocates the memory.
Instead of using the "Append Array" vi I used the "replace array subset" vi which is more efficient.

The next test I plan to use is to replace a "block" of data (2 d array subset) instead of replaceing a 1 d array at a time.

I am also sure it is not the computer speed as I have run this on a P4, 3.2 GHZ dual core with 2 GB RAM.

I have also tried to run this on an AMD 64 bit dual core, 2 GB RAM, hyperthreading technology, with 533 MHZ bus, & PCI express video.

I am running LV 8.0.1 with NI vision on W2K & WXP SP1 & SP2 (different computers).

Earl

[eweltmer]

You are probably going to need the following sub vi's as well.

Thank You,

Earl