Optimizing queue/for loop

[dhlNILABVIEW]

I built a subVI for creating an en face projection of a 3D image. This is basically done by taking the max signal of each 2D slice and concatenating that with each other slice to create a new array. This new array is then reshaped to match the dimensions of the surface being imaged. I am having trouble getting it to run efficently, though. While I am processing hundreds of thousands of voxels, it seems to take an excess amount of time to construct the new image. Can anyone point me in the right direction to optomize or diagnose the problem? Also, the data must be queued in due to how the imaging system is set up. 

[nathand]

Your VI doesn't make any sense. Why have an outer For loop that only executes once? What's the point of the queue, when you dequeue exactly the same data immediately? Just use a wire. Also, any time you obtain a queue, you should also release it. In general you should obtain a queue outside a loop, use it inside the loop, then release it after the loop finishes - although in your case, since your for loop only executes once, it doesn't matter. The VI is also broken due to the unused Index Array inside one of the interior For loops. Can you post a better example of what you're trying to do?

[dhlNILABVIEW]

Sorry, I just drew this code up really quickly. The single for loop is not necessary. 

I am using a queue since, apparently, "single element queue is a way to pass data by ref and avoid making copies" (https://lavag.org/topic/12676-performance-of-queue-of-large-arrays/). In the link, they are using a DVR, though, of which I am not quite sure how it operates. The input into this queue is actually a continuous stream of 2D arrays that map out a 2D slice of what I am imaging. The input to the queue is constantly being overwritten by new 2D slices. I am trying to find the max values of each column of these 2D arrays and log that information into a stream of 1D arrays. This 1D array can then be reshaped into a 2D array corresponding to the width and depth of the surface of the object that I am imaging. I hoped this helped explain it a bit. I don't think uploading the actual code will be very helpful since it is upwards of 50 MB and requires imaging hardware to operate. 

[crossrulz]

This sounds more like an application for a Producer/Consumer.

[nathand]

If you're not sure why you're using a particular structure, then you probably should not be using that structure. You don't have a single-element queue here (you aren't setting the queue size), and even if you did, is the only reason you're using it because you read somewhere that it "avoids making copies"? Do you need to pass the data between loops? If you're acquiring and processing the data in the same loop, you can probably lose the queue without making unnecessary copies.

Is your actual code 50mb? That suggests that either you've saved a large image or array as default data in a control or indicator, or your code is much more complicated than necessary (50mb of VIs is a lot). Can you strip out just the VI that does the work you're trying to optimize? Even if we can't run your VI without the right hardware, we can learn a lot from seeing the real code. An overly-simplified, confusing, and broken VI isn't helpful.

What's the source of the data? You can't "overwrite" the input to a queue; you can only enqueue more elements - if there's space in the queue for them. However, if you are actually using a single-element queue, and you can't process the data as fast as it arrives, then you'll discard most of the data since there won't be room in the queue for it (again, single-element).

Do you know the eventual size of the 2D output array ahead of time? If so, probably best to allocate that in advance. Then, as each new 2D input array arrives, get the maximum for each column and replace the existing element in the 2D output array with that new value.