LabVIEW spawning new threads with every call to DLL via CLN, but same DLL works fine being called from CVI

[acerenzio]

I have been hunting down an issue with a call library node (CLN) for several weeks now and I'm hoping someone can offer some guidance. I have a 3rd party c++ DLL we'll call "Camera.dll". It has two functions that I'm using:

1. Configure 

2. GrabFrameAverage

What I want to do is call the GrabFrameAverage function in a loop in LabVIEW. The function basically just returns pixel data from a camera. I can return data fine for about 50 loops and then LabVIEW will freeze and crash. After many days of troubleshooting I've been able to identify that with each call to this function I see ~5-8 new threads spawn when I watch the Windows Resource Monitor. The crash seems to happen when the thread count hits around 600.

As one troubleshooting step, I tried to see if I could call the function from TestStand. When I do I get the same behavior where it will work fine for several loops and then crash, all while increasing thread count. 

I was ready to fully blame the DLL (and haven't ruled that out yet), but when I run this same test in CVI I can run it indefinitely and I see no new threads spawn which has me questioning if the DLL is really to blame.

Some additional troubleshooting info that has all led to the same results:

1. In LabVIEW
   1. Tried slowing the loop down by manually stepping through
   2. Changing thread from to "any" instead of UI (then returned to UI thread for all additional attempts)
   3. Tried both calling conventions (the import wizard defaults to C)
   4. Wrote a CVI wrapper around the 3rd party DLL and called the wrapper from LabVIEW
   5. Instead of initializing a new array each loop I have tried doing an initialization outside the loop and then making the array into a local variable that I can pass in and and then use as the output. The thought here was that maybe somehow the threads were related to creating 'new' arrays each time.
2. In TestStand
   1. Slowed loop down
   2. Changed adapter settings between CVI and C/C++
   3. Called DLL directly and using CVI wrapper

• Is there any way that I could be messing up a the CLN and that be what's causing the threads?
• Is there a way to manually close out the threads? 
• Could it be related to the way I'm creating the Array Data Pointers?
• Do I need to "free" the Array Data Pointers somehow?

I'm kind of at a loss at the moment so any help or guidance would be appreciated.

This is all using TS/LV/CVI 2019. 

[acerenzio]

One additional detail that may be helpful:

I cannot make the threads go away except for fully closing out LabVIEW. I somewhat expected that when I "unloaded" the DLL by wiring an empty path to the same CLN that first calls the DLL that the threads would end if they were being held by the DLL.

[rolfk]

LabVIEW's multithreading is static. That means it allocates a number of threads at startup and keeps them in a thread pool to use but it won't create new threads afterwards. So those additional threads definitely and positively are allocated by your DLL for some reason. How that happens without knowing more about the DLL in question is just as simple as reading the weather forecast for next year in your crystal ball.

Something, somewhere in your DLL allocates those threads for some strange reason. You will need to contact the manufacturer of that DLL about what can cause it to allocate new threads on every Grab call.

[acerenzio]

Thank you for the reply.

Do you have any theories on why CVI would be able to avoid this issue? I'm definitely on board with accepting this as a "feature" of the DLL, I just can't reconcile how LabVIEW / TestStand display the symptom, but CVI seems to be free of it. 

I will probably post this to CVI board as well to see if anyone over there can offer some additional insight.

[rolfk]

Well, LabVIEW is a bit special in terms of multithreading. It does this automatically unless you specifically disable it by forcing everything into the UI thread. In C you have to explicitly create threads and run your code in them to have multithreading. TestStand is a bit in bitween. It uses under the hood ActiveX/COM for a lot of its workings and that is a bit of a schimare. It theoretically allows multithreading but requires some extra work to do so, so it may very well be that TestStand executes its test steps all in one single threaded engine, or it may not.

My suspicion is that the DLL detects if you call the Grab function from a different thread than the Configure function and then does some extra stuff. It may also spawn a single (or a few threads) in the background in CVI to monitor the camera but since your program is calling the DLL always from the same thread its internal threading management doesn't get confused. In LabVIEW if the Grab and/or Configure function are not called always from the same thread, it my somehow detect that its own threads haven't been allocated yet and spawn a new one. Or maybe even worse, no Thread Local Storage at all. Just a simple GetThreadID() and compare it with a globally stored ID and if it is not the same, assume that we haven't been initialized and allocate an entire new threading environment and store the new ID in that global storage. On next call rinse and repeat!!

My guess is that they do a bad mixture of Thread Local Storage and Global data storage for this management information and with a single threaded app it doesn't matter but with multiple threads it completely messes up.