thread function + open menu = crash!

[ElectroLund]

I'm using CMT library and have a serial port data-intensive function that runs in the default thread pool. This function updates a progress bar in my main panel, as well as prints out live data collection to a little string control. The function runs for 1+ minute. This workflow has been stable for quite a while, but one of my users discovered a critical bug. If they have a context menu popped up (which I do with RunPopupMenu on the EVENT_RIGHT_CLICK event of a table control) *before* this CMT function begins, then the function grinds through until near the end but never returns. If I stop the debugger, we're stuck at CmtWaitForThreadPoolFunctionCompletion. I can't step at all, it's just stuck here. Any ideas for next step? I suppose one approach would be to force close the popped up menu? Although I'm not sure that can be done programmatically. I'd rather understand why this menu being open locks up the thread function.

[ElectroLund]

I've got more details now. I've found that if I have any menu open while a new threaded function is running, the CmtWaitForThreadPoolFunctionCompletion will hang until that menu is closed. I have created a sample project which demonstrates this behavior. It's repeatable. Just click the Start button, then open any of the menus (which have no callbacks).

[JoseTorre]

I think the problem is with "DEFAULT_THREAD_POOL_HANDLE" ) If you create your own pool for your threads you won't have that problem. ( CmtNewThreadPool )

[ElectroLund]

Ok, I've added an option to use either the DEFAULT or a new thread pool.  Fails in either case if menu is left open.  I've also added a MessagePopup after the CmtWaitForThreadPoolFunctionCompletion to make it more clear on pass/fail.

Strange, huh?  Why do the menus behave like this?  It's as if they lock all threads while open?

A couple workarounds that I can think of are:

• Dim all menubars that might be opened during these CMT functions. However! There is still the possibility of a menu being left open when the CMT function begins.
• Programmatically select some other control when CMT function begins.  Tried this, but that's not the same thing as a mouse click, i.e., the other control is made active, but the menu remains open.
• Close the menu? I can't find a way to programmatically close a menu once it's open.

[JoseTorre]

The problem is that the events and the sequence of the program is supposed to be in the main thread, when the callback is called in that main thread, that thread keeps running until the threads have not been released, at the same time the interface in that same thread is waiting for events from the menu, hence the blocking. I don't know if I'm explaining myself correctly, I'm attaching a version trying to make a variation of your example taking the end-of-thread functions out of the main thread to try to explain it. i don't let me to upload the code. I write the code below: (Simple.c) /*****************************************************************************/ /* Simple example illustrating usage of CVI multi-threading functions */ /*****************************************************************************/ #include #include #include #include "Simple.h" static int gPanel = 0; /* panel handle */ static int gExiting = 0; /* flag to indicate user is exiting */ static int CVICALLBACK ThreadFunction1 (void *functionData); static int CVICALLBACK ThreadFunction2 (void *functionData); static int CVICALLBACK ThreadFunction3 (void *functionData); int wait(double p_delay); int threadPoolID; int threadFunctionId1 = 0; int threadFunctionId2 = 0; int threadFunctionId3 = 0; int close = 0; int flagthreadFunctionId1 = 0; int flagthreadFunctionId2 = 0; int main (int argc, char *argv[]) { if (InitCVIRTE (0, argv, 0) == 0) return -1; /* out of memory */ if ((gPanel = LoadPanel (0, "Simple.uir", PANEL)) < 0) return -1; CmtNewThreadPool(4, &threadPoolID); CmtScheduleThreadPoolFunction (threadPoolID, ThreadFunction3, NULL, &threadFunctionId3); DisplayPanel (gPanel); RunUserInterface (); DiscardPanel (gPanel); CmtDiscardThreadPool(threadPoolID); return 0; } /* First thread function */ static int CVICALLBACK ThreadFunction1 (void *functionData) { for (int i=0; i<50; i++) { SetCtrlVal (gPanel, PANEL_NUM1, i++); wait (0.5); } flagthreadFunctionId1 = 0; return 0; } /* Second thread function */ static int CVICALLBACK ThreadFunction2 (void *functionData) { for (int i=0; i<1000; i++) { SetCtrlVal (gPanel, PANEL_NUM2, i++); wait ( 0.001); } flagthreadFunctionId2 = 0; return 0; } int CVICALLBACK PanelCallback (int panel, int event, void *callbackData, int eventData1, int eventData2) { switch (event) { case EVENT_CLOSE: gExiting = 1; if (threadFunctionId3) { close = 1; CmtWaitForThreadPoolFunctionCompletion (threadPoolID, threadFunctionId3, OPT_TP_PROCESS_EVENTS_WHILE_WAITING); DebugPrintf("threadFunctionId3 completed! \n"); MessagePopup("Done!", "threadFunctionId3 completed! \n"); CmtReleaseThreadPoolFunctionID (threadPoolID, threadFunctionId3); threadFunctionId3 = 0; } QuitUserInterface (0); break; } return 0; } int CVICALLBACK StartThreads (int panel, int control, int event, void *callbackData, int eventData1, int eventData2) { int checkbox; switch (event) { case EVENT_COMMIT: if (gExiting == 0) { gExiting = 1; flagthreadFunctionId1 = 1; /* Schedule two thread functions */ CmtScheduleThreadPoolFunction (threadPoolID, ThreadFunction1, NULL, &threadFunctionId1); flagthreadFunctionId2 = 1; CmtScheduleThreadPoolFunction (threadPoolID, ThreadFunction2, NULL, &threadFunctionId2); } break; } return 0; } static int CVICALLBACK ThreadFunction3 (void *functionData) { while(close == 0) { if (threadFunctionId1 && flagthreadFunctionId1 == 0) { CmtWaitForThreadPoolFunctionCompletion (threadPoolID, threadFunctionId1, OPT_TP_PROCESS_EVENTS_WHILE_WAITING); DebugPrintf("threadFunctionId1 completed! \n"); MessagePopup("Done!", "threadFunctionId1 completed! \n"); CmtReleaseThreadPoolFunctionID (threadPoolID, threadFunctionId1); threadFunctionId1 = 0; } if (threadFunctionId2&& flagthreadFunctionId2 == 0) { CmtWaitForThreadPoolFunctionCompletion (threadPoolID, threadFunctionId2, OPT_TP_PROCESS_EVENTS_WHILE_WAITING); DebugPrintf("threadFunctionId2 completed! \n"); MessagePopup("Done!", "threadFunctionId2 completed! \n"); CmtReleaseThreadPoolFunctionID (threadPoolID, threadFunctionId2); threadFunctionId2 = 0; } if (threadFunctionId1 == 0 && threadFunctionId2 == 0) gExiting = 0; wait ( 0.5); ProcessSystemEvents(); ProcessDrawEvents(); } return 0; } int wait(double p_delay) { double time1 = Timer(); double time2 = Timer(); while (time2 - time1 < p_delay) { ProcessDrawEvents(); ProcessSystemEvents(); time2 = Timer(); } return 0; }

[rolfk]

If you click on the ellipsis (three dots at the end of the toolbar above your edit window) you get additiional icons and can select the one with the < / > on it to get a code display box. There you can add your source code, select the format to be C code and the forum software will not remove line breaks and even do a bit of syntax highlighting.

[ElectroLund]

I put Jose's code through a formatter.  Thanks, Jose!  I haven't tested this yet, but you've given me a lot to chew on.  I may have some gaps in my understanding of how the multi-threading system works.

/*****************************************************************************/
/* Simple example illustrating usage of CVI multi-threading functions */ 
/*****************************************************************************/

#include "Simple.h"

static int gPanel = 0; /* panel handle */
static int gExiting = 0; /* flag to indicate user is exiting */
static int CVICALLBACK ThreadFunction1(void * functionData);
static int CVICALLBACK ThreadFunction2(void * functionData);
static int CVICALLBACK ThreadFunction3(void * functionData);
int wait(double p_delay);
int threadPoolID;
int threadFunctionId1 = 0;
int threadFunctionId2 = 0;
int threadFunctionId3 = 0;
int close = 0;
int flagthreadFunctionId1 = 0;
int flagthreadFunctionId2 = 0;

int main(int argc, char * argv[]) 
{
	if (InitCVIRTE(0, argv, 0) == 0) 
		return -1; /* out of memory */

	if ((gPanel = LoadPanel(0, "Simple.uir", PANEL)) < 0) 
		return -1;

	CmtNewThreadPool(4, & threadPoolID);
	CmtScheduleThreadPoolFunction(threadPoolID, ThreadFunction3, NULL, & threadFunctionId3);

	DisplayPanel(gPanel);
	RunUserInterface();
	
	DiscardPanel(gPanel);
	CmtDiscardThreadPool(threadPoolID);
	
	return 0;
} /* First thread function */

static int CVICALLBACK ThreadFunction1(void * functionData) 
{
   for (int i = 0; i < 50; i++) 
   {
      SetCtrlVal(gPanel, PANEL_NUM1, i++);
      wait(0.5);
   }
   
   flagthreadFunctionId1 = 0;
   return 0;
} /* Second thread function */

static int CVICALLBACK ThreadFunction2(void * functionData)
{
   for (int i = 0; i < 1000; i++)
   {
      SetCtrlVal(gPanel, PANEL_NUM2, i++);
      wait(0.001);
   }
   
   flagthreadFunctionId2 = 0;
   return 0;
}

int CVICALLBACK PanelCallback(int panel, int event, void * callbackData, int eventData1, int eventData2) 
{
   switch (event) 
   {
	   case EVENT_CLOSE:
			gExiting = 1;
			if (threadFunctionId3) 
			{
				close = 1;
				CmtWaitForThreadPoolFunctionCompletion(threadPoolID, threadFunctionId3, OPT_TP_PROCESS_EVENTS_WHILE_WAITING);

				DebugPrintf("threadFunctionId3 completed! \n");
				MessagePopup("Done!", "threadFunctionId3 completed! \n");
				CmtReleaseThreadPoolFunctionID(threadPoolID, threadFunctionId3);

				threadFunctionId3 = 0;
			}

			QuitUserInterface(0);
			break;
   }
   return 0;
}

int CVICALLBACK StartThreads(int panel, int control, int event, void * callbackData, int eventData1, int eventData2) 
{
   int checkbox;
   
   switch (event) 
   {
	   case EVENT_COMMIT:
		  if (gExiting == 0) 
		  {
			 gExiting = 1;
			 
			 flagthreadFunctionId1 = 1; /* Schedule two thread functions */
			 CmtScheduleThreadPoolFunction(threadPoolID, ThreadFunction1, NULL, & threadFunctionId1);
			 
			 flagthreadFunctionId2 = 1;
			 CmtScheduleThreadPoolFunction(threadPoolID, ThreadFunction2, NULL, & threadFunctionId2);
		  }
      break;
   }
   return 0;
}

static int CVICALLBACK ThreadFunction3(void * functionData) 
{
   while (close == 0) 
   {
      if (threadFunctionId1 && flagthreadFunctionId1 == 0) 
	  {
         CmtWaitForThreadPoolFunctionCompletion(threadPoolID, threadFunctionId1, OPT_TP_PROCESS_EVENTS_WHILE_WAITING);
		 
         DebugPrintf("threadFunctionId1 completed! \n");
         MessagePopup("Done!", "threadFunctionId1 completed! \n");
		 
         CmtReleaseThreadPoolFunctionID(threadPoolID, threadFunctionId1);
         threadFunctionId1 = 0;
      }
      if (threadFunctionId2 && flagthreadFunctionId2 == 0) 
	  {
         CmtWaitForThreadPoolFunctionCompletion(threadPoolID, threadFunctionId2, OPT_TP_PROCESS_EVENTS_WHILE_WAITING);
		 
         DebugPrintf("threadFunctionId2 completed! \n");
         MessagePopup("Done!", "threadFunctionId2 completed! \n");
		 
         CmtReleaseThreadPoolFunctionID(threadPoolID, threadFunctionId2);
         threadFunctionId2 = 0;
      }
      if (threadFunctionId1 == 0 && threadFunctionId2 == 0) 
		  gExiting = 0;
	  
      wait(0.5);
      ProcessSystemEvents();
      ProcessDrawEvents();
   }
   return 0;
}

int wait(double p_delay)
{
   double time1 = Timer();
   double time2 = Timer();
   
   while (time2 - time1 < p_delay) 
   {
      ProcessDrawEvents();
      ProcessSystemEvents();
      time2 = Timer();
   }
   return 0;
}

[RobertoBozzolo]

In my opinion this problem is not related to multithreading, I mean that it is evidenced by your multithreaded approach but is not caused by it.

First of all, undergoing threads are executed regularly even with the menu shown, as evidenced by the counters. The problem arises in CmtWaitForThreadPoolFunctionCompletion but is not related to this function: a well known characteristic of CVI is that menus stop event processing when shown, and this is what prevents StartThreads function to terminate. (Even UI timers are blocked by menus, and this is definitely a characteristic to keep in mind when coding!)

A way to overcome this problem is to use CmtScheduleThreadPoolFunctionAdv to start your threads specifying a function to execute at thread end to do whatever you may need to execute like clean up the process, save data and so on.

The code can look like the following:

/* Schedule two thread functions */
//	CmtScheduleThreadPoolFunction (threadPoolID,
//		ThreadFunction1, NULL,
//		threadFunctionId1);
CmtScheduleThreadPoolFunctionAdv (threadPoolID, ThreadFunction1, NULL, DEFAULT_THREAD_PRIORITY,
		cbkFunction, EVENT_TP_THREAD_FUNCTION_END, (void *)1, RUN_IN_SCHEDULED_THREAD,
		&threadFunctionId1);
//	CmtScheduleThreadPoolFunction (threadPoolID,
//		ThreadFunction2, NULL,
//		&threadFunctionId2);
CmtScheduleThreadPoolFunctionAdv (threadPoolID, ThreadFunction2, NULL, DEFAULT_THREAD_PRIORITY,
		cbkFunction, EVENT_TP_THREAD_FUNCTION_END, (void *)2, RUN_IN_SCHEDULED_THREAD,
		&threadFunctionId2);

together with the function to complete at thread end:

void CVICALLBACK cbkFunction (CmtThreadPoolHandle poolHandle, CmtThreadFunctionID functionID, unsigned int event, int value, void *callbackData)
{
	DebugPrintf ("Thread %d terminated.\n", (int)callbackData);
	return;
}

This is the output of the program (I added a debugPrintf in thread functions to keep track of program activity: function name, Timer() and iteration #):

ThreadFunction2 8.89: iteration 988
ThreadFunction2 8.90: iteration 990
ThreadFunction2 8.93: iteration 992
ThreadFunction2 8.94: iteration 994
ThreadFunction1 8.96: iteration 34
ThreadFunction2 8.98: iteration 996
ThreadFunction2 8.99: iteration 998
Thread 2 terminated.
ThreadFunction1 9.47: iteration 36
ThreadFunction1 9.98: iteration 38
ThreadFunction1 10.50: iteration 40
ThreadFunction1 11.01: iteration 42
ThreadFunction1 11.52: iteration 44
ThreadFunction1 12.04: iteration 46
ThreadFunction1 12.55: iteration 48
Thread 1 terminated.
threadFunctionId2 completed! 
threadFunctionId1 completed! 

[ElectroLund]

This is fascinating. Jose's code will in face fully run regardless of menu's being drawn. In face, the termination of the thread functions actually proceed to the MessagePopup, which is impressive. Roberto's also finishes the thread functions, but then get hung up at the Wait and following MessagePopups, until I remove the menu manually. I'm not clear about the importance of where I place these various CMT functions. Jose's solution seems to be that I should place my CmtWaitForThreadPoolFunctionCompletion itself inside another threaded function. This makes the calling of these very cumbersome, given that mine are wrapped up in general purpose libraries.

[RobertoBozzolo]

I was under the impression that MessagePopup was put there for demonstrative purposes only, so I didn't pay attention to it and did the minimum code modifications to demonstrate how to address the issue.

Simply moving the MessagePopup call into cbkFunction permits the message to be shown even when the menu is open.

I am attaching a modified version of the sample program with all modifications needed to have a clean solution.