Ring control lockup

[gdargaud]

Hi all,

I have a (large complex)  CVI linux program that sometimes lockup while opening large ring selectors (large being defined by a hundred or so items, more than the vertical screen can handle but not insanely so). It fairly often locks up at this stage and needs to be killed.

Reading about this issue, I've seen that it may happen if InstallPopup is called in the same thread. Is it still the case as the message is from 2006 ? There are many conditions in my prog when InstallPopup (or 2nd instances of RunUserInterface) are called by timers or external conditions, but I don't think it is the case here.

I wrote the following test case, but I doesn't exhibit the problem. Anything else I can check ? Any way to know if a ring is currently expanded before calling InstallPopup ? And NO, I do NOT want to multithread my program, it's already nightmarish enough !

#include <ansi_c.h>
#include <userint.h>
#include <cvirte.h>

static int Led=0;

int CVICALLBACK cb_Ring(int panel, int control, int event,
		void *callbackData, int eventData1, int eventData2) {
	int Val;
	switch (event) {
		case EVENT_COMMIT:
			GetCtrlVal(panel, control, &Val);
			if (Val==0) // Select 0 to quit
				QuitUserInterface (0);
			break;
	}
	return 0;
}

int CVICALLBACK cbt_Timer(int panel, int control, int event, 
		void *callbackData, int eventData1, int eventData2) { 
	static int Val=0;
	switch (event) {
		case EVENT_TIMER_TICK:
			SetCtrlVal(panel, Led, Val++&1);
			if (Val%10==0) ConfirmPopup ("Crash?", "Do you think this will lock up?");
			break;
	}
	return 0;
}

static void InsertLabels(int Panel, int Ctrl, int Nb) {
	char Label[10];
	for (int i=0; i<Nb; i++) {
		sprintf(Label, "-%03d-", i);
		InsertListItem (Panel, Ctrl, -1, Label, i);
	}
}

int main (int argc, char *argv[]) {
	if (InitCVIRTE (0, argv, 0) == 0) return -1;
	int Pnl  = NewPanel(0, "Test Ring", 20, 20, 100, 100);

	int Tmr  = NewCtrl (Pnl, CTRL_TIMER, "Timer", 50, 50);
	SetCtrlAttribute (Pnl, Tmr, ATTR_CALLBACK_FUNCTION_POINTER, cbt_Timer);
	
	Led = NewCtrl (Pnl, CTRL_ROUND_LED_LS, "Led", 50, 2);

	int Ring = NewCtrl (Pnl, CTRL_RING_LS, "Ring", 20, 2);
	SetCtrlAttribute (Pnl, Ring, ATTR_CALLBACK_FUNCTION_POINTER, cb_Ring);
	InsertLabels(Pnl, Ring, 200);
	
	DisplayPanel (Pnl);
	return RunUserInterface ();
}

[gdargaud]

Replying to myself here. The above code does indeed lockup sometimes (both on Windows and Linux). Racking my two brain cells left, I remembered writing something related in the '90s:

           They are safe in respect with timers. If a popup is called while there
           are asynchronous calls, you can't close it until the timers get closed.
           With those calls, the timers are paused while the popup is open.
           This assumes that either the timers are less important than the popups,
           or that the popups relate to error conditions and that the process 
           shouldn't keep on going anyway.


int SafeMessagePopup(char *popupTitle, char *messageString) {
    int i;
    SuspendTimerCallbacks();
    i=MessagePopup(popupTitle, messageString);
    ResumeTimerCallbacks();
    return i;
}

[LuisG]

I tried for quite a bit, trynig as many possible interactions with the ring popup as I could think of just before the confirm popup, but I was unable to reproduce the hang. And for what it's worth, I don't see any obvious cause for a hang.

Your comment above your SafeMessagePopup function suggests that timers are not fired while popups are active, but that's not really the case. In your program, operating the ring results in a popup (because there are too many items) and yet the timers still occur. Operating a normal ring menu, or a menubar menu, also does not suppress timers. The only thing I'm aware that prevents timers from occurring would be moving the window itself (and, of course, not having yet returned from the previous timer event, since timer callbacks are non-reentrant)

In any case, even if timers were suppressed I still wouldn't see an obvious reason for the hang. So, if a hang is happening, it might be because of some hard-to-reproduce bug in the CVI run-time engine. Do you happen to have a detailed sequence of actions that can reproduce the hang -- even if not reliably -- that I can try to mimic, in order to investigate this further?

Luis

[gdargaud]

Hello LuisG and thank you for looking into this.

I refined the test prog a bit and just managed to lock it 3 times on Linux out of about 20 tries. But I cannot get it to consistently lock up. I tried it a few times on XP whitout lockup.

#include <ansi_c.h>
#include <userint.h>
#include <cvirte.h>

static int Led=0, Pop=0;

// Just comment the lines whose events you want to ignore
void ShowEvent(int Event) {
	char *Str=NULL;
	switch (Event) {
		default:								printf("\nEvent: Unknown %d", Event); return;
		case EVENT_NONE:						Str="None"; break;
		case EVENT_COMMIT:						Str="Commit"; break;
		case EVENT_VAL_CHANGED:					Str="Val changed"; break;
		case EVENT_LEFT_CLICK:					Str="Left click"; break;
		case EVENT_LEFT_DOUBLE_CLICK:			Str="Left double clik"; break;
		case EVENT_RIGHT_CLICK:					Str="Right click"; break;
		case EVENT_RIGHT_DOUBLE_CLICK:			Str="Right double clik"; break;
		case EVENT_KEYPRESS:					Str="Keypress"; break;
		case EVENT_GOT_FOCUS:					Str="Got focus"; break;
		case EVENT_LOST_FOCUS:					Str="Lost focus"; break;
		case EVENT_IDLE:						Str="Idle"; break;
		case EVENT_CLOSE:						Str="Close"; break;
		case EVENT_PANEL_SIZE:					Str="Panel size"; break;
		case EVENT_PANEL_MOVE:					Str="Panel move"; break;
		case EVENT_END_TASK:					Str="End task"; break;
		case EVENT_TIMER_TICK:					Str="Timer tick"; break;
		case EVENT_DISCARD:						Str="Discard"; break;
		case EVENT_EXPAND:						Str="Expand"; break;
		case EVENT_COLLAPSE:					Str="Collapse"; break;
		case EVENT_DRAG:						Str="Drag"; break;
		case EVENT_DROP:						Str="Drop"; break;
		case EVENT_DROPPED:						Str="Dropped"; break;
		case EVENT_SORT:						Str="Sort"; break;
		case EVENT_SELECTION_CHANGE:			Str="Selection change"; break;
		case EVENT_HSCROLL:						Str="H scroll"; break;
		case EVENT_VSCROLL:						Str="V scroll"; break;
		case EVENT_MARK_STATE_CHANGE:			Str="Mark state change"; break;
		case EVENT_COMBO_BOX_INSERT:			Str="Combo box insert"; break;
		case EVENT_ACTIVE_CELL_CHANGE:			Str="Active cell change"; break;
		case EVENT_ROW_SIZE_CHANGE:				Str="Row size change"; break;
		case EVENT_COLUMN_SIZE_CHANGE:			Str="Column size change"; break;
		case EVENT_ACTIVE_TAB_CHANGE:			Str="Active tab change"; break;
		case EVENT_EDIT_MODE_STATE_CHANGE:		Str="Edit mode state change"; break;
		case EVENT_MOUSE_POINTER_MOVE:			/*Str="Mouse pointer move";*/ break;
		case EVENT_LEFT_CLICK_UP:				Str="Left click up"; break;
		case EVENT_RIGHT_CLICK_UP:				Str="Right click up"; break;
		case EVENT_MOUSE_WHEEL_SCROLL:			Str="Mouse wheel scroll"; break;
		case EVENT_PANEL_SIZING:				Str="Panel sizing"; break;
		case EVENT_PANEL_MOVING:				Str="Panel moving"; break;
		case EVENT_RADIX_CHANGE:				Str="Radix change"; break;
		case EVENT_ZOOM:						Str="Zoom"; break;
		case EVENT_AXIS_VAL_CHANGE:				Str="Axis val change"; break;
		case EVENT_INTERACTIVE_LEGEND:			Str="Interactive legend"; break;
		case EVENT_TABLE_ROW_COL_LABEL_CLICK:	Str="Table row col label click"; break;
		case EVENT_BEGIN_EDIT_TREE_CELL:		Str="Begin edit tree cell"; break;
		case EVENT_TREE_CELL_COMMIT:			Str="Tree cell commit"; break;
		case EVENT_TREE_CELL_BEGIN_MENU:		Str="Tree cell begin menu"; break;
		case EVENT_TREE_CELL_ACTIVE_ITEM_CHANGE:Str="Tree cell active item change"; break;
		case EVENT_TREE_CELL_VAL_CHANGED:		Str="Tree cell val changed"; break;
		case EVENT_CTRL_MENU:					Str="Ctrl menu"; break;
		case EVENT_DROP_COPY:					Str="Drop copy"; break;
		case EVENT_VAL_COERCED:					Str="Val coerced"; break;
		case EVENT_PANEL_MINIMIZE:				Str="Panel minimize"; break;
		case EVENT_PANEL_MAXIMIZE:				Str="Panel maximize"; break;
		case EVENT_PANEL_RESTORE:				Str="Panel restore"; break;
		case EVENT_RING_BEGIN_MENU:				Str="Ring begin menu"; break;
	}
	if (Str!=NULL) printf("\nEvent: %s", Str);
}


int CVICALLBACK cb_Ring(int panel, int control, int event,
		void *callbackData, int eventData1, int eventData2) {
	int Val;

	ShowEvent(event);
	
	switch (event) {
		case EVENT_COMMIT:
			GetCtrlVal(panel, control, &Val);
			printf("\nValue: %d", Val);
			break;
	}
	return 0;
}

int CVICALLBACK cb_Quit(int panel, int control, int event,
		void *callbackData, int eventData1, int eventData2) {
	switch (event) {
		case EVENT_COMMIT:
			QuitUserInterface (0);
			break;
	}
	return 0;
}

int CVICALLBACK cbt_Timer(int panel, int control, int event, 
		void *callbackData, int eventData1, int eventData2) { 
	static int Val=0;
	switch (event) {
		case EVENT_TIMER_TICK:
			SetCtrlVal(panel, Led, Val++&1);
			if (Val%10==0) {
//				SuspendTimerCallbacks ();
//				ConfirmPopup ("Crash?", "Do you think this will lock up?");
				
				InstallPopup(Pop);
				RunUserInterface();
				RemovePopup(0);
				//DiscardPanel(Pop);
				
//				ResumeTimerCallbacks ();
			}
			break;
	}
	return 0;
}

static void InsertLabels(int Panel, int Ctrl, int Nb) {
	char Label[10];
	for (int i=0; i<Nb; i++) {
		sprintf(Label, "-%03d-", i);
		InsertListItem (Panel, Ctrl, -1, Label, i);
	}
}

int main (int argc, char *argv[]) {
	if (InitCVIRTE (0, argv, 0) == 0) return -1;
	int Pnl  = NewPanel(0, "Test Ring", 20, 20, 100, 100);
	    Pop  = NewPanel(0, "Popup panel", 200, 200, 100, 100);

	int CtrlOK = NewCtrl (Pop, CTRL_SQUARE_COMMAND_BUTTON_LS, "OK", 5, 5);
	SetCtrlAttribute (Pop, CtrlOK, ATTR_CALLBACK_FUNCTION_POINTER, cb_Quit);
	SetPanelAttribute(Pop, ATTR_CLOSE_CTRL, CtrlOK);
	
	int Tmr  = NewCtrl (Pnl, CTRL_TIMER, "Timer", 50, 50);
	SetCtrlAttribute (Pnl, Tmr, ATTR_CALLBACK_FUNCTION_POINTER, cbt_Timer);
	
	Led = NewCtrl (Pnl, CTRL_ROUND_LED_LS, "Led", 50, 2);

	int CtrlQuit = NewCtrl (Pnl, CTRL_SQUARE_COMMAND_BUTTON_LS, "Quit", 75, 2);
	SetCtrlAttribute (Pnl, CtrlQuit, ATTR_CALLBACK_FUNCTION_POINTER, cb_Quit);
	SetPanelAttribute (Pnl, ATTR_CLOSE_CTRL, CtrlQuit);
	
	int Ring = NewCtrl (Pnl, CTRL_RING_LS, "Ring", 20, 2);
	SetCtrlAttribute (Pnl, Ring, ATTR_CALLBACK_FUNCTION_POINTER, cb_Ring);
	InsertLabels(Pnl, Ring, 200);
	
	DisplayPanel (Pnl);
	return RunUserInterface ();
}

 About workarounds, so I can catch the event EVENT_RING_BEGIN_MENU and maybe set a global that I test before opening any Popup window.

Or maybe do the opposite: when opening a popup, I loop through all user interface controls looking for rings and disable them (dimmed or indicator).

A better way would be to force close the ring menu when I need to display a popup, as if [Cancel] was pressed. But I don't think that's possible, is it ?

Any thought about that ?

[gdargaud]

Minor bug report: UILEventString () is missing some event.

[gdargaud]

Other bug reports: the word 'hoizontal' in userint.h (yeah, hard to be more minor than that !)

Missing event: EVENT_RING_END_MENU, making it impossible to use the 1st workaround I described above as cancelling the ring menu doesn't trigger any kind of event.

[LuisG]

I couldn't reproduce the hang with this variation either, unfortunately. When you can reproduce it, have you tried breaking into the hang in the debugger to see what the source position is? That might give us some clues.

Another thing you can do is to log as much as the code as possible, at least the timer callback before and after the popup, and the begin menu. That way, even if you can't debug the hang, you could reconstruct some of the sequence of actions just before the hang.

Before we start focusing on workarounds, I think we need to have an understanding of what exactly is causing the hang, so I recommend continuing to investigate it, if you have the time for it. If the debugger and the logging don't help, we might need to try some other tools, such as process explorer, for example, that will give us a better snapshot of what all the process threads are doing, including the CVI run-time engine.

The reason there is no end menu event is because the begin menu isn't intended to track the lifetime of the menu. It's there to give you a chance to programmatically alter the contents of the menu, at the time that your end user invokes it. It's basically a menu dimmer callback.Having said that, if it were really necessary, we could probably come up with a way to find out when the menu goes away.

Yes, the UILEventString bug is a known issue. The hoizontal, however, wasn't. 

[gdargaud]

Hello Luis,

just to be sure, you HAVE tested it on Linux, right ?

There seem to be slightly distinct cases. One where I can click and bring to foreground the popup and the ring menu while the program hangs. And another similar one where the popup is not repainted (it turns black after the first time it's sent to the background). And yet another one where the popup goes away when clicked on, but the ring menu hangs.

I just did a backtrace in the last situation but it's not very helpful:

(gdb) bt
#0  0x00b21402 in __kernel_vsyscall ()
#1  0x006cecc5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#2  0x008d35a4 in ?? () from /usr/lib/libX11.so.6
#3  0x008ea84c in _XReply () from /usr/lib/libX11.so.6
#4  0x008db6b0 in XQueryPointer () from /usr/lib/libX11.so.6
#5  0x00f43fac in ?? () from /usr/local/lib/libcvi.so
#6  0x09f7d208 in ?? ()
#7  0x00000260 in ?? ()
#8  0xbfba4754 in ?? ()
#9  0xbfba4750 in ?? ()
#10 0xbfba4768 in ?? ()
#11 0xbfba4764 in ?? ()
#12 0xbfba4760 in ?? ()
#13 0xbfba475c in ?? ()
#14 0xbfba4758 in ?? ()
#15 0xbfba5c18 in ?? ()
#16 0x00000003 in ?? ()
#17 0x00fc9a28 in ?? () from /usr/local/lib/libcvi.so
#18 0xbfba4760 in ?? ()
#19 0xbfba475c in ?? ()
#20 0xbfba4758 in ?? ()
#21 0xbfba4750 in ?? ()
#22 0xbfba4768 in ?? ()
#23 0xbfba4764 in ?? ()
#24 0xb7f0df28 in ?? ()
#25 0x00f3a468 in ?? () from /usr/local/lib/libcvi.so
#26 0x00000000 in ?? ()

(gdb) finish
Run till exit from #0  0x009a8402 in __kernel_vsyscall ()
Program received signal SIGTSTP, Stopped (user).
0x009a8402 in __kernel_vsyscall ()

(gdb) finish
Run till exit from #0  0x009a8402 in __kernel_vsyscall ()
0x00655cc5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0

(gdb) finish
Run till exit from #0  0x00655cc5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0

 The prog seems to be stuck in pthread_cond_wait.

One thing I've noticed is that if the bug doesn't happen on the first popup and/or first ring menu, it doesn't seem to happen at all.

As for workaround, the one I would prefer is to have a way to programatically close a ring menu (similar to pressing cancel). This way if something urgent comes up, I can interrupt. If you know of a way to do this please let me know.

I've devised and tested a workaround, but I haven't gone to prod with it yet.

It goes this way:

- catch ring menu begin events for all rings and set a global var

- catch lost focus and commit event for all rings and unset said global var

- catch left click up for panels containing rings and unset same global var

I do the above by looping through all panels and controls and chaining a control and a panel callback, so there's not much code modification.

The paintul part is to find all popups and not call them if the global var is set.

Like I said, it works but it changes the program behavior.

EVENT_RING_BEGIN_MENUEVENT_RING_BEGIN_MENUEVENT_RING_BEGIN_MENUEVENT_RING_BEGIN_MENU

[LuisG]

No, I hadn't tested in Linux, because I read in one of your earlier posts that the hang also happened in Windows. I'm pretty limited right now in how much investigation I can do in Linux. But I did set up a system today and tried your program in Linux. I was able to reproduce a hang, which might be one of the ones you're describing. To reproduce this one, I don't need to display the popup from the timer at all. All I need to do is to scroll the list in the ring popup up and down for a few minutes. Also, this hang would not happen in Windows.

I've created a bug report for further investigation: 477444. Hopefully someone will post an update here soon with some additional information about this hang.

[gdargaud]

Woah! I just reproduced it like you said on the first try, without popup. It's not even necessary to have a timer. Just a panel with a ring.

This is a MAJOR bug that I'm sure wasn't present in the previous version of CVI for Linux, and it's a HUGE problem for me. I have several CVI programs that run critical infrastructures. If they crash whenever you select a ring... Well you can see where this is going.

I don't think there's anything I can do besides replacing all the rings with... what ?

Rant mode: There were several critical bugs in the previous version of CVI for Linux but I managed to write workarounds while waiting 3 years for the update. This is not going to go this time and I'll require a patch.