Lockups in CVI for Linux

[Guillaume Dargaud]

Hello all,
I just spent another few days investigation random lockups in a complex
application. I cannot reproduce them in a simpler system which is
infuriating: my test system works fine, my production system locks up!

They seem to happen sometimes when resizing windows.

I've gone through all the possibilities I could think of and eliminated
them:
- ATTR_FIXED_PLOT_AREA to TRUE on all graphs/strip charts
- removed all panel sizing calls from panel/control callbacks
- they don't seem to trigger on windows which aren't resizeable.
Still I get random freezes and gdb tells me it's in some nameless functions
deep in CVI libs.


Possible related bug (although it's marked as fixed in CVI2010):
http://zone.ni.com/devzone/cda/tut/p/id/12158#200558_by_Category


Yeah, I know this all is very vague... Grasping at straws here...
--
Guillaume Dargaud
http://www.gdargaud.net/

[Guillaume Dargaud]

This is driving me absolutely crazy.
The lockups happen inside RunUserInterface() and I have no clue what in my
code is causing them. They are somehow reproducible (currently every time I
restart the program). But I cannot reproduce it in a controlled environment,
only on the production system!


When it happens Xorg and my program use most of the CPU. It must be some
window repainting problem (?!?):
$ top
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2999 root 15 0 77736 37m 19m R 77 1.1 3770:56 Xorg
2454 myuser 15 0 98.6m 56m 7400 S 49 1.7 1:05.46 myprog

Here are two basically identical debugger traces:

$ gdb ./myprog 1823
GNU gdb Red Hat Linux (6.5-37.el5_2.2rh)
Copyright (C) 2006 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu"...Using host libthread_db
library "/lib/libthread_db.so.1".

Attaching to program: /home/myuser/GenepiClient/20120312-1636/myprog,
process 1823
Reading symbols from /usr/local/lib/libcvi.so...done.
Loaded symbols for /usr/local/lib/libcvi.so
Reading symbols from /usr/local/lib/libnianlys.so...done.
Loaded symbols for /usr/local/lib/libnianlys.so
Reading symbols from /lib/libpthread.so.0...done.
[Thread debugging using libthread_db enabled]
[New Thread -1208756528 (LWP 1823)]
[New Thread 32402320 (LWP 1825)]
[New Thread 61463440 (LWP 1824)]
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /usr/local/lib/libcvintwrk.so...done.
Loaded symbols for /usr/local/lib/libcvintwrk.so
Reading symbols from /usr/local/lib/libninetv.so...done.
Loaded symbols for /usr/local/lib/libninetv.so
Reading symbols from /lib/libdim.so...done.
Loaded symbols for /lib/libdim.so
Reading symbols from /lib/libc.so.6...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/libm.so.6...done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /lib/libdl.so.2...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/librt.so.1...done.
Loaded symbols for /lib/librt.so.1
Reading symbols from /usr/lib/libX11.so.6...done.
Loaded symbols for /usr/lib/libX11.so.6
Reading symbols from /usr/lib/libXext.so.6...done.
Loaded symbols for /usr/lib/libXext.so.6
Reading symbols from /usr/lib/libstdc++.so.6...done.
Loaded symbols for /usr/lib/libstdc++.so.6
Reading symbols from /usr/local/lib/libLV100000_BLASLAPACK.so...done.
Loaded symbols for /usr/local/lib/libLV100000_BLASLAPACK.so
Reading symbols from /usr/local/lib/libiomp5.so...done.
Loaded symbols for /usr/local/lib/libiomp5.so
Reading symbols from /lib/ld-linux.so.2...done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from /usr/local/lib/liblkdynam.so.5...done.
Loaded symbols for /usr/local/lib/liblkdynam.so.5
Reading symbols from /usr/local/lib/liblkrealt.so.5...done.
Loaded symbols for /usr/local/lib/liblkrealt.so.5
Reading symbols from /usr/local/lib/liblksock.so.5...done.
Loaded symbols for /usr/local/lib/liblksock.so.5
Reading symbols from /usr/local/lib/liblkbrow.so.5...done.
Loaded symbols for /usr/local/lib/liblkbrow.so.5
Reading symbols from /lib/libgcc_s.so.1...done.
Loaded symbols for /lib/libgcc_s.so.1
Reading symbols from /usr/lib/libXau.so.6...done.
Loaded symbols for /usr/lib/libXau.so.6
Reading symbols from /usr/lib/libXdmcp.so.6...done.
Loaded symbols for /usr/lib/libXdmcp.so.6
Reading symbols from /usr/local/lib/liblksec.so.5...done.
Loaded symbols for /usr/local/lib/liblksec.so.5
Reading symbols from /usr/lib/libXcursor.so.1...done.
Loaded symbols for /usr/lib/libXcursor.so.1
Reading symbols from /usr/lib/libXrender.so.1...done.
Loaded symbols for /usr/lib/libXrender.so.

[Guillaume Dargaud]

It was, again, the ATTR_FIXED_PLOT_AREA bug.
I had missed one of the graphics.

In a past post (http://forums.ni.com/t5/LabWindows-CVI/CVI-8-0-Linux-
runtime-crashing-in-PlotStripChart/td-p/1285684), I see this bug tracking
number as 255697, but I don't find it on ni.com
--
Guillaume Dargaud
http://www.gdargaud.net/

[LuisG]

Guillaume,

The post you're linking to is for a different issue. Bug 255697 was filed as a result of that post, but because the crash turned out to be caused by a corrupted .uir (the root cause of which we never found) this ended up not being reported as a bug in ni.com.

The bug that you actually ran into is recorded as two separate bugs: 316930 on Windows and 344028 on Linux. There are two records for it, even though there is only a single root-cause bug, because the bug has very  different symptoms in Windows and Linux (in Windows it's barely noticeable). These also aren't in ni.com yet because there hasn't been an update of the Known Issues lists since this bug was processed. But on the next update they will appear.

Luis

[Guillaume Dargaud]

Well...

A critical app locked up several times while in full use and now won't even
start without locking up. It displays the panels and then hangs.
So working around the ATTR_PLOT_AREA bug didn't solve it completely.

Once, after being left to run for a while I got this message on the console:

XIO: fatal IO error 104 (Connection reset by peer) on X server ":0.0"
after 2377724291 requests (2377724288 known processed) with 25722
events remaining.

Looks like it keeps repainting the panels or something like that.
I (and the team which should be using this soft) am stumped.

I need a fix for that ASAP.
--
Guillaume Dargaud
http://www.gdargaud.net/

[LuisG]

I'm not sure if you're saying that this is the ATTR_FIXED_PLOT_AREA bug or isn't. Does it always lock up when you plot to a graph and does it never lock up when you don't plot?

[Kevin_B.]

Guillaume, 

I can't tell from your description if this is exactly the same issue or not. It sounded like what I suggested to you earlier had fixed the problem, and so we need to determine if this is the same issue, or a different one.

The easiest way to do this is to disable all axis names for all Graphs/Stripcharts/Digital Graphs. Go into the Edit Axis Settings dialog for each axis and make sure that the first field, Axis Name has nothing in it. If you're using a Right Y-Axis or a Top X-Axis, those names need to be removed as well. Then try your program and see if that fixes the problem.

If it doesn't fix the problem, and you're sure you removed all axis names, then we're looking at a completely different issue.

If it does fix the problem, then you can start adding Axis Names back into your graphs/stripcharts one at a time until we find exactly which one is causing the hang. We also need to determine if it's an Y-axis name that's causing the problem or a X-axis name. I'd enable Fixed Plot Area on the control. If it's the left Y-axis name I'd suggest adding 5 to the Plot Area Left and subtracting 5 from Plot Area Width using the property browser. Keep increasing Plot Area Left until the problem stops happening. If it's the Bottom X-Axis you'd probably want to keep Plot Area Top the same, and just subtract 5 from Plot Area Height.

Try those things and let me know what you find and we'll determine how to proceed from here.

[Guillaume Dargaud]

Thank you for your prompt reply.

I've disabled all the axis names (but not the numeric or text labels) with
the following macro just to be sure (lots and lots of graphs in this app, so
I just loop on all panels/controls to be sure):

#define NO_AXIS(Pnl, Ctrl) { \
int B = SetBreakOnLibraryErrors (0); \
SetCtrlAttribute(Pnl, Ctrl, ATTR_ACTIVE_YAXIS, VAL_RIGHT_YAXIS); \
SetCtrlAttribute(Pnl, Ctrl, ATTR_YNAME, ""); \
SetCtrlAttribute(Pnl, Ctrl, ATTR_ACTIVE_YAXIS, VAL_LEFT_YAXIS); \
SetCtrlAttribute(Pnl, Ctrl, ATTR_YNAME, ""); \
SetCtrlAttribute(Pnl, Ctrl, ATTR_ACTIVE_XAXIS, VAL_TOP_XAXIS); \
SetCtrlAttribute(Pnl, Ctrl, ATTR_XNAME, ""); \
SetCtrlAttribute(Pnl, Ctrl, ATTR_ACTIVE_XAXIS, VAL_BOTTOM_XAXIS);\
SetCtrlAttribute(Pnl, Ctrl, ATTR_XNAME, ""); \
SetBreakOnLibraryErrors (B); \
}

So far so good, but the program is in night mode and not doing much. I'll
let you know tomorrow when the operators start playing with it again.

Thanks.
--
Guillaume Dargaud
http://www.gdargaud.net/

[Guillaume Dargaud]

'morning,
So far so good. No hang reported after hours of use.

I can live with a fixed plot area until the release of CVI 2012, it's a bit
ugly but no big deal. But the lack of titles on the axes is more problematic
as they convey quite a bit of info.

Where do we go from here ? Other things to test ? Patch ?
--
Guillaume Dargaud
http://www.gdargaud.net/

[Kevin_B.]

My guess is that it's the Left Y-Axis that is causing this issue. It's theoretically possible to happen with all the other axes, but I've never seen it manifest itself with them. You could modify your macro to only hide the left Y-Axis name (I realize this is often the most important one), but that would give us a little more information, and hopefully give your operators more info as well.

One other thing you could try is to change ATTR_XYNAME_FONT or ATTR_XYNAME_POINT_SIZE. I'd probably try increasing point size 1 or 2 points, and if that doesn't work maybe using a different font with a couple different point sizes. This method is probably the least likely to work, but it's possible it might solve your problem in the interim.

You could also try this step from my previous post:

"If it does fix the problem, then you can start adding Axis Names back into your graphs/stripcharts one at a time until we find exactly which one is causing the hang. We also need to determine if it's an Y-axis name that's causing the problem or a X-axis name. I'd enable Fixed Plot Area on the control. If it's the left Y-axis name I'd suggest adding 5 to the Plot Area Left and subtracting 5 from Plot Area Width using the property browser. Keep increasing Plot Area Left until the problem stops happening. If it's the Bottom X-Axis you'd probably want to keep Plot Area Top the same, and just subtract 5 from Plot Area Height."