Multithread lockup after call to CmtScheduleThreadPoolFunction

[BrianGShea]

I'm really stuck, so any information would be great help. I can't post the code as it exceeds a few thousand lines of C.

System: WinXP (not WinXP-64bit), AMD Athlon 64, 1GB Ram.

I have tracked the problem to a specific point. It is the point after I exit from Foo after calling CmtScheduleThreadPoolFunction and before the thread executes. Meaning if i exit the Foo for some other error, the app will continue to run.

Here is a snippet of the code:
void Foo( pMyStructDef controlConn )
{
    // ConnectToTCPServer : Used for slow control and status monitoring
    ...  
    // Load WinSock, connect to remote IP, Highspeed data transfers.
    ...
    // Start Read Thread
    ret = CmtScheduleThreadPoolFunction( DEFAULT_THREAD_POOL_HANDLE, MFCReadThread,
           &controlConn->threadRead, &controlConn->threadRead.fID );
    if( ret < 0 ) { // Clean up on failure.
   
        // Close TCP Client
        DisableBreakOnLibraryErrors ();
        ret = DisconnectFromTCPServer( tcpConn->ctrl_sock );
        EnableBreakOnLibraryErrors();
       
        destroyQ( controlConn->CtrlRxQ );
       
        // Free up the read/write child Queues
        CmtDiscardTSQ( controlConn->threadRead.TSQHandle );
        CmtDiscardTSQ( controlConn->threadWrite.TSQHandle );
       
        // Make sure theads don't run.
        controlConn->threadRead.control = 0;
        controlConn->threadWrite.control = 0;
       
        // Close Socket
        closesocket(sock);
        WSACleanup();
        return 0;
    }

return 0; // Program will lock up after executing this.

} // End of Foo()

See Next Post to this message (5000 Char Limit)

[BrianGShea]

Continued from last post

CmtScheduleThreadPoolFunction executes and returns sucessful. However, my CVI program will lock the system 100% after leaving Foo(), ALT-CTRL-DEL does not work, i use the hard reset button.

I know the thread does not execute, MFCReadThread is never enter, breakpoint is not reached. Lockes up with or without breakpoint, single stepping or just running.

This code was working 100%, but sometimes it just stops for a few days until i comment out lots of code then slowly uncomments it section by section.

All structs are allocated on the heap so it's not a problem of not malloc'ing memory for them, they are passed in to foo as a pointer.

The thread will read the data in the struct ONLY and use it to access allocated sockets and terminate/pause when instructed to do so.

I use the NI TSQ's to pass all data read from the socket to the main applicaion.

One more after this. I hope 😉

[BrianGShea]

I can't post the thread function, >5000 Chars, so i hope the above is enough info.

Thanks

[BrianGShea]

Lesson for all:

In the above the lockup is being caused by the fact that i did this in my code

void Foo( pMyStructDef tcpConn )
{

    pMyStructDef controlConn = tcpConn; // This is the problem statement.

    // Same as above

    ret = CmtScheduleThreadPoolFunction( DEFAULT_THREAD_POOL_HANDLE, MFCReadThread,
           &controlConn->threadRead, &controlConn->threadRead.fID );

    ...

    return 0;
}

I had made a copy of the pointer, even though it is a pointer to the same location as tcpConn it causes a lockup.

After simplifing to what i had posted it started to work. I switched it back and locked up, then back to the first post and it works.

I haven't the faintest clue what the problem is, CmtScheduleThreadPoolFunction should write info to memory locations that are occupied by fID, as that is where the pointer points to and not the controlConn varible. Same with &controlConn->threadRead, The memory location of threadRead should be stored and not the loation of controlConn, which is lost after Foo exits.

Any comments on this would be welcome.

[Mert_A.]

Your problem could be memory corruption that is catching up with you at a later time.  If it seems like "random" changes are causing mysterious behavior, double check pairing of allocations/deallocations, perilous casts, uses of void *, and pointer arithmetic.

Try leaving the declaration in, but use the pointer passed into your function:

void Foo( pMyStructDef tcpConn )
{
    pMyStructDef controlConn = 0;
    if (controlConn)   // just to avoid possibility of compiler optimizing it away
      controlConn = 0;
    ...
    ret = CmtScheduleThreadPoolFunction( DEFAULT_THREAD_POOL_HANDLE, MFCReadThread,
           &tcpConn->threadRead, &tcpConn->threadRead.fID );
    ...
}

If this causes problems, your problem has nothing to do with the controlConn assignment.

The only other thing would be to make really sure that pMyStructDef is actually a pointer, and that you're not trashing it somewhere between the beginning of Foo and the call to CmtScheduleThreadPoolFunction.

Good luck.

Mert A.
National Instruments

[BrianGShea]

Good suggestions:

This is my standard struct make up, esp. for structs that get used and passed as pointers, it cuts out some typing.

typedef strut MyDefStruct {
      // Struct member defs (too many to list to be relivent
} MyDefStruct, *pMyDefstruct;

In main.c

MyDefStruct globalConn;

Later in callback function for command button in callbacks.c
// Testing only
extern MyDefStruct globalConn;
// end testing only
int CVICALLBACK CommConnect( ... ) // <- removed for readability
// This is ugly, cause it was never change to use the WinSock IPAddress Structs.
globalConn.ipaddr[0] = ipList[ipIndex].S_un.S_un_b.s_b1;
globalConn.ipaddr[1] = ipList[ipIndex].S_un.S_un_b.s_b2;
globalConn.ipaddr[2] = ipList[ipIndex].S_un.S_un_b.s_b3;
globalConn.ipaddr[3] = ipList[ipIndex].S_un.S_un_b.s_b4;

// Woops this should not be here, because it is done in Foo, will remove, but this wont
// cause a problem execpt for un-freed memory. 
globalConn.CtrlRxQ = createQ();
globalConn.PacketcallBack = &MyControlCallback; // Used for ConnectToTCPServer
   
Foo( &globalConn );

return 0;
} // End of CmnButton Callback

---
The point of passing a pointer to a Comm struct was to allow multiple connections to different units, and maintaining separate storage locations for all revelant connection data, such as thread id's, TSQ id's what unit the connection belongs to and so on.

FYI: createQ is an in house Linked List FIFO queue. Quick with minimal overhead. It is not used in any threads, only in the call back "MyControlCallback" setup with ConnectToTCPServer, it is mainly for queue non-time cirtical status packets from out remote unit. It has beed tested, an used for several projects.

Anyway, I don't think it's memory corruption of controlConn (tcpConn), I've already set a breakpoint prior to calling CmtScheduleThreadPoolFunction and checked the validity of the pointer and data in the pointer to the global var created in main.c

I do agree it does seem like memory corruption because it is a random occurrence, in happening, but always after the threads are started. I'm continuing to work through this. About 10min after i posted the last message the program started locking up again. So that pointer change did not help.

I'm going to try a different global var for fID then assign it to the struct after a successful return.

[BrianGShea]

Hmm, this problem has nothing to do with the thread or my code. I replaced my ReadThread with a dummy thread that counts down from 5 using a for loop then returns 0;

Once i exit the callback that addeds the thread to the default thread pool the system bogs down untill the thread finishs. About 30secsonds to do so.

If i breakpoint on the return 0 statement, the system is 100% usage and it takes time to register keystrokes, pressing F10 (single step) twice will exit the thread and all is fine again, but takes upwards to 2minutes to do so. It seems like the dubugger does not allow the thread to release control back to the OS as it should. Or the dubugger is not releasing control back to the OS (doubt this is the case).

No data was passed to thread function, no global varibles are accessed.

The program works fine when built as non-debug. It has to be a problem with CVI 7.1 and debugging threads.

Anyone have same type of issues?  I really need to have working threads to deal with data processings. Please HELP!!! Any sugestions to program flow to get threads up and running would be very helpful, NI has poor examples and limited documentation on the thread support.

At this point i just wish i had a fork() fucntion and some pipes to transfer data 😉

[Mert_A.]

This is pretty tough to diagnose, and it's not really something I've heard of before.  I can make some suggestions of things to try, just in hopes of better characterizing the problem.  Is the problem reliably reproducable?  From the pointer assignment confusion earlier, it sounds like it is not.  You could try writing a very simple app that schedules a thread pool function, then see if you can reproduce the issue while debugging that.  Also, you mentioned that when built as non-debug, the program works fine. What if you just run the debug executable without actually debugging it in CVI?  Can you reproduce the problem if you have no breakpoints or watch variables?  You might also want to try creating a new thread pool (CmtNewThreadPool) with various numbers of threads and running the thread function in one of that pool's threads.  Have you confirmed in the Windows task manager that it is your app's process that is eating all the cpu time?

You can also try patching your version of CVI from 7.1 to 7.1.1.  It is unlikely that this will solve your problem, but it can't hurt.

Good luck.

Mert A.
National Instruments

[BrianGShea]

Mert A. ,

Okay, i figured it out. You'll never beleive it. The randomness came from me adding a break point before or after the CmtScheduleThreadPoolFunction in the current callback or leaving it out and placing it elsewere like in the main loop. (Good guess on the breakpoints)

Porgram flow:

User clicks connect button. Callback is entered, Foo() is called, CmtScheduleThreadPoolFunction() is called and all fucntions return back to OS after thread is installed.

If i have a breakpoint in the callback or Foo() the system hangs after pressing F5 (run). No breakpoint, no hang, everything works fine as it should.

I can put breakpoints in the thread or my main loop without problems. (Main loop calles all Process functions that are required to draw and run system)

This happens with a new program designed just to start a thread an add it to a pool with a comman button callback, the thread runs and return immediately, or it should anyway. But locks up if the program is halted during the callback that addes the thread.

Give me a few I will post the watered down version.

Nop, I cannot access the Task manager to verify, the GUI is complete locked, only thing that works are mouse cursor and reset button.

I have CVI 7.1.1, just checked.

My system again is WinXP (not WinXP 64bit) AMD Athlon 64bit. No speical Hardware installed, and not working off network or USB drive all programs run from loal HD/Memory.

Thanks for the suggestions,

[BrianGShea]

WARNING THIS PROGRAM CAN CAUSE YOUR COMPUTER TO LOCK UP!!!
Okay, if you run the program with no breakpoints you get a threaded banner that scrolls across the dialog box. Nice right 🙂

Okay, if you put a breakpoint in ThreadStarter (main.c) or in ThreadControl (main.c) and run the program, when the The thread is started (Must click on Run Thread Command Button) you will get to the break point, that is fine, you can single step out of the functions and callback, but once the program returns to the system CVI & program lock up. Not sure who is causing the problem, my guess CVI.

Okay, if you uncomment CmtReleaseThreadPoolFunctionID in ThreadStarter (main.c) and run to breakpoint and continue to run, all is good.

One last observation, if you, instead of passing ThreadFunctionID in to ThreadStart as a pointer, hardcode the it as :
ret = CmtScheduleThreadPoolFunction( DEFAULT_THREAD_POOL_HANDLE, DummyThread, NULL, &ThreadFunctionID );

It is fine, as it should be.

The problem, is when a pointer to my global var to store the Function ID is used. fid is a pointer to a varible that is always avaiable. Eventhough fid is in the function's stack, we don't care we just want the memory location that fid stores (i.e. a pointer). These two methods are equivilant. Weather i use the address of operater to pass ThreadFunctionID as a pointer or feed the fucntion the memory location directly as a pointer.

Now if i passed &fid, this will be bad since fid is only avaiable during the execution of ThreadStarter and will be lost once the fucntion exits, resulting in access to an illegal memory address (possible).