unexplained error in cvirte.dll causes application to crash

[mdhgarcia]

We're running a test sequencer that's been working for months and recently began crashing.  The output from Windows is attached below.  The application is written in LabWindows 6.0, and the run-time engine is 8.1.1.361.  We have an IMAQ capture device running along with a third-party USB-to-serial break-out box.

Unfortunately, the software doesn't crash when run with the debugger (presumably using an earlier version of the RTE?) so we can't trace the call that triggers it.

Has anyone experienced this type of crash before?  Any solutions or suggestions?

Any way we can open up cvirte.dll in a debugger to see what's going on at address 0x00227163?

Thanks,

Matias Garcia

Event Type: Error
Event Source: Application Error
Event Category: None
Event ID: 1000
Date:  9/10/2009
Time:  2:37:50 AM
User:  N/A
Computer: TE_BRADLEYRNO
Description:
Faulting application 085156.exe, version 1.0.0.0, faulting module cvirte.dll, version 8.1.1.361, fault address 0x00227163.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 41 70 70 6c 69 63 61 74   Applicat
0008: 69 6f 6e 20 46 61 69 6c   ion Fail
0010: 75 72 65 20 20 30 38 35   ure  085
0018: 31 35 36 2e 65 78 65 20   156.exe
0020: 31 2e 30 2e 30 2e 30 20   1.0.0.0
0028: 69 6e 20 63 76 69 72 74   in cvirt
0030: 65 2e 64 6c 6c 20 38 2e   e.dll 8.
0038: 31 2e 31 2e 33 36 31 20   1.1.361
0040: 61 74 20 6f 66 66 73 65   at offse
0048: 74 20 30 30 32 32 37 31   t 002271
0050: 36 33 0d 0a               63..   

[jr_2005]

On two occasions in the past, I have managed to figure out a faulting function from the standard Windows crash dump. Try the following steps:

• Make a note of the faulting address
• Wade through the list of loaded modules until you (hopefully!) find one that is loaded at an address just a bit lower than the fault location
• Calculate the offset address into this module, at which the fault occurs, by simple subtraction of the load address from the fault one
• Using Dependancy Walker on the module, sort its functions by Entry Point
• Look through the list of functions for the one that has an address just a bit lower than the offset fault address

Voila! you have identified the function that caused the error. Hopeully, it will have a meaningful name and suggest a line of investigation. In my case, it turned out to be CloseFile in Windows that had an invalid pointer. (From a third party library, I hasten to add!) Once I realised this I could track down the problem to be a network file issue.

Not guaranteed to work, but worth a try.

JR

[mdhgarcia]

Thanks, JR.

Where did you find the crash dump, exactly?  We pulled this one out of the event viewer, but I'm not sure if I can recreate the list of loaded modules.  I know the module that cried wolf (cvirte.dll), but I don't know the address into which it was loaded, only the fault address.  I have sorted the entry points in there using dependency walker, which is a revelation (thanks for the tip), but I need to know the offset now.

Cheers,

Matias

[mdhgarcia]

Okay, I found out the entry point is the fault address listed.  The function is the innocuously-named "_UP_WriteLine".  I never would have guessed, and it doesn't appear to be called directly from our code...  It was worth a shot.

Cheers,

Matias

[nickb]

Hello Matias, 

If you'd like we can take a bit more specific of a look here with a little bit of your help. If you could download a debug version of the CVI run time I've created and posted here and place it in the same directory as your application, we should be able to start to make some sense of the crash you are seeing.  There's a couple things you'll need to do to get us the files we need, but the steps should be clearly laid out here.  Make sure (as indicated in my blog post) that you include a map file for your application with the .dmp file, so that we can get an idea what calls you are making to cause a crash in the runtime.

Let me know when the files are up, and I'll be happy to take a look.

Thanks!

NickB

National Instruments 

[mdhgarcia]

Thanks, Nick.

Do we need to run the debug executable, or will the release executable provide the same information since the library is now instrumented?

Out of curiosity, is it only debug symbols that are added in the debug executable, or is there instrumentation added?

Matias

[nickb]

Hello Matias,

You can run the release executable.  What is important to us is not actually instrumentation, but the symbols that I have generated for the debug version of the CVI run time engine I have made available for you to download.  These symbols will allow me to get a clean look at the stack trace when the crash occurs on your machine.  When the crash occurs in your release exe, I will be able to load the .dmp file that is created by Dr. Watson into Visual Studio, and see a clear stack trace at the time of the crash.  Also, as long as you remember to create a full dump, I will be able to see the state of our global data structures and local variables at the time of the crash as well.  With this information in hand, we'll be well on our way to finding the source of the crash.

NickB

National Instruments 

[mdhgarcia]

Thanks again.

Off-hand, does _UP_Writeline mean anything to you?  This test happens to be extremely high-priority right now, and it may be up to 10 hours before I can get crash data (the length of the test cycle).  Last time, it crashed around interval 22 of 26.

Matias

Message Edited by mdhgarcia on 09-11-2009 09:39 AM

[nickb]

_UP_Writeline doesn't ring any bells as far as issues that I've seen in the past.  However, when I match that offset of the reported crash against the map file that was generated for the runtime that was officially released, it appears that the crash is happening in snprintf, and not _UP_Writeline.  I'm also not immediately familiar with any issues with snprintf, but that may be something you can check out in your code. 

NickB

National Instruments