GPIB Communication to Multiple Instruments is Dropping Out/Locking Up

[VRMan]

Hi,

I'm developing software that periodically queries multiple instruments through GPIB. However, after some time ranging from 5 minutes to 24 hours, communication to an instrument fails and causes communication to all other instruments to drop out. Sometimes, it causes the computer to completely freeze as well. And communication to the instruments can only be restored by powering down all the hardware and rebooting the computer.

I have been able to reproduce the problem with the following hardware/software setup:

- I have 2 NI Instrument Simulators connected to the computer. Each instrument is on its own GPIB-USB-HS channel with no other instruments present.

- The computer is running Windows XP with Service Pack 2 and all the necessary software from NI Developer Suite August 2005 (NI-VISA 3.3.1, NI-488.2 2.30, etc...)

- I wrote a small test program that connects to both instruments and periodically sends a MEAS:DC? query every 100 ms. The queries are locked by the same mutex so that there are no simultaneous queries despite having each instrument on separate GPIB channels.

Any comments or help are greatly appreciated.

Thanks!

VRMan

[VRMan]

Here's the source code for the test program if anyone wants to try and reproduce the problem. Thanks again!

#include <string.h>
#include <stdlib.h>
#include <stdio.h>

#include <windows.h>
#include <process.h>

#include <visa.h>

#define VISA_DLL     "visa32.dll"

#define INSTR1_RESOURCE_NAME  "GPIB1::5::INSTR"
#define INSTR2_RESOURCE_NAME  "GPIB2::5::INSTR"

typedef ViStatus (_VI_FUNC *viOpenDefaultRMPtr)(ViPSession vi);
typedef ViStatus (_VI_FUNC *viOpenPtr)(ViSession sesn, ViRsrc name, ViAccessMode mode, ViUInt32 timeout, ViPSession vi);
typedef ViStatus (_VI_FUNC *viReadPtr)(ViSession vi, ViPBuf buf, ViUInt32 cnt, ViPUInt32 retCnt);
typedef ViStatus (_VI_FUNC *viWritePtr)(ViSession vi, ViBuf  buf, ViUInt32 cnt, ViPUInt32 retCnt);

viOpenDefaultRMPtr g_pOpenDefaultRM = NULL;
viOpenPtr   g_pOpen = NULL;
viReadPtr   g_pRead = NULL;
viWritePtr   g_pWrite = NULL;

HANDLE    g_mutex = NULL;

unsigned __stdcall InstrThread( void *pArg )
{
 DWORD  dwEvent = 0;

 char  szBuffer[1024];
 ViSession session = VI_NULL;
 ViUInt32 retCount = 0;
 ViStatus status = VI_NULL;

 double  dDC = 0.0;

 // get the instrument session
 session = *(ViSession*) pArg;

 while ( true )
 {
  // attempt to acquire the mutex
  dwEvent = WaitForSingleObject( g_mutex, INFINITE );
  if ( dwEvent != WAIT_OBJECT_0 )
  {
   // force a context switch and then try to re-acquire the mutex
   Sleep( 0 );
   continue;
  }

  printf( "session 0x%08X acquired the mutex\n", session );

  // format the DC query command
  sprintf( szBuffer, "MEAS:DC?" );

  // write the query command
  retCount = 0;
  status = (*g_pWrite)( session, (ViBuf) szBuffer, (ViUInt32) strlen( szBuffer ), &retCount );
  printf( "\tcall to viWrite returned\n" );

  if ( status < 0 )
   printf( "\tviWrite returned an error: %08X\n", status );
  else if ( retCount != strlen( szBuffer ) )
   printf( "\tviWrite did not write the entire command\n" );
  else
  {
   ZeroMemory( szBuffer, sizeof( szBuffer ) );

   // read the query response
   retCount = 0;
   status = (*g_pRead)( session, (ViPBuf) szBuffer, sizeof( szBuffer ) - 1, &retCount );
   printf( "\tcall to viRead returned\n" );

   if ( status < 0 )
    printf( "\tRead returned an error: %08X\n", status );
   else
   {
    // add null-terminator to response
    szBuffer[ retCount ] = '\0';

    dDC = 0.0;
    if ( sscanf( szBuffer, "%lf", &dDC ) != 1 )
     printf( "\tcould not read DC value from return buffer: %s\n", szBuffer );
    else
     printf( "\tDC = %lf\n", dDC );
   }
  }

  printf( "session 0x%08X is releasing the mutex\n", session );

  // release the mutex
  ReleaseMutex( g_mutex );

  Sleep( 100 );
 }

 return (0);
}

[VRMan]

int main( int argc, char** argv )
{
 HMODULE    hVisaDLL = NULL;

 ViSession   rmSession = VI_NULL;
 ViSession   instr1Session = VI_NULL;
 ViSession   instr2Session = VI_NULL;
 ViStatus   status = VI_NULL;

 uintptr_t   instrThread1 = NULL;
 uintptr_t   instrThread2 = NULL;
 unsigned   threadID = 0;

 // load the Visa DLL
 hVisaDLL = LoadLibrary( VISA_DLL );
 if ( hVisaDLL == NULL )
 {
  printf( "can not load Visa DLL (%s)\n", VISA_DLL );
  exit( 0 );
 }

 // load the required functions from the DLL
 g_pOpenDefaultRM = (viOpenDefaultRMPtr) GetProcAddress( hVisaDLL, "viOpenDefaultRM" );
 g_pOpen = (viOpenPtr) GetProcAddress( hVisaDLL, "viOpen" );
 g_pRead = (viReadPtr) GetProcAddress( hVisaDLL, "viRead" );
 g_pWrite = (viWritePtr) GetProcAddress( hVisaDLL, "viWrite" );
 if ( ( g_pOpenDefaultRM == NULL ) ||
   ( g_pOpen == NULL ) ||
   ( g_pRead == NULL ) ||
   ( g_pWrite == NULL ) )
 {
  printf( "can not load functions from Visa DLL\n" );
  exit( 0 );
 }

 // open the default resource manager
 status = (*g_pOpenDefaultRM)( &rmSession );
 if ( status < 0 )
 {
  printf( "viOpenDefaultRM returned the error: 0x%08X\n", status );
  exit( 0 );
 }

 // attempt to open instrument1
 status = (*g_pOpen)( rmSession, INSTR1_RESOURCE_NAME, (ViAccessMode) VI_NULL, (ViUInt32) 10000, &instr1Session );
 if ( status < 0 )
 {
  printf( "could not open the resource: %s\n", INSTR1_RESOURCE_NAME );
  printf( "viOpen returned the error: 0x%08X\n", status );
  exit( 0 );
 }

 // attempt to open instrument2
 status = (*g_pOpen)( rmSession, INSTR2_RESOURCE_NAME, (ViAccessMode) VI_NULL, (ViUInt32) 10000, &instr2Session );
 if ( status < 0 )
 {
  printf( "could not open the resource: %s\n", INSTR2_RESOURCE_NAME );
  printf( "viOpen returned the error: 0x%08X\n", status );
  exit( 0 );
 }

 // create global mutex
 g_mutex = CreateMutex( NULL, FALSE, "GPIBMutex" );
 if ( g_mutex == NULL )
 {
  printf( "could not create global mutex\n" );
  exit( 0 );
 }

 // start a polling thread for instrument1
 instrThread1 = _beginthreadex( NULL, 0, InstrThread, &instr1Session, 0, &threadID );
 if ( instrThread1 == NULL )
 {
  printf( "could not start polling thread for %s\n", INSTR1_RESOURCE_NAME );
  exit( 0 );
 }

 // start a polling thread for instrument2
 instrThread2 = _beginthreadex( NULL, 0, InstrThread, &instr2Session, 0, &threadID );
 if ( instrThread2 == NULL )
 {
  printf( "could not start polling thread for %s\n", INSTR2_RESOURCE_NAME );
  exit( 0 );
 }

 // let the other threads access the GPIB devices
 while ( true )
 {
  Sleep( 1000 );
 }

 // no need to clean up since the while loop goes on indefinitely

 return (0);
}

[Matt_S.]

Hi VRMan,

Could you post a spy capture taken during this freeze up? That will help us see if you're getting some weird data back on the lines when this occurs.

Also, when you lose communication, what errors are you getting?(if any).

Regards,
Matt S.

[VRMan]

Hi Matt,

The NI-Spy capture file ended up being over 2GB, so I haven't been able to isolate the exact moment when the errors start occuring.

When the errors start occuring, calls to viWrite to one instrument will return VI_ERROR_INV_OBJECT, and the calls to viWrite to the other instrument will return VI_ERROR_RSRC_NFOUND. If I exit the application, go to MAX, and try to connect to the instruments using the GPIB Communicator, I can reconnect to the instrument that was returning VI_ERROR_RSRC_NFOUND errors but not the instrument returning VI_ERROR_INV_OBJECT.

Sometimes, the test application freezes the entire system completely including the mouse and keyboard.

I'll try and see if I can get an NI-Spy capture file that isolates the errors at the exact moment the errors start occuring.

Thanks a lot.

VRMan

[ScottieB]

NI-Spy does have an option to stop capturing when it sees an error, which may help you pare down your spy capture.  I would also be curious to know if it takes a specific number of iterations every time before it fails.

Scott B.
GPIB Software
National Instruments

[VRMan]

Sorry about the long break, but I've done some more testing on the issue by narrowing down the test case down to only a single NI Instrument Simulator device, a single NI USB-GPIB adapter, and a simple CVI program (source code is attached) that queries for the DC value in a loop. The NI-Spy captures are still hundreds of MB in size, but I've also attached a screenshot of the error when it first occurs. I've ran this test many times and the error definitely does not occur after the same number of iterations.

The computer I that I used to test was cleanly reformatted and had only Windows XP Professional SP2 installed with the latest security patches, NI Measurement Studio, and NI LabWindows CVI installed from the Feb 2006 NI Dev Kit. As for the hardware, the computer has an AMD 3400+ CPU, Shuttle SK83G motherboard with the VIA K8M800 chipset (has integrated video and ethernet), 1 GB of RAM, no PCI cards installed, no AGP cards installed, a Western Digital 80GB HD, and a Sony DVD RW drive.

I have also run the tests with Agilent VISA installed and NI-VISA uninstalled and the problem still occurs. So I plan on running some tests with other computer hardware configurations and PCI GPIB interfaces.

Thanks in advance to anyone who can help 🙂

[Matt_S.]

That error usually occurs when your session with that device has been interrupted for some reason. Are there any power lines or anything that could be causign your system to be disconnecting or getting too much noise? If you stop your application and run it again, does it work immediately, or do you have to restart first?

Regards,
Matt S.

[VRMan]

There aren't any power lines near the instrument simulator, and I'm certain there weren't any electrical disruptions, because the test computer is connected to the same power strip and has not had any problems.

After the error occurs, I have to shutdown the computer, unplug the USB-GPIB adapter, restart the computer, plug the USB-GPIB adapter back, and then it works again. I do not have to restart the instrument simulator. Without taking those steps, I can't reconnect to the instrument through the test program or the GPIB communicator in MAX.

I've setup another test computer with a 2.8 GHz Intel P4 and an Intel 875P chipset motherboard, and the tests have gone for almost 24 hours without any problems.  Hopefully, it will run for the whole week without any problems.

Thanks,

VRMan

[JasonSmith]

Please post whether or not you are able to replicate the problem on another computer.  If it works initially with the PCI-GPIB, try the other computer with the GPIB-USB-HS.  This will help us determine if the problem is associated to a particular type of GPIB interface.

Jason S.
Applications Engineer
National Instruments