Intermittent timeout error with linescan camera. Get error after scanning hundreds or as much as 400,000 frames

[glen.meiring]

I am using a Dalsa Spyder 3 line scan camera.  It will scan for a while and return a timeout error.  I am using LabVIEW 8.5 and I think I have the latest IMAQdx.  My application needs to capture images continuously.  At first I was doing development using a VMWare workstation which seemed to work fine.  Then I started getting these intermittent timeout errors.  So I installed LabVIEW on real hardware.  It was better, but still got errors usually within scanning a few thousand frames (10 line frames).  I installed the Intel Pro 1000 card, and I thought all was well.  It ran for a couple of hours, almost 400,000 frames with no errors.

Once I receive the timeout error, the program will not capture any more images from the camera without re-initializing.

I am using the low level vi's so I can buffer frames.  I call "IMAQdx Open Camera.vi", "IMAQdx Configure Acquisition.vi" and "IMAQdx Start Acquisition.vi" in a separate vi called "InitCamera.vi".  After InitCamera.vi, I start an image acquisition subvi which runds in a loop calling "IMAQdx Get Image.vi".  This subvi runs at an above normal priority to capture the images and put them in a queue.  I have a separate loop to dequeue the images and process them.

Once I recieve the timeout error, every call to "IMAQdx Get Image.vi" returns a timeout error.

Does anyone know why I am getting this timeout error and how to make it stop?  Also, I know it's likely to recieve have a network problem from time-to-time, so what is the proper way to recover from a timeout error?

Note: I have an older version of this program running on another Dalsa camera which has been running for over a week.  It doesn't use queueing or parellel loops.

Thanks a lot!

Glen Meiring

[BlueCheese]

Hi Glen,

I'm fairly doubtful that your application differences you mentioned (queing or parallel loops) would be causing timeout issues (unless you had some logic error in your code that was causing you to wait on a buffer index that was far enough in the future to cause you to timeout while waiting for it). Could you show a sample of your acquisition code just to be sure?

Are there other differences between your two setups? Are there differences in the firmware versions of the cameras? Do you have a different network topology, network hardware (PCI vs PCI-express), different PC specs, etc? What version of the IMAQdx driver are you using?

My hunch would be that perhaps you're having an intermittent connection issue that is causing you to lose packets. This can happen even on an isolated network if you were using a PCI network card that shares bandwidth with other devices on the PCI bus. Most network cards have very shallow FIFOs for DMA'ing packets into memory and instead rely on the bus they sit on to be fast enough at all times to accomodate the flow of ethernet traffic. On 32-bit/33Mhz PCI, you can saturate the entire bus with a single gigabit NIC transferring data in only one direction (and you could potentially have bidirectional traffic that fast). If you are in fact losing packets, the IMAQdx driver will try to request resends of the missing data. Normally, this should get the data and be fine (assuming you have enough bandwidth for the added resent data). Its possible that if you are really starved for bandwidth and/or CPU time, the flood of resent data starts clogging the pipe so much that it interferes with the normal image flow and degrades the situation further. It generally should clear up on its own I'd think (at a cost of potentially skipping some buffers) but I have seen issues with certain older cameras eventually just going completely dead if we saturate them with resend requests. Maybe that is what is happening.

I'd suggest a couple of things. First, switch to PCI Express network cards if you haven't (or 66Mhz PCI-X ones). You're asking for trouble if you're trying to use normal PCI for gigabit ethernet without lots of tweaking. Next, have your application query the "Resends Requested" and "Lost Packets" property nodes. See if those show any intermittent loss, especially when you get into the timeout state.

If you are seeing packet loss, you could also try throttling the peak bandwidth used by the camera. This is useful in cases such as running over a congested network. The problem is that there is a difference between average bandwidth and peak bandwidth. You could be transmitting an image frame once every 15 seconds, but when the camera transmits the frame it could be using 100% of the 1000Mbit bandwidth for a couple dozen milliseconds. On newer versions of IMAQdx you can adjust the Peak Bandwidth Requested on the camera. Depending on the camera's implementation, this may or may not throttle the image frame frate (if it doesn't, it will just show the Actual Bandwidth Requested to be higher than the amount you requested). Finally, you could try disabling packet resends. This will eliminate any interaction with the camera when you have packet loss and prevent problems if the camera were going into a bad state due to too many resends

Unfortunately, there's no clear answer as to what the problem is without more information to debug it. Hopefully I've given you some places to start and if you give some more details about your setup and try out some of the things above we should be able to determine what is going on.

-Eric

[glen.meiring]

Differences: The supplier sent me a different camera than the prototype I have running on the shop floor (running over a week with no error - older version of my application though).  The PC's are the same.   The network topology is slightly different.  IMAQdx = 3.1.0

I am monitoring the lost packets by comparing the buffer number out.  I don't show any lost packets.

I don't have any PCI Express slots (I don't think).  I would have to get some new hardware, which is fine if necessary.  The thing is, this is a pretty low demand application.  I really only need to acquire 30 to 120 lines per second.  My camera won't go that slow, so I set up for 10 line frames at 10x my desired line rate.  The point is, this is pretty low bandwidth application, I think.

I'll look into the resend count.

It seems to be system performance related.  When I was developing on the VMWare virtual machine, the timeout problem was really bad.  Since I started running on the real hardware with the Intel Pro 1000 NIC, it is much better, but still exists.  It takes an hour or two before there is an error.

I'm really beginning to suspect the camera.  Maybe I'll try running this new camera with the older version of the program which has been running for over a week with no errors.

Thanks for all your good suggestions.  I'm going to use as many as possible.

Glen

[BlueCheese]

Hi Glen,

I'm suspicious that the camera is flaking out when IMAQdx sends resend requests. Your experience with the VMWare case seems to suggest this. I'd disable the resends completely and see if your timeout problem goes away. You'll potentially lose packets under congestion conditions though. If you do, lower the peak bandwidth feature I mentioned and it should help quite a bit. While you are using a slow frame rate, the camera is potentially still sending the data very fast when it becomes available.

-Eric