Issues with acquiring from multiple GigE cameras

[thisguy]

Hi,

I'm trying to build a simple app where three cameras are running simultaneously and can optionally record AVIs with timestamped data appended to each frame. I'm not having the recognition issue (all three cameras are seen through either a network bridge on the PXI system I'm using or an external GigE switch), but they seem to be failing during the grab process. When I have more than one running, green noise in the form of horizontal lines appear on the images, and usually all but one of the cameras will eventually drop out. No such problem exists when triggering only one camera. It seems a lot like the frames are colliding somehow; can grabs from multiple cameras not occur in the same loop?

As for the program set up, it's mainly a modified version of the "write" case in the AVI Read/Write with Data example. The user is prompted to save the avi's and a text file, three camera and IMAQ sessions are opened (with unique names), and then the program loops until the stop is hit, with a true/false case for actually recording the data. Frame/sample rate is controlled by the millisecond timer wait function. The problem happens regardless of acquisition speed (even at 1 frame/sec), and the CPU nor the network link seem to be anywhere near fully taxed. 

Thanks,

Brian

[toto26]

Hi there,

It seems to be a bandwith problem. But there are some solutions...;)

First, check in you camera parameteres if you can set up the maximum allowed bandwith. If you have 3 cameras, then divide the max bandwith by 3 and enter the value in this parameter. 

Another option is to trigger your cameras in order to limit the number of frame sent on the network. A software trigger also do the job.

Hope this helps

[Photon_Dan]

I am experiencing the exact same issue here.  I just called in to NI Support and walked through reproducing those nasty horizontal lines while on the call.  The AE was able to see those lines and has escalated the issue to the developers.  Hopefully there is something that we are all missing.  I did try setting the Bytes per Packet parameter in MAX to 2000 from its normal 8000 (I'm using 4 cameras).

Intel Pro1000 Ethernet - Yes!

NI GigE Ethernet Driver loaded - Yes!

Gigabit Switch with Jumbo Frame support - Yes, up to 9,216 bytes

Cameras - 2 Basler Scouts (1Mp) and 2 Basler Pilots (5Mp)

Driver - NI-IMAQdx 3.8.1

VBAI 2011

I actually get the green lines when acquiring frames on just one camera if another camera is included in an open insection in VBAI.  I'm talking about just launching VBAI and opening a inspection, not even running it.  Once I do that, if I open MAX and select a different camera, I see the green lines when I acquire a snap or grab.

There appear to be other bandwidth settings available under the "Transport Layer" section of the camera attributes, but it looks like Bandwidth is set to 10% with a 10X multiplier for "burst" traffic.  I changed the multiplier to 2X, but I still see the problem.

I am awaiting a response from NI, but in the meantime, if anyone has any suggestions...

Thanks,

Dan Press

PrimeTest Automation

[Brad]

You can try two things:

1. Use the Snapshot mode in the VBAI acquisition step. This will free up bandwidth so the camera only acquires when the step runs. This takes a little longer for the acq step to run, so another option would be ...

2. Configure the camera for SW trigger mode and use the Read/Write Camera Attribute step to send the SW trigger before the acquisition step (make sure to use the Immediate mode and note Wait for Next in this case). I was getting green lines from my Basler camera with this approach, but I also didn't have GigE network connections (was going through some 100MB routers), so not sure what would happen with better network connections. 

There may be some network settings you can play with also, but you can also try these VBAI options in the mean time.

Hope that helps,

Brad

[BlueCheese]

Hi Dan,

I think you're confusing a few thingd about packet size. This doesn't work like FireWire where you share the total packet size on the bus. Thus, reducing the packet size doesn't share bandwidth better, it just creates more work! Keep the packet size set to the 8k. Next, go to the attribute tree and have it show all attributes. You'll see one called Max Peak Bandwidth under Advanced Ethernet. Here is where you divide the 1000Mbit bandwidth among your cameras as you see fit. Depending on the camera, the "Actual Peak Bandwidth" might not reduce past a certain amount unless you reduce the frame rate.

Eric

[Photon_Dan]

Thanks for the suggestions guys.  I put the packet size back to 8000 on both cameras.  So far, I'm still seeing green horizontal lines.  Here's what I've tried and what I've learned.

1.  Set Acquisition Mode in VBAI to Continuous - Immediate

2.  Tried to reduce the bandwidth demand of the cameras by setting Acquisition Frame Rate (Abs) to 3.0

3.  Checked the Device Max Throughput under Transport Layer in MAX

a. Color Pilot camera is 30.4 MB/sec

b. Mono Pilot camera is 11.1 MB/sec

4.  Checked the Bandwidth Assigned under Transport Layer in MAX, and it is 125 MB/sec

5.  Add 30.4 to 11.1 and get 41.5 MB/sec so there is 83.5 MB/sec bandwidth available.

6.  Subtract 5 MB/sec for overhead and 6 MB/sec for each camera for Bandwidth Reserve (set at 5%) and there is still 66.5 MB/sec left over

7.  Open VBAI, and then open an inspection in which the Mono camera is configured (mad sure to load the defaults that were saved in MAX)

8.  Launch MAX and try a Grab from the Color camera

9.  See green lines of death.... :manmad:

So, I've narrowed my experimentation down to just two cameras and I still have acquisition problems.  I've got to be missing something else.  Note that there is no camera attribute section called "Advanced Ethernet" only "Transport Layer".  Perhaps that was camera-specific.

I think I'm going to have to use dedicated Ethernet ports for each camera.  It is disappointing since this architecture should work given those bandwidth concessions made for frame rate.

 - Dan

[Brad]

Did you try snapshot mode in the VBAI step?

[Photon_Dan]

Snapshot Mode clears up the horizontal lines while the inspection is not running, but they come back if there are two acquisitions requested at the same time.  In my process I do not want to restrict acquisition to just one camera at a time.  Even though I just need one single frame, I do not have a good way to serialize the cameras.  My intent is to give my customer one VBAI inspection for each camera.  I could do it with a big inspection using states for each camera, but it adds that extra layer of complexity and takes away the possibility of parallel execution of inspections.  Why won't the cameras share bandwidth the way they are told to?  I feel like I should be able to run Grabs on four cameras a long as the frame rates are low enough.  Image quality is paramount.  It is no good to serve up an image if it is broken up by those lines everywhere.  I'd rather wait another fraction of a second for the image to be complete.  The driver / abstraction layers should handle the bandwidth sharing by design.  Wishful thinking I guess.

 - Dan

[BlueCheese]

Hi Dan,

Sorry I wasn't more clear in my instructions before. The underlying driver does provide such a mechanism, although it is not completely automatic (I won't get too deep into the reasons of why it can't/shouldn't be fully automatic). I'm guessing you just were not able to find the relevent property. I've attached screenshot showing how to access the property in VBAI. You need to click the "Show All Attributes", then scroll up since the newly-visible categories are at the top of the tree and will not be visible, then expand the categories as shown. Let me know if you have further questions.

Eric

[Photon_Dan]

Thanks Eric, I believe that hidden and buried setting is the key.  How fast can that be added to the KnowledgeBase article about multiple GigE cameras?

I have switched to multiple Ethernet ports on the PC with one dedicated to each camera, and everything is running full speed with no noise now.

 - Dan