Setting camera's ROI on the fly

[lcarlson]

Hello,

I am trying to set my camera's region/area of interest on the fly.  My scenario is as follows: I have information regarding the whereabouts of a ~100 µm laser spot that will appear somewhere within a 3 mm box on the camera's chip.  I know this information ~100 ms from when the camera will trigger to capture the spot.  I cannot download the entire 3 mm x 3 mm range of pixels because it will take too long (well, not with a CameraLink camera, but I am trying to stay with a GigE camera).  I would like to have the image available at the computer in about a millisecond so I can process it in another millisecond.  This should be feasible with a ~100x100 pixel image, but again, the spot can appear anywhere, so the ROI must be shifted ahead of time in anticipation to where the spot will appear.  My question is this: can I change or move the (or any) camera's ROI in ~50-100 ms?  What vi would be able to accomplish this?

So far I have tried two ways to do this:

1) Download the entire image, then mask the desired region.  This takes 20-30 ms due to downloading the 659x494 pixels.  Masking the region only helps with the image-processing algorithm time.

2) Set the ROI as soon as the spot's information becomes available.  This takes ~200 ms to write new ROI information to the camera with IMAQ 1394 grab setup.vi and snap an image.

Setup Information:
LV and Vision 8.0
Basler A311f firewire camera (for mock-up)
Windows XP (could possibly imprement into a real-time operating system that I have)

Any help or ideas would be appreciated.

Thanks!
lcarlson

[BlueCheese]

Hi LCarlson,

Interesting application you have there! I have a few ideas and suggestions...

Firstly, GigE Vision works a bit differently from firewire. There is more potential bandwidth than 1394a/b for one, so you can potentially get images transferred faster than 1394 can. In the latest version of IMAQdx we expose some controls which allow you to adjust the peak bandwidth used by the camera to adjust the tradeoff between latency it takes for the entire image to be transferred versus how much of bus bandwidth it takes up. If you are using a dedicated network connection to the camera with a fast PCI Express gigabit card, you should be able to adjust the camera to use every last byte of bandwidth when transferring the image (rather than spacing the packets out to make it less "bursty"). This may reduce the time it takes to download an entire image considerably compared to your firewire camera (my back-of-the-envelope calculation says ~3ms could be possible at your resolution).

Also, writing the new ROI may be a faster operation on GigE Vision cameras as well. For firewire cameras when you write the new ROI we need to write many other registers as well when we start the acquisition in order to ensure consistent behavior on all cameras from different vendors. On GigE Vision, the camera includes an XML file that specifies all the interdependencies between registers and allows intelligent caching to reduce the number of register writes needed to set the ROI and restart the acquisition. While Ethernet is potentially a higher latency bus than firewire, the reduction in registers to write may cut the configuration time considerably in some cases.

Lastly, since you seem to be only grabbing a single image on each iteration, you may want to consider using the Snap functionality instead of a Grab. The advantage is that you will only be allocating 1 buffer during the configuration time instead of the minimum needed for a continuous acquisition. It might not make a huge difference in the configuration time for small image sizes, but you may as well try it too.

Hope this helps,

Eric

[lcarlson]

Eric,

Thank you for your response.  A GigE camera seems like it would fit my bill.  I had a few followup questions.

1) For maximum bandwidth, do you recommend an Intel-chip GigE card in a PCIexpress x1 slot?  To me this seems like it would ensure the highest bandwidth.  If I cannot set the ROI fast enough, I might just have to download a full ~300x300 pixel image covering the entire area the spot could occur.  Do you have any favorite GigE, high framerate cameras?

2) Do IMAQdx drivers work in RT?  I have used up all the slots on the host computer (FireWire card, GigE card to talk to RT computer, and DAQ card).  The RT computer, which predicts the spot's anticipated location on the camera, has an available PCIe and PCI slot.  I would like to snap the image on the RT computer and send over the results to the host.

3) The snap function did not have much of an influence on the time to configure and grab the image, but I will use every advantage I can.

4) How are the IMAQdx driver acquired?  As an upgrade?  As part of a capture card?

Thanks,
Lcarlson

[BlueCheese]

Hi Lcarlson,

Yes, an Intel-based GigE card in a PCI Express slot is the best solution currently. NI directly sells and supports the PCIe-8231 and PXIe-8234, both PCI-express based, just different form factors (and the 8234 has 2 ports). Both should be able to be bundled with a copy of Vision Acquisition which includes IMAQdx.

The Basler cameras are very good all-around. Their basic VGA camera does 70fps full-frame, but if you cut down the image size can go much higher. At 300x300 it goes around 108 frames/sec. If you can cut down the ROI even more it goes faster. I believe they have some other cameras as well with faster sensors if you need even faster.

While IMAQdx is supported under LabVIEW RT, it only currently supports 1394. However, the API is the same for GigE Vision and 1394 when using IMAQdx so you can always start with your 1394 camera and move to GigE Vision once it is available under RT.

The IMAQdx driver is a licensed part of the Vision Acquisition software (just like the previous NI-IMAQ for 1394 driver was). Any hardware such as the gigabit cards I mentioned above will include a license for IMAQdx as well. It can however be purchased separately as well. Depending on what version of your current Vision Acquisition you have, you may be able to upgrade to a version of IMAQdx with your same license. Also, if you are enrolled in SSP you can always upgrade to the latest versions of Vision Acquisition as they are released.

Hope this helps,

Eric

[lcarlson]

Hello Eric,

Thanks for the helpful information.
Do you (or does anyone else) know when GigE cameras will be supported by IMAQdx under LV Real Time?  I would really like the determinism of the RT system, and the bandwidth and price of a GigE caerma as opposed to CameraLink.

Thanks,
Lcarlson

[VGA_CD-ROM]

Hi Lcarlson,
You are right in that we currently do not support Gigabit Ethernet cameras in LabVIEW RT, although some GigE network cards are now supported. I have not seen anything which indicates that support will be added for GigE cameras, but I unfortunately cannot confidently say yes or no on the issue. However, IEEE 1394 is currently supported, and the 1394b (also called FireWire 800) does have a maximum bandwidth of up to.800 Mbps, so that could potentially server as an alternative for you.

[lcarlson]

I am still trying to determine how fast it would take to set an ROI on a GigE camera.  Eric has assured me that it will be faster than FireWire, but has anyone ever timed this?  Are we looking at hundreds of milliseconds or just a few milliseconds?  Using a "performace" or "hardware-specific" driver is suppposed to reduce the overhead and get the registers written to the camera as fast as possible.  In my experience, it takes upwards of 200 ms using a FireWire camera and LabView.  How about a GigE camera with a hardware-specific driver?

Thanks,
Lcarlson

[BlueCheese]

Hi Lcarlson,

As to the overhead of changing the ROI and re-configuring the acquisition, keep in mind the performance is very camera-dependent. This is true even on FireWire because register writes vary in time they take to complete between different vendors and models. I only said GigE Vision has the potential to be a bit faster because of the intelligence the driver can use based on how the camera describes its features. When I get a chance I'll try benchmarking a Snap in a loop with a ROI change each iteration on a specific camera and see what kind of overhead it entails and let you know.

Sorry, I cannot yet give you a timeframe as to when GigE Vision will be supported on LabVIEW RT. However, as Vijay mentioned, the 1394b cameras are a good alternative that is fully supported under RT in the meantime.

I do have one option I'll throw out considering the unique application you have. Many Firewire cameras do have the option of changing the ROI parameters like the left and top offset while the camera is running. This can theoretically happen while the IMAQdx driver is acquiring provided the actual image configuration (like width and height) does not change. While the IMAQdx driver will currently not let you change the ROI at all while running (it has other complications that could cause other parameters to become invalid), if you are careful and have a flexible camera you may be able to change just the offsets without the IMAQdx driver knowing the difference. For this you would need to look up the register specifications for your camera and determine which registers you would have to change at runtime to move that offset. You could then use the raw register writing capability of the IMAQdx driver to change that. Potentially you could also put the camera into a mode (if supported) that it will take one frame per software trigger based on a write to a software trigger register. Your application could then trigger the frame signal after changing the ROI, then Grab the resulting image. The advantage to this approach would be that there would only be a few register writes and no hardware/software reconfiguration necessary for each iteration.

Hope this helps,

Eric

[lcarlson]

Hello Eric,

After a long hiatus waiting for the camera to arrive from Germany, we procured our Basler pia640-210gm GigE camera.  The camera shares a GigE switch with a real-time OS computer and the host.  The host computer, running LV and Vision 8.0, has the Intel Pro/1000GT ethernet card and I've followed all the recommendations such as jumbo frames, etc to get the highest bandwidth.  (On a side note, is there any better way to check on the network activity besides hitting control-alt-delete and watching the network usage?  This seems to "read low" - I see only ~10% usage when streaming images at 210 fps...supposedly ~60 MB/s, or roughly half of the GigE bandwidth...)

My objective now is to MOVE or reposition the area of interest on-the-fly like you suggested.  I can set the AOI to ~100x100 pixels ahead of time but the spot on the camera that I am trying to capture can move +/- 1mm, so the AOI must follow it.  At time= -150 ms from camera trigger, I will have a rough idea as to where the spot will be on the sensor.  I must then move the AOI to that approximate location, either by changing the property node and resetting the camera, or writing directly to the camera's registers.  I would like the AOI "set" and ready when the hardware trigger comes into the camera at t=0.  Then I have a simple centroid-finding task to complete in ~2ms.

I've attached a trial vi which tries to reset the AOI parameters as fast as possible.  These initial trials take ~250 ms to cancel the old parameters and then write new ones.  The ideal time would be <150 ms from when the information is available to when the camera is able to accept a trigger.  Perhaps this isn't the best way to do it.  The "unconfigure acquisition" is necessary to remove the old parameters or else the new ones can't be written, but it takes too much time.

Is there available a look up table of all the registers for GigE?  Maybe I am getting confused with IEEE 1394, but I believe there must be a base register address to which attributes are added.  I like your idea of changing the raw registers but cannot find any of this in the documentation.  I know that the AOI can be moved while capturing because it does it quite quickly in the Basler Pylon viewer with a manual slider bar (the dimensions of the AOI are fixed).

I appreciate your help,
Lcarlson

[BlueCheese]

Hi Lcarlson,

With that Basler camera you can definitely do what you want. Their camera properly exposes the OffsetX and OffsetY in a way that they can be safely moved while the camera is acquiring without invalidating the acquisition parameters. I have tried a simple prototype that was able to move the AOI while acquiring on a Basler Scout in approx 2ms.

Unfortunately, IMAQdx currently blocks write access to all the AOI controls including the OffsetX/Y while acquiring since (if I remember correctly) some earlier cameras from some vendors allowed those attributes to do things such as coercing the Width and Height and making the acquisition parameters invalid. I will make a note that we should re-evaluate this decision and possibly allow those features to be written at all times. If the camera's XML description file written by the manufacturer is designed properly it should work correctly as the Basler one does.

To get around the IMAQdx locking right now you have a couple of options. You can write directly to the register the value lives at, but this can cause some problems since there can be caching issues with the interaction with other attributes as well as no range checking involved. To determine the register address requires some hunting through the camera's XML file as well, and it is not guaranteed to have consistent behavior across camera firmware versions. A better option is to add an additional attribute to the camera's XML file that is identical to the OffsetX/Y but has a different name. This will get around the blocking that IMAQdx does and allow you to change the feature while acquiring using the normal attribute configuration methods. The modifications to this file are somewhat complex to describe, and I don't have your particular Pioneer model available, so if you could go to the C:\Documents and Settings\All Users\Documents\National Instruments\NI-IMAQdx\Data\XML directory and attach the XML file from the camera that we cache in that directory to this forum post I can make the modifications for you.

-Eric