Stereo library 2012 pointers

[Klemen]

Dear programmers,

I am trying to evaluate the new Stereo Vision Library and since I am fairly new at this, I would greatly appreciate an advice or two.

My (simple) setup is:

- Two DSLR cameras (different models, Cannon 50D and 350D)

- Different lenses, but the focal lengths were the manually set to 35 mm

- Same lighting (outdoor)

- Baseline distance ~200 mm

I calibrated single cameras using a camera model and also corrected perspective distortion. I used a grid shown above in 20 different postitions. Here, I had no problem.

A static scene was considered, and the cameras were set on 10 s timer.

Moving on, I used the stereo library to calibrate the stereo system and saved the stereo calibration data.

 I used the semi-global block matching alghoritm to find the disparity image. Based on the equation

where f = 35 mm and b = 200 mm.

I calculated the minimum disparity and number of disparities for my working distance (approx. 500-3000 mm) -> min disp = 0, number of disp. = 16.

Using a small window size 7x7, I calculated the disparity for the following acquired images (an example of acquired image pair):

                                             Left image                                                                                                      Right image

The disparity image i get (non-interpolated) is:

The depth image isn't calculated (blank image) for some reason.

What am I doing wrong? I know I made some errors by my simplified system (the cameras are not fixed, the optics is not the same etc.), but what is wrong with the resulting disparity?

Is the problem in the  vertical misalignment of the images? From what I can gather, the biggest problem is the texture matching. I expected more segmented images (more areas of equal depth). Is there not enough texture to match both images?

If i try the example which is shipped with the Vision Module (Stereo Vision.vi) it works ok.

Please help with an explanation. And if I made a mistake, please be patient with me.

Thank you and best regards,

K

[Klemen]

I am wondering if the problem is in the different field of views of the two cameras?

[ValkoA]

Hi Klemen!

    Indeed, from what I see here, the calibration of the two image sources did not succeed, so the disparity image generator gets false data about the alignment of you cameras, and so the resulting disparity  data is unusable. Could you link in some calibration images as well? With those I could simulate your setup.

    Also, you've mentioned that you tried the example VI. Could you test your own data using the example VI? To do this you need to make a copy of it, the examine the setup

of the National Instruments\Vision\Examples\Images\Stereo Vision  folder, set up your data in the same alignment, then change the inport location of the block diagram from this

to a path of your own folder. You can also use the images of the example to test your own code with.

Kind regards:

Andrew Valko

National Instruments

[Klemen]

Dear Andrew,

thank you for the answer.

I have already tried the steps you recommended. The example images shipped with the Stereo Library produce the same result with my program as in the example.

I am linking the calibration grid images, some test images and the code. There you will see two different "Top VI's". One is for calibration, the other is for acquiring the depth image.

Hope you can offer some assistance. It is obvious that I am making some mistake, just don't know at which stage...

Thank you very much,

K

[ValkoA]

Dear Klemen!

 I've been checking your code and so far, it appears that the calibration data it generates is incorrect. If you check the calibration output with this VI:

you will find that while the output matrices seem to be correct, but the Rectified ROI areas both return the whole image as opposed to the section that is visible from both cameras. This could definitely cause the blurred distortion image we got, because the two coordinate systems for the two images are not linked to each other. I'm not exactly sure as to what causes this behaviour, but I'll look into it in more detail tomorrow.

Aside from this, your depth image is calculated, but you have to put the indicator (graph) inside the event structure for it to display correctly. You can also play around with some of the settings, see what works for your images.

Best regards:

Andrew Valko

National Instruments

[Klemen]

Hello Andrew,

thank you for this. I see what you are talking about... This is indeed very strange - moreso because this happens also with the example Stereo Vision.vi. The rectified ROI's are the same and are equal to images size.

I haven't had time to further investigate this, since I am fighting battles on multiple fronts now.

But I will get to it, since this interests me very much.

I really appreciate your help and would love to hear what you found out.

Best regards,

K

[ronak25]

Hi Klemen,

   I have checked the grid images you have provided. While grid images are good, they are flipped. That is, Left camera grid images should be right camera grid images and vice-versa. Also, for object images for which you want to measure 3D, you need to flip left to right and vice-versa. Using your grid images (by flipping) I could get a good calibration (0.97 quality) and I am passing you the stereo calibration here. Kindly find it attached. Use this calibration file and capture some more object images with more texture (yes the images that you captured had a lot less texture, e.g. in clouds) for the same set-up and you will get 3D for that. I could get some values here myself. Try out and let me know how it goes.

Regards,

Ronak.

[Klemen]

Dear Ronak,

thank you for the answer and the time you took to test this.

Could you please elaborate on the calibration part a little bit more (did you use my attached VI's)?  What parameters/settings did you use to obtain the depth image? I tried your calibration file and flipped the acquired images, but to no avail. It is strange that the depth image is reduced in size (600x400 instead of original size of 3456x2304). What calibration parameters did you use?

I know there is not enough texture in the clouds, but I was focusing on the objects I've put in front of the cameras (the box, etc...). I thought that there was enough texture there???

It would really be helpful to attach some VI's with default settings that work.

This is really getting to me. It seems I do not understand this in the slightest...

Best regards,

K

[ronak25]

Ohh.. I forgot to mention that I reduced image sizes to 600x400. My fault. Also, I made them gray-scale. I am passing your images (rescaled and gray-scaled) in the attachement (Grid as well as object images, again for this set also make left as right and vice-versa). Use them. For calibration, I did the same thing (rescaled and gray-scaled the grid images). Finally, I used the Calibration Interface to calibrate the images. I passed the stero calibration file to you in the last response; use that along with the images I have sent. You will then have to change the number of disparities and see whether you can get the disparity and depth properly or not. I have tried here but due to different sensors, responses are different. So, while it gives something; it is not too clear. For example, for image-set 18, I am getting depth of the table starting from 57 inches to ending at 76 inches, if I take distance between two dots in your grid as 1 inch. I would recommend the following things for your future endeavours:

1). If possible use the similar sensors (similar cameras).

2). Capture smaller images (600x400) for your convenience.

3). You do not require 20 grid images. 6-7 grid images with sufficient translation and angle coverage should do.

4). Work in monochrome mode.

5). Use textured images (or use patterned lighting, if possible).

While it is slightly involved to setup stereo, it is very easy thereafter. So, bear with the process for now. Typically, for an informed user (having already set the system in past), it takes 15-20 minutes to set-up, capture images and calibrate.

Regards,

Ronak.

[Klemen]

Thank you for this explanation and the attachments! I appreciate it greatly... I will try this today or tommorow.

I know the diffrenent sensors are causing some miscalibration. Since this was just a test I tried to make this as simple as possible (my setup was very crude as it can be seen from the first post). I used DSLR cameras for single reason only -  to obtain higer accuracy via a higher pixel resolution.

By monochrome you mean 8-bit right?

Thank you for the recommendations and pointers. I will get back to you after I test this some more.

Regards,

K