How to align Depth image and RGB image from Kinect.

[AnkitRamani]

Hello, 

I am facing one problem regarding to Picture alignment. I am using Kinect Xbox 360 and Kinesthesia tool kit to get Depth array and color array.

Depth array is Unsigned 16 bit Integer and RGB array is Unsigned 24 bit integer.

Here in attachment I have attached screen shot in which I am trying to detect one object and after detecting that object, I am getting its coordinates and then using the same coordinates from RGB image I am drawing rectangle in Depth image to just check whether both image has same ROI or not.

But unfortunately both are different.

Please guide me how can I solve this problem.

Thank you very much in advance.

[JGoebel]

Hello AnkitRamani,

can you please attach the code of the VI used in the screenshots? If this VI continously aquires pictures from the kinect, please modify it to read the two pictures from two files (best is to use diagram disable structure to replace the kinect-driver-VIs).

I guess you just have a constant offset in your depth-array. Maybe some parameters for the driver are set wrong.

I will have a closer look as soon as i got the code.

Best,

Jan Göbel

Applications Engineer

[AnkitRamani]

Hello Jan Göbel, 

Thank you very much for your reply.

Here in attachment I have attached the code.

This vi continously aquires pictures from the kinect. If possible then please explain me little bit more to do mapping from RGB image to depth image.

There is direct function like MapDepthFrameToColorFrame,  but I have no idea how to implement with Labview.

Thank you once again.

Regards, 

Ankit Ramani.

[JGoebel]

Hello Ankit,

thanks for your quick reply.

I just had a look at your code, and though, there could be some optimization, I don't see anything that could cause this offset.

Is the offset constant?

Right now I guess, that the kinect just does not aquire the two pictures syncronized, and you might need to be adjusted. This would explain the MapDepthFrameToColorFrame-Function.

I see you are using the kinect toolkit, wich is a 3rd-party toolkit and not supported by NI. Nevermind, if this toolkit contains a VI "MapDepthFrameToColorFrame" feel free to attach this VI, and I will have a look at how to use it in LabVIEW.

If you are talking about the MapDepthFrameToColorFrame-.NET-Function, you can use the .NET-invokenodes in LabVIEW for using it. See this tutorial for more information on how to do so:
https://decibel.ni.com/content/docs/DOC-9138

You can also call the assemblies with the DLL-Node:

http://digital.ni.com/public.nsf/allkb/DCB90714981A1F148625731E00797C33

Best,

Jan Göbel

Applications Engineer

[AnkitRamani]

Hello Jan Göbel, 

Thank you very much for your reply.

Actually there is no "MapDepthFrameToColorFrame" is provided by toolkit.

But If we go to the Event Depth and Video vi, from that there is depth frame call back function and right click on it >> create >> Method for MicrosoftKinect.Kinectsensorclass and from there we have lots of option regarding to the depth frame.

and from there there is call back functin called MapDepthFrameToColorFrame and mapDepthToColorPoint. I tried to use it. But somehow I am not getting accurate mapping. I hope this might help you to suggest me something, so I can access the mapping from RGB to depth or Depth to RGB.

Thank you very much once again for your time.

Regards, 

Ankit.

[JGoebel]

Hello Ankit,

thanks for the screenshot. What you are seeing there is the .NET-API for the Kinect Sensor. Sadly this is providet by Microsoft, and I don't have detailed knowledge on how to use the driver functions. I found the documentation for the method here:

https://msdn.microsoft.com/en-us/library/microsoft.kinect.kinectsensor.mapdepthframetocolorframe.asp...

but this might not be enough documentation for using the method.

Are there Examples providet with the Toolkit to see how to use the function?

Best,

Jan Göbel

Applications Engineer

[AnkitRamani]

Hello Jan Göbel, 

Thank you very much for your reply.

Only one example is provided by toolkit provider, that I have attached here in attachment.

Regards, 

Ankit.

[JGoebel]

Hello Ankit,

sadly this problem propably combined to the construction of the Kinect Sensor(s). So there will be no way around the Mapping-Methods. For how to use them you propably will have to contact microsoft support for kinect.

Another idea would be, if the offset is constant everywhere on the picture, just to remove the offset manually. If the offset changes depending on the position on the picture, this might get a little difficult.

Best,
Jan Göbel

Applications Engineer

[AnkitRamani]

Hello Jan Göbel, 

Thank you very much for your time and effort.

I will try on your suggestion regarding to the offset.

Regards, 

Ankit.