User Tracking with LabVIEW and Kinect based on the OpenNI Interface

[Naity]

I found a way to get the user recognition to work in  LabVIEW using Microsoft Kinect. It is based on the OpenNI interface with PrimeSense  middleware components and drivers. A short video presenting some of the  features is available:

To get it working, you will need to install the  previously named components. For an optimal use of the functions, it is better  to have NI Vision in addition to LabVIEW 2010 SP1 (or higher), although all data  can be read and used in numeric form, allowing its use by people not owing an  NI Vision license.

I -  Installation

You will need to download the following files in order  for it to function:

[1] The Kinect Drivers compatible with PrimeSense: https://github.com/avin2/SensorKinect

[2] OpenNI latest installation file: http://www.openni.org/downloadfiles/opennimodules/openni-binaries/20-latest-unstable

[3] PrimeSense NITE Middleware installation file: http://www.openni.org/downloadfiles/opennimodules/openni-compliant-middleware-binaries/33-latest-uns...

[4] PrimeSense Sensor installation file:

http://www.openni.org/downloadfiles/opennimodules/openni-compliant-hardware-binaries/31-latest-unsta...

/!\ Note that OpenNI drivers can theoretically not be  installed while the official Kinect SDK drivers are. You can try to get both  working together looking at this  discussion in the OpenNI discussion Group, but it seems that the  method does not always work. If not, you will need to remove the Kinect SDK  Drivers before installing the OpenNI ones.

* Assuming you have the Kinect device plugged in and  that you have no drivers installed on your computer, unzip the file downloaded  in [1] and then change to the device manager. You should only see the Kinect  motor as unknown device. You can manually look for its drivers in <path to the [1] unzipped  folder>\Platform\Win32\Driver\. I could only test on x86 systems, thus I  do not know if it works on x64 systems as well. Repeat the process for Kinect  Camera and Kinect Audio once they appear. At the end it should look like the  following:   * Now install OpenNI (file downloaded in [2]). This is  a simple Microsoft installer.

* Repeat the process with the PrimeSense files  downloaded in [3] and [4].

* Once those steps are completed, install the  PrimeSense Sensor interface for Kinect. It can be found under <path to the [1] unzipped folder>\Bin\.  This again, is a simple msi file.

* To verify if the installation has been successfully  completed, try to run some of the OpenNI examples. The default path to them is C:\Program Files\OpenNI\Samples\Bin\Release\.  You can run NiSimpleViewer.exe which displays the depth information of the depth  sensor or NiUserTracker.exe which allows testing the user tracking. The  calibration is done taking a pose looking like the Greek letter PSI as shown in  the picture below.

   

* Finally, try the UserTracker.net.exe from the same  folder to verify that the .NET assemblies have been correctly installed, as  they will be used in LabVIEW.

If those examples do not work, it can either mean  there is a conflict with another driver, that the installation has not been successful  or that you have a graphic card that is too old/you have an integrated graphic  chip (it does not work with all of them).

II – Tracking  Users in LabVIEW

Now that the device seems to be working with OpenNI,  you can try to make it work with LabVIEW. The project I did can be downloaded  at the bottom of this document. Extract it somewhere and open Kinect  Wrapper.lvproj.

The important parts of the project are the Examples  and Functions folders. In the Example folder you can find ready-to-use VIs that  are programmed with the ones from the Functions folder.

*  TestDepth.vi: This VI allows easy testing of  the depth sensor of the camera. It returns the middle point of the depth image.  You can just move away or wave in front of it to see it move. Note that the  sensor can do distance measurements starting from 50cm only.

*  TestMap.vi: This VI gets the depth map and  displays it in a 3D graph. It runs slowly because of the 3D graph which needs  time to display its 640*480 points. If you own NI Vision, it can be replaced by  DisplayDepth.vi for better performance.

*  TestImage.vi: Returns the image of the  Kinect’s camera. Requires NI Vision.

*  TestMultipleUserTracking.vi: The example where  the tracking is shown. You will need NI Vision if you want to visualize the  tracking. If you do not have it, the data should still be available but not for  visualization. You will then have to delete the variable_image.vi.

Note: This is all based on OpenNI .NET Assembly. It  means it should also theoretically work with Asus Xtion instead of Kinect. I  could not try as I do not own the device. If somebody tries it, please let me  have your feedback.

Overview of one of the VI (clic to enlarge):

[Elmar_NI]

I like it

[Ben_J1]

This is awesome. Who wrote this?

[Naity]

I did . Do you have questions regarding it?

[Ben_J1]

We have some guys in IMAQ R&D who want to email you to follow up. So what is your email?

[KrisnaWisnu]

This is awesome, thanks for sharing. I actually ran into this error after following the steps:

   One or more of the following nodes could not be enumerated:

   Device: PrimeSense/SensorV2/5.0.0.25: The device is not connected!

The solution that I've found is by installing <path to the [1] unzipped  folder>\avin2-SensorKinect-2d13967\Bin\SensorKinect-Win-OpenSource32-5.0.3.4.msi

The last problem that I had, running TestDepth.vi throw me error 1172 (some dll cannot be found). Apparently, I need to restart my computer!

[Naity]

Hi Krisna

Thanks for the report. The first point is normal, I described it in the docment, and without the SensorKinect sensor fil from PrimeSense it can not work.

Your second point it quite weird. Did it work after restarting? Can you nameme the DLL which could not be found? And which OS are you working with?

Thanks

[NClark]

Hi, this looks very good, but I have issues getting it to work.  I installed OpenNI and drivers and all the provided examples work, but when I try to run the Labview project's examples, it asks for OpenNI.net.dll.  So I directed it to the dll in the OpenNI folder.  It then loads fine, but it is full of errors (the run command is broken).  It looks like it might be one little thing wrong that is causing multiple vi's to error (the function vi's contain the problem errors that make the examples unrunable), but I cannot find what the problem is.

For example GetMap.vi contains the following types of errors: Property Node: Invalid property, Invalid refnum class, Invoke Node: Invalid mehod, Invalid refnum class

Unfortunately, I have no experience with USB devices.

[NClark]

Nevermind, I got it to work by using the x86 files instead of the x64.

[Naity]

Hi Clark.

It seems that you are right. I was however unable to track down where it does come from. I wrote it with windows x86 and tested also on widows x64 with LabVIEW x86 and OpenNI x86. That way it should work.

[Bmvalarezo]

Congratulations and thanks for your help

[Naity]

Hi.

Thanks for sharing the result of your development. I'm glad to see my code is of any use

[Tobiey]

Hi.

I have finally got LabView to run the example VIs. But when I run them I just can't see any data. It feels like the screen freezes and when I press stop it shows this error:

I'd really like to get the VI to start working.

I had the same problem as NClark at the beginning when using x64 but changed to x86.

[Naity]

Hi,

Thanks for posting your screenshot. The first question I would like to ask is did u ever had the Official Kinect SDK installed on your computer or only OpenNI? If only OpenNI, could you open the subvi in highlight execution modus and tell me at which node does the error happen?

Oh, and before all thatm do you manage to run the example delivered with the OpenNI library (like NISimpleViewer and NIUserTracker at C:\Program Files\OpenNI\Samples\Bin\Release\ ?)

[Tobiey]

I've istalled the Official Kinect SDK earlier, but I unistalled the drivers and installed Primesense drivers.

If the picture got too poor quallity. The error came in the sub VI CreateContext.vi at the second Constructor Node (Context) path to configuration -> xmlFile. The following error occured ERR(1172).

I can run all the Open NI examples that stood in this tut.

EDIT :

I just found my problem. I was so eggerd to get started with the Kinect so I didn't read the comments. I just had to change the directory for the SamplesConfig.xml in te block Diagram.

Thank you for the quick replys and I'm sorry for being soo eggerd.

[Naity]

Tobley,

I tried to make a new installation on a computer to see if everything is still working with the new openni interface. Poit is that it is not. The last version of the unstable build looks like not supporting Kinect. I could only advise you to uninstall OpenNI, NITE and PrimeSenson from your computer and thwen download the stable build of the OpenNI Package (The OpenNI Package contains all 3 useful installers in one file) as shown on the following picture:

Thanks for reporting this to me, I'll probably edit the article in the upcoming days to write this incompatibility.

Regards

EDIT: We cross posted. I didnt had time now to try with my labview library and probably won't today. I'll keep you up to date.

[Naity]

So I just tried. The wrapper still works fine for me. So if it still doesnt work after you install the last stable version...  I hope it is not some leftovers of the SDK to stop compatibility with other drivers. I'll tell you if I manage to reproduce it. Can you also keep me up to date on your status?

Thanks

Regards

[prisonerhu]

I have a same problem (Tobey - err1172)

But after the driver downloading, doesn't work the context.

Which SDK do you prefer? I have this (officially) 32 bit (my windows is the same)

I downloaded many SampleConfig.xml files, and I tried with many path, but the result was the same.

Could you send me the perfect xml file?

Any Idea, what the problem is?

Please help!

Thx!

[Naity]

Hi Prisonerhu,

There is two things: First, I think you can not really have the Kinect SDK and OpenNI working on the same comp (some people found a workaroud but it doesn't work on all computers). Beside I would avise you, if you did not, to download the last stable version of OpenNI and NITE. I tried to reinstall the whole system with the last unstable drivers and I was experiencing the same problem. Last of all, try with the examples delivered with the OpenNI drivers. (C:\Program Files\OpenNI\Samples\Bin\Release\.). If Simple Viewer does not work, it means than sompething went wrong with the instalaltion of the drivers or the recognition of your device.

I tried to run a new installation on a blank system 2 weeks ago and with the latest stable drivers it worked like a charm.

[prisonerhu]

Thx the fast response!

I tried that you said. The all driver and sdk's was uninstalled, after a install in order. But the Simple Viewer doesn't work: (One or more of the following noed couldn not be enumerated: Device: PrimeSense/SensorV2/5.0.5.1: The device is not connected) - The kinect was connected (I'm sure, because the sdk-s sample working - not perfectly, but work)

I don't have any idea. You?

[Naity]

I tried to make a new installation on another comnputer but it worked without problem. I found it quite surprising.

If the simple Viewer doesn't work it means that even before the LabVIEW code, the OpenNI interface does not work. An idea I am thinking about is, first, to verify in the device manager than the device is registered as on the picture of the DevZone article. If you only see Kinect Motor but not Kinect Audio and Kinect Camera (or written differently / showned as a question mark) run the driver file again ( https://github.com/avin2/SensorKinect then <path to the unzipped  folder>\Platform\Win32\Driver\ ). If everything look alright on that side, verify that u also installed the specificNITE Sensor compatibility drivers for Kinect (from the same archive, under <path to the unzipped folder>\Bin . You can also tryx to uninstall all the openni and primesense programs installed on your coöp and run it again (dont't know if that last point will really help, but windows is known as being a moody OS, it doesnt' cost more than 15 mins to try).

Regards

[MicMac89]

I followed instructions described in this document and unfortunately I cannot run this cool project. I've downloaded and installed the newest version of files and installed them in correct order. I got the following error:

Error says that it could not find an entrance point of procedure ?V... in library XnDDK.dll. Do you have any Idea how to solve this problem. Kinect is properly installed I can see it in device manager. Perhaps I need older instalation files of openni and primesense.

I did it on computers with x64 (Windows 7) and x86 (Windows 7 but in on this system I had problem with Windows SDK also) and XP x86. It's important for me to solve this problem because I would like to use it on robotics competition on Saturday

[NClark]

Hi again,

I got the Kinect data working in my project and all was happy, but this Monday LabVIEW suddenly decided to stop giving me Kinect data.  I literally ran it fine, changed a couple of int values in my code, then ran it again at which point it ceased to give me joint coordinates.  I tried a reinstall and I get the same thing.  The examples that come with OpenNI work just fine, but when LabVIEW tries to use the Kinect, I get nothing.  For instance, the TestImage.vi just sits there with a white screen when I run it and TestMultipleUserTracking.vi stays "waiting" forever.  I am just confounded about what is wrong especially since I did not touch the files that retrieve the data.  Any ideas on what might have happened/how to fix it?

Thanks

[jimmy1234]

I am running LV 9. Is there anyway you could save the above files so I am able to open them. I get a message "File version later than the current labview version".

Thank you,

Jim

[Naity]

@MicMac89: We have already discussed your issue on the forum. Is it still working?

@NIClark: This behaviour is weird. I had the programm running on a daily basis for over 2 month and never had such an issue. I had similar behaviour only in the early steps of the development but they were supposed to be resolved on the published version (and when it used to happen, a reboot was usually enough to solve it). Is it still hapening?

@Jimmy1234: I tried to save the project for LabVIEW 2009. Can you confirm me weither it worjks or not?

http://www.mediafire.com/?50tw4soenhqtqbc

Regards

[NClark]

Rebooting had no effect.  To get it running, I had to reinstall it using old PrimeSense Sensor interface for Kinect installer along with the newest version of OpenNI.  It is fixed now, but it really still baffles me too.

[Naity]

I am glad you got it running. If you have some time, could you document it a little (which LV version, .net framework, openni version and primesense drivers). I would like to try to reproduce it because I found it baffleing too (especially than the code is not supposed to be  more than a translation of the openni examples). I am sorry for the behaviour aniway.

[NClark]

I am using LabVIEW 2011 on a 64-bit windows machine.  I used the current (got them Friday, Dec 16), unstable, 32-bit OpenNI binary (openni-win32-1.4.0.2-dev), middleware (nite-win32-1.5.0.2-dev), and hardware (sensor-win32-5.0.5.1-redist) modules.  After that, I installed SensorKinect-Win-OpenSource32-5.0.3.4 (from an old download of the Kinect Drivers).

[CyGa]

Hi Naity,

Does the code work when running in a VM ?

I followed all instructions noted, read careful all comments, tried all tha was advised... but it stil doesn't work ! I'm trying to use your code and OpenNI binairies in a virtual machine (Win7 x64 ; LV2011). When running LV VIs it generates the error 1172 when building the context (constructor Context with the path to the XML File as an input)... And when lauching .exe from OpenNI samples it says that Kinect wasn't discovered (even if I can see it well detected in the Device Manager).

Any idea ? Would it come from the VM ?

[Naity]

.

[Naity]

Hi Zyl,

I have not tested it so far in a virtual machine. If the OpenNI examples do not work either, the problem comes from there. However, I ran into the same problem yesterday  and it was caused by... the cleaning lady who unplugged the device while cleaning (with the USB powering the LED I didn't realised at first).

In the device manager, do you only see Kinect Motor or do you also see Kinect Camera / Kinec Audio ?

EDIT: sorry for double post

[CyGa]

From the device manager I can see the Kinect Motor, Camera and Audio in the PrimeSense node...

Each time I'm trying to launch one of the samples it says that "The procedure entry point xxx (too long to name it here ) could not be located in the dynamic library XnCore.dll"

Activating the log in the SampleConfig.xml just lead into a txt file saying the XnLog.cpp was well started...

[Julio_HDT]

HI Nclark, could you resolve this problem,

"Hi, this looks very good, but I have issues getting it to work. I installed OpenNI and drivers and all the provided examples work, but when I try to run the Labview project's examples, it asks for OpenNI.net.dll. "

I have the same it, I cant fix it, i have dll.

[NClark]

Hi jchdt,

What version of OpenNI are you using?  When I had this problem, I resolved it by uninstalling the 64 bit software and drivers and replacing them with the 32 bit software and drivers.  That is about all I can suggest.

[Julio_HDT]

i'm using "openni-win64-1.5.2.23-dev" with labview 2011 and windows 7 with 64 bit system

[CyGa]

I confirm that it's not working in a virtual machine. I finally installed everything on the native OS of my computer, and it works fine.

[camille31]

Hi,

I'm a french Student in Network and Telecommunication. I have to make a project with Labview and Kinect. I saw your project and i want to try it to see how it works because i'm a novice and I want to learn how this software works.
But I've got a problem. I've 2 errors. I take two screenshot to show you these errors. Can you explain me these errors please? Thanks.

[Bmvalarezo]

Hi, in windows vista 32 bits it's OK.

But I install in Windows 7 of 64 and labview of 32 bits but i can not work, I install OPEN NI of 32 and 64 bist but don´t work. the driver of 32 bits don´t install, please help.

[toto26]

Hi,

I've found a way to use the motor in parallel, just install the CodeLaboratories driver http://codelaboratories.com/downloads/ for the motor only.

You can then access the function through the CLNUIDevice.dll

Is there a way to use the Gesture Recognition function with LabVIEW?

Regards

[Naity]

@ JC HDT:

Unfortunately the code does not seem to work with the 64 bit version of the OpenNI Driver. You should use the 32 bit.

@Bmvalarezo:

Same that JC. Try to uninstall the 64bits drivers are they are not gonna work in LabVIEW. I ran some test on Windows 7 x64 with LabVIEW 32 bits and OpenNI 32 bits and it work without problem. Maybe uninstall everything first (OpenNI, the different Prime Sense components and the drivers in the device manager) and reinstall it. Then perform a test running their examples to see if it works.

@Camille:The first thing I would advise is, again, to check if the installed version of the OpenNI components and drivers are the 32 bit ones. I would also run the delivered .net based examples to determine where the problem comes from. A new installation of the OpenNI Components sometimes work (if you read those comment through, NClark had a similar problem once who appeared without reason and disappeared without more reason after reinstalling the drivers).

@toto26: Thanks for sharing. However, what motor are you talking about?

Thanks all for your feedbacks

[toto26]

@Naity:

There's a motor in the Kinect base that allows to point up and down the device.

Then the dll I talked about allows to access the integrated accelerometer datas and to control the LED in front of the Kinect.

Have a look at my post: https://decibel.ni.com/content/docs/DOC-20563

[Naity]

Oh, this motor!!! Awesome .

I have problem getting it to work so I let a message in the comment of your community message.

[Jeyminee]

Still trying to make it work...Win7 64b with OpenNI fullPackage 32b (NITE 1.5.2.21 / Sensor 5.1.0.41 / KinectMod 5.1.0.25) and LabView 2011 32b

Installed and uninstalled several times with different version of the programs and the drivers.

I have the kinect and the xtion pro too.

It's running but i have no image... Ni Vision Acquisition and NI Vision Development installed.

It's like LabView doesn't try to communicate with the cameras cause i don't see the IR...

[Naity]

Still weird. I retested yesterday after updating to 2011 SP1 and it still works.

Are you able to see it using the examples from OpenNI (the .NET based ones).

Beside I wrote the drivers so that vision is not necessary to communicate with the camera (the IR sensors examples do not use it and the tracking one use it only for having a visual feedback but it can be let aside and its not necessary to get the array of articulation).

Beside, I have been observing lately some problem with the communication between kinect and the unstable version of the OpenNI drivers. Maybe it comes from there (and I could never try with ASUS xtion. If you get it to work with Kinect, could you try and tell me if it works with xtion too?)

Best regards

[trojanbrandon]

We're having an error saying that the expected assembly is OpenNI.net version 1.3.2.3, but we have OpenNI.net version 1.5.2.23. Do you have any solutions?

Thanks

[patbarnes1989]

Thanks for this software, a group and I got it to work great; however, we are working on a project where the Kinect cannot be stationary and has to rotate back and forth.  Through testing, we found the tracking works great when the Kinect is stationary, but when it starts moving, the tracking program constantly goes back to the "Waiting" and then "Calibrating" screens and does not work properly again until the user clicks the reset button.  For our project, we need to ensure tracking remains continuous, is there a part of the code we can modify to make the tracking less sensitive and make it more difficult for the tracker to want to go to the "Waiting" and "Calibrating" modes?  Thanks!

[Rea_Rea]

I am having major trouble installing this.  I am using Parallels on a Mac running 64 bit Windows 7.  I cannot get anything to work.  I have followed your directions, but I seem to be confused.  Also, I cannot see any VIs in this path:C:\Program Files\OpenNI\Samples\Bin\Release\.   If you have any suggestions please let me know.  Thank you.

[carter12s]

Hey,

I can verify that this works perfectly with the Xtion Pro Live. I am having some errors with with labview releasing references and have to frequently restart my computer to keep it working, but out of the box you can get data with this code from the Xtion.

-Carter

[Rea_Rea]

@Naity I cannot download the Kinect Drivers compatible with PrimeSense.  I followed the link and clicked dowload as a zip file but there is a downloading error every time.  What should I do?

[slivse]

Thanks, i´ll show you my final alication.

[djj790]

Nice work Naity! Has anyone calibrated the position coordinates to be given in meters (or inches)?