Using Microsoft Kinect with myRIO (update)

[xkingkiller]

This guide will describe how to use Kinect on myRIO through the USB port. This involves setting up libfreenect libraries on the myRIO, and testing it with an example project. This library only contains basic APIs to read raw RGB image and depth image from Kinect. Other functions such as audio, 3D skeleton abstraction are not included.

Instructions:

A. Download the attached 2 zip files.

     The "KinectLib.zip" contains libfreenect library files and the "Kinect Project.zip" contains the example Kinect VIs to use the library to read data from Kinect.

B. Setting up libfreenect Library.

     Requirements:

     In order to complete this part of the tutorial you will need software to SSH into your myRIO. This example uses PuTTY. You will also need to use SFTP to transfer files to the myRIO. This example uses FileZilla.

     Steps:

     1. Connect myRIO through its device USB port to your PC.

     2. Enable SSH. Go into NI MAX and find your myRIO under Remote Systems item in the left pane. Browse to your myRIO. In the System Settings tab of the device, under Startup Settings, there should be several checkboxes. Enable the box to the right of “Enable Secure Shell Server(sshd)” if it is not already. Hit Save.

     3. Connect to the myRIO with SFTP. Username is 'admin' and Password is left blank.

     4. Copy over libfreenect.so.0.1.2, libfreenect_sync.so.0.1.2 and libkinect_wrapper.so in /user/lib folder on your myRIO. And copy liblibusb.so.1.0.8 to /usr/local/lib

     5. Open an SSH connection with the myRIO. User is admin and password is left blank.

    

     6. Create symbol link to these files through the following commands in your PuTTY.   

     =============================================================

     #ln -s /usr/lib/libfreenect.so.0.1.2 /usr/lib/libfreenect.so.0.1

     #ln -s /usr/lib/libfreenect_sync.so.0.1.2 /usr/lib/libfreenect_sync.so.0.1

     #ln -s /usr/local/lib/liblibusb.so.1.0.8 /lib/libusb-1.0.so.0

     =============================================================

     7. Reboot your myRIO and your libfreenct library should be ready to go.

C. Run Example

Software Requirements:

• LabVIEW 2013
• LabVIEW Real-Time 13.0.0 - NI myRIO 13.0
• Kinect Example VIs (attached)

Hardware Setup:

Connect your Kinect to the myRIO USB port and powered it by the wall adapter. Also, you can power Kinect with a external 12V DC. To make sure your Kinect is working properly, again SSH to your myRIO and type the following commands to check whether all Kinect USB devices are correctly mounted. And you should be able to see three USB devices in the list: "Xbox NUI Audio", "Xbox NUI Motor", "Xbox NUI Camera". If you cannot see them, make sure your Kinect is correctly connected.

     ====================================================

     #cat /proc/bus/usb/devices

     ====================================================

Software Overview:

In the included project, there are two top level VIs which can be run after the Kinect is connected, and the libraries are installed on the myRIO.

Kinect Depth Controller has controls for the Depth Resolution and Depth Format, as well as the LED option and Tilt Degree. All depth formats and resolutions should work (NOTE: resolution and format have to be set before the program is run, and don’t have effect while the program is running). Tilt degree controls the motor inside of the Kinect that rotates it. LED option controls the LED on the front of the Kinect.

Depth resolutions: Tiny (160x120), Low(320x240), and Medium(640x480)

After Init Kinect Systems has run, any of the Vis that read depth or video data from the Kinect should work (exceptions for video resolutions noted below). It is important that Close Kinect Systems is called (use stop button, don’t abort); if it’s not executed, on next execution of the VI the myRIO will lose connection immediately. However, running it again after the disconnection will work.

Kinect Video Controller has the same controls as Kinect Depth Controller, but this time the Video Resolution and Video Format controls will take effect (again, these must be set before executing the VI). NOTE: as of right now, none of the video formats work in Low(320x240) resolution, all of the formats work in Medium(640x480) resolution, and only RGB and Bayer formats work in High(1280x1024) resolution.

If you will be doing image processing, or converting the depth to physical units, the projects on Obstacle Avoidance and Color Following with the Kinect have examples of this. Basically, you will want to modify the Read Video or some Read Depth VI so that you get the 2D array of points instead of the picture. From there you can convert it to an IMAQ image and process it with IMAQ functions.

The lower resolutions for both Video and Depth are much faster than higher resolutions. In the other projects, we’ve been getting about 15fps on tiny depth resolution, and ~3fps on medium resolution video.

Kinect VIs & Wrapper library

Credit for the Kinect VIs and wrapper code goes to Vicentiu Neagoe, Mihai Serban, and Adrian Papp.

This is a updated version of the original documentation written by erik.j.o

The attached "Async_wrapper.h" is the c header file of this library.

[probs]

Hi!

This is a very good instruction. I have successfully controlled kinect by following the steps above!

For more flexbility, Is it possible to share the full fuction list of libkinect_wrapper with us? or where can we get it?

Thanks,

Polin Liao

[xkingkiller]

If you see the myRIO->subVIs->library calls of the project I attached, you can see most of the VI-warppers for the APIs. I also attached the header file of this library - Async_wrapper.h now, you can refer to the header file.

[probs]

Hi Xkingkiller,

Thanks for your reply. I'll check it out.

Thanks,

Polin Liao

[PeteFromQuanser]

Hey Xkingkiller,

Thanks a lot for putting this together. I followed your instructions, but I'm running into a problem. When I run #cat /proc/bus/usb/devices I don't see the motor device, and when I run either of the sample VIs I immediately get "Connection to the target (myRIO) has been lost". Here is what I see when I check the installation:

Any help would be very much appreciated.

Thanks,

Peter

[xkingkiller]

Hi Peter,

    Can you try to make sure your Kinect is working properly with your PC host, say using a MicroSoft SDK, and your myRIO is initialized with the "Getting Started Wizard". Normally, the Audio and Camera device will not show up due to disconnection of the 12 external DC Power. Never came across the missing of the Motor device.

[PeteFromQuanser]

I installed the Kinect SDK for Windows and was able to get the Kinect up and running. I was also able to run a couple sample VIs for myRIO with no problem. I reinstalled both the myRIO software, and the Kinect drivers to no avail. I'm not sure what this indicates in terms of debugging, but the light on the front of the sensor is flashing when connected to myRIO, as opposed to solid when connected to my PC. Does it matter that I'm not using the specific Kinect for Windows, but the XBOX 360 Kinect? I will try to test with another Kinect this afternoon to see if it's a model problem (the current one I have is 1473, but we have a 1414 as well).

Any other thoughts are appreciated.

Thanks

[xkingkiller]

It seems to be a known issue with libfreenect for the new Kinect model 1473. I found a discussion about this issues: https://github.com/OpenKinect/libfreenect/issues/316

Can you successfully running the examples on your Kinect 1414?

[PeteFromQuanser]

Thanks for finding that. I seems that the more recent builds of the libfreenect libraries support the newer model. What is the feasibility of using the newer releases of the libraries? If a modification to your examples would be time consuming to support v0.3.0, it's not a big deal. Implementing Kinect in the LabVIEW for Windows environment is also a feasible option for me.

Thanks again for your help,

Peter

[WallyFromDilbert]

Ok, so I am in need of some help. I have followed the instructions several times know but I keeping getting an error saying "Connection to the target (myRIO) has been lost". When I run the #cat /proc/bus/usb/devices command I do not see the "Xbox NUI Audio", "Xbox NUI Motor", or "Xbox NUI Camera” ….. this is what I see. I am using a kinect model 1517, for windows.

[xkingkiller]

Well, currently, this library is only working for Kinect 1414, all other newer Kinect are not supported. Sorry, I think I should marked it in the tutorial.

[djonas92]

Is there been a update on the lib file. I'm using a 1517 model, and i need to get it running using Myrio.

[WallyFromDilbert]

I do not think there was been an update to the lib fiel for model 1517 to work, you can always try...... but I doubt it will work. I had the same problem (read post above), the code only works for model 1414 to my knowledge.

[djonas92]

I tried it before reading the comments, i get the same error "Connection to the target (myRIO) has been lost" so the didn't update it. That's why i asked.

[racoon90]

I want to use this program on Window.

but .so file is not available.

what shoud i do?

[Jorge_Valerio]

Hi,

I followed the instructions and tested using #cat and apparently everything is Ok, but when I run the example got this message:

__________________________________________

Initializing...

Calculating dependencies...

Checking items for conflicts. This operation could take a while...

Preparing items for download. This operation could take a while...

Deploying myRIO(successfully deployed target settings)

Deploying Kinect_init(SubVI).viKinect_init(SubVI).vi loaded with errors on the target and was closed.

LabVIEW:  Failed to load shared library libkinect_wrapper.*:kinect_set_log_level:C          . Ensure that the library is present on the RT target. Use MAX to install NI software or FTP to transfer custom libraries to the RT target.

LabVIEW:  Failed to load shared library libkinect_wrapper.*:kinect_init:C          . Ensure that the library is present on the RT target. Use MAX to install NI software or FTP to transfer custom libraries to the RT target.

Deployment completed with errors

________________________________________

What did I do wrong?

Thank you in advanced.

[BradM]

It sounds like the hardware is being detected correctly but, as the error message states, the LabVIEW application is unable to locate the libkinect_wrapper.so file (which also means that there may be other library issues). Make sure to follow the step

...

4. Copy over libfreenect.so.0.1.2, libfreenect_sync.so.0.1.2 and libkinect_wrapper.so in /user/lib folder on your myRIO. And copy liblibusb.so.1.0.8 to /usr/local/lib

...

You can check to make sure that things are copacetic by issuing the following commands

ldconfig

ldconfig -p | grep -E "lib(free|ki)nect"

You should see listings for all three libraries

[Jorge_Valerio]

Hi BradM,

I can see all the libraries:

but when I run the example get the following error:

Any suggestions?

[Jorge_Valerio]

It works!

Thank you guys.

[trogen]

in step 2 u said enable SSH what is it?Where can i find it? Where can i find NI MAX??

please help...

[BradM]

https://decibel.ni.com/content/docs/DOC-30232

[M.Arafuka]

Hi,  I am not familiar with Linux and having the same proble as Jorge_Valerio got the error message. It seems that I am doing wrong on the step 4 of copying files. Could you give me step-by-step instructions of how to do this by using FileZilla? Or any useful URL?  Thanks.

[BradM]

I have not used Filezilla, but based on the screenshots, it looks like the older style FTP client interface. I would ssh to the target and check for the library files in the locations listed in step 4 for a better idea of how things are going awry.

Once ssh'd into the target using, e.g., PuTTY

(check to make sure that the libraries have been copied to the correct locations)

ls /usr/lib/libfree*

ls /usr/lib/libkin*

ls /usr/local/lib/*usb*

(check to make sure that the library loading cache knows of these new libraries)

ldconfig -p | grep -E "lib(free|ki)nect"

[approgramer]

I am using Kinesthesia toolkit for kinect to measure angles between joints...It worked fine on computer but when i load these vi to myrio the deploy botton is brocken,,.. Why doesnt it run on myrio.

I did all the above steps and the depth imaging and videio worked, but my project requirs skeletal tracking. how can i get .so libraries for skeletal tracking.

if i import the kinesthesia library( those are .vi files) into myrio using filezilla following the same above steps will those VIs work on myrio.

can i import .vi files into myrio and will it work?

if not is it possible to convert .vi file into .so? if yes how can it be done??

[trogen]

how can i get 3D skeleton abstraction libraries??

[trogen]

If i ssh .vi files into Myrio will it work??

[BradM]

I would direct those questions to the group as they are more familiar with the toolkit

[M.Arafuka]

Hi, Jorge_Valerio,

I have the same troble that you faced.  I could finally do the step4 and see all three devices of Kinect and all three libraries in PuTTY. It seems that I did it correctly. When I run the sample VI, I got the same massage of "libusb-1.0.so.0:cannot open shared object file: No such file or directory" that Jorge_Valerio faced. How did you overcome?

[M.Arafuka]

I could finally copy the files stated in the step4. I used Win SCP instead of FileZilla. Now I can see three libraries below.

libkinect_wrapper.so (libc6) => /usr/lib/libkinect_wrapper.so

libfreenect_sync.so.0.1 (libc6) => /usr/lib/libfreenect_sync.so.0.1

libfreenect.so.0.1 (libc6) => /usr/lib/libfreenect.so.0.1

But when I run the example VIs, I get the error message of "libusb-1.0.so.0: cannot open shared object file; No such file or directory". What should I do? It seems like the same problem that Jorge_Valerio faced.

[approgramer]

thanks bradm

[BradM]

masaman,

You show the listing for three of the four libraries, did you copy over liblibusb.so.1.0.8 and create the noted symlink?

[M.Arafuka]

I could finally use Kinect with myRIO.  I checked the symbol link again. Is it wrong statement in step6? I re-created symbol link below. Then, it worked and I could use the examples.

wrong:   #ln -s /usr/local/lib/liblibusb.so.1.0.8 /lib/libusb-1.0.so.0

correct:  #ln -s /usr/local/lib/liblibusb.so.1.0.8 /lib/libusb-1.0.so.1

But anyway, thank you all who supported me.

Thanks

[Lexicondi]

Hey xkingkiller!

Thank you so much for this libraries. Are there pallettes in labview that I can use in conjuction with this libraries to adapt your code to my needs? It doesn't work with the Kinesthesia toolkit certainly and I don't think it works with the IMAQ Vision toolkit as well.

Basically, I am truing to acquire the maximum pixel from the depth map. I will be glad if you could point me in the direction by which I could do this.

Thanks!

[chokkk]

hello when I try to make the correction still have the same problem. Needless to note that the specification to the library " usr " Might help

[Baarath12]

hi..i have went through the instructions...but im facing a problem since im using the newer model of kinect. Does anyone found a way to solve the problem for connecting the newer model.

[Baarath12]

hi...do u knw where i can get the recent builds of the libfreenect libraries?

[Fatik.Hasan]

I've done all the work required and verified that the Kinect is installed on my myRIO, but everytime I run a vi, I get this error. Some help would be greatly appreciated! Please tell me how to resolve this issue.

[BradM]

Fatik.Hasan,

It seems like you are trying to run a VI intended for running on the target (where you have just setup the noted libraries from KinectLib.zip) on your desktop (from the My Computer context). The VI is looking for libkinect_wrapper.dll, which is likely a result of the VI using a CLFN configured to use libkinect_wrapper.*. Also, the dependency search dialog is a giveaway.

[fatikhasan]

Oh, thanks for the info. I'm actually a newbie. Can you guide me how to fix this? What should I do?

[BradM]

Make sure to follow the instructions carefully, specifically the bits about setting up the target (part B in the instructions above).

Once you've setup your target, you can follow any of the NI documenation on how to develop applications for RT targets (basically, from your project, add your RT target, as described on page 12 of http://www.ni.com/pdf/manuals/371375a.pdf )

[fatikhasan]

BradM,

I've followed all the required steps.

Verified all the libraries learning from your comments,

and I can also see my Kinect devices detected by myRIO.

but whenever I run a VI the following happens:

it gets stuck here each and every time and when I cancel it, the following error appears:

What did I do wrong? and how I can troubleshoot this issue?

Some help would be greatly appreciated.

[BradM]

Can you check to make sure that liblibusb-1.0.so.0 exists in the ldcache?

ldconfig -p | grep libusb

Also, note that you created the link in the wrong location, which is likely leading to your issue.

From the original instructions:

ln -s /usr/local/lib/liblibusb.so.1.0.8 /lib/libusb-1.0.so.0

[SIDRAAKEEL]

Here are the things you requested for:

There seems to be an issue here. Why is there "so.1" here instead of "so.0"?

[Fatik.Hasan]

Here are the things you requested for:

[BradM]

The ".so.1" vs. ".so.0" is due to the internal SONAME being different. Note that it is using liblibusb-1.0.so.1 and that it has found it in the /usr/local/lib location. This is the symlink that was created by ldconfig, not the symlink you manually created.

It is likely the case that ldconfig sees the symlink in /lib that matches an automatically-generated symlink that ldconfig created in the past, removed the symlink, and regenerates the SONAME symlinks. This is done to facilitate when libraries are removed or updated on Linux systems to ensure that there are not dangling symlinks to libraries that no longer exist (or are a different version, so a different filename.)

So, tl:dr(but you should, if you care about library loading on Linux), the created symlink will work so long as you don't run ldconfig or reboot your target. This is due to decisions made when the original author created the VIs that use CLFN. I have no idea how the original implementation worked, it should be subject to the same behavior, but the underlying OS has changed in the time since originally written, so I suppose I could attribute the change in behavior to that.

My recommendation would be to open the VIs and adjust the CLFNs.

[Fatik.Hasan]

I finally got it to work! You've been really helpful. Thanks alot BardM!

[DDHITO]

Hello Fatik

I had the same problem as you had,could you plz tell me how you solved the problem?

[DDHITO]

Hello SID

I had the same problem as you had,could you plz tell me how you solved the problem?

[DDHITO]

hello BradM

Accroding to your instructions, i think everthing is right,but i still can't run any vi,and this problem keep coming out(in fig1)

[BradM]

The link creation failed, you need to add a space:

//You have

ln -s/usr/local/...

//You need

ls -s /usr/local/...

[fatikhasan]

After you have transferred the libraries to myRIO, use the following lines of code in PuTTY:

ln -s /usr/lib/libfreenect.so.0.1.2 /usr/lib/libfreenect.so.0.1

ln -s /usr/lib/libfreenect_sync.so.0.1.2 /usr/lib/libfreenect_sync.so.0.1

ln -s /usr/local/lib/liblibusb.so.1.0.8 /lib/libusb-1.0.so.0

Verify if you have copied the libraries in the right location, and if the libraries are cahced, using the methods in the conversation between BradM and I.

For verification of cache:

ldconfig -p | grep -E "lib(free|ki)nect" (3 files should appear when you run this)

ldconfig -p | grep libusb (1 file should appear when you run this)

Do tell me if you get it to work.