Student Projects

cancel
Showing results for 
Search instead for 
Did you mean: 

Using Microsoft Kinect with myRIO (update)

This guide will describe how to use Kinect on myRIO through the USB port. This involves setting up libfreenect libraries on the myRIO, and testing it with an example project. This library only contains basic APIs to read raw RGB image and depth image from Kinect. Other functions such as audio, 3D skeleton abstraction are not included.

Instructions:

A. Download the attached 2 zip files.

     The "KinectLib.zip" contains libfreenect library files and the "Kinect Project.zip" contains the example Kinect VIs to use the library to read data from Kinect.

B. Setting up libfreenect Library.

     Requirements:

     In order to complete this part of the tutorial you will need software to SSH into your myRIO. This example uses PuTTY. You will also need to use SFTP to transfer files to the myRIO. This example uses FileZilla.

     Steps:

     1. Connect myRIO through its device USB port to your PC.

     2. Enable SSH. Go into NI MAX and find your myRIO under Remote Systems item in the left pane. Browse to your myRIO. In the System Settings tab of the device, under Startup Settings, there should be several checkboxes. Enable the box to the right of “Enable Secure Shell Server(sshd)” if it is not already. Hit Save.

     3. Connect to the myRIO with SFTP. Username is 'admin' and Password is left blank.

Capture.PNG

     4. Copy over libfreenect.so.0.1.2, libfreenect_sync.so.0.1.2 and libkinect_wrapper.so in /user/lib folder on your myRIO. And copy liblibusb.so.1.0.8 to /usr/local/lib

     5. Open an SSH connection with the myRIO. User is admin and password is left blank.

     Capture2.PNGCapture3.PNG

     6. Create symbol link to these files through the following commands in your PuTTY.   

     =============================================================

     #ln -s /usr/lib/libfreenect.so.0.1.2 /usr/lib/libfreenect.so.0.1

     #ln -s /usr/lib/libfreenect_sync.so.0.1.2 /usr/lib/libfreenect_sync.so.0.1

     #ln -s /usr/local/lib/liblibusb.so.1.0.8 /lib/libusb-1.0.so.0

     =============================================================

     7. Reboot your myRIO and your libfreenct library should be ready to go.

C. Run Example


Software Requirements:

  • LabVIEW 2013
  • LabVIEW Real-Time 13.0.0 - NI myRIO 13.0
  • Kinect Example VIs (attached)

Hardware Setup:

Connect your Kinect to the myRIO USB port and powered it by the wall adapter. Also, you can power Kinect with a external 12V DC. To make sure your Kinect is working properly, again SSH to your myRIO and type the following commands to check whether all Kinect USB devices are correctly mounted. And you should be able to see three USB devices in the list: "Xbox NUI Audio", "Xbox NUI Motor", "Xbox NUI Camera". If you cannot see them, make sure your Kinect is correctly connected.

     ====================================================

     #cat /proc/bus/usb/devices

     ====================================================

Capture4.PNG

Software Overview:

In the included project, there are two top level VIs which can be run after the Kinect is connected, and the libraries are installed on the myRIO.


Kinect Depth Controller has controls for the Depth Resolution and Depth Format, as well as the LED option and Tilt Degree. All depth formats and resolutions should work (NOTE: resolution and format have to be set before the program is run, and don’t have effect while the program is running). Tilt degree controls the motor inside of the Kinect that rotates it. LED option controls the LED on the front of the Kinect.


Depth resolutions: Tiny (160x120), Low(320x240), and Medium(640x480)


After Init Kinect Systems has run, any of the Vis that read depth or video data from the Kinect should work (exceptions for video resolutions noted below). It is important that Close Kinect Systems is called (use stop button, don’t abort); if it’s not executed, on next execution of the VI the myRIO will lose connection immediately. However, running it again after the disconnection will work.


Kinect Video Controller has the same controls as Kinect Depth Controller, but this time the Video Resolution and Video Format controls will take effect (again, these must be set before executing the VI). NOTE: as of right now, none of the video formats work in Low(320x240) resolution, all of the formats work in Medium(640x480) resolution, and only RGB and Bayer formats work in High(1280x1024) resolution.


If you will be doing image processing, or converting the depth to physical units, the projects on Obstacle Avoidance and Color Following with the Kinect have examples of this. Basically, you will want to modify the Read Video or some Read Depth VI so that you get the 2D array of points instead of the picture. From there you can convert it to an IMAQ image and process it with IMAQ functions.


The lower resolutions for both Video and Depth are much faster than higher resolutions. In the other projects, we’ve been getting about 15fps on tiny depth resolution, and ~3fps on medium resolution video.

Kinect VIs & Wrapper library

Credit for the Kinect VIs and wrapper code goes to Vicentiu Neagoe, Mihai Serban, and Adrian Papp.

This is a updated version of the original documentation written by erik.j.o

The attached "Async_wrapper.h" is the c header file of this library.

Download All
Comments
probs
NI Employee (retired)
on

Hi!

This is a very good instruction. I have successfully controlled kinect by following the steps above!

For more flexbility, Is it possible to share the full fuction list of libkinect_wrapper with us? or where can we get it?

lib.png

Thanks,

Polin Liao

xkingkiller
NI Employee (retired)
on

If you see the myRIO->subVIs->library calls of the project I attached, you can see most of the VI-warppers for the APIs. I also attached the header file of this library - Async_wrapper.h now, you can refer to the header file.

probs
NI Employee (retired)
on

Hi Xkingkiller,

Thanks for your reply. I'll check it out.

Thanks,

Polin Liao

PeteFromQuanser
Member
Member
on

Hey Xkingkiller,

Thanks a lot for putting this together. I followed your instructions, but I'm running into a problem. When I run #cat /proc/bus/usb/devices I don't see the motor device, and when I run either of the sample VIs I immediately get "Connection to the target (myRIO) has been lost". Here is what I see when I check the installation:

usb_status.png

Any help would be very much appreciated.

Thanks,

Peter

xkingkiller
NI Employee (retired)
on

Hi Peter,

    Can you try to make sure your Kinect is working properly with your PC host, say using a MicroSoft SDK, and your myRIO is initialized with the "Getting Started Wizard". Normally, the Audio and Camera device will not show up due to disconnection of the 12 external DC Power. Never came across the missing of the Motor device.

PeteFromQuanser
Member
Member
on

I installed the Kinect SDK for Windows and was able to get the Kinect up and running. I was also able to run a couple sample VIs for myRIO with no problem. I reinstalled both the myRIO software, and the Kinect drivers to no avail. I'm not sure what this indicates in terms of debugging, but the light on the front of the sensor is flashing when connected to myRIO, as opposed to solid when connected to my PC. Does it matter that I'm not using the specific Kinect for Windows, but the XBOX 360 Kinect? I will try to test with another Kinect this afternoon to see if it's a model problem (the current one I have is 1473, but we have a 1414 as well).

Any other thoughts are appreciated.

Thanks

xkingkiller
NI Employee (retired)
on

It seems to be a known issue with libfreenect for the new Kinect model 1473. I found a discussion about this issues: https://github.com/OpenKinect/libfreenect/issues/316

Can you successfully running the examples on your Kinect 1414?

PeteFromQuanser
Member
Member
on

Thanks for finding that. I seems that the more recent builds of the libfreenect libraries support the newer model. What is the feasibility of using the newer releases of the libraries? If a modification to your examples would be time consuming to support v0.3.0, it's not a big deal. Implementing Kinect in the LabVIEW for Windows environment is also a feasible option for me.

Thanks again for your help,

Peter

WallyFromDilbert
Member
Member
on

Ok, so I am in need of some help. I have followed the instructions several times know but I keeping getting an error saying "Connection to the target (myRIO) has been lost". When I run the #cat /proc/bus/usb/devices command I do not see the "Xbox NUI Audio", "Xbox NUI Motor", or "Xbox NUI Camera” ….. this is what I see. I am using a kinect model 1517, for windows.

Kinect_Image.png

xkingkiller
NI Employee (retired)
on

Well, currently, this library is only working for Kinect 1414, all other newer Kinect are not supported. Sorry, I think I should marked it in the tutorial.

djonas92
Member
Member
on

Is there been a update on the lib file. I'm using a 1517 model, and i need to get it running using Myrio.

WallyFromDilbert
Member
Member
on

I do not think there was been an update to the lib fiel for model 1517 to work, you can always try...... but I doubt it will work. I had the same problem (read post above), the code only works for model 1414 to my knowledge.

djonas92
Member
Member
on

I tried it before reading the comments, i get the same error "Connection to the target (myRIO) has been lost" so the didn't update it. That's why i asked.

racoon90
Member
Member
on

I want to use this program on Window.

but .so file is not available.

what shoud i do?

Jorge_Valerio
Member
Member
on

Hi,

I followed the instructions and tested using #cat and apparently everything is Ok, but when I run the example got this message:

__________________________________________

Initializing...

Calculating dependencies...

Checking items for conflicts. This operation could take a while...

Preparing items for download. This operation could take a while...

Deploying myRIO(successfully deployed target settings)

Deploying Kinect_init(SubVI).viKinect_init(SubVI).vi loaded with errors on the target and was closed.

LabVIEW:  Failed to load shared library libkinect_wrapper.*:kinect_set_log_level:C          . Ensure that the library is present on the RT target. Use MAX to install NI software or FTP to transfer custom libraries to the RT target.

LabVIEW:  Failed to load shared library libkinect_wrapper.*:kinect_init:C          . Ensure that the library is present on the RT target. Use MAX to install NI software or FTP to transfer custom libraries to the RT target.

Deployment completed with errors

________________________________________

What did I do wrong?

Thank you in advanced.

Ing. Jorge Valerio Araoz
Ingeniero de Ventas
SOCIEDAD INDUCONTROL INGENIERÍA S.A.C.
http://www.inducontrol.com.pe/
jvalerio@inducontrol.com.pe
Celular/RPC: +51-986666633
RPM: *271058
Teléfono: +511-4405225
Calle Manuel Fuentes 976B - Lima 27 - Perú
"No es lo que hacemos, es lo que hacemos posible." - National Instruments

BradM
Active Participant
Active Participant
on

It sounds like the hardware is being detected correctly but, as the error message states, the LabVIEW application is unable to locate the libkinect_wrapper.so file (which also means that there may be other library issues). Make sure to follow the step

...

4. Copy over libfreenect.so.0.1.2, libfreenect_sync.so.0.1.2 and libkinect_wrapper.so in /user/lib folder on your myRIO. And copy liblibusb.so.1.0.8 to /usr/local/lib

...

You can check to make sure that things are copacetic by issuing the following commands

ldconfig

ldconfig -p | grep -E "lib(free|ki)nect"

You should see listings for all three libraries

Jorge_Valerio
Member
Member
on

Hi BradM,

I can see all the libraries:

libraries.png

but when I run the example get the following error:

myRIO error.png

Any suggestions?

Ing. Jorge Valerio Araoz
Ingeniero de Ventas
SOCIEDAD INDUCONTROL INGENIERÍA S.A.C.
http://www.inducontrol.com.pe/
jvalerio@inducontrol.com.pe
Celular/RPC: +51-986666633
RPM: *271058
Teléfono: +511-4405225
Calle Manuel Fuentes 976B - Lima 27 - Perú
"No es lo que hacemos, es lo que hacemos posible." - National Instruments

Jorge_Valerio
Member
Member
on

It works!

Thank you guys.

Ing. Jorge Valerio Araoz
Ingeniero de Ventas
SOCIEDAD INDUCONTROL INGENIERÍA S.A.C.
http://www.inducontrol.com.pe/
jvalerio@inducontrol.com.pe
Celular/RPC: +51-986666633
RPM: *271058
Teléfono: +511-4405225
Calle Manuel Fuentes 976B - Lima 27 - Perú
"No es lo que hacemos, es lo que hacemos posible." - National Instruments

trogen
Member
Member
on

in step 2 u said enable SSH what is it?Where can i find it? Where can i find NI MAX??

please help...

BradM
Active Participant
Active Participant
on
M.Arafuka
NI Employee (retired)
on

Hi,  I am not familiar with Linux and having the same proble as Jorge_Valerio got the error message. It seems that I am doing wrong on the step 4 of copying files. Could you give me step-by-step instructions of how to do this by using FileZilla? Or any useful URL?  Thanks.

BradM
Active Participant
Active Participant
on

I have not used Filezilla, but based on the screenshots, it looks like the older style FTP client interface. I would ssh to the target and check for the library files in the locations listed in step 4 for a better idea of how things are going awry.

Once ssh'd into the target using, e.g., PuTTY

(check to make sure that the libraries have been copied to the correct locations)

ls /usr/lib/libfree*

ls /usr/lib/libkin*

ls /usr/local/lib/*usb*

(check to make sure that the library loading cache knows of these new libraries)

ldconfig -p | grep -E "lib(free|ki)nect"

approgramer
Member
Member
on

I am using Kinesthesia toolkit for kinect to measure angles between joints...It worked fine on computer but when i load these vi to myrio the deploy botton is brocken,,.. Why doesnt it run on myrio.

I did all the above steps and the depth imaging and videio worked, but my project requirs skeletal tracking. how can i get .so libraries for skeletal tracking.

if i import the kinesthesia library( those are .vi files) into myrio using filezilla following the same above steps will those VIs work on myrio.

can i import .vi files into myrio and will it work?

if not is it possible to convert .vi file into .so? if yes how can it be done??

trogen
Member
Member
on

how can i get 3D skeleton abstraction libraries??

trogen
Member
Member
on

If i ssh .vi files into Myrio will it work??

BradM
Active Participant
Active Participant
on

I would direct those questions to the group as they are more familiar with the toolkit

M.Arafuka
NI Employee (retired)
on

Hi, Jorge_Valerio,

I have the same troble that you faced.  I could finally do the step4 and see all three devices of Kinect and all three libraries in PuTTY. It seems that I did it correctly. When I run the sample VI, I got the same massage of "libusb-1.0.so.0:cannot open shared object file: No such file or directory" that Jorge_Valerio faced. How did you overcome?

M.Arafuka
NI Employee (retired)
on

I could finally copy the files stated in the step4. I used Win SCP instead of FileZilla. Now I can see three libraries below.

libkinect_wrapper.so (libc6) => /usr/lib/libkinect_wrapper.so

libfreenect_sync.so.0.1 (libc6) => /usr/lib/libfreenect_sync.so.0.1

libfreenect.so.0.1 (libc6) => /usr/lib/libfreenect.so.0.1

But when I run the example VIs, I get the error message of "libusb-1.0.so.0: cannot open shared object file; No such file or directory". What should I do? It seems like the same problem that Jorge_Valerio faced.

approgramer
Member
Member
on

thanks bradm

BradM
Active Participant
Active Participant
on

masaman,

You show the listing for three of the four libraries, did you copy over liblibusb.so.1.0.8 and create the noted symlink?

M.Arafuka
NI Employee (retired)
on

I could finally use Kinect with myRIO.  I checked the symbol link again. Is it wrong statement in step6? I re-created symbol link below. Then, it worked and I could use the examples.

wrong:   #ln -s /usr/local/lib/liblibusb.so.1.0.8 /lib/libusb-1.0.so.0

correct:  #ln -s /usr/local/lib/liblibusb.so.1.0.8 /lib/libusb-1.0.so.1

But anyway, thank you all who supported me.

Thanks

Lexicondi
Member
Member
on

Hey xkingkiller!

Thank you so much for this libraries. Are there pallettes in labview that I can use in conjuction with this libraries to adapt your code to my needs? It doesn't work with the Kinesthesia toolkit certainly and I don't think it works with the IMAQ Vision toolkit as well.

Basically, I am truing to acquire the maximum pixel from the depth map. I will be glad if you could point me in the direction by which I could do this.

Thanks!

chokkk
Member
Member
on

hello when I try to make the correction still have the same problem. Needless to note that the specification to the library " usr " Might help

Baarath12
Member
Member
on

hi..i have went through the instructions...but im facing a problem since im using the newer model of kinect. Does anyone found a way to solve the problem for connecting the newer model.

Baarath12
Member
Member
on

hi...do u knw where i can get the recent builds of the libfreenect libraries?


Fatik.Hasan
Member
Member
on

I've done all the work required and verified that the Kinect is installed on my myRIO, but everytime I run a vi, I get this error. Some help would be greatly appreciated! Please tell me how to resolve this issue.Capture222.PNG

BradM
Active Participant
Active Participant
on

Fatik.Hasan,

It seems like you are trying to run a VI intended for running on the target (where you have just setup the noted libraries from KinectLib.zip) on your desktop (from the My Computer context). The VI is looking for libkinect_wrapper.dll, which is likely a result of the VI using a CLFN configured to use libkinect_wrapper.*. Also, the dependency search dialog is a giveaway.

fatikhasan
Member
Member
on

Oh, thanks for the info. I'm actually a newbie. Can you guide me how to fix this? What should I do?

BradM
Active Participant
Active Participant
on

Make sure to follow the instructions carefully, specifically the bits about setting up the target (part B in the instructions above).

Once you've setup your target, you can follow any of the NI documenation on how to develop applications for RT targets (basically, from your project, add your RT target, as described on page 12 of http://www.ni.com/pdf/manuals/371375a.pdf )

fatikhasan
Member
Member
on

BradM,

I've followed all the required steps.
1.JPG

Verified all the libraries learning from your comments,

2.JPG

and I can also see my Kinect devices detected by myRIO.

3.JPG

but whenever I run a VI the following happens:

4.JPG

it gets stuck here each and every time and when I cancel it, the following error appears:

5.JPG

What did I do wrong? and how I can troubleshoot this issue?

Some help would be greatly appreciated.

BradM
Active Participant
Active Participant
on

Can you check to make sure that liblibusb-1.0.so.0 exists in the ldcache?

ldconfig -p | grep libusb

Also, note that you created the link in the wrong location, which is likely leading to your issue.

From the original instructions:

ln -s /usr/local/lib/liblibusb.so.1.0.8 /lib/libusb-1.0.so.0
SIDRAAKEEL
Member
Member
on

Here are the things you requested for:

1.JPG3.JPG

There seems to be an issue here. Why is there "so.1" here instead of "so.0"?

Fatik.Hasan
Member
Member
on

Here are the things you requested for:

1.JPG

3.JPG

BradM
Active Participant
Active Participant
on

The ".so.1" vs. ".so.0" is due to the internal SONAME being different. Note that it is using liblibusb-1.0.so.1 and that it has found it in the /usr/local/lib location. This is the symlink that was created by ldconfig, not the symlink you manually created.

It is likely the case that ldconfig sees the symlink in /lib that matches an automatically-generated symlink that ldconfig created in the past, removed the symlink, and regenerates the SONAME symlinks. This is done to facilitate when libraries are removed or updated on Linux systems to ensure that there are not dangling symlinks to libraries that no longer exist (or are a different version, so a different filename.)

So, tl:dr(but you should, if you care about library loading on Linux), the created symlink will work so long as you don't run ldconfig or reboot your target. This is due to decisions made when the original author created the VIs that use CLFN. I have no idea how the original implementation worked, it should be subject to the same behavior, but the underlying OS has changed in the time since originally written, so I suppose I could attribute the change in behavior to that.

My recommendation would be to open the VIs and adjust the CLFNs.

Fatik.Hasan
Member
Member
on

I finally got it to work! You've been really helpful. Thanks alot BardM!

DDHITO
Member
Member
on

Hello Fatik

I had the same problem as you had,could you plz tell me how you solved the problem?

DDHITO
Member
Member
on

Hello SID

I had the same problem as you had,could you plz tell me how you solved the problem?

DDHITO
Member
Member
on

hello BradM

Accroding to your instructions, i think everthing is right,but i still can't run any vi,and this problem keep coming out(in fig1)

5.jpgQQ图片20160413214801.png

BradM
Active Participant
Active Participant
on

The link creation failed, you need to add a space:

//You have

ln -s/usr/local/...

//You need

ls -s /usr/local/...

fatikhasan
Member
Member
on

After you have transferred the libraries to myRIO, use the following lines of code in PuTTY:

ln -s /usr/lib/libfreenect.so.0.1.2 /usr/lib/libfreenect.so.0.1

ln -s /usr/lib/libfreenect_sync.so.0.1.2 /usr/lib/libfreenect_sync.so.0.1

ln -s /usr/local/lib/liblibusb.so.1.0.8 /lib/libusb-1.0.so.0

Verify if you have copied the libraries in the right location, and if the libraries are cahced, using the methods in the conversation between BradM and I.

For verification of cache:

ldconfig -p | grep -E "lib(free|ki)nect" (3 files should appear when you run this)

ldconfig -p | grep libusb (1 file should appear when you run this)

Do tell me if you get it to work.

Contributors