Kinect 2 - Haro3D VI Library

[MarcDub]

****************************************************

Edit January 6, 2016

Thank you for over 1100 downloads of the Haro3D library version 1.0.

The Haro3D library is now certified LabVIEW Compatible and is part of the LabVIEW tool network. Version 1.2 can be downloaded using VIPM.

The permanent license can be purchased at the bottom of the following page: www.harotek.com/products.

Version 1.2 of the Haro3D library includes improvements to the Kinect V2 API, an Oculus Rift DK2 API, and a converter of .obj format files into LabVIEW 3D picture control objects.

The manual of version 1.2 is available for download from this current document.

****************************************************

Overview

The Kinect is a sensor developed by Microsoft for the Xbox game console. Its main goal is to be able to interpret human positions and gestures. To accomplish this task, the Kinect is equipped with a depth measurement system based on active illumination. This feature makes the Kinect a low cost three-dimensional camera that can be used for applications outside the gaming industry. More information about the Kinect can be found at http://www.microsoft.com/en-us/kinectforwindows/meetkinect/features.aspx.

When the first Kinect came out, the  great Kinesthesia library was rapidly made available by University of Leeds. That library is based on .NET assemblies that can be used directly from Labview. The Kinect for Windows v2 was first made available at the end of 2013 within a beta program, and the public release was July 2014. To our knowledge, no VI library to access the Kinect v2 features from Labview has been available so far. One reason might be that there are apparently some issues to access the Kinect v2 from within Labview using the .NET assemblies.   

We believe that the Kinect is a great piece of hardware, and that Labview has great, but under-exploited 3D Visualization tools. At HaroTek, we developed a VI library called Haro3D™. This VI library contains API VI's to access some features of the Kinect v2. These API VI's are basically wrapper around two DLL's (one for 32-bit and the other for 64-bit versions of Labview) that were developed in C++.

Redistribution and use in source and binary forms of the Haro3D™ library, with or without modification, are permitted provided that the conditions expressed in the accompanying license are met (3-clause BSD license with add-ons for NI and Microsoft).

HaroTek is committed to maintain and keep growing the Haro3D™ library over time.

The Haro3D™ library can be downloaded from the current page. A copy of the manual is also available. The manual is simply the pdf version of the help file accessible from Labview.

Comments are welcome.

More information can be obtained at www.harotek.com.

Description

The Haro3D™ library gives access to the following functionality of theiesKinect v2:

• Bodies (people and joints tracking)
• Colored 3D cloud of points.
• Depth sensing
• Color high-definition camera
• Active infrared imaging
• 3D volume reconstruction
  
  
  

In addition, the Haro3D™ library offers Utilities VI's to complement the functionalities of the Kinect 2 like VI's to display and interact with the 3D data, and VI's to save and read back the 3D data (clouds of points and meshes). Fully functional examples for each of the functionalities are also provided.

Installation

The library comes within a VIPM package from JKI. VIPM installs the library, the examples (in the Example folder of Labview), and a help file that can be accessed through the Labview context help window.

To access the Kinect v2 functions, the Visual C++ redistributable (x86 or x64 depending on your version of Labview) must be installed on your system. Additional requirements are given below.

Requirements

Software

For the Microsoft Kinect for Windows V2:

Operating System: Windows 8 or later, 64-bit.

Upon connecting the Kinect for Windows V2 for the first time to your computer, the drivers should be downloaded through Windows Update. If the computer is not connected to the Internet, download and run the Kinect runtime 2.0 or later (http://www.microsoft.com/en-us/download/details.aspx?id=44559).

For the Haro3D™ library

The use of the Kinect V2 with the Haro3D™ library requires Visual C++ 2013 Redistributable x86 or x64 (http://www.microsoft.com/en-us/download/details.aspx?id=40784),  (or Visual C++ 2013, Ultimate, Premium, Professional, and Express for Windows Desktop) corresponding to the LabVIEW™ bitness version.

Hardware

Below is an excerpt of the requirements given by Microsoft at (http://msdn.microsoft.com/en-us/library/dn782036.aspx)

1. Kinect for Windows V2 sensor, which includes a power hub and USB cabling

2. Computer processor: I7 3.1 GHz (or higher), x64 core

3. Memory: 4GB or more

4. Operating System: Windows 8 or later, 64-bit.

5. Video card: DX11 capable graphics adapter (see list of known good adapters below)

• Intel HD 4400 integrated display adapter
• ATI Radeon HD 5400 series
• ATI Radeon HD 6570
• ATI Radeon HD 7800
• (256-bit GDDR5 2GB/1000Mhz)
• NVidia Quadro 600
• NVidia GeForce GT 640
• NVidia GeForce GTX 660
• NVidia Quadro K1000M
  

6.Built-in USB 3.0 host controller (Intel or Renesas chipset). If you're adding USB 3.0 functionality to your existing PC through an adapter, please ensure that it is a Windows 8 compliant device and that it supports Gen-2.

Steps to Implement or Execute Code

1. Install the library using the VIPM file.
2. Make sure that the Kinect v2 is correctly configured
3. Look at the  examples using the NI Example finder (use Directory Structure, and opens HaroTek-> Haro3D  Examples
4. Access the API and Utilities VI's using the function palettes: Addons -> HaroTek -> Haro3D -> Kinect v2 or -> Utilities.

Programming Steps

There are 7 API VI's. One VI is for the Kinect control. It turns ON and OFF the Kinect. This VI should be called first. The 6 other VI's  gives access to the Bodies, Clouds, Color Image, Depth, Fusion, and Infrared functionalities of the Kinect.

The Haro3D Utilities consist in VI's to display Bodies, Clouds, and Meshes in Labview 3D pictures, and to save and read Clouds and Meshes in PLY, XYZ, and STL formats.

1. Starts the Kinect using the KinectControl API
2. Initializes the desired functionalities (using the Init command). More than one functionality can be simultaneously used.
3. Access the data.
4. Use the Haro3D Utilities VI's to display, save, or load the 3D data if necessary
5. Close the desired functionalities
6. Stops the Kinect.

Below is a very simple example of those steps for 2 Kinect functionalities: Clouds and Bodies.

The resulting output of the code above is shown below:

Examples

The examples are installed by VIPM in the Examples directory of Labview under HaroTek/Haro3D Examples. The examples can be accessed through NI Example Finder (see below)

Front panels of the Bodies and Cloud examples are shown below:

Additional Images or Video

We are going to post videos on youtube on the examples VIs and other topics. Check the HaroTek channel: https://www.youtube.com/channel/UCmxaabC-nkffasAzzlppyKw/feed or check on www.harotek.com/videos.

[vmax711]

Fantastic work! Thank you! Would you mind sharing on how I can just access Joint positions?

[MarcDub]

The Haro3D library VIPM package includes an example on how to access the joint information. I give below a more explicit example on how to access the joint informatin (joint type, tracking state, position, orientation) of each joint. 

First, install the Kinect 2. You will need a computer running Windows 8 64-bit or later. When connecting the Kinect 2, the drivers should be downloaded from the internet. If not, you can download the drivers using the link given in the document above.

Second, you have to install the library and the Visual C++ 2013 redistributable x86 or x64 (matching your LabVIEW version).

Finally, the programming is relatively easy. You start the Kinect using the Kinect_control VI, then initializes the bodies functions using the Body VI, and then you can access information of the 25 joints of the up to 6 bodies at up to 30 Hz. Notice that if there are less than 6 detected body, the detected bodies do not necessary start at index 0 of the body array. You need to go through all 6 possible bodies of the array even if only 1 person is in front of the Kinect.

The joint information includes the joint type, tracking state, position, and orientation. There are 25 different joints per body and the names of the joints provided by the library typedef are self-explanatory. The position are simply the X, Y, Z values  of the joints in the Kinect coordinate system. The orientation is the a quaternion giving the absolute orientation of limb between the parent joint and the current joint. Notice that in the example provided with the Haro3D library, the orientation quarternion is not used; the orientation is calculated using the positions of the parent and child joints.

Have a look at the snippet below (LabVIEW 2013) and at the libray example and documentation. Hopefully that will help you.

[simons_84]

Hi,

thank you so much!!

I can't find Haro3D library in VIPM, could you check?

I'll try to install it manually.

Ops, I've seen now the vips file...:-))

thank you, again!!

[Yutaka]

It's a great library! Thank you very much. It is really easy to use Kinect. Now I am using your library with IMAQ vision to handle the image taken by Kinect.

I have a question that do you have a plan to publish C++ codes of your dll?

[MarcDub]

For the moment, I do not intend to provide the C++ code because I cannot maintain, support, and document the C++ code the same way I intend to do for the APIs.

However, If you let me know what features you are looking for, I am currently gathering suggestions for the next release of the Haro3D Library.

[Yutaka]

Hi! Thank you for the reply. I am trying to develop dlls by myself and I am curious about how other people write codes to control Kinect.

Actually, I am interested in realtime processing for obstacle avoidance of a robot. So the function should run fast. I hope that I can directory connect an imaq image to your aquire function. I think tha it makes easy to save and to do other machine vision functions.

[MarcDub]

How fast do you need to acquire the data? The depth data will provide the fastest processing (it is a simple 512x424 2D array of 32-bit integers). You should be able to get 30 Hz rates (the limit) no matter what.

If you use the clouds of points, processing will be slower because there are 3 coordinates per point plus the color. Furthermore, because of a problem with the registration in the Kinect SDK, the AP uses an array that matches the size of HD camera (either 1920x1080, 960x540, or 480x270). The depth sensor is 512x424 so you either oversample or undersample (there is no benefit using the 1920x1080 resolution with the clouds of points). Depending on your computer, you should be able to get 30 Hz rates with at least the 480x270 resolution. However, at low light levels, the HD camera acquisition might limit the rates to 15 Hz.

I am not sure what you mean by "directly connect an imaq image to your acquire function". If you mean to obtain an imaq image directly from the dll, I don't think it is possible. You will have to go acquire raw data and convert them into an imaq image using Labview function.

Let me know if you need more help.

[MarcDub]

It is possible to transform the depth array into a cloud of points. You have to get the registration table from the Kinect_control API with the "Characteristics" command. You then multiply the depth values by the corresponding coefficient of the table to get the X and Y values (the depth array values are the Z values). This is the exact process used by the Kinect SDK to convert depth into X,Y,Z values. There are 2 advantages with this approach: you get the full depth sensor resolution (512x424 pixels) and it is faster than using the Cloud of points API. The drawback is that you don't get the colors (colors are available but are currently badly registered to the depth values because of a bug in the Kinect SDK).

Below is an image showing how to calculate the cloud of points from the depth array. With my computer, I can get acquisition at the 30 Hz limit if 3D-data display is disabled. It drops to 23-24 Hz with the display enabled. I will post the VI with this project document.

[TY131]

Great work! But I have a problem when use this library, I can run the example VI successfully for the first time. If I stop the program or even close the VI, then reopen and run the same example. The image will not update and in the device manager, the icon for kinect will disappear, I check the block diagram, it enters the no kinect image case. If I stop the VI and run again, it shows no kinect connected. even uplug the kinect usb and reconnect it can't solve it. only after restart the computer,the PC can detect the kinect again. VI can run successfully again. But the situation mentioned above repeats . I check the requirement in the manual. The only unsatisfied condition is that I m using i5 2.9Ghz CPU. is it the problem? do u have the same situation before?

Thank you.