Robotic Application with the Hololens

[MarcDub]

Overview

This is the second document presenting an example application of LabVIEW with the Microsoft Hololens, an augmented reality helmet (see www.hololens.com). The link for the first document is provided here: Hololens for Labview: a new toolkit.

This document presents an example of robotic application with the Hololens, using a VI that was developed to load robots from manufacturer CAD files (OBJ File Format for 3D Robot Simulations). This VI was slightly modified, using the soon-to-be released toolkit, to upload the robot axis meshes in the Hololens as well as in the LabVIEW 3D Picture control.

Video

A video demonstrating this modified VI with the Hololens can be seen below.

Notice that the video presented shows the mixed reality capture of the Hololens (virtual objects plus video camera), and not the actual view of the user. The virtual reality as seen by the user is limited to a ~30°x17° field of view (user can see the actual environment with a nearly full field of view).

Virtual Robot

Figure 1 shows the VI used to upload and control the robot in the LabVIEW 3D picture control.

Figure 1     Front panel of the VI used to generate and control robot in LabVIEW 3D picture control.

In addition of adding the Vis so that the 3D objects representing the robot axes are also uploaded on the Hololens, a simple Data Dashboard application was created and linked to the VI so that the robot can be controlled by the user wearing the Hololens. Figure 2 shows the interface of the Data Dashboard application on a tablet.

Figure 2     To control the robot in the virtual environment, a simple tablet application was developed with NI Data Dashboard. This application illustrates the benefits of leveraging LabVIEW to rapidly create applications for the Hololens.

Once uploaded in the Hololens, the robot axes can be moved in any position (see Figure 3).

Figure 3   Robot as displayed in the Hololens.

The Hololens maps the environment. It is therefore possible to scale down the robot and put it on a table, if this can be useful (see figure 4).

Figure 4   Robot as displayed in the Hololens.

Summary

LabVIEW provides a large variety of tools that makes using the Hololens easier. Without LabVIEW, user interaction must be integrated within the Hololens application, with its limited range of inputs (gaze and air-tapping). LabVIEW provides more flexibility, like Data Dashboard, as shown in this document. Additionally, LabVIEW developers will appreciate not to have to deal with Unity and/or DirectX.

Please, do not hesitate to post feedbacks here or directly to me.

[ASharma]

Brilliant!  Can't wait for the HoloLens Toolkit for LabVIEW!