This is the third document presenting an example application of LabVIEW with the Microsoft Hololens, an augmented reality helmet (see www.hololens.com). The first two documents can be found in the Virtual group.
The range of possible scientific and engineering applications for the Microsoft Hololens is very wide. Unfortunately, the Hololens requires a good knowledge of Unity with C# or DirectX with C++ to develop an application. Some generic applications are being developed for scientists and engineers but those applications will probably rarely fully meet the requirements. Scientists and engineers who have no formal programming formation have been using LabVIEW for decades now to develop applications that meet their requirements more closely than generic software. This document is an example on how this flexibility of LabVIEW can be extended to the Hololens and to augmented reality.
This document presents an example of application related to the field of non-destructive testing (NDT). NDT is probably a field that would greatly benefit from augmented reality. This project shows how an application developed with LabVIEW that already includes 3D objects, can be extended to be used with the Hololens. This application is called HaroUT™ and has been presented in a previous document. The 3D portion of the HaroUT application was modified, using the soon-to-be released toolkit, so that the 3D results can also be uploaded on the Hololens.
A video showing the use of the HaroUT application with the Hololens can be seen below.
Notice that the video presented shows the mixed reality capture of the Hololens (virtual objects plus video camera), and not the actual view of the user. The virtual reality as seen by the user is limited to a ~30°x17° field of view).
The HaroUT application is a calculator of ultrasonic delays and ray tracer for phased arrays that was developed with LabVIEW. The calculated ultrasonic beams can be plotted in 2D or 3D using LabVIEW 2D or 3D picture controls. Figure 1 shows the HaroUT interface along with the window for 3D display of the results.
Figure 1 HaroUT application interface.
The 3D objects created in LabVIEW can also be uploaded on the Hololens, thanks to a soon-to-be released toolkit from HaroTek (see Hololens for Labview: a new toolkit for a quick introduction on how the toolkit can be used).
To make the control of the 3D objects easier when using the Hololens, a simple Data Dashboard application was created and linked to the HaroUT application. Figure 2 shows the interface of the Data Dashboard application on a tablet.
Figure 2 Simple tablet interface to control the HaroUT application. The tablet interface was developed with NI Data Dashboard. The use of Data Dashboard here illustrates the benefits of leveraging LabVIEW to rapidly develop scientific and industrial applications for the Hololens.
The Hololens has sensors that maps the actual environment. The data from these sensors are used to stabilize the virtual 3D objects (or holograms) within the actual environment. The surfaces mapped by the Hololens can also be used to position the virtual 3D objects in the actual environment. The green cursor moves to the first mapped surface in the line of sight of the user.
The HaroUT application calculated ultrasonic rays for a 5-inch diameter, 0.75-inch wall pipe. The surfaces mapped by the Hololens were used to align the virtual pipe with an actual 5-inch diameter pipe (see Figure 3). The Hololens therefore acts as a virtual X-ray machine, showing the calculated ultrasonic beams and focal points within the wall of the actual pipe.
Figure 3 Virtual pipe and ultrasonic beams aligned to an actual 5-inch diameter pipe. The blue section of the pipe corresponds to a virtual weld.
The 3D objects can be scaled or moved around to make some features easier to see. Figure 4 shows how the virtual pipe, the ultrasonic beams, and the virtual defects are enlarged to improve the visibility of the details. In the present example, defects can be manually added at each of the beam focal points using the "Flag Defect" button of the tablet interface. The defects show as orange squares in Figure 4. In an actual inspection however, the time-of-flight or amplitude information from the ultrasonic signals could be integrated in the application to show the operator where the defects are located in the actual structures being inspected.
Figure 4 Virtual pipe and ultrasonic beams scaled up for better visibility by the user. The orange squares on the virtual blue weld are defects manually added at the locations of different beam focal points.
This document shows an example of how augmented reality could benefit an ultrasonic inspection, and NDT in general. The Microsoft Hololens can be used to show in 3D on the actual structure (a pipe in the present case) the results of an inspection, possibly reducing the time required to locate defects, and helping assessing the over health of the structure.
LabVIEW provides a large variety of tools that makes using the Hololens easier. With the flexibility of programming with LabVIEW, scientists and engineers can start thinking about applications that could benefit from the Hololens in their own fields without having to deal with Unity and/or DirectX.
Please, do not hesitate to post feedbacks here or directly to me.