Student Projects
Contact Information
Competition Year: 2016
University: Loughborough University
Project Supervisors: David Kerr, Kaddour Bouazza-Marouf, Atul Gaur
Email Address: r.green2-12@student.lboro.ac.uk
Country: UK
Title
Turning Breath into Words: A Novel Method of Augmented Communication
Project Description
Development of a prototype system that can learn an impaired user’s breath patterns and produce spoken words on their behalf. Ethical approval has been granted by The University Hospitals of Leicester NHS Trust for intial tests of the system on selected patients in an ITU (Intensive Treatment Unit).
Products Used
- LabVIEW
- NI myDAQ
- Advanced Signal Processing Toolkit (now included as part of LabVIEW Professional)
- NI Labs Machine Learning Toolkit
- Vision Development Module
The Challenge
AAC (augmentative and alternative communication) systems are those that are used by people with serious disabilities that impair their ability to speak and communicate. One such famous use is the cheek movement system utilized by Professor Stephen Hawking.
Figure 1: A ltter board user and Interpreter (By Poule at English Wikipedia, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=15786923)
The challenge was to design and build a system that is capable of producing spoken words on behalf of a user from learning and recognizing their breath patterns.
It was required to develop hardware that would record the user’s breath in conjunction with a software platform to perform the processing and return synthesised speech from a computer.
The solution required a two way learning process, one in which the user learns how to create the patterns with their breath but at the same time, the system learns the user’s breath and recognizes it when they wish to produce speech.
In conjunction with the University Hospitals of Leicester NHS Trust and researchers from Loughborough University, a process was developed to produce information from the user’s breath that the machine can learn from and interpret.
With the view of using the system in intensive care units for those with little experience of AAC systems, it was vital to the success of the project to produce as simple as possible an interface for the users.
As mentioned above, the pressure was on to produce a working system that could be tested on people and within a hospital environment, as such it was imperative to have minimal time to completion.
The Solution
In essence, the Breath to Words system is designed around a straightforward process: acquire data, process data, produce output.
Figure 2: The entire breath communication system
The user breaths into the face mask, this is subsequently connected to a pressure transducer and custom bridge and amplification circuit for interfacing with the myDAQ.
The raw, acquired breath data is then processed using a number of techniques which currently remain confidential, however, the purpose is to extract information from the breath data in a way that is agnostic to the patient's breathing strength and at the time that the pattern began.
The resulting application is intelligent and versatile - meaning the system can adapt to any user. It also ensures that the software is fairly hardware agnostic - allowing the use of various breath sensing equipment, whilst ensuring that the current machine learning data remainds valid.
We are currently extending the system to enable a loved one’s recorded voice to be used for the speech, to increase the familiarity and personalization of the system.
Figure 3: A typical breathing apparatus
Before the Breath to Words system is used, a two-way learning process takes place between the machine and the user. By repeating the desired pattern for a word or phrase, the machine increases it’s understanding of the user’s breath whilst, simultaneously, the user learns to modulate their breath more preciseley. As both do so, the overall accuracy of the process improves.
To further improve usibility, we are working to increase the amount of feedback the machine provides to the user, so that the user can better understand how to improve their abilitythemselves.
Figure 3: The welcome screen interface
.png)
Figure 4: Teaching the system a new phrase
.png)
Figure 5: The system interpretting a breath and producing speech
With a focus on developing the novel software and mathematical process, it was imperative that minimal time be spent developing a data acquisition solution. The myDAQ became the perfect device in the prototyping stage, with minimal setup time and the simplicity of integrating it with the software developed in LabVIEW, I was able to have an incredibly short time to first measurement. This allowed me to focus on the development of the software and have confidence in the pressure measurements that I was taking of breath.
Using LabVIEW during the development had a number of outstanding advantages, the first was the speed that I could actually develop and implement the system. The wealth of built-in functions combined with extensive function libraries created by the wider LabVIEW developer community meant that I rarely got bogged down in developing any mathematical functions from the ground up.
The user interface was a key factor in the final solution - LabVIEW's rich set of tools and customization options minimized development time and allowed me to create an intuative, stream-lined user experience. The ability to rapidly make changes and see them almost instantly in the interface was a huge boon throughout the development.
Beyond the advantages of the software and hardware themselves, the tight integration between the NI platform and the wider NI ecosystem provided a number of advantages. I had guaranteed compatibility between the device I used and the software I developed, and the flexibility of LabVIEW and the DAQmx drivers ensured I could keep that confidence as the development moved from the prototype phase and hardware changes made.
With a strong focus on the graphical user interface of the application, it was essential to create one that was scalable and highly flexible. To do so, I created some custom code that implemented sub panels and a state machine to programmatically alter the interface dependant upon what functionalility the user required.
Hospital Trials and Near-Future Developments
As an ongoing research and development project, refinements continue to be implimented, however the time to develop an entire initial software prototype was significantly small. Within a period of around 2-3 weeks, as a solo developer, I created a complete application, including the core data acquisition, advanced signal processing and output of the system. In addition to the core functionality, I was also able to produce a robust user interface, in-application error handling, and the modularity to ensure future ease of development.
Although this unique medical device is still in the early developement phase, ethical approval has been granted by the University Hospitals of Leicester NHS Trust for intial tests of the system on selected patients in an ITU. This will allow us to verify usibility in patient-trials in British hospitals. Upon completion of the patient-trials, we will have a much clearer idea of the performance of the machine and people’s ability to learn the process.
Ultimately, the long-term view is that with the stable software platform that we have developed will allow the Breath to Words system to grow and improve with ease, until ready to be implemented as a product.
Figure 9: Breath Communication Prototype Poster (PDF Attached Below)
Nominate Your Professor
Nominee: David Kerr
David is an excellent lecturer who frequently uses National Instruments software to demonstrate concepts and give a real world indication of how concepts are utilised and implemented. David taught taught me for a module on Digital Image Processing, the module was made more engaging by the use of labs that allowed us to get hands on with the Vision Builder software and implement real world solutions to existing problems. The use of the software made learning the concepts a joy and much more memorable
