a low-cost digital letter board to speech system, which enables victims of paralysis to communicate
Figure 1: m(eye)DAQ User Interface and Blink Detector
The m(eye)DAQ is a digital letter board that enables victims of paralysis to communicate. The system utilises a National Instruments myDAQ to detect eye blinks or finger movements. Detected movements are fed into a LabVIEW application, which converts the movement into sentences before reading them aloud (removing the need for a human interpreter).
m(eye)DAQ is an example of a augmentative and alternative communication (AAC) system. Existing AAC systems require significant processing power and expensive components – with basic models costing between $4,000-10,000 and more advanced versions, which monitor patient eye movement, costing between $10,000-25,000. This represents a serious financial burden on users and health services. Therefore, I set out to produce a low-cost alternative from readily available components.
Total cost of m(eye)DAQ = £164 = myDAQ + LabVIEW + electrical components/sensors
Many victims of paralysis, including degenerative conditions such as Amyotrophic Lateral Sclerosis (ALS) and Locked-in syndrome (LIS), are unable to speak and have very limited body movement. They may only be able to move a cheek muscle, finger or eye. How can such sufferers converse with carers and loved-ones, when they can only effective give binary signals?
The traditional solution is a low-tech Letter Board (aka. Alphabet Board or Spelling Board).
What is a Letter Board?
A letter board is a physical board that arranges the letters of the alphabet in to a 2D grid.
A nurse/carer moves their finger down the rows of letters. The patient signals that a particular row includes a desired letter using whatever movement they can (a bink, twitch or tap of a finger). Subsequently, the carer then points to each letter in the selected row and, again, the patient signals when the finger arrives at a desired letter.
This process is repeated until words and sentences are formed.
The m(eye)DAQ project is based upon this layout of letter board, although there are more versions available. The project produces a digital version similar to this letter board that automates the process of scrolling and selecting as well as giving the patient the ability to have the sentences read aloud by the computer.
Famous Letter Boards
The recent Stephen Hawkings biopic, The Theory of Everything, featured an emotive scene where Dr Hawkings learns to use a Letter Board. Before he had his computer based system, Dr Hawkings' partner Jane would help him spell out words by choosing the colours that related to his chosen letter.
Figure 3: Stephen and Jane Hawking using the letter board in "The Theory of Everything"
Dr. Christopher Pendergast, an ALS sufferer and president of Ride For Life, reviewed The Theory of Everything for the Huffington Post. Dr. Pendergast was especially moved by Hawking's lack of independance during the letterboard scene.
"I resonated with the frustration and anger shown in the scene where Hawking was being encouraged to use an augmentative communication device, an alphabet board, to spell out his thoughts. The audience have no idea the isolation and terror that scene illustrated."
Additionally, a letterboard is seen throughout the critically acclaimed movie, The Diving Bell and The Butterfly. This is the true story of a French journalist, Jean-Dominique Bauby, who, following a severe stroke, was left with only the ability to blink his left eye. Using a letter board, Jean-Dominique and his assistant managed to complete his now famous memoirs.
Figure 4: Jean-Dominique Bauby completes his memoirs using a Letter Board
Famously, a letter board was also seen in Breaking Bad. Paralyzed Hector Salamanca is a once high-ranking member of the Juárez Cartel, turned police informant. He commutates with police officers using a letter board by tapping a brass service bell taped onto his wheelchair armrest.
Figure 5: Hector Salamanca communicating with his bell in Breaking Bad
The issue with all of these manual letter boards is that the end-user is completely reliant on a carer/nurse to communicate. As a digital letterboard, m(eye)DAQ automates the letter highlighting and can promote independance for paralysis sufferers.
m(eye)DAQ: The Hardware
The aim of the project was to build the system for as little money as possible, for that reason, the blink detection hardware simply utilises a single TCRT5000 optical reflectance sensor, a pair of 3D cinema glasses and two resistors. The finger movement detection circuit is a very simple switch that in this case is an upcycled door bell to provide a large and sure surface to press on.
The prototyped circuit for the TCRT5000 sensor is shown below:
Figure 6: Prototype Circuit Layout
Figure 7: The TCRT5000 Sensor on the Glasses
The TCRT5000 is mounted onto the frame of the glasses to hold it steady and utilises the flexibility of the wires to allow positioning specific to the user’s eyes.
The sensor itself works by firing an infra red signal at the white of the user’s eye, it then measures the reflected signal using an infra red photo transistor, and this reflected value varies depending on whether the signal is reflected off of the eye ball or the eye lid. The software analyses the signal to detect whether or not a significant enough change in signal has taken place.
Figure 8: Sensor's Reflected Signal
m(eye)DAQ: The Software
The following is the digital letter board layout, from the image it can be seen that the blue highlighted row/column is the active and selectable section.
As can be seen from the user interface, the digital letter board provides options for editing the written text in case a mistake is made or a change is needed.
Figure 9: The User Interface/Front Panel
The board also has room for expansion should common words wish to be used or new options added.
Further to the movement detection and speech systems, included in the software is a predictive text function. The predictive text takes a text file of the 1000 most popular words in the English dictionary and arranges them based on their popularity, based on the letters that have been typed of a word, it will return the most popular word that begins with those letters.
Figure 10: Predictive Text Sub VI Code
The overall architecture was simply a state machine where shift registers handle the increments of the scrolling and also the passing of the strings of text for the speech system.
If you are interested in the code or would like to implement elements of it into your own projects, then please feel free to download the attached Zip file and check it out.
Steps to Execute Code
- LabVIEW 2014
- myDAQ USB Device
- Movement interpretation device (have fun experimenting with these!)
The application currently gives the user the ability to make the computer speak, this means that for a message to be conveyed, there must be someone else present. I'm currently working on making this a connected appliction to remove this need and provide more freedom to the user. Please feel free to expand on the application, the following are some steps to implement (some of which I will be working on) to make the application an inavuable utility:
Any constructive improvements are more than welcome!
**This document has been updated to meet the current required format for the NI Code Exchange.**
About the Developer:
My name is Rob, I am an intern Applications Engineer at National Instruments. I am currently studying for a Masters in Mechanical Engineering at Loughborough University. I have a passion for developing projects that can positively influence and change the way people interact with the world.
Through the use of National Instruments tools I have been able to rapidly prototype and implement these projects such as above and look forward to sharing some awesome news projects, watch this space!