Tasting Vision
The story of the BrainPort V100 – from vibrating plates, to balance aids, to a visual aid that helps users “see” with their tongue
At a Glance
- Visual information can be sent, via the tongue, in a manner that allows visually impaired people to receive and interpret it in a meaningful way
- BrainPort works by sending information from a camera to an array of sensors on the tongue, which form patterns by electrostimulation which, after training, are interpreted by the user
- In test subjects without visual impairment, tongue electrostimulation results in activation of the somatosensory cortex (as expected). But in those who are blind, it’s the visual cortex that becomes activated
- The technology could offer a less expensive and nonsurgical alternative to retinal implants, and might even be able to be used alongside other technologies to help improve independence for the profoundly blind
Neuroplasticity – the brain’s ability to reorganize, adapt, and form new neural connections throughout adulthood – is an area of neurology with a checkered past. For many years it was thought that the adult brain was “hard-wired” with fixed neuronal circuits – a notion that’s now disproven. Today, there’s solid evidence of adult neuronal plasticity and an active, experience-dependent reorganization of the synaptic networks of the brain involving multiple interrelated structures – and this includes the cerebral cortex, the region of the brain where most visual input is processed. The BrainPort V100 is a device that was developed to exploit exactly these plastic processes, in order to help users “see” objects with their tongues…
Brain barriers
American neuroscientist Paul Bach-y-Rita had a great interest in neuronal plasticity (1). In 1958, his father suffered an ischemic stroke that affected his mobility and ability to speak. His family was told he was unlikely to ever fully recover. The treatment options were fairly limited back in the 1950s, but Paul’s brother George, a psychiatrist, worked hard to provide rehabilitation treatment for their father. Happily (and against all expectations) their father went on to live a normal life for the rest of his years. When he died decades later, an autopsy revealed an unrepaired, severely stroke-damaged brainstem.
To Bach-y-Rita, this presented strong evidence of the existence of neuroplasticity: he believed his father’s recovery could only be explained by reorganization of the brain. Indeed, he’s recognized as one of the first to propose the concept of sensory substitution to induce neuroplasticity as a therapeutic modality. In the late 1960s, Bach-y-Rita and his colleagues at the Smith-Kettlewell Institute of Visual Sciences in San Francisco used some spare equipment to design and build a chair with a bank of 400 vibrating plates that rested against the user’s back. The plates were connected to a video camera placed above the chair, and the pattern in which the vibrating plates was stimulated enabled blindfolded users to “see” the objects that the camera recorded – they had produced one of the earliest examples of a haptic feedback device (2). Bach-y-Rita suggested what was occurring was an example of neuroplasticity, as he believed the signals sent to the brain from the skin were being processed in the visual cortex. As he famously said, “You don’t ‘see’ with your eyes, you ‘see’ with your brain.”
On balance
Originally, Bach-y-Rita designed a device to help patients with balance disorders. It detected the position of patients’ heads with an accelerometer, and relayed that information to the patient, by way of electrostimulation of the tongue. Stimuli on the right side of the tongue meant the body was leaning to the right, stimuli on the left side meant that the body was leaning to the left (3). This information was then used in conjunction with a series of exercises to treat balance disorders. But could it be used for anything more demanding? His next step was to evaluate this technology as a vision aid. Sadly, Bach-y-Rita died in 2006, aged 72, but his research continued.
Figure 1. BrainPort consists of a pair of sunglasses (with an integrated video camera), a 20 × 20 array of stainless steel electrodes that is placed on the tongue, and a handheld computer that processes the video camera’s input, contains contrast and stimulation intensity controls, and activates the electrode array.
“Just give the brain information and it will figure it out”
Vision substitution technology isn’t new: Louis Braille developed his code back in 1824 and the white cane rose to prominence in the 1930s. Exploiting the tongue is the place where the device meets the body – as BrainPort does – is a newer phenomenon. It consists of three interconnected units: a pair of sunglasses with an integrated video camera, a 20 × 20 array of stainless steel electrodes that are placed on the tongue, and a handheld computer that processes the video camera’s input, contains contrast and stimulation intensity controls, and activates the electrode array with patterns based on the camera input (Figure 1). In terms of the camera feed, a black pixel results in no tongue stimulation, and as a pixel gets brighter, so does the intensity of tongue electrostimulation. Through training, users learn to understand these levels of stimulation and form “pictures” in their mind of what the camera is viewing (see Figure 2).
Neuroimaging studies have shown that blind users activate the visual cortex to transfer information from the tongue to the brain after even a short period of training, whereas, sighted control subjects activated the somatosensory cortex (as you would expect with tongue stimulation) (4–6). Over time, and with training, blind users start to learn how to interpret what they feel on their tongue as real world objects and scenarios. Bach-y-Rita said, “Just give the brain information and it will figure it out” – and certainly with training, it does.
Unlike the original balance-correcting device (which involved a chair), BrainPort needed to be small, portable, and the tongue stimulation component was required to sit on the tongue comfortably, and (understandably) be moisture resistant. Fortunately, advances in technology over the years have helped achieve these goals.
Figure 2. Supervised training is central to people deriving benefit from the BrainPort device. Typically, most individuals can start to recognize shapes after a few hours of training, and after more training, users can identify familiar objects and avoid obstacles.
Figure 3. Learning curves for a four alternative forced choice shape recognition test in congenitally blind (n=8) and blindfolded control (n=10) subjects. (a) Mean percentage changes ± standard error of the mean (SEM) of correct responses and (b) mean reaction times ±SEM. No significant differences in performance were observed between the groups. Reproduced from (5).
A different angle
There are a number of similarities between BrainPort and some of the retinal prostheses that are available today: principally, the camera-embedded sunglasses and the small portable computer, although the main difference is the interface to the body – BrainPort sits on the tongue, rather than requiring surgical implantation on or under the retina. But does it work?
There are signs that it does. A number of relatively small studies (n=11–42) have been performed that have examined BrainPort’s visuo-tactile performance, including assessments of direction of motion, shape recognition and orientation and motility performance in blind subjects and non-blind controls (see Figure 3; reviewed in 7). While on the whole these studies have not demonstrated that blind subjects perform significantly better than blindfolded, sighted controls, they have shown a trend towards improved performances on these tests with training and repeated experience. However, early anecdotal reports from BrainPort users suggest that the device has been able to help profoundly blind patients regain some level of mobility, independence and confidence, and BrainPort’s manufacturer, Wicab, state that “some users with congenital blindness have even found that BrainPort has changed their understanding of how sighted people see – such as objects appearing closer when nearer, and that objects look different when viewed from different angles.”
Echolocation and apps
There’s no reason why, in principle, devices like BrainPort cannot work synergistically with other products like retinal prostheses or assistive aids like ultrasound or echolocation devices – and according to Wicab, some of these combinations are currently under evaluation.
The combination of digital video input and computer processing also raises possibilities of building upon the feature set of these devices. It’s reported that one of the top requests from BrainPort users was “the identification of exit and bathroom signs, without needing to ask for assistance,” and its manufacturers are looking into developing an app for the device that will allow the user to select the type of sign they wish to “see,” and the device will vibrate or make a noise whenever it identifies it. This is something that could be expanded to specific structures too, like pedestrian crossings and traffic lights. Finally, outside of its primary purpose, there’s also been interest from the military and the gaming market. Who knows what it might be used for in five or 10 years’ time?
- University of Wisconsin-Madison, “Memorial resolution of the faculty of the University of Wisconsin-Madison: on the death of Professor Paul Bach-y-Rita”, (2007). Available at: bit.ly/29R08LJ. Accessed July 21, 2016.
- P Bach-y-Rita et al., “Vision substitution by tactile image projection”, Nature, 221, 963–964 (1969). PMID: 5818337.
- YP Danilov et al., “Efficacy of electrotactile vestibular substitution in patients with peripheral and central vestibular loss”, J Vestib Res, 17, 119–130 (2007). PMID: 18413905.
- VK Lee et al., “Successful tactile based visual sensory substitution use functions independently of visual pathway integrity”, Front Hum Neurosci, 8, [ePub] (2014). PMID: 24860473.
- M Ptito et al., “Crossmodal recruitment of the ventral visual stream in congenital blindness”, Neural Plast, 2012 [ePub] (2012). PMID: 22779006.
- M Ptito and R Kupers, “Cross-modal plasticity in early blindness”, J Integr Neurosci, 4, 479–488 (2005). PMID: 16385642.
- HC Stronks, “The role of visual deprivation and experience on the performance of sensory substitution devices”, Brain Res. 1624, 140–152 (2015). PMID: 26183014.
I have an extensive academic background in the life sciences, having studied forensic biology and human medical genetics in my time at Strathclyde and Glasgow Universities. My research, data presentation and bioinformatics skills plus my ‘wet lab’ experience have been a superb grounding for my role as a deputy editor at Texere Publishing. The job allows me to utilize my hard-learned academic skills and experience in my current position within an exciting and contemporary publishing company.
