Real Advances from Virtual Reality
Combining VR ‘gaming’ technology with advanced wireless electroencephalography may transform the monitoring and diagnosis of visual disorders – particularly glaucoma.
At a Glance
- Standard automated perimetry (SAP) is a cumbersome and poorly objective method of assessing and monitoring visual field deficits
- Recent improvements in virtual reality technology and non-invasive electroencephalography suggest we can directly monitor visual field function at the neurophysiological level, thus eliminating subjectivity
- We have combined the best of these advances into a single, portable device – the nGoggle
- The system has potential to not only improve diagnosis and monitoring, but also broaden patient access and ultimately make a ‘game’ out of visual field assessments
I have always been interested in virtual reality (VR) technology and its potential applications to the evaluation and management of eye conditions; to me, the potential in this area seemed self-evident. But recently, the dramatic improvements we’ve seen in VR systems – in the news almost every day – have been matched by radical advances in data processing algorithms and electroencephalography (EEG) systems. The combination gives us a golden opportunity to develop something of real benefit to glaucoma patients: an accurate, portable and objective method of assessing visual field (VF) loss.
It’s no game…
At present, the accepted and broadly used technique for monitoring glaucoma progression is standard automated perimetry (SAP), which can sometimes be an unreliable guide to the extent of visual function damage. As it depends on inherently subjective responses from the patient, the data can be inaccurate, and test-retest variability can be high, making it difficult to identify genuine glaucomatous changes over time.
SAP is also limited by practical drawbacks. Notably, the devices are bulky and cumbersome, which restricts patient assessments to well-equipped clinical settings and means that patients in underserved or remote locations may not be assessed at all. Furthermore, in the resource-limited environment typical of clinical settings, SAP tests may not be performed frequently enough to permit timely diagnosis or detection of progression. How can advanced VR/EEG technology change this?
Part of the answer lies in the ability of new generation VR devices to allow for portable and convenient systems that can present different kinds of stimuli to test visual function. The VR system can be integrated with EEG to measure the so-called visual evoked potentials – the brain electrical responses generated by the visual stimuli. In a technique known as steady-state visual evoked potentials (SSVEP), the stimuli are not one-off events, but rapidly-flickering, on-off signals, which evoke characteristic electrical waveforms in the brain. Multifocal VR systems can stimulate many different areas of the retina simultaneously, thereby evoking concurrent responses in each area – we call these multifocal SSVEPs (mfSSVEPs). If stimuli presented to different parts of the VF flicker at different rates, mfSSVEPs that are specific to each flicker rate will be induced. Thus, multifocal stimuli covering the VF will trigger multiple specific mfSSVEPs accordingly; by comparing the observed mfSSVEPs with the expected pattern, we can precisely identify areas of VF loss.
The other part of the answer involves improved techniques for non-invasively monitoring brain activity. Until recently, data could only be collected in the clinical or laboratory settings. In addition, the technology relied on methods that were time-consuming and sometimes uncomfortable: for example, electrode placement on the scalp required skin preparation and gel application. The mfSSVEPs that we generate, however, allow the use of more advanced monitoring methods. Recordings of mfSSVEPs are acquired wirelessly, and do not require cumbersome preparation techniques such as application of conductive gels. The mfSSVEP waveforms have other desirable properties too: they are less susceptible to background noise and to artefacts arising from blinking or eye movement. Finally, mfSSVEP detection has been significantly improved by the application of sophisticated data processing techniques such as machine learning.
Building the nGoggle
Starting point: Samsung Gear VR Goggle headset, to which we added the following:
- mA current stimulators
- six flexible, polymer-based, wireless dry EEG electrodes
- our foam-based, wireless dry electroculogram (EOG) electrodes
- Bluetooth 4 radio
- Systems for simultaneous detection of 3D linear acceleration and 3D angular velocity (200 samples / second)
- Wireless neuromonitoring system with dual core processor running Yocto Linux
Result: a novel brain-computer interface that can generate, record and transmit neurophysiological responses to visual field stimuli
Mix ‘n’ match device development: the nGoggle
We wanted to combine these sets of advances into a single device that would elicit, capture and transmit mfSSVEP data in real time. To develop a portable device with all the required functionality to objectively assess loss of visual function, we had to combine disparate technologies in a single unit (1). In brief, we incorporated a VR goggle with head-mounted display, a wireless EEG system and a computer processing unit into a device we called nGoggle (NGoggle, Inc., San Diego, CA). (Box 1). The initial prototype system used a Samsung Gear VR goggle. More recent iterations of the nGoggle have incorporated other goggles and the development of a customized VR goggle is underway.
In the original prototype system, VF testing included sending visual stimuli to the headset display through a connected cell phone. The visual stimuli included 20 sectors covering 35 degrees of the field of view; each sector flickering at a specific frequency – from 8 to 11.8 Hz – inducing different mfSSVEPs from different VF sectors. We capture the mfSSVEPs using 6 EEG electrodes that can be applied and worn without gel preparation, located at the occipital region. The system also included 4 electrooculogram (EOG) electrodes for monitoring fication losses. The computer processing unit processes the data and transmit it to the cloud via wireless or Bluetooth technology. Real-time monitoring can be done by a tablet. The entire system is completely wireless, and very portable (Figure 1). And of course there is no need for any subjective patient input like button-pressing – it is all objective.
We successfully developed a phone-based, head-mounted system that enables effective – and objective – detection of specific mfSSVEPs associated with stimulation of specific parts of the visual field. But does it provide clinically meaningful information?
Addressing real questions
One of the first questions we addressed was: can our system distinguish between glaucomatous and healthy eyes? We designed a pilot clinical study (Box 2) to uncover the answer, and to get some idea of the accuracy and repeatability of the nGoggle system. We summarized our results on diagnostic performance by means of receiver operating characteristic curves (ROC curves), which despite the complicated name are just simple plots that provide information on the sensitivity/specificity trade-off (the area under the ROC curve summarized the diagnostic accuracy of each parameter – a value of 1 represents 100% accuracy, whereas 0.5 represents pure chance).
The data from this pilot study were very encouraging: they showed that nGoggle was at least as good as, if not better than, SAP. Specifically, our ROC curve data indicated that nGoggle mfSSVEPs were lower (p<0.001) for glaucoma eyes (0.289) than healthy eyes (0.334). The area under the ROC curve for nGoggle mfSSVEP was 0.92, which was larger than for SAP mean deviation (0.81), SAP mean sensitivity (0.80) and SAP pattern standard deviation (0.77). At an 80% specificity, the mfSSVEP parameter had a sensitivity of 85%, as compared with 64% for SAP; at a specificity of 90%, the corresponding figures were 71% and 43%.
Testing the nGoggle
- Inclusion criteria: Eyes with POAG or healthy eyes.
- Exclusion criteria: BCVA < 20/40; spherical refraction outside +/-5 D; cylinder correction outside 3 D; any coexisting disease that might affect optic nerve or VF.
- Recruitment: 33 glaucoma patients (62 eyes); 17 healthy participants (30 eyes), all patients were recruited and tested within three months of diagnosis.
- Test procedure: Patients were presented with visual stimuli comprising two patterns of 20 sectors covering the central 35 degrees of the field of view. Five seconds of visual stimulation were followed by a one second break; this six-second cycle was repeated 30 times, so that each patient was assessed over a total of three minutes.
- Comparison procedure: Patients also underwent SAP tests with a Humphrey Visual Field Analyzer II (Carl Zeiss Meditec).
- Repeatability of nGoggle data was assessed in 20 glaucomatous eyes (10 patients) by collecting three sets of measurements, each set separated by one week
The games we can play
In summary, we have developed a device, nGoggle, which removes the subjectivity from conventional VF assessments, and can discriminate between healthy and glaucomatous eyes in a clinical setting. Our preliminary data also suggested that the nGoggle was more accurate than global parameters from SAP. In addition, the portability of the device may allow it to be used in home-based settings, where many more tests could be acquired over time than what can be done nowadays with SAP, potentially leading to earlier detection of progression. Such application, however, still requires validation.
Looking ahead, we believe that nGoggle may have applications beyond the assessment of VF loss. For example, it has potential to assess higher cognitive functions via the development of VR-based tests. It could also be modified to allow assessment of contrast sensitivity. Long term, our goal is to make a game of VF assessment: can you imagine a patient playing Candy Crush on nGoggle while the device tests their VF? Well, it’s possible – the subject could play the game using central vision while a type of imperceptible flickering stimuli are presented peripherally. The patient wouldn’t subjectively notice, but the retina and brain would, and the nGoggle would record that response. Just because assessing VF is a serious part of glaucoma diagnosis and management doesn’t mean patients shouldn’t enjoy it!
- FA Medeiros et al., “The nGoggle: a portable brain-based method for assessment of visual function deficits in glaucoma”, IOVS, 57, 3940.
- M Nakanishi et al., “Detecting glaucoma with a portable brain-computer interface for objective assessment of visual function”, JAMA Ophthalmology, 135, 550-557 (2017). PMID: 28448641.