Rise of the Machines
A recent study shows that ChatGPT is becoming increasingly adept at answering ophthalmological board certification exam questions. Where does that leave us humans?
Alun Evans | | 4 min read | Future
The “AI Revolution” was first touted by Silicon Valley as the modern-day panacea for all our global issues – curing diseases, accelerating technological innovation, improving human creativity with its unstoppable and incrementally powerful capabilities. Since this utopian philosophy – also known as Technological Solutionism – revitalized interest in AI, wider society has become concerned with machine learning and its ever-growing capacity to act – or think – like a human being.
The growing paranoia of machines gradually outsmarting humans was further cemented by the defeat of chess grandmaster Garry Kasparov at the hands (processors?) of Deep Blue, the IBM-built computer, back in 1997. The widely publicized match was viewed as a major landmark for machine learning, with IBM estimating approximately five million people had tuned in to watch the matches live online (a substantial Internet audience for the early days of the modern Internet).
Since Kasparov’s defeat by a machine that he claimed was “as intelligent as your alarm clock,” the evolution of artificial intelligence and machine learning has hurtled forward, with advances, such as voice recognition (now commonplace in smartphones), the first “humanoid robot,” Sophia, making regular TV appearances and gaining Saudi citizenship (1, 2), and an AI-generated music track ostensibly by Drake and The Weeknd topping the charts (3). In light of the pace of progress, there’s been a steady debate between scientists, policymakers, governments, philosophers, media types, and academics regarding the ethics, benefits, and dangers of artificial intelligence (4, 5). Most recently, in May 2023, the “Godfather of AI,” Geoffrey Hinton, left Google (6), explicitly tweeting about the reasons for his departure: “I left so that I could talk about the dangers of AI without considering how this impacts Google.”
One of the latest and much-hyped additions to the armory for the machines in the upcoming human–robot civil war is ChatGPT – the automated AI chatbot that suddenly seems to be everywhere. (Interestingly, ChatGPT was a system born out of Hinton’s own research). At first glance, this AI subtype appears somewhat innocuous – and certainly less dangerous than, say, Lethal Autonomous Weapons Systems (LAWS) – but where does it fit into the healthcare picture?
A recent study revealed that ChatGPT recently scored an admirable 58 percent in a board certification test in ophthalmology, with researchers using a dataset of multiple choice questions garnered from OphthoQuestions (7). This latest test was performed in February 2023, and showed a 12 percent increase from an initial test conducted in January 2023. The researchers noted that the system performed most accurately on general medicine questions, with 79 percent accuracy, while its performance was considerably less accurate on questions relating to ophthalmology subspecialties (only 20 percent accuracy for oculoplastics, and 0 percent accuracy for retina-based questions).
However, “as the knowledge corpus of ChatGPT evolves, we believe its accuracy will improve significantly, especially in these niche subspecialty areas,” notes Andrew Mihalache, lead author of the study and resident at The Schulich School of Medicine and Dentistry in Western Ontario, Canada. “It’s also anticipated that new versions of ChatGPT will be able to interpret images. So we look forward to characterizing its performance on image-based questions and evaluating the extent to which newer iterations of ChatGPT may be used as a support tool for medical students, clinicians, and patients in the future.”
Asked whether ChatGPT’s continued evolution might lead to an increase of cheating in ophthalmic exams, Mihalache replies, “Certainly, the use of ChatGPT in an examination setting should be explicitly prohibited.”
In the long-term, however, Mihalache believes ChatGPT and other forms of AI present undeniable benefits for medical education. “AI has the potential to revolutionize medicine in the long-term,” he says. “It could help clinicians diagnose certain conditions with greater accuracy, improve preventative care measures, advance current treatment paradigms, and increase the efficiency and accuracy of medical research.”
Although all of these ascribed benefits seem to counter the AI doomsayers’ claims of robot world domination, Mihalache is also keen to address some of the concerns surrounding artificial intelligence and its continued expansion, particularly in the field of medicine. “Recent surveys have found that a high proportion of the public feels uncomfortable towards the use of AI in medicine,” he says. “As AI continues to advance, we should work towards addressing major concerns surrounding privacy, accountability, and medical ethics, while also harnessing its potential to improve patient outcomes.”
- Hanson Robotics, “Sophia,” (2023). Available at: https://bit.ly/3VVGSVl.
- Wired, “The agony of Sophia, the world’s first robot citizen condemned to a lifeless career in marketing,” (2018). Available at: https://bit.ly/44RgwrB.
- YouTube, “ghostwriter - heart on my sleeve (Drake feat. The Weeknd AI),” (2023). Available at: https://bit.ly/3LTz4yP.
- The Alan Turing Institute, “Understanding artificial intelligence ethics and safety: A guide for the responsible design and implementation of AI systems in the public sector,” (2019). Available at: https://bit.ly/3nPJdoi.
- Future of Life Institute, “Pause Giant AI Experiments: An Open Letter,” (2023). Available at: https://bit.ly/42sehJN.
- The New York Times, “‘The Godfather of AI’ Quits Google and Warns of Danger Ahead,” (2023). Available at: https://bit.ly/41r1qWT.
- A Mihalache et al., “Performance of an Artificial Intelligence Chatbot in Ophthalmic Knowledge Assessment, “ JAMA Ophthalmol, [Online ahead of print] (2023). PMID: 37103928.