Ophthalmologists and AI team up to improve the diagnosis of diabetic retinopathy.
Recent advances in AI promise to improve access to diabetic retinopathy screening and improve its accuracy. But it’s less clear how AI will work in the physician’s office or other clinical settings. Previous attempts to use computer-assisted diagnosis shows that some screeners rely on the machine too much, which leads to repeating the machine’s errors or under-rely on it and ignore accurate predictions. Researchers at Google AI believe some of these pitfalls may be avoided if the computer can “explain” its predictions.
To test this theory, the researchers developed two types of assistance to help physicians read the algorithm’s predictions.
- Grades: A set of five scores that represent the strength of evidence for the algorithm’s prediction.
- Grades + heatmap: Enhance the grading system with a heatmap that measures the contribution of each pixel in the image to the algorithm’s prediction.
Ten ophthalmologists (four general ophthalmologists, one trained outside the US, four retina specialists, and one retina specialist in training) were asked to read each image once under one of three conditions: unassisted, grades only, and grades + heatmap.
Both types of assistance improved physicians’ diagnostic accuracy. It also improved their confidence in the diagnosis. But the degree of improvement depended on the physician’s level of expertise.
“With assistance, general ophthalmologists match but do not exceed the model’s accuracy, while retina specialists start to exceed the model’s performance.”
Without assistance, general ophthalmologists are significantly less accurate than the algorithm, while retina specialists are not significantly more accurate than the algorithm. With assistance, general ophthalmologists match but do not exceed the model’s accuracy, while retina specialists start to exceed the model’s performance.
“There’s an analogy in driving,” says lead researcher, Rory Sayres, Ph.D. “There are self-driving vehicles, and there are tools to help drivers, like Android Auto. The first is automation, the second is augmentation. The findings of our study indicate that there may be space for augmentation in classifying medical images like retinal fundus images. When the combination of clinician and assistant outperforms either alone, this provides an argument for up-leveling clinicians with intelligent tools.”