Eye Doctors Find WebMD Symptom Checker Wrong More Than Half the Time

Enlarge / Give online symptom checkers the hairy eyeball.

In 2015, The BMJ (Previously British medical journal) conducted a thorough audit of online symptom checkers. He found that, on average, the sites gave the correct diagnosis first only about a third of the time. Carl Shen, an ophthalmology resident at McMaster University, led a team of researchers in a small-scale follow-up looking specifically at eye health and found equally concerning results: the correct diagnoses only appeared a quarter of the time.

The results, presented this week at the annual meeting of the American Academy of Ophthalmology, are early and tentative, but Dr. Shen and his team are planning a larger follow-up study. In the meantime, WebMD has updated its algorithm.

Inconvenience stickers

To assess WebMD’s accuracy, Shen and colleagues compiled 42 eye health “clinical vignettes” based on the medical literature. A decidedly unpleasant vignette of someone with acute angle-closure glaucoma, for example, describes a “44-year-old woman with[ing] in the ER… with severe pain around her right eye lasting four hours… She also has nausea and has vomited once… Intraocular pressure is extremely high.

These thumbnails were then reduced to bare symptoms so that they could easily be entered into the symptom checker twice: once by a medical professional and once by a lay person. This allowed for consideration of both how non-specialists would interact with the tool and how different people might describe the same condition in different ways. Since the diagnoses of the clinical cases were already known, the result of the symptom checker could be compared to the known correct diagnosis.

Of the 42 cases, the correct diagnosis appeared somewhere in the list 18 times, or just under half. The correct result only appeared at the top of the list in 11 cases. And because the primary diagnosis was incorrect most of the time, the recommended course of action was also often wrong. Of the 18 vignettes based on clinical urgency, the symptom checker correctly recommended urgent care needed in only seven of them.

These results are a little worse than those of 2015 BMJ survey, which reported the correct diagnosis appearing in the top 20 symptom check options 58% of the time and appropriate recommendations for emergency care given in 80% of medical emergencies. But Dr. Shen’s eye study is also much smaller than the BMJ‘s, which looked at 23 different symptom checkers and compiled results from many different people with the same symptoms. The ophthalmological study could therefore be less reliable. Shen and his team are also much earlier in the research process than the BMJ survey, and they envision crowdsourcing as a way to capture how different people would describe the same condition differently.

Software updates

It’s also important to note that since completing these two studies, WebMD has updated its algorithm, in part due to the BMJ research. Where the old symptom checker required users to start by clicking on a body part, the new version only needs a description of the symptoms. “After testing the new version of the WebMD Symptom Checker, our analysis of 77 varied cases showed a 70% accuracy rate for the top 3 condition,” says Michael Smith, Medical Director of WebMD.

Independent evaluation of this success rate and exploration of whether it is the same across different health domains could follow in future research. As it stands, the picture may be a bit sunnier than Shen and his colleagues suggest, but it’s clear that online symptom checkers shouldn’t be used in isolation to make medical decisions. They are “intended for informational purposes only”, says Dr Smith, emphasizing that physicians should “engage in conversation with their patients to help them understand how best to use the information”.

Helping doctors have this conversation from an informed starting point is precisely why Shen became interested in the issue. As more and more patients come to the doctor’s office after self-researching their symptoms, he and his colleagues write in their conference presentation, “It is important for ophthalmologists to be aware of the capabilities and limitations of diagnostic tools. ‘health information available’.