High “Star Ratings” from Users Don't Mean a Mobile Medical App Works

By Sondra Forsyth

By screening 250 user reviews and comments for a once popular — but proven inaccurate — mobile app claiming to change your iPhone into a blood pressure monitor, Johns Hopkins researchers have added to evidence that a high “star rating” doesn’t necessarily reflect medical accuracy or value.

A release from Johns Hopkins Medcine explains that the June 2018 analysis found that even the presence of app disclaimers warning users not to rely on the app for medical monitoring, diagnosis or treatment effectiveness failed to uniformly dissuade people from doing so, or stop health care professionals from recommending the app. That app, called Instant Blood Pressure and made by AuraLife, was withdrawn from the market in July 2015.

The release quotes study author Timothy Plante, M.D., M.H.S., a former fellow at the Johns Hopkins University School of Medicine and current assistant professor of medicine in the Larner College of Medicine at the University of Vermont, as saying, “People tend to trust user reviews when shopping online and use them to decide which products to purchase, but that doesn’t cut it for medical apps. There are certain thresholds of accuracy that need to be maintained, and a five-star rating doesn’t replace clinical validation studies and FDA review.”

The researchers say that unregulated mobile health app use could give people a false sense of security, which could lead to dire health consequences.

In a report on the study, published on June 7^th 2018 in the Journal of the American Medical Informatics Association, the physicians call for medical professionals to be more skeptical of apps that don’t have scientific studies backing them, and warn members of the public that just because an app has been endorsed by a health care professional doesn’t mean it works.

Their new findings are a follow-up to the authors’ March 2016 JAMA Internal Medicine report that analyzed the commercially available Instant Blood Pressure app, showing that four out of five times someone with high blood pressure used the app, they received falsely reassuring, nonelevated results.

For the new study, the researchers analyzed 261 user ratings and reviews downloaded from the Apple iTunes store before the app was taken off the market for reasons that were not made public.

The average star rating of the latest version of the app was four out of five stars, and 59 percent of the reviews assigned the app five stars. Commentary praising the accuracy of the app based on anecdotal experience comprised 42 percent of the reviews, and 10 percent of the reviews mention inaccuracy.

The Instant Blood Pressure app included a disclaimer — as do many mobile health app companies hoping to avoid FDA oversight — that the app shouldn’t be used as a medical device and is for “recreational” purposes only. Yet, 24 reviews –10 percent — claimed to use the app for medical purposes, with 11 people using it to manage their high blood pressure treatments, one person using it to manage kidney disease and another person using it to monitor blood pressure after a heart transplant.

“The data showed that disclaimers aren’t a complete solution. Consumers will continue to use these devices to manage their health care, which could be dangerous if they are substituting the app for medical care with a professional,” says Seth Martin, M.D., M.H.S., assistant professor of medicine at the Johns Hopkins University School of Medicine and one of the study’s investigators.

Six reviews came from people who claimed to be health care professionals including four nurses and one physician. These people gave an average rating of 4.2 stars out of five. Eleven reviews came from people who said their health care provider — a total of four physicians and seven nurses — approved of the app. Two people said that a nurse and an emergency room physician disapproved of the app, and as a result gave a one-star rating.

“Physicians need to be careful; if you’re saying you personally use an app, people will trust it,” says Plante. “Our take-home message to health care providers is to make sure you hold apps to the highest standards, and not just give out recommendations willy-nilly for apps that you think are neat. And consumers should be skeptical of reviews from self-proclaimed health professionals in product reviews.”

The Instant Blood Pressure app sold for $4.99 on iTunes and was downloaded 140,000 times before it was withdrawn. It ranked among the best-selling apps on all of iTunes for much of its stretch on the app store. After the app was withdrawn and following the authors’ last study, the Federal Trade Commission settled with the company for $600,000, with payment suspended for lack of funds in the company. Many medical apps are still available for consumers, some downloading as many as 1,000,000 times each.

“I’m glad for the high demand from consumers to be able to monitor their health from day to day,” says Martin. “Now we need to actually make sure mobile apps work as intended, in properly designed and conducted clinical studies, before they are relied on for health care.”

The authors recommend using home blood pressure devices proven to be accurate by independent reviewers.