Quantity and type of peer-reviewed evidence for popular free medical apps: Cross-sectional review

Abstract

Introduction– Mobile apps are being increasingly used as a tool to deliver clinical care. Evidence of efficacy for such apps varies, and appropriate levels of evidence may depend on the app’s intended use. The UK’s National Institute for Health and Care Excellence (NICE) recently developed an evidence standards framework, aiming to explicitly set out the required standards of evidence for different categories of digital health technologies. To determine current compliance with the evidence standards framework, the current study quantified the amount and type of peer-reviewed evidence associated with a cross-section of popular medical apps. Methods– Apps were identified by selecting the top 100 free medical apps in the Apple App Store and all free apps in the NHS Apps Library. Each app was assigned to one of the four tiers (1, 2, 3a, 3b) in the NICE evidence standards framework. For each app, we conducted searches in Ovid-MEDLINE, Web of Science, Google Scholar, and via manufacturer websites to identify any published articles that assessed the app. This allowed us to determine our primary outcome, whether apps in tiers 3a/3b were more likely than apps in tier 1/2 to be associated with academic peer-reviewed evidence. Results– We reviewed 125 apps in total (Apple App Store (n = 72), NHS Apps Library (n = 45), both (n = 8), of which 54 were categorized into the higher evidence standards framework tiers, 3a/3b. After screening, we extracted 105 relevant articles which were associated with 25 of the apps. Only 6 articles, pertaining to 3 apps, were reports of randomised controlled trials. Apps in tiers 3a/3b were more likely to be associated with articles than apps in lower tiers (χ2 = 5.54, p = .01). The percentage of tier 3a/3b apps with associated articles was similar for both the NHS Apps Library (10/28) and Apple App store (7/24), (χ2 = 0.042, p = .84). Discussion– Apps that were in higher tiers 3a and 3b, indicating higher clinical risk, were more likely to have an associated article than those in lower categories. However, even in these tiers, supporting peer-reviewed evidence was missing in the majority of instances. In our sample, Apps from the NHS Apps Library were more no more likely to have supporting evidence than popular Apple App Store apps. This is of concern, given that NHS approval may influence uptake of app usage.