Sensitivity and specificity of alternative screening methods for systematic reviews using text mining tools

Abstract

ObjectivesTo evaluate the impact of text mining (TM) on the sensitivity and specificity of title and abstract screening strategies for systematic reviews (SRs). Study Design and SettingTwenty reviewers each evaluated a 500-citation set. We compared five screening methods: conventional double screen (CDS), single screen, double screen with TM, combined double screen and single screen with TM, and single screen with TM. Rayyan, Abstrackr, and SWIFT-Review were used for each TM method. The results of a published SR were used as the reference standard. ResultsThe mean sensitivity and specificity achieved by CDS were 97.0% (95% confidence interval [CI]: 94.7, 99.3) and 95.0% (95% CI: 93.0, 97.1). When compared with single screen, CDS provided a greater sensitivity without a decrease in specificity. Rayyan, Abstrackr, and SWIFT-Review identified all relevant studies. Specificity was often higher for TM-assisted methods than that for CDS, although with mean differences of only one-to-two percentage points. For every 500 citations not requiring manual screening, 216 minutes (95% CI: 169, 264) could be saved. ConclusionTM-assisted screening methods resulted in similar sensitivity and modestly improved specificity as compared to CDS. The time saved with TM makes this a promising new tool for SR.