A text-mining tool generated title-abstract screening workload savings: performance evaluation versus single-human screening

Abstract

Background and ObjectivesText-mining tool, Abstrackr, may potentially reduce the workload burden of title and abstract screening (Stage 1), using screening prioritization and truncation. This study aimed to evaluate the performance of Abstrackr's text-mining functions (‘Abstrackr-assisted screening’; screening undertaken by a single-human screener and Abstrackr) vs. Single-human screening. MethodsA systematic review of treatments for relapsed/refractory diffuse large B cell lymphoma (n = 7,723) was used. Citations, uploaded to Abstrackr, were screened by a human screener until a pre-specified maximum prediction score of 0.39540 was reached. Abstrackr's predictions were compared with the judgments of a second, human screener (who screened all citations in Covidence). The performance metrics were sensitivity, specificity, precision, false negative rate, proportion of relevant citations missed, workload savings, and time savings. ResultsAbstrackr reduced Stage 1 workload by 67% (5.4 days), when compared with Single-human screening. Sensitivity was high (91%). The false negative rate at Stage 1 was 9%; however, none of those citations were included following full-text screening. The high proportion of false positives (n = 2,001) resulted in low specificity (72%) and precision (15.5%). ConclusionAbstrackr-assisted screening provided Stage 1 workload savings that did not come at the expense of omitting relevant citations. However, Abstrackr overestimated citation relevance, which may have negative workload implications at full-text screening.