PDB90 IS IT POSSIBLE TO AUTOMATE A SYSTEMATIC LITERATURE REVIEW?

Abstract

The aim was to investigate whether screening abstracts for a systematic literature review could be automated by a program using natural language processing. A recently published Cochrane systematic review in 2018 by Lo et al was selected as the control due to its high quality and level of interest. We extracted the list of 185 abstracts included in the initial review. These were then subjected to a Python application, which analysed the text of each abstract for relevance and quality. Our program aimed to replicate the PICO framework, whereby it looked for the appropriate diabetic population with chronic kidney disease in the abstract text. It then looked for the appropriate interventions of insulin or other antidiabetic agents. Finally, the program searched for the following outcomes of interest: HbA1c, FBG, weight, all-cause death, CV death, eGFR and discontinuation. Of the 185 abstracts identified, 44 abstracts in the original publication were screened in and included for full-text review. Our program resulted in a positive-predictive value of 71.4%, a false-positive rate of 11.0% and a false-negative rate of 28.6%. The results suggest that it is possible to automate the screening of a literature review to a reasonable accuracy. However, further improvements are required to ensure reliability. It is important to keep in mind that programs will always bear the inherent bias of its developers. In our case, we tried to minimise bias by remaining blinded to those abstracts that passed screening during development of the program. Only upon completion of the program did we then compare those screened in vs out. We are also in the process of automating full text reviews, numerical data extraction and analysis, which we hope to incorporate into this process, resulting in an accurate and efficient method of automating the entire literature review process.