AI2 LEVERAGING MACHINE-ASSISTANCE TO REPLICATE A SYSTEMATIC REVIEW

Abstract

Systematic literature review (SLR) is an important, but time-consuming approach to synthesizing the literature around a topic. This study utilized Artificial Intelligence (AI) capable of extracting numeric results (e.g. “4 of 10”), interventions used (e.g., “vedolizumab”), outcomes (e.g., “remission at 14 weeks”), and diseases from text. We combined the machine extracted results with manual cleaning and compared the results against a published Systematic Literature Review (SLR). A target SLR was chosen for its recency and clinical importance (a 2017 SLR focused on vedolizumab to treat ulcerative colitis and Crohn’s disease). Two primary end-points from the SLR were chosen as targets to replicate (14 week and 12-month remission, respectively). The AI was pointed at PubMed and retrieved 383 papers related to vedolizumab. The AI filtered down to 58 papers that appeared to be Randomized Controlled Trials or Real-World studies with a minimum quality score (using our previously published method for automated screening). This final set of papers was run through the AI extraction component and the results were manually verified and cleaned, if needed. The SLR comparison set contained 14 papers from PubMed, and 4 papers collected manually (missed by the automation initially). These papers covered 94% of those in the SLR (one paper’s result was communicated scientist-to-scientist). All extracted and cleaned results matched the values exactly as reported in the SLR. The process of gathering the papers and running the AI took a single work day. The manual cleaning took 5 days of effort, for a total effort of 6 days. We demonstrated that using AI to assist people in crafting SLR can reduce the time from an average of 3-6 months to 6 days. This time will only decrease as AI methods increase in sophistication and breadth of accessible data.