MSR117 Are Classifiers the Future of Artificial Intelligence in Systematic Literature Reviews?

Abstract

Over the past decade, AI algorithms have been increasingly incorporated into literature review software and used at many stages of the review process, from searches and screening to extraction of data and synthesis of results. AI is most often utilized to facilitate title and abstract screening; however, its current use as a secondary or independent reviewer is largely limited to binary decisions of reference inclusion or exclusion. While this is a valuable asset in reducing the screening burden for human reviewers, the lack of AI’s ability to provide a rationale for its decision making can be a substantial obstacle to its more widespread adoption, particularly when conducting systematic reviews for submission to Health Technology Assessment (HTA) agencies. In this sense, it is imperative that AI be able to provide more granularity around its decision making by providing reasons for inclusion or exclusion in a similar manner to which decisions and assumptions are typically made by human reviewers during the screening process. The use of machine learning classifiers, which are specific algorithms capable of classifying data into different categories, could enhance the existing binary include/exclude approach to more closely align with complex and often multi-faceted selection criteria. Implementation of classifiers can thereby provide more confidence in the accuracy of the algorithm or indicate where more references or better examples are needed to train the AI. Greater visibility into AI decisions may lead to increased uptake by researchers, particularly within highly complex reviews. This may also make a step towards future acceptance of AI technology within the stringent review frameworks required by HTA bodies. With classifiers, AI edges closer to being able to replicate human reviewers, improving efficiency in the screening process and reducing time and resources required to systematically review large bodies of literature.