302: QUANTITATIVE AND QUALITATIVE EVALUATION OF ARTIFICIAL INTELLIGENCE-BASED CRITICAL CARE PUBLICATIONS

Abstract

Introduction: Critical care utilizes complex data platforms to inform time-sensitive decisions for appropriate patient management. Unsurprisingly, increasing number of Artificial Intelligence (AI) models are developed and published in critical care medicine. However, despite the ongoing increase in interest in using AI, the general translation of academic research into deployable AI solutions has been challenging and limited. In this work, we provide a quantitative and qualitative assessment of AI based critical care research publications over time using state-of-the-art NLP(Natural Language Processing) models. Methods: We performed a PubMed search using the terms, “machine learning” or “artificial intelligence”, “healthcare”, restricted to English language and human subject research for the years 2019, 2020 and 2021. We excluded publications due to errors in the PubMed search, limited gene studies and short commentaries. From the final set of selected articles, we manually selected publications pertaining to critical care. We then used a Bidirectional Encoder Representation from Transformer (BERT)-based maturity classification model, that was pre-trained and validated on manually labeled data for ‘Mature’ and ‘Not Mature’ publications, to assess the level of maturity of each selected critical care publication. The level of maturity was determined by the answer to the question, “Does the output of the proposed model have a direct, actionable impact on patient care by providing information to healthcare providers or automated systems?” Results: Our initial PubMed search for the number of AI based publications in healthcare yielded, 1720, 3222, and 2182 articles for the years 2019, 2020, and 2021, respectively. Out of these 32, 41, and 24 publications were critical care related in their respective years. Using the BERT the maturity model, overall 99, 250, and 36 of articles were classed as mature; however, only 0, 2 and 1 of critical care articles published in 2019, 2020, and 2021 respectively, were labeled as mature. Conclusions: With the growing number of publications of AI models in critical care, there is an opportunity to research and publish models with higher levels of maturity and likelihood of clinical deployment, through enhanced collaboration between computer scientists and clinicians.