135 Acceptance and Barriers of AI-Based Decision Support Systems in Emergency Departments: A Quantitative and Qualitative Evaluation

Abstract

Computerized Decision Support Systems (CDSSs) are developed to assist clinicians in their decision-making processes with the ultimate goal of improving patients’ outcomes. Although many studies have discussed the acceptance and barriers of CDSSs, few have been investigated with theoretically grounded frameworks, especially AI-based CDSSs. We aimed to identify the acceptance and barriers of our newly developed CDSS by using the Unified Theory of Acceptance and Use of Technology (UTAUT) and the Consolidated Framework for Implementation Research (CFIR) frameworks. This mixed-methods study was performed from March to April 2021. Transitional year residents (n=6), emergency medicine residents (n=5), and emergency physicians (n=3) from two community, tertiary care hospitals in Japan were included. We first developed a real-time CDSS for predicting aortic dissection based on numeric and text information from medical charts (eg, chief complaints, medical history, vital signs) with natural language processing. To address anchoring bias, the system displays an alert with an explanation on the screen when the probability of aortic dissection changes sharply or exceeds the prespecified threshold. This system was deployed on the Internet, and the participants used the system with clinical vignettes based on actual cases. Participants were then involved in a mixed-methods evaluation consisting of a UTAUT-based questionnaire with a 5-point Likert scale (quantitative) and a CFIR-based semi-structured interview (qualitative). Interviews were sampled, transcribed, and analyzed using the MaxQDA software. A deductive approach to qualitative content analysis was adopted incorporating four of the CFIR’s domains. The qualitative data were interpreted through coding and a summary matrix of the domains. All 14 participants completed the questionnaires and interviews. Quantitative analysis revealed generally positive responses for user acceptance. Effort expectancy (median [IQR], 4 [3-4]; 5 indicates most positive) was especially high which led to high behavior intention (median [IQR], 3 [3-4]). Qualitative analysis suggested that the design quality was high and was perceived to be neither interruptive nor effortful. In addition, it implied that the system was reliable enough. Despite its reliability and ease of use, alerts by the CDSS tended to be ignored for patients with typical presentations, for whom participants felt more confident about the diagnostic algorithms. Moreover, while real-time updates of alerts were perceived useful, it led to concerns that the heavy system load may lead to errors in the system. Using theoretically grounded frameworks enabled us to thoroughly understand the acceptance and barriers of implementation of CDSS. Although our real-time CDSS was qualitatively shown to be less effective on typical cases with concerns of system failure, the mixed-methods evaluation demonstrated that the CDSS was intuitive and reliable and thus has good user acceptance.