What Are the Users’ Needs? Design of a User-Centered Explainable Artificial Intelligence Diagnostic System

Abstract

The application of artificial intelligence (AI) systems has surged in the high-risk area of medicine, and these systems must explain their decisions to different users. However, existing explainable AI (XAI) design practices in the medical domain are mostly focused on domain experts, such as physicians, and there is a lack of XAI design practices for consumer users. Therefore, we developed a library of XAI user needs in the medical domain, which can be used as an auxiliary tool for the development of user-centered XAI design solutions in this domain. Moreover, through empirical research, based on our XAI user Needs Library, we designed an XAI-based electrocardiogram diagnostic system prototype for consumer users and conducted a user evaluation. The results provide the empirical experience of the design space of XAI and promote consumer user-centered XAI practices.