Developing and validating a knowledge-based AI assessment system for learning clinical core medical knowledge in otolaryngology

Abstract

BackgroundClinical core medical knowledge (CCMK) learning is essential for medical trainees. Adaptive assessment systems can facilitate self-learning, but extracting experts' CCMK is challenging, especially using modern data-driven artificial intelligence (AI) approaches (e.g., deep learning). ObjectivesThis study aims to develop a multi-expert knowledge–aggregated adaptive assessment scheme (MEKAS) using knowledge-based AI approaches to facilitate the learning of CCMK in otolaryngology (CCMK-OTO) and validate its effectiveness through a one-month training program for CCMK-OTO education at a tertiary referral hospital. MethodsThe MEKAS utilized the repertory grid technique and case-based reasoning to aggregate experts' knowledge to construct a representative CCMK base, thereby enabling adaptive assessment for CCMK-OTO training. The effects of longitudinal training were compared between the experimental group (EG) and the control group (CG). Both groups received a normal training program (routine meeting, outpatient/operation room teaching, and classroom teaching), while EG received MEKAS for self-learning. The EG comprised 22 UPGY trainees (6 postgraduate [PGY] and 16 undergraduate [UGY] trainees) and 8 otolaryngology residents (ENT-R); the CG comprised 24 UPGY trainees (8 PGY and 16 UGY trainees). The training effectiveness was compared through pre- and post-test CCMK-OTO scores, and user experiences were evaluated using a technology acceptance model-based questionnaire. ResultsBoth UPGY (z = −3.976, P < 0.001) and ENT-R (z = −2.038, P = 0.042) groups in EG exhibited significant improvements in their CCMK-OTO scores, while UPGY in CG did not (z = −1.204, P = 0.228). The UPGY group in EG also demonstrated a substantial improvement compared to the UPGY group in CG (z = −4.943, P < 0.001). The EG participants were highly satisfied with the MEKAS system concerning self-learning assistance, adaptive testing, perceived satisfaction, intention to use, perceived usefulness, perceived ease of use, and perceived enjoyment, rating it between an overall average of 3.8 and 4.1 out of 5.0 on all scales. ConclusionsThe MEKAS system facilitates CCMK-OTO learning and provides an efficient knowledge aggregation scheme that can be applied to other medical subjects to efficiently build adaptive assessment systems for CCMK learning. Larger-scale validation across diverse institutions and settings is warranted further to assess MEKAS's scalability, generalizability, and long-term impact.