Aim: This study aims to assess the capabilities of artificial intelligence (AI) in answering online Continuing Medical Education (CME) courses to find the resistant to AI-misuse strategies. Materials and Methods: The study evaluated 30 CME online courses from popular American (ACCME), European (EACCME), and German Medical Association accredited online platforms, including Medscape, eaccme.uems.eu, Springer Nature, der-niedergelassene-arzt, and Aerzteblatt. ChatGPT Version 4.0 with integrated plugins for interactive AI chats with documents, web access to scientific databases, and interactive AI chats with videos was used to answer the CME evaluation questions. A special scoring system, referred to as "complexity score," was introduced in the study. This system has two main objectives: first, to assess strategies that prevent the misuse of AI in medical online education; second, to measure the effort that physicians must invest to answer CME questions using AI. Results: AI was used to answer a total of 248 questions, divided into three categories: ACCME accredited courses: 7 credits; EACCME accredited courses: 9.5 credits; German CME courses: 28 credits. AI successfully completed the quiz in 90% of cases (27 courses) and showed an accuracy rate of 86%. 213 out of 248 questions were correctly answered: 38 out of 48 ACCME questions; 85 out of 100 EACCME questions; 90 out of 100 CME questions. The outcome "AI error" was significantly associated only with a higher number of questions in the quiz: p-value 0.01. However, this predictor had no impact on the AI's ability to successfully complete the entire quiz. The AI failure rate was significantly associated with learning materials based on new studies without open access: p-value 0.02 and the need to view all learning materials to gain access to the quiz: p-value 0.02. A higher complexity score of the course was significantly associated with AI failure rates: p-value 0.0034. Conclusion: This study has shown that AI can successfully answer medical quiz questions even without access to learning materials. Therefore, the best strategy to prevent the misuse of AI in CME online training is to align human learning with AI feeding. Access to the quiz should only be possible after a complete review of the learning materials. This could be achieved by setting a fixed time or through multiple slides with separate access to each slide and subsequent quiz access.
Search from 250 M+ research papersSearch from 250 M+ research papers
Open Access
May 1, 2024
Ryan K Fawley 
