GPT-4/4V's performance on the Japanese National Medical Licensing Examination

Abstract

Background Recent advances in Artificial Intelligence (AI) are changing the medical world, and AI will likely replace many of the actions performed by medical professionals. The overall clinical ability of the AI has been evaluated by its ability to answer a text-based national medical examination. This study uniquely assesses the performance of Open AI's ChatGPT against all Japanese National Medical Licensing Examination (NMLE), including images, illustrations, and pictures. Methods We obtained the questions of the past six years of the NMLE (112th to 117th) from the Japanese Ministry of Health, Labour and Welfare website. We converted them to JavaScript Object Notation (JSON) format. We created an application programming interface (API) to output correct answers using GPT-4 for questions without images and GPT4-V(ision) or GPT4 console for questions with images. Results The percentage of image questions was 723/2400 (30.1%) over the past six years. In all years, GPT-4/4V exceeded the minimum score the examinee should score. In total, over the six years, the percentage of correct answers for basic medical knowledge questions was 665/905 (73.5%); for clinical knowledge questions, 1143/1531 (74.7%); and for image questions 497/723 (68.7%), respectively. Conclusions Regarding medical knowledge, GPT-4/4V met the minimum criteria regardless of whether the questions included images, illustrations, and pictures. Our study sheds light on the potential utility of AI in medical education.