Performance of generative pre-trained transformer-4 on the certification test for mental health management: A factorial design

Abstract

This study aimed to investigate the performance of generative pre-trained transformer-4 (GPT-4) on the Certification Test for Mental Health Management and whether tuned prompts could improve its performance. This study used a 3 × 2 factorial design to examine the performance according to test difficulty (courses) and prompt conditions. We prepared 200 multiple-choice questions (600 questions overall) for each course using the Certification Test for Mental Health Management (levels I-III) and essay questions from the level I test for the previous four examinations. Two conditions were used: a simple prompt condition using the questions as prompts and tuned prompt condition using techniques to obtain better answers. GPT-4 (gpt-4-0613) was adopted and implemented using the OpenAI API. The simple prompt condition scores were 74.5, 71.5, and 64.0 for levels III, II, and I, respectively. The tuned and simple prompt condition scores had no significant differences (OR = 1.03, 95% CI; 0.65-1.62, p = 0.908). Incorrect answers were observed in the simple prompt condition because of the inability to make choices, whereas no incorrect answers were observed in the tuned prompt condition. The average score for the essay questions under the simple prompt condition was 22.5 out of 50 points (45.0%). GPT-4 had a sufficient knowledge network for occupational mental health, surpassing the criteria for levels II and III tests. For the level I test, which required the ability to describe more advanced knowledge accurately, GPT-4 did not meet the criteria. External information may be needed when using GPT-4 at this level. Although the tuned prompts did not significantly improve the performance, they were promising in avoiding unintended outputs and organizing output formats. UMIN trial registration: UMIN-CTR ID = UMIN000053582.