Screening for common mental health disorders: a psychometric evaluation of a chatbot system

Abstract

ABSTRACT The current study presents the psychometrics and screening accuracy properties of a chatbot system that understands free-text responses to mental health screening questions using natural language processing (NLP). The aiCARE system was tested against web-based versions of the Patient Health Questionnaire- 9 (PHQ-9), General Anxiety Disorder-7 (GAD-7), and Posttraumatic Stress Disorder Checklist (PCL-5). The study included 773 volunteers (Mage = 21.28, SD = 5.34) who answered the same free-text (chatbot version) and closed-ended survey questions (standard survey version). Overall, the research found that the proposed chatbot system is reliable in determining whether clinical symptomatology is present or absent based on free-text responses to PHQ-9, GAD-7, and PCL-5 questions. It had comparable sensitivity, specificity, total accuracy, and AUC values to standard web-based survey methods, as well as good internal consistency and convergent validity. The general implications are that chatbot systems could be used to identify common psychopathology as part of a stepped care model. It is not intended to be used in place of clinical diagnosis. Future research is needed to assess its effectiveness in more clinically and demographically diverse populations.