A multimodal dialogue system for customer service based on user personality adaptation and dialogue strategies

Abstract

ABSTRACT This paper presents a multimodal dialogue system with user personality adaptation and multiple nonverbal behavior generation for a conversational android robot that plays a customer service role in recommending travel plans. It is useful to estimate the user's personality to realize a dialogue system that adapts the dialogue strategy to individual users. To improve the user's impression of the conversation robot as a customer service agent, we must control the nonverbal behavior of the robot, such as its facial expressions and motions. Moreover, we need to design the dialogue strategy to provide information and make users enjoy the conversation by adding an ice-breaking step. Against this background, we develop a dialogue system prototype that adjusts the dialogue strategy based on the user's personality, as estimated by a pre-trained multimodal sensing model. Furthermore, we implement appropriate nonverbal behavior patterns for each situation, including the voice, motion, and facial expressions of the android robot, to provide it appropriate and polite service agent behaviors. A personality assessment game is introduced into the dialogue to prevent the user from becoming bored. The game plays an ice-breaking role in the dialogue. To let the user engage with the robot, we implement a complement to the user as a dialogue act. We evaluate the proposed multimodal dialogue system for customer service by means of participation in a dialogue robot competition. The fair evaluation results from 26 dialogue users who conversed with multiple systems show that the proposed dialogue system achieved the best score among 13 teams participating in the preliminary round of the competition, indicating that the user personality adaptation and various elements implemented in our dialogue system improve the dialogue experience of general users. The results of a third-party evaluation and multiple regression analysis show that natural response is the most important factor contributing to user impression.