Talking with a Virtual Human: Controlling the Human Experience and Behavior in a Virtual Conversation

Abstract

Virtual humans are often designed to replace real humans in virtual reality applications for e.g., psychotherapy, education and entertainment. In general, applications with virtual humans are created for modifying a person's knowledge, beliefs, attitudes, emotions or behaviors. Reaching these intended goals, however, strongly depends on being able to control the conversation in these applications. Obviously important aspects to control such a conversation are speech recognition and natural language understanding and generation, but besides these aspects also the behavior of virtual humans and objects in the virtual environment may potentially influence the simulated conversation, and therefore, its effectiveness. Understanding which factors in a virtual environment may affect the dialog between a human and a virtual human, and finding ways to control the human experience and behavior during the conversation are the main aims of this thesis. Three main elements that characterize a conversation between a human and a virtual human were identified, i.e., the surrounding environment, the virtual conversation partner, and the virtual bystanders. Four separated empirical studies were conducted to investigate the effect of these three main elements in the domain of virtual reality exposure therapy for treating social anxiety disorders. The results show that priming materials in the virtual environment such as videos and pictures have a guiding effect on humans having a conversation with a virtual human. Also, emotions expressed when the virtual human speaks are perceived as more intense than emotions expressed when the virtual human listens, and emotions expressed while speaking had a larger effect on people's valence and discussion satisfaction. Furthermore, a positive attitude of the virtual conversation partner, i.e., a happy facial expression while constantly looking at the human conversation partner, and speaking with a positive voice intonation, elicits a more positive emotional state in humans as compared to a negative attitude, i.e., an angry facial expression while looking at the human conversation partner, and speaking with a negative voice intonation. Similarly, a positive attitude of virtual bystanders towards a person, i.e., happy facial expressions and whispering positive comments about the person's behavior, evokes more self-efficacy and less anxiety showing less avoidance behavior in the person compared to a negative attitude of the bystanders, i.e., angry facial expressions and whispering negative comments. In conclusion, by manipulating virtual objects, the virtual conversation partner or virtual bystanders, a therapist may affect the behavior, emotions and beliefs of a person.