Introducing CASUX: A Standardized Scale for Measuring the User Experience of Artificial Intelligence Based Conversational Agents

Abstract

Conversational agents are growing in popularity, and as such they must provide a good user experience and meet the needs of the users. Yet, how to measure the user experience in the conversational AI scenario remains an open and urgent question to be solved, which may hinder further empirical studies on human-agent interactions. In fact, there have been very few studies that explore how users perceive interacting with these conversational agents, which is important to ensure their sustainability. Accordingly, in this work, we used a subjective technique by following a measurement approach to develop a standardized measurement instrument/scale called the Conversational Agent Scale for User Experience. In terms of the methodology, we used a mixed-method approach involving an iterative process spanning across six different user studies (three qualitative and three quantitative) at different points of time for the purpose of dimension identification, item generation and subsequent item refinements. As a part of the qualitative studies we conducted a Systematic Literature Review, semi-structured interviews with users of conversational agents, and expert interviews. For the quantitative studies a lab-based experiment was performed for the pre-test, followed by two online surveys as a part of pilot testing and scale development. The qualitative studies initially identified a total of 13 distinct user experience dimensions and 418 measurement items. Finally, the Exploratory Factor Analysis as a part of the main survey resulted in 9 user experience dimensions (practicality perception, proficiency, humanness, sentiment, robustness, etiquette & mannerism, personality, anthropomorphism, and ease of use) and 34 measurement items. This was supplemented by conducting another Confirmatory Factor Analysis for establishing the reliability and validity of the proposed scale, together with checking the model-fit indices. All the 9 dimensions had sufficient validity, and a reasonable level of statistical reliability. Some of the user experience dimensions like humanness, personality, anthropomorphism, and etiquette & mannerism show the uniqueness of the conversational AI scenario from the traditional usage factors used commonly while evaluating graphical user interface-based systems. This work fills the gap of a lack of research on how to classify and measure the conversational experience of users and provides a reference for practitioners and designers in developing these agents and continuously improving the usage experience.