Users' experiences with chatbots: findings from a questionnaire study

Asbjørn Følstad,Petter Bae Brandtzaeg

doi:10.1007/s41233-020-00033-2

Abstract

For chatbots to be broadly adopted by users, it is critical that they are experienced as useful and pleasurable. While there is an emerging body of research concerning user uptake and use of chatbots, there is a lack of theoretically grounded studies detailing what constitutes good or poor chatbot user experiences. In this paper, we present findings from a questionnaire study involving more than 200 chatbot users who reported on episodes of chatbot use that they found particularly satisfactory or frustrating. The user reports were analysed with basis in theory on user experience, with particular concern for pragmatic and hedonic attributes. We found that pragmatic attributes such as efficient assistance (positive) and problems with interpretation (negative) were important elements in user reports of satisfactory and frustrating episodes. Hedonic attributes such as entertainment value (positive) and strange and rude responses (negative) were also frequently mentioned. Older participants tended to report on pragmatic attributes more often, whereas younger participants tended to report on hedonic attributes more often. Drawing on the findings, we propose four high-level lessons learnt that may benefit chatbot service providers, and we suggest relevant future research.

Highlights

Chatbots are machine agents with which users interact through natural language dialogue, by text or voice [4]
We found an appreciation for both pragmatic attributes and hedonic attributes in participants’ detailing of positive chatbot user experiences; 42% of the participant reports reflected pragmatic chatbot attributes and 36% highlighted hedonic attributes
We foresee more in-depth investigations into the relative differences between task-oriented chatbots and chatbots oriented towards social interaction with respect to how pragmatic and hedonic attributes can be leveraged to improve user experience

Summary

Introduction

Chatbots are machine agents with which users interact through natural language dialogue, by text or voice [4]. While chatbots were initially developed to mimic human conversation [47], often in the form of social chatter, current chatbots are typically task-oriented, enabling their users to achieve specific goals or outcomes [41]. Current task-oriented chatbots are typically designed in consideration of the emotional or social aspects of conversational interaction [22]. We position our understanding of user experience within what Forlizzi and Battarbee [18] would refer to as user-centred models, as we mainly consider the perspective of people who use chatbots. Following the international standard for human-centred design of interactive systems, we understand user experience as a ‘person’s perceptions and responses resulting from the use and/or anticipated use of a product, system or service’ [29, p. In line with Law et al [35], we consider user experience to be dynamic, context-dependent and subjective

Objectives

Methods

Results

Discussion

Conclusion