User experience with conversational agent: a systematic review of assessment methods

Abstract

ABSTRACT Conversational agents are becoming popular for providing a more natural and realistic user experience. New studies have become significant to understand how to assess this experience, mainly because of the increase in applications of this nature. We systematically reviewed the literature to identify how the user experience is assessed when interacting with conversational agents. A total of 443 studies were identified in the ACM, IEEE, Springer, and Scopus databases. Of these, 27 studies met the eligibility criteria. Most studies used their own evaluation methods, without adopting questionnaires validated for UX evaluation. Few studies used assessment tools before participants interacted with agents, and only two carried out assessments before, during, and after use. The results of the assessments can be better if specific instruments for UX are adopted. Furthermore, it is necessary to assess the experience at different times and use combined methods, to understand aspects related to the participants’ feelings and behaviours.