Embodied Conversational Agents Research Articles

Promoting Transparent and Consistent Frameworks for Interactive Digital Testimonies: A Case Study on Preserving Zilli Schmidt’s Story

Interactive Digital Testimonies (IDTs) combine digital archives of purpose-made recordings, conversational agents, and immersive display technology to preserve and recreate interactive conversations with contemporary witnesses in a lifelike manner. IDTs represent a specific subcategory of (Embodied) Conversational Agents (ECAs) due to the constraint of not including AI-generated or otherwise synthetic responses or actions. While numerous IDTs have been developed over the last few years, the descriptions of these systems and their respective evaluations frequently lack consistency and transparency, which has led to considerable heterogeneity and a lack of comparability. To counteract these developments, we present the IDT of Holocaust survivor and member of the German-speaking Romani community Zilli Schmidt, which we developed since 2021. We transparently share both content-related and technical features of this IDT.

Reengineering eADVICE for Long Waitlists: A Tale of Two Systems and Conditions

Long outpatient waiting times pose a significant global challenge in healthcare, impacting children and families with implications for health outcomes. This paper presents the eHealth system called eADVICE (electronic Advice and Diagnosis Via the Internet following Computerised Evaluation) that is designed to address waiting list challenges for paediatricians. Initially designed for children’s incontinence, the system’s success in terms of health goals and user experience led to its adaptation for paediatric sleep problems. This paper focuses on user experiences and the development of a working alliance with the virtual doctor, alongside health outcomes based on a randomised controlled trial (N = 239) for incontinence. When reengineering eADVICE to sleep disorders, the promising results regarding the reciprocal relationship between user experience and building a working alliance encouraged a focus on the further development of the embodied conversational agent (ECA) component. This involved tailoring the ECA discussion to patient cognition (i.e., beliefs and goals) to further improve engagement and outcomes. The proposed eADVICE framework facilitates adaptation across paediatric conditions, offering a scalable model to enhance access and self-efficacy during care delays.

An Economic Evaluation of a Web-Based Management Support System for Children With Urinary Incontinence: The eADVICE Trial.

Children who require specialist outpatient care typically wait substantial periods during which their condition may progress, making treatment more difficult and costly. Timely and effective therapy during this period may reduce the need for lengthy specialist care. This study evaluated the cost-effectiveness of an individualized, evidence-informed, web-based program for children with urinary incontinence awaiting a specialist appointment (Electronic Advice and Diagnosis Via the Internet following Computerized Evaluation [eADVICE]) compared to usual care. eADVICE was supervised by a primary physician and delivered by an embodied conversational agent. A trial-based cost-effectiveness analysis was performed from the perspective of the health care funder as a substudy of eADVICE, a multicenter, waitlist-controlled, randomized trial. Outcomes measures were incremental cost per incremental change in continence status and quality of life on an intention-to-treat basis. Uncertainty was examined using cost-effectiveness planes, scenarios, and 1-way sensitivity analyses. Costs were valued in 2021 Australian dollars. The use of eADVICE was found to be cost saving and beneficial (dominant) over usual care, with a higher proportion of children dry over 14 days at 6 months (risk difference 0.13; 95%CI 0.02-0.23, P = .03) and mean health care costs reduced by $188 (95%CI $61-$315) per participant. An individualized, evidence-informed, web-based program delivered by an embodied conversational agent is likely cost saving for children with urinary incontinence awaiting a specialist appointment. The potential economic impact of such a program is favorable and substantial, and may be transferable to outpatient clinic settings for other chronic health conditions.

Adapting the Number of Questions Based on Detected Psychological Distress for Cognitive Behavioral Therapy With an Embodied Conversational Agent: Comparative Study.

The high prevalence of mental illness is a critical social problem. The limited availability of mental health services is a major factor that exacerbates this problem. One solution is to deliver cognitive behavioral therapy (CBT) using an embodied conversational agent (ECA). ECAs make it possible to provide health care without location or time constraints. One of the techniques used in CBT is Socratic questioning, which guides users to correct negative thoughts. The effectiveness of this approach depends on a therapist's skill to adapt to the user's mood or distress level. However, current ECAs do not possess this skill. Therefore, it is essential to implement this adaptation ability to the ECAs. This study aims to develop and evaluate a method that automatically adapts the number of Socratic questions based on the level of detected psychological distress during a CBT session with an ECA. We hypothesize that this adaptive approach to selecting the number of questions will lower psychological distress, reduce negative emotional states, and produce more substantial cognitive changes compared with a random number of questions. In this study, which envisions health care support in daily life, we recruited participants aged from 18 to 65 years for an experiment that involved 2 different conditions: an ECA that adapts a number of questions based on psychological distress detection or an ECA that only asked a random number of questions. The participants were assigned to 1 of the 2 conditions, experienced a single CBT session with an ECA, and completed questionnaires before and after the session. The participants completed the experiment. There were slight differences in sex, age, and preexperimental psychological distress levels between the 2 conditions. The adapted number of questions condition showed significantly lower psychological distress than the random number of questions condition after the session. We also found a significant difference in the cognitive change when the number of questions was adapted based on the detected distress level, compared with when the number of questions was fewer than what was appropriate for the level of distress detected. The results show that an ECA adapting the number of Socratic questions based on detected distress levels increases the effectiveness of CBT. Participants who received an adaptive number of questions experienced greater reductions in distress than those who received a random number of questions. In addition, the participants showed a greater amount of cognitive change when the number of questions matched the detected distress level. This suggests that adapting the question quantity based on distress level detection can improve the results of CBT delivered by an ECA. These results illustrate the advantages of ECAs, paving the way for mental health care that is more tailored and effective.

Talk with Ted: an embodied conversational agent for caregivers

ABSTRACT Communication is key to the success of any relationship. When it comes to caregivers, having a conversation with a person living with some form of cognitive impairment, such as dementia, can be a struggle. Most people living with dementia experience some form of communication impairment that reduces their ability to express their needs. In this case study, we present the design of an embodied conversation agent (ECA), Ted, designed to educate caregivers about the importance of good communication principles when engaging with people living with dementia. This training tool was trialed and compared to an online training tool, with 23 caregivers divided into two cohorts (12 in the ECA condition, and 11 in the online training tool condition), over a period of 8 weeks using a mixed evaluation approach. Our findings suggest that (a) caregivers developed an emotional connection with the ECA and retained the learning from their interactions with Ted even after 8 weeks had elapsed, (b) caregivers implemented the learnings in their practice, and (c) the changes in care practice were well received by people living with dementia.

Inclusive Learning through Immersive Virtual Reality and Semantic Embodied Conversational Agent: A case study in children with autism

Sustainable Development Goal 4 from the United Nations 2030 Agenda, focus on ensuring inclusive and equitable quality education and promoting lifelong learning opportunities for all. Inclusive education is a key component of sustainable development goal 4, and assistive technology is a critical factor in achieving it. In this context, this paper introduces the development of an immersive virtual reality system with conversational skills for inclusive learning processes. The idea of this work is to provide an assistive technology to motivate, educate, and train students with disabilities in a more inclusive way. In order to improve the user experience, the system was developed to work in a CAVE-like immersive environment using Natural Language Processing through a Semantic Embodied Conversational Agent. This work highlights that assistive technology can promote educational, psychological, and social benefits for students with disabilities. The use of assistive technology can enable academic engagement and social participation and be transformative from a psychological perspective. A case study was conducted on children with autism, which showed encouraging results of the system as an assistive technology for teaching and learning.

The Co-Design of an Embodied Conversational Agent to Help Stroke Survivors Manage Their Recovery

Whilst the use of digital interventions to assist patients with self-management involving embodied conversational agents (ECA) is emerging, the use of such agents to support stroke rehabilitation and recovery is rare. This iTakeCharge project takes inspiration from the evidence-based narrative style self-management intervention for stroke recovery, the ‘Take Charge’ intervention, which has been shown to contribute to significant improvements in disability and quality of life after stroke. We worked with the developers and deliverers of the ‘Take Charge’ intervention tool, clinical stroke researchers and stroke survivors, to adapt the ‘Take Charge’ intervention tool to be delivered by an ECA (i.e., the Taking Charge Intelligent Agent (TaCIA)). TaCIA was co-designed using a three-phased approach: Stage 1: Phase I with the developers and Phase II with people who delivered the original Take Charge intervention to stroke survivors (i.e., facilitators); and Stage 2: Phase III with stroke survivors. This paper reports the results from each of these phases including an evaluation of the resulting ECA. Stage 1: Phase I, where TaCIA V.1 was evaluated by the Take Charge developers, did not build a good working alliance, provide adequate options, or deliver the intended Take Charge outcomes. In particular, the use of answer options and the coaching aspects of TaCIA V.1 were felt to conflict with the intention that Take Charge facilitators would not influence the responses of the patient. In response, in Stage 1: Phase II, TaCIA V.2 incorporated an experiment to determine the value of providing answer options versus free text responses. Take Charge facilitators agreed that allowing an open response concurrently with providing answer options was optimal and determined that working alliance and usability were satisfactory. Finally, in Stage 2: Phase III, TaCIA V.3 was evaluated with eight stroke survivors and was generally well accepted and considered useful. Increased user control, clarification of TaCIA’s role, and other improvements to improve accessibility were suggested. The article concludes with limitations and recommendations for future changes based on stroke survivor feedback.

Optimizing appreciation and persuasion of embodied conversational agents for health behavior change: A design experiment and focus group study.

Embodied Conversational Agents (ECAs) can increase user engagement and involvement and can strengthen the effect of an intervention on health outcomes that is provided via an ECA. However, evidence regarding the effectiveness of ECAs on health outcomes is still limited. In this article, we report on a study that has the goal to identify the effect of a match between a health topic and the ECAs' appearance on ratings of personality characteristics, persuasiveness and intention to use. We report on an online experiment with three different ECAs and three different health topics, conducted among 732 older adults. We triangulated the quantitative results with qualitative insights from a focus group. The results reveal that older adults prefer an ECA that has an appearance matching a certain health topic, resulting in higher ratings on persuasiveness and intention to use. Personality characteristics should be measured embedded within a health topic, but are not rated higher because of a match. We furthermore provide guidelines for designing the content of the ECA.

From deadpan machine to relating socially: Middle school students’ experiences speaking English with embodied conversational agents

Speaking with an artificial counterpart in simulated environments has been shown beneficial for foreign language learning. Still, few studies have explored how it is experienced by younger students. We report on a study of Swedish students (N = 22) speaking English with embodied conversational agents (ECAs) in everyday-life scenarios. Data were collected on students’ rankings, choices, and open-response items in logbooks and questionnaires. Self-reported data on experiences were analyzed through three dimensions: cognitive, emotional, and social. Findings show that students were generally satisfied with the activity and emotionally engaged with large individual differences within the social dimension. We unpack aspects regarding social relating to the ECA, analyzing the space between experiencing the ECA as a socially distant “deadpan machine” to humanizing and relating socially.

This embodied conversational agent looks very human and as old as I feel! The effect of perceived agent anthropomorphism and consumer-agent age difference on brand attitude

ABSTRACT In a study with 320 Canadian participants, we explore the effect of perceived physical and non-physical anthropomorphism of an Embodied Conversational Agent (ECA) and the perceived actual and subjective age difference between an individual and this ECA on the ECA’s likeability and credibility. We also explore the effect of likeability and credibility on the attitude towards the website on which the ECA appears and the brand on the website. Perceived physical anthropomorphism has a positive effect on ECA likeability and credibility. The perceived differences between a consumer’s subjective and actual age and perceived ECA age have a negative effect on ECA likeability. This effect is attenuated by an interaction effect between the subjective age difference and perceived non-physical anthropomorphism: for a given level of the subjective age difference, the more the ECA is perceived as having human emotions and motivations, the more positive the effect on ECA likeability is. ECA likeability and credibility lead to more positive attitudes towards the website and the brand. For the conceptualization of our study, we draw upon the Attraction to Similarity Theory, complemented by insights from anthropomorphism studies, Self-Congruity Theory, and relational characteristics research. Theoretical and managerial contributions are discussed.

80.3 The Behavioral Health Avatar for Automatic Nonverbal Behavioral Analysis for Personalizing Treatment of Bipolar Disorder

Improving diagnostic accuracy of bipolar spectrum disorder (BSD) is clearly indicated to better inform treatment decisions. This proof-of-concept study aims to advance and improve treatment options for youth with BSD. This innovative proposal introduces the Behavioral Health AVATAR (BHA), which is an embodied conversational agent (ECA) that conducts natural and noncontact diagnostic interviews. During the interview, the BHA measures pupil dilation, eye fixation location, facial expression, body rigidity, vocal acoustics, and linguistic content.

A Review of Evaluation Practices of Gesture Generation in Embodied Conversational Agents

Embodied conversational agents (ECA) are often designed to produce nonverbal behavior to complement or enhance their verbal communication. One such form of nonverbal behavior is co-speech gesturing, which involves movements that the agent makes with its arms and hands that are paired with verbal communication. Co-speech gestures for ECAs can be created using different generation methods, divided into rule-based and data-driven processes, with the latter gaining traction because of the increasing interest from the applied machine learning community. However, reports on gesture generation methods use a variety of evaluation measures, which hinders comparison. To address this, we present a systematic review on co-speech gesture generation methods for iconic, metaphoric, deictic, and beat gestures, including reported evaluation methods. We review 22 studies that have an ECA with a human-like upper body that uses co-speech gesturing in social human-agent interaction. This includes studies that use human participants to evaluate performance. We found most studies use a within-subject design and rely on a form of subjective evaluation, but without a systematic approach. We argue that the field requires more rigorous and uniform tools for co-speech gesture evaluation, and formulate recommendations for empirical evaluation, including standardized phrases and example scenarios to help systematically test generative models across studies. Furthermore, we also propose a checklist that can be used to report relevant information for the evaluation of generative models, as well as to evaluate co-speech gesture use.

Changing users’ health behaviour intentions through an embodied conversational agent delivering explanations based on users’ beliefs and goals

ABSTRACT Interventions to improve health and well-being abound. Whether they are designed for prevention, maintenance or improvement, a key challenge is the motivation of the user to change their current behaviours, such as persisting or taking new actions. To encourage someone to change their behaviour requires persuading them to change their goals and/or their beliefs about the behaviour or their ability to perform it. Our embodied conversational agent (ECA) uses explanations based on the goals and beliefs of the user to promote a sense of personalisation and engagement with the treatment plan which could form a bond as the dyad develop shared goals and tasks together. To keep our message minimal and understand whether belief-based or goal-based explanations are more efficacious in changing behaviour intention, we collected data in the context of a scenario where the ECA seeks to change four behaviours recommended to help students manage their study stress. Our findings suggest that when the behaviour requires a change in desire, we need goal-based explanation, when adoption of the behaviour requires addressing a barrier we need belief-based explanation and warrant future investigation. Further, the stratified analysis suggested that more tailoring to the student’s context could provide more motivation to change.

The multimodal EchoBorg: not as smart as it looks

In this paper we present a Multimodal Echoborg interface to explore the effect of different embodiments of an Embodied Conversational Agent (ECA) in an interaction. We compared an interaction where the ECA was embodied as a virtual human (VH) with one where it was embodied as an Echoborg, i.e, a person whose actions are covertly controlled by a dialogue system. The Echoborg in our study not only shadowed the speech output of the dialogue system but also its non-verbal actions. The interactions were structured as a debate between three participants on an ethical dilemma. First, we collected a corpus of debate sessions with three humans debaters. This we used as baseline to design and implement our ECAs. For the experiment, we designed two debate conditions. In one the participant interacted with two ECAs both embodied by virtual humans). In the other the participant interacted with one ECA embodied by a VH and the other by an Echoborg. Our results show that a human embodiment of the ECA overall scores better on perceived social attributes of the ECA. In many other respects the Echoborg scores as poorly as the VH except copresence.

An embodied conversational agent coach to support societal participation learning by low-literate users

People of low literacy could benefit from automated support when learning about societal participation. We design an Embodied Conversational Agent (ECA) ‘coach’ that can provide effective learning support to low-literate learners, develop a prototype virtual learning environment, and evaluate this prototype with low-literate end users. First, we inventory the learning support benefits of ECA coaching. Second, we update existing requirements to better specify functional demands for the coach ECA. Third, we write use cases and develop the prototype. Finally, we evaluate the prototype with low-literate users in a mixed-method within-subjects experiment. Results show that the coach influences the subjective learning experience: Participants report higher positive affect, higher user-system engagement, and increased self-efficacy regarding online banking. These results particularly apply to the domain of challenging information skills exercises. Caveats apply: One of four exercises was significantly more difficult than the other three; and coach support rules were not clearly formalized.

A Systematic Review on Healthcare Artificial Intelligent Conversational Agents for Chronic Conditions.

This paper reviews different types of conversational agents used in health care for chronic conditions, examining their underlying communication technology, evaluation measures, and AI methods. A systematic search was performed in February 2021 on PubMed Medline, EMBASE, PsycINFO, CINAHL, Web of Science, and ACM Digital Library. Studies were included if they focused on consumers, caregivers, or healthcare professionals in the prevention, treatment, or rehabilitation of chronic diseases, involved conversational agents, and tested the system with human users. The search retrieved 1087 articles. Twenty-six studies met the inclusion criteria. Out of 26 conversational agents (CAs), 16 were chatbots, seven were embodied conversational agents (ECA), one was a conversational agent in a robot, and another was a relational agent. One agent was not specified. Based on this review, the overall acceptance of CAs by users for the self-management of their chronic conditions is promising. Users’ feedback shows helpfulness, satisfaction, and ease of use in more than half of included studies. Although many users in the studies appear to feel more comfortable with CAs, there is still a lack of reliable and comparable evidence to determine the efficacy of AI-enabled CAs for chronic health conditions due to the insufficient reporting of technical implementation details.

Aging Adults' Motivation to Use Embodied Conversational Agents in Instrumental Activities of Daily Living: Results of Latent Profile Analysis.

The rapidly increasing share of ageing adults in the population drives the need and interest in assistive technology, as it has the potential to support ageing individuals in living independently and safely. However, technological development rarely reflects how needs, preferences, and interests develop in different ways while ageing. It often follows the strategy of “what is possible” rather than “what is needed” and “what preferred”. As part of personalized assistive technology, embodied conversational agents (ECAs) can offer mechanisms to adapt the technological advances with the stakeholders’ expectations. The present study explored the motivation among ageing adults regarding technology use in multiple domains of activities of daily living. Participants responded to the questionnaire on the perceived importance of instrumental activities of daily living and acceptance of the idea of using ECAs to support them. Latent profile analysis revealed four profiles regarding the motivation to use ECAs (i.e., a low motivation profile, two selective motivation profiles with an emphasis on physical and psychological well-being, and a high motivation profile). Profiles were compared in terms of their acceptance of ECA usage in various life domains. The results increase the knowledge needed in the development of assistive technology adapted to the expectations of ageing adults.

“I felt her company”: A qualitative study on factors affecting closeness and emotional support seeking with an embodied conversational agent

Embodied conversational agents (ECAs) show promise for use as companions in emotional support applications, although real world uptake can be low. Incorporating closeness building techniques to interactions may improve engagement. Research has identified a range of linguistic, behavioural, emotional, and appearance strategies to build rapport and working alliance with ECAs, however substantially less research has investigated closeness; a relationship quality that is more suited to ECAs in supportive companion roles. Exploratory qualitative research may help to inform theoretical models and the design of ECAs for closeness in supportive applications. Qualitative data were collected as part of a mixed-method experimental study. As part of the study, a community sample of 197 adults provided written responses to two open-ended questions assessing barriers and facilitators to closeness and emotional support seeking with an ECA. Data were analysed by two independent raters using conventional content analysis. Parent themes and sub-themes were derived from the data and refined to achieve agreement. Factors affecting closeness with an ECA included its physical characteristics, conversational ability, rapport building behaviours, and program errors. These factors, plus features of ECA specific support and preferences for human support affected willingness to seek emotional support from the ECA. The results contribute to the theoretical framework of embodied agent-patient communication and may inform the design of companion ECAs. The findings suggest that companion ECAs should deliver rapport building behaviours and high quality conversations, be constructed with humanlike facial features, and be error free. Although, these features should be further evaluated experimentally.

Exploiting Evolutionary Algorithms to Model Nonverbal Reactions to Conversational Interruptions in User-Agent Interactions

In social interactions between humans and Embodied Conversational Agents (ECAs) conversational interruptions may occur. ECAs should be prepared to detect, manage and react to such interruptions in order to keep the interaction smooth, natural and believable. In this paper, we examined nonverbal reactions exhibited by an interruptee during conversational interruptions and we propose a novel technique driven by an evolutionary algorithm to build a computational model for ECAs to manage user's interruptions. We propose a taxonomy of conversational interruptions adapted from social psychology, an annotation schema for semi-automatic detection of user's interruptions and a corpus-based observational analysis of human nonverbal reactions to interruptions. Then we present a methodology for building an ECA behavioral model including the design and realization of an interactive study driven by an evolutionary algorithm, where participants interactively built the most appropriate set of multimodal reactive behaviours for an ECA to display interpersonal attitudes (friendly/hostile) through nonverbal reactions to a conversational interruption.

Effectiveness of embodied conversational agents for managing academic stress at an Indian University (ARU) during COVID‐19

AbstractStress has become one of the major reasons for many mental health related issues among students of all age groups, which has resulted in devastating personal losses including suicide. Societal and familial pressure to succeed is high, particularly in developing countries where education is highly valued as a key enabler. As part of stress management during the COVID‐19 pandemic, demand for online intelligent virtual advisors has risen and, consequently, the need for personalised explanation that is culturally sensitive to the user's context is essential to improve the user's understanding of and trust in the recommendations provided by the virtual advisor. This paper presents the mAnaging stRess at University embodied conversational agent (ECA) that has been adapted for Indian university students from an explainable agent that was found to help Western students reduce their stress by providing study tips with explanations based on the student's beliefs and/or goals. We conducted a research study with sixty students which measured the impact of providing three different patterns of tailored explanations (belief‐based, goal‐based, and belief and goal‐based explanation) on the students' intentions to change the recommended behaviours and the relationship built with the ECA. The experimental results indicate that there was stress reduction across all student groups provided with different types of explanations. Further, the students showed trust and a good working alliance with the conversational agent, along with an intention to change behaviour across all types of explanations. However, it was observed that the user context played an important role in behaviour change intention and hence explanations could be tailored further, making them culturally more relevant to Indian students. Practitioner notesWhat is already known about this topic Embodied conversational agents (ECAs) have been mostly developed, applied and shown to be effective in developed countries. Hence, their design and development are mostly guided by the intended user's needs and preferences. In a Western context, ECAs have been found to be beneficial for reducing study stress in university students. There is a pertinent need for use of low cost, effective technology that can aid academic stress reduction in higher educational institutions in developing countries owing to their high youth populations, lack of adequate mental healthcare facilities and associated social stigma. What this paper adds The adaptation and use of ECAs to reduce study stress in higher education students in a developing country is evaluated. The ECA technology is adapted for an Indian context in terms of its physical appearance, colour, speech dialect and dialog content so that it is culturally more aligned to the target population. The ECA engages in an empathic conversation tailored for the Indian students and their COVID‐19 context providing them with explanation‐backed behaviour recommendations that take their beliefs and goals into account. The ECA provides three types of explanation: belief‐only; goal‐only; and both belief and goal. Results of a study carried out in an Indian university with 61 students, randomly assigned to one of the explanation types, to capture their demographics, study stress statistics, behaviour change intentions and trust/working alliance with the conversational agent. The major findings include stress reduction across all explanation groups, development of a positive relationship between the ECA and the students regardless of its explanation pattern, and changes in behaviour intentions across all types of explanations for all recommended behaviours. However, differences in change intentions for certain behaviours indicate further tailoring of explanations is required based on the user context. Implications for practice and/or policy The ECA technology has shown promise in terms of stress reduction amongst Indian students. Higher Education Institutions in developing countries could utilise low‐cost and widely accessible ECAs to overcome lack of access to human‐based support and reluctance to use available services due to stigmatized attitudes to mental health issues. This technology can be further improved and deployed into a larger number of Indian educational institutions leading to a widespread impact on overall student health and wellbeing. Digital technologies to support mental health have become more prominent during the COVID‐19 pandemic, at least in Western countries. The ECA technology evaluated in our study demonstrates its viability and potential value for use in developing countries, with appropriate tailoring.