On-screen Agent Research Articles

Human Elements in Pedagogical Assistants that Aid in English Language Teaching & Learning: An Offline Study

Technological development continuously advances with additional innovations added to the existing account. Yesterday’s apps, today’s ChatGPT, and tomorrow’s extended reality applications are proof of continuous innovation in the Information Technology (IT) sector. During the 2020 pandemic, the world’s education system shifted to online teaching. By then, Artificial Intelligence (AI) had already invested in introducing pedagogical agents (PAs) in teaching through multimedia learning. There are multiple attributable studies about human-comparable on-screen agents that support teaching and improve learning performance with their human characteristics like bodily moments, spoken narration, emotional expression, and gestures. Besides, AI provided pedagogical agents with an exuberant quality of surveilling students during classes. Foreign or second language teachers typically observe emotionality in the classrooms to motivate, encourage, and provide feedback. Thus, the current study aims to investigate and compare the characteristic features of pedagogical agents with those of on-screen teachers. It will explore to what extent the features stimulate the learning process in English as a Foreign Language (EFL) / English as a second language (ESL) learning. Additionally, the present study attempts to cover the classification of emotionality and agents’ contextual appearance in teaching English classes. The study will compare the pedagogical agents’ characteristics with human screen teachers’ characteristics using one-tailed, right-tailed F-test variance. The study proves that pedagogical agents are considered equal partners with on-screen teachers based on the available characteristics. Additionally, the study signals the need for future teachers to acquire technical skills.

14-month-old infants detect a semantic mismatch when occluded objects are mislabeled.

When infants start mastering their first language, they may start to notice when words are used incorrectly. Around 14-months of age, infants detect incorrect labeling when they are presented with an object which is labeled while still visible. However, things that are referred to are often out of sight when we communicate about them. The present study examined infants' detection of semantic mismatch when the object was occluded at the time of labeling. Specifically, we investigated whether mislabeling that referred to an occluded object could elicit a semantic mismatch. We showed 14-month-old Danish-speaking infants events where an onscreen agent showed an object and then hid it in a box. This was followed by another agent's hand pointing at the box, and a concurrent auditory category label played, which either matched or did not match the hidden object. Our results indicate that there is an effect of semantic mismatch with a larger negativity in incongruent trials. Thus, infants detected a mismatch, as indicated by a larger n400, when occluded objects were mislabeled. This finding suggests that infants can sustain an object representation in memory and compare it to a semantic representation of an auditory category label.

Recognizing and Relating to the Race/Ethnicity and Gender of Animated Pedagogical Agents

This study examined how well people can recognize and relate to animated pedagogical agents of varying ethnicities/races and genders. For both Study 1 (realistic-style agents) and Study 2 (cartoon-style agents), participants viewed brief video clips of virtual agents of varying racial/ethnic categories and gender types and then identified their race/ethnicity and gender and rated how human-like and likable the agent appeared. Participants were highly accurate in identifying Black and White agents but were less accurate for Asian, Indian, and Hispanic agents. Participants were accurate in recognizing gender differences. Participants rated all types of agents as moderately human-like, except for White agents. Likability ratings were lowest for White and male agents. The same pattern of results was obtained across two independent studies with different participants and different onscreen agents, which indicates that the results are not solely due to one specific set of agents. Consistent with the Media Equation Hypothesis and the Alliance Hypothesis, this work shows that people are sensitive to the race/ethnicity and gender of onscreen agents and relate to them differently. These findings have implications for how to design animated pedagogical agents for improved multimedia learning environments in the future and serve as a crucial first step in highlighting the possibility and feasibility of incorporating diverse onscreen virtual agents into educational computer software.

The Power of Voice to Convey Emotion in Multimedia Instructional Messages

This study examines an aspect of the role of emotion in multimedia learning, i.e., whether participants can recognize the instructor’s positive or negative emotion based on hearing short clips involving only the instructor’s voice just as well as also seeing an embodied onscreen agent. Participants viewed 16 short video clips from a statistics lecture in which an animated instructor, conveying a happy, content, frustrated, or bored emotion, stands next to a slide as she lectures (agent present) or uses only her voice (agent absent). For each clip, participants rated the instructor on five-point scales for how happy, content, frustrated, and bored the instructor seemed. First, for happy, content, and bored instructors, participants were just as accurate in rating emotional tone based on voice only as with voice plus onscreen agent. This supports the voice hypothesis, which posits that voice is a powerful source of social-emotional information. Second, participants rated happy and content instructors higher on happy and content scales and rated frustrated and bored instructors higher on frustrated and bored scales. This supports the positivity hypothesis, which posits that people are particularly sensitive to the positive or negative tone of multimedia instructional messages.

Learning with human and virtual instructors who display happy or bored emotions in video lectures

In this study, we investigate whether the affective state (happy or bored) of a human or virtual instructor in an instructional video on statistics yields different learning processes and outcomes. The positivity principle states that the emotional state of the instructor is recognized by students (hypothesis 1) and affects their emotional state (hypothesis 2), motivational state (hypothesis 3), and learning outcomes (hypothesis 4). The equivalence principle states that people respond to the emotional tone of computerized onscreen agents in the same way as for human instructors (hypothesis 5). In a 2 × 2 between subjects design, participants were assigned to watch a statistics video in one of four groups: Happy Human (HH), Happy Agent (HA), Bored Human (BH), and Bored Agent (BA). Then, they rated the emotional state of the instructor, rated their own emotional state, rated their motivational state, and took a posttest. The findings support predictions 1, 2, and 3 of the positivity principle but not prediction 4, perhaps because an immediate test was not sensitive enough. The equivalence principle (hypothesis 5) was partially supported as the human and agent groups showed similar rating patterns although the effects were stronger for human instructors for recognizing the instructor's emotion. As education transitions online, it is important to note that students are happier and more motivated when they learn from happy instructors than from bored instructors.

Younger and older users׳ recognition of virtual agent facial expressions

As technology advances, robots and virtual agents will be introduced into the home and healthcare settings to assist individuals, both young and old, with everyday living tasks. Understanding how users recognize an agent׳s social cues is therefore imperative, especially in social interactions. Facial expression, in particular, is one of the most common non-verbal cues used to display and communicate emotion in on-screen agents (Cassell et al., 2000). Age is important to consider because age-related differences in emotion recognition of human facial expression have been supported (Ruffman et al., 2008), with older adults showing a deficit for recognition of negative facial expressions. Previous work has shown that younger adults can effectively recognize facial emotions displayed by agents (Bartneck and Reichenbach, 2005; Courgeon et al., 2009, 2011; Breazeal, 2003); however, little research has compared in-depth younger and older adults’ ability to label a virtual agent׳s facial emotions, an import consideration because social agents will be required to interact with users of varying ages. If such age-related differences exist for recognition of virtual agent facial expressions, we aim to understand if those age-related differences are influenced by the intensity of the emotion, dynamic formation of emotion (i.e., a neutral expression developing into an expression of emotion through motion), or the type of virtual character differing by human-likeness. Study 1 investigated the relationship between age-related differences, the implication of dynamic formation of emotion, and the role of emotion intensity in emotion recognition of the facial expressions of a virtual agent (iCat). Study 2 examined age-related differences in recognition expressed by three types of virtual characters differing by human-likeness (non-humanoid iCat, synthetic human, and human). Study 2 also investigated the role of configural and featural processing as a possible explanation for age-related differences in emotion recognition. First, our findings show age-related differences in the recognition of emotions expressed by a virtual agent, with older adults showing lower recognition for the emotions of anger, disgust, fear, happiness, sadness, and neutral. These age-related difference might be explained by older adults having difficulty discriminating similarity in configural arrangement of facial features for certain emotions; for example, older adults often mislabeled the similar emotions of fear as surprise. Second, our results did not provide evidence for the dynamic formation improving emotion recognition; but, in general, the intensity of the emotion improved recognition. Lastly, we learned that emotion recognition, for older and younger adults, differed by character type, from best to worst: human, synthetic human, and then iCat. Our findings provide guidance for design, as well as the development of a framework of age-related differences in emotion recognition.

An embodiment effect in computer-based learning with animated pedagogical agents.

How do social cues such as gesturing, facial expression, eye gaze, and human-like movement affect multimedia learning with onscreen agents? To help address this question, students were asked to twice view a 4-min narrated presentation on how solar cells work in which the screen showed an animated pedagogical agent standing to the left of 11 successive slides. Across three experiments, learners performed better on a transfer test when a human-voiced agent displayed human-like gestures, facial expression, eye gaze, and body movement than when the agent did not, yielding an embodiment effect. In Experiment 2 the embodiment effect was found when the agent spoke in a human voice but not in a machine voice. In Experiment 3, the embodiment effect was found both when students were told the onscreen agent was consistent with their choice of agent characteristics and when inconsistent. Students who viewed a highly embodied agent also rated the social attributes of the agent more positively than did students who viewed a nongesturing agent. The results are explained by social agency theory, in which social cues in a multimedia message prime a feeling of social partnership in the learner, which leads to deeper cognitive processing during learning, and results in a more meaningful learning outcome as reflected in transfer test performance.

Designing motivational agents: The role of praise, social comparison, and embodiment in computer feedback

The present study draws on theories of attribution, social comparison, and social facilitation to investigate how computers might use principles of motivation and persuasion to provide user feedback. In an online experiment, 192 participants performed a speed-reading task. The independent variables included whether or not the verbal feedback from the computer involved praise, whether the objective feedback showed that the participants were performing better or worse from their peers, and whether or not the feedback was presented by an on-screen agent. The main dependent variables included a subjective measure of participants’ intrinsic motivation and an objective measure of their task persistence. Results showed that providing participants with praise or comparative information on others’ performance improved intrinsic motivation. When praised, participants whose performances were comparatively low persisted in the task longer than those whose performances were comparatively high did. Additionally, the mere presence of an embodied agent on the screen increased participants’ motivation. Together, these results indicate that praise and social comparison can serve as effective forms of motivational feedback and that humanlike embodiment further improves user motivation.

Towards a science of motivated learning in technology-supported environments

This commentary reviews seven papers that study motivation with new media, contained in this special issue of Educational Technology Research & Development edited by Ruth Small. For each paper, this commentary summarizes exemplary contributions, offers an assessment of what is exciting, and suggests directions for future research. Some exciting contributions include using new media in an attempt to promote motivation such as game-making, collaborative wiki construction, and interaction with onscreen agents. Some important challenges for future research on motivation with new media are to develop testable theories and to collect evidence based on rigorous methodologies including evidence concerning the relation between motivation and learning outcomes

Differences in effect of robot and screen agent recommendations on human decision-making

This paper compares the effect of a robot's and on-screen agent's recommendations on human decision-making using a quantitative evaluation method. We are interested in whether a robot's physical body produces some differences in the effect or not. Previous research investigated the advantage of a physical body; however, the advantage was not clarified quantitatively and there was not enough evidence to give the results credibility. Our method based on quantitative evaluation clarifies the effect of a robot's and on-screen agent's behavior on user decision-making. Comparing a robot's behavior with an on-screen agent's, we show that the degree of the effect firmly depends on the interaction environment and that geometrical consistency between the interaction environment and embodied social agents (ESAs), which include robots and on-screen agents, is important in the recommendation situation.

Personalized Messages That Promote Science Learning in Virtual Environments.

College students learned how to design the roots, stem, and leaves of plants to survive in five different virtual reality environments through an agent-based multimedia educational game. For each student, the agent used personalized speech (e.g., including I and you) or nonpersonalized speech (e.g., 3rd-person monologue), and the game was presented via desktop computer (low immersion) or head-mounted display (high immersion). Across both levels of immersion, students who received personalized agent messages performed better on retention and problem-solving transfer tests. Although students reported higher levels of physical presence with high rather than low immersion, higher immersion did not lead to better performance on tests of retention or transfer. Students learn more deeply from computer-based simulation games when an on-screen agent speaks to them in a personalized style rather than a nonpersonalized style.