Back-channel Feedback Research Articles

Evaluation of back-channel feedback timing in the speech-driven embodied entrainment character system

Evaluation of back-channel feedback timing in the speech-driven embodied entrainment character system

Role of Speaker Cues in Attention Inference

Current state-of-the-art approaches to emotion recognition primarily focus on modeling the nonverbal expressions of the sole individual without reference to contextual elements like the co-presence of the partner. In this paper, we demonstrate that the accurate inference of listeners' social-emotional state of attention depends on also accounting for the nonverbal behaviors of their storytelling partner, namely their speaker cues. To gain a deeper understanding of the role of speaker cues in attention inference, we conduct investigations into real world interactions of children storytelling with their peers. Through in-depth analysis of human-human interaction data, we first identify nonverbal speaker cues (i.e., backchannel-inviting cues) and listener responses (i.e., backchannel feedback) to later demonstrate how speaker cues can modify the interpretation of attention-related backchannels as well as serve as a means to regulate the responsiveness of listeners. We then discuss the design implications of our findings toward our primary goal of developing attention recognition models for storytelling robots. Social robots can use speaker cues to form more accurate inferences about the attentive state of their human partner.

ASEに基づく相槌によるロボットとの対話体験の向上

In this paper, we describe an investigation into users' experiences of a simple talking robot with back-channel feedbacks that is designed based on an artificial subtle expression (ASE). In the experiments with participants, they are divided into six conditions based on an expression factor (three levels; human-like speech, blinking light, and beeping sound) and a timing decision method factor (two levels; a linguistic method and an acoustic method) for investigating participants' impressions on the dialogue experience. We developed an electric pedestal to show the blinking expression, on which a simple cubic robot was fixed. Participants engaged in a task of explaining a cooking procedure with a spoken dialogue system coupled with the robot on the pedestal. The robots responded to them by making the back-channel feedbacks in accordance with the expression factor. The results of questionnaire analyses suggested that the ASE-based expressions of back-channel feedback provide positive experiences for users.

Backchannel Prediction for Mandarin Human-Computer Interaction

In recent years, researchers have tried to create unhindered human-computer interaction by giving virtual agents human-like conversational skills. Predicting backchannel feedback for agent listeners has become a novel research hot-spot. The main goal of this paper is to identify appropriate features and methods for backchannel prediction in Mandarin conversations. Firstly, multimodal Mandarin conversations are recorded for the analysis of backchannel behaviors. In order to eliminate individual difference in the original face-to-face conversations, more backchannels from different listeners are gathered together. These data confirm that backchannels occurring in the speakers' pauses form a vast majority in Mandarin conversations. Both prosodic and visual features are used in backchannel prediction. Four types of models based on the speakers' pauses are built by using support vector machine classifiers. An evaluation of the pause-based prediction model has shown relatively high accuracy in consideration of the optional nature of backchannel feedback. Finally, the results of the subjective evaluation validate that the conversations performed between humans and virtual listeners using backchannels predicted by the proposed models is more unhindered compared to other backchannel prediction methods.

A cross-cultural examination of the backchannel behavior of Japanese and Americans: Considerations for Japanese EFL learners

Listenership (consisting of backchannel feedback) and its e�耀 ect on intercultural communication were investigated in 30 dyadic conversations in English between Japanese and American participants. The �耀 ndings of this study demonstrate several di�耀 erences in how members of each culture used back- channels in terms of frequency, variability, placement, and function. This study also found evidence supporting the hypothesis that backchannel conventions that are not shared between cultures contribute to negative perceptions across cultures. Thus, the �耀 ndings of this study support the conclusion that listenership warrants more attention in EFL classes in Japan. Further, toward creating a peda- gogical framework, this study also provides EFL professionals with a comprehen- sive account of the listenership of Japanese EFL speakers. Finally, this study also o�耀 ers potential insights into the linguistic variation of native English speakers. That is, the negative perceptions that the American native English speakers re- ported of their Japanese EFL speaking interlocutors' listener responses in this study were not as pronounced as those reported by the British native English speakers in a previous study conducted by the same researcher.

Multimodal plan representation for adaptable BML scheduling

Natural human interaction is characterized by interpersonal coordination: interlocutors converge in their speech rates, smoothly switch speaking turns with virtually no delay, provide their interlocutors with verbal and nonverbal backchannel feedback, wait for and react to such feedback, execute physical tasks in tight synchrony, etc. If virtual humans are to achieve such interpersonal coordination they require very flexible behavior plans that are adjustable on-the-fly. In this paper we discuss how such plans are represented, maintained and constructed in our BML realizer Elckerlyc. We argue that behavior scheduling for Virtual Humans can be viewed as a constraint satisfaction problem, and show how Elckerlyc uses this view in its flexible behavior plan representation that allows one to make on-the-fly adjustments to behaviors while keeping the specified constraints between them intact.

Cultural transmission in the laboratory: agent interaction improves the intergenerational transfer of information

Cumulative cultural evolution requires that information is faithfully transmitted from generation to generation. The present study examines the role of agent interaction as a social learning mechanism through which information is transmitted across multiple generations. The performance of two types of linear transmission chains was compared: noninteractive (agents in adjacent chain positions were not permitted to interact) and interactive (adjacent agents freely interacted with one another). In both conditions, information (details of a narrative text) was lost as it was passed along the transmission chain. However, interactive transmission chains promoted more accurate recall of information than noninteractive chains. A content analysis revealed that most listeners actively participated in the information transfer process by seeking clarification and providing backchannel feedback to the narrator. Furthermore, the extent to which listeners engaged with the narrator was associated with narrator recall accuracy. Our results indicate that bidirectional agent interaction is an important consideration for studies of cultural transmission and cumulative cultural evolution.

American and Arab Perceptions of an Arabic Turn-Taking Cue

Languages differ in the way that speakers coordinate their interaction moment by moment, and this can cause intercultural misunderstandings. The authors explore this in the domain of listening behavior. One way that listeners show interest and attention is by producing back-channel feedback (short utterances such as okay and hmm) at appropriate times, and these times are determined, in part, by the interlocutor, who signals when such feedback is welcome with various cues. In Arabic these cues include a prosodic feature in the form of a steep continuous drop in pitch. This article shows that English speakers can misinterpret this, perceiving it as an expression of negative affect, and that this tendency can be substantially alleviated by training.

Mapping and Manipulating Facial Expression

Nonverbal visual cues accompany speech to supplement the meaning of spoken words, signify emotional state, indicate position in discourse, and provide back-channel feedback. This visual information includes head movements, facial expressions and body gestures. In this article we describe techniques for manipulating both verbal and nonverbal facial gestures in video sequences of people engaged in conversation. We are developing a system for use in psychological experiments, where the effects of manipulating individual components of nonverbal visual behavior during live face-to-face conversation can be studied. In particular, the techniques we describe operate in real-time at video frame-rate and the manipulation can be applied so both participants in a conversation are kept blind to the experimental conditions.

A probabilistic multimodal approach for predicting listener backchannels

During face-to-face interactions, listeners use backchannel feedback such as head nods as a signal to the speaker that the communication is working and that they should continue speaking. Predicting these backchannel opportunities is an important milestone for building engaging and natural virtual humans. In this paper we show how sequential probabilistic models (e.g., Hidden Markov Model or Conditional Random Fields) can automatically learn from a database of human-to-human interactions to predict listener backchannels using the speaker multimodal output features (e.g., prosody, spoken words and eye gaze). The main challenges addressed in this paper are automatic selection of the relevant features and optimal feature representation for probabilistic models. For prediction of visual backchannel cues (i.e., head nods), our prediction model shows a statistically significant improvement over a previously published approach based on hand-crafted rules.

Learning to show you're listening

Good listeners generally produce back-channel feedback, that is, short utterances such as uh-huh which signal active listening. As the rules governing back-channeling vary from language to language, second-language learners may need help acquiring this skill. This paper is an initial exploration of how to provide this. It presents a training sequence which enables learners to acquire a basic Arabic back-channel skill, namely, that of producing feedback immediately after the speaker produces a syllable or two with a sharply falling pitch. This training sequence includes an explanation, audio examples, the use of visual signals to highlight occurrences of this pitch downslope, auditory and visual feedback on learners' attempts to produce the cue themselves, and feedback on the learners' performance as they play the role of an attentive listener in response to one side of a pre-recorded dialog. Experiments show that this enables learners to better approximate proper Arabic back-channeling behavior.

AutoTutor: Incorporating Back-Channel Feedback and Other Human-Like Conversational Behaviors into an Intelligent Tutoring System

This paper describes our recent attempts to incorporate human-like conversational behaviors into the dialog moves delivered by an animated pedagogical agent that simulates human tutors. We first present a brief overview of the modules comprising AutoTutor, an intelligent tutoring system. The second section describes a set of conversational behaviors that are being incorporated into AutoTutor. The behaviors of interest involve variations in intonation, head movements, arm and hand movements, facial expressions, eye blinking, gaze direction, and back-channel feedback. The final section presents a recent empirical study concerned with back-channel feedback events during human-to-human tutoring sessions. The back-channel feedback events emitted by tutors are mostly positive (63%), mostly verbal (77%), and immediately follow speech-act boundaries or noun-phrase boundaries (83%). Tutors also deliver back-channelevents at a very high rate when students are emitting dialog, about 13 events per minute. Conversely, 88% of students' back-channel feedback events are head nods, and they occur at unbounded locations (63%).

Prosodic features which cue back-channel responses in English and Japanese

Back-channel feedback, responses such as uh-uh from a listener, is a pervasive feature of conversation. It has long been thought that the production of back-channel feedback depends to a large extent on the actions of the other conversation partner, not just on the volition of the one who produces them. In particular, prosodic cues from the speaker have long been thought to play a role, but have so far eluded identification. We have earlier suggested that an important prosodic cue involved, in both English and Japanese, is a region of low pitch late in an utterance (Ward, 1996). This paper discusses issues in the definition of back-channel feedback, presents evidence for our claim, surveys other factors which elicit or inhibit back-channel responses, and mentions a few related phenomena and theoretical issues.

Fully Embodied Conversational Avatars: Making Communicative Behaviors Autonomous

Although avatars may resemble communicative interface agents, they have for the most part not profited from recent research into autonomous embodied conversational systems. In particular, even though avatars function within conversational environments (for example, chat or games), and even though they often resemble humans (with a head, hands, and a body) they are incapable of representing the kinds of knowledge that humans have about how to use the body during communication. Humans, however, do make extensive use of the visual channel for interaction management where many subtle and even involuntary cues are read from stance, gaze, and gesture. We argue that the modeling and animation of such fundamental behavior is crucial for the credibility and effectiveness of the virtual interaction in chat. By treating the avatar as a communicative agent, we propose a method to automate the animation of important communicative behavior, deriving from work in conversation and discourse theory. BodyChat is a system that allows users to communicate via text while their avatars automatically animate attention, salutations, turn taking, back-channel feedback, and facial expression. An evaluation shows that users found an avatar with autonomous conversational behaviors to be more natural than avatars whose behaviors they controlled, and to increase the perceived expressiveness of the conversation. Interestingly, users also felt that avatars with autonomous communicative behaviors provided a greater sense of user control.

RESPONSIVENESS IN DIALOG AND PRIORITIES FOR LANGUAGE RESEARCH

In human language there seems to be a direct link between perception and action for back-channel feedback, that is, the grunts and other acknowledgments produced by a listener in a conversation. In particular, in Japanese, detection of a low pitch region in the speaker's utterance often cues the listener to produce back-channel feedback. This fact has been exploited in a system that can interact with unsuspecting speakers and appear to converse, in real time. This finding suggests that Brooks' 1991a, 1991b methodology for the study and synthesis of physical behavior is relevant also for social behavior. If so, then real-time responsiveness, rather than better phoneme recognition,is a top priorityfor the construction of spoken language systems and for the scientific study of language.

Sex, Communication Structures and Role Specification

Abstract This study investigates variations in male and female approaches to leadership in small discussion groups. Three aspects of leadership are investigated: (1) the nature of participation in the turn-taking system (cf. Duncan, 1972), (2) leadership role specialization (cf. Burke, 1967) and (3) the differential bases of attribution of leadership by sex. Data are obtained from 10 four-member groups, each composed of two males and two females. In half of the groups a male was appointed as coordinator, while a female was appointed as coordinator in the other half. Results indicate no differences between male and female coordinators in the number of turns taken in the discussions, but near significant differences in providing back-channel feedback to current speakers and in turn-collaborations, with male coordinators engaging in more of these activities than non-coordinators, and female coordinators engaging in fewer of these activities than non-coordinators. In terms of leadership role specialization, b...