Multiparty Conversations Research Articles

Lexical speaker identification in TV shows

It is possible to use lexical information extracted from speech transcripts for speaker identification (SID), either on its own or to improve the performance of standard cepstral-based SID systems upon fusion. This was established before typically using isolated speech from single speakers (NIST SRE corpora, parliamentary speeches). On the contrary, this work applies lexical approaches for SID on a different type of data. It uses the REPERE corpus consisting of unsegmented multiparty conversations, mostly debates, discussions and Q&A sessions from TV shows. It is hypothesized that people give out clues to their identity when speaking in such settings which this work aims to exploit. The impact on SID performance of the diarization front-end required to pre-process the unsegmented data is also measured. Four lexical SID approaches are studied in this work, including TFIDF, BM25 and LDA-based topic modeling. Results are analysed in terms of TV shows and speaker roles. Lexical approaches achieve low error rates for certain speaker roles such as anchors and journalists, sometimes lower than a standard cepstral-based Gaussian Supervector - Support Vector Machine (GSV-SVM) system. Also, in certain cases, the lexical system shows modest improvement over the cepstral-based system performance using score-level sum fusion. To highlight the potential of using lexical information not just to improve upon cepstral-based SID systems but as an independent approach in its own right, initial studies on crossmedia SID is briefly reported. Instead of using speech data as all cepstral systems require, this approach uses Wikipedia texts to train lexical speaker models which are then tested on speech transcripts to identify speakers.

Deficient gaze pattern during virtual multiparty conversation in patients with schizophrenia

Virtual reality has been used to measure abnormal social characteristics, particularly in one-to-one situations. In real life, however, conversations with multiple companions are common and more complicated than two-party conversations. In this study, we explored the features of social behaviors in patients with schizophrenia during virtual multiparty conversations. Twenty-three patients with schizophrenia and 22 healthy controls performed the virtual three-party conversation task, which included leading and aiding avatars, positive- and negative-emotion-laden situations, and listening and speaking phases. Patients showed a significant negative correlation in the listening phase between the amount of gaze on the between-avatar space and reasoning ability, and demonstrated increased gaze on the between-avatar space in the speaking phase that was uncorrelated with attentional ability. These results suggest that patients with schizophrenia have active avoidance of eye contact during three-party conversations. Virtual reality may provide a useful way to measure abnormal social characteristics during multiparty conversations in schizophrenia.

Sociable Spotlight: The Impact of Interactive Artifacts to Ground the Social Interactions

Understanding why people treat simple geometric animations like real agent which has intention to interact with people even if its geometry is artificial thing will aid the `agency' problems of human-agent interaction. This paper explores effects of treating simple geometric animations as a real participant to facilitate multi-party conversation in social interaction. Observational study was conducted with groups of two or three persons using simple circle (sociable spotlight) which moves based on dynamic information in the current multi-party conversation, with the goal of discovering how participants are utilizing the behaviors of sociable spotlight as an other party for organizing the conversational sequences in talk-in-interaction. In addition, we motivated to explore how the sociable spotlight is embedded within the organization of conversation and how the user's behaviors are changed according to the sociable spotlight's behaviors by investigate through conversation analysis of a video-recording. Finally, we conclude how the agency of artificial things constructed in multi-party conversation from minimal designing point of view.

Adjusting conceptual pacts in three-party conversation.

During conversation, partners develop representations of jointly known information--the common ground--and use this knowledge to guide subsequent linguistic exchanges. Extensive research on 2-party conversation has offered key insights into this process, in particular, its partner-specificity: Common ground that is shared with 1 partner is not always assumed to be shared with other partners. Conversation often involves multiple pairs of individuals who differ in common ground. Yet, little is known about common ground processes in multi-party conversation. Here, we take a 1st step toward understanding this problem by examining situations in which simple dyadic representations of common ground might cause difficulty--situations in which dialogue partners develop shared labels (entrained terms), and then a 3rd (naïve) party joins the conversation. Experiment 1 examined unscripted, task-based conversation in which 2 partners entrained on terms. At test, speakers referenced game-pieces in a dialogue with their partner, or in a 3-party conversation including a new, naïve listener. Speakers were sensitive to the 3rd party, using longer, disfluent expressions when additionally addressing the new partner. By contrast, analysis of listener eye-fixations did not suggest sensitivity. Experiment 2 provided a stronger test of sensitivity and revealed that listeners do cancel expectations for terms that had been entrained before when a 3rd, naïve party joins the conversation. These findings shed light on the mechanisms underlying common ground, showing that rather than a unitary construct, common ground is flexibly adapted to the needs of a naïve 3rd party.

A Multiparty Conversation System with an Addressee Identification Mechanism based on Nonverbal Information

A Multiparty Conversation System with an Addressee Identification Mechanism based on Nonverbal Information

Modeling topic control to detect influence in conversations using nonparametric topic models

Identifying influential speakers in multi-party conversations has been the focus of research in communication, sociology, and psychology for decades. It has been long acknowledged qualitatively that controlling the of a conversation is a sign of influence. To capture who introduces new topics in conversations, we introduce SITS--Speaker Identity for Topic Segmentation--a nonparametric hierarchical Bayesian model that is capable of discovering (1) the topics used in a set of conversations, (2) how these topics are shared across conversations, (3) when these topics change during conversations, and (4) a speaker-specific measure of topic control. We validate the model via evaluations using multiple datasets, including work meetings, online discussions, and political debates. Experimental results confirm the effectiveness of SITS in both intrinsic and extrinsic evaluations.

Gaze behavior of pre-adolescent children afflicted with Asperger Syndrome

Asperger syndrome (AS) is a form of high-functioning autism characterized by qualitative impairment in social interaction. People afflicted with AS typically have abnormal nonverbal behaviors which are often manifested by avoiding eye contact. Gaze constitutes an important interactional resource, and an AS person's tendency to avoid eye contact may affect the fluidity of conversations and cause misunderstandings. For this reason, it is important to know the precise ways in which this avoidance is done, and in what ways it affects the interaction. The objective of this article is to describe the gaze behavior of preadolescent AS children in institutional multiparty conversations. Methodologically, the study is based on conversation analysis and a multimodal study of interaction. The findings show that three main patterns are used for avoiding eye contact: (1) fixing one's gaze straight ahead; (2) letting one's gaze wander around; and (3) looking at one's own hands when speaking. The informants of this study do not look at the interlocutors at all in the beginning or the middle of their turn. However, sometimes they turn to look at the interlocutors at the end of their turn. This proves that these children are able to use gaze as a source offeedback. When listening, looking at the speaker also seems to be easier for them than looking at the listeners when speaking

Towards a Semantic-Based Approach for Affect and Metaphor Detection

Affect detection from open-ended virtual improvisational contexts is a challenging task. To achieve this research goal, the authors developed an intelligent agent which was able to engage in virtual improvisation and perform sentence-level affect detection from user inputs. This affect detection development was efficient for the improvisational inputs with strong emotional indicators. However, it can also be fooled by the diversity of emotional expressions such as expressions with weak or no affect indicators or metaphorical affective inputs. Moreover, since the improvisation often involves multi-party conversations with several threads of discussions happening simultaneously, the previous development was unable to identify the different discussion contexts and the most intended audiences to inform affect detection. Therefore, in this paper, the authors employ latent semantic analysis to find the underlying semantic structures of the emotional expressions and identify topic themes and target audiences especially for those inputs without strong affect indicators to improve affect detection performance. They also discuss how such semantic interpretation of dialog contexts is used to identify metaphorical phenomena. Initial exploration on affect detection from gestures is also discussed to interpret users’ experience of using the system and provide an extra channel to detect affect embedded in the virtual improvisation. Their work contributes to the journal themes on affect sensing from text, semantic-based dialogue processing and emotional gesture recognition.

Wordless Sounds: Robust Speaker Diarization Using Privacy-Preserving Audio Representations

This paper investigates robust privacy-sensitive audio features for speaker diarization in multiparty conversations: i.e., a set of audio features having low linguistic information for speaker diarization in a single and multiple distant microphone scenarios. We systematically investigate Linear Prediction (LP) residual. Issues such as prediction order and choice of representation of LP residual are studied. Additionally, we explore the combination of LP residual with subband information from 2.5 kHz to 3.5 kHz and spectral slope. Next, we propose a supervised framework using deep neural architecture for deriving privacy-sensitive audio features. We benchmark these approaches against the traditional Mel Frequency Cepstral Coefficients (MFCC) features for speaker diarization in both the microphone scenarios. Experiments on the RT07 evaluation dataset show that the proposed approaches yield diarization performance close to the MFCC features on the single distant microphone dataset. To objectively evaluate the notion of privacy in terms of linguistic information, we perform human and automatic speech recognition tests, showing that the proposed approaches to privacy-sensitive audio features yield much lower recognition accuracies compared to MFCC features.

Identifying the Addressee using Head Orientation and Speech Information in Multiparty Human-Agent Conversations

Identifying the Addressee using Head Orientation and Speech Information in Multiparty Human-Agent Conversations

Resources for turn competition in overlapping talk

Overlapping talk occurs frequently in multi-party conversations, and is a domain in which speakers may pursue various communicative goals. The current study focuses on turn competition. Specifically, we seek to identify the phonetic differences that discriminate turn-competitive from non-competitive overlaps. Conversation analysis techniques were used to identify competitive and non-competitive overlaps in a corpus of multi-party recordings. We then generated a set of potentially predictive features relating to prosody (F0, intensity, speech rate, pausing) and overlap placement (overlap duration, point of overlap onset, recycling etc.). Decision tree classifiers were trained on the features and tested on a classification task, in order to determine which features and feature combinations best differentiate competitive overlaps from non-competitive overlaps. It was found that overlap placement features played a greater role than prosodic features in indicating turn competition. Among the prosodic features tested, F0 and intensity were the most effective predictors of turn competition. Also, our decision tree models suggest that turn competitive and non-competitive overlaps can be initiated by a new speaker at many different points in the current speaker’s turn. These findings have implications for the design of dialogue systems, and suggest novel hypotheses about how speakers deploy phonetic resources in everyday talk.

MM-Space: Recreating Multiparty Conversation Space by Using Dynamic Displays

MM-Space: Recreating Multiparty Conversation Space by Using Dynamic Displays

Production, circulation and deconstruction of gender norms in LGBTQ speech practices

This paper investigates interdiscursivity in talk about gender norms in the Lesbian, Gay, Bisexual, Trans, Queer (LGBTQ) community. Drawing from field work conducted among gay and lesbian parents, transgender people, and the Drag King community, I show how accounts like les hommes sont comme ça (‘Men are like that’) or les femmes sont comme ça (‘Women act that way’) emerging in discourse about pregnancy reveal gender-based norms and their relationship to social conditions. Adopting a linguistic anthropology perspective, while paying particular attention to agency, categorization, circulation, and recontextualization processes, I show how gender norms are constructed and deconstructed through interactional, semantic, and morphosyntactic patterns found in a variety of multisemiotic mediums (e.g. texts, multiparty conversations and forum exchanges, images) and in a diversity of contexts. In addition to its contribution to the study of gender and language, this article takes new steps in the ongoing development of membership categorization analysis and queer linguistics.

Conversational learning integration in technology enhanced classrooms

Today’s college students have grown up with technology. These digital natives typically gravitate toward group activities in technology embedded social contexts. However, despite this multidimensional evolution, little has changed in conventional classrooms where they build their education experience. We investigate learning models in a classroom environment which still remains the main driver of education today. We describe a conversational learning model based on group activities which involve multi-party conversations. We implement this model in a technology-enhanced studio-classroom to “visualize” conversations which otherwise would remain abstract to learners. Teachers are empowered with instructional patterns to guide their changing role in this novel classroom environment. Based on standard assessment indicators, we conduct an experimental analysis which results show interesting tradeoffs of learning performance that favor the proposed conversational learning approach compared to those obtained from conventional instruction.

Automatic Role Recognition in Multiparty Conversations: An Approach Based on Turn Organization, Prosody, and Conditional Random Fields

Roles are a key aspect of social interactions, as they contribute to the overall predictability of social behavior (a necessary requirement to deal effectively with the people around us), and they result in stable, possibly machine-detectable behavioral patterns (a key condition for the application of machine intelligence technologies). This paper proposes an approach for the automatic recognition of roles in conversational broadcast data, in particular, news and talk shows. The approach makes use of behavioral evidence extracted from speaker turns and applies conditional random fields to infer the roles played by different individuals. The experiments are performed over a large amount of broadcast material (around 50 h), and the results show an accuracy higher than 85%.

Spotting laughter in natural multiparty conversations

It is essential for the advancement of human-centered multimodal interfaces to be able to infer the current user's state or communication state. In order to enable a system to do that, the recognition and interpretation of multimodal social signals (i.e., paralinguistic and nonverbal behavior) in real-time applications is required. Since we believe that laughs are one of the most important and widely understood social nonverbal signals indicating affect and discourse quality, we focus in this work on the detection of laughter in natural multiparty discourses. The conversations are recorded in a natural environment without any specific constraint on the discourses using unobtrusive recording devices. This setup ensures natural and unbiased behavior, which is one of the main foci of this work. To compare results of methods, namely Gaussian Mixture Model (GMM) supervectors as input to a Support Vector Machine (SVM), so-called Echo State Networks (ESN), and a Hidden Markov Model (HMM) approach, are utilized in online and offline detection experiments. The SVM approach proves very accurate in the offline classification task, but is outperformed by the ESN and HMM approach in the online detection (F 1 scores: GMM SVM 0.45, ESN 0.63, HMM 0.72). Further, we were able to utilize the proposed HMM approach in a cross-corpus experiment without any retraining with respectable generalization capability (F 1 score: 0.49). The results and possible reasons for these outcomes are shown and discussed in the article. The proposed methods may be directly utilized in practical tasks such as the labeling or the online detection of laughter in conversational data and affect-aware applications.

Across languages and cultures: Brokering problems of understanding in conversational repair

AbstractThis article examines the interactional construction of language competence in bilingual immigrant communities. The focus is on how participants in social interaction resolve problems of understanding that are demonstrably rooted in their divergent linguistic and cultural expertise. Using the methodology of conversation analysis to examine mundane video-recorded conversations in Russian-American immigrant families, I describe a previously unanalyzed communicative practice for solving understanding problems: by one participant enacting the role of a language broker in a repair sequence. The article thus contributes to the existing research on the interactional construction of language competence, on the one hand, and on the organization of repair and its relationship to social epistemics, on the other. (Language brokering, repair, conversation analysis, social epistemics, multiparty conversation, Russian, immigrant families, intercultural communication)*

Privacy-Sensitive Audio Features for Speech/Nonspeech Detection

The goal of this paper is to investigate features for speech/nonspeech detection (SND) having low linguistic information from the speech signal. Towards this, we present a comprehensive study of privacy-sensitive features for SND in multiparty conversations. Our study investigates three different approaches to privacy-sensitive features. These approaches are based on: 1) simple, instantaneous feature extraction methods; 2) excitation source information based methods; and 3) feature obfuscation methods such as local (within 130 ms) temporal averaging and randomization applied on excitation source information. To evaluate these approaches for SND, we use multiparty conversational meeting data of nearly 450 hours. On this dataset, we evaluate these features and benchmark them against standard spectral shape based features such as Mel frequency perceptual linear prediction (MFPLP). Fusion strategies combining excitation source with simple features show that comparable performance can be obtained in both close-talking and far-field microphone scenarios. As one way to objectively evaluate the notion of privacy, we conduct phoneme recognition studies on TIMIT. While excitation source features yield phoneme recognition accuracies in between the simple features and the MFPLP features, obfuscation methods applied on the excitation features yield low phoneme accuracies in conjunction with SND performance comparable to that of MFPLP features.

Managing multiple actions through multimodality: Doctors' involvement in interpreter-mediated interactions

AbstractMany studies (Heath 1984b, 1986; Robinson 1998; Robinson & Stivers 2001; Greatbach 2006; Frers 2009) have shown that, while interacting with their patients, doctors fulfill several tasks (speaking, reading, writing prescriptions, reporting data on the computer, etc.) through an interplay of diverse multimodal resources (speech, gesture, body posture, gaze, object manipulation, etc.). In comparison, multimodality in interpreter-mediated medical encounters has received scant attention. This contribution examines a corpus of interactions involving Italian doctors, Chinese patients, and an interpreter. Having to manage multiparty and multitask conversations with patients with whom no direct communication is possible, doctors are forced to rely heavily on multimodality. The analysis here shows how the simultaneous actions performed by the doctors are coordinated through the use of multiple modalities. The outcomes on the global organization of interpreter-mediated interaction are also discussed. (Conversation analysis, doctor-patient interaction, interpreting, multitasking, multimodality)*

A-STAR: Toward translating Asian spoken languages

This paper outlines the first Asian network-based speech-to-speech translation system developed by the Asian Speech Translation Advanced Research (A-STAR) consortium. Eight research groups comprising the A-STAR members participated in the experiments, covering nine languages, i.e., eight Asian languages (Hindi, Indonesian, Japanese, Korean, Malay, Thai, Vietnamese, and Chinese) and English. Each A-STAR member contributed one or more of the following spoken language technologies: automatic speech recognition, machine translation, and text-to-speech through Web servers. The system was designed to translate common spoken utterances of travel conversations from a given source language into multiple target languages in order to facilitate multiparty travel conversations between people speaking different Asian languages. It covers travel expressions including proper nouns that are names of famous places or attractions in Asian countries. In this paper, we describe the issues of developing spoken language technologies for Asian languages, and discuss the difficulties involved in connecting different heterogeneous spoken language translation systems through Web servers. This paper also presents speech-translation results including subjective evaluation, from the first A-STAR field testing which was carried out in July 2009.