Vocal Modalities Research Articles

Communicative efficiency in multimodal language directed at children and adults.

The ecology of human communication is face to face. In these contexts, speakers dynamically modify their communication across vocal (e.g., speaking rate) and gestural (e.g., cospeech gestures related in meaning to the content of speech) channels while speaking. What is the function of these adjustments? Here we ask whether speakers dynamically make these adjustments to increase communicative success, and decrease cognitive effort while speaking. We assess whether speakers modulate word durations and produce iconic (i.e., imagistically evoking properties of referents) gestures depending on the predictability of each word they utter. Predictability is operationalized as surprisal and computed from computational language models trained on corpora of child-directed, or adult-directed language. Using data from a novel corpus (Ecological Language Corpus) of naturalistic interactions between adult-child (aged 3-4), and adult-adult, we show that surprisal predicts speakers' multimodal adjustments and that some of these effects are modulated by whether the comprehender is a child or an adult. Thus, communicative efficiency applies generally across vocal and gestural communicative channels not being limited to structural properties of language or vocal modality. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Audio–Visual Fusion for Emotion Recognition in the Valence–Arousal Space Using Joint Cross-Attention

Automatic emotion recognition (ER) has recently gained lot of interest due to its potential in many real-world applications. In this context, multimodal approaches have been shown to improve performance (over unimodal approaches) by combining diverse and complementary sources of information, providing some robustness to noisy and missing modalities. In this paper, we focus on dimensional ER based on the fusion of facial and vocal modalities extracted from videos, where complementary audio-visual (A-V) relationships are explored to predict an individual’s emotional states in valence-arousal space. Most state-of-the-art fusion techniques rely on recurrent networks or conventional attention mechanisms that do not effectively leverage the complementary nature of A-V modalities. To address this problem, we introduce a joint cross-attentional model for A-V fusion that extracts the salient features across A-V modalities, that allows to effectively leverage the inter-modal relationships, while retaining the intra-modal relationships. In particular, it computes the cross-attention weights based on correlation between the joint feature representation and that of the individual modalities. By deploying the joint A-V feature representation into the cross-attention module, it helps to simultaneously leverage both the intra and inter modal relationships, thereby significantly improving the performance of the system over the vanilla cross-attention module. The effectiveness of our proposed approach is validated experimentally on challenging videos from the RECOLA and AffWild2 datasets. Results indicate that our joint cross-attentional A-V fusion model provides a cost-effective solution that can outperform state-of-the-art approaches, even when the modalities are noisy or absent. Code is available at https://github.com/praveena2j/Joint-Cross-Attention-for-Audio-Visual-Fusion.

Multi-modal humor segment prediction in video

Humor can be induced by various signals in the visual, linguistic, and vocal modalities emitted by humans. Finding humor in videos is an interesting but challenging task for an intelligent system. Previous methods predict humor in the sentence level given some text (e.g., speech transcript), sometimes together with other modalities, such as videos and speech. Such methods ignore humor caused by the visual modality in their design, since their prediction is made for a sentence. In this work, we first give new annotations to humor based on a sitcom by setting up temporal segments of ground truth humor derived from the laughter track. Then, we propose a method to find these temporal segments of humor. We adopt an approach based on sliding window, where the visual modality is described by pose and facial features along with the linguistic modality given as subtitles in each sliding window. We use long short-term memory networks to encode the temporal dependency in poses and facial features and pre-trained BERT to handle subtitles. Experimental results show that our method improves the performance of humor prediction.

Still-face redux: Infant responses to a classic and modified still-face paradigm in proximal and distal care cultures

Literature on infant emotion is dominated by research conducted in Western, industrialized societies where early socialization is characterized by face-to-face, vocal communication with caregivers. There is a dearth of knowledge of infant emotion in the context of social interaction outside of the visual and vocal modalities. In a three-population cross-cultural comparison, we used the still-face task to measure variation in behavior among infants from proximal care (practicing high levels of physical contact) communities in Bolivia and distal care (emphasizing vocal and visual interaction) communities in the U.S. and Fiji. In a modified version of the face-to-face still-face (FFSF), Study 1, infants in the U.S. and Fiji displayed the typical behavioral response to the still-face episode: increased negative affect and decreased social engagement, whereas infants in Bolivia showed no change. For tactile behavior, infants in Bolivia showed an increase in tactile self-stimulation from the interaction episode to the still-face episode, whereas U.S. infants showed no change. In Study 2, we created a novel body-to-body version of the still-face paradigm (“still-body”) with infants in US and Bolivia, to mimic the near-constant physical contact Bolivian infants experience. The U.S. and Bolivian infant response was similar to Study 1: US infants showed decreased positive affect and increased negative affect and decreased social engagement from the interaction to the still-body episode and Bolivian infants showed no change. Notably, there were overall differences in infant behaviors between the two paradigms (FFSF and Still-Body). Infants in Bolivia and the U.S. showed increased positive facial affect during the FFSF paradigm in comparison with the Still-Body paradigm. Our results demonstrate the need for more globally representative developmental research and a broader approach to infant emotion and communication.

Gesture is the primary modality for language creation.

How language began is one of the oldest questions in science, but theories remain speculative due to a lack of direct evidence. Here, we report two experiments that generate empirical evidence to inform gesture-first and vocal-first theories of language origin; in each, we tested modern humans' ability to communicate a range of meanings (995 distinct words) using either gesture or non-linguistic vocalization. Experiment 1 is a cross-cultural study, with signal Producers sampled from Australia (n = 30, Mage = 32.63, s.d. = 12.42) and Vanuatu (n = 30, Mage = 32.40, s.d. = 11.76). Experiment 2 is a cross-experiential study in which Producers were either sighted (n = 10, Mage = 39.60, s.d. = 11.18) or severely vision-impaired (n = 10, Mage = 39.40, s.d. = 10.37). A group of undergraduate student Interpreters guessed the meaning of the signals created by the Producers (n = 140). Communication success was substantially higher in the gesture modality than the vocal modality (twice as high overall; 61.17% versus 29.04% success). This was true within cultures, across cultures and even for the signals produced by severely vision-impaired participants. The success of gesture is attributed in part to its greater universality (i.e. similarity in form across different Producers). Our results support the hypothesis that gesture is the primary modality for language creation.

Overlooked evidence for semantic compositionality and signal reduction in wild chimpanzees (Pan troglodytes)

Recent discoveries of semantic compositionality in Japanese tits have enlivened the discussions on the presence of this phenomenon in wild animal communication. Data on semantic compositionality in wild apes are lacking, even though language experiments with captive apes have demonstrated they are capable of semantic compositionality. In this paper, I revisit the study by Boesch (Hum. Evol. 6:81–89, 1991) who investigated drumming sequences by an alpha male in a chimpanzee (Pan troglodytes) community in the Taï National Park, Côte d’Ivoire. A reanalysis of the data reveals that the alpha male produced semantically compositional combined messages of travel direction change and resting period initiation. Unlike the Japanese tits, the elements of the compositional expression were not simply juxtaposed but displayed structural reduction, while one of the two elements in the expression coded the meanings of both elements. These processes show relative resemblance to blending and fusion in human languages. Also unlike the tits, the elements of the compositional expression did not have a fixed order, although there was a fixed distribution of drumming events across the trees used for drumming. Because the elements of the expression appear to carry verb-like meanings, the compositional expression also resembles simple verb-verb constructions and short paratactic combinations of two clauses found across languages. In conclusion, the reanalysis suggests that semantic compositionality and phenomena resembling paratactic combinations of two clauses might have been present in the communication of the last common ancestor of chimpanzees and humans, not necessarily in the vocal modality.

A Cross-Sectional Test of Sign Creation by Children in the Gesture and Vocal Modalities.

Naturalistic studies show that children can create language-like communication systems in the absence of conventional language. However, experimental evidence is mixed. We address this discrepancy using an experimental paradigm that simulates naturalistic sign creation. Specifically, we tested if a sample of 6- to 12-year-old children (52 girls and 56 boys drawn from an urban, predominantly white population in Western Australia) can comprehend and create novel gestural and vocal signs. Experiment 1 tested children's ability to comprehend novel signs. Experiment 2 tested children's ability to create novel signs. Results show that children can comprehend and create gestural and vocal signs, that communication is more successful in the gesture modality, and that older children outperform younger children.

A Tale of Two Modalities: Sign and Speech Influence Each Other in Bimodal Bilinguals

Bimodal bilinguals are hearing individuals fluent in a sign and a spoken language. Can the two languages influence each other in such individuals despite differences in the visual (sign) and vocal (speech) modalities of expression? We investigated cross-linguistic influences on bimodal bilinguals’ expression of spatial relations. Unlike spoken languages, sign uses iconic linguistic forms that resemble physical features of objects in a spatial relation and thus expresses specific semantic information. Hearing bimodal bilinguals (n = 21) fluent in Dutch and Sign Language of the Netherlands and their hearing nonsigning and deaf signing peers (n = 20 each) described left/right relations between two objects. Bimodal bilinguals expressed more specific information about physical features of objects in speech than nonsigners, showing influence from sign language. They also used fewer iconic signs with specific semantic information than deaf signers, demonstrating influence from speech. Bimodal bilinguals’ speech and signs are shaped by two languages from different modalities.

Organization of Oral Dialogical Discourse: Feedback and Ways of Its Explication

The results of the study of feedback in the Russian oral dialogical discourse are presented in the article. The relevance of this study is due to several factors. First, the debatable issue is related to the qualification of fragments of oral discourse from the point of view of their relevance to the genre of monologue, dialogue, polylogue. Secondly, there is a number of theoretical problems arising from such qualifications, in particular, in the scientific description of feedback markers (based on the material of the Russian language), a rather large number of gaps are found. The definitions of the terms feedback and feedback markers are provided in the article. The authors note that it seems advisable to consider feedback markers in a somewhat expanded way, including in this group not only units used to express positive attention, but also units that can signal difficulties in perceiving the interlocutor’s utterance, as well as illocutionary forced units that are a reaction to the initial a replica of the interlocutor. Criteria for the inclusion of units functioning within the vocal modality in the group of feedback markers are proposed. A classification of feedback markers based on the ability of units to collapse the proposition contained in the statement of the interlocutor is proposed. The quantitative characteristics of the described subgroups are presented.

The hard work of doing nothing: Accounting for inhibitory costs during multiple action control.

Performing many actions at the same time is usually associated with performance costs. However, recent eye-tracking evidence indicates that under specific conditions, inhibiting a secondary response can be more costly than executing it, resulting in dual-action benefits. Here, we show that performance gains due to the absence of inhibitory control demands in dual-action trials are not limited to saccades as a response modality. In our study, participants had to react to a visually presented directional word by either reading the stimulus aloud (vocal modality), pressing the corresponding arrow key on a keyboard (manual modality), or both. Crucially, manual error rates were significantly lower when participants had to respond with both a button press and naming than when they had to respond with naming only. More specifically, in vocal-only conditions we observed a significant percentage of false-positive manual responses, suggesting difficulties with inhibiting an unwarranted manual action. Thus, our results indicate that difficulties associated with single- (vs. dual-) action control are a stable, domain-general phenomenon that likely arises whenever execution-related demands are accompanied by substantial additional inhibitory control demands.

Family expressiveness relates to happy emotion matching among 9-month-old infants

Perceiving and understanding the emotions of those around us is an imperative skill to develop early in life. An infant’s family environment provides most of their emotional exemplars in early development. However, the relation between the early development of emotion perception and family expressiveness remains understudied. To investigate this potential link to early emotion perception development, we examined 38 infants at 9 months of age. We assessed infants’ ability to match emotions across facial and vocal modalities using an intermodal matching paradigm for angry–neutral, happy–neutral, and sad–neutral pairings. We also attained family expressiveness information via parent report. Our results indicate a significant positive relation between emotion matching and family expressiveness specific to the happy–neutral condition. However, we found no evidence for emotion matching for the infants as a group in any of the three conditions. These results suggest that family expressiveness does relate to emotion matching for the earliest developing emotional category among 9-month-old infants and that emotion matching with multiple emotions at this age is a challenging task.

Facial Expression Enhances Emotion Perception Compared to Vocal Prosody: Behavioral and fMRI Studies.

Facial and vocal expressions are essential modalities mediating the perception of emotion and social communication. Nonetheless, currently little is known about how emotion perception and its neural substrates differ across facial expression and vocal prosody. To clarify this issue, functional MRI scans were acquired in Study 1, in which participants were asked to discriminate the valence of emotional expression (angry, happy or neutral) from facial, vocal, or bimodal stimuli. In Study 2, we used an affective priming task (unimodal materials as primers and bimodal materials as target) and participants were asked to rate the intensity, valence, and arousal of the targets. Study 1 showed higher accuracy and shorter response latencies in the facial than in the vocal modality for a happy expression. Whole-brain analysis showed enhanced activation during facial compared to vocal emotions in the inferior temporal-occipital regions. Region of interest analysis showed a higher percentage signal change for facial than for vocal anger in the superior temporal sulcus. Study 2 showed that facial relative to vocal priming of anger had a greater influence on perceived emotion for bimodal targets, irrespective of the target valence. These findings suggest that facial expression is associated with enhanced emotion perception compared to equivalent vocal prosodies.

Using a Prerequisite Skills Assessment to Identify Optimal Modalities for Mand Training.

This study examined the utility of a brief prerequisite assessment in predicting the subsequent effectiveness and rate of acquisition of mand training in each of three response modalities (sign, picture exchange, and vocalizations). Overall, the picture exchange was the most effective and efficient modality for acquiring the targeted mand. The vocal modality was the least effective except when the prerequisite assessment indicated that two-syllable vocal imitation was intact. The implications for selection of response modality for early mand training are discussed.

Different Approaches to Meaning in Primate Gestural and Vocal Communication.

In searching for the roots of human language, comparative researchers investigate whether precursors to language are already present in our closest relatives, the non-human primates. As the majority of studies into primates’ communication use a unimodal approach with focus on one signal type only, researchers investigate very different aspects depending on whether they are interested in vocal, gestural, or facial communication. Here, we focus on two signal types and discuss how meaning is created in the gestural (visual, tactile/auditory) as compared to the vocal modality in non-human primates, to highlight the different research foci across these modalities. First, we briefly describe the defining features of meaning in human language and introduce some debates concerning meaning in non-human communication. Second, with focus on these features, we summarize the current evidence for meaningful communication in gestural as compared to vocal communication and demonstrate that meaning is operationalized very differently by researchers in these two fields. As a result, it is currently not possible to generalize findings across these modalities. Rather than arguing for or against the occurrence of semantic communication in non-human primates, we aim at pointing to gaps of knowledge in studying meaning in our closest relatives, and these gaps might be closed.

Debunking two myths against vocal origins of language

Gesture-first theories of language origins often raise two unsubstantiated arguments against vocal origins. First, they argue that great ape vocal behavior is highly constrained, limited to a fixed, species-typical repertoire of reflexive calls. Second, they argue that vocalizations lack any significant potential to ground meaning through iconicity, or resemblance between form and meaning. This paper reviews the considerable evidence that debunks these two “myths”. Accumulating evidence shows that the great apes exercise voluntary control over their vocal behavior, including their breathing apparatus, larynx, and supralaryngeal articulators. They are also able to learn new vocal behaviors, and even show some rudimentary ability for vocal imitation. In addition, an abundance of research demonstrates that the vocal modality affords rich potential for iconicity. People can understand iconicity in sound symbolism, and they can produce iconic vocalizations to communicate a diverse range of meanings. Thus, two of the primary arguments against vocal origins theories are not tenable. As an alternative, the paper concludes that the origins of language – going as far back as our last common ancestor with great apes – are rooted in iconicity in both gesture and vocalization.

On Vocal Assemblages: From Edison to Miku

For all its hermeneutic density the voice is a profoundly malleable object. By drawing our attention away from the ‘grain’ of the voice to networks of vocality, this article explores how modern mediations of the voice––from the microphone to auto-tune––illustrate its co-evolution with a history of non-human circuits and exchanges. The voice is categorized as an assemblage in the sense suggested by Deleuzian theoretical currents and actor network theory: a multi-scalar, human–non-human hybrid that emerges through nested constellations. Four vocal modalities––synthesis, deconstruction, auto-correlation, and simulation––are identified as integral to a modern history of vocal breaching experiments that transform what voices can be and how they are heard. The final part of the paper explores how these extensions play out in relation to the Japanese virtual singer, performer, and idol, Hatsune Miku. Miku’s voice, it will be argued, is constituted at the intersection of crowd-sourcing, corporate investment and the algorithm. But it also illustrates the distributed ontology of all voices.

The Still, Sad Music of Humanity in Doom Metal's Romanticizing Machine

The Still, Sad Music of Humanity in Doom Metal's Romanticizing Machine Atene Mendelyte (bio) The still, sad music of humanity,Nor harsh, nor grating, though of ample power —William Wordsworth, Tintern Abbey This essay was inspired by Ronald Bogue's pioneering Deleuzian reading of the three types of metal music—death, doom, and black. The scholar avoids the traps of oversimplification obvious in previous (mostly sociological) studies of such music that tend to concentrate on the threat of inducing violent behaviour. Such a stance by now has become a cliché and has little analytical value. I see a great potential in further developing this type of Deleuzian listening as it can offer a better understanding of metal music. It needs to be mentioned that there have been other analyses hinting at a more positive potential of metal music, such as musicologist Robert Walser's against the grain observation that metal music "creates communal attachments, enacts collective empowerment, and works to assuage entirely reasonable anxieties"1 or Nicola Masciandaro's experimental attempt to merge the discourses of black metal and theory.2 The essay collection edited by Niall W. R. Scott and Imke Von Helden The Metal Void: First Gatherings (2010)3 includes many varied explorations of metal music. The journal Glossator has a number of interesting articles on black metal, and another online journal Helvete is entirely dedicated to black metal theory. But such a surge of analytical readings tends to cluster around black and death metal while doom metal remains on the outskirts of scholarly interest. What is then lacking, I suggest, is a close(r) attention paid to individual doom metal practices. This is where my approach differs from Bogue's, since, unlike him, I think that each type of metal practice has to be understood on its own terms instead of comparing them based on a more transcendental or general basis. The practice that will be scrutinized in this essay is that of the [End Page 469] legendary British doom metal band My Dying Bride since, as mentioned above, doom metal, in comparison to death and black metal, has received by far the least interest in academia and non-academia alike. Even in Bogue's illuminating essay it remains the least explored subgenre, which is a direct consequence of the basis chosen for comparison; if one specifically discusses musical intensities and avoids focusing on the connection between lyrics and voice, one is bound to conclude that doom metal is both less technical, less virtuoso than death metal and less visceral than black metal. The lack of interest and doom metal's underdog status follow from that. But the center of gravity in doom metal is precisely the relation between lyrics, vocal modalities, and musical accompaniment, which makes it an entirely different kind of assemblage from the other types of metal. Moreover, I wish to argue, it is exactly where the greatest and most interesting musical thought potential lies. The machinic assemblage produced by My Dying Bride's practice is of a peculiar sort. I shall be referring to it as "the romanticizing machine" because it brings to fruition the principal logic of British Romantic poetry and European Romantic music. The processual aspect of the given title is pivotal as well, since the machine works by combining musical, textual, conceptual, affective, temporal, dramatic, and other elements and repurposing them for the sake of drawing its own plane of immanence and consistency, engaging in its musical world-making. Musical Deterritorialization and Reterritorialization My Dying Bride strives to create a total poetic experience. The band provides atmospheric musical plateaus that work to produce a mood, an affective space rather than a metric, rhythmic, musically progressive experience. The lyrics are highly poetic, often spoken in a narrative voice. Vocal delivery is used in a multiplicity of its modalities: sung vocals, death metal grunts, spoken word, whispers. Speech itself is modulated; it can either be a narrative voice, as in the closing lines of "Hail Odysseus," or a more poetically expressive voice, as in most songs on A Map of All Our Failures (Peaceville Records, 2012). The voice is made to exude despair, sadness, and other strong feelings via emphatic intonation...

Realization of sign language motion using a dual-arm/hand humanoid robot

The recent increase in technological maturity has empowered robots to assist humans and provide daily services. Voice command usually appears as a popular human---machine interface for communication. Unfortunately, deaf people cannot exchange information from robots through vocal modalities. To interact with deaf people effectively and intuitively, it is desired that robots, especially humanoids, have manual communication skills, such as performing sign languages. Without ad hoc programming to generate a particular sign language motion, we present an imitation system to teach the humanoid robot performing sign languages by directly replicating observed demonstration. The system symbolically encodes the information of human hand---arm motion from low-cost depth sensors as a skeleton motion time-series that serves to generate initial robot movement by means of perception-to-action mapping. To tackle the body correspondence problem, the virtual impedance control approach is adopted to smoothly follow the initial movement, while preventing potential risks due to the difference in the physical properties between the human and the robot, such as joint limit and self-collision. In addition, the integration of the leg-joints stabilizer provides better balance of the whole robot. Finally, our developed humanoid robot, NINO, successfully learned by imitation from human demonstration to introduce itself using Taiwanese Sign Language.

The influence of the visual modality on language structure and conventionalization: insights from sign language and gesture.

For humans, the ability to communicate and use language is instantiated not only in the vocal modality but also in the visual modality. The main examples of this are sign languages and (co-speech) gestures. Sign languages, the natural languages of Deaf communities, use systematic and conventionalized movements of the hands, face, and body for linguistic expression. Co-speech gestures, though non-linguistic, are produced in tight semantic and temporal integration with speech and constitute an integral part of language together with speech. The articles in this issue explore and document how gestures and sign languages are similar or different and how communicative expression in the visual modality can change from being gestural to grammatical in nature through processes of conventionalization. As such, this issue contributes to our understanding of how the visual modality shapes language and the emergence of linguistic structure in newly developing systems. Studying the relationship between signs and gestures provides a new window onto the human ability to recruit multiple levels of representation (e.g., categorical, gradient, iconic, abstract) in the service of using or creating conventionalized communicative systems.

Biases in facial and vocal emotion recognition in chronic schizophrenia.

There has been extensive research on impaired emotion recognition in schizophrenia in the facial and vocal modalities. The literature points to biases toward non-relevant emotions for emotional faces but few studies have examined biases in emotional recognition across different modalities (facial and vocal). In order to test emotion recognition biases, we exposed 23 patients with stabilized chronic schizophrenia and 23 healthy controls (HCs) to emotional facial and vocal tasks asking them to rate emotional intensity on visual analog scales. We showed that patients with schizophrenia provided higher intensity ratings on the non-target scales (e.g., surprise scale for fear stimuli) than HCs for the both tasks. Furthermore, with the exception of neutral vocal stimuli, they provided the same intensity ratings on the target scales as the HCs. These findings suggest that patients with chronic schizophrenia have emotional biases when judging emotional stimuli in the visual and vocal modalities. These biases may stem from a basic sensorial deficit, a high-order cognitive dysfunction, or both. The respective roles of prefrontal-subcortical circuitry and the basal ganglia are discussed.