Nonverbal Vocalizations Research Articles

Voices without words: the spectrum of nonverbal vocalisations

ABSTRACT Nonverbal vocalisations are a fundamental part of human life. Here, we use Tinbergen’s ethological framework to examine the functions, ontogenetic trajectories, evolutionary history, and underlying mechanisms of five types of vocalisations: cries, laughter, moans, screams, and sighs. Integrating insights from evolutionary biology and social psychology, we demonstrate how biological functions and social factors shape vocalisations. Tracing the development of vocalisations from infancy to adulthood emphasises the role of innate tendencies, as well as delineating processes of social learning. For each vocalisation type, we map their distinct nature and communicative potential: cries are crucial for survival and caregiving responses; laughter fosters social bonding and group cohesion; moans convey a wide range of internal states from pleasure to discomfort; screams serve as urgent alarms in critical situations; and sighs regulate emotions and signal shifts in emotional states. We conclude that both biological and social factors must be considered to understand nonverbal vocalisations.

What Is Faster than Where in Vocal Emotional Perception.

Voices carry a vast amount of information about speakers (e.g., emotional state; spatial location). Neuroimaging studies postulate that spatial ("where") and emotional ("what") cues are processed by partially independent processing streams. Although behavioral evidence reveals interactions between emotion and space, the temporal dynamics of these processes in the brain and its modulation by attention remain unknown. We investigated whether and how spatial and emotional features interact during voice processing as a function of attention focus. Spatialized nonverbal vocalizations differing in valence (neutral, amusement, anger) were presented at different locations around the head, whereas listeners discriminated either the spatial location or emotional quality of the voice. Neural activity was measured with ERPs of the EEG. Affective ratings were collected at the end of the EEG session. Emotional vocalizations elicited decreased N1 but increased P2 and late positive potential amplitudes. Interactions of space and emotion occurred at the salience detection stage: neutral vocalizations presented at right (vs. left) locations elicited increased P2 amplitudes, but no such differences were observed for emotional vocalizations. When task instructions involved emotion categorization, the P2 was increased for vocalizations presented at front (vs. back) locations. Behaviorally, only valence and arousal ratings showed emotion-space interactions. These findings suggest that emotional representations are activated earlier than spatial representations in voice processing. The perceptual prioritization of emotional cues occurred irrespective of task instructions but was not paralleled by an augmented stimulus representation in space. These findings support the differential responding to emotional information by auditory processing pathways.

Investigating Acoustic and Psycholinguistic Predictors of Cognitive Impairment in Older Adults: Modeling Study.

About one-third of older adults aged 65 years and older often have mild cognitive impairment or dementia. Acoustic and psycho-linguistic features derived from conversation may be of great diagnostic value because speech involves verbal memory and cognitive and neuromuscular processes. The relative decline in these processes, however, may not be linear and remains understudied. This study aims to establish associations between cognitive abilities and various attributes of speech and natural language production. To date, the majority of research has been cross-sectional, relying mostly on data from structured interactions and restricted to textual versus acoustic analyses. In a sample of 71 older (mean age 83.3, SD 7.0 years) community-dwelling adults who completed qualitative interviews and cognitive testing, we investigated the performance of both acoustic and psycholinguistic features associated with cognitive deficits contemporaneously and at a 1-2 years follow up (mean follow-up time 512.3, SD 84.5 days). Combined acoustic and psycholinguistic features achieved high performance (F1-scores 0.73-0.86) and sensitivity (up to 0.90) in estimating cognitive deficits across multiple domains. Performance remained high when acoustic and psycholinguistic features were used to predict follow-up cognitive performance. The psycholinguistic features that were most successful at classifying high cognitive impairment reflected vocabulary richness, the quantity of speech produced, and the fragmentation of speech, whereas the analogous top-ranked acoustic features reflected breathing and nonverbal vocalizations such as giggles or laughter. These results suggest that both acoustic and psycholinguistic features extracted from qualitative interviews may be reliable markers of cognitive deficits in late life.

When to Laugh, When to Cry: Display Rules of Nonverbal Vocalisations Across Four Cultures

AbstractNonverbal vocalisations like laughter, sighs, and groans are a fundamental part of everyday communication. Yet surprisingly little is known about the social norms concerning which vocalisations are considered appropriate to express in which context (i.e., display rules). Here, in two pre-registered studies, we investigate how people evaluate the appropriateness of different nonverbal vocalisations across locations and relationships with listeners. Study 1, with a U.S. sample (n = 250), showed that certain vocalisations (e.g., laughter, sighs, cries) are consistently viewed as more socially acceptable than others (e.g., roars, groans, moans). Additionally, location (private vs. public) and interpersonal closeness (close vs. not close) significantly influenced these perceptions, with private locations and close relationships fostering greater expressive freedom. Study 2 extended this investigation across four societies with divergent cultural norms (n = 1120 in total): the U.S. (for direct replication), Türkiye, China, and the Netherlands. Findings largely replicated those from Study 1 and supported the existence of cross-culturally consistent patterns in display rules for nonverbal vocalisations, though with some variation across cultures. This research expands our understanding of how social norms affect auditory communication, extending beyond the visual modality of facial expressions to encompass the rich world of nonverbal vocalisations.

Humans need auditory experience to produce typical volitional nonverbal vocalizations

Human nonverbal vocalizations such as screams and cries often reflect their evolved functions. Although the universality of these putatively primordial vocal signals and their phylogenetic roots in animal calls suggest a strong reflexive foundation, many of the emotional vocalizations that we humans produce are under our voluntary control. This suggests that, like speech, volitional vocalizations may require auditory input to develop typically. Here, we acoustically analyzed hundreds of volitional vocalizations produced by profoundly deaf adults and typically-hearing controls. We show that deaf adults produce unconventional and homogenous vocalizations of aggression and pain that are unusually high-pitched, unarticulated, and with extremely few harsh-sounding nonlinear phenomena compared to controls. In contrast, fear vocalizations of deaf adults are relatively acoustically typical. In four lab experiments involving a range of perception tasks with 444 participants, listeners were less accurate in identifying the intended emotions of vocalizations produced by deaf vocalizers than by controls, perceived their vocalizations as less authentic, and reliably detected deafness. Vocalizations of congenitally deaf adults with zero auditory experience were most atypical, suggesting additive effects of auditory deprivation. Vocal learning in humans may thus be required not only for speech, but also to acquire the full repertoire of volitional non-linguistic vocalizations.

Perception and discrimination of real-life emotional vocalizations in early blind individuals.

The capacity to understand others' emotions and react accordingly is a key social ability. However, it may be compromised in case of a profound sensory loss that limits the contribution of available contextual cues (e.g., facial expression, gestures, body posture) to interpret emotions expressed by others. In this study, we specifically investigated whether early blindness affects the capacity to interpret emotional vocalizations, whose valence may be difficult to recognize without a meaningful context. We asked a group of early blind (N = 22) and sighted controls (N = 22) to evaluate the valence and the intensity of spontaneous fearful and joyful non-verbal vocalizations. Our data showed that emotional vocalizations presented alone (i.e., with no contextual information) are similarly ambiguous for blind and sighted individuals but are perceived as more intense by the former possibly reflecting their higher saliency when visual experience is unavailable. Our study contributes to a better understanding of how sensory experience shapes ememotion recognition.

Sources of Sound Exposure in Pediatric Critical Care.

Sound levels in the pediatric intensive care unit (PICU) are often above recommended levels, but few researchers have identified the sound sources contributing to high levels. To identify sources of PICU sound exposure. This was a secondary analysis of continuous bedside video and dosimeter data (n = 220.7 hours). A reliable coding scheme developed to identify sound sources in the adult ICU was modified for pediatrics. Proportions of sound sources were compared between times of high (≥45 dB) and low (<45 dB) sound, during day (7 AM to 6:59 PM) and night (7 PM to 6:59 AM) shifts, and during sound peaks (≥70 dB). Overall, family vocalizations (38% of observation time, n = 83.9 hours), clinician vocalizations (32%, n = 70.6 hours), and child nonverbal vocalizations (29.4%, n = 64.9 hours) were the main human sound sources. Media sounds (57.7%, n = 127.3 hours), general activity (40.7%, n = 89.8 hours), and medical equipment (31.3%, n = 69.1 hours) were the main environmental sound sources. Media sounds occurred in more than half of video hours. Child nonverbal (71.6%, n = 10.2 hours) and family vocalizations (63.2%, n = 9 hours) were highly prevalent during sound peaks. General activity (32.1%, n = 33.2 hours), clinician vocalizations (22.5%, n = 23.3 hours), and medical equipment sounds (20.6, n = 21.3 hours) were prevalent during night shifts. Clinicians should partner with families to limit nighttime PICU noise pollution. Large-scale studies using this reliable coding scheme are needed to understand the PICU sound environment.

Non-speech emotion recognition based on back propagation feed forward networks

Non-speech emotion recognition involves identifying emotions conveyed through non-verbal vocalizations such as laughter, crying, and other sound signals, which play a crucial role in emotional expression and transmission. This paper employs a nine-category discrete emotion model encompassing happy, sad, angry, peaceful, fearful, loving, hateful, brave, and neutral. A proprietary non-speech dataset comprising 2337 instances was utilized, with 384-dimensional feature vectors extracted. The traditional Backpropagation Neural Network (BPNN) algorithm achieved a recognition rate of 87.7% on the non-speech dataset. In contrast, the proposed Whale Optimization Algorithm - Backpropagation Neural Network (WOA-BPNN) algorithm, applied to a self-made non-speech dataset, demonstrated a remarkable accuracy of 98.6%. Notably, even without facial emotional cues, non-speech sounds effectively convey dynamic information, and the proposed algorithm excels in their recognition. The study underscores the importance of non-speech emotional signals in communication, especially with the continuous advancement of artificial intelligence technology. The abstract thus encapsulates the paper’s focus on leveraging AI algorithms for high-precision non-speech emotion recognition.

Brief Report: Quantifying Speech Production Coordination from Non- and Minimally-Speaking Individuals.

Non-verbal utterances are an important tool of communication for individuals who are non- or minimally-speaking. While these utterances are typically understood by caregivers, they can be challenging to interpret by their larger community. To date, there has been little work done to detect and characterize the vocalizations produced by non- or minimally-speaking individuals. This paper aims to characterize five categories of utterances across a set of 7 non- or minimally-speaking individuals. The characterization is accomplished using a correlation structure methodology, acting as a proxy measurement for motor coordination, to localize similarities and differences to specific speech production systems. We specifically find that frustrated and dysregulated utterances show similar correlation structure outputs, especially when compared to self-talk, request, and delighted utterances. We additionally witness higher complexity of coordination between articulatory and respiratory subsystems and lower complexity of coordination between laryngeal and respiratory subsystems in frustration and dysregulation as compared to self-talk, request, and delight. Finally, we observe lower complexity of coordination across all three speech subsystems in the request utterances as compared to self-talk and delight. The insights from this work aid in understanding of the modifications made by non- or minimally-speaking individuals to accomplish specific goals in non-verbal communication.

Identification of Non-Speaking and Minimal-Speaking Individuals Using Nonverbal Vocalizations

Identification of Non-Speaking and Minimal-Speaking Individuals Using Nonverbal Vocalizations

Analysis of Illocutionary Act in A Quite Place Movie

This research aims to describe the use of illocutionary acts in the film A Quite Place. This study used a descriptive qualitative method. This research uses Searle's theory to find the verbal and nonverbal utterances of all the characters included in the illocutionary act theory. Data collection is carried out by downloading, reading, watching, and classifying. The data analysis technique is to present, describe, interpret, and conclude the results of the analysis. Based on the theories provided, the data is analyzed one by one to determine the context and type of illocutionary act used. The results of the research show that there are four types of illocutionary acts used by all the characters in the film A Quiet Place 2018. The instructions present the dominant frequencies of the five types of illocutionary acts. There are six that contain directive actions. The assertive one has four data. Expressive has four data, the lowest frequency of illocutionary acts is commissive, there are two data. This type of declaration was not found in this study. The data most often found in this film is directive with six data and the least is commissive with two data. Based on the explanation above, it can be concluded that the problems that occurred in this research have been answered so that the research objective has been found and explained clearly, namely how illocutionary acts are used in the film A Quite Place using Searle's theory.

‘Should we laugh?’ Acoustic features of (in)voluntary laughters in spontaneous conversations

Laughter is one of the most common non-verbal features; however, contrary to the previous assumptions, it may also act as signals of bonding, affection, emotional regulation agreement or empathy (Scott et al. Trends Cogn Sci 18:618–620, 2014). Although previous research agrees that laughter does not form a uniform group in many respects, different types of laughter have been defined differently by individual research. Due to the various definitions of laughter, as well as their different methodologies, the results of the previous examinations were often contradictory. The analysed laughs were often recorded in controlled, artificial situations; however, less is known about laughs from social conversations. Thus, the aim of the present study is to examine the acoustic realisation, as well as the automatic classification of laughter that appear in human interactions according to whether listeners consider them to be voluntary or involuntary. The study consists of three parts using a multi-method approach. Firstly, in the perception task, participants had to decide whether the given laughter seemed to be rather involuntary or voluntary. In the second part of the experiment, those sound samples of laughter were analysed that were considered to be voluntary or involuntary by at least 66.6% of listeners. In the third part, all the sound samples were grouped into the two categories by an automatic classifier. The results showed that listeners were able to distinguish laughter extracted from spontaneous conversation into two different types, as well as the distinction was possible on the basis of the automatic classification. In addition, there were significant differences in acoustic parameters between the two groups of laughter. The results of the research showed that, although the distinction between voluntary and involuntary laughter categories appears based on the analysis of everyday, spontaneous conversations in terms of the perception and acoustic features, there is often an overlap in the acoustic features of voluntary and involuntary laughter. The results will enrich our previous knowledge of laughter and help to describe and explore the diversity of non-verbal vocalisations.

Sounds like a fight: listeners can infer behavioural contexts from spontaneous nonverbal vocalisations

ABSTRACT When we hear another person laugh or scream, can we tell the kind of situation they are in – for example, whether they are playing or fighting? Nonverbal expressions are theorised to vary systematically across behavioural contexts. Perceivers might be sensitive to these putative systematic mappings and thereby correctly infer contexts from others’ vocalisations. Here, in two pre-registered experiments, we test the prediction that listeners can accurately deduce production contexts (e.g. being tickled, discovering threat) from spontaneous nonverbal vocalisations, like sighs and grunts. In Experiment 1, listeners (total n = 3120) matched 200 nonverbal vocalisations to one of 10 contexts using yes/no response options. Using signal detection analysis, we show that listeners were accurate at matching vocalisations to nine of the contexts. In Experiment 2, listeners (n = 337) categorised the production contexts by selecting from 10 response options in a forced-choice task. By analysing unbiased hit rates, we show that participants categorised all 10 contexts at better-than-chance levels. Together, these results demonstrate that perceivers can infer contexts from nonverbal vocalisations at rates that exceed that of random selection, suggesting that listeners are sensitive to systematic mappings between acoustic structures in vocalisations and behavioural contexts.

JNV corpus: A corpus of Japanese nonverbal vocalizations with diverse phrases and emotions

We present JNV (Japanese Nonverbal Vocalizations) corpus, a corpus of Japanese nonverbal vocalizations (NVs) with diverse phrases and emotions. Existing Japanese NV corpora either lack phrase diversity or focus on a small number of emotions, which makes it difficult to analyze the characteristics of Japanese NVs and support downstream tasks like emotion recognition. We first propose a corpus-design method that contains two phases: (1) collecting NVs phrases based on crowd-sourcing; (2) recording NVs by stimulating speakers with emotional scenarios. We then collect 420 audio clips from 4 speakers that cover 6 emotions based on the proposed method. Results of comprehensive objective and subjective experiments demonstrate that (1) the emotions of the collected NVs can be recognized with high accuracy by both human evaluators and statistical models; (2) the collected NVs have a high authenticity comparable to previous corpora of English NVs. Additionally, we analyze the distributions of vowel types in Japanese and conduct feature importance analysis to show discriminative acoustic features between emotion categories in Japanese NVs. We publicate JNV to advance further development in this field.

Beyond speech: Exploring diversity in the human voice

Humans have evolved voluntary control over vocal production for speaking and singing, while preserving the phylogenetically older system of spontaneous nonverbal vocalizations such as laughs and screams. To test for systematic acoustic differences between these vocal domains, we analyzed a broad, cross-cultural corpus representing over 2h of speech, singing, and nonverbal vocalizations. We show that, while speech is relatively low-pitched and tonal with mostly regular phonation, singing and especially nonverbal vocalizations vary enormously in pitch and often display harsh-sounding, irregular phonation owing to nonlinear phenomena. The evolution of complex supralaryngeal articulatory spectro-temporal modulation has been critical for speech, yet has not significantly constrained laryngeal source modulation. In contrast, articulation is very limited in nonverbal vocalizations, which predominantly contain minimally articulated open vowels and rapid temporal modulation in the roughness range. We infer that vocal source modulation works best for conveying affect, while vocal filter modulation mainly facilitates semantic communication.

The Discourse of Sighs in Medieval English Literature

ABSTRACT The very ubiquity of references to sighing in medieval English literary culture has meant that the significance of sighs has remained hidden in plain sight. Yet the introduction of sighs into medieval reimaginings of biblical and other sources underlines how much sighing is part of late medieval affect. A dual quality to sighs makes their interpretation depend on context: sighs may express negative or positive responses. As nonverbal vocalizations, sighs constitute a mode of communication, while in devotional tradition the penitent’s sighs may offer a means of bringing into being a desired spiritual state. Chaucer’s poems show him aligning sighs with significant speech, most intensively in Troilus, where Chaucer suggests sighs’ communicative role and their different kinds, while exploring their interconnections with breathing and emotion, and how sighs can powerfully transcend speech with what is left unsaid.

ReCANVo: A database of real-world communicative and affective nonverbal vocalizations

Nonverbal vocalizations, such as sighs, grunts, and yells, are informative expressions within typical verbal speech. Likewise, individuals who produce 0–10 spoken words or word approximations (“minimally speaking” individuals) convey rich affective and communicative information through nonverbal vocalizations even without verbal speech. Yet, despite their rich content, little to no data exists on the vocal expressions of this population. Here, we present ReCANVo: Real-World Communicative and Affective Nonverbal Vocalizations - a novel dataset of non-speech vocalizations labeled by function from minimally speaking individuals. The ReCANVo database contains over 7000 vocalizations spanning communicative and affective functions from eight minimally speaking individuals, along with communication profiles for each participant. Vocalizations were recorded in real-world settings and labeled in real-time by a close family member who knew the communicator well and had access to contextual information while labeling. ReCANVo is a novel database of nonverbal vocalizations from minimally speaking individuals, the largest available dataset of nonverbal vocalizations, and one of the only affective speech datasets collected amidst daily life across contexts.

Adolescent Asthma Monitoring: A Preliminary Study of Audio and Spirometry Modalities.

Asthma patients' sleep quality is correlated with how well their asthma symptoms are controlled. In this paper, deep learning techniques are explored to improve forecasting of forced expiratory volume in one second (FEV1) by using audio data from participants and test whether auditory sleep disturbances are correlated with poorer asthma outcomes. These are applied to a representative data set of FEV1 collected from a commercially available sprirometer and audio spectrograms collected overnight using a smartphone. A model for detecting nonverbal vocalizations including coughs, sneezes, sighs, snoring, throat clearing, sniffs, and breathing sounds was trained and used to capture nightly sleep disturbances. Our preliminary analysis found significant improvement in FEV1 forecasting when using overnight nonverbal vocalization detections as an additional feature for regression using XGBoost over using only spirometry data.Clinical relevance- This preliminary study establishes up to 30% improvement of FEV1 forecasting using features generated by deep learning techniques over only spirometry-based features.

Multi-modal speech emotion detection using optimised deep neural network classifier

ABSTRACT Speech emotion recognition (SER) has received significant attention recently and seems to be a critical aspect of human-computer interaction. The performance of the recent methods is not at the expected level there are many different strategies have been developed for SER. In the present situation, communication, non-verbal vocalisation, with a vocal sound plays an essential role in emotional expression. The multimodal database is considered for which the hybridised model of the audio-visual dependent emotion identification model is proposed. In this research, the optimised deep NN is used to recognise emotions from multimodal input data. For better performance in emotion recognition, the learner memorising optimisation fine-tunes the hyperparameters involved in the deep NN network classifier. The hybrid texture feature descriptor is proposed to improve classification outcomes from audio-video signals as well as accuracy. The learner memorising optimisation is used in the deep NN classifier to obtain the ideal parameters involved in exploring space with high accuracy. The metric values of the learner memorising optimisation-based deep NN have been evaluated using the Enter face’05 database and the proposed method obtained 97.490% accuracy, 98% sensitivity, and 97.490% specificity for the K-fold value 10% and 96.928%, 98.80%, and 96.928% for 90% training data.

Modeling Real-World Affective and Communicative Nonverbal Vocalizations From Minimally Speaking Individuals

Nonverbal vocalizations from non- and minimally speaking individuals who speak fewer than 20 words (mv* individuals) convey important communicative and affective information. While nonverbal vocalizations that occur amidst typical speech and infant vocalizations have been studied extensively in the literature, there is limited prior work on vocalizations by mv* individuals. Our work is among the first studies of the communicative and affective information expressed in nonverbal vocalizations by mv* children and adults. We collected labeled vocalizations in real-world settings with eight mv* communicators, with communicative and affective labels provided in-the-moment by a close family member. Using evaluation strategies suitable for messy, real-world data, we show that nonverbal vocalizations can be classified by function (with 4- and 5-way classifications) with F1 scores above chance for all participants. We analyze labeling and data collection practices for each participating family, and discuss the classification results in the context of our novel real-world data collection protocol. The presented work includes results from the largest classification experiments with nonverbal vocalizations from mv* communicators to date.