Vocal Cues Research Articles

Paying attention to uncertainty: A stochastic multimodal transformers for post-traumatic stress disorder detection using video

Background and Objectives:Post-traumatic stress disorder is a debilitating psychological condition that can manifest following exposure to traumatic events. It affects individuals from diverse backgrounds and is associated with various symptoms, including intrusive thoughts, nightmares, hyperarousal, and avoidance behaviors. Methods:To address this challenge this study proposes a decision support system powered by a novel multimodal deep learning approach, based on a stochastic Transformer and video data. This Transformer has the ability to take advantage of its stochastic activation function and layers that allow it to learn sparse representations of the inputs. The method leverages a combination of low-level features extracted using three modalities, including Mel-frequency cepstral coefficients extracted from audio recordings, Facial Action Units captured from facial expressions, and textual data obtained from the audio transcription. By considering these modalities, our proposed model captures a comprehensive range of information related to post-traumatic stress disorder symptoms, including vocal cues, facial expressions, and linguistic content. Results:The deep learning model was trained and evaluated on the eDAIC dataset, which consists of clinical interviews with individuals with and without post-traumatic disorder. The model achieved state-of-the-art results, demonstrating its effectiveness in accurately detecting PTSD, showing an impressive Root Mean Square Error of 1.98, and a Concordance Correlation Coefficient of 0.722, signifying the model’s superior performance compared to existing approaches. Conclusion:This work introduces a new method for post-traumatic stress disorder detection from videos by utilizing a multimodal stochastic Transformer model. The model makes use of a variety of modalities, such as text, audio, and visual data, to gather comprehensive and varied information in order to make the detection.

How fund manager speaks values: evidence from China

ABSTRACT Confidence, especially in the form of overconfidence, is a crucial psychological factor that influences fund managers’ investment behaviours. However, how to reliably quantify it remains an open question. The self-marketing endeavours via video-sharing platforms of mutual fund companies in China have unexpectedly provided persuasive avenue to gauge confidence through video appearances. This paper proposes a novel data integration framework, combining with multimodal content understanding techniques, and offers calculations of nonverbal personal traits from videos circulated in social media. In particular, we focus on managers’ vocal cues and construct a confidence measure from vocal pitch. Our findings reveal a substantial relation between vocal confidence and fund managers’ investment decisions. Our work also underscores the potential of multimodal data from social media in disclosing nuanced findings in the interdisciplinary studies of finance and psychology.

Validation of behavioral measures of social cognition in individuals diagnosed with schizophrenia.

Schizophrenia, a complex neuropsychiatric disorder, manifests severe impairments in social cognition, notably in Theory of Mind (ToM), empathy, and emotion recognition, which significantly influence social competence and overall functioning. These aspects are crucial for prognosis in individuals diagnosed with schizophrenia (SZ). This study validates a comics strip paradigm for ToM and empathy assessment, the Montreal Affective Voices (MAV) for measuring emotion recognition, and a Go-NoGo task for inhibition control estimation in individuals diagnosed with SZ, comparing their performance with healthy controls. SZ participants exhibited diminished abilities in the comics strip task, especially in ToM and empathy conditions, alongside challenges in identifying emotions from vocal cues in MAV. They responded slower and tended to be less accurate in the Go-NoGo task. The validated behavioral battery addresses the limitations of previous measures and emerges as a promising tool for future investigations into the neural systems underlying social cognition in schizophrenia. Such insights can lead to the development of long-needed treatment for negative symptoms and social dysfunctions in schizophrenia.

A Louder Call for the Integration of Multiple Nonverbal Channels in the Study of Affect.

Affective science has increasingly sought to represent emotional experiences multimodally, measuring affect through a combination of self-report ratings, linguistic output, physiological measures, and/or nonverbal expressions. However, despite widespread recognition that non-facial nonverbal cues are an important facet of expressive behavior, measures of nonverbal expressions commonly focus solely on facial movements. This Commentary represents a call for affective scientists to integrate a larger range of nonverbal cues-including gestures, postures, and vocal cues-alongside facial cues in efforts to represent the experience of emotion and its communication. Using the measurement and analysis of vocal cues as an illustrative case, the Commentary considers challenges, potential solutions, and the theoretical and translational significance of working to integrate multiple nonverbal channels in the study of affect.

Reconfigurable Robotic Exercising Companion

Regular exercise plays a crucial role in promoting overall well-being in today’s lifestyle. However, individuals often find it challenging to properly execute exercises, including maintaining correct postures and appropriate movement speeds. Robotic companions have emerged as potential solutions to assist and motivate users during exercise sessions. This research paper proposes a novel robot companion designed for exercise scenarios using a reconfigurable robot. In contrast to existing non-reconfigurable robotic companions, the use of a reconfigurable robot provides added flexibility in generating emotions. The system incorporates a module that utilizes fuzzy logic to evaluate the correctness of exercise performance based on posture variations and movement speeds. The robot generates emotions and provides feedback to users based on the exercise correctness score. The robot expresses emotions through reconfigurations, motion patterns, and variations in robot speed. This emotion-based feedback could be helpful for creating engaging and interactive exercise experiences. Apart from emotion generation, the robot utilizes vocal cues as feedback. Experimental results validate the effectiveness of the proposed system in evaluating exercise correctness and demonstrating meaningful emotion transitions. The findings of this work contribute to the development of innovative robotic companions for improving exercise adherence and overall well-being.

Do sheep (Ovis aries) discriminate human emotional odors?

While sheep can detect and discriminate human emotions through visual and vocal cues, their reaction to human body odors remains unknown. The present study aimed to determine whether sheep (Ovis aries) can detect human odors, olfactorily discriminate stressed from non-stressed individuals, and behave accordingly based on the emotional valence of the odors. Axillary secretions from 34 students were collected following an oral examination (stress odor) or a regular class (non-stress odor). Fourteen female and 15 male lambs were then exposed to these odors through a habituation-dishabituation procedure. The habituation stimulus was presented four times for one minute, followed by the dishabituation stimulus presented once for one minute. Behavioral variables included spatiality relative to target odors, approach/withdrawal, ear positioning, sniffing, ingestion, and vocalization. Both female and male lambs more often positioned their ears backwards/forwards, and asymmetrically when exposed to the dishabituation stimulus, but regardless of their stress or non-stress value. They also changed their approach behavior when exposed to the dishabituation stimuli. Lambs displayed some behavioral signs of discrimination between the habituation and dishabituation odors, but regardless of their relation to stress or non-stress of human donors. In sum, this exploratory study suggests that young sheep respond negatively to the odor of unfamiliar humans, without showing any specific emotional contagion related to the stress odor. This exploratory study suggests young ovines can detect human body odor, a further step toward understanding the human-sheep relationship.

Fast and friendly: The role of vocal cues in adolescents' responses to and perceptions of peer provocation.

Adolescents self-report using different strategies to respond to peer provocation. However, we have a limited understanding of how these responses are behaviorally enacted and perceived by peers. This study examined the extent to which adolescents' self-reported responses to peer provocation (i.e., aggressive, assertive, and withdrawn) predicted how their vocal enactments of standardized responses to peer provocation were perceived by other adolescents. Three vocal cues relevant to the communication of emotional intent-average pitch, average intensity, and speech rate-were explored as moderators of these associations. Adolescent speakers (n = 39; Mage = 12.67; 66.7% girls) completed a self-report measure of how they would choose to respond to scenarios involving peer provocation; they also enacted standardized vocal responses to hypothetical peer provocation scenarios. Recordings of speakers' vocal responses were presented to a separate sample of adolescent listeners (n = 129; Mage = 12.12; 52.7% girls) in an online listening task. Speakers who self-reported greater use of assertive response strategies enacted standardized vocal responses that were rated as significantly friendlier by listeners. Vocal responses enacted with faster speech rates were also rated as significantly friendlier by listeners. Speakers' self-reported use of aggression and withdrawal was not significantly related to listeners' ratings of their standardized vocal responses. These findings suggest that adolescents may be perceived differently by their peers depending on the way in which their response is enacted; specifically, faster speech rate may be perceived as friendlier and thus de-escalate peer conflict. Future studies should consider not only what youth say and/or do when responding to peer provocation but also how they say it.

Effects of testosterone on speech production and perception: Linking hormone levels in males to vocal cues and female voice attractiveness ratings

This study sets out to investigate the potential effect of males’ testosterone level on speech production and speech perception. Regarding speech production, we investigate intra- and inter-individual variation in mean fundamental frequency (fo) and formant frequencies and highlight the potential interacting effect of another hormone, i.e. cortisol. In addition, we investigate the influence of different speech materials on the relationship between testosterone and speech production. Regarding speech perception, we investigate the potential effect of individual differences in males’ testosterone level on ratings of attractiveness of female voices.In the production study, data is gathered from 30 healthy adult males ranging from 19 to 27 years (mean age: 22.4, SD: 2.2) who recorded their voices and provided saliva samples at 9 am, 12 noon and 3 pm on a single day. Speech material consists of sustained vowels, counting, read speech and a free description of pictures. Biological measures comprise speakers’ height, grip strength, and hormone levels (testosterone and cortisol). In the perception study, participants were asked to rate the attractiveness of female voice stimuli (sentence stimulus, same-speaker pairs) that were manipulated in three steps regarding mean fo and formant frequencies.Regarding speech production, our results show that testosterone affected mean fo (but not formants) both within and between speakers. This relationship was weakened in speakers with high cortisol levels and depended on the speech material. Regarding speech perception, we found female stimuli with higher mean fo and formants to be rated as sounding more attractive than stimuli with lower mean fo and formants. Moreover, listeners with low testosterone showed an increased sensitivity to vocal cues of female attractiveness.While our results of the production study support earlier findings of a relationship between testosterone and mean fo in males (which is mediated by cortisol), they also highlight the relevance of the speech material: The effect of testosterone was strongest in sustained vowels, potentially due to a strengthened effect of hormones on physiologically strongly influenced tasks such as sustained vowels in contrast to more free speech tasks such as a picture description. The perception study is the first to show an effect of males’ testosterone level on female attractiveness ratings using voice stimuli.

Can perpetrators discern survivors from voice?

ABSTRACT Background: Research has shown that potential perpetrators and individuals high in psychopathic traits tend to body language cues to target a potential new victim. However, whether targeting occurs also by tending to vocal cues has not been examined. Thus, the role of voice in interpersonal violence merits investigation. Objective: In two studies, we examined whether perpetrators could differentiate female speakers with and without sexual and physical assault histories (presented as rating the degree of ‘vulnerability’ to victimization). Methods: Two samples of male listeners (sample one N = 105, sample two, N = 109) participated. Each sample rated 18 voices (9 survivors and 9 controls). Listener sample one heard spontaneous speech, and listener sample two heard the second sentence of a standardized passage. Listeners’ self-reported psychopathic traits and history of previous perpetration were measured. Results: Across both samples, history of perpetration (but not psychopathy) predicted accuracy in distinguishing survivors of assault. Conclusions: These findings highlight the potential role of voice in prevention and intervention. Gaining a further understanding of what voice cues are associated with accuracy in discerning survivors can also help us understand whether or not specialized voice training could have a role in self-defense practices.

Redistribution of songbirds within a migratory stopover site as a response to sylviid warbler song playback

Assessing habitat quality is critical for migrating birds, with implications for fuel loading, predation risk and timing of departure. To efficiently distribute within stopover sites, migrants rely on various cues that facilitate habitat quality assessment. In this study, we conducted a playback experiment at a mid‐route stopover site to test the effects of vocal cues, specifically bird songs, on habitat redistribution during spring migration. To estimate the effect of bird songs on bird abundance we mist‐netted and ringed birds for 60 days on alternating playback treatment (playing songs of three sylviid warbler species) and no playback (control) days. A nearby site was used to control for fluctuations caused by migration waves. Overall, total bird abundance was significantly increased by the playback treatment. This increase resulted from the attraction of conspecifics (species whose songs were played) rather than of other species – sylviids or others. Correspondingly, species richness was seemingly unaffected. To investigate the relationship between internal‐state factors and the attraction to playback in the most abundantly attracted species, Eurasian Blackcap Sylvia atricapilla, we tested associations with age, sex and body condition index but found no significant effects, possibly because of the limited sample size. Our findings challenge established ideas regarding heterospecific attraction and contradict previous studies. Attraction to bird song is discussed in the context of geographical location, timing and internal drivers. Vocal cues, specifically bird songs, may affect micro‐habitat selection by migrating songbirds during stopovers. Consequently, we call for consideration of the influence of the acoustic environment on birds during migratory stopover in future studies, habitat management and conservation efforts.

Seasonal and diurnal marine mammal presence in the proposed marine protected area of Southampton Island, Hudson Bay, Nunavut as revealed by passive acoustic monitoring

Marine protected areas contribute to mitigating the effects of human activities on marine ecosystems. The waters of Southampton Island, Nunavut, Canada are under consideration to become a marine protected area, but baseline information of marine mammal presence and habitat use is lacking. This study represents the first passive acoustic monitoring of marine mammals in this area, with data collected in 2018 and 2019. Bioacoustics analyses and generalized linear models were used to investigate species seasonal and temporal vocalization trends; infer behavior and habitat use from vocal cues; and explore vocal activity correlation with sea ice. Over a diel period, bearded seals ( Erignathus barbatus (Erxleben, 1777)) and walruses ( Odobenus rosmarus (Linnaeus, 1758)) were found to increase their vocal activity at night, whereas belugas ( Delphinapterus leucas (Pallas, 1776)) mainly vocalized during daytime. Seasonally, bearded seals were recorded only during their breeding season at sea ice break-up, while walruses vocalized consistently throughout the open-water period outside their breeding season. Beluga vocal trends suggest their use of the area as a migratory corridor during sea ice break-up and freeze-up. Finally, bowhead whales ( Balaena mysticetus Linnaeus, 1758) were recorded later than expected, concurrently with the onset of their mating vocal displays during the fall migration at sea ice freeze-up.

Do vocal cues matter in information disclosure? Evidence from IPO online roadshows in the SSE STAR market

In this study, we investigate whether vocal cues expressed by management in IPO online roadshows in the SSE STAR Market play an information role. Using deep learning techniques, we classify the emotional tendencies of vocal cues in IPO online roadshows and construct quantitative proxy variables. After a series of empirical analyses, we find that vocal emotions are positively associated with first-day stock returns after IPO. In addition, we find that vocal emotions have predictive power for future operating performance after IPO. Our findings still hold after a series of robustness checks. Through additional analyses, we provide evidence that information content in the prospectus and investor sentiment play moderating and mediating roles, respectively, in the relationship between vocal emotions and the first-day stock returns after IPO. Moreover, we find that the vocal emotions expressed in the presentation section and the management tone in the Q&A section develop synergistic effects. Finally, we find that the information role played by vocal emotions exhibits timeliness. Our findings elucidate the information role of vocal cues and highlight the importance of IPO online roadshows in the SSE STAR Market.

Decoding emotions: Exploring the validity of sentiment analysis in psychotherapy

Objective Given the importance of emotions in psychotherapy, valid measures are essential for research and practice. As emotions are expressed at different levels, multimodal measurements are needed for a nuanced assessment. Natural Language Processing (NLP) could augment the measurement of emotions. The study explores the validity of sentiment analysis in psychotherapy transcripts. Method We used a transformer-based NLP algorithm to analyze sentiments in 85 transcripts from 35 patients. Construct and criterion validity were evaluated using self- and therapist reports and process and outcome measures via correlational, multitrait-multimethod, and multilevel analyses. Results The results provide indications in support of the sentiments’ validity. For example, sentiments were significantly related to self- and therapist reports of emotions in the same session. Sentiments correlated significantly with in-session processes (e.g., coping experiences), and an increase in positive sentiments throughout therapy predicted better outcomes after treatment termination. Discussion Sentiment analysis could serve as a valid approach to assessing the emotional tone of psychotherapy sessions and may contribute to the multimodal measurement of emotions. Future research could combine sentiment analysis with automatic emotion recognition in facial expressions and vocal cues via the Nonverbal Behavior Analyzer (NOVA). Limitations (e.g., exploratory study with numerous tests) and opportunities are discussed.

Cerebral response to emotional working memory based on vocal cues: an fNIRS study.

Humans mainly utilize visual and auditory information as a cue to infer others' emotions. Previous neuroimaging studies have shown the neural basis of memory processing based on facial expression, but few studies have examined it based on vocal cues. Thus, we aimed to investigate brain regions associated with emotional judgment based on vocal cues using an N-back task paradigm. Thirty participants performed N-back tasks requiring them to judge emotion or gender from voices that contained both emotion and gender information. During these tasks, cerebral hemodynamic response was measured using functional near-infrared spectroscopy (fNIRS). The results revealed that during the Emotion 2-back task there was significant activation in the frontal area, including the right precentral and inferior frontal gyri, possibly reflecting the function of an attentional network with auditory top-down processing. In addition, there was significant activation in the ventrolateral prefrontal cortex, which is known to be a major part of the working memory center. These results suggest that, compared to judging the gender of voice stimuli, when judging emotional information, attention is directed more deeply and demands for higher-order cognition, including working memory, are greater. We have revealed for the first time the specific neural basis for emotional judgments based on vocal cues compared to that for gender judgments based on vocal cues.

Human perception of cats' communicative cues: human‐cat communication goes multimodal

As a central key to apprehend the human-cat relationship, the question of how humans decode the behaviours expressed by their feline companions has drawn scientists’ attention in the past decades. To this point, previous studies have not investigated humans’ understanding of cats’ communicative cues in a multimodal perspective. While communication generally implies multiple channels, the influence of signal modality as relates to human-cat communication, is still poorly understood. Therefore, the primary aim of the present study was to investigate whether people can identify cats’ communicative information using unimodal (visual or vocal) and bimodal (visual and vocal) signals. Participants (n=630) were recruited via online advertisement on social media. Each participant viewed 24 carefully operationalised video clips of cats in different emotional/behavioural conditions. Four categories were included: contentment, discontentment, solicitation (food or attention) and predatory behaviour. Clips were presented as vocal only, visual only, or bimodal (visual and vocal cues). Video clips showing a bimodal expression were identified with the highest score (91,8% of correct ratings) compared with the visual only (87,3%) and the vocal only (72,2%). All modalities considered, contentment returned the highest identification score (90,1% correct), followed by solicitation (87,2%), then predatory behaviour (86,3%). Discontentment was the most difficult behaviour to be correctly identified (71,6%). Finally, for all behavioural categories and modalities, professionals working with animals returned a higher score than lay people (86.2% vs 82.9%). Taken together, our data underline the influence of the signal modality on interspecific communication between cats and humans. Information emitted by cats as a bimodal signal (visual and vocal) is better understood by humans than visual signals. The most difficult to decipher for humans are vocal signals emitted alone. A better understanding of humans’ abilities to understand their feline counterparts, could potentially help pet owners, and animal care practitioners to optimise cat care and welfare.

Blindness influences emotional authenticity perception in voices: Behavioral and ERP evidence

The ability to distinguish spontaneous from volitional emotional expressions is an important social skill. How do blind individuals perceive emotional authenticity? Unlike sighted individuals, they cannot rely on facial and body language cues, relying instead on vocal cues alone. Here, we combined behavioral and ERP measures to investigate authenticity perception in laughter and crying in individuals with early- or late-blindness onset. Early-blind, late-blind, and sighted control participants (n = 17 per group, N = 51) completed authenticity and emotion discrimination tasks while EEG data were recorded. The stimuli consisted of laughs and cries that were either spontaneous or volitional. The ERP analysis focused on the N1, P2, and late positive potential (LPP). Behaviorally, early-blind participants showed intact authenticity perception, but late-blind participants performed worse than controls. There were no group differences in the emotion discrimination task. In brain responses, all groups were sensitive to laughter authenticity at the P2 stage, and to crying authenticity at the early LPP stage. Nevertheless, only early-blind participants were sensitive to crying authenticity at the N1 and middle LPP stages, and to laughter authenticity at the early LPP stage. Furthermore, early-blind and sighted participants were more sensitive than late-blind ones to crying authenticity at the P2 and late LPP stages.Altogether, these findings suggest that early blindness relates to facilitated brain processing of authenticity in voices, both at early sensory and late cognitive-evaluative stages. Late-onset blindness, in contrast, relates to decreased sensitivity to authenticity at behavioral and brain levels.

The irrepressible influence of vocal stereotypes on trust.

There is a reciprocal relationship between trust and vocal communication in human interactions. On one hand, a predisposition towards trust is necessary for communication to be meaningful and effective. On the other hand, we use vocal cues to signal our own trustworthiness and to infer it from the speech of others. Research on trustworthiness attributions to vocal characteristics is scarce and contradictory, however, being typically based on explicit judgements which may not predict actual trust-oriented behaviour. We use a game theory paradigm to examine the influence of speaker accent and prosody on trusting behaviour towards a simulated game partner, who responds either trustworthily or untrustworthily in an investment game. We found that speaking in a non-regional standard accent increases trust, as does relatively slow articulation rate. The effect of accent persists over time, despite the accumulation of clear evidence regarding the speaker's level of trustworthiness in a negotiated interaction. Accents perceived as positive for trust can maintain this benefit even in the face of behavioural evidence of untrustworthiness.

Vocal Cues to Assess Arousal State of Bottlenose Dolphins (Tursiops spp.) Involved in Public Presentations

Emotions in animals may be expressed by arousal and understanding this often relies upon the monitoring of their behaviour. Under human care, animals’ arousal states may be linked to husbandry decisions, whereby animals may display arousal responses to scheduled events such as feeding and human interaction. Here, we investigate vocal correlates of arousal associated with public presentations of bottlenose dolphins (Tursiops spp.) in human care by comparing vocal production rates and characteristics between high and low arousal contexts. Elevated arousal during the day compared with overnight was characterised by increased signature and non-signature whistle production. High intensity broadband crack vocalisations were produced less than whistles during the day and did not correlate with increased arousal around presentation times. Three of ten dolphins increased signature whistle production before and/or after presentation sessions, indicating elevated arousal and variation in individual responses. Many individuals elevated minimum frequency and suppressed maximum frequency of signature whistles in a way that correlated with higher arousal contexts, indicating that these may therefore be good indicators of changes in arousal state. Overall, our study demonstrates that passive acoustic monitoring can provide a useful indication of arousal linked to husbandry decisions, and that individual variation in vocal responses, likely linked to personality, is important to consider.

How neurotypical listeners recognize emotions expressed through vocal cues by speakers with high-functioning autism.

We conducted an investigation to explore how neurotypical (NT) listeners perceive the emotional tone of voice in sentences spoken by individuals with high-functioning autism spectrum disorders (ASD) and NT speakers. The investigation included both male and female speakers from both groups. In Study 1, NT listeners were asked to identify the emotional prosody (anger, fear, happiness, surprise or neutral) conveyed by the speakers. Results revealed that emotional expressions produced by male ASD speakers were generally less accurately recognized compared to male NT speakers. In contrast, emotions expressed by female ASD speakers were more accurately categorized compared to female NT speakers, except when expressing fear. This suggests that female ASD speakers may not express emotional prosody in the same way as their male counterparts. In Study 2, a subset of produced materials was rated for valence, voice modulation, and voice control to supplement Study 1 results: Female ASD speakers sounded less negative when expressing fear compared to female NT speakers. Male ASD speakers were perceived as less positive than NT speakers when expressing happiness. Voice modulation also differed between groups, showing a tendency for ASD speakers to follow different display rules for both positive emotions (happiness and surprise) tested. Finally, male ASD speakers were rated to use voice cues less appropriately compared to NT male speakers, an effect less pronounced for female ASD speakers. Together, the results imply that difficulties in social interactions among individuals with high-functioning ASD could be due to non-prototypical voice use of male ASD speakers and emphasize that female individuals do not show the same effects.

Evidence for vocal signatures and voice-prints in a wild parrot.

In humans, identity is partly encoded in a voice-print that is carried across multiple vocalizations. Other species also signal vocal identity in calls, such as shown in the contact call of parrots. However, it remains unclear to what extent other call types in parrots are individually distinct, and whether there is an analogous voice-print across calls. Here we test if an individual signature is present in other call types, how stable this signature is, and if parrots exhibit voice-prints across call types. We recorded 5599 vocalizations from 229 individually marked monk parakeets (Myiopsitta monachus) over a 2-year period in Barcelona, Spain. We examined five distinct call types, finding evidence for an individual signature in three. We further show that in the contact call, while birds are individually distinct, the calls are more variable than previously assumed, changing over short time scales (seconds to minutes). Finally, we provide evidence for voice-prints across multiple call types, with a discriminant function being able to predict caller identity across call types. This suggests that monk parakeets may be able to use vocal cues to recognize conspecifics, even across vocalization types and without necessarily needing active vocal signatures of identity.