Nonverbal Social Signals Research Articles

The influence of social signals on the self-experience of pain: A neuroimaging review.

Researchers in cognitive neuroscience have investigated extensively how psychological factors shape the processing and perception of pain using behavioral, physiological, and neuroimaging methods. However, social influences of pain, an essential part of biopsychosocial pain models, have received relatively little attention. This is particularly true for the neurobiological mechanisms underlying social modulations on pain. Therefore, this review discusses the findings of recent neuroimaging studies measuring the effects of social manipulations on pain perception (e.g., verbal and non-verbal social signals, social interaction style, conformity, social support, and sociocultural mediators). Finally, a schematic summary of the different social modulatory themes is presented.

Correlates of individual voice and face preferential responses during resting state

Human nonverbal social signals are transmitted to a large extent by vocal and facial cues. The prominent importance of these cues is reflected in specialized cerebral regions which preferentially respond to these stimuli, e.g. the temporal voice area (TVA) for human voices and the fusiform face area (FFA) for human faces. But it remained up to date unknown whether there are respective specializations during resting state, i.e. in the absence of any cues, and if so, whether these representations share neural substrates across sensory modalities. In the present study, resting state functional connectivity (RSFC) as well as voice- and face-preferential activations were analysed from functional magnetic resonance imaging (fMRI) data sets of 60 healthy individuals. Data analysis comprised seed-based analyses using the TVA and FFA as regions of interest (ROIs) as well as multi voxel pattern analyses (MVPA). Using the face- and voice-preferential responses of the FFA and TVA as regressors, we identified several correlating clusters during resting state spread across frontal, temporal, parietal and occipital regions. Using these regions as seeds, characteristic and distinct network patterns were apparent with a predominantly convergent pattern for the bilateral TVAs whereas a largely divergent pattern was observed for the bilateral FFAs. One region in the anterior medial frontal cortex displayed a maximum of supramodal convergence of informative connectivity patterns reflecting voice- and face-preferential responses of both TVAs and the right FFA, pointing to shared neural resources in supramodal voice and face processing. The association of individual voice- and face-preferential neural activity with resting state connectivity patterns may support the perspective of a network function of the brain beyond an activation of specialized regions.

Taking a Computational Cultural Neuroscience Approach to Study Parent-Child Similarities in Diverse Cultural Contexts.

Parent-child similarities and discrepancies at multiple levels provide a window to understand the cultural transmission process. Although prior research has examined parent-child similarities at the belief, behavioral, and physiological levels across cultures, little is known about parent-child similarities at the neural level. The current review introduces an interdisciplinary computational cultural neuroscience approach, which utilizes computational methods to understand neural and psychological processes being involved during parent-child interactions at intra- and inter-personal level. This review provides three examples, including the application of intersubject representational similarity analysis to analyze naturalistic neuroimaging data, the usage of computer vision to capture non-verbal social signals during parent-child interactions, and unraveling the psychological complexities involved during real-time parent-child interactions based on their simultaneous recorded brain response patterns. We hope that this computational cultural neuroscience approach can provide researchers an alternative way to examine parent-child similarities and discrepancies across different cultural contexts and gain a better understanding of cultural transmission processes.

Laughter as language

Understanding the import of laughter, has interested philosophers and literary scholars for millennia and, more recently, psychologists, biologists, neuroscientists, and linguists. However, the assumption has been that laughter lacks meaning akin to what words and phrases possess and that it does not contribute to the compositional construction of meaning. In this paper, we argue that, in fact, laughter (and other non-verbal social signals like smiling, sighing, frowning) has propositional content—it involves reference to external real world events, has stand alone meanings, and participates in semantic and pragmatic processes like repair, implicature, and irony. We show how to develop a formal semantic and pragmatic account of laughter embedded in a general theory of conversational interaction and emotional reasoning and show how to explain the wide, indeed in principle unbounded range of uses laughter exhibits. We show how our account can be extended to other non-verbal social signals like smiling, sighing, eye rolling, and frowning. Should laughter and its ilk be incorporated in the grammar? We suggest that they probably should be, if one assumes a conversationally–oriented view of grammar. But various open issues remain.

Pre-exposure to Ambiguous Faces Modulates Top-Down Control of Attentional Orienting to Counterpredictive Gaze Cues.

Understanding and reacting to others’ nonverbal social signals, such as changes in gaze direction (i.e., gaze cue), are essential for social interactions, as it is important for processes such as joint attention and mentalizing. Although attentional orienting in response to gaze cues has a strong reflexive component, accumulating evidence shows that it can be top-down controlled by context information regarding the signals’ social relevance. For example, when a gazer is believed to be an entity “with a mind” (i.e., mind perception), people exert more top-down control on attention orienting. Although increasing an agent’s physical human-likeness can enhance mind perception, it could have negative consequences on top-down control of social attention when a gazer’s physical appearance is categorically ambiguous (i.e., difficult to categorize as human or nonhuman), as resolving this ambiguity would require using cognitive resources that otherwise could be used to top-down control attention orienting. To examine this question, we used mouse-tracking to explore if categorically ambiguous agents are associated with increased processing costs (Experiment 1), whether categorically ambiguous stimuli negatively impact top-down control of social attention (Experiment 2), and if resolving the conflict related to the agent’s categorical ambiguity (using exposure) would restore top-down control to orient attention (Experiment 3). The findings suggest that categorically ambiguous stimuli are associated with cognitive conflict, which negatively impact the ability to exert top-down control on attentional orienting in a counterpredicitive gaze-cueing paradigm; this negative impact, however, is attenuated when being pre-exposed to the stimuli prior to the gaze-cueing task. Taken together, these findings suggest that manipulating physical human-likeness is a powerful way to affect mind perception in human-robot interaction (HRI) but has a diminishing returns effect on social attention when it is categorically ambiguous due to drainage of cognitive resources and impairment of top-down control.

Aggression modulates neural correlates of hostile intention attribution to laughter in children

The tendency to interpret nonverbal social signals as hostile in intention is associated with aggressive responding, poor social functioning and mental illness, and can already be observed in childhood. To investigate the neural correlates of such hostile attributions of social intention, we performed a functional magnetic imaging study in 10–18 year old children and adolescents. Fifty healthy participants rated videos of laughter, which they were told to imagine as being directed towards them, as friendly versus hostile in social intention. Hostile intention ratings were associated with neural response in the right temporal voice area (TVA). Moreover, self-reported trait physical aggression modulated this relationship in both the right TVA and bilateral lingual gyrus, with stronger associations between hostile intention ratings and neural activation in children with higher trait physical aggression scores. Functional connectivity results showed decreased connectivity between the right TVA and left dorsolateral prefrontal cortex with increasing trait physical aggression for making hostile social intention attributions. We conclude that children's social intention attributions are more strongly related to activation of early face and voice-processing regions with increasing trait physical aggression.

In the presence of conflicting gaze cues, fearful expression and eye-size guide attention

ABSTRACTHumans are social beings that often interact in multi-individual environments. As such, we are frequently confronted with nonverbal social signals, including eye-gaze direction, from multiple individuals. Yet, the factors that allow for the prioritisation of certain gaze cues over others are poorly understood. Using a modified conflicting gaze paradigm, we tested the hypothesis that fearful gaze would be favoured amongst competing gaze cues. We further hypothesised that this effect is related to the increased sclera exposure, which is characteristic of fearful expressions. Across three experiments, we found that fearful, but not happy, gaze guides observers’ attention over competing non-emotional gaze. The guidance of attention by fearful gaze appears to be linked to increased sclera exposure. However, differences in sclera exposure do not prioritise competing gazes of other types. Thus, fearful gaze guides attention among competing cues and this effect is facilitated by increased sclera exposure – but increased sclera exposure per se does not guide attention. The prioritisation of fearful gaze over non-emotional gaze likely represents an adaptive means of selectively attending to survival-relevant spatial locations.

Social influence on metacognitive evaluations: The power of nonverbal cues.

Metacognitive evaluations refer to the processes by which people assess their own cognitive operations with respect to their current goal. Little is known about whether this process is susceptible to social influence. Here we investigate whether nonverbal social signals spontaneously influence metacognitive evaluations. Participants performed a two-alternative forced-choice task, which was followed by a face randomly gazing towards or away from the response chosen by the participant. Participants then provided a metacognitive evaluation of their response by rating their confidence in their answer. In Experiment 1, the participants were told that the gaze direction was irrelevant to the task purpose and were advised to ignore it. The results revealed an effect of implicit social information on confidence ratings even though the gaze direction was random and therefore unreliable for task purposes. In addition, nonsocial cues (car) did not elicit this effect. In Experiment 2, the participants were led to believe that cue direction (face or car) reflected a previous participant's response to the same question-that is, the social information provided by the cue was made explicit, yet still objectively unreliable for the task. The results showed a similar social influence on confidence ratings, observed with both cues (car and face) but with an increased magnitude relative to Experiment 1. We additionally showed in Experiment 2 that social information impaired metacognitive accuracy. Together our results strongly suggest an involuntary susceptibility of metacognitive evaluations to nonverbal social information, even when it is implicit (Experiment 1) and unreliable (Experiments 1 and 2).

Systematic analysis of video data from different human-robot interaction studies: a categorization of social signals during error situations.

Human–robot interactions are often affected by error situations that are caused by either the robot or the human. Therefore, robots would profit from the ability to recognize when error situations occur. We investigated the verbal and non-verbal social signals that humans show when error situations occur in human–robot interaction experiments. For that, we analyzed 201 videos of five human–robot interaction user studies with varying tasks from four independent projects. The analysis shows that there are two types of error situations: social norm violations and technical failures. Social norm violations are situations in which the robot does not adhere to the underlying social script of the interaction. Technical failures are caused by technical shortcomings of the robot. The results of the video analysis show that the study participants use many head movements and very few gestures, but they often smile, when in an error situation with the robot. Another result is that the participants sometimes stop moving at the beginning of error situations. We also found that the participants talked more in the case of social norm violations and less during technical failures. Finally, the participants use fewer non-verbal social signals (for example smiling, nodding, and head shaking), when they are interacting with the robot alone and no experimenter or other human is present. The results suggest that participants do not see the robot as a social interaction partner with comparable communication skills. Our findings have implications for builders and evaluators of human–robot interaction systems. The builders need to consider including modules for recognition and classification of head movements to the robot input channels. The evaluators need to make sure that the presence of an experimenter does not skew the results of their user studies.

Context-Aware Automated Analysis and Annotation of Social Human--Agent Interactions

The outcome of interpersonal interactions depends not only on the contents that we communicate verbally, but also on nonverbal social signals. Because a lack of social skills is a common problem for a significant number of people, serious games and other training environments have recently become the focus of research. In this work, we present NovA ( No n v erbal behavior A nalyzer), a system that analyzes and facilitates the interpretation of social signals automatically in a bidirectional interaction with a conversational agent. It records data of interactions, detects relevant social cues, and creates descriptive statistics for the recorded data with respect to the agent's behavior and the context of the situation. This enhances the possibilities for researchers to automatically label corpora of human--agent interactions and to give users feedback on strengths and weaknesses of their social behavior.

NOCOA+: Multimodal Computer-Based Training for Social and Communication Skills

SUMMARY Non-verbal communication incorporating visual, audio, and contextual information is important to make sense of and navigate the social world. Individuals who have trouble with social situations often have difficulty recognizing these sorts of non-verbal social signals. In this article, we propose a training tool NOCOA+ (Non-verbal COmmuniation for Autism plus) that uses utterances in visual and audio modalities in nonverbal communication training. We describe the design of NOCOA+ ,a nd further perform an experimental evaluation in which we examine its potential as a tool for computer-based training of non-verbal communication skills for people with social and communication difficulties. In a series of four experiments, we investigated 1) the effect of temporal context on the ability to recognize social signals in testing context, 2) the effect of modality of presentation of social stimulus on ability to recognize non-verbal information, 3) the correlation between autistic traits as measured by the autism spectrum quotient (AQ) and non-verbal behavior recognition skills measured by NOCOA+ ,4 ) the effectiveness of computer-based training in improving social skills. We found that context information was helpful for recognizing non-verbal behaviors, and the effect of modality was different. The results also showed a significant relationship between the AQ communication and socialization scores and non-verbal communication skills, and that social skills were significantly improved through computer-based training.

Using non-verbal social signals and degree centrality to optimise a covert actor's detection scheme for a healthy networked community

Passive attacks are of the nature of eavesdropping on, monitoring of, transmissions where the attacker’s goal is to unnoticeably obtain the information in transmission. Passive attacks are difficult to detect and therefore hard to prevent. The main focus for most research in this area has been on preventing the attacks rather than detecting the attacker for example by data encryption. But encryption by itself falls short because passive attack can occur in more ways than just observing exposed data. Encryption is also not always applicable for example in open wireless communication protocols. With these observations, we aim to design a scheme that reduces such attackers’ capability. We introduce the non-verbal social signal deception detection in the investigative module while we optimise the scheme by putting in consideration a participant’s centrality given the topology. The results are an intuitive scalable and optimised scheme that detects, investigates, and expels the guilty suspects.

On the detection of the level of attention in an orchestra through head movements

Results from a study of non-verbal social signals in an orchestra are presented. Music is chosen as an example of interactive and social activity, where non-verbal communication plays a fundamental role. The orchestra is adopted as a social group with a clear leader (the conductor) of two groups of musicians (the first and second violin sections). It is shown how a reduced set of simple movement features – head movements – can be used to measure the levels of attention of the musicians with respect to the conductor and the music stand under various conditions (different conductors/pieces/sections of the same piece).

How do shared-representations and emotional processes cooperate in response to social threat signals?

Research in social cognition has mainly focused on the detection and comprehension of others’ mental and emotional states. Doing so, past studies have adopted a “contemplative” view of the role of the observer engaged in a social interaction. However, the adaptive problem posed by the social environment is first and foremost that of coordination, which demands more of social cognition beyond mere detection and comprehension of others’ hidden states. Offering a theoretical framework that takes into account the dynamical aspect of social interaction – notably by accounting for constant interplay between emotional appraisal and motor processes in socially engaged human brain – thus constitutes an important challenge for the field of social cognition. Here, we propose that our social environment can be seen as presenting opportunities for actions regarding others. Within such a framework, non-verbal social signals such as emotional displays are considered to have evolved to influence the observer in consistent ways. Consequently, social signals can modulate motor responses in observers. In line with this theoretical framework we provide evidence that emotional and motor processes are actually tightly linked during the perception of threat signals. This is ultimately reflected in the human brain by constant interplay between limbic and motor areas.

A Polymorphism in the Serotonin Transporter Gene Moderates Cardiovascular Reactivity to Psychosocial Stress

To examine whether a polymorphism (5-HTTLPR: serotonin transporter linked polymorphic region) in the promoter of the serotonin transporter gene (SLC6A4) moderates cardiovascular reactivity to social threat. Psychologically healthy young adults delivered a speech and performed mental arithmetic in one of three conditions: a) an evaluative audience condition that gave disapproving and negative nonverbal social signals (n = 59); b) an evaluative audience condition that provided supportive social signals (n = 60); or c) a no audience condition (n = 65). Heart rate (HR) and systolic and diastolic blood pressures (DBP) were measured before, during, and after the stress tasks to assess cardiovascular reactivity and recovery. In the negative audience condition, there was a significant association between the 5-HTTLPR and systolic blood pressure, DBP, and HR reactivity. Individuals with the short/short genotype showed the greatest reactivity. The DBP and HR reactivity of short/short individuals in the negative audience condition was also greater than that of individuals with the short/short genotype in the no audience condition. These associations of the 5-HTLPR with HR reactivity were moderated by gender, being limited to females. With respect to cardiovascular recovery, short/short individuals in the negative audience condition exhibited impaired DBP recovery relative to other genotypes in the same condition, as well as short/short individuals in the no audience condition. The 5-HTTLPR moderates cardiovascular reactivity to stress in a threatening evaluative social context, which suggests that the serotonin system may be involved in the processes by which stressful, conflict-ridden social environments affect risk for cardiovascular-related health outcomes.

Modelling and Analysing Creative Communication within Groups of People: The Artistic Event at FET11

Theatre stage and artistic performances aim at joining participants to act together, to share, shape, and co-create cultural content by means of active experiences. Future networked and social media technologies promote new ways of experiencing cultural events and artefacts. Nonverbal social signals are major components of future social media characterised by embodiment and physical engagement of users. This contribution introduces research results from the EU-ICT-FET SIEMPRE Project, presenting how these results were shown in the final performance at FET11.

Bilateral amygdala damage impairs the acquisition and use of common ground in social interaction.

The development of "common ground," or mutual knowledge of shared information, is believed to require the ability to update a mental representation of another person's thoughts and knowledge based on verbal information and nonverbal social and emotional signals, to facilitate economical communication. As in other forms of everyday social communication, the development of common ground likely requires the orchestration of multiple cognitive processes supported by various neural systems. Here, we investigate the contribution of the amygdala to these processes. SM, a patient with complete, focal, bilateral amygdala damage, and deficits in social and emotional processing, and five healthy comparison participants, each interacted with a familiar partner. We investigated the participants' ability to develop and use referential labels across 24 dynamic, collaborative interactions. Participants verbally directed their partner how to arrange a set of 12 abstract tangrams while separated by a low barrier, allowing them to see each other but hiding their tangrams. In contrast to comparison participants, SM exhibited an impaired rate of learning across trials and did not show the typical simplification in the labels generated during the interactions. Detailed analyses of SM's interactional discourse and social behavior suggested that she has impaired perspective-taking or what can be interpreted as deficient "theory of mind," manifested in abnormal "language-in-use." These results support the conclusion that the amygdala, a structure critical for social and emotional processing, plays an important role in the acquisition and use of common ground and in social communication more broadly.

Association of trait emotional intelligence and individual fMRI‐activation patterns during the perception of social signals from voice and face

Multimodal integration of nonverbal social signals is essential for successful social interaction. Previous studies have implicated the posterior superior temporal sulcus (pSTS) in the perception of social signals such as nonverbal emotional signals as well as in social cognitive functions like mentalizing/theory of mind. In the present study, we evaluated the relationships between trait emotional intelligence (EI) and fMRI activation patterns in individual subjects during the multimodal perception of nonverbal emotional signals from voice and face. Trait EI was linked to hemodynamic responses in the right pSTS, an area which also exhibits a distinct sensitivity to human voices and faces. Within all other regions known to subserve the perceptual audiovisual integration of human social signals (i.e., amygdala, fusiform gyrus, thalamus), no such linked responses were observed. This functional difference in the network for the audiovisual perception of human social signals indicates a specific contribution of the pSTS as a possible interface between the perception of social information and social cognition.

Implicit learning of social predictions

Humans exchange a range of nonverbal social signals in every interaction. It is an open question whether people use these signals, consciously or unconsciously, to guide social behavior. This experiment directly tested whether participants could learn to predict another person’s behavior using nonverbal cues in a single interaction, and whether explicit knowledge of the cue–outcome relationship was necessary for successful prediction. Participants played a computerized game of rock–paper–scissors against an avatar they believed was another participant. Sometimes the avatar generated a predictive facial cue before the play. On these trials, participants’ win-frequency increased over time, even if they did not acquire explicit knowledge of the predictive cue. The degree to which participants could predict the avatar (wins on cued trials) related to their self-reported liking of the avatar. These findings demonstrate the importance of implicit associative learning mechanisms in guiding social behavior on a moment-to-moment basis during interaction.

Explicit and implicit neural mechanisms for processing of social information from facial expressions: A functional magnetic resonance imaging study

The processing of changing nonverbal social signals such as facial expressions is poorly understood, and it is unknown if different pathways are activated during effortful (explicit), compared to implicit, processing of facial expressions. Thus we used fMRI to determine which brain areas subserve processing of high-valence expressions and if distinct brain areas are activated when facial expressions are processed explicitly or implicitly. Nine healthy volunteers were scanned (1.5T GE Signa with ANMR, TE/TR 40/3,000 ms) during two similar experiments in which blocks of mixed happy and angry facial expressions ("on" condition) were alternated with blocks of neutral faces (control "off" condition). Experiment 1 examined explicit processing of expressions by requiring subjects to attend to, and judge, facial expression. Experiment 2 examined implicit processing of expressions by requiring subjects to attend to, and judge, facial gender, which was counterbalanced in both experimental conditions. Processing of facial expressions significantly increased regional blood oxygenation level-dependent (BOLD) activity in fusiform and middle temporal gyri, hippocampus, amygdalohippocampal junction, and pulvinar nucleus. Explicit processing evoked significantly more activity in temporal lobe cortex than implicit processing, whereas implicit processing evoked significantly greater activity in amygdala region. Mixed high-valence facial expressions are processed within temporal lobe visual cortex, thalamus, and amygdalohippocampal complex. Also, neural substrates for explicit and implicit processing of facial expressions are dissociable: explicit processing activates temporal lobe cortex, whereas implicit processing activates amygdala region. Our findings confirm a neuroanatomical dissociation between conscious and unconscious processing of emotional information.