Communicative Hand Gestures Research Articles

Talking with Your (Artificial) Hands: Communicative Hand Gestures as an Implicit Measure of Embodiment.

SummaryWhen people talk, they move their hands to enhance meaning. Using accelerometry, we measured whether people spontaneously use their artificial limbs (prostheses) to gesture, and whether this behavior relates to everyday prosthesis use and perceived embodiment. Perhaps surprisingly, one- and two-handed participants did not differ in the number of gestures they produced in gesture-facilitating tasks. However, they did differ in their gesture profile. One-handers performed more, and bigger, gesture movements with their intact hand relative to their prosthesis. Importantly, one-handers who gestured more similarly to their two-handed counterparts also used their prosthesis more in everyday life. Although collectively one-handers only marginally agreed that their prosthesis feels like a body part, one-handers who reported they embody their prosthesis also showed greater prosthesis use for communication and daily function. Our findings provide the first empirical link between everyday prosthesis use habits and perceived embodiment and a novel means for implicitly indexing embodiment.

Energy flows in gesture-speech physics: The respiratory-vocal system and its coupling with hand gestures.

Expressive moments in communicative hand gestures often align with emphatic stress in speech. It has recently been found that acoustic markers of emphatic stress arise naturally during steady-state phonation when upper-limb movements impart physical impulses on the body, most likely affecting acoustics via respiratory activity. In this confirmatory study, participants (N = 29) repeatedly uttered consonant-vowel (/pa/) mono-syllables while moving in particular phase relations with speech, or not moving the upper limbs. This study shows that respiration-related activity is affected by (especially high-impulse) gesturing when vocalizations occur near peaks in physical impulse. This study further shows that gesture-induced moments of bodily impulses increase the amplitude envelope of speech, while not similarly affecting the Fundamental Frequency (F0). Finally, tight relations between respiration-related activity and vocalization were observed, even in the absence of movement, but even more so when upper-limb movement is present. The current findings expand a developing line of research showing that speech is modulated by functional biomechanical linkages between hand gestures and the respiratory system. This identification of gesture-speech biomechanics promises to provide an alternative phylogenetic, ontogenetic, and mechanistic explanatory route of why communicative upper limb movements co-occur with speech in humans.

Gesture's body orientation modulates the N400 for visual sentences primed by gestures.

Body orientation of gesture entails social‐communicative intention, and may thus influence how gestures are perceived and comprehended together with auditory speech during face‐to‐face communication. To date, despite the emergence of neuroscientific literature on the role of body orientation on hand action perception, limited studies have directly investigated the role of body orientation in the interaction between gesture and language. To address this research question, we carried out an electroencephalography (EEG) experiment presenting to participants (n = 21) videos of frontal and lateral communicative hand gestures of 5 s (e.g., raising a hand), followed by visually presented sentences that are either congruent or incongruent with the gesture (e.g., “the mountain is high/low…”). Participants underwent a semantic probe task, judging whether a target word is related or unrelated to the gesture‐sentence event. EEG results suggest that, during the perception phase of handgestures, while both frontal and lateral gestures elicited a power decrease in both the alpha (8–12 Hz) and the beta (16–24 Hz) bands, lateral versus frontal gestures elicited reduced power decrease in the beta band, source‐located to the medial prefrontal cortex. For sentence comprehension, at the critical word whose meaning is congruent/incongruent with the gesture prime, frontal gestures elicited an N400 effect for gesture‐sentence incongruency. More importantly, this incongruency effect was significantly reduced for lateral gestures. These findings suggest that body orientation plays an important role in gesture perception, and that its inferred social‐communicative intention may influence gesture‐language interaction at semantic level.

Speeding up the detection of non-iconic and iconic gestures (SPUDNIG): A toolkit for the automatic detection of hand movements and gestures in video data

In human face-to-face communication, speech is frequently accompanied by visual signals, especially communicative hand gestures. Analyzing these visual signals requires detailed manual annotation of video data, which is often a labor-intensive and time-consuming process. To facilitate this process, we here present SPUDNIG (SPeeding Up the Detection of Non-iconic and Iconic Gestures), a tool to automatize the detection and annotation of hand movements in video data. We provide a detailed description of how SPUDNIG detects hand movement initiation and termination, as well as open-source code and a short tutorial on an easy-to-use graphical user interface (GUI) of our tool. We then provide a proof-of-principle and validation of our method by comparing SPUDNIG’s output to manual annotations of gestures by a human coder. While the tool does not entirely eliminate the need of a human coder (e.g., for false positives detection), our results demonstrate that SPUDNIG can detect both iconic and non-iconic gestures with very high accuracy, and could successfully detect all iconic gestures in our validation dataset. Importantly, SPUDNIG’s output can directly be imported into commonly used annotation tools such as ELAN and ANVIL. We therefore believe that SPUDNIG will be highly relevant for researchers studying multimodal communication due to its annotations significantly accelerating the analysis of large video corpora.

Tracing the time course of emotion perception: the impact of stimulus physics and semantics on gesture processing

Numerous event-related brain potential (ERP) studies reveal the differential processing of emotional and neutral stimuli. Yet, it is an ongoing debate to what extent the ERP components found in previous research are sensitive to physical stimulus characteristics or emotional meaning. This study manipulated emotional meaning and stimulus orientation to disentangle the impact of stimulus physics and semantics on emotional stimulus processing. Negative communicative hand gestures of Insult were contrasted with neutral control gestures of Allusion to manipulate emotional meaning. An elementary physical manipulation of visual processing was implemented by presenting these stimuli vertically and horizontally. The results showed dissociable effects of stimulus meaning and orientation on the sequence of ERP components. Effects of orientation were pronounced in the P1 and N170 time frames and attenuated during later stages. Emotional meaning affected the P1, evincing a distinct topography to orientation effects. Although the N170 was not modulated by emotional meaning, the early posterior negativity and late positive potential components replicated previous findings with larger potentials elicited by the Insult gestures. These data suggest that the brain processes different attributes of an emotional picture in parallel and that a coarse semantic appreciation may already occur during relatively early stages of emotion perception.

Time course and specificity of sensory-motor alpha modulation during the observation of hand motor acts and gestures: a high density EEG study

The main aim of the present study was to explore, by means of high-density EEG, the intensity and the temporal pattern of event-related sensory-motor alpha desynchronization (ERD) during the observation of different types of hand motor acts and gestures. In particular, we aimed to investigate whether the sensory-motor ERD would show a specific modulation during the observation of hand behaviors differing for goal-relatedness (hand grasping of an object and meaningless hand movements) and social relevance (communicative hand gestures and grasping within a social context). Time course analysis of alpha suppression showed that all types of hand behaviors were effective in triggering sensory-motor alpha ERD, but to a different degree depending on the category of observed hand motor acts and gestures. Meaningless gestures and hand grasping were the most effective stimuli, resulting in the strongest ERD. The observation of social hand behaviors such as social grasping and communicative gestures, triggered a more dynamic time course of ERD compared to that driven by the observation of simple grasping and meaningless gestures. These findings indicate that the observation of hand motor acts and gestures evoke the activation of a motor resonance mechanism that differs on the basis of the goal-relatedness and the social relevance of the observed hand behavior.

Merging of Phonological and Gestural Circuits in Early Language Evolution

In the monkey, cortical auditory projections subdivide into a dorsal stream mostly involved in spatiotemporal processing, that projects mainly to dorsal frontal areas; and a ventral stream involved in stimulus identification, connected to the ventrolateral prefrontal cortex (VLPFC). We propose that in the human lineage, part of the dorsal auditory pathway has specialized in vocalization processing, enhancing vocal repetition and short-term memory capacities that are crucial for linguistic development. In the human, the vocalization-related dorsal auditory component tends to converge in the VLPFC with the ventral auditory stream and with projections involved in gestural control; and consists of a direct connection between the auditory cortex and the VLPFC via the arcuate fasciculus, and an indirect pathway via the supramarginal gyrus. Additionally, intraparietal and inferior parietal afferents to the VLPFC are associated with communicative hand gestures, with manipulation skills and with early tool-making. Although in general terms compatible with the mirror-neuron gestural hypothesis for language origins, this proposal underlines the participation of the dorsal auditory pathway in voice processing as a key event that marked the beginning of human phonology and the subsequent evolution of language. Instead, the mirror neuron system for gestures and the primitive vocalization network (ventral pathway) contributed to provide a communicative scaffolding that facilitated the emergence of human-like phonology. Furthermore, we emphasize the phylogenetic continuity (homology) between non-human and human vocalization and their neural substrates, something that is not usually stressed in the mirror neuron perspective.

Communicative hand gestures and object-directed hand movements activated the mirror neuron system

Humans produce hand movements to manipulate objects, but also make hand movements to convey socially relevant information to one another. The mirror neuron system (MNS) is activated during the observation and execution of actions. Previous neuroimaging experiments have identified the inferior parietal lobule (IPL) and frontal operculum as parts of the human MNS. Although experiments have suggested that object-directed hand movements drive the MNS, it is not clear whether communicative hand gestures that do not involve an object are effective stimuli for the MNS. Furthermore, it is unknown whether there is differential activation in the MNS for communicative hand gestures and object-directed hand movements. Here we report the results of a functional magnetic resonance imaging (fMRI) experiment in which participants viewed, imitated and produced communicative hand gestures and object-directed hand movements. The observation and execution of both types of hand movements activated the MNS to a similar degree. These results demonstrate that the MNS is involved in the observation and execution of both communicative hand gestures and object-direct hand movements.