Gesture Comprehension Research Articles

The Role of Gesture in Multimodal Communication

Purpose: The general objective of the study was to examine the role of gesture in multimodal communication.
 Methodology: The study adopted a desktop research methodology. Desk research refers to secondary data or that which can be collected without fieldwork. Desk research is basically involved in collecting data from existing resources hence it is often considered a low cost technique as compared to field research, as the main cost is involved in executive’s time, telephone charges and directories. Thus, the study relied on already published studies, reports and statistics. This secondary data was easily accessed through the online journals and library.
 Findings: The findings reveal that there exists a contextual and methodological gap relating to the role of gesture in multimodal communication. Preliminary empirical review revealed that that gestures play a crucial role in multimodal communication, serving as more than mere supplements to verbal discourse. Through theoretical frameworks like social semiotics and empirical research across various contexts, it was found that gestures enhance comprehension, memory retention, and social interactions. The findings emphasized the importance of recognizing and leveraging gestures in communication, urging educators, communicators, and policymakers to develop strategies that harness the full potential of multimodal communication for fostering understanding and connection in diverse settings.
 Unique Contribution to Theory, Practice and Policy: The Cognitive Load theory, Symbolic Interactionism and Social Semiotics model may be used to anchor future studies on the role of gestures in multimodal communication. The study recommended integrating gesture theories within existing communication frameworks, enriching theoretical discourse and offering new perspectives on multimodal communication dynamics. Practically, it advocated for incorporating gesture-based interventions in education and professional settings to enhance communication effectiveness and build rapport. At the policy level, it called for the integration of multimodal communication strategies into educational curricula and workplace training programs to support inclusive and effective communication practices. The study also suggested further research into the cognitive mechanisms of gesture production and comprehension, alongside the development of technology-mediated communication tools and guidelines for inclusive communication practices. Additionally, it proposed interdisciplinary collaborations to advance understanding of multimodal communication processes and drive innovation in the field.

An ultra-low-computation model for understanding sign languages

In artificial intelligence applications, advanced computational models, such as deep learning, are employed to achieve high accuracy, often requiring the execution of numerous operations. Conversely, lightweight computational models are typically more resource-efficient, making them suitable for various devices, including smartphones, tablets, and wearable technology. This paper presents an ultra-low-computation solution for interpreting sign languages to assist deaf and hard-of-hearing individuals without needing specialized hardware or significant computational resources. The proposed approach initially performs data abstraction on the input data. During this process, the image is systematically scanned from various perspectives, and the collected information is then encoded into a one-dimensional vector. Subsequently, the abstracted information undergoes processing through a Fully Connected Neural Network (FCN), resulting in highly accurate output. We also introduced two abstraction methods, namely Opaque and Glass, inspired by the interaction of light with different types of objects. The proposed abstractions facilitate the comprehension of the hand gesture’s outer boundary as well as its row-wise and column-wise density of pixels. Our experiments on three datasets confirm the efficiency of the proposed method, achieving an accuracy of 99.4% in recognizing American Sign Language, 99.96% accuracy in recognizing Indian Sign Language, and 99.95% accuracy in recognizing Bangla Sign Language. Notably, the model size and the number of MAC operations are significantly smaller than state-of-the-art computational models trained on the same datasets.

Descriptive Statistical Features-Based Improvement of Hand Gesture Identification

One of the most significant tools for Human–Computer Interaction (HCI) is body language, namely gestures. Therefore, Gesture recognition using Mechanomyography (MMG) for controlling prostheses and rehabilitation devices has been researched for years. The MMG is gauged as the mechanical vibrations engendered by the muscle in its activation. Hence, MMG data provides information regarding the mechanical activity of muscles. It is obtained from the limb and facilitates the comprehension of hand gestures. A MMG dataset named “MMG DATA” that has already been produced is utilized in this paper. To improve the accuracy of classification and the speed of processing, we have extracted the twelve descriptive statistical features of the eight game motions and their iterations contained in the “MMG DATA” for each subject. Five distinct machine learning algorithms—including Decision Tree (DT), Linear Discriminant Analysis (LDA), Naive Bayes (NB), Support Vector Machine (SVM), and k-Nearest Neighbor (k-NN)—were used to categorize the hand gestures, with the results compared using several performance metrics. Several experiments have been carried out in order to validate the presented models. Additionally, two scenarios have been achieved by splitting the dataset into two different and far apart ratios: the first by randomly partitioning the dataset into 80% for training the models and 20% for testing them, while the second by evenly dividing the dataset into 50% for both training and testing phases. Drawing on the results of the performance metrics, it can be argued that this study effectively distinguishes and classifies the eight hand motions. Furthermore, it is worth noting that the results exhibit significant strength and stability, particularly when transitioning from the initial scenario to the subsequent scenario across all proposed models. Out of the classifiers that were chosen, the SVM emerged as the most successful, achieving a classification accuracy of 97.1%.

Hypoconnectivity networks in schizophrenia patients: A voxel-wise meta-analysis of Rs-fMRI

BackgroundIn recent years several meta-analyses regarding resting-state functional connectivity in patients with schizophrenia have been published. The authors have used different data analysis techniques: regional homogeneity, seed-based data analysis, independent component analysis, and amplitude of low frequencies. Hence, we aim to perform a meta-analysis to identify connectivity networks with different activation patterns between people diagnosed with schizophrenia and healthy controls using voxel-wise analysis. Method: We collected primary studies exploring whole brain connectivity by functional magnetic resonance imaging at rest in patients with schizophrenia compared with healthy controls. We identified 25 studies included high-quality studies that included 1285 patients with schizophrenia and 1279 healthy controls. Results: The results indicate hypoactivation in the right precentral gyrus and the left superior temporal gyrus of patients with schizophrenia compared with healthy controls. Conclusions: These regions have been linked with some clinical symptoms usually present in Plea with schizophrenia, such as auditory verbal hallucinations, formal thought disorder, and the comprehension and production of gestures.

Identifying Patterns of Similarities and Differences between Gesture Production and Comprehension in Autism and Typical Development

Production and comprehension of gesture emerge early and are key to subsequent language development in typical development. Compared to typically developing (TD) children, children with autism spectrum disorders (ASD) exhibit difficulties and/or differences in gesture production. However, we do not yet know if gesture production either shows similar patterns to gesture comprehension across different ages and learners, or alternatively, lags behind gesture comprehension, thus mimicking a pattern akin to speech comprehension and production. In this study, we focus on the gestures produced and comprehended by a group of young TD children and children with ASD—comparable in language ability—with the goal to identify whether gesture production and comprehension follow similar patterns between ages and between learners. We elicited production of gesture in a semi-structured parent–child play and comprehension of gesture in a structured experimenter-child play across two studies. We tested whether young TD children (ages 2–4) follow a similar trajectory in their production and comprehension of gesture (Study 1) across ages, and if so, whether this alignment remains similar for verbal children with ASD (Mage = 5 years), comparable to TD children in language ability (Study 2). Our results provided evidence for similarities between gesture production and comprehension across ages and across learners, suggesting that comprehension and production of gesture form a largely integrated system of communication.

Humanoid Robot Application as COVID-19 Symptoms Checker Using Computer Vision and Multiple Sensors

Novel Corona Virus (nCoV) infects human’s respiratory system. It spreads easily when an infected person makes a close contact with other people. To prevent its massive spread, it is necessary to ensure anyone coming to a certain place is not being infected. The symptoms include high body temperature (≥37.5°C) and low oxygen saturation level (≤95%). This day, most places only check the human body temperature. Thus, the authors are interested to make an attempt to design a system that is able to measure both human body temperature and oxygen saturation level. This work also applies the 7-DoF Upper-Body of Humanoid Robot to prevent virus spread from and to the employee. The system will detect the coming of visitors by using face detection. It requires 7.24 seconds to detect the visitor without a mask, and 1.26 second when the visitor wears a mask. The body temperature measurement was done using GY-906 temperature sensor with an error of 0.51%. For the oxygen saturation level measurement, MAX30100 pulse oximeter module was applied and showed an error of 0.78%. In addition, the upper-body of humanoid robot will perform some gestures to instruct the visitors in every process of the system. The implemented 7-DoF upper-body of humanoid robot has 93.33% gesture comprehension rate. In conclusion, the overall system has been tested and showed success rate up to 75%.

Production and comprehension of co-speech gestures in Parkinson's disease

This study examined how impairments in sensorimotor abilities of individuals with Parkinson's Disease (PD) can be related to the use and understanding of co-speech hand gestures involving literal and figurative actions. We tested individuals with PD (n = 18, 12 males, Mage = 56.5, SDage = 8.16, PD duration since onset: M = 5.36 years, SD = 3.51, Hoehn and Yahr Scale:MH&Y = 2.09, SDH&Y = 0.50) and age- and education-matched neurotypical controls (n = 18, 14 males, Mage = 56.61, SDage = 8.88) with two experimental tasks. In the gesture production task, participants retold the narratives presented to them in a written format. In the gesture comprehension task, participants were asked to match a gesture with a novel verb in literal and figurative sentence contexts. Results showed that patients with PD gestured significantly less than the neurotypical controls. No group differences were found for the type of gesture use. Individuals with PD performed worse than controls on matching gestures with novel verbs, particularly for figurative meanings. Individuals' severity in the disease negatively correlated with their performance for these figurative novel verb-gesture matches. The performances in the two tasks did not correlate. These findings suggest that problems in sensorimotor abilities resulting from PD can influence overall gesture production and gesture comprehension, providing further evidence on the relations between PD and the impaired use of multimodal language.

Differences in the production and perception of communicative kinematics in autism.

In human communication, social intentions and meaning are often revealed in the way we move. In this study, we investigate the flexibility of human communication in terms of kinematic modulation in a clinical population, namely, autistic individuals. The aim of this study was twofold: to assess (a) whether communicatively relevant kinematic features of gestures differ between autistic and neurotypical individuals, and (b) if autistic individuals use communicative kinematic modulation to support gesture recognition. We tested autistic and neurotypical individuals on a silent gesture production task and a gesture comprehension task. We measured movement during the gesture production task using a Kinect motion tracking device in order to determine if autistic individuals differed from neurotypical individuals in their gesture kinematics. For the gesture comprehension task, we assessed whether autistic individuals used communicatively relevant kinematic cues to support recognition. This was done by using stick‐light figures as stimuli and testing for a correlation between the kinematics of these videos and recognition performance. We found that (a) silent gestures produced by autistic and neurotypical individuals differ in communicatively relevant kinematic features, such as the number of meaningful holds between movements, and (b) while autistic individuals are overall unimpaired at recognizing gestures, they processed repetition and complexity, measured as the amount of submovements perceived, differently than neurotypicals do. These findings highlight how subtle aspects of neurotypical behavior can be experienced differently by autistic individuals. They further demonstrate the relationship between movement kinematics and social interaction in high‐functioning autistic individuals.Lay SummaryHand gestures are an important part of how we communicate, and the way that we move when gesturing can influence how easy a gesture is to understand. We studied how autistic and typical individuals produce and recognize hand gestures, and how this relates to movement characteristics. We found that autistic individuals moved differently when gesturing compared to typical individuals. In addition, while autistic individuals were not worse at recognizing gestures, they differed from typical individuals in how they interpreted certain movement characteristics.

Gesture Influences Resolution of Ambiguous Statements of Neutral and Moral Preferences.

When faced with an ambiguous pronoun, comprehenders use both multimodal cues (e.g., gestures) and linguistic cues to identify the antecedent. While research has shown that gestures facilitate language comprehension, improve reference tracking, and influence the interpretation of ambiguous pronouns, literature on reference resolution suggests that a wide set of linguistic constraints influences the successful resolution of ambiguous pronouns and that linguistic cues are more powerful than some multimodal cues. To address the outstanding question of the importance of gesture as a cue in reference resolution relative to cues in the speech signal, we have previously investigated the comprehension of contrastive gestures that indexed abstract referents – in this case expressions of personal preference – and found that such gestures did facilitate the resolution of ambiguous statements of preference. In this study, we extend this work to investigate whether the effect of gesture on resolution is diminished when the gesture indexes a statement that is less likely to be interpreted as the correct referent. Participants watched videos in which a speaker contrasted two ideas that were either neutral (e.g., whether to take the train to a ballgame or drive) or moral (e.g., human cloning is (un)acceptable). A gesture to the left or right side co-occurred with speech expressing each position. In gesture-disambiguating trials, an ambiguous phrase (e.g., I agree with that, where that is ambiguous) was accompanied by a gesture to one side or the other. In gesture non-disambiguating trials, no third gesture occurred with the ambiguous phrase. Participants were more likely to choose the idea accompanied by gesture as the stimulus speaker’s preference. We found no effect of scenario type. Regardless of whether the linguistic cue expressed a view that was morally charged or neutral, observers used gesture to understand the speaker’s opinion. This finding contributes to our understanding of the strength and range of cues, both linguistic and multimodal, that listeners use to resolve ambiguous references.

A audiodescrição na dança em cadeira de rodas

O estudo teve como objetivo elaborar um roteiro de Audiodescrição (AD) de um espetáculo de dança em cadeira de rodas (DECR); e verificar a significância e viabilidade deste recurso, quanto a compreensão da descrição dos movimentos, por pessoas com deficiência visual e profissionais na área da dança. Trata-se de uma pesquisa descritiva e exploratória ao qual foi construído e apresentado o roteiro da AD do espetáculo de dança em cadeira de rodas. O roteiro foi aplicado em um software de legendagem Workshop (SW), versão 2.51 e a AD da dança em cadeira de rodas foi apreciada pelos participantes da pesquisa. Como instrumento de coleta de dados todos responderam a um questionário sobre os significados/entendimento da coreografia somado a uma entrevista quanto a compreensão dos gestos corporais narrados na AD. As respostas foram comparadas entre si buscando identificar as concordâncias e discrepâncias assinaladas pelos participantes. Pelos resultados podemos analisar que este estudo trouxe contribuições expressivas, sistematizando termos referentes à dança em cadeira de rodas no estilo Rumba, permitindo que esta modalidade de dança seja acessível ao deficiente visual em futuras competições nacionais e internacionais.

Towards a reliable measure of motor working memory: revisiting Wu and Coulson's (2014) movement span task.

Some researchers have argued that motor working memory is relatively independent from visuospatial working memory and underlies the learning and processing of motor tasks, like gesture comprehension. To allow systematic testing of these claims, Wu & Coulson 2014 Psychol. Sci. 26, 1717–1727. (doi:10.1177/0956797615597671) proposed a novel measure of motor working memory, the movement span task. Some studies have reported that the movement span task has a high degree of validity. The purpose of the present study was to attempt to replicate Wu & Coulson 2014 Psychol. Sci. 26, 1717–1727. (doi:10.1177/0956797615597671) in the following ways: (1) the high correlation between movement span and movement recall scores and (2) the lack of correlation between the movement span task on the one hand and visuospatial and verbal working memory on the other. In the present study, we found a high correlation between the movement span and recall scores as well as most measures of visuospatial memory. However, the size of these correlations was similar to that reported by Wu and Coulson, suggesting that the significance may be related to sample size. In other words, motor working memory may be weakly related to visuospatial memory. By contrast, there were weak correlations between the movement span task and verbal memory. In sum, we found the same pattern of results observed by Wu & Coulson 2014, 1717–1727. (doi:10.1177/0956797615597671).

Integrated Communication System: Gesture and Language Acquisition in Typically Developing Children and Children With LD and DLD

Gesture and language development are strongly connected to each other. Two types of gestures in particular are analyzed regarding their role for language acquisition: pointing and iconic gestures. With the present longitudinal study, the predictive values of index-finger pointing at 12 months and the comprehension of iconic gestures at 3;0 years for later language skills in typically developing (TD) children and in children with a language delay (LD) or developmental language disorder (DLD) are examined. Forty-two monolingual German children and their primary caregivers participated in the study and were followed longitudinally from 1;0 to 6;0 years. Within a total of 14 observation sessions, the gestural and language abilities of the children were measured using standardized as well as ad hoc tests, parent questionnaires and semi-natural interactions between the child and their caregivers. At the age of 2;0 years, 10 of the 42 children were identified as having a LD. The ability to point with the extended index finger at 1;0 year is predictive for language skills at 5;0 and 6;0 years. This predictive effect is mediated by the language skills of the children at 3;0 years. The comprehension of iconic gestures at 3;0 years correlates with index-finger pointing at 1;0 year and also with earlier and later language skills. It mediates the predictive value of index-finger pointing at 1;0 year for grammar skills at 5;0 and 6;0 years. Children with LD develop the ability to understand the iconicity in gestures later than TD children and score lower in language tests until the age of 6;0 years. The language differences between these two groups of children persist partially until the age of 5;0 years even when the two children with manifested DLD within the group of children with LD are excluded from analyses. Beyond that age, no differences in the language skills between children with and without a history of LD are found when children with a manifest DLD are excluded. The findings support the assumption of an integrated speech–gesture communication system, which functions similarly in TD children and children with LD or DLD, but with a time delay.

The communicative advantage: how kinematic signaling supports semantic comprehension

Humans are unique in their ability to communicate information through representational gestures which visually simulate an action (eg. moving hands as if opening a jar). Previous research indicates that the intention to communicate modulates the kinematics (e.g., velocity, size) of such gestures. If and how this modulation influences addressees’ comprehension of gestures have not been investigated. Here we ask whether communicative kinematic modulation enhances semantic comprehension (i.e., identification) of gestures. We additionally investigate whether any comprehension advantage is due to enhanced early identification or late identification. Participants (n = 20) watched videos of representational gestures produced in a more- (n = 60) or less-communicative (n = 60) context and performed a forced-choice recognition task. We tested the isolated role of kinematics by removing visibility of actor’s faces in Experiment I, and by reducing the stimuli to stick-light figures in Experiment II. Three video lengths were used to disentangle early identification from late identification. Accuracy and response time quantified main effects. Kinematic modulation was tested for correlations with task performance. We found higher gesture identification performance in more- compared to less-communicative gestures. However, early identification was only enhanced within a full visual context, while late identification occurred even when viewing isolated kinematics. Additionally, temporally segmented acts with more post-stroke holds were associated with higher accuracy. Our results demonstrate that communicative signaling, interacting with other visual cues, generally supports gesture identification, while kinematic modulation specifically enhances late identification in the absence of other cues. Results provide insights into mutual understanding processes as well as creating artificial communicative agents.

Stopping Eating and Drinking by Advance Directives: Choose Your Injustice

Stopping Eating and Drinking by Advance Directives: Choose Your Injustice

The role of conflict, feedback, and action comprehension in monitoring of action errors: Evidence for internal and external routes

The mechanisms and brain regions underlying error monitoring in complex action are poorly understood, yet errors and impaired error correction in these tasks are hallmarks of apraxia, a common disorder associated with left hemisphere stroke. Accounts of monitoring of language posit an internal route by which production planning or competition between candidate representations provide predictive signals that monitoring is required to prevent error, and an external route in which output is monitored using the comprehension system. Abnormal reliance on the external route has been associated with damage to brain regions critical for sensory-motor transformation and a pattern of gradual error ‘clean-up’ called conduite d'approche (CD). Action pantomime data from 67 participants with left hemisphere stroke were consistent with versions of internal route theories positing that competition signals monitoring requirements. Support Vector Regression Lesion Symptom Mapping (SVR-LSM) showed that lesions in the inferior parietal, posterior temporal, and arcuate fasciculus/superior longitudinal fasciculus predicted action conduite d'approche, overlapping the regions previously observed in the language domain. A second experiment with 12 patients who produced substantial action CD assessed whether factors impacting the internal route (action production ability, competition) versus external route (vision of produced actions, action comprehension) influenced correction attempts. In these ‘high CD’ patients, vision of produced actions and integrity of gesture comprehension interacted to determine successful error correction, supporting external route theories. Viewed together, these and other data suggest that skilled actions are monitored both by an internal route in which conflict aids in detection and correction of errors during production planning, and an external route that detects mismatches between produced actions and stored knowledge of action appearance. The parallels between language and action monitoring mechanisms and neuroanatomical networks pave the way for further exploration of common and distinct processes across these domains.

The Relation between the Ability and the Performance on Integrated Comprehension of Speech and Iconic Gestures: From the Effect of Directive Words Point of View

The Relation between the Ability and the Performance on Integrated Comprehension of Speech and Iconic Gestures: From the Effect of Directive Words Point of View

Natural reference: A phylo- and ontogenetic perspective on the comprehension of iconic gestures and vocalizations.

The recognition of iconic correspondence between signal and referent has been argued to bootstrap the acquisition and emergence of language. Here, we study the ontogeny, and to some extent the phylogeny, of the ability to spontaneously relate iconic signals, gestures, and/or vocalizations, to previous experience. Children at 18, 24, and 36months of age (N=216) and great apes (N=13) interacted with two apparatuses, each comprising a distinct action and sound. Subsequently, an experimenter mimicked either the action, the sound, or both in combination to refer to one of the apparatuses. Experiments 1 and 2 found no spontaneous comprehension in great apes and in 18-month-old children. At 24months of age, children were successful with a composite vocalization-gesture signal but not with either vocalization or gesture alone. At 36months, children succeeded both with a composite vocalization-gesture signal and with gesture alone, but not with vocalization alone. In general, gestures were understood better compared to vocalizations. Experiment 4 showed that gestures were understood irrespective of how children learned about the corresponding action (through observation or self-experience). This pattern of results demonstrates that iconic signals can be a powerful way to establish reference in the absence of language, but they are not trivial for children to comprehend and not all iconic signals are created equal.

Developmental and cognitive aspects of children’s disbelief comprehension through intonation and facial gesture

This article investigates how children leverage intonational and gestural cues to an individual’s belief state through unimodal (intonation-only or facial gesture-only) and multimodal (intonation + facial gesture) cues. A total of 187 preschoolers (ages 3–5) participated in a disbelief comprehension task and were assessed for Theory of Mind (ToM) ability using a false belief task. Significant predictors included age, condition and success on the ToM task. Performance improved with age, and was significantly better for the multimodal condition compared to both unimodal conditions, suggesting that even though unimodal cues were useful to children, the presence of reinforcing information for the multimodal condition was more effective for detecting disbelief. However, results also point to the development of intonational and gestural comprehension in tandem. Children that passed the ToM task significantly outperformed those that failed it for all conditions, showing that children who can attribute a false belief to another individual may more readily access these intonational and gestural cues.

Communicative intent modulates production and comprehension of actions and gestures: A Kinect study

Actions may be used to directly act on the world around us, or as a means of communication. Effective communication requires the addressee to recognize the act as being communicative. Humans are sensitive to ostensive communicative cues, such as direct eye gaze (Csibra & Gergely, 2009). However, there may be additional cues present in the action or gesture itself. Here we investigate features that characterize the initiation of a communicative interaction in both production and comprehension.We asked 40 participants to perform 31 pairs of object-directed actions and representational gestures in more- or less- communicative contexts. Data were collected using motion capture technology for kinematics and video recording for eye-gaze. With these data, we focused on two issues. First, if and how actions and gestures are systematically modulated when performed in a communicative context. Second, if observers exploit such kinematic information to classify an act as communicative.Our study showed that during production the communicative context modulates space–time dimensions of kinematics and elicits an increase in addressee-directed eye-gaze. Naïve participants detected communicative intent in actions and gestures preferentially using eye-gaze information, only utilizing kinematic information when eye-gaze was unavailable.Our study highlights the general communicative modulation of action and gesture kinematics during production but also shows that addressees only exploit this modulation to recognize communicative intention in the absence of eye-gaze. We discuss these findings in terms of distinctive but potentially overlapping functions of addressee directed eye-gaze and kinematic modulations within the wider context of human communication and learning.

Comprehension of conventional gestures in typical children, children with autism spectrum disorders and children with language disorders

BackgroundConventional gestures (CG) contribute to communication and are a precursor of verbal language. The aim of our study was: (a) to analyse the developmental course of CG comprehension in typical children; (b) from a clinical perspective, to document how children with autism spectrum disorders (ASD) and language disorder (LD) can understand CG. MethodsComprehension of CG was evaluated through a video-based testing. The protocol consisted of 15 arbitrary gestures and 15 iconic gestures. We recruited 262 typical children aged from 3 to 10years old and 55 children with communication impairments aged from 4 to 8years old: 36 children with ASD and 19 with LD. ResultsCG comprehension increased during the early development of typical children and the improvement of iconic gestures comprehension occurred earlier than that of arbitrary ones. Children with ASD received lower CG scores than typical children and LD children. For each gesture taken separately, children with LD performed better than not only ASD children but typical children as well. Furthermore, when we performed the analysis with both arbitrary and iconic gesture, children with LD outperformed both children with ASD and TD children. ConclusionsOur results suggest a dissociation between oral language comprehension and language gesture comprehension in children with LD and a deficit in both oral language and language gesture comprehension in children with ASD.