Types Of Gestures Research Articles

Visual Static Hand Gesture Recognition Using Convolutional Neural Network

Hand gestures are an essential part of human-to-human communication and interaction and, therefore, of technical applications. The aim is increasingly to achieve interaction between humans and computers that is as natural as possible, for example, by means of natural language or hand gestures. In the context of human-machine interaction research, these methods are consequently being explored more and more. However, the realization of natural communication between humans and computers is a major challenge. In the field of hand gesture recognition, research approaches are being pursued that use additional hardware, such as special gloves, to classify gestures with high accuracy. Recently, deep learning techniques using artificial neural networks have been increasingly proposed for the problem of gesture recognition without using such tools. In this context, we explore the approach of convolutional neural network (CNN) in detail for the task of hand gesture recognition. CNN is a deep neural network that can be used in the fields of visual object processing and classification. The goal of this work is to recognize ten types of static hand gestures in front of complex backgrounds and different hand sizes based on raw images without the use of extra hardware. We achieved good results with a CNN network architecture consisting of seven layers. Through data augmentation and skin segmentation, a significant increase in the model’s accuracy was achieved. On public benchmarks, two challenging datasets have been classified almost perfectly, with testing accuracies of 96.5% and 96.57%.

Automated Gesture Recognition Using African Vulture Optimization with Deep Learning for Visually Impaired People on Sensory Modality Data

Gesture recognition for visually impaired persons (VIPs) is a useful technology for enhancing their communications and increasing accessibility. It is vital to understand the specific needs and challenges faced by VIPs when planning a gesture recognition model. But, typical gesture recognition methods frequently depend on the visual input (for instance, cameras); it can be vital to discover other sensory modalities for input. The deep learning (DL)-based gesture recognition method is effective for the interaction of VIPs with their devices. It offers a further intuitive and natural way of relating with technology, creating it more available for everybody. Therefore, this study presents an African Vulture Optimization with Deep Learning-based Gesture Recognition for Visually Impaired People on Sensory Modality Data (AVODL-GRSMD) technique. The AVODL-GRSMD technique mainly focuses on the utilization of the DL model with hyperparameter tuning strategy for a productive and accurate gesture detection and classification process. The AVODL-GRSMD technique utilizes the primary data preprocessing stage to normalize the input sensor data. The AVODL-GRSMD technique uses a multi-head attention-based bidirectional gated recurrent unit (MHA-BGRU) method for accurate gesture recognition. Finally, the hyperparameter optimization of the MHA-BGRU method can be performed by the use of African Vulture Optimization with Deep Learning (AVO) approach. A series of simulation analyses were performed to demonstrate the superior performance of the AVODL-GRSMD technique. The experimental values demonstrate the better recognition rate of the AVODL-GRSMD technique compared to that of the state-of-the-art models.

More than words: a study on the visibility of hand gestures in public spaces

Hand gestures play an important role in human communication. Although the study of their repertoires and roles for past communities is a popular field of research, there has been no attempt so far to study their visibility during public events. The aim of this study was to determine the maximum number of people who could see hand gestures well enough to understand their meaning. Using gestures taken from ancient Roman rhetorical treatises, which we divided into three classes related to the detail of the gestures (fingers, hand, arm, or arms), we conducted a series of experiments to determine the maximum distance from which each class of gestures could be seen. We used the results, including regression analysis, to conduct visibility analyses for two case studies: one on the rostra on the Late Republican Forum Romanum in Rome; and the other on Pyramid No 3 in the centre of Late-Classical Mayan Tikal. We used the calculation of the areas where gestures were visible to estimate crowd sizes by drawing on crowd behaviour observation during contemporary public gatherings. They show not only how many people could have potentially seen the gestures, but also what percentage of the theoretically available space could have been occupied by people who had the potential to see them. According to the findings, only a little under half (44.8%) of the maximum possible audience were able to detect all types of gestures (various levels of detail) at the LR Roman Forum, while at Pyramid No 3 in Tikal, just a mere 16.7% were able to do so. We believe that the results presented and the methodology used can be applied to analyse any public space, regardless of place and time, thus providing a valuable tool to comprehend past public assemblies.

Теоретичний аналіз проблеми жестової стимуляції в роботі з пацієнтами з постінсультною афазією

The article contains the analyses of theoretical aspects of gesture usage as a therapeutic tool in the speech therapy rehabilitation of patients with aphasia. The paper characterizes studies that indicate a close relation between verbal and non-verbal communication. There are defined main types of gestures and their classification, as well as the communicative function of gesticulation for people with preserved speech and the ones with aphasia. There is a description of gesture functioning peculiarities for people with aphasia and their impact on the process of speech therapy rehabilitation. There were analyzed results of previously conducted studies on the effectiveness of gestural stimulation as a method of restoring communication in patients with aphasia. It allowed to state following. Most studies indicate that the use of gestures in impaired communication contributes to the restoration of verbal speech. As well as improves the search for needed vocabulary (both nouns and verbs), helps the patient to remember words better, and in the case of unintelligible speech to convey and recognize the content of the message. It also provides an opportunity for the patient to maintain contact with the surrounding people. Difficulties identified while the research are the use of gestural stimulation in remedial learning, namely motor disorders, particularly limb apraxia, which is common in stroke patients. The article contains the description of tests generally accepted in the neurological clinic, with the help of which it is possible to assess the preservation of complex forms of praxis in patients with organic lesions of the central nervous system. Furthermore, there is the analysis of studies that deny the positive influence of gestures on communication in people with aphasia, or determine the effectiveness of their use only in working with patients with a mild degree of aphasia.

Using virtual reality to assess gesture performance deficits in schizophrenia patients.

Gesture performance deficits are prevalent in schizophrenia patients and are strongly associated with poor social communication skills and community functioning, affecting their overall quality of life. Currently, video-recording technology is widely used in clinical settings to assess gesture production deficits in schizophrenia patients. Nevertheless, the subjective evaluation of video-recordings can encumber task assessment. The present study will aim to use virtual reality to examine its potential use as an alternative tool to objectively measure gesture performance accuracy in schizophrenia patients and healthy controls. Gesture performance in the virtual reality setting will be based on the well-established Test of Upper Limb Apraxia. Participants will be immersed in a virtual environment where they will experience themselves being embodied in a collocated virtual body seen from a first-person perspective. Motion trackers will be placed on participants' hands and elbows to track upper body movements in real-time, and to record gesture movement for later analysis. Participants will see a virtual agent sitting across from them, with a virtual table in between. The agent will perform various types of gestures and the participants' task will be to imitate those gestures as accurately as possible. Measurements from the tracking devices will be stored and analyzed to address gesture performance accuracy across groups. This study aims to provide objective measurements of gesture performance accuracy in schizophrenia patients. If successful, the results will provide new knowledge to the gesture literature and offer the potential for novel therapeutic interventions using virtual reality technologies. Such interventions can improve gesturing and thus advance social communication skills in schizophrenia patients.

Exploring Students’ Use of Gestures to Create Scientific Models during Geological Field Trips and Modeling Activities

Abstract This study contributes to the literature by shedding light on students’ learning by examining and comparing students’ gestures in two educational contexts: geological field trips and classroom-based modeling activities. This study engaged 10 middle-school students in two different geological field trips and two classroom-based modeling activities and employed a hermeneutic approach to understand students’ use of gestures in both settings. Using multimodal interaction analysis (MIA) students’ gestures were explored to understand how students created and conveyed meaning using verbal and nonverbal interactions. The study found students’ gestures conveyed scientific content and differing social roles in the two settings. The results suggest the frequency and type of gestures used in each setting may be complementary to students’ development of accurate models. This study presents new classification criteria and characteristics to identify students’ gestures. This study demonstrates gestures can be an important resource for learning when students are engaging in modeling activities.

Nonverbal Communication And Sales Performance: Do High and Low-Performing Sales Students Gesture in the Same Way?

ABSTRACT Nonverbal communication impacts a broad range of human interactions and bears significant influence on understanding and perceptions. One of the most influential categories of nonverbal communication is kinesics, which includes physical gestures and other body movements. The current study develops a foundation for future kinesics education in sales by examining the use of four types of gestures (illustrators, beats, adaptors, and regulators) and coinciding measures of sales performance within the context of a national student sales competition. Results include the finding that highly rated sales students use one specific kind of gesture – illustrators - more frequently than low performing sales students. No significant relationships were uncovered between beats, adaptors, or regulators and performance. Educational implications and recommendations for sales courses are provided.

Audible gestures

Abstract This study focuses on one type of audible gesture, designedly single claps (DSCs), as used by different people at an educational institution. The institution is designed to provide second language English users with opportunities to use English in various situations. Through the use of Multimodal Conversation Analysis, the analysis first focuses on the shape of DSCs and what makes them visible as not projecting further claps. Next, the analysis focuses on how DSCs are used within their sequential context. DSCs can take a variety of shapes, in that there are different ways not to project further claps; they can be used to attract attention of multiple recipients, and thus as one resource to manage interaction; and they are used as such a resource by representatives of the educational institution, who take on teacher roles within the interaction, with responsibility and deontic authority to manage shifts in activity and participation framework.

Gesture-speech coupling in persons with aphasia: A kinematic-acoustic analysis.

Aphasia is a profound language pathology hampering speech production and/or comprehension. People With Aphasia (PWA) use more manual gestures than Non-Brain Injured (NBI) individuals. This intuitively invokes the idea that gesture is compensatory in some way, but there is variable evidence of a gesture-boosting effect on speech processes. The status quo in gesture research with PWA is an emphasis on categorical analysis of gesture types, focusing on how often they are recruited, and whether more or less gesturing aids communication or speaking. However, there are increasingly louder calls for the investigation of gesture and speech as continuous entangled modes of expression. In NBI adults, expressive moments of gesture and speech are synchronized on the prosodic level. It has been neglected how this multimodal prosody is instantiated in PWA. In the current study, we perform the first acoustic-kinematic gesture-speech analysis in Persons With Aphasia (i.e., Wernicke's, Broca's, Anomic) relative to age-matched controls, where we apply several multimodal signal analysis methods. Specifically, we related the speech peaks (smoothed amplitude envelope change) with that of the nearest peaks in the gesture acceleration profile. We obtained that the magnitude of gesture versus speech peaks are positively related across the groups, though more variably for PWA, and such coupling was related to less severe Aphasia-related symptoms. No differences were found between controls and PWA in terms of temporal ordering of speech envelope versus acceleration peaks. Finally, we show that both gesture and speech have slower quasi-rhythmic structure, indicating that next to speech, gesture is slowed down too. The current results indicate that there is a basic gesture-speech coupling mechanism that is not fully reliant on core linguistic competences, as it is found relatively intact in PWA. This resonates with a recent biomechanical theory of gesture, which renders gesture-vocal coupling as fundamental and a priori to the (evolutionary) development of core linguistic competences. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Sign Language Based Video Calling App

Deaf and hard of hearing people communicate with others and within their own communities by using sign language. Computer recognition of sign language begins with the learning of sign gestures and continues through the production of text and speech. There are two types of sign gestures: static and dynamic. Both gesture recognition systems are crucial to the human community, even if static gesture recognition is easier than dynamic gesture recognition. We have conducted research on the steps required to convert static American sign language (ASL) to readable text and selected the best available methods to do so. Examined general steps are the data collection, pre-processing, transformation, feature extraction, and classification. There are also some recommendations for further study in this field.

No gesture too small: An investigation into the ability of gestural information to mislead eyewitness accounts by 5- to 8-year-olds

The accuracy of eyewitness interviews has legal and clinical implications within the criminal justice system. Leading verbal suggestions have been shown to give rise to false memories and inaccurate testimonies in children, but only a small body of research exists regarding non-verbal communication. The present study examined whether 5- to 8-year-olds in the UK could be misled about their memory of an event through exposure to leading gestural information, which suggested an incorrect response, using a variety of question and gesture types. Results showed that leading gestures significantly corrupted participants’ memory compared to the control group (MD = 0.60, p < 0.001), with participants being misled by at least one question nearly three-quarters of the time. Questions about peripheral details, and gestures that were more visible and expressive, increased false memory further, with even subtle gestures demonstrating a strong misleading influence. We discuss the implications of these findings for the guidelines governing eyewitness interviews.

Distributed embodiment of metaphorical hope in hand, head, and eyebrow gestures.

This study aimed to examine the embodied conceptualization of hope through metaphors. We asked a group of participants to discuss their hopes in a semi-structured interview. We examined the types of hand, head, and eyebrow gestures produced when they were talking about their future hopes. The obtained results showed that when participants talked about their future hopes, they mainly used forward hand gestures, rightward head gestures, and upward eyebrow gestures. Based on these results, it is suggested that various semantic components and emotional associations of hope are metaphorically embodied in different manners in various parts of the body. The future aspect of hope is conceptualized as a forward movement and is embodied as a forward hand gesture. The good or positive emotional aspect associated with future hopes is metaphorically conceptualized as a rightward head gesture or an upward eyebrow gesture. We call this process distributed embodiment of a metaphorical concept. Our proposal is supported by the findings of past studies that have found future is metaphorically embodied as something in front of us (or forward movement), and good is metaphorically embodied as upper space (or upward movement) or right side (or rightward movement).

Dynamic Visualization of VR Map Navigation Systems Supporting Gesture Interaction

With the rapid development of information technology, virtual reality and gesture interaction have been gradually applied in the research and development of map navigation systems. Traditional visualization methods are no longer suitable for this novel interactive map. This research offers a dynamic visualization plan for a virtual reality (VR) navigation map focusing on natural gesture interaction to give examples for creating similar systems. The principal work is composed of two experiments. The first experiment focuses on designing map navigation gestures (moving, rotating, and zooming). Heuristic experiments are used to collect users’ subjective preferences and design suggestions about gestures. The second experiment is designed as a behavioral study to investigate which types of gestures and visualizations, among those obtained from the heuristic experiment in the first part, yield higher performance in our specific scenario. This result offers a practical VR map dynamic display approach through experimental validation. It also provides the basis for a human factor and technology support for future investigations.

GesturalOrigins: A bottom-up framework for establishing systematic gesture data across ape species

Current methodologies present significant hurdles to understanding patterns in the gestural communication of individuals, populations, and species. To address this issue, we present a bottom-up data collection framework for the study of gesture: GesturalOrigins. By “bottom-up”, we mean that we minimise a priori structural choices, allowing researchers to define larger concepts (such as ‘gesture types’, ‘response latencies’, or ‘gesture sequences’) flexibly once coding is complete. Data can easily be re-organised to provide replication of, and comparison with, a wide range of datasets in published and planned analyses. We present packages, templates, and instructions for the complete data collection and coding process. We illustrate the flexibility that our methodological tool offers with worked examples of (great ape) gestural communication, demonstrating differences in the duration of action phases across distinct gesture action types and showing how species variation in the latency to respond to gestural requests may be revealed or masked by methodological choices. While GesturalOrigins is built from an ape-centred perspective, the basic framework can be adapted across a range of species and potentially to other communication systems. By making our gesture coding methods transparent and open access, we hope to enable a more direct comparison of findings across research groups, improve collaborations, and advance the field to tackle some of the long-standing questions in comparative gesture research.

Multimodal language in bilingual and monolingual children: Gesture production and speech disfluency

AbstractBilingual and monolingual children might have different styles of using multimodal language. This study investigates speech disfluency and gesture production of 5- and 7-year-old Turkish monolingual (N = 61) and Turkish–English bilingual children (N = 51). We examined monolinguals’ Turkish narratives and bilinguals’ Turkish and English narratives. Results indicated that bilinguals were more disfluent than monolinguals, particularly for silent and filled (e.g., umm) pauses. Bilinguals used silent pauses and repetitions (e.g., cat cat) more frequently in English than in Turkish. Gesture use was comparable across language and age groups, except for iconic gestures. Monolinguals produced more iconic gestures than bilinguals. Children's overall gesture frequency predicted disfluency rates only in Turkish. Different gesture types might be orchestrated in the multimodal system, contributing to narrative fluency. The use of disfluency and gesture types might provide insight into bilingual and monolingual children's language development and communication strategies.

Multimodal Development in Children's Narrative Speech: Evidence for Tight Gesture-Speech Temporal Alignment Patterns as Early as 5 Years Old.

This study aims to analyze the development of gesture-speech temporal alignment patterns in children's narrative speech from a longitudinal perspective and, specifically, the potential differences between different gesture types, namely, gestures that imagistically portray or refer to semantic content in speech (i.e., referential gestures) and those that lack semantic content (i.e., non-referential gestures). This study uses an audiovisual corpus of narrative productions (n = 332) from 83 children (43 girls, 40 boys) who participated in a narrative retelling task at two time points in development (at 5-6 and 7-9 years of age). The 332 narratives were coded for both manual co-speech gesture types and prosody. Gestural annotations included gesture phasing (i.e., preparation, stroke, hold, and recovery) and gesture types (in terms of referentiality, i.e., referential and non-referential), whereas prosodic annotations included pitch-accented syllables. Results revealed that by ages 5-6 years, children already temporally aligned the stroke of both referential and non-referential gestures with pitch-accented syllables, showing no significant differences between these two gesture types. The results of the present study contribute to the view that both referential and non-referential gestures are aligned with pitch accentuation, and therefore, this is not only a characteristic of non-referential gestures. Our results also add support to McNeill's phonological synchronization rule from a developmental perspective and indirectly back up recent theories about the biomechanics of gesture-speech alignment, suggesting that this is an inherent ability of oral communication.

A Review on Methods and Applications in Multimodal Deep Learning

Deep Learning has implemented a wide range of applications and has become increasingly popular in recent years. The goal of multimodal deep learning (MMDL) is to create models that can process and link information using various modalities. Despite the extensive development made for unimodal learning, it still cannot cover all the aspects of human learning. Multimodal learning helps to understand and analyze better when various senses are engaged in the processing of information. This article focuses on multiple types of modalities, i.e., image, video, text, audio, body gestures, facial expressions, physiological signals, flow, RGB, pose, depth, mesh, and point cloud. Detailed analysis of the baseline approaches and an in-depth study of recent advancements during the past five years (2017 to 2021) in multimodal deep learning applications has been provided. A fine-grained taxonomy of various multimodal deep learning methods is proposed, elaborating on different applications in more depth. Last, main issues are highlighted separately for each domain, along with their possible future research directions.

Best of Both Worlds? Combining Different Forms of Mixed Reality Deictic Gestures

Mixed Reality provides a powerful medium for transparent and effective human-robot communication, especially for robots with significant physical limitations (e.g., those without arms). To enhance nonverbal capabilities for armless robots, this article presents two studies that explore two different categories of mixed reality deictic gestures for armless robots: a virtual arrow positioned over a target referent (a non-ego-sensitive allocentric gesture) and a virtual arm positioned over the gesturing robot (an ego-sensitive allocentric gesture). In Study 1, we explore the tradeoffs between these two types of gestures with respect to both objective performance and subjective social perceptions. Our results show fundamentally different task-oriented versus social benefits, with non-ego-sensitive allocentric gestures enabling faster reaction time and higher accuracy, but ego-sensitive gestures enabling higher perceived social presence, anthropomorphism, and likability. In Study 2, we refine our design recommendations by showing that in fact these different gestures should not be viewed as mutually exclusive alternatives, and that by using them together, robots can achieve both task-oriented and social benefits.

The context of infants' giving gestures in mother-infant dyads: Typical giving gestures and those contingent on exploration and play.

This study aimed to focus on a niche that has not yet been investigated in infants' gesture studies that is the effect of the prior context of one specific gestural behavior (gives) on maternal behavior. For this purpose, we recruited 23 infants at 11 and 13months of age yielded 246 giving gesture bouts that were performed in three contexts: typical when the object was offered immediately, contingent on exploration, and contingent on play. The analysis revealed that maternal responses to infants' giving gestures varied and were affected by their age and gesture context. Hence, mothers amended their responses according to the background that generated each gesture. The number of verbal responses to infants' giving gestures decreased as the infants aged, whereas the number of pretense responses increased. For infants aged 11months, mothers generally provided motor responses to typical gestures. However, for infants aged 13months, this trend declined and was replaced by a strong positive correlation between giving gestures contingent on play and verbal responses. We concluded that the type of activity with objects prior to employing giving gestures could enhance infants' symbolic skills because caregivers monitor the contingent act that yields the gesture that shapes their response.

Assessing cats’ (Felis catus) sensitivity to human pointing gestures.

A wide range of nonhuman animal species has been shown to be able to respond to human referential signals, such as pointing gestures. The aim of the present study was to replicate previous findings showing cats to be sensitive to human pointing cues (Miklósi et al., 2005). In our study, we presented two types of human pointing gestures-momentary ipsilateral (direct pointing) and momentary cross-body pointing. We tested nine rescue cats in a two-way object-choice task. On a group level, the success rate of cats was 74.4%. Cats performed significantly above chance level in both the ipsilateral and cross-body pointing conditions. Trial number, rewarded side, and type of gesture did not significantly affect the cats' performances in the experiment. On an individual level, five out of seven cats who completed 20 trials performed significantly above chance level. Two cats only completed 10 trials. One of them succeeded in eight and the other in six of these trials. The results of our study replicate previous findings of cats being responsive to human ipsilateral pointing cues and add additional knowledge about their ability to follow cross-body pointing cues. Our results highlight that a domestic species, socialized in a group setting, may possess heterospecific communication skills. Further research is needed to exclude alternative parsimonious explanations, such as local and stimulus enhancements. (PsycInfo Database Record (c) 2023 APA, all rights reserved).