Meaningful Gestures Research Articles

Data glove-based gesture recognition using CNN-BiLSTM model with attention mechanism

As a novel form of human machine interaction (HMI), hand gesture recognition (HGR) has garnered extensive attention and research. The majority of HGR studies are based on visual systems, inevitably encountering challenges such as depth and occlusion. On the contrary, data gloves can facilitate data collection with minimal interference in complex environments, thus becoming a research focus in fields such as medical simulation and virtual reality. To explore the application of data gloves in dynamic gesture recognition, this paper proposes a data glove-based dynamic gesture recognition model called the Attention-based CNN-BiLSTM Network (A-CBLN). In A-CBLN, the convolutional neural network (CNN) is employed to capture local features, while the bidirectional long short-term memory (BiLSTM) is used to extract contextual temporal features of gesture data. By utilizing attention mechanisms to allocate weights to gesture features, the model enhances its understanding of different gesture meanings, thereby improving recognition accuracy. We selected seven dynamic gestures as research targets and recruited 32 subjects for participation. Experimental results demonstrate that A-CBLN effectively addresses the challenge of dynamic gesture recognition, outperforming existing models and achieving optimal gesture recognition performance, with the accuracy of 95.05% and precision of 95.43% on the test dataset.

Reaching beneath the tip of the iceberg: A guide to the Freiburg Multimodal Interaction Corpus

Abstract Most corpora tacitly subscribe to a speech-only view filtering out anything that is not a ‘word’ and transcribing the spoken language merely orthographically despite the fact that the “speech-only view on language is fundamentally incomplete” (Kok 2017, 2) due to the deep intertwining of the verbal, vocal, and kinesic modalities (Levinson and Holler 2014). This article introduces the Freiburg Multimodal Interaction Corpus (FreMIC), a multimodal and interactional corpus of unscripted conversation in English currently under construction. At the time of writing, FreMIC comprises (i) c. 29 h of video-recordings transcribed and annotated in detail and (ii) automatically (and manually) generated multimodal data. All conversations are transcribed in ELAN both orthographically and using Jeffersonian conventions to render verbal content and interactionally relevant details of sequencing (e.g. overlap, latching), temporal aspects (pauses, acceleration/deceleration), phonological aspects (e.g. intensity, pitch, stretching, truncation, voice quality), and laughter. Moreover, the orthographic transcriptions are exhaustively PoS-tagged using the CLAWS web tagger (Garside and Smith 1997). ELAN-based transcriptions also provide exhaustive annotations of re-enactments (also referred to as (free) direct speech, constructed dialogue, etc.) as well as silent gestures (meaningful gestures that occur without accompanying speech). The multimodal data are derived from psychophysiological measurements and eye tracking. The psychophysiological measurements include, inter alia, electrodermal activity or GSR, which is indicative of emotional arousal (e.g. Peräkylä et al. 2015). Eye tracking produces data of two kinds: gaze direction and pupil size. In FreMIC, gazes are automatically recorded using the area-of-interest technology. Gaze direction is interactionally key, for example, in turn-taking (e.g. Auer 2021) and re-enactments (e.g. Pfeiffer and Weiss 2022), while changes in pupil size provide a window onto cognitive intensity (e.g. Barthel and Sauppe 2019). To demonstrate what opportunities FreMIC’s (combination of) transcriptions, annotations, and multimodal data open up for research in Interactional (Corpus) Linguistics, this article reports on interim results derived from work-in-progress.

Publicking/Privating: The Gestural Politics of Digital Spaces

The distinction between public and private spheres has become increasingly blurred in the digital age. As more aspects of life move online, where information is potentially visible to anyone, traditional barriers dividing public and private realms dissolve. This creates a default condition of publicness for much online activity. In response, Internet users have developed novel ways of demarcating contexts as more public or more private through “gestures” (Vilém Flusser) that range from literal bodily movements to highly context-dependent and media-specific signs. This essay argues that in digital environments, the notions of “public” and “private” take on active, performative dimensions as verbs — “publicking” and “privating.” Users engage in ongoing yet subtle negotiations to establish “spaces of appearance” (Hannah Arendt) and signify communication as directed toward distinct audiences. The essay criticizes classic theories of the public sphere as inadequate for digital life. It proposes recasting the distinction as fluid and gestural rather than stable and institutional. Although overlooked as trivial, micro-gestures of publicking and privating enable users to perform publicness and privateness in a time in which the public has become an act rather than a place. Studying these fleeting yet meaningful gestures provides insight into how users resist the default publicness of contemporary digital life.

Cebir Öğretim Sürecinin Jestler Bağlamında Göstergebilimsel Olarak İncelenmesi

This research aims to examine the gestures used by teachers and students in the algebra learning-teaching process and the purpose of these signs. The participants of this case study are 25 eigth graders and their mathematics teacher. The data were collected through video recordings and clinical interviews. Deductive analysis was used to analyze the data, while synchronic and diachronic analyses were utilized to analyze the signs in terms of semiotics. Findings revealed that the teacher used a bunch of iconic, metaphoric, deictic, and beat gestures during the algebra teaching process. The meaning of the iconic gestures that resulted in the study can be listed as centering and itemization, illustration of a table, constant term, preceding element, decrease, separation, base and power, rectangle and side, length, and vertical-horizontal. On the other hand, the metaphorical gestures attribute holding, equality, moving the term from one side to the other, numbers or letters, equality, distributing and hiding. The deictic gestures imply pointing out boards with fingers, board markers or hands, and indicating students and notebooks with fingers, while the beat gestures refer to lowering the hands from top to bottom, waving the fist up and down and moving the hand back and forth. It is seen that the teacher aims to strengthen her expression and support students' mathematical understanding through the gestures she uses.

Unlocking the Power of Gesture: Using Movement-Based Instruction to Improve First Grade Children's Spatial Unit Misconceptions.

Gestures are hand movements that are produced simultaneously with spoken language and can supplement it by representing semantic information, emphasizing important points, or showing spatial locations and relations. Gestures' specific features make them a promising tool to improve spatial thinking. Yet, there is recent work showing that not all learners benefit equally from gesture instruction and that this may be driven, in part, by children's difficulty understanding what an instructor's gesture is intended to represent. The current study directly compares instruction with gestures to instruction with plastic unit chips (Action) in a linear measurement learning paradigm aimed at teaching children the concept of spatial units. Some children performed only one type of movement, and some children performed both: Action-then-Gesture [AG] or Gesture-then-Action [GA]. Children learned most from the Gesture-then-Action [GA] and Action only [A] training conditions. After controlling for initial differences in learning, the gesture-then-action condition outperformed all three other training conditions on a transfer task. While gesture is cognitively challenging for some learners, that challenge may be desirable-immediately following gesture with a concrete representation to clarify that gesture's meaning is an especially effective way to unlock the power of this spatial tool and lead to deep, generalizable learning.

The parietal architecture binding cognition to sensorimotor integration: a multimodal causal study.

Despite human's praxis abilities are unique among primates, comparative observations suggest that these cognitive motor skills could have emerged from exploitation and adaptation of phylogenetically older building blocks, namely the parieto-frontal networks subserving prehension and manipulation. Within this framework, investigating to which extent praxis and prehension-manipulation overlap and diverge within parieto-frontal circuits could help in understanding how human cognition shapes hand actions. This issue has never been investigated by combining lesion mapping and direct electrophysiological approaches in neurosurgical patients. To this purpose, 79 right-handed left-brain tumour patient candidates for awake neurosurgery were selected based on inclusion criteria. First, a lesion mapping was performed in the early postoperative phase to localize the regions associated with an impairment in praxis (imitation of meaningless and meaningful intransitive gestures) and visuo-guided prehension (reaching-to-grasping) abilities. Then, lesion results were anatomically matched with intraoperatively identified cortical and white matter regions, whose direct electrical stimulation impaired the Hand Manipulation Task. The lesion mapping analysis showed that prehension and praxis impairments occurring in the early postoperative phase were associated with specific parietal sectors. Dorso-mesial parietal resections, including the superior parietal lobe and precuneus, affected prehension performance, while resections involving rostral intraparietal and inferior parietal areas affected praxis abilities (covariate clusters, 5000 permutations, cluster-level family-wise error correction P < 0.05). The dorsal bank of the rostral intraparietal sulcus was associated with both prehension and praxis (overlap of non-covariate clusters). Within praxis results, while resection involving inferior parietal areas affected mainly the imitation of meaningful gestures, resection involving intraparietal areas affected both meaningless and meaningful gesture imitation. In parallel, the intraoperative electrical stimulation of the rostral intraparietal and the adjacent inferior parietal lobe with their surrounding white matter during the hand manipulation task evoked different motor impairments, i.e. the arrest and clumsy patterns, respectively. When integrating lesion mapping and intraoperative stimulation results, it emerges that imitation of praxis gestures first depends on the integrity of parietal areas within the dorso-ventral stream. Among these areas, the rostral intraparietal and the inferior parietal area play distinct roles in praxis and sensorimotor process controlling manipulation. Due to its visuo-motor 'attitude', the rostral intraparietal sulcus, putative human homologue of monkey anterior intraparietal, might enable the visuo-motor conversion of the observed gesture (direct pathway). Moreover, its functional interaction with the adjacent, phylogenetic more recent, inferior parietal areas might contribute to integrate the semantic-conceptual knowledge (indirect pathway) within the sensorimotor workflow, contributing to the cognitive upgrade of hand actions.

Technological and social distractions at unsignalized and signalized campus crosswalks: A multi-stage naturalistic observation study

The use of personal listening devices (PLDs) is most prevalent among the college attending population. PLDs can be quite distracting, especially when its users are performing focused tasks such as street crossings. Such scenarios are commonplace on campuses of rural higher education institutes, where students must cross multiple unsignalized crosswalks to get to their destination. To evaluate the dangers of PLD use and pedestrian behavior while navigating crosswalks, 1274 pedestrians were systematically observed over nine hours at four crosswalks; naturalistic observations were followed by a survey of 135 pedestrians, and two focus groups with a total of eight participants following a unique three-stage protocol. Results show that societal distractions such as crossing in group settings or talking with other members of a group while crossing led to more risky pedestrian behavior, than did technological distractors such as PLDs or cellphone use. There was also a consensus on hand gesture meanings for pedestrian-driver communication despite the presence of relatively diverse population. Findings from this study can serve as guidelines to develop external human machine interfaces (eHMIs) for automated vehicles, and appropriate countermeasures to reduce pedestrian distractions at crosswalks of higher education institutes.

Multimodality Discourses Of The Miss Universe Beauty Pageant As An Arena Of Cultural Production

The Miss Universe beauty pageant is a cultural production arena where the construction of female beauty is formed through an ideal image that is slowly socialized by the media, becoming a standard in society. This research aims to find out and describe the construction of the meaning of beauty built through the Miss Universe beauty contest through the perspective of Discourse Analysis Multimodality, reveal the implied meaning of gestures, and understand the self-identity of the Miss Universe contestants. This qualitative research uses a critical paradigm using document studies, interviews, and literature. Data analysis used multimodality discourse analysis through data reduction, presentation, and conclusion. The results of this study indicate that: 1) The meaning of beauty at the Miss Universe 2021 pageant is constructed through the depiction of the contestants' perfection, which consists of their physique, attitude, and insight. The contestants' appearances have created beauty criteria that shape the idealization of the female figure and influence the audience's perspective on self-appreciation; 2) Overall, the most frequently shown contestant gestures give confidence and grace according to the message in the Miss Universe slogan, namely confidently beautiful; 3) The identity of the Miss Universe contestants reveals who, personality, where, and what culture is being performed with various styles and uniqueness. This uniqueness can be in the form of personality, interests, favorite food, way of talking, way of walking, and even how the contestants behave when responding to things and questions.

Nurse-Patient Communication and Relationship When Wearing Personal Protective Equipment: Nurses' Experience in a COVID-19 Ward.

Little is known about which communication strategies nurses carried out and whether the nurse-patient relationship has been altered due to the mandated use of personal protective equipment during the COVID-19 pandemic. This study describes how nurse-patient communication and relationships took place from the point of view of nurses engaged in caring for patients with COVID-19. A qualitative descriptive study design following COREQ guidelines was conducted. Semi-structured telephone interviews with nurses working in the COVID ward of an Italian university hospital were performed between September 2020 and June 2021. Ten nurses were recruited using convenience sampling. One overarching theme, three main themes, and nine sub-themes were identified. The overarching theme 'The in-out relationship: 'in here and out there' and 'inside me and out of me' included the main themes 'A closed system different from normal', 'Uncovering meaningful human gestures', and 'A deep experience to live''. The relational nature of nursing-where 'me and you' and the context are the main elements-leads nurses to find new ways of interacting and communicating with patients, even in a new situation that has never been experienced. Enhancing human gestures, thinking about new contexts of care, and educating new generations to maintain human-to-human interaction, regardless of the context of care, are the directives to be explored for creating the future of nursing care.

Shared and distinct routes in speech and gesture imitation: Evidence from stroke

Dual-route models of high-level (praxis) actions distinguish between an “indirect” semantic route mediating meaningful gesture imitation, and a “direct” sensory-motor route mediates meaningless gesture imitation. Similarly, dual-route language models distinguish between an indirect route mediating production and repetition of words, and a direct route mediating non-word repetition. Although aphasia and limb apraxia frequently co-occur following left-hemisphere cerebrovascular accident (LCVA), it is unclear which aspects of these functional-neuroanatomic dual-route architectures are shared across praxis and language domains. This study focused on gesture imitation to test the hypothesis that semantic information (and portions of the indirect route) are shared across domains, whereas two distinct dorsal routes mediate sensory-motor mapping. Forty chronic LCVA and 17 neurotypical controls completed semantic memory and language tasks and imitated 3 types of gesture stimuli: (1) labeled/“named” meaningful, (2) unnamed meaningful, and (3) meaningless gestures. The comparison of accuracy between meaningless versus unnamed meaningful gestures examined the benefits of semantic information, while the comparison of unnamed meaningful versus named meaningful imitation examined additional benefits of linguistic cueing. Mixed-effects models examined group by task interaction effects on gesture ability. We found that for patients with LCVA, unnamed meaningful gestures were imitated more accurately than meaningless gestures, suggesting that semantic information was beneficial, but there was no benefit of labeling. Reduced benefit of semantic information on gesture accuracy was associated with lesions to inferior frontal and posterior temporal regions as well as semantic memory performance on a pictorial (non-gesture) task. In contrast, there was no relationship between meaningless gesture imitation and nonword repetition, indicating that measures of direct route performance are not associated across language and action. These results provide preliminary evidence that portions of the indirect semantic route are shared across the language and action domains, while two direct sensory-motor mapping routes mediate word repetition and gesture imitation.

Towards a great ape dictionary: Inexperienced humans understand common nonhuman ape gestures.

In the comparative study of human and nonhuman communication, ape gesturing provided the first demonstrations of flexible, intentional communication outside human language. Rich repertoires of these gestures have been described in all ape species, bar one: us. Given that the majority of great ape gestural signals are shared, and their form appears biologically inherited, this creates a conundrum: Where did the ape gestures go in human communication? Here, we test human recognition and understanding of 10 of the most frequently used ape gestures. We crowdsourced data from 5,656 participants through an online game, which required them to select the meaning of chimpanzee and bonobo gestures in 20 videos. We show that humans may retain an understanding of ape gestural communication (either directly inherited or part of more general cognition), across gesture types and gesture meanings, with information on communicative context providing only a marginal improvement in success. By assessing comprehension, rather than production, we accessed part of the great ape gestural repertoire for the first time in adult humans. Cognitive access to an ancestral system of gesture appears to have been retained after our divergence from other apes, drawing deep evolutionary continuity between their communication and our own.

Gesture meaning modulates the neural correlates of effector-specific imitation deficits in left hemisphere stroke

BackgroundPrevious studies on left hemisphere (LH) stroke patients reported effector-specific (hand, fingers, bucco-facial) differences in imitation performance. Furthermore, imitation performance differed between meaningless (ML) and meaningful (MF) gestures. Recent work suggests that a gesture’s meaning impacts the body-part specificity of gesture imitation. MethodsWe tested the hypothesis that the gesture’s meaning (ML vs MF) affects the lesion correlates of effector-specific imitation deficits (here: bucco-facial vs arm/hand gestures) using behavioural data and support vector regression-based lesion-symptom mapping (SVR-LSM) in a large sample of 194 sub-acute LH stroke patients. ResultsBehavioural data revealed a significant interaction between the effector used for imitation and the meaning of the imitated gesture. SVR-LSM analyses revealed shared lesion correlates for impaired imitation independent of effector or gesture meaning in the left supramarginal (SMG) and superior temporal gyri (STG). Besides, within the territory of the left middle cerebral artery, impaired imitation of bucco-facial gestures was associated with more anterior lesions, while arm/hand imitation deficits were associated with more posterior lesions. MF gestures were specifically associated with lesions in the left inferior frontal gyrus and the left insular region. Notably, an interaction of effector-specificity and gesture meaning was also present at the lesion level: A more pronounced difference in imitation performance between the effectors for ML (versus MF) gestures was associated with left-hemispheric lesions in the STG, SMG, putamen, precentral gyrus and white matter tracts. ConclusionThe current behavioural data show that ML gestures are particularly sensitive in assessing effector-specific imitation deficits in LH stroke patients. Moreover, a gesture’s meaning modulated the effector-specific lesion correlates of bucco-facial and arm/hand gesture imitation. Hence, it is crucial to consider gesture meaning in apraxia assessments.

Hand Gestures in Japanese Communication

In nonverbal communication, gestures are performed to be used in conjunction with verbal communication. Japan is a country that uses more nonverbal communication compared to Indonesia. When communicating, the Japanese often use hand gestures. This hand gesture is very important to know for Japanese language learners because it has many variations of use in body language learning. The same hand gesture sometimes has a difference in the intent to be conveyed between the sender and recipient of the message. Therefore, this study was conducted to describe the form of nonverbal communication, especially the form of hand gestures and the meaning of hand gestures. This study used Japanese drama as a source of data that can be seen and studied by Japanese language learners. This study used a descriptive method to describe the differences in the hand gestures of adult Japanese and children in communicating. The results of this study are expected that Japanese language learners, especially students of the Japanese Language Education Program FKIP Universitas Riau, can understand the culture of the Japanese people and that to suppress verbal communication, they often use hand gestures.

A Novel Machine Learning–Based Hand Gesture Recognition Using HCI on IoT Assisted Cloud Platform

Machine learning is a technique for analyzing data that aids the construction of mathematical models. Because of the growth of the Internet of Things (IoT) and wearable sensor devices, gesture interfaces are becoming a more natural and expedient human-machine interaction method. This type of artificial intelligence that requires minimal or no direct human intervention in decision-making is predicated on the ability of intelligent systems to self-train and detect patterns. The rise of touch-free applications and the number of deaf people have increased the significance of hand gesture recognition. Potential applications of hand gesture recognition research span from online gaming to surgical robotics. The location of the hands, the alignment of the fingers, and the hand-to-body posture are the fundamental components of hierarchical emotions in gestures. Linguistic gestures may be difficult to distinguish from nonsensical motions in the field of gesture recognition. Linguistic gestures may be difficult to distinguish from nonsensical motions in the field of gesture recognition. In this scenario, it may be difficult to overcome segmentation uncertainty caused by accidental hand motions or trembling. When a user performs the same dynamic gesture, the hand shapes and speeds of each user, as well as those often generated by the same user, vary. A machine-learning-based Gesture Recognition Framework (ML-GRF) for recognizing the beginning and end of a gesture sequence in a continuous stream of data is suggested to solve the problem of distinguishing between meaningful dynamic gestures and scattered generation. We have recommended using a similarity matching-based gesture classification approach to reduce the overall computing cost associated with identifying actions, and we have shown how an efficient feature extraction method can be used to reduce the thousands of single gesture information to four binary digit gesture codes. The findings from the simulation support the accuracy, precision, gesture recognition, sensitivity, and efficiency rates. The Machine Learning-based Gesture Recognition Framework (ML-GRF) had an accuracy rate of 98.97%, a precision rate of 97.65%, a gesture recognition rate of 98.04%, a sensitivity rate of 96.99%, and an efficiency rate of 95.12%.

Coordinating and sharing gesture spaces in collaborative reasoning

Abstract In collaborative reasoning about what causes the seasons, phases of the moon, and tides, participants (three to four per group) introduce ideas by gesturing depictively in personal space. Other group members copy and vary these gestures, imbuing their gesture spaces with similar conceptual properties. This leads at times to gestures being produced in shared space as members elaborate and contest a developing group model. Gestures in the shared space mostly coincide with conversational turns; more rarely, participants gesture collaboratively as they enact a joint conception. An emergent shared space is sustained by the joint focus and actions of participants and may be repositioned, reoriented, or reshaped to meet changing representational demands as the discourse develops. Shared space is used alongside personal spaces, and further research could shed light on how gesture placement and other markers (such as eye gaze) contribute to the meaning or function of gestures in group activity.

Forward Hand Gesture Spotting and Prediction Using HMM-DNN Model

Automatic key gesture detection and recognition are difficult tasks in Human–Computer Interaction due to the need to spot the start and the end points of the gesture of interest. By integrating Hidden Markov Models (HMMs) and Deep Neural Networks (DNNs), the present research provides an autonomous technique that carries out hand gesture spotting and prediction simultaneously with no time delay. An HMM can be used to extract features, spot the meaning of gestures using a forward spotting mechanism with varying sliding window sizes, and then employ Deep Neural Networks to perform the recognition process. Therefore, a stochastic strategy for creating a non-gesture model using HMMs with no training data is suggested to accurately spot meaningful number gestures (0–9). The non-gesture model provides a confidence measure, which is utilized as an adaptive threshold to determine where meaningful gestures begin and stop in the input video stream. Furthermore, DNNs are extremely efficient and perform exceptionally well when it comes to real-time object detection. According to experimental results, the proposed method can successfully spot and predict significant motions with a reliability of 94.70%.

Differences in Praxis Errors in Autism Spectrum Disorder Compared to Developmental Coordination Disorder.

This study aimed to better understand how autism spectrum disorder (ASD) and developmental coordination disorder (DCD) differ in types of praxis errors made on the Florida Apraxia Battery Modified (FAB-M) and the potential relationships between praxis errors and social deficits in ASD. The ASD group made significantly more timing sequencing errors in imitation of meaningful gestures, as well as more body-part-for-tool errors during gesture-to-command compared to the other two groups. In the ASD group, increased temporal errors in meaningful imitation were significantly correlated with poorer affect recognition and less repetitive behaviors. Thus, in ASD, aspects of imitation ability are related to socioemotional skills and repetitive behaviors.

One Year of School and The Island

One Year of School and The Island

GesSure: A Robust Face-Authentic Enabled Dynamic Gesture Recognition GUI Application

Using physical interactive devices like mouse and keyboards hinders naturalistic human-machine interaction and increases the probability of surface contact during a pandemic. Existing gesturerecognition systems do not possess user authentication, making them unreliable. Static gestures in current gesture-recognition technology introduce long adaptation periods and reduce user compatibility. Our technology places a strong emphasis on user recognition and safety. We use meaningful and relevant gestures for task operation, resulting in a better user experience. This paper aims to design a robust, faceverification-enabled gesture recognition system that utilizes a graphical user interface and primarily focuses on security through user recognition and authorization. The face model uses MTCNN and FaceNet to verify the user, and our LSTM-CNN architecture for gesture recognition, achieving an accuracy of 95% with five classes of gestures. The prototype developed through our research has successfully executed context-dependent tasks like save, print, control video-player operations and exit, and context-free operating system tasks like sleep, shut-down, and unlock intuitively. Our application and dataset are available as open source.

A Review Paper on Real Time Sign Language Detection for the Deaf and Dumb

Hand gesture is one of the styles used in sign language for non-verbal communication. It's most generally used by deaf &amp; dumb people who have hearing or speech problems to communicate among themselves or with normal people. Plathora sign language systems had been developed by numerous makers around the world but they're neither flexible nor cost-effective for the end druggies. Hence, it's a software which presents a system prototype that's suitable to automatically recognize sign language to help deaf and dumb people to communicate more effectively with each other or normal people. Dumb people are generally deprived of normal communication with other people in the society, also normal people find it hard to understand and communicate with them. These people have to depend on an interpreter or on some kind of visual communication. An interpreter won’t be always available and visual communication is substantially delicate to understand. As a normal person is ignorant of the grammar or meaning of numerous gestures that are part of a sign language, it's primarily limited to their families and/ or deaf and dumb community