Gesture Patterns Research Articles

Automatic Gesture-Based Arabic Sign Language Recognition: A Federated Learning Approach

Featuring machine learning algorithms for recognizing hand gesture patterns adjusted for individuals with disabilities is an expanding trend in assisted living. This paper addresses the challenge of interpreting the semantics of image-based hand gestures by introducing a federated deep learning architecture for Arabic sign language recognition. The proposed model manages distributed learning through a client-server paradigm, wherein several edge nodes collaborate to jointly learn the discriminative features of confidential data without breaching its privacy. This model will enable more accessibility for people with deafness or impairment using image gestures. The federated learning procedure is primarily based on the ResNet32 deep backbone and federated averaging mechanism. The experimental results show the effectiveness of the proposed FL model, achieving an accuracy of 98.30% with 33 seconds on average for each client in a single training round. This demonstrates its high capabilities in recognizing Arabic sign language and improving the communication experience for people with disabilities.

Is vision necessary for the timely acquisition of language-specific patterns in co-speech gesture and their lack in silent gesture?

Blind adults display language-specificity in their packaging and ordering of events in speech. These differences affect the representation of events in co-speech gesture-gesturing with speech-but not in silent gesture-gesturing without speech. Here we examine when in development blind children begin to show adult-like patterns in co-speech and silent gesture. We studied speech and gestures produced by 30 blind and 30 sighted children learning Turkish, equally divided into 3 age groups: 5-6, 7-8, 9-10 years. The children were asked to describe three-dimensional spatial event scenes (e.g., running out of a house) first with speech, and then without speech using only their hands. We focused on physical motion events, which, in blind adults, elicit cross-linguistic differences in speech and co-speech gesture, but cross-linguistic similarities in silent gesture. Our results showed an effect of language on gesture when it was accompanied by speech (co-speech gesture), but not when it was used without speech (silent gesture) across both blind and sighted learners. The language-specific co-speech gesture pattern for both packaging and ordering semantic elements was present at the earliest ages we tested the blind and sighted children. The silent gesture pattern appeared later for blind children than sighted children for both packaging and ordering. Our findings highlight gesture as a robust and integral aspect of the language acquisition process at the early ages and provide insight into when language does and does not have an effect on gesture, even in blind children who lack visual access to gesture. RESEARCH HIGHLIGHTS: Gestures, when produced with speech (i.e., co-speech gesture), follow language-specific patterns in event representation in both blind and sighted children. Gestures, when produced without speech (i.e., silent gesture), do not follow the language-specific patterns in event representation in both blind and sighted children. Language-specific patterns in speech and co-speech gestures are observable at the same time in blind and sighted children. The cross-linguistic similarities in silent gestures begin slightly later in blind children than in sighted children.

Game psychotherapy intervention based on entertainment interactive robots for preventing depression in university students

This study designs an entertainment interactive robot system for gaming psychotherapy intervention and evaluates its effectiveness in preventing depression in students. In terms of the architecture and hardware of the entertainment interactive robot system, the robot adopts high-performance processors and sensors to achieve the ability to quickly respond and accurately perceive user actions. In terms of software, a mathematical model was established based on the structure and kinematic principles of the robot to describe its motion. Preprocess user gesture data by analyzing the features and patterns of user gestures, extracting key action information, and using machine learning algorithms or deep learning models to classify and recognize gestures to determine user intent and action types. Evaluate action recognition and effectiveness, and verify the accuracy and reliability of the gesture interaction module through comparison and testing with actual user gestures. The study designed an entertainment interactive robot assisted game that provides a fun and interactive experience to attract students’ attention and stimulate positive emotions. Through game design and behavioral pattern evaluation, the research team evaluated the preventive effect of entertainment interactive robot assisted games on depression and proposed optimization strategies.

Scope of Negation, Gestures, and Prosody: The English Negative Quantifier as a Case in Point.

The present paper examines how English native speakers produce scopally ambiguous sentences and how they make use of gestures and prosody for disambiguation. As a case in point, the participants in the present study produced the English negative quantifiers. They appear in two different positions as (1) The election of no candidate was a surprise (a: 'for those elected, none of them was a surprise'; b: 'no candidate was elected, and that was a surprise') and (2) no candidate's election was a surprise (a: 'for those elected, none of them was a surprise'; b: # 'no candidate was elected, and that was a surprise.' We were able to investigate the gesture production and the prosodic patterns of the positional effects (i.e., a-interpretation is available at two different positions in 1 and 2) and the interpretation effects (i.e., two different interpretations are available in the same position in 1). We discovered that the participants tended to launch more head shakes in the (a) interpretation despites the different positions, but more head nod/beat in the (b) interpretation. While there is not a difference in prosody of no in (a) and (b) interpretation in (1), there are pitch and durational differences between (a) interpretations in (1) and (2). This study points out the abstract similarities across languages such as Catalan and Spanish (Prieto et al. in Lingua 131:136-150, 2013. 10.1016/j.lingua.2013.02.008; Tubau et al. in Linguist Rev 32(1):115-142, 2015. 10.1515/tlr-2014-0016) in the gestural movements, and the meaning is crucial for gesture patterns. We emphasize that gesture patterns disambiguate ambiguous interpretation when prosody cannot do so.

Gentle Transnational Spirits

Kamisato Yudai’s Immigrant Ghost Stories (2022) evokes past generations that suffered violence yet nevertheless haunt the present as gentle spirits, whether as reincarnated animals, reincarnated people, or repeated patterns of physical gesture and movement. Blurring the borders between fiction and documentary, storytelling and physical theatre, the work stages transnationality as both an economic practice and a sociocultural necessity, encouraging us to acknowledge the heterogeneity of Japanese–Latin American and other Japanese transnational identities.

A cross-cultural analysis of the gestural pattern of surprise and surprise-disapproval questions

Abstract In this article, we address the issue concerning the gestural patterns in expressing surprise and disapproval across various languages and cultures. The results obtained so far point to an interesting, and in a sense rather surprising, uniformity. We consider two types of special questions: counter-expectational questions expressing surprise and surprise-disapproval questions, i.e., sentences expressing surprise with a negative orientation, and adopt an experimental design involving sentence repetition and spontaneous production. We focus on the realization of these sentences in Vietnamese, Korean and Japanese, which we compare with the results previously obtained for Italian and replicated for Neapolitan, Spanish and German. Our research is based on the Minimalist theoretical framework developed by Chomsky and scholars in the tradition of generative grammar.

Variable effects of speakers’ visual cues and accent on L2 listening comprehension: A mixed-methods approach

Few studies have explored the influence of a speaker’s accent and visual (facial and gestural) cues on second-language (L2) listening comprehension. The current mixed-methods between-groups design investigated: (1) the effects of accent and visual cues on Arab students’ comprehension of recorded lectures delivered by two speakers: first language (L1) American English and second language (L2) English (strong Vietnamese accent), (2) students’ assessments of the comprehensibility and English nativelikeness of the accents, and (3) students’ perceptions of and preferences for visual cues in communication. A total of 120 Arab university students were distributed across three stimulus conditions: auditory-visual (AV)-gesture-face, AV-face, and A-only within two accent conditions (L1, L2 English). Materials included: a listening comprehension task, 9-point rating scales for comprehensibility and nativelikeness of accent, and a speaker’s accent and visual cues (SAVC) questionnaire. ANCOVAs were conducted on listening comprehension scores and the ratings, with students’ listening proficiency as the covariate. Students with higher listening proficiency received higher comprehension scores and tended to recognize the L1 English accent as nativelike; however, proficiency was not significantly related to their comprehensibility ratings of either accent. The L1 English speaker’s gestures were associated with higher comprehensibility ratings; the L2 English speaker’s gestures were associated with a stronger nonnativelike accent, which students found frustrating to comprehend. Although SAVC responses revealed a general preference for facial cues and gestures in daily communication, the L2 English speaker’s gestures in the study were not considered helpful. Thematic analysis of interviews emphasized: (1) the variable effectiveness of visual cues, (2) the important role of topic and accent familiarity, (3) the relationship between listening proficiency and interlocutor/instructor preference, and (4) strategies to deal with unfamiliar accents. Pedagogical implications include increased awareness of variability in visual cues across speakers and the need to build familiarity with various accents and gesture patterns.

Developmental organization of the graphic gesture with a pre-scruptural task to assess handwriting

IntroductionThe literature mainly focused on spatio-temporal and kinematics parameters of tracing letters or words using digitizing tablets, no recent research has previously studied the developmental prerequisites of the organization of handwriting.ObjectivesWe aimed to aimed to investigate and validate the developmental organization of the graphic gesture with a pre-scruptural task.Methods122 typically developing right-handed elementary school children (grades 1st to 5th) aged from 6 to 11;3 years old were recruited. The axe postural and arm gestural features were video-recorded with analysis in 2D reconstruction. Spatial (length, size, regularity, slope of the line…), temporal (drawing time, pause time) and kinematic measures (velocity, peak velocity) were collected with a digital pen independent connected to an analysis software tool. External validity was studied in relation with the standardized handwriting scale BHK. The child has to draw a line of cycloid loops (from left to right drawn in an anti-clockwise direction) across the width of an A4 size unlined half sheet of white paper (containing non-visible watermarks to provide the location of the pen) free to move on the table, after observing the dynamic model on the iPad placed in front of him.ResultsFive main patterns of inter-segmental displacement gestures were found for the production of the line of loops with a significant developmental progress from grades 1st to 5th. Findings showed significant economic rotation movement with forearm rotation around the elbow in 4th and 5th grade, with the elbow tending significantly to be static on the table (p = 2.43e-16), wrist on the table and (p =0.02) and in half-supine position (p =0·001), tri-digital grasp of the pen (p =3.81e-08). Moreover, the mean pressure applied on the pen decrease at 4 and 5th grades and it is correlated to deleterious spatial-temporal and kinematic parameters.ConclusionsThe results of our study provide the first developmental grade and age-related normative data in the developmental genesis of the graphomotor gesture and with the spatio-temporal /kinematic measures. The more mature the gesture, the more there is a decrease in degrees of freedom of movement and stabilization of the joints that is fixed, as well as the presence of a distal flexion movement extending fingers in synergy with the rotation of the arm around the elbow. Furthermore, the task of copying of loops in ecological settings appears to be a good predictor for legibility and writing speed. Those data can account for the mechanisms of motor programming necessary to the automatization of the future gesture of handwriting.Disclosure of InterestNone Declared

Adaptive Threshold Filtering to Reduce Noise in Elderly Activity Classification Using Bi-LSTM

As the global population ages, there is an increasing need to provide better care and support for older individuals. Deep learning support to accurately predict elderly activities is very important to develop. This research discusses a new model integrating filtering techniques using adaptive thresholds with Bidirectional - Long Short-Term Memory (Bi-LSTM) networks. The problem of activity prediction accuracy, mainly due to noise or irrational measurements in the dataset, is solved with adaptive thresholds. Adaptive characteristics at the threshold are needed because each individual has different activity patterns. Experiments using the HAR70+ dataset describe the activity patterns of 15 elderly subjects and the gesture patterns of 7 activities. Based on body movement patterns, the elderly can be classified as using walking aids. The proposed model design obtains an accuracy of 94.71% with a loss of 0.1984.

Inferring in-air gestures in complex indoor environment with less supervision

People have high demands for comfort and technology in indoor environments. Gestures, as a natural and friendly human computer interaction (HCI) method, have received widespread attention and have been the subject of many research studies. Traditional approaches are based on wearable devices and cameras, which can be cumbersome to operate and infringe upon users’ privacy. Millimeter-wave (mmWave) radar avoids these problems by detecting gestures in a noninvasive manner. However, it encounters practical challenges in complex indoor environments, such as dynamic disturbance from surroundings, variable usage conditions and diverse gesture patterns, which conventionally require considerable manual effort to address. In this paper, we attempt to minimize human supervision and propose a noninvasive gesture recognition method named RaGe that involves a commercial mmWave indoor radar. First, a parameter optimization framework considering gesture prior constraints is proposed for radar configuration, which functions to weaken the disturbance from surroundings. Second, we alleviate data shortages in variable usage conditions and achieve low-cost data augmentation by applying affine transformations. Third, we combine deformable convolution operations with an unsupervised attention mechanism, thus exploring the intrinsic features involved in diverse gesture patterns. Experimental results show that RaGe is able to recognize 7 gestures with 99.3% accuracy and less human supervision, surpassing the state-of-the-art methods in comparative experiments.

Gesture imitation performance in community-dwelling older people: assessment of a gesture imitation task in the screening and diagnosis of mild cognitive impairment and dementia.

Gesture imitation, a simple tool for assessing visuospatial/visuoconstructive functions, is reportedly useful for screening and diagnosing dementia. However, gesture imitation performance in healthy older adults is largely unknown, as are the factors associated with lower performance. To address these unknowns, we examined the gesture imitation performance of a large number of community-dwelling older adults aged ≥65 years in Arao City, Kumamoto Prefecture (southern Japan). The examiner presented the participants with eight gesture patterns and considered it a success if they could imitate them within 10 s. The success rate of each gesture imitation was calculated for three diagnostic groups: cognitively normal (CN) (n = 1184), mild cognitive impairment (MCI) (n = 237), and dementia (n = 47). Next, we reorganised the original gesture imitation battery by combining six selected gestures with the following scoring method: if the participants successfully imitated the gestures, immediately or within 5 s, two points were assigned. If they succeeded within 5-10 s, one point was assigned. The sensitivity and specificity of the battery were investigated to detect the dementia and MCI groups. Factors associated with gesture imitation battery scores were examined. Except one complex gesture, the success rate of imitation in the CN group was high, approximately 90%. The sensitivity and specificity of the gesture imitation battery for discriminating between the dementia and CN groups and between the MCI and CN groups were 70%/88%, and 45%/75%, respectively. Ageing, male sex, and a diagnosis of dementia or MCI were associated with lower scores on the gesture imitation battery. Gesture imitation tasks alone may not be sufficient to detect MCI. However, by combining gestures with set time limits, gesture imitation tasks can be a low-burden and effective method for detecting dementia, even in community medicine, such as during health check-ups.

Surgical gestures can be used to assess surgical competence in robot-assisted surgery

To collect validity evidence for the assessment of surgical competence through the classification of general surgical gestures for a simulated robot-assisted radical prostatectomy (RARP). We used 165 video recordings of novice and experienced RARP surgeons performing three parts of the RARP procedure on the RobotiX Mentor. We annotated the surgical tasks with different surgical gestures: dissection, hemostatic control, application of clips, needle handling, and suturing. The gestures were analyzed using idle time (periods with minimal instrument movements) and active time (whenever a surgical gesture was annotated). The distribution of surgical gestures was described using a one-dimensional heat map, snail tracks. All surgeons had a similar percentage of idle time but novices had longer phases of idle time (mean time: 21 vs. 15 s, p < 0.001). Novices used a higher total number of surgical gestures (number of phases: 45 vs. 35, p < 0.001) and each phase was longer compared with those of the experienced surgeons (mean time: 10 vs. 8 s, p < 0.001). There was a different pattern of gestures between novices and experienced surgeons as seen by a different distribution of the phases. General surgical gestures can be used to assess surgical competence in simulated RARP and can be displayed as a visual tool to show how performance is improving. The established pass/fail level may be used to ensure the competence of the residents before proceeding with supervised real-life surgery. The next step is to investigate if the developed tool can optimize automated feedback during simulator training.

Silent Expressions Unveiled: Deep Learning for British and American Sign Language Detection

Sign language is an efficient means of communication for the hearing impaired. The main goal of this research is to recognize both American and British sign languages using computer vision algorithms and neural networks. Our work addresses the critical need for inclusive and accessible communication for the Deaf community. For the study, we compiled a large dataset of her ASL and BSL gestures. Preprocess the data to improve feature extraction and reduce noise. Our model architecture uses a convolutional neural network (CNN) to capture the temporal patterns of finger gestures. This allows you to use Mediapipe keypoint detection within your model. The end result is a comprehensive model that can accurately and quickly recognize and classify sign language gestures. We further improve the usability of the system by integrating keyword recognition into the model, allowing sign language sentences to be decoded and translated into text. Our test results demonstrate how well our deep learning strategy works to achieve high efficiency and accuracy in both ASL and BSL detection. This program helps bridge the gap between those who use sign language and those who do not. Our new approach not only accurately recognizes American and British sign language, but also seamlessly integrates real-time feedback to help users make immediate improvements to improve their sign language skills. We can do better. Moreover, our approach is flexible enough to be used in a variety of settings, ensuring its usefulness in educational environments and providing sign language learners with a fun and dynamic learning environment. Explore and exploit the full potential of Mediapipe Keypoint using CNNs and also leverage the efficiency of computer vision.

SIBI (Sistem Bahasa Isyarat Indonesia) berbasis Machine Learning dan Computer Vision untuk Membantu Komunikasi Tuna Rungu dan Tuna Wicara

The Indonesian Sign Language System (SIBI) is used to translate sign language into text or speech. SIBI helps improve communication between people using sign language and those who do not understand it. Unlike commonly used languages, SIBI sign language is less known to most people due to a lack of interest. To address this, an artificial intelligence-based application was developed, focusing on deep learning to recognize SIBI sign language hand movements in real-time. The model was created with 20 epochs, a batch size of 16, and a learning rate of 0.001. It consists of 13 layers, with the ReLU activation function used for the input layer, while the output layer uses Sigmoid. The ADAM optimizer was used to expedite the model creation process. The image dataset used had a size of 300x300 pixels. In the classification testing of the SIBI alphabet results in this study, it was tested using distance tests. The distance between the webcam and the SIBI language speaker was divided into two categories: 40 cm and 60 cm. For a 40-cm distance, an accuracy of 87.50% was obtained, and for a 60-cm distance, an accuracy of 79.17% was achieved. One limitation of this study is that two alphabets, J and Z, were not included in the dataset. This is because recognition of these two alphabets requires not only finger pattern recognition but also recognition of their gesture patterns.

Principal's Communication Style and its Impact on the Performance of State Civil Apparatus

The leader is an authority figure in which case every gesture pattern and behavior can have an impact on the conditions and responses of subordinates and employees. The purpose of this study is to determine the impact and influence of the communication style and work motivation that the Principal of State Senior High School 57 Jakarta applies and has on the performance of the State Civil Apparatus (ASN) there. The method used in this research is a qualitative research method with the type of case study, the use of case studies because researchers can understand in depth the context, process, and complexity of the phenomenon being studied. The data collection techniques used are interviews, observation, and documentation. The results of the research found in the field show that the communication that exists between the Principal and all Teachers at SMAN 57 Jakarta is an effective communication style. Effective communication style allows the Principal to convey vision and goals, provide support to Teachers, promote collaboration, build trust, and solve problems together. In addition, based on the research conducted, it can be concluded that effective communication style is a manifestation of one of Horald-Lasswell's communication theories.

Looking at gesture: The reciprocal influence between gesture and conversation

IntroductionThere is limited research in group communication treatment for people with aphasia but existing studies report benefits of gesture to support conversation. Gesture supports conversation through recipient design features and reducing linguistic demands of lexical retrieval and formulation. Additionally, gesture serves an affiliative function. However, the relationship between gesture use and gestural capacity has not been widely examined. As part of a larger study on group cohesiveness and conversation, this investigation examined the patterns of co-speech gesture within authentic conversations among persons with aphasia to discern the functions of gesture use for the participants, changes in the use of gesture over time, and the relationship between gesture use and gesture ability. MethodsConversation Analysis (CA) was applied in an embedded case-study design. Three participants received an academic semester of group and individual conversation-based treatment according to Facilitating Authentic Conversation (Damico et al., 2015). Four conversations from the treatment were selected and transcribed for multi-modality communication with CA conventions applied, and then cyclically analysed for patterns of gesture. ResultsParticipants demonstrated gesture that served social and linguistic functions: ratifying clinicians’ proxy turns, turn-allocation, turn repair, relaying novel visual information, emphasizing content, demonstrating affiliation with the prior speaker, demonstrating their assessment others’ talk, and demonstrating humor. All three participants showed an increased rate of gesture per turn and increasingly used gesture to repair conversation breakdown. Increased gesture use over the course of the semester coincided with increased scores for pantomime on the Porch Index of Communicative Ability (Porch, 1981, PICA). ConclusionIndividuals with aphasia demonstrated increased use of gesture for varied purposes and improved gestural processing following a semester of conversation-based treatment. This is significant because gesture is an effective support for the repair of conversation breakdown typical of persons with aphasia.

Multimodal languaging: Reification profiles in language and gesture

Abstract The paper explores multimodal languaging of objects and words as an encultured practice. We report the results of the experiment where the participants explained the difference between close synonyms, enacting them as either objects or words in speech and gesture. The basic claim of the study is that speech and gesture as second-order language reflect the way they were acquired in developing the knowledge of objects and words in reification image-schemas. We found that i) object reification occurs twice more often; ii) enacting objects and words is affected by the same image-schemas expressed in speech and gesture types, still there are differences in their distribution in speech. This observation evidences that both gesture-specific and language-specific notions are part of languaging since the speech and gesture patterns present the way of packaging the shared idea of objects of reference.

Bilingual mothers and children gesture differently across native and second languages

Children from bilingual families are typically exposed to more diverse communication practices compared to those from monolingual households. Characterizing bilinguals’ gesture use can provide insight into children’s developmental trajectory in their two languages and the cross-linguistic effects of scaffolding from adults. Gestural patterns of Thai-English bilingual mothers and their preschool-age children were examined in three communicative tasks (prompted reminiscing, book sharing, and toy play) during two sessions (one in Thai and one in English). Gestures were categorized into representational, deictic, conventional, beat, and all. The results revealed that, as early as preschool, bilinguals gesticulate differently in their two languages. The findings highlight the importance of evaluating bilinguals’ nonverbal communication in both languages for a comprehensive picture of their linguistic profiles, especially among speakers of understudied languages. Our understanding of how gestures and speech function as an integrated communicative system would be incomplete without the inclusion of linguistic populations that are underrepresented in research.

Android-Based Application for Real-Time Indonesian Sign Language Recognition Using Convolutional Neural Network

Individuals with hearing or speech impairments face challenges in communicating with others, requiring special techniques to express their thoughts and feelings. Sign language is an alternative way to communicate using a specific pattern of hand gestures to deliver messages instead of verbal speaking (oral communication). Unfortunately, most people do not know how to use and read sign language. Because of its complexity and many types of sign language worldwide, only well-trained personnel could use it as a communication medium. This research provides a solution in the form of a machine-learning Android-based application designed to recognize sign language captured by a smartphone camera and translate it into Latin characters. The recognition accommodates Convolutional Neural Network (CNN), one of the popular deep learning algorithms. This application recognizes 26 characters of Indonesian Sign Language (Bahasa Isyarat Indonesia/BISINDO) alphabets using MobileNetV3 architecture. To build the data model, dataset images were collected from 5 different models demonstrating 26 BISINDO characters in various lighting, background, and hand gesture position. These dataset images were also generated using image augmentation process to achieve the randomness by adjusting the image rotation, noise, and brightness. Based on the testing result using 6,240 dataset images, the application has 75.38% accuracy in recognizing Indonesian Sign Language alphabets.

Towards a speech–gesture profile of pragmatic markers: The case of “you know”

Our study introduces a novel approach for examining the functional relationship between the pragmatic marker “you know” and its co-occurring gestures. It draws on 401 instances of “you know” identified in the Nottingham Multimodal Corpus. Based on the speech patterns in the data, a framework of six functions of “you know” was established (i.e., ‘editing’, ‘introducing’, ‘inviting inferences’, ‘elaborating’, ‘marking reported speech’, ‘approximating’). 74 gestures that primarily co-occur with “you know” were selected for analysis based on a functional framework mainly adopted from Kendon (2004): pragmatic, referential, beat and deictic gestures. The main results show that three of the functions (i.e., ‘editing’, ‘introducing’ and ‘elaborating’) account for 81% of the whole dataset, and share a similar function of emphasizing the intention of presenting further information. In addition, they tend to co-occur with the pragmatic gestures that embody a similar pragmatic function. The results illustrate the functional coordination between pragmatic markers and gestures, and demonstrate the value of using a multimodal corpus pragmatic approach for exploring the patterns of speech and gesture, and for developing a more nuanced profile of pragmatic markers that, in the history of pragmatics, have largely been described on the sole basis of their speech instantiation.