Complex Gestures Research Articles

Cascadable optical nonlinear activation function based on Ge-Si.

To augment the capabilities of optical computing, specialized nonlinear devices as optical activation functions are crucial for enhancing the complexity of optical neural networks. However, existing optical nonlinear activation function devices often encounter challenges in preparation, compatibility, and multi-layer cascading. Here, we propose a cascadable optical nonlinear activation function architecture based on Ge-Si structured devices. Leveraging dual-source modulation, this architecture achieves cascading and wavelength switching by compensating for loss. Experimental comparisons with traditional Ge-Si devices validate the cascading capability of the new architecture. We first verified the versatility of this activation function in a MNIST task, and then in a multi-layer optical dense neural network designed for complex gesture recognition classification, the proposed architecture improves accuracy by an average of 23% compared to a linear network and 15% compared to a network with a traditional activation function architecture. With its advantages of cascadability and high compatibility, this work underscores the potential of all-optical activation functions for large-scale optical neural network scaling and complex task handling.

Hand Gesture Recognition for Interactive On-Screen Drawing with Real-Time Subtitling for Teaching

This research presents the development of an innovative educational tool leveraging hand gesture recognition for interactive on-screen drawing with real-time subtitling. Built on OpenCV and Python, the platform enables educators to draw and manipulate digital content through intuitive hand gestures, enhancing classroom engagement. Additionally, real-time subtitle generation offers accessibility for hearing-impaired students by converting drawn content into text notes. This system addresses key challenges in traditional digital drawing tools, such as limited interaction methods and accessibility, aiming to deliver a more inclusive learning environment. Preliminary results indicate high recognition accuracy, smooth interaction, and potential for broader educational applications. Future improvements will involve expanding the gesture set and refining accuracy for complex gestures. Keywords: Hand Gesture Recognition, Interactive Learning, Real-Time Subtitling, Educational Technology, OpenCV

Kinematics of the Tennis Serve Using an Optoelectronic Motion Capture System: Are There Correlations between Joint Angles and Racket Velocity?

The serve is the most important stroke in tennis. It is a complex gesture consisting of numerous rotations with a wide amplitude, which are important to manage for performance. The aim of this study was to investigate whether correlations exist between joint kinematic parameters and racket velocity. A quantitative kinematics analysis of four ranked players (two boys and two girls) was carried out using an optoelectronic system composed of 10 cameras (150 Hz). Five flat serves per player were analyzed. Eighty-two markers were located across the 15 body segments and on the racket. A descriptive statistical analysis including a correlation analysis was carried out between joint angles and racket kinematic parameters (vertical position, velocity, and acceleration) during the cocking and acceleration phases. Ten very high (0.7 < r < 0.9) and three almost perfect (r > 0.9) correlations were found. Shoulder and hip axial rotations, knee flexion, and trunk extension were correlated linearly with racket vertical position and velocity during the cocking phase. For the acceleration phase, elbow flexion, trunk flexion/extension, and trunk axial rotation were linked to racket kinematics. Some of these parameters showed differences between slow and fast serves. These parameters, which are involved in transmitting ball velocity, are important to consider for tennis players and coaches in training programs, education, and performance enhancement.

A channel-fused gated temporal convolutional network for EMG-based gesture recognition

Background:EMG gesture recognition can be widely applied in many fields, such as prosthetic control and human–computer interaction, to enhance the standard of human life. Purpose:This study aims to develop a deep learning-based model to identify multiple complex gestures from raw EMG signals. Methods:We propose a new channel-fused gated temporal convolutional network. First, a channel fusion and gating mechanism is designed to improve temporal convolutional networks, allowing the model to obtain higher-level features. Second, we improve the channel fusion module by the short-term average energy to fuse the EMG signals of multi-channels more accurately. Results:The model is evaluated on a public dataset of EMG gestures, NinaPro DB5. Results demonstrated that the unbalanced accuracy rate of 53 gesture actions reaches 92.71%, and the balanced accuracy rate 74.79%. Further, ablation experiments validates the effectiveness of each model module. Conclusions:This study demonstrates our proposed approach can improve gesture recognition accuracy for complex gestures and has great potential for practical applications.

Gesture imitation performance in community-dwelling older people: assessment of a gesture imitation task in the screening and diagnosis of mild cognitive impairment and dementia.

Gesture imitation, a simple tool for assessing visuospatial/visuoconstructive functions, is reportedly useful for screening and diagnosing dementia. However, gesture imitation performance in healthy older adults is largely unknown, as are the factors associated with lower performance. To address these unknowns, we examined the gesture imitation performance of a large number of community-dwelling older adults aged ≥65 years in Arao City, Kumamoto Prefecture (southern Japan). The examiner presented the participants with eight gesture patterns and considered it a success if they could imitate them within 10 s. The success rate of each gesture imitation was calculated for three diagnostic groups: cognitively normal (CN) (n = 1184), mild cognitive impairment (MCI) (n = 237), and dementia (n = 47). Next, we reorganised the original gesture imitation battery by combining six selected gestures with the following scoring method: if the participants successfully imitated the gestures, immediately or within 5 s, two points were assigned. If they succeeded within 5-10 s, one point was assigned. The sensitivity and specificity of the battery were investigated to detect the dementia and MCI groups. Factors associated with gesture imitation battery scores were examined. Except one complex gesture, the success rate of imitation in the CN group was high, approximately 90%. The sensitivity and specificity of the gesture imitation battery for discriminating between the dementia and CN groups and between the MCI and CN groups were 70%/88%, and 45%/75%, respectively. Ageing, male sex, and a diagnosis of dementia or MCI were associated with lower scores on the gesture imitation battery. Gesture imitation tasks alone may not be sufficient to detect MCI. However, by combining gestures with set time limits, gesture imitation tasks can be a low-burden and effective method for detecting dementia, even in community medicine, such as during health check-ups.

Leveraging Self-Attention Mechanism for Deep Learning in Hand-Gesture Recognition System

This research addresses the complex challenge of recognizing hand gestures irrespective of the user's body posture, a crucial issue in medical treatment for people with speech impairments and human-machine interfaces where precise gesture interpretation is vital. The aim is to engineer an advanced hand gesture recognition system, effective across various body positions and camera viewpoints. A novel flexible camera arrangement was employed, integrating a CNN-Transformer hybrid model, leveraging the strengths of Convolutional Neural Networks and the self-attention mechanism of Transformers. Developed using Python and the PyTorch deep learning framework, the system focuses on sophisticated image processing techniques. A thorough literature review on gesture recognition systems and multi-view analysis was conducted to inform the development. The system demonstrated exceptional accuracy in recognizing hand gestures in diverse body postures and from multiple camera perspectives, significantly outperforming existing methods. It marked a significant advancement in decoding complex gestures, a key aspect for medical applications and intricate human-machine is primarily interactions. This beneficial for people with speech impairments, rehabilitation, and in human-machine interfaces, poised to revolutionize patient care and enhance interaction with advanced machinery and computer systems.

Gesture Recognition in Virtual reality

Virtual Reality (VR) provides users with a sensory experience that is close to reality, creating a sense of interaction. It is widely used, and the gesture recognition in VR also has a great effect. Gesture recognition enriches VR using experience and promotes a more direct and natural interaction. Gesture recognition usually employs sensors to collect data from users and machine leaning algorithms to interpret and respond to human activities. Complex gestures need more complex algorithms and more rigorous operations. The reason is that complex gestures mean larger quantity of data. If data is larger, the harder to get robust and effective datasets. Then, features can also become difficult to extract, contributing to misrecognition or unrecognizable. Though machine leaning algorithms are widely used in gesture recognition, there are still some important challenges need to be addressed, like lack of standardization and limitations of availability of diverse and large datasets. However, VR, gesture recognition and machine leaning algorithms all have excellent prospect, because they are in line with the development of the Times and show the progress of science and technology. This paper not only focuses on their advantages but also does not ignore their shortcomings, and looks at them comprehensively.

Hand Gesture Recognition and Motion Detection System for Interactive Applications

Security systems and interactive multimedia applications are two examples of the many fields in which Gesture Sense (GS) technology, a significant development in human-computer interaction, has found use. This paper introduces the "Arduino Hand Gesture Identification and Motion Detection System," a revolutionary combination of Python, Arduino Uno R3, and ultrasonic sensors intended to improve interactive apps through accurate gesture identification. Real-time tracking of hand movements is made possible by ultrasonic sensors, which are renowned for their fine-grained motion detection capabilities. Python allows the Arduino Uno R3 to run complex gesture detection algorithms. A user's engagement is improved by an LCD that presents recognized gestures and status updates, which provides immediate visual feedback. The system's outstanding accuracy, responsiveness, and usability are demonstrated by severe testing, which shows that it typically recognizes objects with an accuracy of 93%. In the context of interactive applications, this research study demonstrates how GS technology has the potential to revolutionize user experiences by ensuring seamless and immersive interactions. Key Words: Interactive application, Human-computer interaction, Hand gesture recognition, Motion detection.

Enhancing Accessibility with LSTM-Based Sign Language Detection

Sign language serves as a vital means of communication for the deaf and hard of hearing community. However, identifying sign language poses a significant challenge due to its complexity and the lack of a standardized global framework. Recent advances in machine learning, particularly Long Short-Term Memory (LSTM) algorithms, offer promise in the field of sign language gesture recognition. This research introduces an innovative method that leverages LSTM, a type of recurrent neural network designed for processing sequential input. Our goal is to create a highly accurate system capable of anticipating and reproducing sign language motions with precision. LSTM's unique capabilities enhance the recognition of complex gestures by capturing the temporal relationships and fine details inherent in sign language. The results of this study demonstrate that LSTM-based approaches outperform existing state-of-the-art techniques, highlighting the effectiveness of LSTM in sign language recognition and their potential to facilitate communication between the deaf and hearing communities.

Dynamic gesture recognition based on compressed video for UAV control

As a multifunctional aircraft with small size, low cost and easy control, UAVs can be used in many fields such as gardening, plant protection, mapping, logistics, military, etc. For a more convenient human-machine interaction mode, the user interacts with the UAVs in the form of dynamic gestures. The current traditional approach is to train convolutional neural networks with images as direct inputs to the system for the purpose of controlling UAVs. However, this approach leaves the temporal representation information between images missing, resulting in poor training results or over-reliance on computing resources. Therefore, this paper proposes an efficient processing strategy, that is, for simple gesture tasks, the optimized frame extraction algorithm is adopted to process picture-based gesture recognition; for complex gesture tasks, compressed video is used as system input to complete video-based gesture recognition. Based on the training results in action recognition datasets, UCF-101 and HMDB-51, the efficiency of compressed video in gesture recognition tasks has been verified, which can be applied to UAV dynamic gesture control.

High-Sensitive Double-Clad Fiber Bragg Grating Sensors for Real-Time Monitoring of Body Posture and Movement State

Musculoskeletal disorders (MSD) are prevalent in society and currently lack effective treatments. The incidence of MSD appears to be increasing as the worldwide population ages and sedentary lifestyles. Real-time monitoring of the body posture and movement state may be an effective strategy for the prevention and treatment of MSD. We proposed a real-time monitoring system based on a wearable high-sensitive double-clad fiber Bragg grating (FBG) sensors and an integrated demodulation device to recognize the body posture and movement state. The results demonstrate that the double-clad FBG sensors exhibit a high measurement sensitivity of approximately 36 pm/° in elbow joint static tests and 7.5 pm/° in knee joint static tests, which is almost 1.4 times higher than that of a single-mode FBG. Furthermore, a dataset of six gestures was created based on time and wavelength information obtained from six double-clad FBG sensors monitoring subtle deformations of muscles of the arm. The accuracy of recognizing simple and complex gestures was close to 100% and about 88%, respectively. The static and dynamic measurements demonstrate that the double-clad sensors have high accuracy, improved sensitivity, excellent flexibility and good repeatability, and this work is expected to a reliable assessment tool for clinical trials of MSD.

A Wearable Gesture Recognition System With Ultrahigh Accuracy and Robustness Enabled by the Synergy of Multiple Fabric Sensing Devices

Recently, numerous wearable devices for gesture recognition have been reported, however, most of them are fabricated by assembling the separately-designed sensing layers with the substrate through linking agents, which will lead to the shifting or detaching of the sensing layers after long-duration use. Subsequently, over-reliance on multiple filtering algorithms to smooth the collected datasets becomes very popular, unavoidably resulting in the loss of data features and overfitting. In our point of view, it is much more crucial and effective for gesture recognition performance to reinforce the stability of the sensing layers. Therefore, in this work, we designed a type of Integrated MXene/Polyurethane (I-M/PU) sensor based on textile fabrics and further prepared a data glove, a smart wristband and a smart elbow pad. A wearable system for gesture recognition with ultra-high sensitivity to strain, as well as excellent stability and durability is successfully realized by the synergy of those three devices. The collected original data by the integrated sensors was steady with specific features, making the simplification of the classification model feasible. Finally, 19 static gestures (including 9 complex gestures) with the recognition accuracy of 100% in 6998 data samples have been impressively achieved by the simple Multilayer Perceptron (MLP) algorithm, which is the state-of-art best result as we know in the gesture recognition reports without any assistant computer vision algorithms, demonstrating our solution with a promising prospect in Virtual Reality (VR) and Human-Machine Interaction (HMI) fields.

Artificial Intelligence–Enabled Gesture‐Language‐Recognition Feedback System Using Strain‐Sensor‐Arrays‐Based Smart Glove

Wearable smart glove of gesture language provides a novel strategy for the hearing‐impaired people to commutate with the world. Current commercialized solutions of gesture language are limited by the full extent of human interaction beyond operation dexterity, sensory feedback, and the huge cost of fabrication. Herein, a low‐cost, high‐efficient gesture‐language‐recognition feedback system combined with the strain‐sensor arrays and machine‐learning technology is proposed. The strain‐sensor arrays integrated with 3D‐printed glove can extract both spatial and temporal information about the finger's movement. The smart glove achieves gesture‐language recognition using machine learning with an accuracy of over 99%. Integrating with multidimensional manipulation, visual feedback and artificial intelligence (AI)‐based gesture‐language recognition, the smart system can accurately recognize complex gestures and provide real‐time feedback to users. The smart glove system can not only provide an efficient way for hearing‐impaired persons to communicate with the outside world, but also benefit industries in multiple fields such as entertainment, home healthcare, sports training, and the medical industry.

Describing movement learning using metric learning.

Analysing movement learning can rely on human evaluation, e.g. annotating video recordings, or on computing means in applying metrics on behavioural data. However, it remains challenging to relate human perception of movement similarity to computational measures that aim at modelling such similarity. In this paper, we propose a metric learning method bridging the gap between human ratings of movement similarity in a motor learning task and computational metric evaluation on the same task. It applies metric learning on a Dynamic Time Warping algorithm to derive an optimal set of movement features that best explain human ratings. We evaluated this method on an existing movement dataset, which comprises videos of participants practising a complex gesture sequence toward a target template, as well as the collected data that describes the movements. We show that it is possible to establish a linear relationship between human ratings and our learned computational metric. This learned metric can be used to describe the most salient temporal moments implicitly used by annotators, as well as movement parameters that correlate with motor improvements in the dataset. We conclude with possibilities to generalise this method for designing computational tools dedicated to movement annotation and evaluation of skill learning.

Psycholinguistic Study on Expressive Language Disorder of Autistic Characters in Miracle Run Movie

Recently, it has been reported that the prevalence of autism has increased significantly. One of the characteristics of autism is a language problem, or it is well-known linguistically as a language disorder. This study is conducted to analyze categories of expressive language disorder of autism represented in Miracle Run Movie and contexts affecting them in producing those utterances. This study applies a descriptive qualitative approach using Carroll's framework (1985). As the result of analysis on 75 data, Philip and Steven, as the autistic characters, face all kinds of expressive language disorders covering phonology, syntax, semantics, and pragmatics in various cases. They face phonological language disorder in the case of inappropriate stress, flat and singsong intonation, and expressionless speech. They also face syntactical language disorder in the case of echolalia and semantic language disorder involving their disability in using semantic bias. In addition, they have pragmatic language disorder in the case of fewer complex gestures, politeness strategy, spontaneous remarks, misuse of pronouns, limited repertoire devices to initiate a topic in a conversation, and tendency to use a question to gain a response from a listener. Concerning the context when the autistic characters produce expressive language disorder, there are six types of context affecting their utterances. They have expressive language disorder when they are posing a question to someone, answering a question of others, expressing their ambition, expectation, surprise, and something they dislike. This study is an excellent guide for EFL teachers interested in working with autistic students.

Deep‐Learning‐Assisted Noncontact Gesture‐Recognition System for Touchless Human‐Machine Interfaces

AbstractHuman‐machine interfaces (HMIs) play important role in the communication between humans and robots. Touchless HMIs with high hand dexterity and hygiene hold great promise in medical applications, especially during the pandemic of coronavirus disease 2019 (COVID‐19) to reduce the spread of virus. However, current touchless HMIs are mainly restricted by limited types of gesture recognition, the requirement of wearing accessories, complex sensing platforms, light conditions, and low recognition accuracy, obstructing their practical applications. Here, an intelligent noncontact gesture‐recognition system is presented through the integration of a triboelectric touchless sensor (TTS) and deep learning technology. Combined with a deep‐learning‐based multilayer perceptron neural network, the TTS can recognize 16 different types of gestures with a high average accuracy of 96.5%. The intelligent noncontact gesture‐recognition system is further applied to control a robot for collecting throat swabs in a noncontact mode. Compared with present touchless HMIs, the proposed system can recognize diverse complex gestures by utilizing charges naturally carried on human fingers without the need of wearing accessories, complicated device structures, adequate light conditions, and achieves high recognition accuracy. This system could provide exciting opportunities to develop a new generation of touchless medical equipment, as well as touchless public facilities, smart robots, virtual reality, metaverse, etc.

Dance Movement Recognition Based on Modified GMM-Based Motion Target Detection Algorithm

Under the synergistic development of social economy and science and technology, the intelligent teaching of dance has become more and more popular. This teaching method can not only decompose dance movements more specifically, which is easy for students to understand and master, but also get rid of the time and space limitation in traditional dance teaching and provide more independent learning opportunities for students. The problem of low accuracy of dance movement recognition due to complex gesture changes in dance movements is addressed. To this end, this paper proposes a modified motion target detection algorithm based on GMM. The dance movement recognition algorithm first extracts the features of dance movements through a feature pyramid network, then uses a multi-feature fusion module to fuse multiple features to improve the algorithm’s estimation of complex postures, and finally completes the recognition of dance movements. Experiments show that our method can maintain a certain recognition rate in the case where the background and target are easily confused, and can effectively improve the dance action recognition accuracy, thus realizing the action correction function for dancers. This also verifies the effectiveness of the action recognition algorithm for dance movement recognition.

Translating Hand Gestures Using 3D Convolutional Neural Network

Hand gestures are one of the mediums that many people use to communicate with each other. The use of gesture recognition applications has become increasingly popular in recent years especially in computer vision areas. Typically, gestures can easily be recognized from a single image frame (i.e. alphabet from sign language), however the ability to recognize complex gestures with subtle differences between movement requires more works and larger datasets. In this work, we introduce a simple gesture recognition system that translates 5 different hand gestures, namely “doing other things”, “swiping down”, “swiping left”, “zooming out with two fingers” and “drumming fingers”. We used datasets obtained from Jester dataset. The inputs were processed in ‘RGB’ format during the pre-processing phase and a spatiotemporal filter were used as a feature extraction method, which were also the main building block in this system. Next, we trained the features using 3D Convolution Neural Network (3D-CNN). Further, we used real-time video to test the developed recognition system with 5 different actors. Findings show that the developed model can translate hand gestures with accuracy of 85.70% and 0.4% losses.

SpeakLiz by Talov: Toward a Sign Language Recognition mobile application – Supported by Mada Innovation Program

The present article discussion on the Sign language recognition which is part of one of the most challenging tasks for Artificial Intelligence (AI) algorithms: camera-based gesture recognition. Probably, hence its relatively novel practical applications despite its old presence in the general AI field due to the required computing power for a device to be able to track complex gestures (like sign languages) just with a camera along with not-ideal light, focus, and stability conditions or even many of them combined. But, reaching acceptable results on mobile devices with this recognition task is even more elusive than doing it on computers (actually, PC-oriented projects are most of the research out there).

Motion Similarity Evaluation between Human and a Tri-Co Robot during Real-Time Imitation with a Trajectory Dynamic Time Warping Model.

Precisely imitating human motions in real-time poses a challenge for the robots due to difference in their physical structures. This paper proposes a human–computer interaction method for remotely manipulating life-size humanoid robots with a new metrics for evaluating motion similarity. First, we establish a motion capture system to acquire the operator’s motion data and retarget it to the standard bone model. Secondly, we develop a fast mapping algorithm, by mapping the BVH (BioVision Hierarchy) data collected by the motion capture system to each joint motion angle of the robot to realize the imitated motion control of the humanoid robot. Thirdly, a DTW (Dynamic Time Warping)-based trajectory evaluation method is proposed to quantitatively evaluate the difference between robot trajectory and human motion, and meanwhile, visualization terminals render it more convenient to make comparisons between two different but simultaneous motion systems. We design a complex gesture simulation experiment to verify the feasibility and real-time performance of the control method. The proposed human-in-the-loop imitation control method addresses a prominent non-isostructural retargeting problem between human and robot, enhances robot interaction capability in a more natural way, and improves robot adaptability to uncertain and dynamic environments.