Gesture Samples Research Articles

Sea Horse Optimization-Deep Neural Network: A Medication Adherence Monitoring System Based on Hand Gesture Recognition.

Medication adherence is an essential aspect of healthcare for patients and is important for achieving medical objectives. However, the lack of standard techniques for measuring adherence is a global concern, making it challenging to accurately monitor and measure patient medication regimens. The use of sensor technology for medication adherence monitoring has received much attention lately since it makes it possible to continuously observe patients' medication adherence behavior. Sensor devices or smart wearables utilize state-of-the-art machine learning (ML) methods to analyze intricate data patterns and provide predictions accurately. The key aim of this work is to develop a sensor-based hand gesture recognition model to predict medication activities. In this research, a smart sensor device-based hand gesture prediction model is developed to recognize medication intake activities. The device includes a tri-axial gyroscope, geometric, and accelerometer sensors to sense and gather data from hand gestures. A smartphone application gathers hand gesture data from the sensor device, which is then stored in the cloud database in a .csv format. These data are collected, processed, and classified to recognize the medication intake activity using the proposed novel neural network model called Sea Horse Optimization-Deep Neural Network (SHO-DNN). The SHO technique is implemented to update the biases and weights and the number of hidden layers in the DNN model. By updating these parameters, the DNN model is improved in classifying the samples of hand gestures to identify the medication activities. The research model demonstrates impressive performance, with an accuracy of 98.59%, sensitivity of 97.82%, precision of 98.69%, and an F1 score of 98.48%. Hence, the proposed model outperformed the most available models in all the aforementioned aspects. The results indicate that this model is a promising approach for medication adherence monitoring in healthcare applications, instilling confidence in its effectiveness.

Feature decoupling and regeneration towards wifi-based human activity recognition

Gesture recognition using WiFi is vital for human–computer interaction, smart homes, smart spaces, and elderly care. WiFi signals are non-stationary, sensitive to the environment, and traditional pattern recognition-based HAR methods incur exorbitant training and deployment costs due to their reliance on the quality and quantity of training data. While there has been extensive research on enhancing HAR techniques, the efficiency of the system is still limited by the scarcity of training data and identity-action feature coupling. This research focuses on attaining gesture recognition across multiple users with a limited number of samples. Our findings show that the conventional data augmentation methods are incapable of facilitating the system in attaining sample diversity. As a result, we propose a training-free augmentation strategy as a means to provide adequate training data. Unlike the conventional data enhancement approach, this scheme aims to develop data processing methods that differentiate between various samples in practical applications. Consequently, it enhances data pertinent to specific HAR assignments effectively. To effectively extract action features in WiFi samples, an unsupervised cross-user domain sample generation (CUDSG) model is proposed. This model generates virtual gesture samples for new user domains by decoupling and recombining gesture and identity features. This extends the sensing boundary of the system to new user domains without requiring a significant number of users to participate. Model performance was evaluated using various classifiers, such as SVM, KNN and CNN. The results demonstrate a significant improvement in average classification accuracy from 57.3% to 98.4%. This indicates that CUDSG is a highly effective tool for enhancing the performance of existing gesture recognition techniques.

Hand Gesture Gaming

Hand gesture recognition has emerged as a promising interface for interactive gaming applications, offering intuitive and immersive experiences. This research paper explores the development and implementation of a hand gesture recognition system using Python for gaming purposes. The study investigates various machine learning techniques, particularly convolutional neural networks (CNNs), for accurate detection and classification of hand gestures from video input. The research methodology involves collecting a diverse dataset of hand gesture samples, encompassing common gaming gestures such as swipes, taps, and rotations. Preprocessing techniques are employed to enhance the quality and consistency of the dataset. Subsequently, deep learning architectures, including CNNs, are trained on the dataset to recognize and classify hand gestures in real-time.

Empowering human-robot interaction using sEMG sensor: Hybrid deep learning model for accurate hand gesture recognition

In this paper, a novel approach using a Henry Gas Solubility-based Stacked Convolutional Neural Network (HGS-SCNN) for hand gesture recognition using surface electromyography (sEMG) sensors is proposed. The stacked architecture of the CNN model helps to capture both low-level and high-level features, enabling effective representation learning. To begin, we generated a dataset comprising 600 samples of hand gestures. Next, we applied the Discrete Wavelet Transform (DWT) technique to extract features from the filtered sEMG signal. This step allowed us to capture both spatial and frequency information, enhancing the discriminative power of the extracted features. Extensive experiments are conducted to evaluate the performance of the proposed HGS-SCNN model. In addition, the obtained results are compared with state-of-the-art techniques, namely AOA-SCNN, GWO-SCNN, and WOA-SCNN. The comparative analysis demonstrates that the HGS-SCNN outperforms these existing methods, achieving an impressive accuracy of 99.3%. The experimental results validate the effectiveness of our proposed approach in accurately detecting hand gestures. The combination of DWT-based feature extraction and the HGS-SCNN model offers robust and reliable hand gesture recognition, thereby opening new possibilities for intuitive human-machine interaction and applications requiring gesture-based control.

Feasibility study on the application of a spiking neural network in myoelectric control systems.

In recent years, the effectiveness of a spiking neural network (SNN) for Electromyography (EMG) pattern recognition has been validated, but there is a lack of comprehensive consideration of the problems of heavy training burden, poor robustness, and high energy consumption in the application of actual myoelectric control systems. In order to explore the feasibility of the application of SNN in actual myoelectric control systems, this paper investigated an EMG pattern recognition scheme based on SNN. To alleviate the differences in EMG distribution caused by electrode shifts and individual differences, the adaptive threshold encoding was applied to gesture sample encoding. To improve the feature extraction ability of SNN, the leaky-integrate-and-fire (LIF) neuron that combines voltage-current effect was adopted as a spike neuron model. To balance recognition accuracy and power consumption, experiments were designed to determine encoding parameter and LIF neuron release threshold. By conducting the gesture recognition experiments considering different training test ratios, electrode shifts, and user independences on the nine-gesture high-density and low-density EMG datasets respectively, the advantages of the proposed SNN-based scheme have been verified. Compared with a Convolutional Neural Network (CNN), Long Short-Term Memory Network (LSTM) and Linear Discriminant Analysis (LDA), SNN can effectively reduce the number of repetitions in the training set, and its power consumption was reduced by 1-2 orders of magnitude. For the high-density and low-density EMG datasets, SNN improved the overall average accuracies by about (0.99 ~ 14.91%) under different training test ratios. For the high-density EMG dataset, the accuracy of SNN was improved by (0.94 ~ 13.76%) under electrode-shift condition and (3.81 ~ 18.95%) in user-independent case. The advantages of SNN in alleviating the user training burden, reducing power consumption, and improving robustness are of great significance for the implementation of user-friendly low-power myoelectric control systems.

SequenceSense: A Tool for Designing Usable Foot-Based Gestures Using a Sequence-Based Gesture Recognizer

Foot-based gestures enable people to interact with mobile and wearable devices when their hands are unavailable for interaction. For the foot gestures to be truly usable, the gestures should be recognizable by the system without being confused by daily activities and still be easy to perform. However, designing such gestures often requires multiple iterations of gesture design, model training, and evaluation. In this paper, we present SequenceSense, a tool developed to help designers efficiently design a usable gesture set using inertial sensors, which eliminates the need for multiple data collection studies to evaluate the gestures’ usability through gesture modification by sequencing atomic actions and instant false positive analysis, and instead requires only the initial gesture sample collection. Unlike gesture recognizers using complete gestures to train a model, SequenceSense segments gesture into a sequence of atomic actions. For example, a foot tap to the right may have (1) lift the foot, (2) move the foot to the right, and (3) land the foot. SequenceSense also compares the gesture sequence with the sequence database created from the daily activities to identify possible conflicts. This allows gesture designers to build easily usable foot-based gestures without the need for recollecting and evaluating gestures. We validated SequenceSense’s efficacy in designing usable gestures with low false positives through a user study with nine gesture designers.

Continuous Gesture Sequences Recognition Based on Few-Shot Learning

A large number of demands for space on-orbit services to ensure the on-orbit system completes its specified tasks are foreseeable, and the efficiency and the security are the most significant factors when we carry out an on-orbit mission. And it can improve human-computer interaction efficiency in operations with proper gesture recognition solutions. In actual situations, the operations are complex and changeable, so the gestures used in interaction are also difficult to predict in advance due to the compounding of multiple consecutive gestures. To recognize such gestures based on computer vision (CV) requires complex models trained by a large amount of datasets, it is often unable to obtain enough gesture samples for training a complex model in real tasks, and the cost of labeling the collected gesture samples is quite expensive. Aiming at the problems mentioned above, we propose a few-shot continuous gesture recognition scheme based on RGB video. The scheme uses Mediapipe to detect the key points of each frame in the video stream, decomposes the basic components of gesture features based on certain human palm structure, and then extracts and combines the above basic gesture features by a lightweight autoencoder network. Our scheme can achieve 89.73% recognition accuracy on the 5-way 1-shot gesture recognition task which randomly selected 142 gesture instances of 5 categories from the RWTH German fingerspelling dataset.

ML-HGR-Net: A Meta-Learning Network for FMCW Radar Based Hand Gesture Recognition

In recent years, radar-based hand gesture recognition (HGR) has attracted much attention in the field of human-computer interaction (HCI) due to its benefits of high recognition accuracy and independence from lighting conditions. Conventional deep learning (DL) based models for HGR rely on a large amount of labelled data for training to achieve a satisfactory recognition accuracy. However, it is usually time-consuming and labor-intensive for collecting large amounts of radar data and also difficult to cover all possible hand gesture classes. In this paper, we introduce meta-learning to address the few-shot learning problem for frequency modulated continuous wave (FMCW) radar based HGR. We propose a meta-learning network to learn a model that can quickly adapt to unseen hand gesture tasks with few training observations. In particular, we propose a 3D convolutional neural network (CNN) based dual-channel fusion network for feature extraction by exploiting the correlations of multiple features in radar echo signals to improve the recognition accuracy. In addition, we also develop a learnable relation module with neural networks as a non-linear classifier to measure the similarity between the samples of different hand gestures, which can be more applicable for the HGR task. Finally, we evaluate the performance of the proposed model by conducting experiments on an FMCW radar hardware system. Experimental results show that the proposed meta-learning model substantially enhances the recognition accuracy compared with the state-of-the-art methods including DL and meta-learning based models for FMCW radar-based HGR.

Low personality-sensitive feature learning for radar-based gesture recognition

Radar-based sensing of gestures has gained tremendous attention with the recent advancements in radar technologies. However, evident discrepancies exist in the gesture samples due to hand flexibility and individual habits. It is challenging for traditional methods to identify the gestures from unknown data sources. Cross-person (Cross-scenario) recognition refers to a recognition where the training and test samples are from different people (scenarios), respectively. To explore how the recognition performance is affected by the individual habits, the reasons are analyzed and visualized through the experiments. On this basis, HandNet is targeted proposed for the low personality-sensitive feature learning and it has two main contributions. First, a Stepped Data Augmentation (SDA) is proposed to reduce the sample interferences by non-coherent accumulating, and capture the inter-frame dependencies. Second, a Focus on Generalization loss (FoG loss) is proposed to highlight the generalized feature learning by res tricting the distances of inter-source features. Extensive experiments demonstrate that HandNet effectively reduces the classifier’s sensitivity to the personalized habits, and outperforms the existing state-of-the-art methods on the cross-person and cross-scenario gesture recognition. To the best of our knowledge, it is the first time to dedicate to addressing the radar-based gesture recognition with low personal sensitivity, which is more suitable for practical scenarios.

Real-Time Large Vocabulary American Sign Language Recognition System for Mobile Devices

The American Sign Language (ASL) is the only major language used in the educational system of hearing-impaired people in Nigeria. The automatic recognition system of the signs has not been currently used for the teaching of hearing-impaired students. This study developed a realtime large vocabulary sign language for ASL implemented on android devices. Samples of static and dynamic hand gestures were collected from the primary school of handicap, Osogbo. The specific objectives include the collection of hand gestures, examining the specific features for the recognition process, designing a model for the specific features examined, implementing the model, and evaluating the performance of the system. The real-time vocabulary sign language was recognized using Convolution Neural Network (CNN) implemented using Python programming language. The developed system was evaluated using precision, recall and accuracy as metrics. The model prediction carried out using the test image has an overall accuracy of 92.98%. The obtained result showed that the system will enhance the learning skills and provide adequate learning platform for both students and the teachers of hearing-impaired schools.

MTransSee

Gesture recognition using millimeter-wave radios facilitates natural human-computer interactions, but existing works require a consistent environment, i.e., the neural networks for recognition are trained and tested for the same users and at some fixed positions. In this case, their performance will decrease rapidly when they enter into a new environment. To make the model applicable in different environments, a straightforward approach is to collect and re-train the model for the gesture samples on every possible position upon each new user. However, it may ask the users to spend unacceptable time to accomplish such adaptation, which makes it difficult to be widely used in practice. In this paper, we first collect an abundant mmWave gesture dataset containing 59,280 samples as a benchmark to investigate the impact of the environment changes quantitatively. Then we propose a novel transfer-learning approach called mTransSee, which can serve the gestures in practice using pre-learned experience by least adaptation, i.e., retraining using only 8 samples per gesture for the same accuracy. mTransSee reduces dozens of workloads for the environment adaptation. We implement mTransSee on a commodity mmWave sensor and make a user study to compare the advance of mTransSee over the state-of-the-art solution in terms of user experience during adaptation.

Aging, Gesture Production, and Disfluency in Speech: A Comparison of Younger and Older Adults.

Age-related changes are observed in the speech and gestures of neurotypical individuals. Older adults are more disfluent in speech and use fewer representational gestures (e.g., holding two hands close to each other to mean small), compared to younger adults. Using gestures, especially representational gestures, is common in difficult tasks to aid the conceptualization process and to facilitate lexical access. This study investigates how aging can affect gesture production and the co-occurrence between gesture and speech disfluency. We elicited speech and gesture samples from younger and older adults (N = 60) by using a painting description task that provided concrete and abstract contexts. Results indicated that albeit the two age groups revealed comparable overall speech disfluency and gesture rates, they differed in terms of how their disfluencies and gestures were distributed across specific categories. Moreover, the proportion of speech disfluencies that occur with a gesture was significantly higher for younger than older adults. However, the two age groups were comparable in terms of the proportion of gestures that were accompanied by a speech disfluency. These findings suggest that younger adults' language production system might be better at benefiting from other modalities, that is, gesture, to resolve temporary problems in speech planning. However, from a gesture perspective, it might be difficult to differentiate between gestures' self-oriented and communicative functions and understand their role in speech facilitation. Focusing on specific cases where speech disfluency and gestures co-occur and considering individual differences might bring insight into multimodal communication.

Touch Gesture and Emotion Recognition Using Decomposed Spatiotemporal Convolutions

Touch is one of the most essential and effective means to convey affective feelings and intentions in human communication. For a social robot, the ability to recognize human touch gestures and emotions could help realize efficient and natural human–robot interaction. To this end, an affective touch gesture dataset involving ten kinds of touch gestures and 12 kinds of discrete emotions was built by using a pressure sensor array, in which the acquired touch gesture samples are three-dimensional (3-D) spatiotemporal signals that include the shape appearance and motion dynamics. Due to the excellent performance of convolutional neural networks (CNNs), spatiotemporal CNNs have been effectively verified by researchers for 3-D signal classification. However, the large number of parameters and the high complexity of training 3-D convolution kernels remain to be solved. In this article, a decomposed spatiotemporal convolution was designed for feature representation from the raw touch gesture samples. Specifically, the 3-D kernel was factorized into three 1-D kernels by tensor decomposition. The proposed convolution has a simpler but deeper architecture than standard 3-D convolution, which improves the nonlinear expression ability of the model. Besides, the computation cost can be reduced without compromising recognition accuracy. Using a user-dependent test mode, the proposed method yields the accuracies of up to 92.41% and 72.47% for touch gesture and emotion recognitions, respectively. Experimental results demonstrate the effectiveness of the proposed method, and at the same time, preliminarily verify the feasibility of robot perceiving human emotions through touch.

A comparison of convolutional neural networks for Kazakh sign language recognition

For people with disabilities, sign language is the most important means of communication. Therefore, more and more authors of various papers and scientists around the world are proposing solutions to use intelligent hand gesture recognition systems. Such a system is aimed not only for those who wish to understand a sign language, but also speak using gesture recognition software. In this paper, a new benchmark dataset for Kazakh fingerspelling, able to train deep neural networks, is introduced. The dataset contains more than 10122 gesture samples for 42 alphabets. The alphabet has its own peculiarities as some characters are shown in motion, which may influence sign recognition. Research and analysis of convolutional neural networks, comparison, testing, results and analysis of LeNet, AlexNet, ResNet and EffectiveNet – EfficientNetB7 methods are described in the paper. EffectiveNet architecture is state-of-the-art (SOTA) and is supposed to be a new one compared to other architectures under consideration. On this dataset, we showed that the LeNet and EffectiveNet networks outperform other competing algorithms. Moreover, EffectiveNet can achieve state-of-the-art performance on nother hand gesture datasets. The architecture and operation principle of these algorithms reflect the effectiveness of their application in sign language recognition. The evaluation of the CNN model score is conducted by using the accuracy and penalty matrix. During training epochs, LeNet and EffectiveNet showed better results: accuracy and loss function had similar and close trends. The results of EffectiveNet were explained by the tools of the SHapley Additive exPlanations (SHAP) framework. SHAP explored the model to detect complex relationships between features in the images. Focusing on the SHAP tool may help to further improve the accuracy of the model

Frequency of Gesture Use and Language in Typically Developing Prelinguistic Children

PurposeThe aim of this study was to examine the relationship between frequency of gesture use and language with a consideration for the effect of age and setting on frequency of gesture use in prelinguistic typically developing children. MethodParticipants included a total of 54 typically developing infants and toddlers between the ages of 9 months and 15 months separated into two age ranges, 9-12 months and 12-15 months. All participants were administered the Mullen’s Scale of Early Learning and two gesture samples were obtained: one sample in a structured setting and the other in an unstructured setting. Gesture samples were coded by research assistants blind to the purpose of the research study and total frequency and frequencies for the following gesture types were calculated: behavior regulation, social interaction, and joint attention (Bruner, 1983). ResultsResults indicated that both age and setting have a significant effect on frequency of gesture use and frequency of gesture is correlated to receptive and expressive language abilities; however, these relationships are dependent upon the gesture type examined. ConclusionsThese findings further our understanding of the relationship between gesture use and language and support the concept that frequency of gesture is related to language abilities. This is meaningful because gestures are one of the first forms of intentional communication, allowing for early identification of language abilities at a young age.

GAN-based Style Transformation to Improve Gesture-recognition Accuracy

Gesture recognition and human-activity recognition from multi-channel sensory data are important tasks in wearable and ubiquitous computing. In these tasks, increasing both the number of recognizable activity classes and the recognition accuracy is essential. However, this is usually an ill-posed problem because individual differences in the same gesture class may affect the discrimination of different gesture classes. One promising solution is to use personal classifiers, but this requires personal gesture samples for re-training the classifiers. We propose a method of solving this issue that obtains personal gesture classifiers using few user gesture samples, thus, achieving accurate gesture recognition for an increased number of gesture classes without requiring extensive user calibration. The novelty of our method is introducing a generative adversarial network (GAN)-based style transformer to 'generate' a user's gesture data. The method synthesizes the gesture examples of the target class of a target user by transforming of a) gesture data into another class of the same user (intra-user transformation) or b) gesture data of the same class of another user (inter-user transformation). The synthesized data are then used to train the personal gesture classifier. We conducted comprehensive experiments using 1) different classifiers including SVM and CNN, 2) intra- and inter-user transformations, 3) various data-missing patterns, and 4) two different types of sensory data. Results showed that the proposed method had an increased performance. Specially, the CNN-based classifiers increased in average accuracy from 0.747 to 0.822 in the CheekInput dataset and from 0.856 to 0.899 in the USC-HAD dataset. Moreover, the experimental results with various data-missing conditions revealed a relation between the number of missing gesture classes and the accuracy of the existing and proposed methods, and we were able to clarify several advantages of the proposed method. These results indicate the potential of considerably reducing the number of required training samples of target users.

Independent component analysis in multi-channel forearm surface electromyography processing on armband approach for hand gesture classification

Multi-channel surface electromyography acquisition is widely applied for gesture recognition in wearable armband devices and it may contain redundant information between the channels. An issue in this technique is the number of channels, which can improve the classification accuracy, but it requires more computational effort. Independent component analysis is a signal processing technique used in blind source separation problems and can be used to reduce dimension, separating linearly mixed sources. In this study, an analysis of the influence of independent component analysis in hand gesture classification with surface electromyography signals, acquired from the forearm, is proposed. Six gestures were acquired from 10 subjects, 4 time-domain features were extracted, and five classifiers were used in the evaluation. This work compares two approaches to extract the W matrix (the demixing matrix) in independent component analysis: one W matrix for each subject and one W for each gesture sample. Besides, the effect of increasing the number of channels in the classification is analyzed, aiming to find a statistically relevant number of independent components. The results showed that the two approaches of W matrix have no significant difference between them. Moreover, it was observed that the number of independent components affects the classifiers, but five components showed the same distribution of results compared with more components. Concerning the classifiers, extreme learning machine neural network and support vector machines presented the best results (over 90%).

Template Matching Based Sign Language Recognition System for Android Devices

An android based sign language recognition system for selected English vocabularies was developed with the explicit objective to examine the specific characteristics that are responsible for gestures recognition. Also, a recognition model for the process was designed, implemented, and evaluated on 230 samples of hand gestures. The collected samples were pre-processed and rescaled from 3024 ×4032 pixels to 245 ×350 pixels. The samples were examined for the specific characteristics using Oriented FAST and Rotated BRIEF, and the Principal Component Analysis used for feature extraction. The model was implemented in Android Studio using the template matching algorithm as its classifier. The performance of the system was evaluated using precision, recall, and accuracy as metrics. It was observed that the system obtained an average classification rate of 87%, an average precision value of 88% and 91% for the average recall rate on the test data of hand gestures. The study, therefore, has successfully classified hand gestures for selected English vocabularies. The developed system will enhance the communication skills between hearing and hearing-impaired people, and also aid their teaching and learning processes. Future work include exploring state-of-the-art machining learning techniques such Generative Adversarial Networks (GANs) for large dataset to improve the accuracy of results. Keywords— Feature extraction; Gestures Recognition; Sign Language; Vocabulary, Android device.

Robust human gesture recognition by leveraging multi-scale feature fusion

In the state-of-the-art human–computer interaction (HCI) systems, gestures feature is more intuitive, natural and, easy-to-acquire than the other visual features. Due to the high descriptiveness of hand movements and the scarcity of labeled human gesture samples, gesture recognition models is prone to overfitting in practice. In order to handle this problem, to facilitate the model parameter learning during training, this paper introduces a disturbing IoU strategy to alleviate overfitting during the training stage from a ROI discriminative network. Moreover, during the deep gesture feature extraction, feature maps with the same size output by different layers are intelligently fused. This strategy can effectively preserve the key information in a small scale, reduce the information loss, and improve the generalization ability of the high-level gesture features. Comprehensive experimental results on the extended VIVA dataset and the NTU dataset have shown that the proposed model achieves a better mAP performance than its competitors

WristCam: A Wearable Sensor for Hand Trajectory Gesture Recognition and Intelligent Human–Robot Interaction

As a promising component for body sensor networks, the wearable sensors for hand gesture recognition have increasingly received great attention in recent years. By interpreting human intentions through hand gestures, the natural human–robot interaction can be realized in the smart home where the youth and the elderly can perform hand gestures to control the household robot or the robotic wheelchair. Here, a wearable wrist-worn camera sensor was shown to recognize hand trajectory gestures. The moving velocity of the user’s hand was deduced from the matched speeded up robust features keypoints of the moving background of the video sequence. Furthermore, the segmentation of continuous gestures was achieved by detecting the predefined gesture starting signal from the hand region of the image, which was segmented by the lazy snapping algorithm. In this paper, 10 types of gestures and 1350 gesture samples collected from 15 subjects at three different scenes were classified by the dynamic time warping algorithm and the results achieved an average recognition accuracy up to 97.6%. Moreover, the practicability of the proposed system was further demonstrated by controlling a cooperative robot to draw letters on paper.