Gesture Dataset Research Articles

Hand Gesture Enabled Operation for Computer Using Deep Learning

This project aims to develop a system that detects hand gestures and prints corresponding instructions for computer operators. The system utilizes Deep learning algorithms CNN and mediapipe to recognize hand gestures, and then prints instructions for the operator to perform specific computer operations.Operations like Notepad Open,Shutdown,Volume Up,Volume Down etc.The system will consist of a webcam, a Deep learning model, and a printer. The webcam will capture images of hand gestures, which will be processed using the Deep learning model to identify the specific gesture. The system will then print the corresponding instruction on the printer. The system will be designed to be user-friendly and adaptable to different environments and hardware configurations.The system aims to improve efficiency, reduce errors, and enhance accessibility for computer operators. The proposed system consists of a hand gesture detection module, instruction printing module, and computer operation module. The system is trained using a dataset of hand gestures and corresponding computer operations. Experimental results show that the system achieves high accuracy in gesture recognition and instruction printing. The system has potential applications in various fields, including gaming, education, and healthcare. Key Words: Hand gesture detection, Deep learning, instruction printing, computer operation, accessibility, Mediapipe, CNN.

SIGN-LANGUAGE TRANSLATOR USING MACHINE LEARNING

This research introduces a cutting-edge system designed to facilitate communication between individuals who are deaf and those who are hearing. By employing sophisticated machine learning algorithms and specific convolutional neural networks (CNNs), the proposed system can effectively sign language gestures into text or speech. A meticulously curated collection of sign language gestures serves as the foundation for training the model, ensuring its proficiency in accurately classifying several hand shapes and positions. To optimize classification performance, the system incorporates data preprocessing techniques that highlight the most distinctive features of the hands, thereby streamlining the computational process. This paper provides a comprehensive overview of the system's architecture, training methodology, and evaluation results, emphasizing the critical role of machine learning in developing inclusive communication tools that empower the deaf community. Future research initiatives will focus's on expanding the gesture dataset and refining real-time processing capabilities to further enhance the system's effectiveness and accessibility. Key Words: CNN, TensorFlow, Keras, NLTK

MCM-Vb\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$V_b$$\\end{document}F: dance hand gestures recognition with vision based features

To digitize and preserve the cultural heritage in the form of Indian classical dance become apparent area of research. Sattriya classical dance of North-East India (Assam) is one of the eight Indian classical dance forms that requires immediate preservation. Sattriya classical dance consists of 29 Asamyukta hastas (single-hand gestures) and 14 Samyukta hastas (double-hand gestures). Moreover, the foundation of Samyukta hasta depends on understanding Asamyukta hasta. Therefore, the paper aims to classify single-hand gestures of Sattriya classical dance only. Although, a solution based on two level classification method to classify the Sattriya classical dance is available in recent literature, but it requires a trial and error method to select the optimized features. Since, Asamyukta hastas can appear closely similar to each other and therefore misclassification chances are very high. Thus, accuracy rate obtained for the two level classification method was only 75.45%. So, to address this issues in this paper, a Multilevel Classification Model with Vision based Features (MCM-Vb\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$V_b$$\\end{document}F) has been proposed to classify the Asamyukta hastas of Sattriya classical dance. This model uses two types of feature matching, viz., high-level feature matching and low-level feature matching. To extract the high-level features and low-level features different algorithm has been proposed. In this model, features are automatically selected. This proposed MCM-Vb\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$V_b$$\\end{document}F model is also tested on Asamyukta hasta mudras of Bharatanatyam classical dance of South India (Tamil Nadu). This model obtain an accuracy 94.12%, 87.14% for Sattriya classical dance Single-Hand Gestures (SSHG) dataset and Bharatnatyam classical dance Single-Hand Gestures (BHSG) dataset respectively. This paper also provides the comparative study of the proposed model MCM-Vb\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$V_b$$\\end{document}F with traditional bench-mark classifier model such as Naive Bayes, Decision Tree and Support Vector Classifier (SVM) etc.

Fractional-order spike-timing-dependent gradient descent for multi-layer spiking neural networks

Accumulated detailed knowledge about the neuronal activities in human brains has brought more attention to bio-inspired spiking neural networks (SNNs). In contrast to non-spiking deep neural networks (DNNs), SNNs can encode and transmit spatiotemporal information more efficiently by exploiting biologically realistic and low-power event-driven neuromorphic architectures. However, the supervised learning of SNNs still remains a challenge because the spike-timing-dependent plasticity (STDP) of connected spiking neurons is difficult to implement and interpret in existing backpropagation learning schemes. This paper proposes a fractional-order spike-timing-dependent gradient descent (FO-STDGD) learning model by considering a derived nonlinear activation function that describes the relationship between the quasi-instantaneous firing rate and the temporal membrane potentials of nonleaky integrate-and-fire neurons. The training strategy can be generalized to any fractional orders between 0 and 2 since the FO-STDGD incorporates the fractional gradient descent method into the calculation of spike-timing-dependent loss gradients. The proposed FO-STDGD model is tested on the MNIST and DVS128 Gesture datasets and its accuracy under different network structure and fractional orders is analyzed. It can be found that the classification accuracy increases as the fractional order increases, and specifically, the case of fractional order 1.9 improves by 155 % relative to the case of fractional order 1 (traditional gradient descent). In addition, our scheme demonstrates the state-of-the-art computational efficacy for the same SNN structure and training epochs.

Real-Time Hand Gesture Monitoring Model Based on MediaPipe's Registerable System.

Hand gesture recognition plays a significant role in human-to-human and human-to-machine interactions. Currently, most hand gesture detection methods rely on fixed hand gesture recognition. However, with the diversity and variability of hand gestures in daily life, this paper proposes a registerable hand gesture recognition approach based on Triple Loss. By learning the differences between different hand gestures, it can cluster them and identify newly added gestures. This paper constructs a registerable gesture dataset (RGDS) for training registerable hand gesture recognition models. Additionally, it proposes a normalization method for transforming hand gesture data and a FingerComb block for combining and extracting hand gesture data to enhance features and accelerate model convergence. It also improves ResNet and introduces FingerNet for registerable single-hand gesture recognition. The proposed model performs well on the RGDS dataset. The system is registerable, allowing users to flexibly register their own hand gestures for personalized gesture recognition.

Hand Gesture Recognition Using Deep Learning

Hand gesture recognition (HGR) has gained significant attention due to its potential for various applications. This paper explores the use of deep learning, specifically Convolutional Neural Networks (CNNs), for HGR using the TensorFlow library. We investigate existing research on CNN-based HGR, focusing on image classification tasks. We then provide a brief overview of CNNs and their suitability for image recognition. Subsequently, we describe the typical workflow of a deep learning-based HGR system, including data preprocessing, hand detection, feature extraction with CNNs, and classification. We highlight the advantages of using TensorFlow to build and train CNN models for HGR. Finally, we conclude by summarizing the key findings from related work and mentioning the specific dataset and number of gestures classified in our research. This work contributes to the growing body of research on CNN-based HGR using TensorFlow and emphasizes its potential for developing accurate and efficient HGR systems.

Bioplausible Unsupervised Delay Learning for Extracting Spatiotemporal Features in Spiking Neural Networks.

The plasticity of the conduction delay between neurons plays a fundamental role in learning temporal features that are essential for processing videos, speech, and many high-level functions. However, the exact underlying mechanisms in the brain for this modulation are still under investigation. Devising a rule for precisely adjusting the synaptic delays could eventually help in developing more efficient and powerful brain-inspired computational models. In this article, we propose an unsupervised bioplausible learning rule for adjusting the synaptic delays in spiking neural networks. We also provide the mathematical proofs to show the convergence of our rule in learning spatiotemporal patterns. Furthermore, to show the effectiveness of our learning rule, we conducted several experiments on random dot kinematogram and a subset of DVS128 Gesture data sets. The experimental results indicate the efficiency of applying our proposed delay learning rule in extracting spatiotemporal features in an STDP-based spiking neural network.

Optimizing Artificial Neural Network Performance Using Gray Level Co-occurrence Matrix (GLCM) Feature Extraction for Monitoring Elderly Movement

Machine learning methods are used to detect elderly accidents while image processing is used to support machine learning performance so that detection performance can be better. This study used two methods used simultaneously, namely GLCM and ANN. The study consisted of preparation of human gesture datasets, preprocessing stage, application of GLCM, analysis of feature extraction results, classification using ANN and analysis of motion class detection results. Overall, the GLCM method with homogeneity parameters and ANN as a classifier obtained an accuracy of 24.32%. The GLCM method with contrast parameters and ANN as a classifier gets an accuracy of 99.84%. The GLCM method with mean parameters and ANN as the classifier gets an accuracy of 99.99%. The GLCM method with dissimilarity parameters and ANN as a classifier to classify hand movement images gets the best accuracy of 100%.

Comparative Study of sEMG Feature Evaluation Methods Based on the Hand Gesture Classification Performance.

Effective feature extraction and selection are crucial for the accurate classification and prediction of hand gestures based on electromyographic signals. In this paper, we systematically compare six filter and wrapper feature evaluation methods and investigate their respective impacts on the accuracy of gesture recognition. The investigation is based on several benchmark datasets and one real hand gesture dataset, including 15 hand force exercises collected from 14 healthy subjects using eight commercial sEMG sensors. A total of 37 time- and frequency-domain features were extracted from each sEMG channel. The benchmark dataset revealed that the minimum Redundancy Maximum Relevance (mRMR) feature evaluation method had the poorest performance, resulting in a decrease in classification accuracy. However, the RFE method demonstrated the potential to enhance classification accuracy across most of the datasets. It selected a feature subset comprising 65 features, which led to an accuracy of 97.14%. The Mutual Information (MI) method selected 200 features to reach an accuracy of 97.38%. The Feature Importance (FI) method reached a higher accuracy of 97.62% but selected 140 features. Further investigations have shown that selecting 65 and 75 features with the RFE methods led to an identical accuracy of 97.14%. A thorough examination of the selected features revealed the potential for three additional features from three specific sensors to enhance the classification accuracy to 97.38%. These results highlight the significance of employing an appropriate feature selection method to significantly reduce the number of necessary features while maintaining classification accuracy. They also underscore the necessity for further analysis and refinement to achieve optimal solutions.

Development of a Sign Language recognizer using LSTM

Sign language recognition plays a crucial role in bridging communication gaps between the deaf community and the hearing population. This paper presents the development of a sign language recognizer using Long Short-Term Memory (LSTM) networks, a type of recurrent neural network well-suited for sequence prediction tasks. The proposed model aims to accurately interpret hand gestures into text, enhancing accessibility and communication for individuals with hearing impairments. Our approach utilizes a custom-built dataset of sign language gestures, which undergoes preprocessing to extract relevant features such as hand position, orientation, and movement. The LSTM-based architecture is designed to capture the temporal dynamics of these gestures, enabling the recognition of complex sign language patterns. We train the model using a supervised learning approach. Our method demonstrates superior accuracy in recognizing sign language gestures compared to traditional machine learning methods and other deep learning architectures. This advancement contributes significantly to the field of assistive technologies. Future work will explore the integration of additional modalities, such as facial expressions and body movements, to further enhance the system's performance and usability.

Sign Language to Text Conversion using CNN

Sign language is a communication strategy used by those who are unable to hear. So those people who know sign language can communicate with people who are deaf. But a majority of our people don’t know sign language therefore there comes a communication gap between the ones who know sign language and others who don’t know. This project's major purpose is to bridge this gap by developing a systemthat recognizesmultiple sign languages and translates them into text in real-time. We use machine learning technologies to construct this system especially, convolutional neural networks (cnns), which are used to recognize and translate American Sign Language (ASL) into text by capturing it using a webcam. The transformed text is then presented on the screen by which individuals can comprehend and communicate with those who use sign language. The system's performance is evaluated on a dataset of ASL gestures, attaining excellent accuracy and indicating its potential for practical usage in enhancing communication accessibility for the deaf and hard-of-hearing community.

Virtual Mouse

Now a days the use of hand gesture recognition became popular due to the advance technology of artificial Intelligence (AI). In recent years, there has been a growing interest in developing alternative methods for human-computer interaction (HCI) that go beyond traditional input devices like keyboards and mouse. Hand gesture recognition has emerged as a promising approach, offering intuitive and natural ways for users to interact with digital interfaces. The proposed system utilizes computer vision techniques to detect and track hand gestures in real-time. By mapping specific gestures to mouse control commands, users can navigate and interact with graphical user interfaces (GUIs) without the need for physical input devices. In this paper A hand gesture- controlled virtual mouse system utilizes the AI and Machine learning algorithms to recognize the proper hand gestures and translate them into the mouse movements. The system we are designed that the people who have problems or difficulty using a traditional mouse or keyboard it will be appropriate for them. For this we are using camera that capture images of the user’s hand, which are processed by an AI and ML algorithm to recognize the gesture of hands being made. We trained the system by using a dataset of hand gestures. Keywords — Hand gesture recognition, Machine Learning, Artificial Intelligence, Virtual Mouse, Python, Media Pipe

Hand Gesture Recognition System Using Deep Learning

Hand Gesture Recognition Systems have undergone significant advancements, ushering in a new era of human-computer interaction. This paper offers a thorough examination of the current state of the art in hand gesture recognition, addressing both the notable progress achieved and the persistent challenges. By leveraging state-of-the-art technologies such as computer vision and deep learning, the paper explores the methodologies employed in data collection, preprocessing, and the implementation of various algorithms. The research delves into the complexities of popular hand gesture datasets, emphasizing their role in training and testing models. A critical analysis of different algorithms and models, including Hidden Markov Models, Support Vector Machines, and Neural Networks, is presented. The paper scrutinizes their strengths and limitations, providing insights into the delicate balance between accuracy and real-time processing. Furthermore, it investigates the diverse applications of hand gesture recognition, spanning from enriching human-computer interaction to its pivotal role in virtual reality, gaming, and robotics. Despite these advancements, challenges persist, such as occlusion, varying lighting conditions, and the imperative for real-time processing. The hardware utilized in hand gesture recognition systems, including depth sensors, RGB-D cameras, and wearable devices, is examined. Evaluation metrics, such as accuracy, precision, recall, and the F1 score, are employed to evaluate system performance.

Emotion and Gesture Recognition

Human Gestures and emotions play an important role in interpersonal relationships. The automatic recognition of emotions and gestures has been an active research topic from early eras. Emotions are reflected from facial expression, speech and gesture of the body. Hence understanding of emotion and gesture has high importance in interaction between humans(human-human) as well as between human and machine communication.In this system captured images are compared with the trained dataset available in the database and then display emotional state and gesture. For this system, fer13 (Facial Expressions Recognition 2013 dataset) and VGG16 (Visual Geometry Group) datasets are used to compare with captured images. In this system emotion and gesture recognition, both will run simultaneously. EDA(Exploratory Data Analysis) is used to analyse and validate the train data for both emotion as well as gesture dataset.

A channel-fused gated temporal convolutional network for EMG-based gesture recognition

Background:EMG gesture recognition can be widely applied in many fields, such as prosthetic control and human–computer interaction, to enhance the standard of human life. Purpose:This study aims to develop a deep learning-based model to identify multiple complex gestures from raw EMG signals. Methods:We propose a new channel-fused gated temporal convolutional network. First, a channel fusion and gating mechanism is designed to improve temporal convolutional networks, allowing the model to obtain higher-level features. Second, we improve the channel fusion module by the short-term average energy to fuse the EMG signals of multi-channels more accurately. Results:The model is evaluated on a public dataset of EMG gestures, NinaPro DB5. Results demonstrated that the unbalanced accuracy rate of 53 gesture actions reaches 92.71%, and the balanced accuracy rate 74.79%. Further, ablation experiments validates the effectiveness of each model module. Conclusions:This study demonstrates our proposed approach can improve gesture recognition accuracy for complex gestures and has great potential for practical applications.

Sign Language Recognition Using Convolutional Neural Network

This project introduces a real-time sign language detection system powered by Convolutional Neural Networks (CNNs), designed to aid individuals with hearing impairments in communication. Leveraging OpenCV for video capture and hand detection, along with custom modules for hand tracking and image classification, the system seamlessly integrates deep learning methodologies. A pre-trained CNN model, trained on a comprehensive dataset of sign language gestures, forms the core of the system, ensuring accurate classification in real-time. By capturing video frames from a webcam, detecting hands, and processing them through the CNN model, the system provides immediate feedback by overlaying predicted gestures onto the video stream. Through its effective implementation, this project underscores the potential of CNNs in facilitating accessibility and inclusivity, while also paving the way for future enhancements and applications in assistive technology and human-computer interaction. Top of Form Keywords— Real-time, Sign Language Detection, Convolutional Neural Networks (CNNs), Communication, Hearing Impairments, Deep Learning.

Hand Gesture Recognition for Video Player

Hand Gesture Recognition (HGR) is an emerging technology that has numerous applications in various fields, including human-computer interaction and robotics. In this project, we developed a Hand Gesture Recognition system for video players that recognize hand gestures and perform actions such as play, pause, rewind, and fast-forward. The system was developed using Python programming language and the OpenCV, mediapipe, pyautogui libraries. A dataset of hand gestures was created by recording videos using a webcam and annotating the frames with corresponding labels. The Hand Gesture Recognition system achieved an accuracy of 92% on the test set and was able to accurately recognize hand gestures and perform the corresponding actions on a video player. The system has the potential to be used as a novel way to control video players, especially in situations where the user cannot use a mouse or keyboard. In conclusion, the Hand Gesture Recognition system developed in this project provides a promising solution for controlling video players using hand gestures. The system achieved a high level of accuracy in recognizing gestures and performing actions, and can potentially be used in a variety of applications. Further improvements and refinements can be made to the system in the future to make it even more effective and user- friendly.

DEVELOPMENT OF MOBILE APPLICATION FOR SIGN LANGUAGE TRANSLATION USING DEEP LEARNING TECHNIQUE

Sign Language is a visual and gestural language used by deaf and hard-of-hearing people to communicate. However, communication between hearing and non-hearing individuals can be challenging due to the language barrier. In order to overcome this barrier, a Mobile based Translator is being proposed using Convolutional Neural Network (CNN) to recognize and translate hand gestures in real-time. The proposed system consists of a CNN model trained on a large dataset of hand gestures that includes various signs, such as the alphabet, numbers, and common phrases to recognize various signs, and a backend server to handle the translation of the recognized signs into text. The system was implemented using various CNN architectures like ResNet, MobileNet and VGG-16, where the later gave the best accuracy of 97.69%. The trained VGG-16 model recognizes the signs by extracting features from the images of the hand gestures and using these features to classify the gestures. Once a gesture is recognized, the backend server translates it into text using a pre-defined mapping of signs to words or phrases. The translated text will then be displayed to the user in real-time on the mobile app, enabling seamless communication between hearing and non-hearing individuals. The proposed system was implemented as a mobile app using Flutter, which is a cross-platform development framework. This mobile app make communication easier for vulnerable and enable sharing of information without any discrimination.

Spatial–temporal hypergraph based on dual-stage attention network for multi-view data lightweight action recognition

For the problems of irrelevant frames and high model complexity in action recognition, we propose a Spatial–Temporal Hypergraph based on Dual-Stage Attention Network (STHG-DAN) for multi-view data lightweight action recognition. It includes two stages: Temporal Attention Mechanism based on Trainable Threshold (TAM-TT) and Hypergraph Convolution based on Dynamic Spatial–Temporal Attention Mechanism (HG-DSTAM). In the first stage, TAM-TT uses a learning threshold to extract keyframes from multi-view videos, with the multi-view data serving as a guarantee for providing more comprehensive information subsequently; In the second stage, HG-DSTAM divides the human joints into three parts: trunk, hand and leg to build spatial–temporal hypergraphs, extracts high-order features from spatial–temporal hypergraphs constructed of multi-view human body joints, inputs them into the dynamic spatial–temporal attention mechanism, and learns the intra frame correlation of multi-view data between the joint features of body parts, which can obtain the significant areas of action; We use multi-scale convolution operation and depth separable network, which can realize efficient action recognition with a few trainable parameters. We experiment on the NTU-RGB+D, NTU-RGB+D 120 and the imitating traffic police gesture dataset. The performance and accuracy of the model are better than the existing algorithms, effectively improving the machine and human body language interaction cognitive ability.

Real-time Multi-CNN-based Emotion Recognition System for Evaluating Museum Visitors’ Satisfaction

Conventional studies on the satisfaction of museum visitors focus on collecting information through surveys to provide a one-way service to visitors, and thus it is impossible to obtain feedback on the real-time satisfaction of visitors who are experiencing the museum exhibition program. In addition, museum practitioners lack research on automated ways to evaluate a produced content program's lifecycle and its appropriateness. To overcome these problems, we propose a novel multi-convolutional neural network, called VimoNet, which is able to recognize visitors emotions automatically in real-time based on their facial expressions and body gestures. Furthermore, we design a user preference model of content and a framework to obtain feedback on content improvement for providing personalized digital cultural heritage content to visitors. Specifically, we define seven emotions of visitors and build a dataset of visitor facial expressions and gestures with respect to the emotions. Using the dataset, we proceed with feature fusion of face and gesture images trained on the DenseNet-201 and VGG-16 models for generating a combined emotion recognition model. From the results of the experiment, VimoNet achieved a classification accuracy of 84.10%, providing 7.60% and 14.31% improvement, respectively, over a single face and body gesture-based method of emotion classification performance. It is thus possible to automatically capture the emotions of museum visitors via VimoNet, and we confirm its feasibility through a case study with respect to digital content of cultural heritage.