Automatic Gesture Recognition Research Articles

Emotion and Gesture Recognition

Human Gestures and emotions play an important role in interpersonal relationships. The automatic recognition of emotions and gestures has been an active research topic from early eras. Emotions are reflected from facial expression, speech and gesture of the body. Hence understanding of emotion and gesture has high importance in interaction between humans(human-human) as well as between human and machine communication.In this system captured images are compared with the trained dataset available in the database and then display emotional state and gesture. For this system, fer13 (Facial Expressions Recognition 2013 dataset) and VGG16 (Visual Geometry Group) datasets are used to compare with captured images. In this system emotion and gesture recognition, both will run simultaneously. EDA(Exploratory Data Analysis) is used to analyse and validate the train data for both emotion as well as gesture dataset.

EMG-Based Automatic Gesture Recognition Using Lipschitz-Regularized Neural Networks

This article introduces a novel approach for building a robust Automatic Gesture Recognition system based on Surface Electromyographic (sEMG) signals, acquired at the forearm level. Our main contribution is to propose new constrained learning strategies that ensure robustness against adversarial perturbations by controlling the Lipschitz constant of the classifier. We focus on nonnegative neural networks for which accurate Lipschitz bounds can be derived, and we propose different spectral norm constraints offering robustness guarantees from a theoretical viewpoint. Experimental results on four publicly available datasets highlight that a good tradeoff in terms of accuracy and performance is achieved. We then demonstrate the robustness of our models, compared with standard trained classifiers in four scenarios, considering both white-box and black-box attacks.

Football referee gesture recognition algorithm based on YOLOv8s.

Gesture serves as a crucial means of communication between individuals and between humans and machines. In football matches, referees communicate judgment information through gestures. Due to the diversity and complexity of referees' gestures and interference factors, such as the players, spectators, and camera angles, automated football referee gesture recognition (FRGR) has become a challenging task. The existing methods based on visual sensors often cannot provide a satisfactory performance. To tackle FRGR problems, we develop a deep learning model based on YOLOv8s. Three improving and optimizing strategies are integrated to solve these problems. First, a Global Attention Mechanism (GAM) is employed to direct the model's attention to the hand gestures and minimize the background interference. Second, a P2 detection head structure is integrated into the YOLOv8s model to enhance the accuracy of detecting smaller objects at a distance. Third, a new loss function based on the Minimum Point Distance Intersection over Union (MPDIoU) is used to effectively utilize anchor boxes with the same shape, but different sizes. Finally, experiments are executed on a dataset of six hand gestures among 1,200 images. The proposed method was compared with seven different existing models and 10 different optimization models. The proposed method achieves a precision rate of 89.3%, a recall rate of 88.9%, a mAP@0.5 rate of 89.9%, and a mAP@0.5:0.95 rate of 77.3%. These rates are approximately 1.4%, 2.0%, 1.1%, and 5.4% better than those of the newest YOLOv8s, respectively. The proposed method has right prospect in automated gesture recognition for football matches.

Research of Stages, Types of Modeling and Methods of Gesture Recognition

It is impossible to leave aside the problem of communication with the outside world and the social integration of people with disabilities in modern society, where there is a rapid development of communication methods between people. Gesture is the only method of communication for people with disabilities (auditory and language). The research provides information on static and dynamic gestures that allow people with communication problems to exchange information. It is noted that the problem of automatic gesture recognition is solved by various mathematical methods, algorithms and computer systems. The advantages and disadvantages of 2D and 3D models for sequential recognition of hand gestures are shown. It is noted that the creation of a national electronic database that recognizes the national dactyl alphabet and gestures in the Republic of Azerbaijan is an urgent problem. The well-known methods of this category are shown. Artificial neural network method, Hidden Markov method, Random Forest method and method of designating a hand gesture using gloves painted with markers of different colors. The creation of a national electronic database that recognizes the national dactyl alphabet and gestures is proposed, which will partially eliminate the existing barrier in education and problems of communication between people.

Automated Gesture Recognition Using Zebra Optimization Algorithm with Deep Learning Model for Visually Challenged People

Gesture recognition for visually challenged people plays a vital role in improving their convenience and interaction with digital gadgets and environments. It includes improvement of systems that permit them to relate with digital devices by using hand actions or gestures. To improve user-friendliness, these systems select in-built and effortlessly learnable gestures, often integrating wearable devices prepared with sensors for precise detection. Incorporating auditory or haptic feedback devices offers real-time cues about achievement of familiar gestures. Machine learning (ML) and deep learning (DL) methods are useful tools for accurate gesture detection, with customization choices to accommodate individual preferences. In this view, this article concentrates on design and development of Automated Gesture Recognition using Zebra Optimization Algorithm with Deep Learning (AGR-ZOADL) model for Visually Challenged People. The AGR-ZOADL technique aims to recognize the gestures to aid visually challenged people. In the AGR-ZOADL technique, the primary level of data pre-processing is involved by median filtering (MF). Besides, the AGR-ZOADL technique applies NASNet model to learn complex features from the preprocessed data. To enhance performance of NASNet technique, ZOA based hyperparameter procedure performed. For gesture recognition process, stacked long short term memory (SLSTM) model is applied. The performance validation of AGR-ZOADL technique carried out using a benchmark dataset. The experimental values stated that AGR-ZOADL methodology extents significant performance over other present approaches

Automated Gesture Recognition Using African Vulture Optimization with Deep Learning for Visually Impaired People on Sensory Modality Data

Gesture recognition for visually impaired persons (VIPs) is a useful technology for enhancing their communications and increasing accessibility. It is vital to understand the specific needs and challenges faced by VIPs when planning a gesture recognition model. But, typical gesture recognition methods frequently depend on the visual input (for instance, cameras); it can be vital to discover other sensory modalities for input. The deep learning (DL)-based gesture recognition method is effective for the interaction of VIPs with their devices. It offers a further intuitive and natural way of relating with technology, creating it more available for everybody. Therefore, this study presents an African Vulture Optimization with Deep Learning-based Gesture Recognition for Visually Impaired People on Sensory Modality Data (AVODL-GRSMD) technique. The AVODL-GRSMD technique mainly focuses on the utilization of the DL model with hyperparameter tuning strategy for a productive and accurate gesture detection and classification process. The AVODL-GRSMD technique utilizes the primary data preprocessing stage to normalize the input sensor data. The AVODL-GRSMD technique uses a multi-head attention-based bidirectional gated recurrent unit (MHA-BGRU) method for accurate gesture recognition. Finally, the hyperparameter optimization of the MHA-BGRU method can be performed by the use of African Vulture Optimization with Deep Learning (AVO) approach. A series of simulation analyses were performed to demonstrate the superior performance of the AVODL-GRSMD technique. The experimental values demonstrate the better recognition rate of the AVODL-GRSMD technique compared to that of the state-of-the-art models.

Sign Language Recognition Based on Hands Symbol's Classification

One of the most natural and ancient types of conversational language is sign language. The technology that converts sign language into writing for people who have difficulty communicating, such as those who have speech issues, hearing disabilities, or are deaf, is the subject of this study. This paper is based on a real-time method based on finger writing and neural networks for American sign language. An interesting field of vision study is the automatic recognition of human gestures from video images. The Research recommend employing a convolution neural network (CNN) method to recognize human hand motions from a photograph. The objective is to identify hand movements used in human work activities from a camera image. Hand placement and orientation are employed to collect the training and assessment data for CNN. The hand is first put through a filter, and once that has been done, it is put through a classification, which determines what class the hand movements belong to. Then, CNN is trained using the measured pictures.

A Comparative Analysis of Gesture Recording Technologies and Recognition Methods

The dynamic development of computing techniques and communication tools and the improvement of network technology have increased the role of information as the main resource in society. The application of information communication technologies has stimulated the development of intellectual and scientific potential all over the world and has been successfully applied to all fields. Gestures are the only means of communication for hearing and speech impaired people. Automatic recognition of gestures in order to facilitate communication through gestures is an urgent issue from both scientific and practical point of view. This article highlights the static and dynamic gestures. The process of gesture recognition collects data by means of various sensor technologies. The article analyzes the image-based and non-image-based technologies and presents their advantages and disadvantages. It also comparatively analyzes the working principle of existing methods proposed for the gestures identification, explores their advantages and disadvantages and interprets the performance of the software which localizes the hand showing the gesture in the video frame. As a result, it develops a machine learning method based on neural networks for high accuracy identification of gestures. The developed method testing on database open for research shows high performance.

Sign Language Recognition Method Based on Palm Definition Model and Multiple Classification.

Technologies for pattern recognition are used in various fields. One of the most relevant and important directions is the use of pattern recognition technology, such as gesture recognition, in socially significant tasks, to develop automatic sign language interpretation systems in real time. More than 5% of the world’s population—about 430 million people, including 34 million children—are deaf-mute and not always able to use the services of a living sign language interpreter. Almost 80% of people with a disabling hearing loss live in low- and middle-income countries. The development of low-cost systems of automatic sign language interpretation, without the use of expensive sensors and unique cameras, would improve the lives of people with disabilities, contributing to their unhindered integration into society. To this end, in order to find an optimal solution to the problem, this article analyzes suitable methods of gesture recognition in the context of their use in automatic gesture recognition systems, to further determine the most optimal methods. From the analysis, an algorithm based on the palm definition model and linear models for recognizing the shapes of numbers and letters of the Kazakh sign language are proposed. The advantage of the proposed algorithm is that it fully recognizes 41 letters of the 42 in the Kazakh sign alphabet. Until this time, only Russian letters in the Kazakh alphabet have been recognized. In addition, a unified function has been integrated into our system to configure the frame depth map mode, which has improved recognition performance and can be used to create a multimodal database of video data of gesture words for the gesture recognition system.

Body Language Analysis in Healthcare: An Overview

Given the current COVID-19 pandemic, medical research today focuses on epidemic diseases. Innovative technology is incorporated in most medical applications, emphasizing the automatic recognition of physical and emotional states. Most research is concerned with the automatic identification of symptoms displayed by patients through analyzing their body language. The development of technologies for recognizing and interpreting arm and leg gestures, facial features, and body postures is still in its early stage. More extensive research is needed using artificial intelligence (AI) techniques in disease detection. This paper presents a comprehensive survey of the research performed on body language processing. Upon defining and explaining the different types of body language, we justify the use of automatic recognition and its application in healthcare. We briefly describe the automatic recognition framework using AI to recognize various body language elements and discuss automatic gesture recognition approaches that help better identify the external symptoms of epidemic and pandemic diseases. From this study, we found that since there are studies that have proven that the body has a language called body language, it has proven that language can be analyzed and understood by machine learning (ML). Since diseases also show clear and different symptoms in the body, the body language here will be affected and have special features related to a particular disease. From this examination, we discovered that it is possible to specialize the features and language changes of each disease in the body. Hence, ML can understand and detect diseases such as pandemic and epidemic diseases and others.

Superresolution Reconstruction in Automatic Thai Sign Language Feature Extraction Using Adaptive Triangulation Interpolation

Superresolution is an image processing technique for improving image quality and enhancing low-resolution images. This paper presents a novel interpolation method for increasing superresolution reconstruction effectiveness using a triangulation interpolation algorithm for automatic Thai sign language feature extraction. This approach uses three neighboring pixels for estimation. The experiment compared the superresolution reconstruction performance using the triangulation interpolation algorithm and the nearest-neighbor, SRCNN, bilinear, bicubic, GPR, and NEDI methods. The super-resolution reconstruction using the triangulation-improved interpolation technique provided the best PSNR measurements of image quality between the original and superresolution-reconstructed images. The PSNR value of the sign language image was 40.608, improving performance by 13.15%. The Thai sign language gesture recognition using 2D convolutional neural networks showed that the designed model increased the gesture recognition effectiveness with an accuracy of 0.95 and loss of 0.14. Thus, this study provides state-of-the-art superresolution reconstruction for automatic Thai sign language gesture recognition.

Dynamic hand gesture recognition of sign language using geometric features learning

In the sign language alphabet, several hand signs are in use. Automatic recognition of dynamic hand gestures could facilitate several applications such as people with a speech impairment to communi...

Hybrid FiST_CNN approach for Feature Extraction for Vision-Based Indian Sign Language Recognition

Indian sign language (ISL) is the commonly used language by the deaf-mute community in the Indian continent. Effective feature extraction is essential for the automatic recognition of gestures. This paper aims at developing an efficient feature extraction technique using FAST, SIFT, and CNN. Features from Fast Accelerated Segment Test(FAST) with Scale-invariant Feature Transformation(SIFT) are used to detect and compute features, respectively. CNN is used for classification with the hybridization of FAST-SIFT features. The system is implemented and tested using the python-based library Keras. The results of the proposed techniques have been tested on 34 gestures of ISL (24 alphabet sets and 10 digit sets) and then compared with the CNN and SIFT_CNN, and it is also tested on two publicly available datasets on Jochen Trisech Dataset(JTD) and NUS-II dataset. The proposed study outperformed some existing ISLR works with an accuracy of 97.89%, 95.68%, 94.90% and 95.87% for ISL-alphabets, MNIST, JTD and NUS-II, respectively.

Hybrid FiST_CNN approach for Feature Extraction for Vision-Based Indian Sign Language Recognition

Indian sign language (ISL) is the commonly used language by the deaf-mute community in the Indian continent. Effective feature extraction is essential for the automatic recognition of gestures. This paper aims at developing an efficient feature extraction technique using FAST, SIFT, and CNN. Features from Fast Accelerated Segment Test(FAST) with Scale-invariant Feature Transformation(SIFT) are used to detect and compute features, respectively. CNN is used for classification with the hybridization of FAST-SIFT features. The system is implemented and tested using the python-based library Keras. The results of the proposed techniques have been tested on 34 gestures of ISL (24 alphabet sets and 10 digit sets) and then compared with the CNN and SIFT_CNN, and it is also tested on two publicly available datasets on Jochen Trisech Dataset(JTD) and NUS-II dataset. The proposed study outperformed some existing ISLR works with an accuracy of 97.89%, 95.68%, 94.90% and 95.87% for ISL-alphabets, MNIST, JTD and NUS-II, respectively.

Analytical review of models and methods for automatic recognition of gestures and sign languages

Introduction: Currently, the recognition of gestures and sign languages is one of the most intensively developing areas in computer vision and applied linguistics. The results of current investigations are applied in a wide range of areas, from sign language translation to gesture-based interfaces. In that regard, various systems and methods for the analysis of gestural data are being developed. Purpose: A detailed review of methods and a comparative analysis of current approaches in automatic recognition of gestures and sign languages. Results: The main gesture recognition problems are the following: detection of articulators (mainly hands), pose estimation and segmentation of gestures in the flow of speech. The authors conclude that the use of two-stream convolutional and recurrent neural network architectures is generally promising for efficient extraction and processing of spatial and temporal features, thus solving the problem of dynamic gestures and coarticulations. This solution, however, heavily depends on the quality and availability of data sets. Practical relevance: This review can be considered a contribution to the study of rapidly developing sign language recognition, irrespective to particular natural sign languages. The results of the work can be used in the development of software systems for automatic gesture and sign language recognition.

Hybrid FAST-SIFT-CNN (HFSC) approach for Vision-Based Indian Sign Language Recognition

Indian Sign Language (ISL) is the conventional means of communication for the deaf-mute community in the Indian subcontinent. Accurate feature extraction is one of the prime challenges in automatic gesture recognition of ISL gestures. In this paper, a hybrid approach, namely HFSC, integrating FAST and SIFT with CNN has been proposed for automatic and accurate recognition of ISL's static and single-hand gestures. Features from accelerated segment test (FAST) and scale-invariant feature transform (SIFT) provides the basic framework for feature extraction while CNN is used for classification. The performance of HFSC is compared with existing sign language recognition approaches by testing on standard benchmark (MNIST, Jochen-Trisech, and NUS hand posture-II) datasets. The HFSC algorithm's efficiency has been shown by comparing it with CNN and SIFT_CNN for a uniform dataset with an accuracy of 97.89%. Furthermore, the Computational results of the HFSC on complex background dataset achieve comparable accuracy of 95%.

Automatic Recognition System of Motion Posture Based on Information Video Processing Technology

With the rapid development of science and technology, information video processing technology has become a research field that has attracted widespread attention. Automatic recognition of motion gestures is an important research object in the field of information technology. It has been widely used in education, medical treatment and finance, and has promoted the further development of this technology. The application of information video processing technology in motion posture and automatic recognition of motion posture contribute to better scientific education and technical diagnosis. This article is based on the research of automatic motion detection system based on video information processing technology. Based on the summary and analysis of relevant research work at home and abroad, this paper mainly studies the detection of moving posture targets, the segmentation of moving posture targets, and the automatic recognition of moving posture tracking in video sequences based on the characteristics of the motion posture. This research aims to effectively improve the effect of automatic recognition of sports high poses while appropriately reducing the computational complexity. The research focuses on finding simpler and more informative automatic recognition methods of sports poses, and a large number of targeted methods have been carried out. This research uses these algorithms to extract moving images from videos, and proposes an automatic recognition method of motion gestures based on information video processing technology. Experimental results show that the average error rate of the system designed in this paper is 89.5%, which can meet the current application requirements for automatic recognition of motion gestures.

Approaches to Automatic Gesture Recognition: Hardware and Methods Overview.

In this paper, hardware and software solutions addressed to automatic gesture recognition are considered. Trends in image analysis in the current computer vision-based approaches are analysed. Each of the considered approaches was addressed, in order to reveal their advantages and drawbacks. Research papers on the usability of gesture interfaces were reviewed. It was revealed that sensor-based systems, being quite accurate and demonstrating high speed of recognition, have limited application due to the specificity of devices (gloves, suit) and their relatively narrow distribution. At the same time, computer vision-based approaches can be successfully applied only when problems of occlusions and datasets are solved. The results obtained can be used for designing training systems.

Real-Time Hand Gesture Recognition

The automatic recognition of gestures enriches Human – Computer Interaction by offering a natural intuitive method of data input. Compared with the traditional interaction approaches, such as keyboard, mouse, pen etc. Vision based hand interaction is more natural and efficient. As this methodology is aided with cameras and computer vision techniques. And besides normal use for handling objects and manipulating tools, the hand of human being can be used as the mean of communication here. This paper presents a new method or technique for a real time static hand gesture recognition on plain uniform background, for the HCI, based on pattern recognition using K-nearest Neighbor (KNN). Major consideration here is the stable lighting condition. Here work is done to recognize 10 gestures from ASL. Concept of connected components or objects is used here for hand segmentation, canny edge detector is used for pattern creation and classification is done by using K-nearest Neighbor algorithm. This algorithm gives 84% accuracy with recognition time of 0.3s. Accuracy decreases as distance of hand from camera increases above 1.5feets. The gestures recognized here can be used further to develop an application of controlling the devices which will be very much helpful to the physically disabled and older persons who can’t move easily from one place to another.

Gesture Recognition in Robotic Surgery: A Review.

Surgical activity recognition is a fundamental step in computer-assisted interventions. This paper reviews the state-of-the-art in methods for automatic recognition of fine-grained gestures in robotic surgery focusing on recent data-driven approaches and outlines the open questions and future research directions. An article search was performed on 5 bibliographic databases with the following search terms: robotic, robot-assisted, JIGSAWS, surgery, surgical, gesture, fine-grained, surgeme, action, trajectory, segmentation, recognition, parsing. Selected articles were classified based on the level of supervision required for training and divided into different groups representing major frameworks for time series analysis and data modelling. A total of 52 articles were reviewed. The research field is showing rapid expansion, with the majority of articles published in the last 4 years. Deep-learning-based temporal models with discriminative feature extraction and multi-modal data integration have demonstrated promising results on small surgical datasets. Currently, unsupervised methods perform significantly less well than the supervised approaches. The development of large and diverse open-source datasets of annotated demonstrations is essential for development and validation of robust solutions for surgical gesture recognition. While new strategies for discriminative feature extraction and knowledge transfer, or unsupervised and semi-supervised approaches, can mitigate the need for data and labels, they have not yet been demonstrated to achieve comparable performance. Important future research directions include detection and forecast of gesture-specific errors and anomalies. This paper is a comprehensive and structured analysis of surgical gesture recognition methods aiming to summarize the status of this rapidly evolving field.