Vision-based Hand Gesture Recognition Research Articles

Single Shot Detector CNN and Deep Dilated Masks for Vision-Based Hand Gesture Recognition From Video Sequences

Single Shot Detector CNN and Deep Dilated Masks for Vision-Based Hand Gesture Recognition From Video Sequences

Computer vision-based hand gesture recognition for human-robot interaction: a review

As robots have become more pervasive in our daily life, natural human-robot interaction (HRI) has had a positive impact on the development of robotics. Thus, there has been growing interest in the development of vision-based hand gesture recognition for HRI to bridge human-robot barriers. The aim is for interaction with robots to be as natural as that between individuals. Accordingly, incorporating hand gestures in HRI is a significant research area. Hand gestures can provide natural, intuitive, and creative methods for communicating with robots. This paper provides an analysis of hand gesture recognition using both monocular cameras and RGB-D cameras for this purpose. Specifically, the main process of visual gesture recognition includes data acquisition, hand gesture detection and segmentation, feature extraction and gesture classification, which are discussed in this paper. Experimental evaluations are also reviewed. Furthermore, algorithms of hand gesture recognition for human-robot interaction are examined in this study. In addition, the advances required for improvement in the present hand gesture recognition systems, which can be applied for effective and efficient human-robot interaction, are discussed.

DeReFNet: Dual-stream Dense Residual Fusion Network for static hand gesture recognition

Vision-based hand gesture recognition (HGR) system provides the most effective and natural way of interaction between humans and machines. However, the recognition performance of such an HGR system is challenging due to the variations in illumination, complex backgrounds, the shape of the user’s hand, and inter-class similarity. This work proposes a compact dual-stream dense residual fusion network (DeReFNet) to address the above challenges. The proposed convolutional neural network architecture mainly utilizes the strength of global features from each residual block of the residual stream and spatial information from the other stream using dense connectivity. Both the streams are fused to gather enriched information using the feature concatenation module. The efficacy of the DeReFNet is validated using a subject-independent cross-validation technique on four publicly available benchmark datasets. Furthermore, the qualitative and quantitative analysis of the benchmarked datasets illustrates that the DeReFNet outperforms state-of-the-art methods in terms of accuracy and computational time.

Patient Assistance System Based on Hand Gesture Recognition

We propose a two-stage hand gesture recognition architecture to support a patient assistance system. Some medical conditions limit mobility, and the patient must rely on medical staff to meet their needs. In such cases, a phone or intercom is not convenient to call for help. A vision-based system operated by changing the orientation of fingers can be used to send specific messages without making arm movements. However, vision-based hand gesture recognition is hindered by occlusion, background clutter, and variations in illumination. Therefore, we developed a two-stage architecture: The first stage produces a saliency map to simplify recognition and the second stage performs classification. A novel combined loss function optimizes the saliency detection model and makes the saliency map more precise. An adaptive kernel-based channel attention layer is used to emphasize salient features. The proposed architecture achieved precise saliency detection on four benchmark datasets and high-accuracy recognition on two. We designed an interface for patients to send specific messages to the medical staff using hand gestures. The interface help patients request assistance and connect with medical staff without leaving the bed or involving a third party.

Hand Gesture Recognition via Lightweight VGG16 and Ensemble Classifier

Gesture recognition has been studied for a while within the fields of computer vision and pattern recognition. A gesture can be defined as a meaningful physical movement of the fingers, hands, arms, or other parts of the body with the purpose to convey information for the environment interaction. For instance, hand gesture recognition (HGR) can be used to recognize sign language which is the primary means of communication by the deaf and mute. Vision-based HGR is critical in its application; however, there are challenges that will need to be overcome such as variations in the background, illuminations, hand orientation and size and similarities among gestures. The traditional machine learning approach has been widely used in vision-based HGR in recent years but the complexity of its processing has been a major challenge—especially on the handcrafted feature extraction. The effectiveness of the handcrafted feature extraction technique was not proven across various datasets in comparison to deep learning techniques. Therefore, a hybrid network architecture dubbed as Lightweight VGG16 and Random Forest (Lightweight VGG16-RF) is proposed for vision-based hand gesture recognition. The proposed model adopts feature extraction techniques via the convolutional neural network (CNN) while using the machine learning method to perform classification. Experiments were carried out on publicly available datasets such as American Sign Language (ASL), ASL Digits and NUS Hand Posture dataset. The experimental results demonstrate that the proposed model, a combination of lightweight VGG16 and random forest, outperforms other methods.

A Review on Vision-based Hand Gesture Recognition Targeting RGB-Depth Sensors

With the advancement of automation, vision-based hand gesture recognition (HGR) is gaining popularity due to its numerous uses and ability to easily communicate with machines. However, identifying hand positions is the most difficult assignment due to the fact of crowded backgrounds, sensitivity to light, form, speed, size, and self-occlusion. This review summarizes the most recent studies on hand postures and motion tracking using a vision-based approach by applying Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA). The parts and subsections of this review article are organized into numerous categories, the most essential of which are picture acquisition, preprocessing, tracking and segmentation, feature extraction, collation of key gesture identification phases, and classification. At each level, the various algorithms are evaluated based on critical key points such as localization, largest blob, per pixel binary segmentation, depth information, and so on. Furthermore, the datasets and future scopes of HGR approaches are discussed considering merits, limitations, and challenges.

A Structured and Methodological Review on Vision-Based Hand Gesture Recognition System.

Researchers have recently focused their attention on vision-based hand gesture recognition. However, due to several constraints, achieving an effective vision-driven hand gesture recognition system in real time has remained a challenge. This paper aims to uncover the limitations faced in image acquisition through the use of cameras, image segmentation and tracking, feature extraction, and gesture classification stages of vision-driven hand gesture recognition in various camera orientations. This paper looked at research on vision-based hand gesture recognition systems from 2012 to 2022. Its goal is to find areas that are getting better and those that need more work. We used specific keywords to find 108 articles in well-known online databases. In this article, we put together a collection of the most notable research works related to gesture recognition. We suggest different categories for gesture recognition-related research with subcategories to create a valuable resource in this domain. We summarize and analyze the methodologies in tabular form. After comparing similar types of methodologies in the gesture recognition field, we have drawn conclusions based on our findings. Our research also looked at how well the vision-based system recognized hand gestures in terms of recognition accuracy. There is a wide variation in identification accuracy, from 68% to 97%, with the average being 86.6 percent. The limitations considered comprise multiple text and interpretations of gestures and complex non-rigid hand characteristics. In comparison to current research, this paper is unique in that it discusses all types of gesture recognition techniques.

Motion Estimation and Hand Gesture Recognition-Based Human-UAV Interaction Approach in Real Time.

As an alternative to traditional remote controller, research on vision-based hand gesture recognition is being actively conducted in the field of interaction between human and unmanned aerial vehicle (UAV). However, vision-based gesture system has a challenging problem in recognizing the motion of dynamic gesture because it is difficult to estimate the pose of multi-dimensional hand gestures in 2D images. This leads to complex algorithms, including tracking in addition to detection, to recognize dynamic gestures, but they are not suitable for human–UAV interaction (HUI) systems that require safe design with high real-time performance. Therefore, in this paper, we propose a hybrid hand gesture system that combines an inertial measurement unit (IMU)-based motion capture system and a vision-based gesture system to increase real-time performance. First, IMU-based commands and vision-based commands are divided according to whether drone operation commands are continuously input. Second, IMU-based control commands are intuitively mapped to allow the UAV to move in the same direction by utilizing estimated orientation sensed by a thumb-mounted micro-IMU, and vision-based control commands are mapped with hand’s appearance through real-time object detection. The proposed system is verified in a simulation environment through efficiency evaluation with dynamic gestures of the existing vision-based system in addition to usability comparison with traditional joystick controller conducted for applicants with no experience in manipulation. As a result, it proves that it is a safer and more intuitive HUI design with a 0.089 ms processing speed and average lap time that takes about 19 s less than the joystick controller. In other words, it shows that it is viable as an alternative to existing HUI.

Towards a Feasible Hand Gesture Recognition System as Sterile Non-contact Interface in the Operating Room with 3D Convolutional Neural Network

Operating surgeons are constrained when interacting with computer systems as they traditionally utilize hand-held devices such as keyboard and mouse. Studies have previously proposed and shown the use of hand gestures is an efficient, touchless way of interfacing with such systems to maintain a sterile field. In this paper, we propose a Deep Computer Vision-based Hand Gesture Recognition framework to facilitate the interaction. We trained a 3D Convolutional Neural Network with a very large scale dataset to classify hand gestures robustly. This network became the core component of a prototype application requiring intraoperative navigation of medical images of a patient. Usability evaluation with surgeons demonstrates the application would work and a hand gesture lexicon that is germane to Medical Image Navigation was defined. By completing one cycle of usability engineering, we prove the feasibility of using the proposed framework inside the Operating Room.

Architecture Design and VLSI Implementation of 3D Hand Gesture Recognition System.

With advancements in technology, more and more research is being focused on enhancing daily life quality and convenience. Along with the increase in the development of gesture control systems, many controllers, such as the keyboard, mouse, and other devices, have been replaced with remote control products, which are gradually becoming more intuitive for users. However, vision-based hand gesture recognition systems still have many problems to overcome. Most hand detection methods adopt a skin filter or motion filter for pre-processing. However, in a noisy environment, it is not easy to correctly extract interesting objects. In this paper, a VLSI design with dual-cameras has been proposed to construct a depth map with a stereo matching algorithm and recognize hand gestures. The proposed system adopts an adaptive depth filter to separate interesting foreground objects from the background. We also propose dynamic gesture recognition using depth and coordinate information. The system can perform static and dynamic gesture recognition. The ASIC design is implemented in TSMC 90 nm with about 47.3 K gate counts, and 27.8 mW of power consumption. The average accuracy of each gesture recognition is 83.98%.

Methods, Databases and Recent Advancement of Vision-Based Hand Gesture Recognition for HCI Systems: A Review.

Hand gesture recognition is viewed as a significant field of exploration in computer vision with assorted applications in the human–computer communication (HCI) community. The significant utilization of gesture recognition covers spaces like sign language, medical assistance and virtual reality–augmented reality and so on. The underlying undertaking of a hand gesture-based HCI framework is to acquire raw data which can be accomplished fundamentally by two methodologies: sensor based and vision based. The sensor-based methodology requires the utilization of instruments or the sensors to be genuinely joined to the arm/hand of the user to extract information. While vision-based plans require the obtaining of pictures or recordings of the hand gestures through a still/video camera. Here, we will essentially discuss vision-based hand gesture recognition with a little prologue to sensor-based data obtaining strategies. This paper overviews the primary methodologies in vision-based hand gesture recognition for HCI. Major topics include different types of gestures, gesture acquisition systems, major problems of the gesture recognition system, steps in gesture recognition like acquisition, detection and pre-processing, representation and feature extraction, and recognition. Here, we have provided an elaborated list of databases, and also discussed the recent advances and applications of hand gesture-based systems. A detailed discussion is provided on feature extraction and major classifiers in current use including deep learning techniques. Special attention is given to classify the schemes/approaches at various stages of the gesture recognition system for a better understanding of the topic to facilitate further research in this area.

Vision-based hand gesture recognition using deep learning for the interpretation of sign language

Hand gestures have been the key component of communication since the beginning of an era. The hand gestures are the foundation of sign language, which is a visual form of communication. In this paper, a deep learning based convolutional neural network (CNN) model is specifically designed for the recognition of gesture-based sign language. This model has a compact representation that achieves better classification accuracy with a fewer number of model parameters over the other existing architectures of CNN. In order to evaluate the efficacy of this model, VGG-11 and VGG-16 have also been trained and tested in this work. To evaluate the performance, 2 datasets have been considered. First, in this work, a large collection of Indian sign language (ISL) gestures consisting of 2150 images is collected using RGB camera, and second, a publicly available American sign language (ASL) dataset is used. The highest accuracy of 99.96% and 100% is obtained by the proposed model for ISL and ASL datasets respectively. The performance of the proposed system, VGG-11, and VGG-16 are experimentally evaluated and compared with the existing state-of-art approaches. In addition to accuracy, other efficiency indices have been also used to ascertain the robustness of the proposed work. The findings indicate that the proposed model outperforms the existing techniques as it has the potential to classify maximum gestures with a minimal rate of error. The model is also tested with the augmented data and is found as invariant to rotation and scaling transformation.

Hand gesture recognition via enhanced densely connected convolutional neural network

Hand gesture recognition (HGR) serves as a fundamental way of communication and interaction for human being. While HGR can be applied in human computer interaction (HCI) to facilitate user interaction, it can also be utilized for bridging the language barrier. For instance, HGR can be utilized to recognize sign language, which is a visual language represented by hand gestures and used by the deaf and mute all over the world as a primary way of communication. Hand-crafted approach for vision-based HGR typically involves multiple stages of specialized processing, such as hand-crafted feature extraction methods, which are usually designed to deal with particular challenges specifically. Hence, the effectiveness of the system and its ability to deal with varied challenges across multiple datasets are heavily reliant on the methods being utilized. In contrast, deep learning approach such as convolutional neural network (CNN), adapts to varied challenges via supervised learning. However, attaining satisfactory generalization on unseen data is not only dependent on the architecture of the CNN, but also dependent on the quantity and variety of the training data. Therefore, a customized network architecture dubbed as enhanced densely connected convolutional neural network (EDenseNet) is proposed for vision-based hand gesture recognition. The modified transition layer in EDenseNet further strengthens feature propagation, by utilizing bottleneck layer to propagate the features being reused to all the feature maps in a bottleneck manner, and the following Conv layer smooths out the unwanted features. Differences between EDenseNet and DenseNet are discerned, and its performance gains are scrutinized in the ablation study. Furthermore, numerous data augmentation techniques are utilized to attenuate the effect of data scarcity, by increasing the quantity of training data, and enriching its variety to further improve generalization. Experiments have been carried out on multiple datasets, namely one NUS hand gesture dataset and two American Sign Language (ASL) datasets. The proposed EDenseNet obtains 98.50% average accuracy without augmented data, and 99.64% average accuracy with augmented data, outperforming other deep learning driven instances in both settings, with and without augmented data.

A Review of the Hand Gesture Recognition System: Current Progress and Future Directions

This paper reviewed the sign language research in the vision-based hand gesture recognition system from 2014 to 2020. Its objective is to identify the progress and what needs more attention. We have extracted a total of 98 articles from well-known online databases using selected keywords. The review shows that the vision-based hand gesture recognition research is an active field of research, with many studies conducted, resulting in dozens of articles published annually in journals and conference proceedings. Most of the articles focus on three critical aspects of the vision-based hand gesture recognition system, namely: data acquisition, data environment, and hand gesture representation. We have also reviewed the performance of the vision-based hand gesture recognition system in terms of recognition accuracy. For the signer dependent, the recognition accuracy ranges from 69% to 98%, with an average of 88.8% among the selected studies. On the other hand, the signer independent’s recognition accuracy reported in the selected studies ranges from 48% to 97%, with an average recognition accuracy of 78.2%. The lack in the progress of continuous gesture recognition could indicate that more work is needed towards a practical vision-based gesture recognition system.

Multi‐view hand gesture recognition via pareto optimal front

Computational cost-ineffectiveness and ambiguity due to self-occlusion are bottlenecks of vision-based hand gesture recognition. In this study, the authors address these issues by proposing a novel multi-view hand gesture recognition method based on Pareto optimal front. They first present an oriented gradient local binary pattern operator to generate a groupwise gesture feature data set for multi-view hand gesture images. Then they take hand gesture recognition over multi-view hand images as a multi-query image retrieval problem and Pareto optimal front is constructed based on the dissimilarities between the testing images and sample images. The gesture corresponding to the point with the shortest distance to the origin on the Pareto optimal front is the final recognised result. Extensive experiments verify the accuracy and efficiency of their Pareto optimal font-based method.

Convolutional neural network with spatial pyramid pooling for hand gesture recognition

Hand gesture provides a means for human to interact through a series of gestures. While hand gesture plays a significant role in human–computer interaction, it also breaks down the communication barrier and simplifies communication process between the general public and the hearing-impaired community. This paper outlines a convolutional neural network (CNN) integrated with spatial pyramid pooling (SPP), dubbed CNN–SPP, for vision-based hand gesture recognition. SPP is discerned mitigating the problem found in conventional pooling by having multi-level pooling stacked together to extend the features being fed into a fully connected layer. Provided with inputs of varying sizes, SPP also yields a fixed-length feature representation. Extensive experiments have been conducted to scrutinize the CNN–SPP performance on two well-known American sign language (ASL) datasets and one NUS hand gesture dataset. Our empirical results disclose that CNN–SPP prevails over other deep learning-driven instances.

Vision-Based Hand Gesture Recognition Techniques using Smartphones

In the past few years, the computational performance of smartphone devices has seen tremendous growth. Due to which the smartphone has become a suitable platform for various computer-vision based applications which earlier was not possible. In this paper, we study various methods through which we can achieve computer vision-based hand gesture recognition natively on smartphones. If smartphones can support hand gesture recognition it can provide a new way to interact with mobile devices and overcome the hurdles of voice and touch-based user interface improving the user experience at the same time also supports other gesture-based applications. The techniques we study are mainly vision-based since camera module is present on most of the smartphones and it does not require other additional sensors or other hardware. We have compared the various methods available based on algorithms used and corresponding accuracy.

RETRACTED ARTICLE: Development of hand gesture recognition system using machine learning

Human computer interaction (HCI) systems are increasing due to the demand for non-intrusive methods for communicating with machines. In this research article, vision based hand gesture recognition (HGR) System has been proposed using machine learning. This proposed system consists of three stages: segmentation, feature extraction and classification. The developed system is to be trained and tested using Sebastian Marcel static hand posture database which is available online. Discrete wavelet transform (DWT) along with modified Speed Up Robust Feature extraction technique has been used to extract rotation and scale invariant key descriptors. Then Bag of Word technique is used to develop the fixed dimension input vector that is required for the support vector machine. The classification accuracy of class 2 and class 4 which corresponds to the ‘No’ and ‘grasp’ gesture has reached 98%. The overall classification accuracy of the HGR system using SVM classifier is 96.5% with a recognition time of 0.024 s. Due to fast recognition time, this system can be employed in real time gesture image recognition system. Our HGR system addresses the complex background problem and also improves the robustness of hand gesture recognition.

A survey on hand gesture recognition for mobile devices

As smartphones and mobile devices become ubiquitous in the daily life of human beings, facilitating interaction with them will have a positive impact on their use. This paper presents an emerging and important area for research on human-machine interaction, involving interaction with mobile devices. Hand gesture recognition is already a quite mature field and its integration on advanced technologies like smartphones makes a survey an interesting and needed contribution. The design of interactive mobile applications should differ from traditional desktop applications. For this purpose, the paper discusses some concepts that help to understand new challenges and introduces techniques that may be useful to properly exploit these devices. This paper gives also an overview of recent works done on vision-based and contact-based hand gesture recognition systems for mobile devices. Used techniques and approaches are also discussed. At the end, we conclude with some reflections.

Text Generation through Hand Gesture Recognition

Development of Vision-based Hand Gesture Recognition systems has proved to be beneficial for disabled people over the past. These systems are user friendly and inexpensive. They also make room for contemporary endeavours in the Human-Computer Interaction area. Gestures can control the cursor, the music player, or even play games. Despite all the improvements made in Vision-based recognition systems, a fast, accurate, and reliable system is yet to see the light of the day. This study aims to understand the previous work in this field and develop a fast and accurate alternative to perform the same task. To render the system faster, we propose to replace the histograms from the previous approach and uses native OpenCV functions. The Use of these functions can reduce the time for calculating the histogram and loading it when required. The final system developed, succeeded in predicting 37 gestures with a 98% accuracy. The system also has a calculator mode, where some gestures are reserved for operators and others for operands. This study tells us how we can use different computer vision techniques to make the recognition process more memory efficient and lightweight. In the future, this could be used as a commercially viable option by putting it on app stores.