Real-time Hand Gesture Recognition Research Articles

Quantized CNN-based efficient hardware architecture for real-time hand gesture recognition

Nowadays, Convolutional Neural Networks (CNN) have been widely adopted for vision-based hand gesture recognition. Several existing CNN architectures designed for gesture classification perform well with high accuracy but require a high memory footprint and processing when deployed on low-power embedded devices. To address this issue, we present a quantized CNN-based efficient framework for meeting real-life hand gesture recognition challenges. The proposed quantized CNN is designed and implemented using the FINN-based pipelined streaming architecture on an FPGA. Moreover, hardware-based optimizations are used to minimize the resources needed and to achieve fast memory access. Experimental results demonstrate that the developed recognition system achieves an average accuracy of 92% on the numeral database of Indian Sign Language (ISL). Additionally, our optimized design attains an inference latency of 0.85ms for real-time single gesture prediction on the PYNQ Zynq Ultrascale (ZU) FPGA, consuming only 3.63 W of power. The proposed design achieves a better trade-off between hardware resource utilization and speed performance, over previous designs.

Adapted Contrastive Predictive Coding Framework for Accessible Smart Home Control System Based on Hand Gestures Recognition

Smart home control systems have been widely used to control multiple smart home devices such as smart TVs, smart HVAC, and smart bulbs. While targeting the design of a smart home environment, the accessibility of the entire system is crucial to achieving a comfortable and accessible environment for all home individuals. Thus, in this paper, we are aiming to improve the daily quality of life of the elderly and Disabled individuals by improving the user experience of a vision-based accessible home control system based on an integrated self-supervised and supervised learning framework that provides a robust capability of extracting the features to enhance the overall performance via adapting the contrastive predictive coding concepts and deep learning for real-time hand gesture recognition system. The proposed framework provides an accessible smart home control system with a significant improvement in the user experience.

Real-time sign language recognition based on YOLO algorithm

This study focuses on real-time hand gesture recognition in the Turkish sign language detection system. YOLOv4-CSP based on convolutional neural network (CNN), a state-of-the-art object detection algorithm, is used to provide real-time and high-performance detection. The YOLOv4-CSP algorithm is created by adding CSPNet to the neck of the original YOLOv4 to improve network performance. A new object detection model has been proposed by optimizing the YOLOv4-CSP algorithm in order to provide more efficient detection in Turkish sign language. The model uses CSPNet throughout the network to increase the learning ability of the network. However, Proposed YOLOv4-CSP has a learning model with Mish activation function, complete intersection of union (CIoU) loss function and transformer block added. The Proposed YOLOv4-CSP algorithm has faster learning with transfer learning than previous versions. This allows the proposed YOLOv4-CSP algorithm to perform a faster restriction and recognition of static hand signals simultaneously. To evaluate the speed and detection performance of the proposed YOLOv4-CSP model, it is compared with previous YOLO series, which offers real-time detection, as well. YOLOv3, YOLOv3-SPP, YOLOv4-CSP and proposed YOLOv4-CSP models are trained with a labeled dataset consisting of numbers in Turkish Sign language, and their performances on the hand signals recognitions are compared. With the proposed method, 98.95% precision, 98.15% recall, 98.55 F1 score and 99.49% mAP results are obtained in 9.8 ms. The proposed method for detecting numbers in Turkish sign language outperforms other algorithms with both real-time performance and accurate hand sign prediction, regardless of background.

Comparative analysis on real-time hand gesture and sign language recognition using convexity defects and YOLOv3

Comparative analysis on real-time hand gesture and sign language recognition using convexity defects and YOLOv3

REAL-TIME HAND GESTURE DETECTION AND RECOGNITION USING CONVOLUTIONAL ARTIFICIAL NEURAL NETWORKS

Real-time hand gesture detection is an interesting topic in pattern recognition and computer vision. In this study, we propose the use of a Convolutional Neural Network (CNN) to detect and recognize hands in real-time. Our goal is to develop a system that can accurately identify and interpret user gestures in real-time. The proposed approach involves two main stages, namely hand gesture recognition and gesture recognition. For stage detection, we use the CNN architecture to recognize hands in the video. We train the CNN model using a dataset containing various hand gestures. Once a hand is detected, we extract the relevant hand region and proceed to the gesture recognition stage. The gesture recognition stage involves training and testing CNN models for different hand signal recognition. We use a hand gesture dataset that contains a variety of common hand signals. The experimental results show that the proposed system can detect and recognize hand movements in real-time with satisfactory accuracy. Although there are still some challenges that need to be overcome, this research provides a solid foundation for further development in real-time hand gesture recognition.

Static Hand Gesture Recognition Using Novel Convolutional Neural Network and Support Vector Machine

Hand tracking and identification through visual means pose a challenging problem. To simplify the identification of hand gestures, some systems have incorporated position markers or colored bands, which are not ideal for controlling robots due to their inconvenience. The motion recognition problem can be solved by combining object identification, recognition, and tracking using image processing techniques. A wide variety of target detection and recognition image processing methods are available. This paper proposes novel CNN-based methods to create a user-free hand gesture detection system. The use of synthetic techniques is recommended to improve recognition accuracy. The proposed method offers several advantages over existing methods, including higher accuracy and real-time hand gesture recognition suitable for sign language recognition and human-computer interaction. The CNN automatically extracts high-level characteristics from the source picture, and the SVM is used to classify these features. This study employed a CNN to automatically extract traits from raw EMG images, which is different from conventional feature extractors. The SVM classifier then determines which hand gestures are being made. Our tests demonstrate that the proposed strategy achieves superior accuracy compared to using only CNN.

Real-time hand gesture recognition using multiple deep learning architectures

Real-time hand gesture recognition using multiple deep learning architectures

Real-Time Hand Gesture Recognition by Decoding Motor Unit Discharges Across Multiple Motor Tasks From Surface Electromyography.

Surface electromyography (EMG) decomposition techniques have been developed to decode motor neuron activities non-invasively in the past decades, showing superior performance in human-machine interfaces such as gesture recognition and proportional control. However, neural decoding across multiple motor tasks and in real-time remains challenging, which limits its wide application. In this work, we proposed a real-time hand gesture recognition method by decoding motor unit (MU) discharges across multiple motor tasks ( 10) in a motion-wise way. The EMG signals were first divided into numerous segments related to motions. The convolution kernel compensation algorithm was applied for each segment individually. The local MU filters, which indicate the MU-EMG correlation for each motion, were calculated iteratively in each segment and reused for global EMG decomposition to trace the MU discharges across motor tasks in real-time. The motion-wise decomposition method was applied on the high-density EMG signals recorded during twelve hand gesture tasks from eleven non-disabled participants. The neural feature of discharge count was extracted for gesture recognition based on five common classifiers. On average, 164 ±34 MUs were identified for twelve motions from each subject, with a pulse-to-noise ratio of 32.1 ±5.6 dB. The average time cost of EMG decomposition in a sliding window of 50 ms was less than 5 ms. The average classification accuracy using a linear discriminant analysis classifier was 94.6 ±8.1%, which was significantly higher than that of a time-domain feature called root mean square. The superiority of the proposed method was also validated with a previously published EMG database comprising 65 gestures. These results indicate the feasibility and superiority of the proposed method for MU identification and hand gesture recognition across multiple motor tasks, extending the potential applications of neural decoding in human-machine interfaces.

AI-Based Sign Language Recognition System for Deaf and Hard-of-Hearing Individuals

Abstract: Sign language is a vital form of communication for the deaf and hard-of-hearing community. The development of sign language recognition systems plays a crucial role in bridging the communication gap between hearing and non-hearing populations. This paper presents a Sign Language Recognition System (SLRS) built using the MediaPipe Hands model and web technologies such as HTML, CSS, and JavaScript. The system utilizes real-time hand tracking and gesture recognition to interpret sign language gestures and provides corresponding textual output. The proposed system demonstrates the potential for accessible and inclusive communication through the integration of machine learning and web-based technologies.

Discerning of Gestures using Machine Learning

Abstract: Hand gestures can improve computer vision, and gesture-based innovation can provide a natural way to engage. Deafness is a hearing loss that prevents people fromunderstanding spoken language. Thanks to sign language, thegap in spoken language has also shrunk. Hand gestures are assessed in the same way as sign language is, displaying howdeaf people naturally communicate. We propose real-time hand gesture recognition in our study. Deaf people profit from the proposed CNN model design since it will help themovercome communication barriers. The proposed model wasable to recognise 11 different gestures with 94.61 percent accuracy using depth images.

MediaPipe Hands 모델 기반의 손 제스처 인식을 이용한 사용자 인터페이스

This paper is related to a user interface based on hand gestures. The purpose of this paper is to propose a real-time hand gesture recognition method that recognizes hand point, hand mouse, hand dial, and handwriting gestures using MediaPipe Hands model. In this paper, various static poses are detected using landmarks, which are the output values of MediaPipe Hands model, and then dynamic gestures are detected through the movement and change of these static poses. Then, using these dynamic gestures, hand point, hand mouse, hand dial, and hand writing functions are implemented. In this paper, when a usability test was performed on the proposed hand gesture recognition method, an average recognition rate of 96.4% was obtained. A computer simulation was performed focusing on the scenario of controlling a virtual smart home device using the proposed hand gesture recognition method.

A 184-μW Error-Tolerant Real-Time Hand Gesture Recognition System With Hybrid Tiny Classifiers Utilizing Edge CNN

This article proposes a low-power real-time hand gesture recognition (HGR) system with high recognition accuracy for smart edge devices. This design balances accuracy and power consumption by utilizing computation-efficient hybrid classifiers assisted with a majority voting scheme. By combining the recognition results of consecutive frames, the HGR system shows improved immunity to misclassification. In addition, the compressed input data before high-level processing dramatically reduce the on- chip memory and computational load. The proposed Edge-convolutional neural network (CNN) core with interactable processing engines reduces the memory accessing and the feature register toggling rate by 27% and 50%, respectively. The sequence analyzer based on majority voting improves the static and dynamic gesture recognition accuracy by ~7% and ~8% only with 9.4% hardware overhead. The test chip was fabricated in 65-nm CMOS technology, occupying the area of 1 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\times $ </tex-math></inline-formula> 1.5 mm2. It consumes the lowest power of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$184~\mu \text{W}$ </tex-math></inline-formula> at 25 MHz and 0.6 V. The proposed HGR system can recognize six static gestures and 24 dynamic hand gestures with an average accuracy of 87.25%–95% and 85.4%–94.9%, respectively.

Hand Gesture Recognition Using Densely Connected Deep Residual Network and Channel Attention Module for Mobile Robot Control

Navigating a mobile robot in an indoor or outdoor environment is an interesting research area for human-robot collaboration (HRC). In such a scenario, hand gesture offers some unique abilities for HRC to provide non-verbal communication between the user and robot. This paper proposes a novel real-time hand gesture recognition (HGR) technique for mobile robot control application. A compact convolutional neural network (CNN) based HGR system denoted as densely connected residual channel attention module (DRCAM) is proposed to recognize the vision-based hand gestures effectively. Since fingers are the most vital sign for HGR; an attention mechanism using multi-scale representation is proposed, which emphasizes finger information more effectively. The proposed CNN model employs the cascading structure of residual blocks with a multi-scale channel attention module to learn low to high-level information of hand gestures. In addition, the cascading structures are connected through dense connectivity, which strengthens the feature propagation and facilitates feature reuse. Experiments are conducted on an ingenuously developed dataset and a publicly available American sign language finger-spelling (ASL-FS) benchmarked dataset to evaluate the performance of the proposed technique. The experimental result illustrates that the proposed DRCAM network outperforms the state-of-the-art methods in terms of mean accuracy using the leave-one-subject-out cross-validation test. Finally, the training model is used to develop a software-based user interface system for the control of a mobile robot in a real-time environment.

Hand Gesture-based Teleoperation Control of a Mecanum-wheeled Mobile Robot

The aim of this study is to design an interface for the teleoperation control of a Mecanum-wheeled Mobile Robot (MMR). To make a cost-effective and non-invasive teleoperation interface approach, a hand gesture recognition strategy via a webcam is adopted. In this study, seven hand gestures are recognized on a desktop PC and then the corresponding velocity commands are sent to the MMR via WiFi. The SqueezeNet architecture is employed for the hand gesture recognition and MMR teleoperation task after various Convolutional Neural Network (CNN) models are compared. The comparison considers some aspects, i.e., accuracy, memory usage, and recognition speed, to find the most feasible method for real-time hand gesture recognition and teleoperation control. Experimental results are presented to show the excellence of the proposed SqueezeNet architecture.

Voice Conversion and Hand Gesture Recognition for Aponic People

Aphonia, a condition resulting in the loss of voice, presents significant challenges in interpersonal interactions. This project proposes a dual-pronged approach involving hand gesture recognition and voice conversion techniques to facilitate effective communication for aphonic individuals. The integration of real-time hand gesture recognition provides an alternative means of expressing ideas and emotions. By capturing and translating hand gestures into textual or auditory output, this approach offers a versatile mode of communication. Additionally, advanced voice conversion algorithms are employed to synthesize natural and intelligible speech from typed or selected text. This innovative coupling of technologies empowers aphonic individuals to engage in fluid conversations, fostering improved social interactions and enhancing their overall quality of life. A webcam is used to communicate with deaf and aphonic people. When there are modalities of communication, such as speech, that are unavailable, the human hand is the preferred option. Hand gestures that transmit concepts utilizing diverse forms and finger alignment enable human-machine interaction. The purpose of this work is to develop a hand gesture detection model and translate the results to text and audio formats. The model also responds to user voice commands and displays hand signs from the database.

Home Automation-Based Health Assessment Along Gesture Recognition via Inertial Sensors

Hand gesture recognition (HGR) is used in a numerous applications, including medical health-care, industrial purpose and sports detection. We have developed a real-time hand gesture recognition system using inertial sensors for the smart home application. Developing such a model facilitates the medical health field (elders or disabled ones). Home automation has also been proven to be a tremendous benefit for the elderly and disabled. Residents are admitted to smart homes for comfort, luxury, improved quality of life, and protection against intrusion and burglars. This paper proposes a novel system that uses principal component analysis, linear discrimination analysis feature extraction, and random forest as a classifier to improve HGR accuracy. We have achieved an accuracy of 94% over the publicly benchmarked HGR dataset. The proposed system can be used to detect hand gestures in the healthcare industry as well as in the industrial and educational sectors.

Light-Weight Deep Learning Techniques with Advanced Processing for Real-Time Hand Gesture Recognition

In the discipline of hand gesture and dynamic sign language recognition, deep learning approaches with high computational complexity and a wide range of parameters have been an extremely remarkable success. However, the implementation of sign language recognition applications for mobile phones with restricted storage and computing capacities is usually greatly constrained by those limited resources. In light of this situation, we suggest lightweight deep neural networks with advanced processing for real-time dynamic sign language recognition (DSLR). This paper presents a DSLR application to minimize the gap between hearing-impaired communities and regular society. The DSLR application was developed using two robust deep learning models, the GRU and the 1D CNN, combined with the MediaPipe framework. In this paper, the authors implement advanced processes to solve most of the DSLR problems, especially in real-time detection, e.g., differences in depth and location. The solution method consists of three main parts. First, the input dataset is preprocessed with our algorithm to standardize the number of frames. Then, the MediaPipe framework extracts hands and poses landmarks (features) to detect and locate them. Finally, the features of the models are passed after processing the unification of the depth and location of the body to recognize the DSL accurately. To accomplish this, the authors built a new American video-based sign dataset and named it DSL-46. DSL-46 contains 46 daily used signs that were presented with all the needed details and properties for recording the new dataset. The results of the experiments show that the presented solution method can recognize dynamic signs extremely fast and accurately, even in real-time detection. The DSLR reaches an accuracy of 98.8%, 99.84%, and 88.40% on the DSL-46, LSA64, and LIBRAS-BSL datasets, respectively.

An interactive game for rehabilitation based on real-time hand gesture recognition.

Currently, cardiovascular and cerebrovascular diseases have become serious global health problems related to their high incidence and fatality rate. Some patients with cardiovascular cerebro-cardiovascular diseases even may face motor or cognitive dysfunction after surgery. In recent years, human-computer interactive systems with artificial intelligence have become an important part of human well-being because they enable novel forms of rehabilitation therapies. We propose an interactive game utilizing real-time skeleton-based hand gesture recognition, which aims to assist rehabilitation exercises by improving the hand-eye coordination of the patients during a game-like experience. For this purpose, we propose a lightweight residual graph convolutional architecture for hand gesture recognition. Furthermore, we designed the whole system using the proposed gesture recognition module and some third-party modules. Finally, some participants were invited to test our system and most of them showed an improvement in their passing rate of the game during the test process.

An Integrated Real-Time Hand Gesture Recognition Framework for Human–Robot Interaction in Agriculture

Incorporating hand gesture recognition in human–robot interaction has the potential to provide a natural way of communication, thus contributing to a more fluid collaboration toward optimizing the efficiency of the application at hand and overcoming possible challenges. A very promising field of interest is agriculture, owing to its complex and dynamic environments. The aim of this study was twofold: (a) to develop a real-time skeleton-based recognition system for five hand gestures using a depth camera and machine learning, and (b) to enable a real-time human–robot interaction framework and test it in different scenarios. For this purpose, six machine learning classifiers were tested, while the Robot Operating System (ROS) software was utilized for “translating” the gestures into five commands to be executed by the robot. Furthermore, the developed system was successfully tested in outdoor experimental sessions that included either one or two persons. In the last case, the robot, based on the recognized gesture, could distinguish which of the two workers required help, follow the “locked” person, stop, return to a target location, or “unlock” them. For the sake of safety, the robot navigated with a preset socially accepted speed while keeping a safe distance in all interactions.

Long Hands gesture recognition system: 2 step gesture recognition with machine learning and geometric shape analysis

A two stage real-time hand gesture recognition system is presented. It combines a machine learning trained detection step with a colour processing contour shape validation step. The detection step is done with either Adaboost Cascades or Support Vector Machines using HOG features. The system achieves a low false positive rate and a sufficient true positive rate necessary for robust real-time performance. It performs well compared to MobileNets a state of the art Neural Network for mobile real-time applications.