Gesture Recognition Module Research Articles

Talking Hands & D and D (Dump & Deaf) Assistant

The largest issue facing the deaf and dumb community these days is communication with both the general public and fellow deaf and dumb people. Ordinary individuals can learn new information from spoken language, everyday conversations, and surrounding sounds. Individuals who are hard of hearing or deaf do not enjoy this luxury. These people will gain from this approach since it provides a means of communication. The term "assistive technology" describes gadgets that are rehabilitative, adaptive, and made for people with disabilities. This project aims to develop a system that can recognize sign language, bridging the communication gap by enabling those with speech impairments to interact with others who do not. Hand gestures are important because, in contrast to other techniques, they enable people to express their ideas rapidly. The current work aims to recognize signs using an image processing-based gesture recognition module and convert the corresponding speech to text using a text-to-speech synthesizer

Gaming using different hand gestures using artificial neural network

INTRODUCTION: Gaming has evolved over the years, and one of the exciting developments in the industry is the integration of hand gesture recognition.
 OBJECTIVES: This paper proposes gaming using different hand gestures using Artificial Neural Networks which allows players to interact with games using natural hand movements, providing a more immersive and intuitive gaming experience.
 METHODS: Introduces two modules: recognition and analysis of gestures. The gesture recognition module identifies the gestures, and the analysis module assesses them to execute game controls based on the calculated analysis.
 RESULTS: The main results obtained in this paper are enhanced accessibility, higher accuracy and improved performance.
 CONCLUSION: To communicate with any of the traditional systems, physical contact is necessary. In the hand gesture recognition system, the same functionality can be interpreted by gestures without requiring physical contact with the interfaced devices.

The real-time hand and object recognition for virtual interaction.

Recognizing hand-object interactions presents a significant challenge in computer vision. It arises due to the varying nature of hand-object interactions. Moreover, estimating the 3D position of a hand from a single frame can be problematic, especially when the hand obstructs the view of the object from the observer's perspective. In this article, we present a novel approach to recognizing objects and facilitating virtual interactions, using a steering wheel as an illustrative example. We propose a real-time solution for identifying hand-object interactions in eXtended reality (XR) environments. Our approach relies on data captured by a single RGB camera during a manipulation scenario involving a steering wheel. Our model pipeline consists of three key components: (a) a hand landmark detector based on the MediaPipe cross-platform hand tracking solution; (b) a three-spoke steering wheel model tracker implemented using the faster region-based convolutional neural network (Faster R-CNN) architecture; and (c) a gesture recognition module designed to analyze interactions between the hand and the steering wheel. This approach not only offers a realistic experience of interacting with steering-based mechanisms but also contributes to reducing emissions in the real-world environment. Our experimental results demonstrate the natural interaction between physical objects in virtual environments, showcasing precision and stability in our system.

Real three-dimensional medical interactive display system based on integral imaging and gesture recognition

In this paper, we developed a real three-dimensional medical interactive display system based on integral imaging and gesture recognition. The system includes a fast image processing module, a high-performance three-dimensional display module, and a gesture recognition module. The three-dimensional model is obtained through gray-scale interval overlay processing. The non-matching element image method and 4 K screen are used for three-dimensional integral imaging display with a large viewing range and high resolution. A pair of binocular cameras is used to capture and recognize gestures, and the maximum interaction frame rate is 22 FPS. Optical experiments were performed to validate the proposed medical interactive three-dimensional display system with existing patient skeletal data. The results showed that the proposed system can effectively solve the problem that surgeons cannot take off their gloves to operate in surgery and provide an intuitive and comfortable three-dimensional display. Our proposed system is expected to be widely used in scenarios such as preoperative simulation, surgical instruction, and medical teaching.

A Deep Learning Approach for ASL Recognition and Text-to-Speech Synthesis using CNN

Abstract: Sign language is a visual language that is used by the deaf and hard-of-hearing community to communicate. However, sign language is not universally understood by non-signers, which can create communication barriers for the deaf and hard-ofhearing individuals. In this paper, we present a novel application for American Sign Language (ASL) to text to speech conversion using deep learning techniques. Our app aims to bridge the communication gap between hearing-impaired individuals who use ASL as their primary mode of communication and individuals who do not understand ASL. The app comprises three main components: a hand gesture recognition module that detects and recognizes ASL signs, a text-tospeech synthesis module that converts recognized ASL signs to text, and a speech synthesis module that converts the text to speech

Improving dynamic gesture recognition in untrimmed videos by an online lightweight framework and a new gesture dataset ZJUGesture

Human–computer interaction technology brings great convenience to people, and dynamic gesture recognition makes it possible for a man to interact naturally with a machine. However, recognizing gestures quickly and precisely in untrimmed videos remains a challenge in real-world systems since: (1) It is challenging to locate the temporal boundaries of performing gestures; (2) There are significant differences in performing gestures among different people, resulting in a variety of gestures; (3) There must be a trade-off between the accuracy and the computational consumption. In this work, we propose an online lightweight two-stage framework, including a detection module and a gesture recognition module, to precisely detect and classify dynamic gestures in untrimmed videos. Specifically, we first design a low-power detection module to locate gestures in time series, then a temporal relational reasoning module is employed for gesture recognition. Moreover, we present a new dynamic gesture dataset named ZJUGesture, which contains nine classes of common gestures in various scenarios. Extensive experiments on the proposed ZJUGesture and 20-bn-Jester dataset demonstrate the attractive performance of our method with high accuracy and a low computational cost.

An interactive game for rehabilitation based on real-time hand gesture recognition.

Currently, cardiovascular and cerebrovascular diseases have become serious global health problems related to their high incidence and fatality rate. Some patients with cardiovascular cerebro-cardiovascular diseases even may face motor or cognitive dysfunction after surgery. In recent years, human-computer interactive systems with artificial intelligence have become an important part of human well-being because they enable novel forms of rehabilitation therapies. We propose an interactive game utilizing real-time skeleton-based hand gesture recognition, which aims to assist rehabilitation exercises by improving the hand-eye coordination of the patients during a game-like experience. For this purpose, we propose a lightweight residual graph convolutional architecture for hand gesture recognition. Furthermore, we designed the whole system using the proposed gesture recognition module and some third-party modules. Finally, some participants were invited to test our system and most of them showed an improvement in their passing rate of the game during the test process.

Person-Following Controller with Socially Acceptable Robot Motion

This work presents a novel stable controller for the person-following task that includes social considerations for a differential drive mobile robot equipped with an RGB-D camera and a laser range finder as main sensors. The proposed controller adapts its behavior based on the knowledge of both: a modified personal space distribution and human user velocity. Control objectives are focused hence on keeping the human user within the camera’s field-of-view while the mobile robot follows it, with a socially acceptable motion through arbitrary paths. To show the good behavior of this proposal, simulation and real experimental results are included and discussed. The asymptotic stability of the overall system is proved through the Lyapunov theory. Also, in our proposal, three state-of-the-art algorithms were integrated with the controller. In particular, a new real-time multi-person skeletal tracking system is used to obtain the relative human–robot position, a text to speech algorithm is used to confirm the commands given by the human, and also, a SLAM algorithm is used to obtain the map of the environment while the main task is being performed. Additionally, a hand gesture recognition module is included to interact with the mobile robot. This way, the robot is allowed to navigate with a socially-aware behavior in environments shared with humans. Finally, subjective and objective metrics are used as a validation method for human perception about the achieved robot motion.

3D Gesture Recognition and Adaptation for Human–Robot Interaction

Gesture-based human-robot interaction has been an important area of research in recent years. The primary aspect for the researchers has always been to create a gesture detection system that is insensitive to lighting and backdrop surroundings. This research proposes a 3D gesture recognition and adaption system based on Kinect for human-robot interaction. The framework consists of four modules, i.e., pointing gesture recognition, 3D dynamic gesture recognition, gesture adaptation, and robot navigation. The proposed dynamic gesture recognition module employs three distinct classifiers: HMM, Multi-class SVM, and CNN. The adaptation module can adapt to new and unrecognized gestures applying semi-supervised self-adaptation or user consent-based adaptation. A Graphical User Interface (GUI) is built for training and testing the proposed system on the fly. A simple simulator along with two different robot-navigation algorithms are developed to test robot navigation based on the recognized gestures. The framework is trained and tested through a five-fold cross-validation method with a total of 3,600 gesture instances of ten predefined gestures performed by 24 persons (Three age categories: Young, Middle Aged, Adult; each with 1,200 gestures). The proposed system achieves a maximum accuracy score of 95.67% with HMM for Middle Aged category, 92.59% with SVM for Middle Aged category, and 89.58% with CNN for Young category in dynamic gesture recognition. Considering all the three age categories, the system achieves an average accuracy of 94.61%, 91.95%, and 88.97% in recognizing dynamic gestures with HMM, SVM, and CNN respectively. Moreover, the system recognizes pointing gestures in real-time.

Hidden Markov Model-Based Video Recognition for Sports

In this paper, we conduct an in-depth study and analysis of sports video recognition by improved hidden Markov model. The feature module is a complex gesture recognition module based on hidden Markov model gesture features, which applies the hidden Markov model features to gesture recognition and performs the recognition of complex gestures made by combining simple gestures based on simple gesture recognition. The combination of the two modules forms the overall technology of this paper, which can be applied to many scenarios, including some special scenarios with high-security levels that require real-time feedback and some public indoor scenarios, which can achieve different prevention and services for different age groups. With the increase of the depth of the feature extraction network, the experimental effect is enhanced; however, the two-dimensional convolutional neural network loses temporal information when extracting features, so the three-dimensional convolutional network is used in this paper to extract features from the video in time and space. Multiple binary classifications of the extracted features are performed to achieve the goal of multilabel classification. A multistream residual neural network is used to extract features from video data of three modalities, and the extracted feature vectors are fed into the attention mechanism network, then, the more critical information for video recognition is selected from a large amount of spatiotemporal information, further learning the temporal dependencies existing between consecutive video frames, and finally fusing the multistream network outputs to obtain the final prediction category. By training and optimizing the model in an end-to-end manner, recognition accuracies of 92.7% and 64.4% are achieved on the dataset, respectively.

Egocentric Gesture Recognition Using 3D Convolutional Neural Networks for the Spatiotemporal Adaptation of Collaborative Robots.

Collaborative robots are currently deployed in professional environments, in collaboration with professional human operators, helping to strike the right balance between mechanization and manual intervention in manufacturing processes required by Industry 4.0. In this paper, the contribution of gesture recognition and pose estimation to the smooth introduction of cobots into an industrial assembly line is described, with a view to performing actions in parallel with the human operators and enabling interaction between them. The proposed active vision system uses two RGB-D cameras that record different points of view of gestures and poses of the operator, to build an external perception layer for the robot that facilitates spatiotemporal adaptation, in accordance with the human's behavior. The use-case of this work is concerned with LCD TV assembly of an appliance manufacturer, comprising of two parts. The first part of the above-mentioned operation is assigned to a robot, strengthening the assembly line. The second part is assigned to a human operator. Gesture recognition, pose estimation, physical interaction, and sonic notification, create a multimodal human-robot interaction system. Five experiments are performed, to test if gesture recognition and pose estimation can reduce the cycle time and range of motion of the operator, respectively. Physical interaction is achieved using the force sensor of the cobot. Pose estimation through a skeleton-tracking algorithm provides the cobot with human pose information and makes it spatially adjustable. Sonic notification is added for the case of unexpected incidents. A real-time gesture recognition module is implemented through a Deep Learning architecture consisting of Convolutional layers, trained in an egocentric view and reducing the cycle time of the routine by almost 20%. This constitutes an added value in this work, as it affords the potential of recognizing gestures independently of the anthropometric characteristics and the background. Common metrics derived from the literature are used for the evaluation of the proposed system. The percentage of spatial adaptation of the cobot is proposed as a new KPI for a collaborative system and the opinion of the human operator is measured through a questionnaire that concerns the various affective states of the operator during the collaboration.

Using Deep Learning Technology to Realize the Automatic Control Program of Robot Arm Based on Hand Gesture Recognition

In this study, the robot arm control, computer vision, and deep learning technologies are combined to realize an automatic control program. There are three functional modules in this program, i.e., the hand gesture recognition module, the robot arm control module, and the communication module. The hand gesture recognition module records the user’s hand gesture images to recognize the gestures’ features using the YOLOv4 algorithm. The recognition results are transmitted to the robot arm control module by the communication module. Finally, the received hand gesture commands are analyzed and executed by the robot arm control module. With the proposed program, engineers can interact with the robot arm through hand gestures, teach the robot arm to record the trajectory by simple hand movements, and call different scripts to satisfy robot motion requirements in the actual production environment.

Cross-Individual Gesture Recognition Based on Long Short-Term Memory Networks

Gestures recognition based on surface electromyography (sEMG) has been widely used for human-computer interaction. However, there are few research studies on overcoming the influence of physiological factors among different individuals. In this paper, a cross-individual gesture recognition method based on long short-term memory (LSTM) networks is proposed, named cross-individual LSTM (CI-LSTM). CI-LSTM has a dual-network structure, including a gesture recognition module and an individual recognition module. By designing the loss function, the individual information recognition module assists the gesture recognition module to train, which tends to orthogonalize the gesture features and individual features to minimize the impact of individual information differences on gesture recognition. Through cross-individual gesture recognition experiments, it is verified that compared with other selected algorithm models, the recognition accuracy obtained by using the CI-LSTM model can be improved by an average of 9.15%. Compared with other models, CI-LSTM can overcome the influence of individual characteristics and complete the task of cross-individual hand gestures recognition. Based on the proposed model, online control of the prosthetic hand is realized.

SMART GLOVE Sign to Speech Conversion and Home Automation Control for Mute Community

Communication plays a vital role for human beings. This project helps in improving the communication with the dumb people using flex device technology. a tool is developed that may translate totally different signs as well as Indian language to text also as voice format. The people who are communicating with dumb people might not perceive their signs and expressions. Through language, communication is feasible for a deaf-mute person without the means of acoustic sounds. The aim behind this work is to develop a system for recognizing the language, that provides communication between people with speech impairment and normal people, thereby reducing the communication gap between them. Compared to different gestures (arm, face, head and body), hand gesture plays a vital role, because it expresses the user's views in less time. within the current work flex sensor-based gesture recognition module is developed to acknowledge English alphabets and few words and a Text-to-Speech. Hence, an approach has been created and changed to listen to the gesture-based communication. it will be very useful to them for conveying their thoughts to others.

Design of Hand Movement based Smart Automation System

There are millions deaf people in the world and around many people in India who are affected by ALS and paralysis, they face difficulty in communicating with others. Therefore in this paper we propose a system that converts simple gesture to speech and also use these gesture to control the home appliances he project will be in the form of glove which makes it handy and reliable. It uses flex sensors for gesture recognition and TTS module for the gesture to speech feature which makes it slender and much viable.

A Signer Independent Sign Language Recognition with Co-articulation Elimination from Live Videos: An Indian Scenario

Due to the high population of hearing impaired and vocal disabled people in India, a sign language interpretation system is becoming highly important for minimizing their isolation in society. This paper proposes a signer independent novel vision-based gesture recognition system which is capable of recognizing single handed static and dynamic gestures, double-handed static gestures and finger spelling words of Indian Sign Language (ISL) from live video. The use of Zernike moments for key frame extraction reduces the computation speed to a large extent. It also proposes an improved method for co-articulation elimination in fingerspelling alphabets. The gesture recognition module comprises mainly three steps – Preprocessing, Feature Extraction, and Classification. In the preprocessing phase, the signs are extracted from a real-time video using skin color segmentation. An appropriate feature vector is extracted from the gesture sequence after co-articulation elimination phase. The obtained features are then used for classification using Support Vector Machine(SVM). The system successfully recognized finger spelling alphabets with 91% accuracy and single-handed dynamic words with 89% accuracy. The experimental results show that the system has a better recognition rate compared to some of the existing methods.

Watch & Do: A Smart IoT Interaction System with Object Detection and Gaze Estimation

The Internet of Things (IoT) attempts to help people access Internet-connected devices, applications, and services anytime and anywhere. However, how providing an efficient and intuitive method of interaction between people and IoT devices is still an open challenge. In this paper, we propose a novel interaction system called Watch & Do , where users can control an IoT device by gazing at it and doing simple gestures. The proposed system mainly consists of: 1) object detection module; 2) gaze estimation module; 3) hand gesture recognition module; and 4) IoT controller module. The target device is identified by various deep learning-based gaze estimation and object detection techniques. Afterwards, hand gesture recognition is applied to generate an IoT device control command which is transmitted to the IoT platform. The experimental results and case studies demonstrate the feasibility of the proposed system and imply the future research directions.

A Portable Driver Assistance System Headset Using Augmented Reality

Vehicle collision leading to life threatening accidents is a common problem which is incrementing noticeably. This necessitated the need for Driver Assistance Systems (DAS) which helps drivers sense nearby obstacles and drive safely. However, it’s inefficiency in unfavorable weather conditions, overcrowded roads, and low signal penetration rates in India posed many challenges during it’s implementation. In this paper, we present a portable Driver Assistance System that uses augmented reality for it’s working. The headset model comprises of five systems working in conjugation in order to assist the driver. The pedestrian detection module, along with the driver alert system serves to assist the driver in focusing his attention to obstacles in his line of sight. Whereas, the speech recognition, gesture recognition and GPS navigation modules together prevent the driver from getting distracted while driving. In the process of serving these two root causes of accidents, a cost effective, portable and holistic driver assistance system has been developed.

ZatLab Gesture Recognition Framework

The main problem this work addresses is the real-time recognition of gestures, particularly in the complex domain of artistic performance. By recognizing the performer gestures, one is able to map them to diverse controls, from lightning control to the creation of visuals, sound control or even music creation, thus allowing performers real-time manipulation of creative events. The work presented here takes this challenge, using a multidisciplinary approach to the problem, based in some of the known principles of how humans recognize gesture, together with the computer science methods to successfully complete the task. This paper is a consequence of previous publications and presents in detail the Gesture Recognition Module of the ZatLab Framework and results obtained by its Machine Learning (ML) algorithms. One will provide a brief review the previous works done in the area, followed by the description of the framework design and the results of the recognition algorithms.

NUI framework based on real-time head pose estimation and hand gesture recognition

The natural user interface (NUI) is used for the natural motion interface without using device or tool such as mice, keyboards, pens and markers. In this paper, we develop natural user interface framework based on two recognition module. First module is real-time head pose estimation module using random forests and second module is hand gesture recognition module, named Hand gesture Key Emulation Toolkit (HandGKET). Using the head pose estimation module, we can know where the user is looking and what the user’s focus of attention is. Moreover, using the hand gesture recognition module, we can also control the computer using the user’s hand gesture without mouse and keyboard. In proposed framework, the user’s head direction and hand gesture are mapped into mouse and keyboard event, respectively.