Interpret Hand Gestures Research Articles

Development of a Sign Language recognizer using LSTM

Sign language recognition plays a crucial role in bridging communication gaps between the deaf community and the hearing population. This paper presents the development of a sign language recognizer using Long Short-Term Memory (LSTM) networks, a type of recurrent neural network well-suited for sequence prediction tasks. The proposed model aims to accurately interpret hand gestures into text, enhancing accessibility and communication for individuals with hearing impairments. Our approach utilizes a custom-built dataset of sign language gestures, which undergoes preprocessing to extract relevant features such as hand position, orientation, and movement. The LSTM-based architecture is designed to capture the temporal dynamics of these gestures, enabling the recognition of complex sign language patterns. We train the model using a supervised learning approach. Our method demonstrates superior accuracy in recognizing sign language gestures compared to traditional machine learning methods and other deep learning architectures. This advancement contributes significantly to the field of assistive technologies. Future work will explore the integration of additional modalities, such as facial expressions and body movements, to further enhance the system's performance and usability.

Recognizing Sign Language using Machine Learning and Deep Learning Models

Individuals with hearing impairments communicate mostly through sign language. Our goal was to create an American Sign Language recognition dataset and utilize it in a neural network-based machine learning model that can interpret hand gestures and positions into natural language. In our study, we incorporated the SVM, CNN and Resnet-18 models to enhance predictability when interpreting ASL signs through this new dataset, which includes provisions such as lighting and distance limitations. Our research also features comparison results between all the other models implemented under invariant conditions versus those using our proposed CNN model. As demonstrated by its high levels of precision at 95.10% despite changes encountered during testing procedures like varying data sets or scene configurations where losses are minimal (0.545), there exists great potential for future applications in image recognition systems requiring deep learning techniques. Furthermore, these advancements may lead to significant improvements within various fields related explicitly to speech-language therapy sessions designed specifically around helping people overcome challenges associated with deafness while building bridges towards improved social integration opportunities.

GestureFusion: A Gesture-based Interaction System

Abstract: In an era of increasingly sophisticated human-computer interaction, the ability to comprehend and interpret hand gestures has become a pivotal element for bridging the gap between humans and machines. This project, titled ”GestureFusion: A Gesture-based Interaction System,” seeks to contribute to this rapidly evolving field by developing a comprehensive system for recognizing and categorizing hand gestures through the application of computer vision techniques. The primary objective of this project is to create a versatile and extensible hand gesture recognition system that can be employed in various domains such as virtual reality, robotics, sign language interpretation, and human-computer interface design. To achieve this, the project employs state-of-the-art computer vision algorithms and deep learning techniques. Key components of this project include data collection and annotation, model training and optimization, the development of a user-friendly application programming interface (API) for integration into diverse applications.

Applying LSTM and GRU Methods to Recognize and Interpret Hand Gestures, Poses, and Face-Based Sign Language in Real Time

This research uses a real-time, human-computer interaction application to examine sign language recognition. This work develops a rule-based hand gesture approach for Indonesian sign language in order to interpret some words using a combination of hand movements, mimics, and poses. The main objective in this study is the recognition of sign language that is based on hand movements made in front of the body with one or two hands, movements which may involve switching between the left and right hand or may be combined with mimics and poses. To overcome this problem, a research framework is developed by coordinating hand gestures with poses and mimics to create features by using holistic MediaPipe. To train and test data in real time, the long short time memory (LSTM) and gated recurrent unit (GRU) approaches are used. The research findings presented in this paper show that hand gestures in real-time interactions are reliably recognized, and some words are interpreted with the high accuracy rates of 94% and 96% for the LSTM and GRU methods, respectively.

Virtual Mouse System using Hand Gesture recognition with Open CV

The use of hand gesture recognition for controlling virtual devices has gained popularity with the advancement of artificial intelligence technology. In this paper, we propose a hand gesture-controlled virtual mouse system that employs AI algorithms to identify and interpret hand gestures, translating them into mouse movements. The primary objective of this system is to provide an alternative interface, particularly beneficial for individuals facing challenges with traditional mouse or keyboard usage.Our proposed system utilizes a camera to capture images of the user's hand. To enable gesture recognition, the system undergoes training using a dataset comprising various hand gestures. Once a gesture is identified, it is translated into a corresponding mouse movement, subsequently executed on the virtual screen. An essential feature of our system is its scalability and adaptability, allowing seamless integration into diverse environments and devices. To enhance user interaction, our system incorporates dynamic/static hand gestures along with a voice assistant for virtually controlling all input operations. Notably, our approach employs Machine Learning (ML) and Computer Vision algorithms to recognize hand gestures and voice commands, eliminating the need for additional hardware requirements. The model is implemented using the MediaPipe framework, demonstrating the feasibility of our proposed system. In conclusion, our hand gesture-controlled virtual mouse system presents a promising avenue for improving user experience and accessibility through human-computer interaction. By leveraging AI, ML, and Computer Vision, we aim to create an intuitive and versatile interface that can be easily adapted to different environments, ultimately enhancing accessibility for a wider range of users. Keywords: Computer vision, hand gesture recognition, Media-pipe, virtual mouse.

Gesture-Based Control of Multimedia Player Using Python and OpenCV

The increasing significance of computers in our daily lives, coupled with the rise of ubiquitous computing, has necessitated effective human-computer interaction. Hand gesture recognition systems have emerged as a real-time video-based solution for detecting and interpreting hand gestures, offering intelligent and natural human-computer interaction (HCI) methods. This project focuses on leveraging human hands as input devices for computer operation. Developed using Python and the OpenCV library, the program utilizes a computer webcam to capture and analyze hand shapes and patterns. The program provides real-time feedback by displaying recognized hand gestures on the live video stream. The ultimate outcome of this project is an application that enhances user experiences in contactless systems. The project recognizes and detects human hand motions using the Python computer language through a process flow that includes background subtraction, hand ROI segmentation, contour detection, and finger recognition. Techniques for processing images are used, including hand gesture detection, pattern recognition, thresholding, and contour detection. The processing of incoming photos and the creation of related keystrokes are made possible by OpenCV, a rich set of image processing tools

Interactive Hand Gesture Control System for Augmented Reality-based Games

This research paper presents the development of an innovative 2D augmented reality (AR) car game utilizing Python, OpenCV, and Pygame. The primary objective of the game is to provide an interactive platform for individuals with motor skill challenges, such as Parkinson’s disease, to enhance their hand and gesture control abilities. The game involves controlling a virtual car superimposed onto the real-world environment using an AR marker. Hand gestures captured by a webcam control the car's movements, creating a novel and accessible gaming experience. The implementation incorporates computer vision techniques from the OpenCV library to detect an AR marker, interpret hand gestures, and map them to specific actions within the game. The Pygame library facilitates the creation of an engaging gaming environment, where users can employ intuitive hand movements to navigate the virtual car. The system's performance is evaluated in varying lighting conditions, and the results demonstrate a promising accuracy rate in gesture detection. The limitations and potential improvements are discussed, emphasizing the system's application in assisting individuals with motor impairments and its broader implications for human-computer interaction. This research provides insights into the intersection of augmented reality, computer vision, and assistive technology, offering a foundation for further advancements in accessible gaming experiences. Keywords: Augmented Reality, Hand Gesture Control, Computer Vision, Game Interaction, Assistive Technology

Hand Gesture Control Robot

In recent years, there has been a growing interest in developing human-machine interfaces that allow users to interact with robots in a more intuitive and natural way. Hand gesture control has emerged as a promising modality for such interfaces, offering a hands-free and immersive interaction experience. This paper presents the design, development, and evaluation of a hand gesture-controlled robot system. The proposed system utilizes computer vision techniques and machine learning algorithms to recognize and interpret hand gestures, enabling users to control the robot's movements and actions seamlessly. The robot is equipped with sensors and actuators, allowing it to respond to the detected gestures in real-time. The goal is to create a user-friendly and efficient interface that enhances the ease of interaction between humans and robots. The findings pave the way for the development of more sophisticated and user-friendly robotic interfaces, fostering the integration of robots into daily human activities. The proposed system's adaptability and versatility make it a promising candidate for further exploration and deployment in real-world scenarios, marking a significant step towards the advancement of gesture-based control in robotics

Real-time gesture-based control of UAVs using multimodal fusion of FMCW radar and vision

Gesture-based control has gained prominence as an intuitive and natural means of interaction with unmanned aerial vehicles (UAVs). This paper presents a real-time gesture-based control system for UAVs that leverages the multimodal fusion of Frequency Modulated Continuous Wave (FMCW) radar and vision sensors, aiming to enhance user experience through precise and responsive UAV control via hand gestures. The research focuses on developing an effective fusion framework that combines the complementary advantages of FMCW radar and vision sensors. FMCW radar provides robust range and velocity measurements, while vision sensors capture fine-grained visual information. By integrating data from these modalities, the system achieves a comprehensive understanding of hand gestures, resulting in improved gesture recognition accuracy and robustness. The proposed system comprises three main stages: data acquisition, gesture recognition, and multimodal fusion. In the data acquisition stage, synchronized data streams from FMCW radar and vision sensors are captured. Then, machine learning algorithms are employed in the gesture recognition stage to classify and interpret hand gestures. Finally, the multimodal fusion stage aligns and fuses the data, creating a unified representation that captures the spatial and temporal aspects of hand gestures, enabling real-time control commands for the UAV. Experimental results demonstrate the system‘s effectiveness in accurately recognizing and responding to hand gestures. The multimodal fusion of FMCW radar and vision sensors enables a robust and versatile gesture-based control interface.

Design of Gesture Controlled Arm – An Overview

The Design of Gesture Controlled Arm project aims to explore the potential of gesture control technology in robotics by designing a functional and reliable robotic arm that can be controlled using hand gestures. The system will use machine learning algorithms and computer vision techniques to recognize and interpret hand gestures, allowing users to control the arm's movement and perform a variety of tasks, such as picking and placing objects. The project will involve designing and fabricating the robotic arm, developing the gesture recognition system, and integrating the two systems to create a cohesive and efficient solution. The project's outcome will demonstrate the practical application of gesture control technology in robotics, potentially opening new avenues for automation in various industries. The project's success will also provide valuable insights into the design and development of advanced robotic systems, paving the way for future advancements in the field.

Camera-based interactive wall display using hand gesture recognition

the recognition of Hand gestures has become a critical point as it is widely used in everyday applications. the challenge in this is to improve the recognition effect and develop a fast recognition method. Glove and led-based methods involve external devices in detecting and interpreting hand gestures, making human-computer interaction less natural. So, different approaches have been used previously that use purely hand gestures in many systems based on human-computer interaction. This system provides a more natural human-computer interaction; it must be made efficient processing speed of classifying the test data (images) from among the training data (database stored for gestures recognition). This speed makes gesture recognition more effective and reliable to use as compared to previously proposed methods. In this research paper, a proposed system based on a camera-based interactive wall display using bare hand gestures with efficient processing speed for controlling the speed of the mouse and other functions. This system has three modules: one uses Genetic Algorithm and Otsu thresholding to identify the query images as the right or wrong gesture and perform the correct action in case of the proper motion, another module controls functions outside of PowerPoint files or Word documents, e.g., to open folders and go through drives, and the third module uses the convexity hull method for finding the number of fingers open in the user's gesture and operates accordingly.

Hand Gesture Controlled Presentation using OpenCV and MediaPipe

In today's digital era, presentations play a crucial role in various domains, ranging from education to business. However, traditional manual presentation methods, reliant on input devices such as keyboards or clickers, have inherent limitations in terms of mobility, interactivity, and user experience. To address these limitations, gesture-controlled presentations have emerged as a promising solution, harnessing the power of computer vision techniques to interpret hand gestures and enable natural interaction with presentation content. This paper presents a comprehensive system for hand gesture-controlled presentations using OpenCV and MediaPipe libraries. OpenCV is employed to capture video input from a webcam, while MediaPipe is utilized for hand tracking and landmark extraction. By analyzing finger positions and movements, the system accurately recognizes predefined gestures. Presenters can seamlessly control the slides, hold a pointer, annotate the content, and engage with the audience in a more interactive manner. The responsiveness and real-time performance contribute to an enhanced presentation experience.

A Smart System using Hand Gestures and Voice

Human-Computer Interaction (HCI) grew enormously over the years. Hand gesture recognition technology brought a brand-new era to the artificial intelligence branch of human-computer interaction. Hand Gesture-based communication is one of the most effortless and natural methods. Identifying and interpreting hand gestures from a continuous sequential stream of input data is called gesture recognition. Amateur or aged people find it hard to identify and press the exact alphabet that they need. Overcoming this difficulty stands as the ultimate goal for our proposed system. Typing has taken many forms, first using a keyboard, gradually changing to touch screens, and now too much easier finger motion tracking systems. The gesture is made by the user is detected by the machine via the image processing techniques and the operation unique to the machine is carried out, thereby minimizing the requirement of any hardware input device. This paper proposes a virtually controlled system that uses hand gestures and voice to perform operations. The system will allow the user to manage cursor, keyboard, and writing functions using hand gestures, and also provides the user with an additional feature to convert the user’s voice to text. The proposed system utilizes a webcam as an input device. This application will add a brand-new era of HCI implementation for educational purposes.

A systematic review on hand gesture recognition techniques, challenges and applications.

BackgroundWith the development of today’s technology, and as humans tend to naturally use hand gestures in their communication process to clarify their intentions, hand gesture recognition is considered to be an important part of Human Computer Interaction (HCI), which gives computers the ability of capturing and interpreting hand gestures, and executing commands afterwards. The aim of this study is to perform a systematic literature review for identifying the most prominent techniques, applications and challenges in hand gesture recognition.MethodologyTo conduct this systematic review, we have screened 560 papers retrieved from IEEE Explore published from the year 2016 to 2018, in the searching process keywords such as “hand gesture recognition” and “hand gesture techniques” have been used. However, to focus the scope of the study 465 papers have been excluded. Only the most relevant hand gesture recognition works to the research questions, and the well-organized papers have been studied.ResultsThe results of this paper can be summarized as the following; the surface electromyography (sEMG) sensors with wearable hand gesture devices were the most acquisition tool used in the work studied, also Artificial Neural Network (ANN) was the most applied classifier, the most popular application was using hand gestures for sign language, the dominant environmental surrounding factor that affected the accuracy was the background color, and finally the problem of overfitting in the datasets was highly experienced.ConclusionsThe paper will discuss the gesture acquisition methods, the feature extraction process, the classification of hand gestures, the applications that were recently proposed, the challenges that face researchers in the hand gesture recognition process, and the future of hand gesture recognition. We shall also introduce the most recent research from the year 2016 to the year 2018 in the field of hand gesture recognition for the first time.

Dual-Core Capacitive Microfiber Sensor for Smart Textile Applications.

Wearable sensors for smart textile applications have garnered tremendous interest in recent years and can have enormous potential for human machine interfaces and digital health monitoring. Here, we report a soft capacitive microfiber sensor that can be woven seamlessly into textiles for strain measurement. Comprising a dual-lumen elastomeric microtube and liquid metallic alloy, the microfiber sensor enables continual strain perception even after being completely severed. In addition, our microfiber sensor is highly stretchable and flexible and exhibits tunable sensitivity, excellent linearity, a fast response, and negligible hysteresis. More importantly, the microfiber sensor is minimally affected by train rate and compression during strain sensing. Even under drastic environmental changes, the microfiber sensor presents good electrical stability. By integrating the microfiber sensor imperceptibly with textiles, we devise smart textile wearables to interpret hand gestures, detect limb motion, and monitor respiration rate. We believe that this sensor presents enormous potential in unobtrusive continuous health monitoring.

Real-time glove and android application for visual and audible Arabic sign language translation

Researchers can develop new systems to capture, analyze, recognize, memorize and interpret hand gestures with machine learning and sensors. Acoustic communication is a way to convey human opinions, feelings, messages, and information. Deaf and mute individuals communicate using sign language that is not understandable by everyone. Unfortunately, they face extreme difficulty in conveying their messages to others. To facilitate the communication between deaf/mute individuals and normal people, we propose a real-time prototype using a customized glove equipped with five flex and one-accelerometer sensors. These sensors are able to detect the bindings of the fingers and the movements of the hand. In addition, we developed an android mobile application to recognize the captured Arabic Sign Language (ArSL) gestures and translate them into displayed texts and audible sounds. The developed prototype is accurate, low cost and fast in response.

Recent Trends and Technologies in Hand Gesture Recognition

Hand gesture recognition is a growing field of research. Gestures are an important aspect of human interaction, both interpersonal and in the context of human computer interface. The interaction between computers and humans are provided by the hand gestures. Gestures provide a way for computers to understand human body language. Hand gesture recognition deals with the goal of interpreting hand gestures via mathematical algorithms. The techniques discussed are based on detection of some shape based features like centroid, orientation of hand, presence of finger, thumb in image and contour based features like convex hull and convexity defects. Hand gesture recognition system provides a user friendly way of interaction with the computer which is more familiar to the human beings. Hand gesture recognition process includes capturing input image which will be processed and segmented so as to extract features to identify different gestures. Different methods of hand gesture recognition techniques are evaluated and proposed a novel method for recognition. This method explains how the users hand is detected and the corresponding gesture is identified in an accurate manner. This paper also explains the various applications that can take advantage of this type of interaction mechanism.

Rapid and efficient hand gestures recognizer based on classes discriminator wavelet networks

The vision based on hand gesture recognition is one of the key challenges in behavior understanding and computer vision. It offers to machines the possibility of identifying, recognizing and interpreting hand gestures in the aim of controlling certain devices, to monitor certain human activities or interacting with certain human machine interfaces (HMI). This paper aims at proposing a rapid and efficient hand gestures recognizer based on classes discriminator wavelet networks. In this work two main contributions were proposed: firstly, by enhancing previous works using wavelet networks (WN) in the classification field, specifically at the learning process of the latest version of WN classifier (WNC) by creating separator WNs discriminating classes (n − 1 WNs to classify n classes) as alternative of constructing a WN corresponding to each training image. This contribution, by minimizing the comparison number between test images WNs and training ones, makes quicker the test phase. Secondly, a new WN architecture based on the cascade notion was proposed, in which the WN is decomposed of a set of stages. The novel architecture aims not only at making recognitions robust and quick but also at rejecting from early stages, as fast as possible, gestures that must not be taken into consideration by the system (spontaneous gestures). To test this work, both proprietary and public hand posture datasets were used. Comparisons with other works are detailed and discussed. Given results showed that the novel hand gesture recognizer performances are comparable to already established ones.

Kinect-based Taiwanese sign-language recognition system

Gesture-recognition is an important component for many intelligent human---computer interaction applications. For example, a realtime sign-language recognition system would detect and interpret hand gestures. Many vision-based sign-language recognition methods have been proposed over the years with mix results of usability. Some system are limited to recognize only a few gestures, while others require the use of 3D camera to provides depth information to improve recognition accuracy. In this paper, a Kinect-based Taiwanese sign-language recognition system is proposed. Three main features are extracted from the signing gestures, namely hand positions, hand signing direction, and hand shapes. The hand positions are readily available through the input sensor. The signing direction is determined using HMM on trajectory of the hand movement, and a SVM is trained and used to recognize the hand shapes. Experimental results show that the proposed system achieved an 85.14 % recognition rate.

3D Imaging for hand gesture recognition: Exploring the software-hardware interaction of current technologies

Interaction with 3D information is one of the fundamental and most familiar tasks in virtually all areas of engineering and science. Several recent technological advances pave the way for developing hand gesture recognition capabilities available to all, which will lead to more intuitive and efficient 3D user interfaces (3DUI). These developments can unlock new levels of expression and productivity in all activities concerned with the creation and manipulation of virtual 3D shapes and, specifically, in engineering design. Building fully automated systems for tracking and interpreting hand gestures requires robust and efficient 3D imaging techniques as well as potent shape classifiers. We survey and explore current and emerging 3D imaging technologies, and focus, in particular, on those that can be used to build interfaces between the users' hands and the machine. The purpose of this paper is to categorize and highlight the relevant differences between these existing 3D imaging approaches in terms of the nature of the information provided, output data format, as well as the specific conditions under which these approaches yield reliable data. Furthermore we explore the impact of each of these approaches on the computational cost and reliability of the required image processing algorithms. Finally we highlight the main challenges and opportunities in developing natural user interfaces based on hand gestures, and conclude with some promising directions for future research.