Real-time Gesture Recognition Research Articles

A Novel Control System for a Laptop with Gestures Recognition

An important lesson learned during the pandemic is that the devices utilized by many people (vending machines, elevators, etc.) can spread the virus and take advantage of contactless programs that can be enabled through gesture recognition. It's a major enabler of smart living and human-computer interaction. It is widely used in controlling equipments. This research presents a novel vision-based real-time gesture recognition system for controlling a laptop. It's a comprehensive system that can do all the required functions needed to remotely control a laptop through hand gesture recognition rather than employing a mouse or keyboard. This system allows changing the position of the mouse point, performing left-click, right-click, double-click, and scrolling up and down. It also provides all the keyboard shortcuts, Print Screen (PrtScr), and Backspace. Furthermore, it allows shutting down as well as creating files and folders. Additionally, it allows playing, muting, pausing media playback, reducing/increasing the brightness, and turning the volume down/up. Moreover, it allows advancing to the next and previous page of the slideshow, as well as zooming in/out. Finally, the system allows gestures to open the most frequently used applications based on the user's choice. In comparison to the previous work, this system is considered unique in that it performs all the functions needed to remotely control the laptop. Previous systems only offer a subset of the tools provided by this proposed system Additionally, it is unique because the user operates all functions solely through gestures, without the need for additional methods.

Ultra-stretchable triboelectric touch pad with sandpaper micro-surfaces for Transformer-assisted gesture recognition

Touch pad based on triboelectric nanogenerator are attracting attention in the field of wearable electronics and human-machine interaction. However, it is still a challenge to realize touch pads combining large-area metamorphic stretchability, high power triboelectric sensing, intelligent free-sliding recognition and availability in extreme environments. Here, we fabricate an ultra-stretchable triboelectric touch pad by integrating liquid metal and high performance hydrogel-based triboelectric sensor arrays, which achieves Transformer-assisted gesture recognition for drone flight direction control. The internal hydrogel electrode has great stretchability, electrical conductivity and anti-freeze properties. Encapsulated in Ecoflex with sandpaper micro-surfaces, it endows high triboelectric power density and pressure sensitivity. With screen-printed liquid metal, the integrated touch pad is fully ultra-stretchable and large-area strain stability. Combining Internet of Things, we build a Transformer-assisted human machine interface. Notably, slide gesture signal peak deviation is reduced by integral processing. The random free-sliding gesture signals are parsed by the Transformer algorithm with 96.83 % accuracy. Multi-threaded signal processing improves the response speed and throughput of the system. The accurate real-time gesture recognition for drone flight direction control in extreme environment is realized. This human machine interface can facilitate development of next generation intelligent remote interactive platforms.

Handwave Harmony: Transforminggestures into Words

Our project introduces a real-time gesture recognition system using ResNet50 and OpenCV. This innovative technology offers precise interpretation of sign movements for individuals with hearing difficulties. This represents a form of interpersonal connection where individuals convey understanding, empathy, or agreement through the subtle movements of their hands. The system is trained on a diverse dataset for flexibility and utilizes ResNet50's capabilities while integrating OpenCV for reduced latency, ensuring effective communication. Rigorous testing validates its accuracy, with user input fosters iterative learning. The study highlights the Importance of continuous enhancement and collaboration with the deaf and hard-of-hearing community for greater inclusiveness.

Static and Dynamic Hand Gestures: A Review of Techniques of Virtual Reality Manipulation.

This review explores the historical and current significance of gestures as a universal form of communication with a focus on hand gestures in virtual reality applications. It highlights the evolution of gesture detection systems from the 1990s, which used computer algorithms to find patterns in static images, to the present day where advances in sensor technology, artificial intelligence, and computing power have enabled real-time gesture recognition. The paper emphasizes the role of hand gestures in virtual reality (VR), a field that creates immersive digital experiences through the Ma blending of 3D modeling, sound effects, and sensing technology. This review presents state-of-the-art hardware and software techniques used in hand gesture detection, primarily for VR applications. It discusses the challenges in hand gesture detection, classifies gestures as static and dynamic, and grades their detection difficulty. This paper also reviews the haptic devices used in VR and their advantages and challenges. It provides an overview of the process used in hand gesture acquisition, from inputs and pre-processing to pose detection, for both static and dynamic gestures.

DMS-yolov8 A gesture recognition algorithm based on improved yolov81

Aiming at the problems of slow speed and low accuracy of traditional neural network systems for real-time gesture recognition in complex backgrounds., this paper proposes DMS-yolov8-a gesture recognition method to improve yolov8. This algorithm replaces the Bottleneck convolution module in the backbone network of yolov8 with variable row convolution DCNV2, and increases the feature convolution range without increasing the computation amount through a more flexible feeling field. in addition, the self-developed MPCA attention module is added after the feature output layer of the backbone layer, which improves the problem of recognizing the accuracy of difference gestures in complex backgrounds by effectively combining the feature information of the contextual framework, taking into account the multi-scale problem of the gestures in the image, this paper introduces the SPPFCSPS module, which realizes multi-feature fusion and improves real-time accuracy of detection. Finally, the model proposed in this paper is compared with other models, and the proposed DMS-yolov8 model achieves good results on both publicly available datasets and homemade datasets, with the average accuracy up to 97.4% and the average mAP value up to 96.3%, The improvements proposed in this paper are effectively validated.

Web Real-Time Communication Using Gesture

Gesture recognition has emerged as a promising technology for enhancing human-computer interaction in various domains. In this paper, we propose a novel approach to gesture recognition using WebRTC (Web Real-Time Communication), a powerful web technology that enables real-time communication between web browsers. Our approach leverages the capabilities of WebRTC to capture live video streams from webcams and process them in real-time for gesture recognition. We present a detailed methodology for integrating WebRTC with a custom deep learning techniques. We evaluate the performance of our system using a diverse set of hand gestures and demonstrate its effectiveness in real-world scenarios. Our experimental results show an average recognition accuracy of 90%, outperforming existing methods in terms of both accuracy and computational efficiency. Potential applications of our approach in interactive web applications, gaming, and augmented reality systems. This work contributes the research on gesture recognition and demonstrates the feasibility of using WebRTC for real-time gesture recognition applications. Key Words: gesture recognition, WebRTC , Real-time communication

Intelligent door unlock system using AI MediaPipe

Prioritizing advanced security, the infusion of AI Mediapipe into the realm of smart door access not only simplifies access control but also guarantees unmatched protection, reshaping the expectation for safety and reliability. Implementing AI Mediapipe facilitates real-time gesture recognition, face detection, and hand tracking, optimizing access control in smart door systems. Integrating software and hardware set a new standard in the domain of smart door access. Here, we have proposed a smart system with a high-resolution camera to capture faces and connect them to a processing unit, i.e., Raspberry Pi. A face detection algorithm has been deployed into the hardware device so that the lock would open once an authorized face is detected. Our proposed system can make accurate predictions compared to existing systems. Integrating advanced computer vision and authentication methods marks a significant stride in the evolution of intelligent entry solutions.

Human–machine collaboration using gesture recognition in mixed reality and robotic fabrication

This research presents an innovative approach that integrated gesture recognition into a Mixed Reality (MR) interface for human–machine collaboration in the quality control, fabrication, and assembly of the Unlog Tower. MR platforms enable users to interact with three-dimensional holographic instructions during the assembly and fabrication of highly custom and parametric architectural constructions without the necessity of two-dimensional drawings. Previous MR fabrication projects have primarily relied on digital menus and custom buttons within the interface for user interaction between virtual and physical environments. Despite this approach being widely adopted, it is limited in its ability to allow for direct human interaction with physical objects to modify fabrication instructions within the virtual environment. The research integrates user interactions with physical objects through real-time gesture recognition as input to modify, update, or generate new digital information. This integration facilitates reciprocal stimuli between the physical and virtual environments, wherein the digital environment is generative of the user’s tactile interaction with physical objects. Thereby providing user with direct, seamless feedback during the fabrication process. Through this method, the research has developed and presents three distinct Gesture-Based Mixed Reality (GBMR) workflows: object localization, object identification, and object calibration. These workflows utilize gesture recognition to enhance the interaction between virtual and physical environments, allowing for precise localization of objects, intuitive identification processes, and accurate calibrations. The results of these methods are demonstrated through a comprehensive case study: the construction of the Unlog Tower, a 36’ tall robotically fabricated timber structure.

A Robust and Real-Time Framework of Cross-Subject Myoelectric Control Model Calibration via Multi-Source Domain Adaptation.

Surface electromyogram (sEMG) has been widely used in hand gesture recognition. However, most previous studies focused on user-personalized models, which require a great amount of data from each new target user to learn the user-specific EMG patterns. In this work, we present a novel real-time gesture recognition framework based on multi-source domain adaptation, which learns extra knowledge from the data of other users, thereby reducing the data collection burdens on the target user. Additionally, compared with conventional domain adaptation methods which treat data from all users in the source domain as a whole, the proposed multi-source method treat data from different users as multiple separate source domains. Therefore, more detailed statistical information on the data distribution from each user can be learned effectively. High-density sEMG (256 channels) from 20 subjects was used to validate the proposed method. Importantly, we evaluated our method with a simulated real-time processing pipeline on continuous sEMG data stream, rather than well-segmented data. The false alarm rate during rest periods in an EMG data stream, which is typically neglected by previous studies performing offline analyses, was also considered. Our results showed that, with only 1 s sEMG data per gesture from the new user, the 10-gesture classification accuracy reached 87.66 % but the false alarm rate was reduced to 1.95 %. Our method can reduce the frustratingly heavy data collection burdens on each new user.

Enhancing HCI Through Real-Time Gesture Recognition with Federated CNNs: Improving Performance and Responsiveness

Enhancing HCI Through Real-Time Gesture Recognition with Federated CNNs: Improving Performance and Responsiveness

Gesture Classification in Electromyography Signals for Real-Time Prosthetic Hand Control Using a Convolutional Neural Network-Enhanced Channel Attention Model.

Accurate and real-time gesture recognition is required for the autonomous operation of prosthetic hand devices. This study employs a convolutional neural network-enhanced channel attention (CNN-ECA) model to provide a unique approach for surface electromyography (sEMG) gesture recognition. The introduction of the ECA module improves the model's capacity to extract features and focus on critical information in the sEMG data, thus simultaneously equipping the sEMG-controlled prosthetic hand systems with the characteristics of accurate gesture detection and real-time control. Furthermore, we suggest a preprocessing strategy for extracting envelope signals that incorporates Butterworth low-pass filtering and the fast Hilbert transform (FHT), which can successfully reduce noise interference and capture essential physiological information. Finally, the majority voting window technique is adopted to enhance the prediction results, further improving the accuracy and stability of the model. Overall, our multi-layered convolutional neural network model, in conjunction with envelope signal extraction and attention mechanisms, offers a promising and innovative approach for real-time control systems in prosthetic hands, allowing for precise fine motor actions.

Lightweight hybrid model based on MobileNet-v2 and Vision Transformer for human–robot interaction

Within convolutional neural networks, convolutional operations are good at extracting local features, but have difficulty in capturing global representations. For Vision Transformer, multi-head self-attention can capture feature dependencies over long distance, but can destruct local feature details. Based on this, we propose a novel lightweight model, named HybridNet, based on MobileNet-v2 and Vision Transformer, capable of combining the advantages of both CNNs and Vision Transformer. In addition, to enhance the capability of HybridNet for temporal information interaction, we incorporate temporal-channel attention in HybridNet. We conducted experiments on Kinetics-400, Jester, and EgoGesture datasets to validate the effectiveness of HybridNet. The experimental results demonstrate that the lightweight model HybridNet achieves 96.3% and 93.9% accuracy on Jester and EgoGesture, respectively, obtaining the performance close to or even comparable with the state-of-the-art methods. Last but not least, we take HybridNet as the real-time gesture recognition model and use the recognition results as commands to control robots in the simulation environment to achieve human–robot interaction. The use of gesture interaction between humans and robots improves communication, facilitates physical collaboration, enables non-verbal expression, enhances accessibility, and creates a more engaging user experience. It adds a dimension of intuitiveness and efficiency to human–robot interaction, making it more dynamic and interactive.

MXene-Induced Flexible, Water-Retention, Semi-Interpenetrating Network Hydrogel for Ultra-Stable Strain Sensors with Real-Time Gesture Recognition.

As water-saturated polymer networks, hydrogels are a growing family of soft materials that have recently become promising candidates for flexible electronics application. However, it remains still difficult for hydrogel-based strain sensors to achieve the organic unity of mechanical properties, electrical conductivity, and water retention. To address this challenge, based on the template, the excellent properties of MXene nanoflakes (rich surface functional groups, high specific surface area, hydrophilicity, and conductivity) are fully utilized in this study to prepare the P(AA-co-AM)/MXene@PDADMAC semi-interpenetrating network (semi-IPN) hydrogel. The proposed hydrogel continues to exhibit excellent strain response and flexibility after 30 days of storage at room temperature, and its performance do not decrease after 1100 cycles. Considering these characteristics, a hydrogel-based device for converting sign language into Chinese characters is successfully developed and optimized using machine learning. Therefore, this study provides novel insight and application directions for hydrogel families.

Recognizing gestures of the ukrainian dactylic alphabet

Sign language is a visual way of communicating used by people who are deaf or hard of hearing. It involves handshapes, facial expressions, and body movements to convey meaning. Sign language helps the deaf community interact with each other and the hearing world, allowing them to participate fully in society. According to the WHO (World Health Organization) over 5 % of the world’s population – or 430 million people — experience problems with hearing. More than 44,000 people with hearing impairments are registered with the Ukrainian Society of the Deaf, an all-Ukrainian public organization for the disabled. Therefore, it is extremely important to develop new software, available to the public, that would allow quickly and effectively learn and understand sign language. This work aims to review gesture recognition techniques and develop a system for detecting and classifying gestures of the Ukrainian dactylic alphabet. Two main approaches to gesture recognition, glove-based and computer vision-based (CV), are explained, with the latter being preferred due to its flexibility and widespread usage. The text elaborates on deep learning-based approaches, particularly LSTM networks, and the advantages they offer in automatically learning features from raw image data. The process of creating a dataset for training the gesture classification model is described, which involves recording videos of hand gestures and extracting keypoints using Google MediaPipe. The mo­del training phase is detailed, covering the architecture of the LSTM-based classifier, optimization algorithms, and loss functions. The resulting model achieves an accuracy of 98.4% on the test dataset. A program for real-time gesture recognition is developed using Python and relevant libraries. The program utilizes a webcam feed to detect and classify hand gestures, displaying the top three predicted letters of the Ukrainian dactylic alphabet. The scientific novelty of the obtained results: the paper presents a method that utilizes hand keypoints for recognizing hand gestures of the Ukrainian dactyl alphabet. Also, as part of the development of the gesture recognition system, a data set was collected, where each gesture corresponds to 50 videos of 65 frames. The practical significance of the results obtained: the model obtained as a result of the study can be used to interpret the gestures of the Ukrainian dactylic alphabet. The dataset collected for training this model can be used in other works to train or validate similar models. The paper might be of use to the ones who are interested in developing similar systems for gesture recognition.

HapticSOUND: An Interactive Learning Experience with a Digital Musical Instrument

In this paper, an interactive learning experience is proposed, aiming to involve museum visitors in a personalized experience of the transmittal of cultural knowledge in an active and creative way. The proposed system, called HapticSOUND, consists of three subsystems: (a) the Information, where visitors are informed about the traditional musical instruments; (b) the Entertainment, where visitors are entertained by playing serious games to virtually assemble traditional musical instruments by a set of 3D objects; and (c) the Interaction, where visitors interact with a digital musical instrument which is an exact 3D-printed replica of a traditional musical instrument, where cameras have been placed to capture user gestures and machine learning algorithms have been implemented for gesture recognition. The museum visitor can interact with the lifelike replica to tactilely and aurally explore the instrument’s abilities, producing sounds guided by the system and receiving real-time visual and audio feedback. Emphasis is given to the Interaction Subsystem, where a pilot study was conducted to evaluate the usability of the subsystem. Preliminary results were promising since the usability was satisfactory, indicating that it is an innovative approach that utilizes sensorimotor learning and machine learning techniques in the context of playing sounds based on real-time gesture and fingering recognition.

Computer Vision based Media Control using Hand Gestures

Abstract: Hand gestures are a form of nonverbal communication that can be used in several fields such as communication between deaf-mute people, robot control, human–computer interaction (HCI), home automation and medical applications. At this time and age, working with a computer in some capacity is a common task. In most situations, the keyboard and mouse are the primary input devices. However, there are several problems associated with excessive usage of the same interaction medium, such as health problems brought on by continuous use of input devices, etc. Humans basically communicate using gestures and it is indeed one of the best ways to communicate. Gesture-based real-time gesture recognition systems received great attention in recent years because of their ability to interact with systems efficiently through human-computer interaction. This project implements computer vision and gesture recognition techniques and develops a vision based low-cost input software for controlling the media player through gestures.

Fine-Grained and Real-Time Gesture Recognition by Using IMU Sensors

Gesture recognition by using Inertial Measurement Unit (IMU) sensors plays an important role in various Internet of Things (IOT) applications, e.g., smart home, intelligent medical system and so on. Traditional technologies usually utilize machine learning algorithms to train different gestures during the offline phase, then recognize the gesture during the online phase. However, such technologies cannot recognize these gestures without prior training. Even for the same gesture, with different gesture amplitude may result in unsuccessful recognition. Also if we change the person to perform the same gesture, the algorithms fails. In order to overcome these drawbacks, we propose an approach, which will be able to track the human body motion in realtime and also recognize complicated gestures. It utilizes the accelerometer information and proposes comprehensive localization algorithms for each deployed sensor attached on the human body. Then, it takes the correlation and limitation among body parts into account to recognize the gesture. Our experiments results show that, the successful recognition rate of our algorithm is 100%. Furthermore, any part of the human body can be well tracked, the tracking accuracy can reach 0.06m.

Sign Language Recognition and Response via Virtual Reality

The main aim of this project is to provide a voice to voiceless people with the help of Python and Unity prototype systems. Aforesaid project work focuses on the problem of gesture recognition in real time that sign language is used by the community of deaf people. The problem addressed is based on Digital image processing using color segmentation, Skin detection, Image Segmentation, Image Filtering, and Template Matching techniques. This system recognizes gestures of ISL(Indian Sign Language) including the alphabet and a subset of its words.

Using Wool Keratin as a Structural Biomaterial and Natural Mediator to Fabricate Biocompatible and Robust Bioelectronic Platforms.

The design and fabrication of biopolymer-incorporated flexible electronics have attracted immense interest in healthcare systems, degradable implants, and electronic skin. However, the application of these soft bioelectronic devices is often hampered by their intrinsic drawbacks, such as poor stability, inferior scalability, and unsatisfactory durability. Herein, for the first time, using wool keratin (WK) as a structural biomaterial and natural mediator to fabricate soft bioelectronics is presented. Both theoretical and experimental studies reveal that the unique features of WK can endow carbon nanotubes (CNTs) with excellent water dispersibility, stability, and biocompatibility. Therefore, well-dispersed and electroconductive bio-inks can be prepared via a straightforward mixing process of WK and CNTs. The as-obtained WK/CNTs inks can be directly exploited to design versatile and high-performance bioelectronics, such as flexible circuits and electrocardiogram electrodes. More impressively, WK can also be a natural mediator to connect CNTs and polyacrylamide chains to fabricate a strain sensor with enhanced mechanical and electrical properties. With conformable and soft architectures, these WK-derived sensing units can be further assembled into an integrated glove for real-time gesture recognition and dexterous robot manipulations, suggesting the great potential of the WK/CNT composites for wearable artificial intelligence.

ShuffleNetv2-YOLOv3: a real-time recognition method of static sign language based on a lightweight network

To better meet the communication needs of hearing impaired people and the public, it is of great significance to recognize sign language more quickly and accurately on embedded platforms and mobile terminals. YOLOv3, raised by Joseph Redmon and Ali Farhadi in 2018, achieved a great improvement in detection speed with considerable accuracy by optimizing Yolo. However, YOLOv3 is still too bloated to use on mobile terminals. A static sign language recognition method based on the ShuffleNetv2-YOLOv3 lightweight model was proposed. The ShuffleNetv2-YOLOv3 lightweight model makes the network lightweight by using ShuffleNetv2 as the backbone network of YOLOv3. The lightweight network improved the recognition speed steeply. Combing with the CIoU loss function, the ShuffleNetv2-YOLOv3 keeps the recognition accuracy while improving the recognition speed. Recognition effectiveness of the self-made sign language images and public database by the ShuffleNetv2-YOLOv3 lightweight model was evaluated by F1 score and mAP value. The performance of the ShuffleNetv2-YOLOv3 model was compared with that of the YOLOv3-tiny, SSD, Faster-RCNN, and YOLOv4-tiny model, respectively. The experimental results show that the proposed ShuffleNetv2-YOLOv3 model achieved a good balance between the accuracy and speed of the gesture detection under the premise of model lightweight. The F1 score and mAP value of the ShuffleNetv2-YOLOv3 model were 99.1% and 98.4%, respectively. The gesture detection speed on the GPU reaches 54 frames per second, which is better than other models. The mobile terminal application of the proposed lightweight model was also evaluated. The minimal inference speed of single frame images on the CPU and GPU is 0.14 and 0.025 s per image, respectively. It is only 1/6.5 and 1/8.5 of the running speed of the original YOLOv3 model. The ShuffleNetv2-YOLOv3 lightweight model is conducive to quick, real time, and similar static sign language gesture recognition, laying a good foundation for real-time gesture recognition in the embedded platforms and mobile terminals.