Accurate Gesture Research Articles

Hand Gesture Recognition for Interactive On-Screen Drawing with Real-Time Subtitling for Teaching

This research presents the development of an innovative educational tool leveraging hand gesture recognition for interactive on-screen drawing with real-time subtitling. Built on OpenCV and Python, the platform enables educators to draw and manipulate digital content through intuitive hand gestures, enhancing classroom engagement. Additionally, real-time subtitle generation offers accessibility for hearing-impaired students by converting drawn content into text notes. This system addresses key challenges in traditional digital drawing tools, such as limited interaction methods and accessibility, aiming to deliver a more inclusive learning environment. Preliminary results indicate high recognition accuracy, smooth interaction, and potential for broader educational applications. Future improvements will involve expanding the gesture set and refining accuracy for complex gestures. Keywords: Hand Gesture Recognition, Interactive Learning, Real-Time Subtitling, Educational Technology, OpenCV

FM‐YOLOv8:Lightweight gesture recognition algorithm

AbstractIn practical production applications, the efficiency and success rate of gesture recognition directly affect the user experience and work efficiency. However, the existing gesture recognition models have the problem of a large number of model parameters and high computational complexity, which makes them unable to meet the needs of end‐to‐end industrial deployment. To solve these problems, this article proposes a gesture recognition model based on YOLOv8. First, FasterNet is adopted as the backbone of YOLOv8, which significantlydecrease the number of parameters and made the model more lightweight. By reducing the number of parameters, the computational complexity of the model can be reduced while maintaining the performance of the model, and the operation efficiency of the model can be improved. Second, recombination convolution ScConv is introduced to replace common convolution operations to further improve the model's efficiency. Recombination convolution can reduce the computation and make up for the loss of precision to some extent. Finally, the MDPIoU loss function is used to optimize target location and prediction, to improve the accuracy of the model. The MDPIoU loss function can better deal with the problem of target boundary frame positioning and prediction so that the model can locate and predict gestures more accurately in gesture recognition tasks. Experiments on a data set containing 10 types of gestures show that the number of parameters and floating point calculations of the improved network model are reduced by 45% and 42.7%, respectively, while the accuracy is unchanged. The improved model can be deployed on edge terminals, providing efficient and accurate gesture recognition.

Human Muscle sEMG Signal and Gesture Recognition Technology Based on Multi-Stream Feature Fusion Network

Surface electromyography signals have significant value in gesture recognition due to their ability to reflect muscle activity in real time. However, existing gesture recognition technologies have not fully utilized surface electromyography signals, resulting in unsatisfactory recognition results. To this end, firstly, a Butterworth filter was adopted to remove high-frequency noise from the signal. A combined method of moving translation threshold was introduced to extract effective signals. Then, a gesture recognition model based on multi-stream feature fusion network was constructed. Feature extraction and fusion were carried out through multiple parallel feature extraction paths, combined with convolutional neural networks and residual attention mechanisms. Compared to popular methods of the same type, this new recognition method had the highest recognition accuracy of 92.1% and the lowest recognition error of 5%. Its recognition time for a single-gesture image was as short as 4s, with a maximum Kappa coefficient of 0.92. Therefore, this method combining multi-stream feature fusion networks can effectively improve the recognition accuracy and robustness of gestures and has high practical value.

EchoGesture Communication: Gesture-based Systems for Individuals with Disabilities

EchoGesture Communication revolutionizes the interaction of differently-abled individuals using hand gestures. People with disabilities often face difficulties in using the conventional electronic gadgets. The proposed study, utilizes sensors, microcontroller, computer vision, and machine learning, to enable real-time recognition of hand gestures, facilitating effective communication. Additionally, Convolutional Neural Network (CNN) is used in the research to achieve accurate gesture recognition. The proposed system allows individuals with disability to communicate effectively using hand gestures.

Stretchable Iontronic Tactile Sensing Fabric.

The iontronic tactile sensing modality has garnered significant attention due to its exceptional sensitivity, immunity to noise, and versatility in materials. Recently, various formats of iontronic tactile sensors have been developed, including droplets, polymer films, paper, ionic gels, and fabrics. However, the stretchability of the current iontronic pressure sensing fabric is inadequate, hindered by the limited stretchiness of the ionic functional fabric. Incorporating a stretchable tactile sensing implement could enhance the wear comfortability by preventing relative movement and ensuring intimate contact between the sensor and the skin. The research focuses on the development of a stretchable iontronic pressure sensing (SIPS) fabric for monitoring diverse aspects of body health and movement in wearable applications. The tactile sensing structure is generated at the iontronic interface between highly stretchable ionic and conductive fabrics. In particular, the ionic fabric is prepared by coating a layer of polyurethane/ionic liquid gel onto a Spandex fabric. To showcase its remarkable sensitivity, stretchability, and ability to detect diverse body information, several application scenarios have been demonstrated including an elastic wristband for precise pulse wave detection, a flexible belt with multitactile sensing channels for respiration and motion tracking purposes, and a stretchable fabric cuff equipped with a high-resolution sensing array comprising 32 × 32 units for accurate gesture recognition.

Low-temperature fabrication of lead-free Ag2BiI5 perovskite photodetector with excellent weak light detection ability for accurate gesture recognition

In recent years, lead halide perovskites have been extensively researched in the optoelectronics field due to their excellent optical and electrical properties, while the inherent toxicity and instability of lead halide perovskites have hindered their commercialization. Stable, lead-free perovskites as a substitute of the Pb-based perovskites show great attention. Herein, the green antisolvent diethyl carbonate is introduced to realize the low-temperature fabrication of nontoxic perovskite Ag2BiI5. The Ag2BiI5 perovskite photodetector (PD) shows excellent performance with a detectivity of as high as 1.13 × 1013 Jones and a linear dynamic range (LDR) of 147 dB. Besides, the weak light with the light intensity of as low as 18.3 nW/cm2 can be detected by our PD. Moreover, the flexible PDs are also prepared with this low-temperature process, which shows good device performance with stable bending resistance characteristics. Based on the excellent optoelectronic properties of the Ag2BiI5 PDs, a gesture recognition system is built which can realize rapid and accurate gesture recognition under ordinary indoor light, with an accuracy of 98.35 %. Our research provides new insights for the development and application of green and environmentally friendly perovskite PD.

A channel-fused gated temporal convolutional network for EMG-based gesture recognition

Background:EMG gesture recognition can be widely applied in many fields, such as prosthetic control and human–computer interaction, to enhance the standard of human life. Purpose:This study aims to develop a deep learning-based model to identify multiple complex gestures from raw EMG signals. Methods:We propose a new channel-fused gated temporal convolutional network. First, a channel fusion and gating mechanism is designed to improve temporal convolutional networks, allowing the model to obtain higher-level features. Second, we improve the channel fusion module by the short-term average energy to fuse the EMG signals of multi-channels more accurately. Results:The model is evaluated on a public dataset of EMG gestures, NinaPro DB5. Results demonstrated that the unbalanced accuracy rate of 53 gesture actions reaches 92.71%, and the balanced accuracy rate 74.79%. Further, ablation experiments validates the effectiveness of each model module. Conclusions:This study demonstrates our proposed approach can improve gesture recognition accuracy for complex gestures and has great potential for practical applications.

DMS-yolov8 A gesture recognition algorithm based on improved yolov81

Aiming at the problems of slow speed and low accuracy of traditional neural network systems for real-time gesture recognition in complex backgrounds., this paper proposes DMS-yolov8-a gesture recognition method to improve yolov8. This algorithm replaces the Bottleneck convolution module in the backbone network of yolov8 with variable row convolution DCNV2, and increases the feature convolution range without increasing the computation amount through a more flexible feeling field. in addition, the self-developed MPCA attention module is added after the feature output layer of the backbone layer, which improves the problem of recognizing the accuracy of difference gestures in complex backgrounds by effectively combining the feature information of the contextual framework, taking into account the multi-scale problem of the gestures in the image, this paper introduces the SPPFCSPS module, which realizes multi-feature fusion and improves real-time accuracy of detection. Finally, the model proposed in this paper is compared with other models, and the proposed DMS-yolov8 model achieves good results on both publicly available datasets and homemade datasets, with the average accuracy up to 97.4% and the average mAP value up to 96.3%, The improvements proposed in this paper are effectively validated.

Self-healing and adhesive MXene-polypyrrole/silk fibroin/polyvinyl alcohol conductive hydrogels as wearable sensor

Conductive hydrogels become increasing attractive for flexible electronic devices and biosensors. However, challenges still remain in fabrication of flexible hydrogels with high electrical conductivity, self-healing capability and adhesion property. Herein, a conductive hydrogel (PSDM) was prepared by solution-gel method using MXene and dopamine modified polypyrrole as conductive enhanced materials, polyvinyl alcohol and silk fibroin as gel networks, and borax as cross-linking agent. Notably, the PSDM hydrogels not only showed high permeability (13.82 mg∙cm−2∙h−1), excellent stretch ability (1235 %), high electrical conductivity (11.3 S/m) and long-term stability, but also exhibited high adhesion performance and self-healing properties. PSDM hydrogels displayed outstanding sensing performance and durability for monitoring human activities including writing, finger bending and wrist bending. The PSDM hydrogel was made into wearable flexible electrodes and realized accurate, sensitive and reliable detection of human electromyographic and electrocardiographic signals. The sensor was also applied in human-computer interaction by collecting electromyography signals of different gestures for machine learning and gesture recognition. According to 480 groups of data collected, the recognition accuracy of gestures by the electrodes was close to 100 %, indicating that the PSDM hydrogel electrodes possessed excellent sensing performance for high precision data acquisition and human-computer interaction interface.

WiVi-GR: Wireless-Visual Joint Representation-Based Accurate Gesture Recognition

WiVi-GR: Wireless-Visual Joint Representation-Based Accurate Gesture Recognition

FPGA-based Lightweight QDS-CNN System for sEMG Gesture and Force Level Recognition.

Deep learning (DL) has been used for electromyographic (EMG) signal recognition and achieved high accuracy for multiple classification tasks. However, implementation in resource-constrained prostheses and human-computer interaction devices remains challenging. To overcome these problems, this paper implemented a low-power system for EMG gesture and force level recognition using Zynq architecture. Firstly, a lightweight network model structure was proposed by Ultra-lightweight depth separable convolution (UL-DSC) and channel attention-global average pooling (CA-GAP) to reduce the computational complexity while maintaining accuracy. A wearable EMG acquisition device for real-time data acquisition was subsequently developed with size of 36mm×28mm×4mm. Finally, a highly parallelized dedicated hardware accelerator architecture was designed for inference computation. 18 gestures were tested, including force levels from 22 healthy subjects. The results indicate that the average accuracy rate was 94.92% for a model with 5.0k parameters and a size of 0.026MB. Specifically, the average recognition accuracy for static and force-level gestures was 98.47% and 89.92%, respectively. The proposed hardware accelerator architecture was deployed with 8-bit precision, a single-frame signal inference time of 41.9μs, a power consumption of 0.317W, and a data throughput of 78.6 GOP/s.

Automated Gesture Recognition Using Zebra Optimization Algorithm with Deep Learning Model for Visually Challenged People

Gesture recognition for visually challenged people plays a vital role in improving their convenience and interaction with digital gadgets and environments. It includes improvement of systems that permit them to relate with digital devices by using hand actions or gestures. To improve user-friendliness, these systems select in-built and effortlessly learnable gestures, often integrating wearable devices prepared with sensors for precise detection. Incorporating auditory or haptic feedback devices offers real-time cues about achievement of familiar gestures. Machine learning (ML) and deep learning (DL) methods are useful tools for accurate gesture detection, with customization choices to accommodate individual preferences. In this view, this article concentrates on design and development of Automated Gesture Recognition using Zebra Optimization Algorithm with Deep Learning (AGR-ZOADL) model for Visually Challenged People. The AGR-ZOADL technique aims to recognize the gestures to aid visually challenged people. In the AGR-ZOADL technique, the primary level of data pre-processing is involved by median filtering (MF). Besides, the AGR-ZOADL technique applies NASNet model to learn complex features from the preprocessed data. To enhance performance of NASNet technique, ZOA based hyperparameter procedure performed. For gesture recognition process, stacked long short term memory (SLSTM) model is applied. The performance validation of AGR-ZOADL technique carried out using a benchmark dataset. The experimental values stated that AGR-ZOADL methodology extents significant performance over other present approaches

Gesture Controlled Virtual Mouse with Voice Assistant

The usage of hand gestures to operate digital devices has grown in recent years, and developing accurate gesture recognition systems using computer vision technology is becoming easier. Voice assistants have also become more common, with developments in artificial intelligence (AI) and natural language processing (NLP) allowing for more accurate voice recognition. Our goal is to create a system which combines both gesture-controlled mouse functionality and voice assistant to execute things that a regular mouse and keyboard can. The GCVA (Gesture Controlled Voice Assistant) comes with a gesture-controlled mouse and a voice assistant. A gesture-controlled mouse uses OpenCV and Media-Pipe, whereas a voice assistant uses AI and NLP concepts to recognize the voice commands given by the user. The system should be used by performing hand gestures and executing the right click, left click, drag, drop, volume control and computer cursor functions, thereby omitting the need for a physical mouse. The success of this system is observed in terms of practicality, stability, and compatibility with physical mechanisms. The voice assistant works and responds to pre-defined commands present in the system. The GCVA system was tested by four different users in order to generate the accuracy scores of mouse gestures and response time graphs for voice assistant. The system yields an accuracy rate of 98%. The system's accuracy and usability showed that hand gesture detection and voice assistants have the potential to replace traditional input devices in the future.

AudioGest: Gesture-based Interaction for Virtual Reality using Audio Devices.

Current virtual reality (VR) system takes gesture interaction based on camera, handle and touch screen as one of the mainstream interaction methods, which can provide accurate gesture input for it. However, limited by application forms and the volume of devices, these methods cannot extend the interaction area to such surfaces as walls and tables. To address the above challenge, we propose AudioGest, a portable, plug-and-play system that detects the audio signal generated by finger tapping and sliding on the surface through a set of microphone devices without extensive calibration. First, an audio synthesis-recognition pipeline based on micro-contact dynamics simulation is constructed to generate modal audio synthesis from different materials and physical properties. Then the accuracy and effectiveness of the synthetic audio are verified by mixing the synthetic audio with real audio proportionally as the training sets. Finally, a series of desktop office applications are developed to demonstrate the application potential of AudioGest's scalability and versatility in VR scenarios.

Recognition of Wh-Question Sign Gestures in Video Streams using an Attention Driven C3D-BiLSTM Network

Human sign language is a visual and gestural means of communication used by people with hearing impairments to interact with others. It has the potential to enable interaction between individuals who struggle with verbal communication and those who lack the essential proficiency to comprehend sign language. Wh-questions constitute a substantial portion of daily sign language interactions. Automatic recognition of Wh-question words in sign language, encompassing interrogative words, has received limited attention in gesture recognition. In this context, a novel dataset named the American Question Sign Video dataset (AQSVd) has been introduced as a significant contribution. Recognizing Wh-question signs in video streams empowers individuals to effectively convey messages through hand movements and gestures, fostering inclusivity and accessibility in communication for deaf and hearing-impaired individuals. The paper proposes a Deep Convolutional-3D BiLSTM Multi-head Attention network for recognizing American Wh-Question word sign gestures in video streams. Incorporating Multi-head attention in the proposed model enhances its ability to capture intricate spatial and temporal features, essential for accurate gesture recognition. The proposed model is thoroughly evaluated across a range of datasets, such as INCLUDE50, WLASL-100, and our AQSVd dataset, achieving remarkable results. Specifically, our model demonstrated an outstanding 98.91% validation accuracy on the AQSVd dataset. The C3D-BiLSTM MHAttention model outperforms other state-of-the-art models, demonstrating its superiority in sign language recognition tasks. The proposed dataset and C3D-BiLSTM Multi-head Attention model contribute significantly to this field, offering potential benefits in education, human-robot interaction, and overall communication for the hearing-impaired community.

A Novel Multi-Feature Fusion Network With Spatial Partitioning Strategy and Cross-Attention for Armband-Based Gesture Recognition.

Effectively integrating the time-space-frequency information of multi-modal signals from armband sensor, including surface electromyogram (sEMG) and accelerometer data, is critical for accurate gesture recognition. Existing approaches often neglect the abundant spatial relationships inherent in multi-channel sEMG signals obtained via armband sensors and face challenges in harnessing the correlations across multiple feature domains. To address this issue, we propose a novel multi-feature fusion network with spatial partitioning strategy and cross-attention (MFN-SPSCA) to improve the accuracy and robustness of gesture recognition. Specifically, a spatiotemporal graph convolution module with a spatial partitioning strategy is designed to capture potential spatial feature of multi-channel sEMG signals. Additionally, we design a cross-attention fusion module to learn and prioritize the importance and correlation of multi-feature domain. Extensive experiment demonstrate that the MFN-SPSCA method outperforms other state-of-the-art methods on self-collected dataset and the Ninapro DB5 dataset. Our work addresses the challenge of recognizing gestures from the multi-modal data collected by armband sensor, emphasizing the importance of integrating time-space-frequency information. Codes are available at https://github.com/ZJUTofBrainIntelligence/MFN-SPSCA.

Motorised Bionic Arm Using SSVEP with Enhanced Hand Gesture Accuracy and Reliability

Motorised Bionic Arm Using SSVEP with Enhanced Hand Gesture Accuracy and Reliability

Piano Playing Gesture Recognition Based on Multiple Intelligences Theory

Abstract In this paper, firstly, in order to solve the piano playing recognition problem in the field of artificial intelligence, based on the theory of multiple intelligences, the VGG-16 deep network migration learning algorithm is applied to estimate and acquire the piano playing gesture posture. Secondly, combined with the Iterative Update Extended Kalman Filter (IUEKF) algorithm, the micro-inertial sensor fixation of the piano-playing gesture is realized, which in turn is conducive to improving the piano-playing gesture recognition accuracy. Then, we obtain real-time piano-playing gesture information through a Kinect somatosensory device, construct a piano-playing gesture recognition model based on migration learning on the basis of obtaining piano-playing gesture features, and confirm the effectiveness of the model through the experimental study of piano-playing recognition. The results show that in piano-playing gesture recognition, the recognition accuracy of this paper’s method remains above 0.9, and the application of this paper’s method can effectively improve the recognition accuracy of piano-playing gestures. On piano playing pedal action recognition, this paper’s method shows that the average F-measure scores of these two strategies are 0.924 and 0.944, respectively, which are better compared to other methods. This study provides an effective case for applying AI techniques to piano performance recognition and broadens the scope of AI applications.

Gesture Classification in Electromyography Signals for Real-Time Prosthetic Hand Control Using a Convolutional Neural Network-Enhanced Channel Attention Model.

Accurate and real-time gesture recognition is required for the autonomous operation of prosthetic hand devices. This study employs a convolutional neural network-enhanced channel attention (CNN-ECA) model to provide a unique approach for surface electromyography (sEMG) gesture recognition. The introduction of the ECA module improves the model's capacity to extract features and focus on critical information in the sEMG data, thus simultaneously equipping the sEMG-controlled prosthetic hand systems with the characteristics of accurate gesture detection and real-time control. Furthermore, we suggest a preprocessing strategy for extracting envelope signals that incorporates Butterworth low-pass filtering and the fast Hilbert transform (FHT), which can successfully reduce noise interference and capture essential physiological information. Finally, the majority voting window technique is adopted to enhance the prediction results, further improving the accuracy and stability of the model. Overall, our multi-layered convolutional neural network model, in conjunction with envelope signal extraction and attention mechanisms, offers a promising and innovative approach for real-time control systems in prosthetic hands, allowing for precise fine motor actions.

Hand Gesture Recognition and Motion Detection System for Interactive Applications

Security systems and interactive multimedia applications are two examples of the many fields in which Gesture Sense (GS) technology, a significant development in human-computer interaction, has found use. This paper introduces the "Arduino Hand Gesture Identification and Motion Detection System," a revolutionary combination of Python, Arduino Uno R3, and ultrasonic sensors intended to improve interactive apps through accurate gesture identification. Real-time tracking of hand movements is made possible by ultrasonic sensors, which are renowned for their fine-grained motion detection capabilities. Python allows the Arduino Uno R3 to run complex gesture detection algorithms. A user's engagement is improved by an LCD that presents recognized gestures and status updates, which provides immediate visual feedback. The system's outstanding accuracy, responsiveness, and usability are demonstrated by severe testing, which shows that it typically recognizes objects with an accuracy of 93%. In the context of interactive applications, this research study demonstrates how GS technology has the potential to revolutionize user experiences by ensuring seamless and immersive interactions. Key Words: Interactive application, Human-computer interaction, Hand gesture recognition, Motion detection.