Gesture Recognition Research Articles

Attention-Based Deep Spiking Neural Networks for Temporal Credit Assignment Problems.

The temporal credit assignment (TCA) problem, which aims to detect predictive features hidden in distracting background streams, remains a core challenge in biological and machine learning. Aggregate-label (AL) learning is proposed by researchers to resolve this problem by matching spikes with delayed feedback. However, the existing AL learning algorithms only consider the information of a single timestep, which is inconsistent with the real situation. Meanwhile, there is no quantitative evaluation method for TCA problems. To address these limitations, we propose a novel attention-based TCA (ATCA) algorithm and a minimum editing distance (MED)-based quantitative evaluation method. Specifically, we define a loss function based on the attention mechanism to deal with the information contained within the spike clusters and use MED to evaluate the similarity between the spike train and the target clue flow. Experimental results on musical instrument recognition (MedleyDB), speech recognition (TIDIGITS), and gesture recognition (DVS128-Gesture) show that the ATCA algorithm can reach the state-of-the-art (SOTA) level compared with other AL learning algorithms.

Single/Multi-Source Black-Box Domain Adaption for Sensor Time Series Data.

Unsupervised domain adaption (UDA), which transfers knowledge from a labeled source domain to an unlabeled target domain, has attracted tremendous attention in many machine learning applications. Recently, there have been attempts to apply domain adaption for sensor time series data, such as human activity recognition and gesture recognition. However, existing methods suffer from some drawbacks that hinder further performance improvement. They often require access to source data or source models during training, which is unavailable in some fields because of privacy protection and storage limit. Typically, the source domains may only provide an application programming interface (API) for the target domain to call. On the other hand, current UDA methods have not considered the temporal consistency and low-signal-to-noise ratio (SNR) of sensor time series. To address the challenges, this article presents a black-box domain adaption framework for sensor time series data (B2TSDA). First, we propose a single/multi-source teacher-student learning framework to distill the knowledge from the source domains to a customized target model. Then we design a new temporal consistency loss by combining an adaptive mask method and dynamic threshold method to maintain consistent temporal information and balance the learning difficulties of different classes. For the multisource black-box domain adaption, we further propose a Shapley-enhanced method to determine the contribution of each source domain. Experimental results on both single-source and multisource domain adaption show that our framework has superior performance compared to other black-box UDA methods.

A Multi-functional Electronic Glove for Multidimensional Environmental Perception and Gesture Recognition

In recent years, wearable electronic gloves with rich sensing functions have garnered widespread attention. Researchers are exploring various aspects of wearable devices including structural design, material innovation, and algorithm optimization. An analysis of demand suggests that wearable electronic gloves should possess sensory functions like hand, which includes roughness of contacted objects, assessing temperature of touched objects, determining grasp force, and interpreting gesture language. This paper presents a multi-sensing wearable flexible electronic glove equipped with graphene foam material sensors that can perceive external pressure, roughness, temperature, and express gesture language. The experimental results show that this glove exhibits excellent approximately linear-range sensing characteristics for force perception (1.475 kPa-1 at 0-0.4 kPa), outstanding temperature sensing capability (temperature sensitivity is -1.22% within the range of 30-80℃), and recognition judgment of 12 different gestures. Additionally, an optimized KNN (K-nearest neighbor) algorithm based on time-domain window is proposed. Compared with traditional KNN algorithm, KNN based on time-domain window improves the accuracy of gesture intent recognition from 86.15% to 91.93%. Wearable electronic gloves represent promising developments for fields such as hand rehabilitation training for patients, intention expression for individuals suffering from language barriers, and wearable multifunctional sensing applications.

An overview of hand gesture recognition based on computer vision

Hand gesture recognition emerges as one of the foremost sectors which has gone through several developments within pattern recognition. Numerous studies and research endeavors have explored methodologies grounded in computer vision within this domain. Despite extensive research endeavors, there is still a need for a more thorough evaluation of the efficiency of various methods in different environments along with the challenges encountered during the application of these methods. The focal point of this paper is the comparison of different research in the domain of vision-based hand gesture recognition. The objective is to find out the most prominent methods by reviewing efficiency. Concurrently, the paper delves into presenting potential solutions for challenges faced in different research. A comparative analysis particularly centered around traditional methods and convolutional neural networks like random forest, long short-term memory (LSTM), heatmap, and you only look once (YOLO). considering their efficacy. Where convolutional neural network-based algorithms performed best for recognizing the gestures and gave effective solutions for the challenges faced by the researchers. In essence, the findings of this review paper aim to contribute to future implementations and the discovery of more efficient approaches in the gesture recognition sector.

Handwave Harmony: Transforminggestures into Words

Our project introduces a real-time gesture recognition system using ResNet50 and OpenCV. This innovative technology offers precise interpretation of sign movements for individuals with hearing difficulties. This represents a form of interpersonal connection where individuals convey understanding, empathy, or agreement through the subtle movements of their hands. The system is trained on a diverse dataset for flexibility and utilizes ResNet50's capabilities while integrating OpenCV for reduced latency, ensuring effective communication. Rigorous testing validates its accuracy, with user input fosters iterative learning. The study highlights the Importance of continuous enhancement and collaboration with the deaf and hard-of-hearing community for greater inclusiveness.

A Low-Power Impedance-to-Frequency Converter for Frequency-Multiplexed Wearable Sensors.

We propose a low-power impedance-to-frequency (I-to-F) converter for wearable transducers that change both its resistance and capacitance in response to mechanical deformation or changes in ambient pressure. At the core of the proposed I-to-F converter is a fixed-point circuit comprising of a voltage-controlled relaxation oscillator and a proportional-to-temperature (PTAT) current reference that locks the oscillation frequency according to the impedance of the transducer. Using both analytical and measurement results we show that the operation of the proposed I-to-F converter is well matched to a specific class of sponge mechanical transducer where the system can achieve higher sensitivity when compared to a simple resistance measurement techniques. Furthermore, the oscillation frequency of the converter can be programmed to ensure that multiple transducer and I-to-F converters can communicate simultaneously over a shared channel (physical wire or virtual wireless channel) using frequency-division multiplexing. Measured results from proof-of-concept prototypes show an impedance sensitivity of 19.66 Hz/ Ω at 1.1 kΩ load impedance magnitude and a current consumption of [Formula: see text]. As a demonstration we show the application of the I-to-F converter for human gesture recognition and for radial pulse sensing.

Multi-View Fusion Network-Based Gesture Recognition Using sEMG Data.

sEMG(surface electromyography) signals have been widely used in rehabilitation medicine in the past decades because of their non-invasive, convenient and informative features, especially in human action recognition, which has developed rapidly. However, the research on sparse EMG in multi-view fusion has made less progress compared to high-density EMG signals, and for the problem of how to enrich sparse EMG feature information, a method that can effectively reduce the information loss of feature signals in the channel dimension is needed. In this article, a novel IMSE (Inception-MaxPooling-Squeeze- Excitation) network module is proposed to reduce the loss of feature information during deep learning. Then, multiple feature encoders are constructed to enrich the information of sparse sEMG feature maps based on the multi-core parallel processing method in multi-view fusion networks, while SwT (Swin Transformer) is used as the classification backbone network. By comparing the feature fusion effects of different decision layers of the multi-view fusion network, it is experimentally obtained that the fusion of decision layers can better improve the classification performance of the network. In NinaPro DB1, the proposed network achieves 93.96% average accuracy in gesture action classification with the feature maps obtained in 300ms time window, and the maximum variation range of action recognition rate of individuals is less than 11.2%. The results show that the proposed framework of multi-view learning plays a good role in reducing individuality differences and augmenting channel feature information, which provides a certain reference for non-dense biosignal pattern recognition.

Gesture recognition technology: a new dimension in human-computer interaction interface

Development of an interface for intelligent gesture control to improve user experience and increase the efficiency of interaction with a computer. This paper proposes a gesture recognition system based on artificial intelligence using convolutional neural networks (CNN). The system comprises three stages: pre-processing, optimal frame determination, and gesture category identification. The extracted features used are independent of movement, scaling, and rotation, providing greater flexibility to the system. The suggested gesture control technology, known as Kazakh Sign Language (KSL) for Kazakh alphabets, eliminates the need for additional devices, enabling users to interact with the system naturally. Experiments demonstrated that the proposed KSL system can accurately recognize Kazakh language alphabet letters with a high precision of 97.3%, owing to the utilization of artificial intelligence and CNN to enhance the accuracy and effectiveness of gesture control. Gestures, a type of visual formation, are perceivable by computers through machine learning models. The selection of methods and systems for recognizing Kazakh sign language gestures was accompanied by addressing various challenges related to language-specific orthographic and gestural features. The developed gesture control interface for human-computer interaction is applied in the field of inclusive education, aiming to assist deaf and hard-of-hearing children in learning sign language.

Innovative healthcare solutions: robust hand gesture recognition of daily life routines using 1D CNN.

Hand gestures are an effective communication tool that may convey a wealth of information in a variety of sectors, including medical and education. E-learning has grown significantly in the last several years and is now an essential resource for many businesses. Still, there has not been much research conducted on the use of hand gestures in e-learning. Similar to this, gestures are frequently used by medical professionals to help with diagnosis and treatment. We aim to improve the way instructors, students, and medical professionals receive information by introducing a dynamic method for hand gesture monitoring and recognition. Six modules make up our approach: video-to-frame conversion, preprocessing for quality enhancement, hand skeleton mapping with single shot multibox detector (SSMD) tracking, hand detection using background modeling and convolutional neural network (CNN) bounding box technique, feature extraction using point-based and full-hand coverage techniques, and optimization using a population-based incremental learning algorithm. Next, a 1D CNN classifier is used to identify hand motions. After a lot of trial and error, we were able to obtain a hand tracking accuracy of 83.71% and 85.71% over the Indian Sign Language and WLASL datasets, respectively. Our findings show how well our method works to recognize hand motions. Teachers, students, and medical professionals can all efficiently transmit and comprehend information by utilizing our suggested system. The obtained accuracy rates highlight how our method might improve communication and make information exchange easier in various domains.

A high-sensitivity and multi-response magnetic nanofiber-aerogel sensor with directionally aligned porous structure based on triple network for interactive human–machine interfaces

Flexible multimodal sensors with a porous structure have attracted tremendous attentions due to their low density, high specific surface area, a wide detection range and satisfied deformability. However, pores disorder and maldistribution issues of these multimodal sensors are major concerns during their sensing response, which often cause poor linearity and unstable sensing characteristics. Herein, a facile and rapid approach was proposed to fabricate a multifunctional magnetic sensor with directionally aligned porous structure based on triple network by combining electrospun nanofibers, bidisperse magnetic particles and sodium alginate/chitosan foam. The electrospun nanofibers were employed as three-dimensional skeletons for the crystallization and vertical growth of ice crystals. Benefiting from the remarkable structure orientation, the homogenized nanofiber-aerogel scaffold possesses excellent flexibility, superior deformation recoverability and biocompatibility. The assembled sensors not only exhibit a high sensitivity (0.40 T−1), rapid response/recovery time and long stability under magnetic stimuli, but also displays satisfied cross-sensitivity, quick response and reliable durability (8000 cycles) in response to external mechanical functions. Importantly, the respective input stimuli could be clearly discriminated via outputting the electrical signals of opposite or different trends. Furthermore, the sensor could be employed as wearable skins to monitor various physiological signals and realize information translation by Morse mode for individuals with disabilities. Additionally, a wearable gesture recognition system with assistance of the deep learning algorithm was testified, with high average recognition accuracy of 99 %. The simple fabrication process and prominent multifunctional characteristics of the proposed sensor endow it with an extensive application prospect in the field of interactive human–machine interfaces.

Implementation of Sibi Sign Language Realtime Detection Program (Case Studi At Sekolah Luar Biasa Negeri 1 Tabanan)

Indonesian deaf people utilize SIBI to communicate using spoken words, gestures, facial expressions, and body language. SIBI, certified for Special Schools (SLB), helps deaf pupils communicate. This project implements SIBI (Indonesian Sign Language System) a real-time detection algorithm at Sekolah Luar Biasa Negeri 1 Tabanan using image processing and YoloV8 ultralytics deep learning. The program trains a sign language gesture detection model on Google Colab's GPU. The SIBI sign language images were used to train a YoloV8 object detection model. The camera captures movements, which the YoloV8 algorithm trained on SIBI gesture data processes. It can recognize gestures in real time and generate text to non-sign language users. The dataset has 107 class vocabulary and 7 class affix prefixes for complete gesture recognition. Shirt color, room brightness, and webcam quality affect detection rates. Optimal detection accuracy is 87.74% and subpar 58.02%. Despite these limitations, the strategy helps deaf students communicate more effectively with non-sign language speakers. This program improves inclusivity and communication in schools, making learning easier for hearing-impaired pupils. This work provides a reliable and quick sign language identification system to help deaf educators and caregivers with daily interactions and education.

DEEP LEARNING-BASED HAND GESTURE RECOGNITION FOR SPEECH SYNTHESIS IN TELUGU

In a world increasingly reliant on technology, individuals had born with hearing impairments face significant communication challenges, leading to feelings of isolation and dependency. This paper addresses the pressing need to empower the deaf and mute community by proposing an innovative solution – Deep Learning-Based Hand Gesture Recognition for Speech Synthesis in Telugu. Deaf and mute individuals encounter barriers in expressing themselves verbally, hindering their integration into mainstream society. Conventional methods often fall short in providing effective communication channels, exacerbating the challenges faced by this community. The critical need for a comprehensive solution that facilitates seamless communication in their native language, Telugu is evident. Current sign language solutions lack precision, failing to capture nuanced gestures, limiting accuracy. Importantly, they overlook converting gestures into spoken Telugu, leaving a gap for the deaf and mute community. These systems also face real-time processing challenges, hindering natural communication in users native language. Our innovative approach utilizes advanced technology, specifically Deep Learning and a Convolutional Neural Network (CNN). This system significantly improves understanding of hand movements, achieving a remarkable 90% accuracy. When someone uses hand gestures, our system converts them into spoken Telugu, enhancing communication for deaf and mute individuals. This substantial improvement empowers native Telugu speakers by 80%, allowing them to express themselves more naturally and actively participate in daily life.

Computer Vision Algorithm Design in Image Processing Based on Projective Geometry

Image processing with computer vision, particularly in the realm of projective geometry, offers remarkable potential for various applications. Through the lens of projective geometry, images can be transformed, augmented, and reconstructed with precision, facilitating tasks such as image rectification, 3D reconstruction, and object tracking. Landmark estimation in computer vision is a vital task with broad applications across various domains. This process involves identifying key points or landmarks within images, enabling tasks such as facial recognition, object tracking, and gesture recognition. This paper, proposed a novel approach for landmark estimation in computer vision using Projective Geometry Landmark Estimation (PGLM). The proposed model aims to estimate the landmark features by a projective geometry model. With the estimation of the geometry features landmarks related to the facial, object, and medical images are computed. The PGLM model uses the point features for the location of the landmark features. In order to compare PGLM's performance to that of more conventional classification methods like Random Forest, K-Nearest Neighbors (KNN), and Support Vector Machine (SVM), simulation analysis is carried out. From what we can see, PGLM routinely beats these alternatives when we compare their accuracy, precision, recall, and F1 score. The findings stated the effectiveness of PGLM as a promising approach for landmark estimation in image processing tasks, paving the way for further advancements in this domain.

Large measurement range curvature sensor based on all-fiber Mach-Zehnder interferometer using triple-ring-core few-mode fiber

This study experimentally demonstrates a large measurement range curvature sensor based on a Mach-Zehnder interferometer (MZI) in a triple-ring-core fiber (TRCF). The sensor is fabricated by fusion splicing a segment of TRCF between two pieces of single-mode fiber (SMF), forming the SMF-TRCF-SMF sandwich structure. Since the TRCF can support the propagation of a few guided modes, the fundamental mode interferes with high-order modes in the sensing part to produce a periodic interference spectrum. Bending causes the effective refractive index of the mode to change, which shifts the interference spectrum. The bending measurement is achieved by monitoring the wavelength variation of interference dips. The proposed sensors can realize curvature measurements up to 100 m-1, with a bending sensitivity of -225.9 pm/m-1. It is believed that the proposed MZI sensor may have potential applications in robot hand gesture recognition and control.

ChatMatch: Exploring the potential of hybrid vision–language deep learning approach for the intelligent analysis and inference of racket sports

Video understanding technology has become increasingly important in various disciplines, yet current approaches have primarily focused on lower comprehension level of video content, posing challenges for providing comprehensive and professional insights at a higher comprehension level. Video analysis plays a crucial role in athlete training and strategy development in racket sports. This study aims to demonstrate an innovative and higher-level video comprehension framework (ChatMatch), which integrates computer vision technologies with the cutting-edge large language models (LLM) to enable intelligent analysis and inference of racket sports videos. To examine the feasibility of this framework, we deployed a prototype of ChatMatch in the badminton in this study. A vision-based encoder was first proposed to extract the meta-features included the locations, actions, gestures, and action results of players in each frame of racket match videos, followed by a rule-based decoding method to transform the extracted information in both structured knowledge and unstructured knowledge. A set of LLM-based agents included namely task identifier, coach agent, statistician agent, and video manager, was developed through a prompt engineering and driven by an automated mechanism. The automatic collaborative interaction among the agents enabled the provision of a comprehensive response to professional inquiries from users. The validation findings showed that our vision models had excellent performances in meta-feature extraction, achieving a location identification accuracy of 0.991, an action recognition accuracy of 0.902, and a gesture recognition accuracy of 0.950. Additionally, a total of 100 questions were gathered from four proficient badminton players and one coach to evaluate the performance of the LLM-based agents, and the outcomes obtained from ChatMatch exhibited commendable results across general inquiries, statistical queries, and video retrieval tasks. These findings highlight the potential of using this approach that can offer valuable insights for athletes and coaches while significantly improve the efficiency of sports video analysis.

Shape-position perceptive fusion electronic skin with autonomous learning for gesture interaction

Wearable devices, such as data gloves and electronic skins, can perceive human instructions, behaviors and even emotions by tracking a hand's motion, with the help of knowledge learning. The shape or position single-mode sensor in such devices often lacks comprehensive information to perceive interactive gestures. Meanwhile, the limited computing power of wearable applications restricts the multimode fusion of different sensing data and the deployment of deep learning networks. We propose a perceptive fusion electronic skin (PFES) with a bioinspired hierarchical structure that utilizes the magnetization state of a magnetostrictive alloy film to be sensitive to external strain or magnetic field. Installed at the joints of a hand, the PFES realizes perception of curvature (joint shape) and magnetism (joint position) information by mapping corresponding signals to the two-directional continuous distribution such that the two edges represent the contributions of curvature radius and magnetic field, respectively. By autonomously selecting knowledge closer to the user's hand movement characteristics, the reinforced knowledge distillation method is developed to learn and compress a teacher model for rapid deployment on wearable devices. The PFES integrating the autonomous learning algorithm can fuse curvature-magnetism dual information, ultimately achieving human machine interaction with gesture recognition and haptic feedback for cross-space perception and manipulation.

Quantized CNN-based efficient hardware architecture for real-time hand gesture recognition

Nowadays, Convolutional Neural Networks (CNN) have been widely adopted for vision-based hand gesture recognition. Several existing CNN architectures designed for gesture classification perform well with high accuracy but require a high memory footprint and processing when deployed on low-power embedded devices. To address this issue, we present a quantized CNN-based efficient framework for meeting real-life hand gesture recognition challenges. The proposed quantized CNN is designed and implemented using the FINN-based pipelined streaming architecture on an FPGA. Moreover, hardware-based optimizations are used to minimize the resources needed and to achieve fast memory access. Experimental results demonstrate that the developed recognition system achieves an average accuracy of 92% on the numeral database of Indian Sign Language (ISL). Additionally, our optimized design attains an inference latency of 0.85ms for real-time single gesture prediction on the PYNQ Zynq Ultrascale (ZU) FPGA, consuming only 3.63 W of power. The proposed design achieves a better trade-off between hardware resource utilization and speed performance, over previous designs.

Personalized Drone Interaction : Adaptive Hand Gesture Control with Facial Authentication

This paper presents a novel system for personalized drone interaction, integrating adaptive hand gesture control with facial authentication. Utilizing the DJI Tello drone equipped with a 5 MP camera, the system employs advanced computer vision and machine learning techniques to ensure secure and intuitive control. Facial recognition using the Histogram of Oriented Gradients (HOG) method and FaceNet model verifies user identity, while MediaPipe and a custom convolutional neural network (CNN) facilitate accurate hand gesture recognition. The system’s real-time processing capabilities ensure seamless and responsive user interaction. Experimental results demonstrate the system’s robustness and accuracy in various scenarios, highlighting its potential for diverse applications such as security, entertainment, and personal assistance.

Stretchable Iontronic Tactile Sensing Fabric.

The iontronic tactile sensing modality has garnered significant attention due to its exceptional sensitivity, immunity to noise, and versatility in materials. Recently, various formats of iontronic tactile sensors have been developed, including droplets, polymer films, paper, ionic gels, and fabrics. However, the stretchability of the current iontronic pressure sensing fabric is inadequate, hindered by the limited stretchiness of the ionic functional fabric. Incorporating a stretchable tactile sensing implement could enhance the wear comfortability by preventing relative movement and ensuring intimate contact between the sensor and the skin. The research focuses on the development of a stretchable iontronic pressure sensing (SIPS) fabric for monitoring diverse aspects of body health and movement in wearable applications. The tactile sensing structure is generated at the iontronic interface between highly stretchable ionic and conductive fabrics. In particular, the ionic fabric is prepared by coating a layer of polyurethane/ionic liquid gel onto a Spandex fabric. To showcase its remarkable sensitivity, stretchability, and ability to detect diverse body information, several application scenarios have been demonstrated including an elastic wristband for precise pulse wave detection, a flexible belt with multitactile sensing channels for respiration and motion tracking purposes, and a stretchable fabric cuff equipped with a high-resolution sensing array comprising 32 × 32 units for accurate gesture recognition.

Human–Robot Collaborative Manufacturing Cell with Learning-Based Interaction Abilities

This paper presents a collaborative manufacturing cell implemented in a laboratory setting, focusing on developing learning-based interaction abilities to enhance versatility and ease of use. The key components of the system include 3D real-time volumetric monitoring for safety, visual recognition of hand gestures for human-to-robot communication, classification of physical-contact-based interaction primitives during handover operations, and detection of hand–object interactions to anticipate human intentions. Due to the nature and complexity of perception, deep-learning-based techniques were used to enhance robustness and adaptability. The main components are integrated in a system containing multiple functionalities, coordinated through a dedicated state machine. This ensures appropriate actions and reactions based on events, enabling the execution of specific modules to complete a given multi-step task. An ROS-based architecture supports the software infrastructure among sensor interfacing, data processing, and robot and gripper controllers nodes. The result is demonstrated by a functional use case that involves multiple tasks and behaviors, paving the way for the deployment of more advanced collaborative cells in manufacturing contexts.