Gesture Recognition Applications Research Articles

Structure and Wiring Optimized TT/MT Double-Helical Fiber Sensors: Fabrication and Applications in Human Motion Monitoring and Gesture Recognition.

A fibrous flexible sensor, with its small size, minimally burdens the human body, ranking among the most user-friendly flexible sensors. However, its application is often limited by damage caused by electrode movement, as flexible sensors are typically attached to joints, which can be greatly alleviated by placing the two electrodes on the same side. Inspired by the hydrogen bonds in the double-helical structure of DNA, the double-helical electrode design is commonly found and applied in fiber-based batteries and supercapacitors into fibrous flexible sensors through coaxial wet-spinning and further treatment. The double helical sensor exhibits high strength and maintains stable operation and is prepared under over 300% strain with gauge factors (GF) of 0.9, 39.5, and 349, respectively, in its working ranges. This unique single-sided electrode structure also enabled applications such as water flow sensing. The sensor into a smart glove capable of real-time is further integrated, five-channel finger motion detection, and used a convolutional neural network (CNN)-based machine learning algorithm to achieve 98.8% accuracy in recognizing six common gestures. This study provides a novel approach to optimize the electrode distribution in fiber-based flexible sensors through an internally encapsulated double-helical structure, making a significant contribution to the field of flexible sensing.

Gesture recognition from surface electromyography signals based on the SE-DenseNet network.

In recent years, significant progress has been made in the research of gesture recognition using surface electromyography (sEMG) signals based on machine learning and deep learning techniques. The main motivation for sEMG gesture recognition research is to provide more natural, convenient, and personalized human-computer interaction, which makes research in this field have considerable application prospects in rehabilitation technology. However, the existing gesture recognition algorithms still need to be further improved in terms of global feature capture, model computational complexity, and generalizability. This paper proposes a fusion model of Squeeze-and-Excitation Networks (SE) and DenseNet, inserting attention mechanism between DenseBlock and Transition to focus on the most important information, improving feature representation ability, and effectively solving the problem of gradient vanishing. This proposed method was tested on the electromyographic gesture datasets NinaPro DB2 and DB4, achieving accuracies of 85.93 and 82.39 % respectively. Through ablation experiments, it was found that the method based on DenseNet-101 as the backbone model produced the best results. Compared with existing models, this proposed method has better robustness and generalizability in gesture recognition, providing new ideas for the development of sEMG signal gesture recognition applications in the future.

A Silver Nanowires‐Based Flexible Capacitive Touch Screen in Tactile Displays for Individuals with Visual Impairment Using Gesture Recognition

AbstractCapacitive touch screens (CTS's) are essential components in most of today's digital devices. However, for the visually impaired (VI) users due to the uneven topography of the tactile surface, CTS's are more challenging to implement and thus this field remains largely underdeveloped. Considering the limited space around the microactuators driving the typical Braille dots for a tactile screen with ten dots‐per‐inch (dpi) resolution, the materials used for CTS should be flexible and durable with high mechanical strength. In this work, a flexible CTS based on polyimide (PI) and silver nanowires (AgNWs) as electrodes with a total thickness of 210 µm is developed. The dimensions of the AgNWs are on average 7.9 ± 2.4 µm in length and 85 ± 24 nm in width. The AgNWs electrodes showed low resistance and good adhesion to the PI substrate. A gesture recognition application is collected from the capacitive data to classify different gestures (including single‐ and double‐click, swipe‐left and ‐right, scroll‐up and ‐down as well as zoom‐in and ‐out) with two different approaches; machine learning and deep learning are implemented. The best performance is obtained using the YOLO model with a high validation accuracy of 97.76%. Finally, a software application is developed with the proposed hand gestures in real‐time to foster interaction of VI users with the tactile display allowing them to navigate a Windows file system and interact with the documents via hand gestures in a similar manner as sighted users on a conventional touch display will be able to do. This work paves the way to utilize CTS for the tactile displays in the market developed for VI users.

Deep learning‐based gesture recognition for surgical applications: A data augmentation approach

AbstractHand gesture recognition and classification play a pivotal role in automating Human‐Computer Interaction (HCI) and have garnered substantial attention in research. In this study, the focus is placed on the application of gesture recognition in surgical settings to provide valuable feedback during medical training. A tool gesture classification system based on Deep Learning (DL) techniques is proposed, specifically employing a Long Short Term Memory (LSTM)‐based model with an attention mechanism. The research is structured in three key stages: data pre‐processing to eliminate outliers and smooth trajectories, addressing noise from surgical instrument data acquisition; data augmentation to overcome data scarcity by generating new trajectories through controlled spatial transformations; and the implementation and evaluation of the DL‐based classification strategy. The dataset used includes recordings from ten participants with varying surgical experience, covering three types of trajectories and involving both right and left arms. The proposed classifier, combined with the data augmentation strategy, is assessed for its effectiveness in classifying all acquired gestures. The performance of the proposed model is evaluated against other DL‐based methodologies commonly employed in surgical gesture classification. The results indicate that the proposed approach outperforms these benchmark methods, achieving higher classification accuracy and robustness in distinguishing diverse surgical gestures.

Large measurement range curvature sensor based on all-fiber Mach-Zehnder interferometer using triple-ring-core few-mode fiber.

This study experimentally demonstrates a large measurement range curvature sensor based on a Mach-Zehnder interferometer (MZI) in a triple-ring-core fiber (TRCF). The sensor is fabricated by fusion splicing a segment of TRCF between two pieces of single-mode fiber (SMF), forming the SMF-TRCF-SMF sandwich structure. Since the TRCF can support the propagation of a few guided modes, the fundamental mode interferes with high-order modes in the sensing part to produce a periodic interference spectrum. Bending causes the effective refractive index of the mode to change, which shifts the interference spectrum. The bending measurement is achieved by monitoring the wavelength variation of interference dips. The proposed sensors can realize curvature measurements up to 100 m-1, with a bending sensitivity of -225.9 pm/m-1. It is believed that the proposed MZI sensor may have potential applications in robot hand gesture recognition and control.

Bioinspired, highly sensitive interlocked flexible textile pressure sensor based on multilayer SWCNTs/PVP/rGO dendritic for gesture recognition

The development of wearable gesture recognition systems, capable of converting intricate human gestures into electric signals and graphical representations, poses significant challenges for future advancements in health monitoring and human-machine interfaces. In response to this demand, we present an approach centered on a cell-inspired, multiscale hierarchical structural piezoresistive pressure sensor designed for gesture recognition applications. The breathable, hierarchically interlocked textile pressure sensor (PFPA), featuring a SWCNTs/polyvinylpyrrolidone (PVP)/reduced graphene oxide (rGO) dendritic structure combined with a PANI/PVA cylindrical array, exhibits remarkable properties. The dendritic electrodes, composed of SWCNTs and PVP in a 1:2 ratio, maintain optimal regularity. Meanwhile, the cylindrical PANI/PVA array is meticulously engineered with a radius of 200 μm, a height of 200 μm, and an inter-cylindrical spacing of 150 μm. The PFPA demonstrates an unprecedented ultrahigh sensitivity of 80.8 kPa−1 across a pressure range of 0–5 kPa, coupled with a rapid response time of 150 ms and minimal hysteresis of 16.1 %. The PFPA also demonstrates outstanding repeatability, with no significant performance degradation observed after 4000 cycles. By correlating the bending angles of finger joints with resistance changes, we successfully integrated these sensors into a glove, enabling precise gesture recognition. The multilayer SWCNTs/PVP/rGO dendritic and PANI/PVA cylindrical array pressure sensor exhibits superior sensing performance and accurate posture capture. This demonstrates its significant potential for applications in wearable electronics and intelligent devices, marking a substantial advancement in the field.

Analysis of gesture recognition applications based on deep learning

With the advent of artificial intelligence, deep learning has been continuously evolving. It has progressed from its initial stages, exemplified by AlphaGo, to the development of related technologies exclusively centred around deep learning for the game of Go. Subsequently, deep learning has found applications in an increasing number of domains, including healthcare, sports, and more. This paper delves into the profound influence and development of deep learning as the foundation for analysing its impact on gesture recognition. Deep learning has brought about significant advancements in this field, enabling more precise and versatile gesture recognition systems. Furthermore, we explore specific use cases and contexts where deep learning has been harnessed, such as in medical diagnostics and sports performance analysis. While scrutinizing these applications, a promising future for deep learning becomes evident, with the potential to revolutionize various industries and enhance technology interaction through the development of more intuitive and sophisticated gesture recognition systems. The continuous growth and evolution of deep learning offer a bright prospect for the future of human-machine interfaces and artificial intelligence-driven solutions.

Hand Gesture Recognition System Using Deep Learning

Hand Gesture Recognition Systems have undergone significant advancements, ushering in a new era of human-computer interaction. This paper offers a thorough examination of the current state of the art in hand gesture recognition, addressing both the notable progress achieved and the persistent challenges. By leveraging state-of-the-art technologies such as computer vision and deep learning, the paper explores the methodologies employed in data collection, preprocessing, and the implementation of various algorithms. The research delves into the complexities of popular hand gesture datasets, emphasizing their role in training and testing models. A critical analysis of different algorithms and models, including Hidden Markov Models, Support Vector Machines, and Neural Networks, is presented. The paper scrutinizes their strengths and limitations, providing insights into the delicate balance between accuracy and real-time processing. Furthermore, it investigates the diverse applications of hand gesture recognition, spanning from enriching human-computer interaction to its pivotal role in virtual reality, gaming, and robotics. Despite these advancements, challenges persist, such as occlusion, varying lighting conditions, and the imperative for real-time processing. The hardware utilized in hand gesture recognition systems, including depth sensors, RGB-D cameras, and wearable devices, is examined. Evaluation metrics, such as accuracy, precision, recall, and the F1 score, are employed to evaluate system performance.

Web Real-Time Communication Using Gesture

Gesture recognition has emerged as a promising technology for enhancing human-computer interaction in various domains. In this paper, we propose a novel approach to gesture recognition using WebRTC (Web Real-Time Communication), a powerful web technology that enables real-time communication between web browsers. Our approach leverages the capabilities of WebRTC to capture live video streams from webcams and process them in real-time for gesture recognition. We present a detailed methodology for integrating WebRTC with a custom deep learning techniques. We evaluate the performance of our system using a diverse set of hand gestures and demonstrate its effectiveness in real-world scenarios. Our experimental results show an average recognition accuracy of 90%, outperforming existing methods in terms of both accuracy and computational efficiency. Potential applications of our approach in interactive web applications, gaming, and augmented reality systems. This work contributes the research on gesture recognition and demonstrates the feasibility of using WebRTC for real-time gesture recognition applications. Key Words: gesture recognition, WebRTC , Real-time communication

Development of Basic Finger Shapes Indonesian Traditional Dance Application Using Color-Based Hand Gesture Recognition

UNESCO stated that Indonesian traditional dance is a world cultural heritage that must be preserved and maintained. Efforts made by the government to preserve traditional dance are still not successful. Research shows that students have difficulty in learning traditional Indonesian dance exercises. Therefore, this study develops a solution that focuses on the difficulty of learning the basic finger postures in traditional Indonesian dances. This is important because fingers are instruments in dancing that have meaningful forms, such as Closed Urang Chopsticks, Nyempurit, Ngruji, Open Urang Chopsticks, Ke-pelan, Baya Mangap, and Ngithing. Learning the practice of basic finger postures aims to provide an independent learning experience through repetitive movements to strengthen or strengthen. This research has succeeded in making a real-time Color-Based Hand Gesture Recognition application for reliable Android devices. This is evidenced by applications that can function well in test scenarios with varying lighting and background conditions. Finally, students can learn the hand gestures of traditional Indonesian dances independently with ease. In addition, the application of the results of this research also contributes to becoming a learning medium in the current era of Blended Learning. Suggestions for future research are to increase the accuracy of Nyempurit and Ngithing attitudes.

3D Motion Gesture Control : Gesture Recognition and Adaptation for Human Computer Interaction

Advancements in human-computer interaction (HCI) have paved the way for more intuitive and immersive interfaces. The first part of the paper delves into the fundamental principles of 3D gesture recognition, including sensor technologies, machine learning algorithms, and computer vision techniques. It discusses the challenges associated with accurate recognition in various environmental conditions and the ways in which these challenges are being addressed by researchers. The second part focuses on the adaptation aspect of the technology. It highlights how 3D gesture recognition can be integrated into adaptive HCI systems, enabling personalized and context-aware interactions. These adaptations can range from adjusting the interface layout to suit the user's preferences to dynamically changing the system's behavior based on the user's gestures. Additionally, the paper discusses the potential applications of 3D gesture recognition in fields such as gaming, virtual reality, healthcare, and beyond. It emphasizes the need for continued research to improve accuracy, robustness, and user-friendliness, ultimately driving the widespread adoption of 3D gesture recognition in HCI.

EchoGest: Soft Ultrasonic Waveguides Based Sensing Skin for Subject-Independent Hand Gesture Recognition.

Gesture recognition is crucial for enhancing human-computer interaction and is particularly pivotal in rehabilitation contexts, aiding individuals recovering from physical impairments and significantly improving their mobility and interactive capabilities. However, current wearable hand gesture recognition approaches are often limited in detection performance, wearability, and generalization. We thus introduce EchoGest, a novel hand gesture recognition system based on soft, stretchable, transparent artificial skin with integrated ultrasonic waveguides. Our presented system is the first to use soft ultrasonic waveguides for hand gesture recognition. EcoflexTM 00-31 and EcoflexTM 00-45 Near ClearTM silicone elastomers were employed to fabricate the artificial skin and ultrasonic waveguides, while 0.1 mm diameter silver-plated copper wires connected the transducers in the waveguides to the electrical system. The wires are enclosed within an additional elastomer layer, achieving a sensing skin with a total thickness of around 500 μ m. Ten participants wore the EchoGest system and performed static hand gestures from two gesture sets: 8 daily life gestures and 10 American Sign Language (ASL) digits 0-9. Leave-One-Subject-Out Cross-Validation analysis demonstrated accuracies of 91.13% for daily life gestures and 88.5% for ASL gestures. The EchoGest system has significant potential in rehabilitation, particularly for tracking and evaluating hand mobility, which could substantially reduce the workload of therapists in both clinical and home-based settings. Integrating this technology could revolutionize hand gesture recognition applications, from real-time sign language translation to innovative rehabilitation techniques.

User Training With Error Augmentation for sEMG-Based Gesture Classification.

We designed and tested a system for real-time control of a user interface by extracting surface electromyographic (sEMG) activity from eight electrodes in a wristband configuration. sEMG data were streamed into a machine-learning algorithm that classified hand gestures in real-time. After an initial model calibration, participants were presented with one of three types of feedback during a human-learning stage: veridical feedback, in which predicted probabilities from the gesture classification algorithm were displayed without alteration; modified feedback, in which we applied a hidden augmentation of error to these probabilities; and no feedback. User performance was then evaluated in a series of minigames, in which subjects were required to use eight gestures to manipulate their game avatar to complete a task. Experimental results indicated that relative to the baseline, the modified feedback condition led to significantly improved accuracy. Class separation also improved, though this trend was not significant. These findings suggest that real-time feedback in a gamified user interface with manipulation of feedback may enable intuitive, rapid, and accurate task acquisition for sEMG-based gesture recognition applications.

GESTURE RECOGNITION FOR INTERACTIVE DASHBOARD: A 3D CONVOLUTIONAL NEURAL NETWORKS WITH LONG-SHORT-TERM-MEMORY APPROACH

Gesture recognition technology has become increasingly popular in various fields due to its potential for controlling consumer devices, robotics, and even translating sign language. However, there is still a lack of practical applications in daily life, especially for consumer products. Previous studies primarily focused on gesture recognition and did not delve into exploring the potential applications of gesture recognition in human interaction. This work aims to address this gap by deploying a gesture recognition model into practical applications and exploring its potential in human interaction. The proposed model combines the use of 3D convolutional neural networks (3DCNN) and long-short-term-memory networks (LSTM) for spatiotemporal feature learning. Six common interactive dashboard gestures were extracted from the 20BN-Jester dataset to train the model. The validation accuracy reached 80.47% after ablation studies. Then, two test cases were conducted to evaluate the effectiveness of the model in controlling Spotify and the Artificial Intelligence of Things (AIoT) smart classroom. The response time for each recognition was approximately 100 to 150 milliseconds. The proposed research aims to develop a simple yet efficient model for hand gesture recognition, which has the potential to be applied in various fields beyond just music and smart classrooms. With the growing popularity of gesture recognition technology, this study contributes to the advancement of practical applications and product innovations that can be integrated into daily life, thereby making it more accessible to the general public.

Interactive projection information generation and feedback for reading behaviour

Interactive reading is a new reading behavior in the age of intelligence. The dual-track development of new media and interactive technology has led to changes in both reading behavior and reading experience. This paper has discussed the information generation and information feedback of interactive reading, and analysed the important applications of projection technology, gesture recognition and QR code technology in interactive reading behavior from various projection interaction technologies. Through the development process of the interactive projection of the atlas series named “City Flowers of Beijing”, we aim to demonstrate the design of interactive and sustainable reading forms, deliver the multi-dimensional, multi-level and diversified reading information, create a reading experience that is both physiologically and psychologically satisfactory to the audience in the age of intelligence, and provide a new direction for the development of human-computer reading behavior in the new media age.

AI Cricket Score Board

Gesture Recognition pertains to recognizing meaningful expressions of motion by a human, involving the hands, arms, face, head, and/or body. It is of utmost important in designing an intelligent and efficient human–computer interface. The applications of gesture recognition are manifold, ranging from sign language through medical rehabilitation, monitoring patients or elder people, surveillance systems, sports gesture analysis, human behavior analysis etc., to virtual reality. In recent years, there has been increased interest in video summarization and automatic sports highlights generation in the game of Cricket. In Cricket, the Umpire has the authority to make important decisions about events on the field. The Umpire signals important events using unique hand on signals and gestures. The primary intention of our work is to design and develop a new robust method for Umpire Action and Non-Action Gesture Identification and Recognition based on the Umpire Segmentation and the proposed Histogram Oriented Gradient (HOG) feature Extraction oriented of Deep Features. Primarily the 80% of Umpire action and non-action images in a cricket match, about 1, 93,000 frames, the Histogram of Oriented Gradient Deep Features are calculated and trained the system having six gestures of Umpire pose using Densenet121.

Laser-Formed Sensors with Electrically Conductive MWCNT Networks for Gesture Recognition Applications.

Currently, an urgent need in the field of wearable electronics is the development of flexible sensors that can be attached to the human body to monitor various physiological indicators and movements. In this work, we propose a method for forming an electrically conductive network of multi-walled carbon nanotubes (MWCNT) in a matrix of silicone elastomer to make stretchable sensors sensitive to mechanical strain. The electrical conductivity and sensitivity characteristics of the sensor were improved by using laser exposure, through the effect of forming strong carbon nanotube (CNT) networks. The initial electrical resistance of the sensors obtained using laser technology was ~3 kOhm (in the absence of deformation) at a low concentration of nanotubes of 3 wt% in composition. For comparison, in a similar manufacturing process, but without laser exposure, the active material had significantly higher values of electrical resistance, which was ~19 kOhm in this case. The laser-fabricated sensors have a high tensile sensitivity (gauge factor ~10), linearity of >0.97, a low hysteresis of 2.4%, tensile strength of 963 kPa, and a fast strain response of 1 ms. The low Young's modulus values of ~47 kPa and the high electrical and sensitivity characteristics of the sensors made it possible to fabricate a smart gesture recognition sensor system based on them, with a recognition accuracy of ~94%. Data reading and visualization were performed using the developed electronic unit based on the ATXMEGA8E5-AU microcontroller and software. The obtained results open great prospects for the application of flexible CNT sensors in intelligent wearable devices (IWDs) for medical and industrial applications.

Flexible Strain Sensor-Based Data Glove for Gesture Interaction in the Metaverse: A Review

The rise of the metaverse concept has brought about widespread attention in wearable gesture recognition devices. Data gloves based on flexible strain sensors have been favoured by researchers owing to their low cost, light weight, direct and continuous monitoring of finger movements. In this review, we first compare the advantages and disadvantages of four different approaches based on the vision sensors, myoelectric sensors, inertial and magnetic sensors, and flexible strain sensors in designing data gloves, and demonstrate the superiority of the flexible strain sensor-based data glove used for metaverse applications. Next, some latest commercial data gloves are exampled and the function modules of the data gloves are presented based on the flexible strain sensors. Meanwhile, the potential applications of gesture recognition in the metaverse are summarized in diversified fields. Finally, the existing problems and development prospects of the current data gloves based on flexible strain sensors are concluded. We are optimistic that novel flexible strain sensor-based data gloves will make transformational impact to realize accurate, low-latency, and immersive gesture interaction in the metaverse.

TripCEAiR: A multi-loss minimization approach for surface EMG based airwriting recognition

Airwriting Recognition refers to the problem of identification of letters written in space with movement of the finger. It can be seen as a special case of dynamic gesture recognition wherein the set of gestures are letters in a particular language. Surface Electromyography (sEMG) is a non-invasive approach used to capture electrical signals generated as a result of contraction and relaxation of the muscles. sEMG has been widely adopted for gesture recognition applications. Unlike static gestures, dynamic gestures are user-friendly and can be used as a method for input with applications in Human Computer Interaction. There has been limited work in recognition of dynamic gestures such as airwriting, using sEMG signals and forms the core of the current work. In this work, a multi-loss minimization framework for sEMG based airwriting recognition is proposed. The proposed framework aims at learning a feature embedding vector that minimizes the triplet loss, while simultaneously learning the parameters of a classifier head to recognize corresponding alphabets. The proposed method is validated on a dataset recorded in the lab comprising of sEMG signals from 50 participants writing English uppercase alphabets. The effect of different variations of triplet loss, triplet mining strategies and feature embedding dimension is also presented. The best-achieved accuracy was 81.26% and 65.62% in user-dependent and independent scenarios respectively by using semihard positive and hard negative triplet mining. The code for our implementation will be made available at https://github.com/ayushayt/TripCEAiR

SEMG signal filtering study using synchrosqueezing wavelet transform with differential evolution optimized threshold

Most gesture recognition studies based on surface electromyography (sEMG) signals focus on filtering, in which the lack of diversity for considered noises can still be the problem. In this work, a denoising method based on Synchrosqueezing Wavelet Transform with Differential Evolution optimized threshold (DEOT-SWT) is proposed. The sEMG signals of ten gestures with three mixed noises, including power line interference (PLI), baseline drift (BW), and white Gaussian noise (WGN), are firstly investigated and filtered by DEOT-SWT, which are collected from seven subjects by utilizing two wearable sEMG signal sensors. Then, the most commonly used Hudgins time-domain feature set is extracted for recognizing ten gestures. Three metrics are adopted to evaluate filtering performance: signal-to-noise ratio (SNR), root mean square error and R-squared value. The gesture recognition accuracy is utilized to verify the practical effect of DEOT-SWT in sEMG-based gesture recognition applications. The results of the experiments demonstrate that the DEOT-SWT algorithm accomplishes desirable denoising performance with an average recognition accuracy of 95.95% (±3.88) in comparison to the classic Infinite Impulse Response (IIR) algorithm and the empirical mode decomposition (EMD) algorithm.