Gesture Control Research Articles

Comparing feature usage in IMU-based gesture control for omnidirectional robot via wearable glove

<span>To improve the intuitiveness of maneuver control on omniwheeled mobile robot, many hand gesture-based robot controls have been developed. The focus of this research is to develop a wearable system for data acquisition from inertial measurement unit (IMU) sensors and compare its features to be used as gesture recognition using the random forest algorithm. With the need of resource constrained device for wearable system based on microcontrollers, we compared the use of Euler and quaternion-based orientation data as input features. As additional comparison, dimension reduction was also carried out using the principal component analysis (PCA) method. Hand gestures are recognized using data obtained by the IMU sensor embedded in the wearable glove. This study compared the accuracy and size of library files embedded in microcontrollers in several feature usage scenarios. The test evaluation results of all scenarios show that the use of all features provides a balance between high accuracy but small file sizes, respectively 99% and 9.2 KB. However, the use of other fewer features, such as by only using 3 Euler data, 4 quaternion data, or by using PCA algorithm (PC=3) can also be used since the accuracy is still above 90%, with a relatively larger file size.</span>

Wi-Fi sensing gesture control algorithm based on semi-supervised generative adversarial network

A Wi-Fi-sensing gesture control system for smart homes has been developed based on a theoretical investigation of the Fresnel region sensing model, addressing the need for non-contact gesture control in household environments. The system collects channel state information (CSI) related to gestures from Wi-Fi signals transmitted and received by network cards within a specific area. The collected data undergoes preprocessing to eliminate environmental interference, allowing for the extraction of complete gesture sets. Dynamic feature extraction is then performed, followed by the identification of unknown gestures using pattern recognition techniques. An improved dynamic double threshold gesture interception algorithm is introduced, achieving a gesture interception accuracy of 98.20%. Furthermore, dynamic feature extraction is enhanced using the Gramian Angular Summation Field (GASF) transform, which converts CSI data into GASF graphs for more effective gesture recognition. An enhanced generative adversarial network (GAN) algorithm with an embedded classifier is employed to classify unknown gestures, enabling the simultaneous recognition of multiple gestures. A semi-supervised learning algorithm designed to perform well even with limited labeled data demonstrates high performance in cross-scene gesture recognition. Compared to traditional fully-supervised algorithms like linear discriminant analysis (LDA), Light Gradient Boosting Machine (LightGBM), and support vector machine (SVM), the semi-supervised GAN algorithm achieves an average accuracy of 95.67%, significantly outperforming LDA (58.20%), LightGBM (78.20%), and SVM (75.67%). In conclusion, this novel algorithm maintains an accuracy of over 94% across various scenarios, offering both faster training times and superior accuracy, even with minimal labeled data.

A low-cost prosthetic arm with real-time haptic feedback and gesture control for enhanced user autonomy

In daily life, manipulating and feeling objects is essential for autonomy and engagement in various activities. However, individuals relying on prosthetic limbs often encounter barriers such as high costs, ineffective feedback, and limited design adaptability. Unlike conventional prosthetic limbs, which frequently struggle to integrate seamlessly and provide intuitive control, the robotic arm outlined in this study represents a leap in functionality and user experience. At the core of its design lies a specialized glove outfitted with a calibrated array of flex-sensing resistors capable of precisely detecting and interpreting the wearer's movements. These sensors input to the robotic arm, translating the user's gestures into motions that follow human hands. The fingertips are equipped with force-sensing resistors to discern the magnitude of force applied during gripping actions. This real-time sensory feedback is conveyed to the user through a haptic system integrated into the glove. By modulating the intensity and pattern of vibrations, users can gauge their grip strength and manipulate their grasp of the object accordingly. Comprehensive experiments were conducted across various scenarios to assess its performance, agility, strength, and sensory accuracy. Through refinement, the author aims to optimize functionality and reliability, enhancing practicality for users. This project has broader implications for the prosthetics community. By utilizing the scalability and cost-effectiveness of this model, the author envisions tailored solutions that empower individuals with limb differences to live more independently.

Towards multisensory control of physical modeling synthesis

Physical models of musical instruments offer an interesting tradeoff between computational efficiency and perceptual fidelity. Yet, they depend on a multidimensional space of user-defined parameters whose exploration by trial and error is impractical. Our article addresses this issue by combining two ideas: query by example and gestural control. Prior publications have presented these ideas separately but never in conjunction. On one hand, we train a deep neural network to identify the resonator parameters of a percussion synthesizer from a single audio example via an original method named perceptual-neural-physical sound matching (PNP). On the other hand, we map these parameters to knobs in a digital controller and configure a musical touchpad with MIDI polyphonic expression. Hence, we propose a multisensory interface between human and machine: it integrates haptic and sonic information and produces new sounds in real time as well as visual feedback on the percussive touchpad. We demonstrate the interest of this new kind of multisensory control via a musical game in which participants collaborate with the machine in order to imitate the sound of an unknown percussive instrument as quickly as possible. Our findings show the challenge and promise of future research in musical "Human-AI parternships".

Analysis of Factors Influencing the Precision of Body Tracking Outcomes in Industrial Gesture Control.

The body tracking systems on the current market offer a wide range of options for tracking the movements of objects, people, or extremities. The precision of this technology is often limited and determines its field of application. This work aimed to identify relevant technical and environmental factors that influence the performance of body tracking in industrial environments. The influence of light intensity, range of motion, speed of movement and direction of hand movement was analyzed individually and in combination. The hand movement of a test person was recorded with an Azure Kinect at a distance of 1.3 m. The joints in the center of the hand showed the highest accuracy compared to other joints. The best results were achieved at a luminous intensity of 500 lx, and movements in the x-axis direction were more precise than in the other directions. The greatest inaccuracy was found in the z-axis direction. A larger range of motion resulted in higher inaccuracy, with the lowest data scatter at a 100 mm range of motion. No significant difference was found at hand velocity of 370 mm/s, 670 mm/s and 1140 mm/s. This study emphasizes the potential of RGB-D camera technology for gesture control of industrial robots in industrial environments to increase efficiency and ease of use.

Sensitivity to Vibrotactile Stimulation in the Hand and Wrist: Effects of Motion, Temporal Patterns, and Biological Sex.

We investigated the impact of low-tempo, repetitive hand movements on vibrotactile sensitivity by employing various temporal and spatial patterns in the hand and wrist area. The investigation of a human's ability to perceive vibrotactile stimuli during dynamic hand movements remains understudied, despite the prevalence of slow to mild hand motions in applications such as hand navigation or gesture control using haptic gloves in Virtual Reality (VR) and Augmented Reality (AR). We investigated vibrotactile sensitivity, analyzing the impact of various factors, including Motion (static and low-tempo repetitive hand movements), Temporal Patterns (Single or Double vibrations with varying onset times), Tactor Placements (hand and wrist), Spatial Patterns, and Biological Sex. Our study revealed that Motion significantly influences vibrotactile sensitivity in the hand and wrist areas, leading to reduced accuracy rates during dynamic conditions. Additionally, as the stimulus onset approached in Double vibrations, accuracy rates markedly decreased. Notably, Hand Placement resulted in significantly higher accuracy rates compared to the Wrist Placement. Our findings underscore the impact of motion in reducing vibrotactile sensitivity on the back of the hand and around the wrist. This research has wide-ranging practical applications, particularly in the field of VR/AR experiences, rehabilitation programs, and accessibility solutions through the use of haptic gloves. Insights from our study can be harnessed to enhance the efficacy of haptic gloves in conveying vibrotactile cues within these contexts.

Multiple Self-Powered Sensor-Integrated Mobile Manipulator for Intelligent Environment Detection.

A multiple self-powered sensor-integrated mobile manipulator (MSIMM) system was proposed to address challenges in existing exploration devices, such as the need for a constant energy supply, limited variety of sensed information, and difficult human-computer interfaces. The MSIMM system integrates triboelectric nanogenerator (TENG)-based self-powered sensors, a bionic manipulator, and wireless gesture control, enhancing sensor data usability through machine learning. Specifically, the system includes a tracked vehicle platform carrying the manipulator and electronics, including a storage battery and a microcontroller unit (MCU). An integrated sensor glove and terminal application (APP) enable intuitive manipulator control, improving human-computer interaction. The system responds to and analyzes various environmental stimuli, including the droplet and fall height, temperature, pressure, material type, angles, angular velocity direction, and acceleration amplitude and direction. The manipulator, fabricated using 3D printing technology, integrates multiple sensors that generate electrical signals through the triboelectric effect of mechanical motion. These signals are classified using convolutional neural networks for accurate environmental monitoring. Our database shows signal recognition and classification accuracy exceeding 94%, with specific accuracies of 100% for pressure sensors, 99.55% for angle sensors, and 98.66, 95.91, 96.27, and 94.13% for material, droplet, temperature, and acceleration sensors, respectively.

Gesture recognition technology: a new dimension in human-computer interaction interface

Development of an interface for intelligent gesture control to improve user experience and increase the efficiency of interaction with a computer. This paper proposes a gesture recognition system based on artificial intelligence using convolutional neural networks (CNN). The system comprises three stages: pre-processing, optimal frame determination, and gesture category identification. The extracted features used are independent of movement, scaling, and rotation, providing greater flexibility to the system. The suggested gesture control technology, known as Kazakh Sign Language (KSL) for Kazakh alphabets, eliminates the need for additional devices, enabling users to interact with the system naturally. Experiments demonstrated that the proposed KSL system can accurately recognize Kazakh language alphabet letters with a high precision of 97.3%, owing to the utilization of artificial intelligence and CNN to enhance the accuracy and effectiveness of gesture control. Gestures, a type of visual formation, are perceivable by computers through machine learning models. The selection of methods and systems for recognizing Kazakh sign language gestures was accompanied by addressing various challenges related to language-specific orthographic and gestural features. The developed gesture control interface for human-computer interaction is applied in the field of inclusive education, aiming to assist deaf and hard-of-hearing children in learning sign language.

Large measurement range curvature sensor based on all-fiber Mach-Zehnder interferometer using triple-ring-core few-mode fiber.

This study experimentally demonstrates a large measurement range curvature sensor based on a Mach-Zehnder interferometer (MZI) in a triple-ring-core fiber (TRCF). The sensor is fabricated by fusion splicing a segment of TRCF between two pieces of single-mode fiber (SMF), forming the SMF-TRCF-SMF sandwich structure. Since the TRCF can support the propagation of a few guided modes, the fundamental mode interferes with high-order modes in the sensing part to produce a periodic interference spectrum. Bending causes the effective refractive index of the mode to change, which shifts the interference spectrum. The bending measurement is achieved by monitoring the wavelength variation of interference dips. The proposed sensors can realize curvature measurements up to 100 m-1, with a bending sensitivity of -225.9 pm/m-1. It is believed that the proposed MZI sensor may have potential applications in robot hand gesture recognition and control.

Towards overcoming barriers to the clinical deployment of mixed reality image-guided navigation systems supporting percutaneous ablation of liver focal lesions

In recent years, we have observed a rise in the popularity of minimally invasive procedures for treating liver tumours, with percutaneous thermoablation being one of them, conducted using image-guided navigation systems with mixed reality technology. However, the application of this method requires adequate training in using the employed system. In our study, we assessed which skills pose the greatest challenges in performing such procedures. The article proposes a training module characterized by an innovative approach: the possibility of practicing the diagnosis, planning, execution stages and the physical possibility of performing the execution stage on the radiological phantom of the abdominal cavity. The proposed approach was evaluated by designing a set of 4 exercises corresponding to the 3 phases mentioned. To the research group included 10 radiologists and 5 residents in the study. Based on 20 clinical cases of liver tumors subjected to percutaneous thermoablation, we developed assessment tasks evaluating four skill categories: head-mounted display (HMD), ultrasound (US)/computed tomography (CT) image fusion interpretation, tracking system use, and the ability to insert a needle. The results were presented using the Likert scale. The results of our study indicate that the most challenging aspect for radiology specialists is adapting to HMD gesture control, while residents point to intraoperative images of fusion and respiratory movements in the liver as the most problematic. In terms of improving the ability to perform procedures on new patients, the module also allows you to create a new hologram for a different clinical case.

Personalized Drone Interaction : Adaptive Hand Gesture Control with Facial Authentication

This paper presents a novel system for personalized drone interaction, integrating adaptive hand gesture control with facial authentication. Utilizing the DJI Tello drone equipped with a 5 MP camera, the system employs advanced computer vision and machine learning techniques to ensure secure and intuitive control. Facial recognition using the Histogram of Oriented Gradients (HOG) method and FaceNet model verifies user identity, while MediaPipe and a custom convolutional neural network (CNN) facilitate accurate hand gesture recognition. The system’s real-time processing capabilities ensure seamless and responsive user interaction. Experimental results demonstrate the system’s robustness and accuracy in various scenarios, highlighting its potential for diverse applications such as security, entertainment, and personal assistance.

Self Operable E-health Diagnosis and Monitoring System with Gesture Control and Voice Assistance

In the present timeline, technology has advanced significantly, and the healthcare sector is constantly expanding. People in rural areas who cannot access corporate hospitals postpone their health check-ups due to the difficulty of traveling long distances. The readily available and expensive nature of diagnostic technologies have prevented many rural residents from seeking clinical examinations, even in cases when they have health issues. Corporate hospitals often make people to wait for long periods of time for health reports and causing delays. These issues highlighted the need for a device which aimed at developing an Arduino based health diagnosis system To resolve this issue, we have introduced a device that offers a comprehensive health diagnosis and monitoring system that uses Arduino technology to store and display vital health information. The system is designed to analyze electrocardiograms (ECG), photoplethysmograms (PPG), body temperature, heart rate and stress levels in real time and then process this information to create meaningful health status assessments and store them in the database for further health assessments. Furthermore, these data could be displayed through web server or an application. Additionally, this system provides voice assistance for the condition of patient when it is critical which helps in early detection of health conditions. Hand gestures are incorporated into this device, which makes it distinctive. Users can control the display of health parameters and easily move through the system's features with simple hand gestures. Users can switch between different types of modes in the system (such as on/off and different operation) by moving their hand in specified patterns. Thus, this makes the system user-friendly so that everyone may use it.

Research on Intelligent Wheelchair Multimode Human–Computer Interaction and Assisted Driving Technology

The traditional wheelchair focuses on the “human-chair” motor function interaction to ensure the elderly and people with disabilities’ basic travel. For people with visual, hearing, physical disabilities, etc., the current wheelchairs show shortcomings in terms of accessibility and independent travel for this group. Therefore, this paper develops an intelligent wheelchair with multimodal human–computer interaction and autonomous navigation technology. Firstly, it researches the multimodal human–computer interaction technology of occupant gesture recognition, speech recognition, and head posture recognition and proposes a wheelchair control method of three-dimensional head posture mapping the two-dimensional plane. After testing, the average accuracy of the gesture, head posture and voice control modes of the motorized wheelchair proposed in this study reaches more than 95 percent. Secondly, the LiDAR-based smart wheelchair indoor autonomous navigation technology is investigated to realize the autonomous navigation of the wheelchair by constructing an environment map, using A* and DWA algorithms for global and local path planning, and adaptive Monte Carlo simulation algorithms for real-time localization. Experiments show that the position error of the wheelchair is within 10 cm, and the heading angle error is less than 5° during the autonomous navigation. The multimode human–computer interaction and assisted driving technology proposed in this study can partially compensate and replace the functional deficiencies of the disabled population and improve the quality of life of the elderly and disabled population.

Enhancing unmanned ground vehicle performance in SAR operations: integrated gesture-control and deep learning framework for optimised victim detection.

In this study, we address the critical need for enhanced situational awareness and victim detection capabilities in Search and Rescue (SAR) operations amidst disasters. Traditional unmanned ground vehicles (UGVs) often struggle in such chaotic environments due to their limited manoeuvrability and the challenge of distinguishing victims from debris. Recognising these gaps, our research introduces a novel technological framework that integrates advanced gesture-recognition with cutting-edge deep learning for camera-based victim identification, specifically designed to empower UGVs in disaster scenarios. At the core of our methodology is the development and implementation of the Meerkat Optimization Algorithm-Stacked Convolutional Neural Network-Bi-Long Short Term Memory-Gated Recurrent Unit (MOA-SConv-Bi-LSTM-GRU) model, which sets a new benchmark for hand gesture detection with its remarkable performance metrics: accuracy, precision, recall, and F1-score all approximately 0.9866. This model enables intuitive, real-time control of UGVs through hand gestures, allowing for precise navigation in confined and obstacle-ridden spaces, which is vital for effective SAR operations. Furthermore, we leverage the capabilities of the latest YOLOv8 deep learning model, trained on specialised datasets to accurately detect human victims under a wide range of challenging conditions, such as varying occlusions, lighting, and perspectives. Our comprehensive testing in simulated emergency scenarios validates the effectiveness of our integrated approach. The system demonstrated exceptional proficiency in navigating through obstructions and rapidly locating victims, even in environments with visual impairments like smoke, clutter, and poor lighting. Our study not only highlights the critical gaps in current SAR response capabilities but also offers a pioneering solution through a synergistic blend of gesture-based control, deep learning, and purpose-built robotics. The key findings underscore the potential of our integrated technological framework to significantly enhance UGV performance in disaster scenarios, thereby optimising life-saving outcomes when time is of the essence. This research paves the way for future advancements in SAR technology, with the promise of more efficient and reliable rescue operations in the face of disaster.

Overview of Gesture Control Technologies

The dynamic development and rapid updating of information and communication technologies (ICT) has created a suitable environment for gesture control of modern computer and robot systems. It is noted that gestures are a necessary component in the communication process. At the same time, it is possible to control the equipment without contact with the support of methods that allow to select and recognize hand gestures. Sufficient work has been done and achievements have been made in the direction of automation and gesture control of technology. The study notes that such systems are more flexible and can easily adapt to people's changing needs. Building a human-machine interface speeds up the communication process and expands the user's capabilities. Gesture control technologies for computer and robotic systems have great potential. A comparative analysis of the parameters of gesture-controlled technologies is shown in the table.

Topology of surface electromyogram signals: hand gesture decoding on Riemannian manifolds

Objective. Decoding gestures from the upper limb using noninvasive surface electromyogram (sEMG) signals is of keen interest for the rehabilitation of amputees, artificial supernumerary limb augmentation, gestural control of computers, and virtual/augmented realities. We show that sEMG signals recorded across an array of sensor electrodes in multiple spatial locations around the forearm evince a rich geometric pattern of global motor unit (MU) activity that can be leveraged to distinguish different hand gestures. Approach. We demonstrate a simple technique to analyze spatial patterns of muscle MU activity within a temporal window and show that distinct gestures can be classified in both supervised and unsupervised manners. Specifically, we construct symmetric positive definite covariance matrices to represent the spatial distribution of MU activity in a time window of interest, calculated as pairwise covariance of electrical signals measured across different electrodes. Main results. This allows us to understand and manipulate multivariate sEMG timeseries on a more natural subspace—the Riemannian manifold. Furthermore, it directly addresses signal variability across individuals and sessions, which remains a major challenge in the field. sEMG signals measured at a single electrode lack contextual information such as how various anatomical and physiological factors influence the signals and how their combined effect alters the evident interaction among neighboring muscles. Significance. As we show here, analyzing spatial patterns using covariance matrices on Riemannian manifolds allows us to robustly model complex interactions across spatially distributed MUs and provides a flexible and transparent framework to quantify differences in sEMG signals across individuals. The proposed method is novel in the study of sEMG signals and its performance exceeds the current benchmarks while being computationally efficient.

76‐3: Invited Paper: Integration of Sensing Technologies into MicroLED Displays

MicroIC enables display attributes unattainable from polycrystalline thin‐film transistors, and the complimentary metal‐oxide semiconductor (CMOS) microICs support a wide range of applications beyond driving pixels. Integrating sensors within displays can add value to products, reduce costs and improve performance versus conventional solutions. As evidenced by touch and display integration in mobile displays via TDDI ICs, in‐cell sensing can reduce cost and improve performance to levels that are not available with all display types. Capacitive touch, optical touch, 3D scanning, and proximity sensing can also benefit from the inherent coordination between sensors and display. We present multiple sensing architectures and demonstrations of integration within microLED displays and describe CMOS microIC circuits for touch sensing applications. With diverse heterogenous semiconductor technology tightly woven into displays, the potential for intuitive gesture controls, adaptive interfaces, and enhanced sensory feedback paves the way for more natural and effortless interactions between humans and machines.

Helping Hands for Handicaps

Abstract: We have more advanced wheelchairs in today's generation. it includes joystick wheelchair, head motion control wheelchair, gesture control wheelchair etc. But with the advance technology it is expensive too, which is not affordable for most of the middle-class families. In order to provide an advanced wheelchair in an affordable price, we have to use less components to reduce the cost so that a person can buy it with a price less than the today’s basic bikes price. To achieve this, we used some basic electronic components such as Arduino MINI and NANO, gyro sensor, Node MCU, Motors etc. To get extra security we integrated some calling and GPS location features too. The person sitting in the wheelchair can control the wheelchair through hand gestures. Additionally, whenever the person sitting in the wheelchair faces any problem, such as if the wheelchair falls, the caretaker will receive a notification and through the GPS location, the caretaker will be able to reach their location.

Implementation of Gesture-Based Virtual Keyboard and Mouse

The development of human-computer interaction has seen a major change in the last few years, moving in the direction of more user-friendly and effective interfaces. This tendency has prompted researchers to investigate new input techniques, especially in the field of gesture-based control systems. By utilizing both hand and eye gestures to create a gesture-based virtual mouse and keyboard, this paper introduces a novel method of human-computer interaction. The suggested system offers users a smooth and simple way to interact with computers by combining hand gesture recognition and eye-tracking technology. Users can control cursor movement and keyboard inputs without the use of physical peripherals by using the system to track their hand and eye movements and interpret them as commands. The incorporation of eye-tracking technology improves the system's accuracy and responsiveness, enabling more precise cursor control and interaction with elements displayed on the screen. Hand gesture recognition, on the other hand, gives users an ergonomic and natural way to input data, allowing them to carry out commands with straightforward motions. The architecture of the system combines real-time processing capabilities to guarantee low latency and high responsiveness with sophisticated machine-learning algorithms for gesture recognition. Additionally, users can define their gesture commands according to their needs and preferences because the system is flexible and customizable. The suggested gesture-based virtual mouse and keyboard system has undergone rigorous testing and evaluation, and the results show promising performance and usability in a range of scenarios and applications. This novel method of interacting with computers has the potential to improve user experience, accessibility, and productivity in a variety of computing contexts. Keywords: Human-Computer Interaction, Gesture Recognition, Eye-Tracking

Gesture speak: Hands-Free Computer Control with Hand Gestures and Voice Commands

In an era of advancing human-computer interaction, "GestureSpeak" emerges as a pioneering project that facilitates hands-free control over computing devices through intuitive hand gestures and voice commands. Leveraging the power of MediaPipe and OpenCV in Python, this innovative system enables users to seamlessly navigate their digital environments without the need for traditional input devices. By harnessing the capabilities of computer vision and machine learning, GestureSpeak interprets and responds to users' gestures and vocal instructions, opening new frontiers in accessibility and user experience. This abstract offers a glimpse into the transformative potential of GestureSpeak in revolutionizing the way we interact with computers, making technology more accessible and intuitive for all users. Advancements in human-computer interaction have led to the development of innovative projects that redefine how users interact with technology. This paper introduces a novel integration of two projects: GestureSpeak and VoiceRobot, which together enable intuitive and hands-free control over computing devices using hand gestures and voice commands. VoiceRobot complements GestureSpeak by adding voice command capabilities to the interaction model. Powered by speech recognition technology, VoiceRobot enables users to control devices and launch applications using natural language. With VoiceRobot, initiating the GestureSpeak project is as simple as speaking a command. The combined system offers unique benefits, including improved accessibility for users with mobility impairments and a more natural way to interact with computing devices. By integrating gesture recognition and voice commands, the project enhances user engagement and efficiency, paving the way for future advancements in human-computer interaction. Launching the Project with VoiceRobot:A notable feature of this integrated system is the ability to launch the GestureGenie project using VoiceRobot. By speaking a predefined command, such as "Activate GestureGenie," users can initiate the hand gesture control system effortlessly. This capability highlights the seamless integration of gesture recognition and voice command technologies, showcasing the project's versatility and user-friendly design. In conclusion, the integration of GestureGenie and VoiceRobot represents a significant advancement in human-computer interaction, offering a comprehensive solution for hands-free device control. This abstract provides insight into the combined capabilities of these projects and their potential impact on accessibility, user experience, and the future of interactive computing.