Set Of Gestures Research Articles

AI-based real-time hand gesture-controlled virtual mouse

ABSTRACT Hand gesture control technology is an emerging alternative to conventional input devices like keyboards and mice, enabling users to interact with computers through natural hand movements. Although, many researchers worked on gesture recognition and its challenges, yet wide range of gestures has not been addressed. In this direction, the author proposed a novel gesture-controlled virtual mouse system that uses hand gestures and hand tip detection with a webcam or internal camera for computer navigation that accurately recognizes various gestures, including cursor movement, left and right clicks, double clicks, scrolling, drag and drop, multiple item selection, volume control. To deal with this real-world problem, the authors proposed a solution for limited space situations and assist individuals with hand impairments who cannot use a physical mouse. Furthermore, it surpasses the limitations of traditional mice, such as mechanical wear and tear, adaptability to different environments, and restricted functionalities. By leveraging hand gestures and tip detection through a camera proposed system offers an efficient and adaptable approach to operating a PC mouse. The proposed methodology uses computer vision and AI for the accurate detection of gestures, even in challenging environments. It also offers a comprehensive set of gestures for various actions and functions.

HaCk: Hand Gesture Classification Using a Convolutional Neural Network and Generative Adversarial Network-Based Data Generation Model

According to global data on visual impairment from the World Health Organization in 2010, an estimated 285 million individuals, including 39 million who are blind, face visual impairments. These individuals use non-contact methods such as voice commands and hand gestures to interact with user interfaces. Recognizing the significance of hand gesture recognition for this vulnerable population and aiming to improve user usability, this study employs a Generative Adversarial Network (GAN) coupled with Convolutional Neural Network (CNN) techniques to generate a diverse set of hand gestures. Recognizing hand gestures using HaCk typically involves a two-step approach. First, the GAN is trained to generate synthetic hand gesture images, and then a separate CNN is employed to classify gestures in real-world data. The evaluation of HaCk is demonstrated through a comparative analysis using Leave-One-Out Cross-Validation (LOO CV) and Holdout Cross-Validation (Holdout CV) tests. These tests are crucial for assessing the model’s generalization, robustness, and suitability for practical applications. The experimental results reveal that the performance of HaCk surpasses that of other compared ML/DL models, including CNN, FTCNN, CDCGAN, GestureGAN, GGAN, MHG-CAN, and ASL models. Specifically, the improvement percentages for the LOO CV Test are 17.03%, 20.27%, 15.76%, 13.76%, 10.16%, 5.90%, and 15.90%, respectively. Similarly, for the Holdout CV Test, HaCk outperforms HU, ZM, GB, GB-ZM, GB-HU, CDCGAN, GestureGAN, GGAN, MHG-CAN, and ASL models, with improvement percentages of 56.87%, 15.91%, 13.97%, 24.81%, 23.52%, 17.72%, 15.72%, 12.12%, 7.94%, and 17.94%, respectively.

Retraction of Tian et al. (2018).

Reports the retraction of "Enacting rituals to improve self-control" by Allen Ding Tian, Juliana Schroeder, Gerald Häubl, Jane L. Risen, Michael I. Norton and Francesca Gino (Journal of Personality and Social Psychology, 2018[Jun], Vol 114[6], 851-876). This retraction follows from a 2023 review of the data reported in the article, which was conducted and reported to the journal by the authors. The authors, who agreed to the retraction, requested a correction after identifying discrepancies between the data analyzed for Study 1 and the data downloaded from Qualtrics. Fifteen participants' condition codes (i.e., control versus experimental) differed between the data reported originally and what participants in the Qualtrics dataset reported doing (e.g., control participants who reported engaging in a ritual). The remaining 69 participants did not present these discrepancies. The authors' reanalysis based on what participants reported doing invalidated the previous conclusion that enacting a ritual improved self-control as measured by food diaries. Specifically, the effect of what participants did on reported calorie consumption was not statistically significant (ritual participants M = 1,563.31, SD = 313.15; control participants M = 1,521.74, SD = 367.79, t[83] = 0.56, p = .576). The authors did not identify any discrepancies in the datafiles for Studies 2-16. The Study 1 participants were recruited at the University of Chicago gym under the supervision of Drs. Juliana Schroeder and Jane Risen. The statistical analyses reported in the article were conducted by Dr. Juliana Schroeder. A dataset containing the data with the original condition codes and the newly analyzed participants' reports in the Qualtrics data appears at https://osf.io/3fk2c. (The following abstract of the original article appeared in record 2018-22169-001.) Rituals are predefined sequences of actions characterized by rigidity and repetition. We propose that enacting ritualized actions can enhance subjective feelings of self-discipline, such that rituals can be harnessed to improve behavioral self-control. We test this hypothesis in 6 experiments. A field experiment showed that engaging in a pre-eating ritual over a 5-day period helped participants reduce calorie intake (Experiment 1). Pairing a ritual with healthy eating behavior increased the likelihood of choosing healthy food in a subsequent decision (Experiment 2), and enacting a ritual before a food choice (i.e., without being integrated into the consumption process) promoted the choice of healthy food over unhealthy food (Experiments 3a and 3b). The positive effect of rituals on self-control held even when a set of ritualized gestures were not explicitly labeled as a ritual, and in other domains of behavioral self-control (i.e., prosocial decision-making; Experiments 4 and 5). Furthermore, Experiments 3a, 3b, 4, and 5 provided evidence for the psychological process underlying the effectiveness of rituals: heightened feelings of self-discipline. Finally, Experiment 5 showed that the absence of a self-control conflict eliminated the effect of rituals on behavior, demonstrating that rituals affect behavioral self-control specifically because they alter responses to self-control conflicts. We conclude by briefly describing the results of a number of additional experiments examining rituals in other self-control domains. Our body of evidence suggests that rituals can have beneficial consequences for self-control. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Optimizing the performance of convolutional neural network for enhanced gesture recognition using sEMG

Deep neural networks (DNNs) have demonstrated higher performance results when compared to traditional approaches for implementing robust myoelectric control (MEC) systems. However, the delay induced by optimising a MEC remains a concern for real-time applications. As a result, an optimised DNN architecture based on fine-tuned hyperparameters is required. This study investigates the optimal configuration of convolutional neural network (CNN)-based MEC by proposing an effective data segmentation technique and a generalised set of hyperparameters. Firstly, two segmentation strategies (disjoint and overlap) and various segment and overlap sizes were studied to optimise segmentation parameters. Secondly, to address the challenge of optimising the hyperparameters of a DNN-based MEC system, the problem has been abstracted as an optimisation problem, and Bayesian optimisation has been used to solve it. From 20 healthy people, ten surface electromyography (sEMG) grasping movements abstracted from daily life were chosen as the target gesture set. With an ideal segment size of 200 ms and an overlap size of 80%, the results show that the overlap segmentation technique outperforms the disjoint segmentation technique (p-value < 0.05). In comparison to manual (12.76 ± 4.66), grid (0.10 ± 0.03), and random (0.12 ± 0.05) search hyperparameters optimisation strategies, the proposed optimisation technique resulted in a mean classification error rate (CER) of 0.08 ± 0.03 across all subjects. In addition, a generalised CNN architecture with an optimal set of hyperparameters is proposed. When tested separately on all individuals, the single generalised CNN architecture produced an overall CER of 0.09 ± 0.03. This study's significance lies in its contribution to the field of EMG signal processing by demonstrating the superiority of the overlap segmentation technique, optimizing CNN hyperparameters through Bayesian optimization, and offering practical insights for improving prosthetic control and human–computer interfaces.

Brave New GES World: A Systematic Literature Review of Gestures and Referents in Gesture Elicitation Studies

How do we determine highly effective and intuitive gesture sets for interactive systems tailored to end users’ preferences? A substantial body of knowledge is available on this topic, among which gesture elicitation studies stand out distinctively. In these studies, end users are invited to propose gestures for specific referents, which are the functions to control for an interactive system. The vast majority of gesture elicitation studies conclude with a consensus gesture set identified following a process of consensus or agreement analysis. However, the information about specific gesture sets determined for specific applications is scattered across a wide landscape of disconnected scientific publications, which poses challenges to researchers and practitioners to effectively harness this body of knowledge. To address this challenge, we conducted a systematic literature review and examined a corpus of N = 267 studies encompassing a total of 187,265 gestures elicited from 6,659 participants for 4,106 referents. To understand similarities in users’ gesture preferences within this extensive dataset, we analyzed a sample of 2,304 gestures extracted from the studies identified in our literature review. Our approach consisted of (i) identifying the context of use represented by end users, devices, platforms, and gesture sensing technology; (ii) categorizing the referents; (iii) classifying the gestures elicited for those referents; and (iv) cataloging the gestures based on their representation and implementation modalities. Drawing from the findings of this review, we propose guidelines for conducting future end-user gesture elicitation studies.

A versatile interaction framework for robot programming based on hand gestures and poses

This paper proposes a framework for industrial and collaborative robot programming based on the integration of hand gestures and poses. The framework allows operators to control the robot via both End-Effector (EE) and joint movements and to transfer compound shapes accurately to the robot. Seventeen hand gestures, which cover the position and orientation controls of the robotic EE and other auxiliary operations, are designed according to cognitive psychology. Gestures are classified by a deep neural network, which is pre-trained for two-hand pose estimation and fine-tuned on a custom dataset, achieving a test accuracy of 99%. The index finger’s pointing direction and the hand’s orientation are extracted via 3D hand pose estimation to indicate the robotic EE’s moving direction and orientation, respectively. The number of stretched fingers is detected via two-hand pose estimation to represent decimal digits for selecting robot joints and inputting numbers. Finally, we integrate these three manners seamlessly to form a programming framework. We conducted two interaction experiments. The reaction time of the proposed hand gestures in indicating randomly given instructions is significantly less than that of other gesture sets, such as American Sign Language (ASL). The accuracy of our method in compound shape reconstruction is much better than that of hand movement trajectory-based methods, and the operating time is comparable with that of teach pendants.

Joint Reconfiguration after Failure for Performing Emblematic Gestures in Humanoid Receptionist Robot.

This study proposed a strategy for a quick fault recovery response when an actuator failure problem occurred while a humanoid robot with 7-DOF anthropomorphic arms was performing a task with upper body motion. The objective of this study was to develop an algorithm for joint reconfiguration of the receptionist robot called Namo so that the robot can still perform a set of emblematic gestures if an actuator fails or is damaged. We proposed a gesture similarity measurement to be used as an objective function and used bio-inspired artificial intelligence methods, including a genetic algorithm, a bacteria foraging optimization algorithm, and an artificial bee colony, to determine good solutions for joint reconfiguration. When an actuator fails, the failed joint will be locked at the average angle calculated from all emblematic gestures. We used grid search to determine suitable parameter sets for each method before making a comparison of their performance. The results showed that bio-inspired artificial intelligence methods could successfully suggest reconfigured gestures after joint motor failure within 1 s. After 100 repetitions, BFOA and ABC returned the best-reconfigured gestures; there was no statistical difference. However, ABC yielded more reliable reconfigured gestures; there was significantly less interquartile range among the results than BFOA. The joint reconfiguration method was demonstrated for all possible joint failure conditions. The results showed that the proposed method could determine good reconfigured gestures under given time constraints; hence, it could be used for joint failure recovery in real applications.

Weakest link or strongest link?

Abstract Teachers often use gestures to connect representations of mathematical ideas. This research examined (1) whether such linking gestures help students understand connections among representations and (2) whether sets of gestures that include repeated handshapes and motions – termed gestural catchments – are particularly beneficial. Undergraduates viewed one of four video lessons connecting two representations of multiplication. In the control lesson, the instructor produced beat gestures that did not link the representations. In the link-only lesson, the instructor used gestures to link representations, but the gestures did not form a catchment. In the consistent-catchment lesson, the instructor highlighted corresponding elements of the two representations using identical gestures. In the inconsistent-catchment lesson, the instructor highlighted non-corresponding elements of the two representations using identical gestures. Participants who saw the lesson with the consistent catchment – which highlighted similarities between representations – were most likely to understand the novel representation and to report learning from the lesson.

Machine Learning-Enabled Intelligent Gesture Recognition and Communication System Using Printed Strain Sensors.

Gesture contains abundant and complicated information in daily life; as a consequence, gesture recognition attracts a wide range of application prospects and academic values as an important way of achieving human-machine interactions (HMIs). Here, we report an intelligent system consisting of a smart glove made by printed CNT-graphene/PDMS strain sensors. The smart glove shows excellent fitness, comfort, and lightness for human hands. Inspired by machine learning strategies, several objects and gestures can be well classified and implemented by a customized artificial neural network. Several data sets of different sign language gestures and object-grabbing gestures were established, and the result shows that the intelligent system can achieve an average accuracy of 97% and up to 99.4% for a number of gesture groups. Moreover, a robot hand is connected to this system, which is able to react to the motion of human hands with certain gestures where simple sign communication is achieved. These features provide a feasible practical application scheme for gesture recognition in HMIs.

Abacus Gestures

Designing an extensive set of mid-air gestures that are both easy to learn and perform quickly presents a significant challenge. Further complicating this challenge is achieving high-accuracy detection of such gestures using commonly available hardware, like a 2D commodity camera. Previous work often proposed smaller, application-specific gesture sets, requiring specialized hardware and struggling with adaptability across diverse environments. Addressing these limitations, this paper introduces Abacus Gestures, a comprehensive collection of 100 mid-air gestures. Drawing on the metaphor of Finger Abacus counting, gestures are formed from various combinations of open and closed fingers, each assigned different values. We developed an algorithm using an off-the-shelf computer vision library capable of detecting these gestures from a 2D commodity camera feed with an accuracy exceeding 98% for palms facing the camera and 95% for palms facing the body. We assessed the detection accuracy, ease of learning, and usability of these gestures in a user study involving 20 participants. The study found that participants could learn Abacus Gestures within five minutes after executing just 15 gestures and could recall them after a four-month interval. Additionally, most participants developed motor memory for these gestures after performing 100 gestures. Most of the gestures were easy to execute with the designated finger combinations, and the flexibility in executing the gestures using multiple finger combinations further enhanced the usability. Based on these findings, we created a taxonomy that categorizes Abacus Gestures into five groups based on motor memory development and three difficulty levels according to their ease of execution. Finally, we provided design guidelines and proposed potential use cases for Abacus Gestures in the realm of mid-air interaction.

An Empirical Comparison of Moderated and Unmoderated Gesture Elicitation Studies on Soft Surfaces and Objects for Smart Home Control

Conducting gesture elicitation studies (GES) in personal spaces such as smart homes is crucial to achieving high ecological validity of elicited gestures. However, supervising such studies is considered intrusive and negatively affects the results' quality. The alternative is to conduct unsupervised GES under similar conditions, but more side-by-side comparisons documenting the similarities and differences between both approaches are necessary. Consequently, we need more data describing the preferred approach and whether the differences or similarities in the results are so significant to cause concern. This research distributed a DIY observation kit, which 30 participants assembled and used to propose gestures for controlling elements in a smart living room using a pillow's surface, with and without supervision. Our results show that gestures from supervised and unsupervised studies differ in quantity and max-consensus but not in gesture Agreement Scores. Our results also show that participants preferred conducting unsupervised studies but proposed fewer gesture sets in this condition.

On the Distribution of Muscle Signals: A Method for Distance-Based Classification of Human Gestures.

We investigate the distribution of muscle signatures of human hand gestures under Dynamic Time Warping. For this we present a k-Nearest-Neighbors classifier using Dynamic Time Warping for the distance estimate. To understand the resulting classification performance, we investigate the distribution of the recorded samples and derive a method of assessing the separability of a set of gestures. In addition to this, we present and evaluate two approaches with reduced real-time computational cost with regards to their effectiveness and the mechanics behind them. We further investigate the impact of different parameters with regards to practical usability and background rejection, allowing fine-tuning of the induced classification procedure.

TripCEAiR: A multi-loss minimization approach for surface EMG based airwriting recognition

Airwriting Recognition refers to the problem of identification of letters written in space with movement of the finger. It can be seen as a special case of dynamic gesture recognition wherein the set of gestures are letters in a particular language. Surface Electromyography (sEMG) is a non-invasive approach used to capture electrical signals generated as a result of contraction and relaxation of the muscles. sEMG has been widely adopted for gesture recognition applications. Unlike static gestures, dynamic gestures are user-friendly and can be used as a method for input with applications in Human Computer Interaction. There has been limited work in recognition of dynamic gestures such as airwriting, using sEMG signals and forms the core of the current work. In this work, a multi-loss minimization framework for sEMG based airwriting recognition is proposed. The proposed framework aims at learning a feature embedding vector that minimizes the triplet loss, while simultaneously learning the parameters of a classifier head to recognize corresponding alphabets. The proposed method is validated on a dataset recorded in the lab comprising of sEMG signals from 50 participants writing English uppercase alphabets. The effect of different variations of triplet loss, triplet mining strategies and feature embedding dimension is also presented. The best-achieved accuracy was 81.26% and 65.62% in user-dependent and independent scenarios respectively by using semihard positive and hard negative triplet mining. The code for our implementation will be made available at https://github.com/ayushayt/TripCEAiR

Living Images and Marian Devotion: Words, Gestures, and Gazes

This article examines the living images of the Virgin through the illustration of one of the most important collections of miracles of the 13th century, Les Miracles de Nostre Dame by Gautier de Coinci. In this case, I will focus my attention on manuscript 551 of Besançon (Besançon, BM, MS 551), which, although it has many flaws in its manufacture, offers an interesting presentation of living images. The study of these miniatures reflects the importance of devotion, the set of gestures, words, and gazes, in the medieval spectator’s experience of Marian images. At a time when these images’ legitimacy as sacred objects was still being debated, the artists in this manuscript show their power without censorship, presenting them as if they were the Virgin herself.

GestureSurface: VR Sketching through Assembling Scaffold Surface with Non-Dominant Hand.

3D sketching in virtual reality (VR) provides an immersive drawing experience for designs. However, due to the lack of depth perception cues in VR, scaffolding surfaces that constrain strokes to 2D are usually used as visual guides to reduce the difficulty of drawing accurate strokes. When the dominant hand is occupied by the pen tool, the efficiency of scaffolding-based sketching can be improved by using gesture input to reduce the idleness of the non-dominant hand. This paper presents GestureSurface, a bi-manual interface that uses non-dominant hand performing gestures to operate scaffolding and the other hand drawing with controller. We designed a set of non-dominant gestures to create and manipulate scaffolding surfaces, which are assembled by automatic combination based on five predefined primitive surfaces. We evaluated GestureSurface through a 20-person user study and found that the method of scaffolding-based sketching using non-dominant hand has the advantages of high efficiency and low fatigue.

Facial electromyogram-based facial gesture recognition for hands-free control of an AR/VR environment: optimal gesture set selection and validation of feasibility as an assistive technology.

The online version contains supplementary material available at 10.1007/s13534-023-00277-9.

GWrite: Enabling Through-the-Wall Gesture Writing Recognition Using WiFi

Recognizing in-air gestures can enable intelligent Human-Computer Interaction (HCI) applications and facilitate human lives. However, existing sensor/camera-based methods for gesture recognition are either non-ubiquitous, intrusive to privacy, or inconvenient to carry around. Contemporary device-free approaches require the person to be in the line of sight and proximity to the sensing device. This paper shows that WiFi signals can recognize hand-drawn in-air gestures even when the gesture location is non-line-of-sight/beyond walls to the WiFi transceivers. The proposed GWrite system utilizes the CSI time-series information from commercial WiFi chipsets. GWrite employs a unique approach for performing hand gestures, thus enabling the design of a hand movement model. Using the model and the time-reversal (TR) technique, this work derives a correspondence between the similarity of CSIs and the relative distance moved by the hand. This relation gave rise to unique features such as the number of segments, angle, and the intersection between segments that can classify a set of gesture shapes consisting of straight-line segments. GWrite achieved an accuracy of 92% on a group of 15 gestures. The proposed approach can be applied to a broader set of gestures, unlike the current systems that function over a limited gesture set.

SequenceSense: A Tool for Designing Usable Foot-Based Gestures Using a Sequence-Based Gesture Recognizer

Foot-based gestures enable people to interact with mobile and wearable devices when their hands are unavailable for interaction. For the foot gestures to be truly usable, the gestures should be recognizable by the system without being confused by daily activities and still be easy to perform. However, designing such gestures often requires multiple iterations of gesture design, model training, and evaluation. In this paper, we present SequenceSense, a tool developed to help designers efficiently design a usable gesture set using inertial sensors, which eliminates the need for multiple data collection studies to evaluate the gestures’ usability through gesture modification by sequencing atomic actions and instant false positive analysis, and instead requires only the initial gesture sample collection. Unlike gesture recognizers using complete gestures to train a model, SequenceSense segments gesture into a sequence of atomic actions. For example, a foot tap to the right may have (1) lift the foot, (2) move the foot to the right, and (3) land the foot. SequenceSense also compares the gesture sequence with the sequence database created from the daily activities to identify possible conflicts. This allows gesture designers to build easily usable foot-based gestures without the need for recollecting and evaluating gestures. We validated SequenceSense’s efficacy in designing usable gestures with low false positives through a user study with nine gesture designers.

User-defined mid-air gestures for multiscale GIS interface interaction

ABSTRACT Multisize geographic information system (GIS) has widely used in social life, and the introduction of natural interactive methods such as gesture interaction further broadens the applications of GIS. Gestures, as a natural means of expression, bring a convenient experience to the use of GIS. The user-elicitation method can determine the user’s original operational intention and take advantage of the natural and convenient gesture interaction. However, current research on elicitation methods mainly focuses on defining the function factors of operation commands and rarely considers the influence of environmental factors such as spatial sizes. This paper takes GIS as the object, which is closely related to multisize space. This paper reports a research project on user-defined gestures for interacting with two-dimensional maps in a virtual reality environment, including three types of GIS operation commands and six sizes of maps. The results showed that the user-defined gestures for the same GIS operation commands varied across map sizes. The increase in map size increased the amount of gestural movement. Drawing on our research, we developed a set of gestures for multisize GIS interface interaction. We hope that this research can provide a guiding foundation for mid-air gesture design for all multisize interfaces.

Development and testing of a virtual simulator for a myoelectric prosthesis prototype – the PRISMA Hand II – to improve its usability and acceptability

Artificial limbs can help people missing body parts to restore some of their daily-life activities. However, the user should spend up to a few months to intuitively control the new device. During this period, she/he may suffer pain due to wearing or using the prosthesis inappropriately. This research presents a virtual simulator that allows the user to carry out training sessions for controlling the prosthesis. A set of Surface Electromyographic (sEMG) sensors are used to acquire the signals from user’s muscles and send them to a recognition algorithm that interprets the patient’s intentions. Simultaneously, the patient observes the response of her/his device on the simulator. Two studies are presented: the first study evaluate the performance of three different recognition algorithms i.e., Linear Discriminant Analysis (LDA), Support Vector Machine (SVM), and Multi-Layer Perceptron (MLP), based on the successful recognition of the patient’s intentions. The second study investigates the least number of sEMG sensors to be used, as having less components improves the patient’s wearability and decreases the processing time. The developed simulator represents a real prosthetic device, PRISMA hand II.The results showed the superiority of the MLP with 80% of successful recognition when 6-sEMG sensors are used. If a reduced set of gestures is considered (frequently needed by the patient), 90% of successful recognition could be achieved. Less sEMG sensors significantly degraded the performance of the recognition algorithm as only 53.8% of successful recognition could be achieved. All experiments were conducted with the help of a patient with below-elbow amputation.