Set Of Gestures Research Articles

A Robust Business Specific Real-Time Sign Language Translator

Abstract: Communication is a great way of expressing yourself but not everyone is capable of communication. No, not everyone chooses to be silent, While some are born deaf and mute, others become one later in life due to certain conditions. The word “dumb” is quite offensive so I will address to someone who cannot speak as mute person throughout this paper. Returning to the topic this paper will enable a more effective way to fill the communication gap between deaf and mute person and normal person especially in places with crowd where special-abled person becomes nervous and anxious to make the business owner understand of his needs easily and are unable to express themselves. In order to remove this barrier and enable the effective communication between the special-abled buyer and the business owner, I have created our own data set of hand gestures and trained using Google Teachable Machine for common sentences that are exchanged between the buyer and seller. Keywords: Sign Language Translation, AI, Special-Abled Communication, Google Teachable Machine Model, OpenCV, Computer Vision, Machine Learning, Deep Learning.

Time Domain Analysis of EMG Signals using KNN and SVM Techniques

The EMG signals that have been processed can mimic human movements. For this study, raw EMG data obtained when the hands are in repose (rest), in a clasp, and when the wrist is buckled and stretched were used to categorise four distinct forms of hand gestures using a MATLAB-based intelligent framework (open access data set). Statistical-time-domain features are applied to sort various hand gestures in this investigation. The K-Nearest-Neighbor (KNN) and Support-Vector-Machine (SVM) classifiers are used for classification and comparison. Furthermore, our method outperforms a state-of-the-art method on other data sets of hand gestures.

L-Sign: Large-Vocabulary Sign Gestures Recognition System

Understanding sign gestures is an essential step to helping individuals with hearing impaired. The existing works can only identify a small set of gestures accurately and the accuracy rate drops sharply with an increasing number of gestures. Because there are two challenges—a large number of similar gestures in sign language and the various signing speed of different people. Based on commercial smart bracelets, this article proposes a large-vocabulary sign language recognition system (which we call L-sign). First, we propose an entropy-based forward and backward matching algorithm to segment each gesture signal. Second, we design a gesture recognizer including a candidate gesture generator and semantic-based voter. The candidate gesture generator is aimed at providing candidate gesture designs based on a 3-branch convolutional neural network. The purpose of a semantic-based voter is to select the target gesture from candidate gestures by scoring, where the semantic distances between the last gesture in the current sentence and any candidate gestures is calculated, and a multilayer k-means algorithm is proposed to obtain a multilayer sign word structure to complete the scores of candidate gestures. Lastly, we deployed L-sign on the MYO bracelet. For 200 commonly used Chinese sign gestures, the experimental results show that the average accuracy rate was greater than 90%.

Touchless Interfaces in the Operating Room: A Study in Gesture Preferences

Touchless interfaces allow surgeons to control medical imaging systems autonomously while maintaining total asepsis in the Operating Room. This is specially relevant as it applies to the recent outbreak of COVID-19 disease. The choice of best gestures/commands for such interfaces is a critical step that determines the overall efficiency of surgeon-computer interaction. In this regard, usability metrics such as task completion time, memorability and error rate have a long-standing as potential entities in determining the best gestures. In addition, previous works concerned with this problem utilized qualitative measures to identify the best gestures. In this work, we hypothesize that there is a correlation between gestures’ qualitative properties and their usability metrics. In this regard, we conducted a user experiment with language experts to quantify gestures’ properties (v). Next, we developed a gesture-based system that facilitates surgeons to control the medical imaging software in a touchless manner. Next, a usability study was conducted with neurosurgeons and the standard usability metrics (u) were measured in a systematic manner. Lastly, multi-variate correlation analysis was used to find the relations between u and v. Statistical analysis showed that the v scores were significantly correlated with the usability metrics with an and p < 0.05. Once the correlation is established, we can utilize either gestures’ qualitative properties or usability metrics to identify the best set of gestures.

A gesture recognition algorithm in a robot therapy for ASD children

Children with Autism Spectrum Disorders (ASDs) exhibit significant impairments in gesture imitation. Newest interventions are based on Human-Robot Interaction (HRI) since children with ASD cope well with stylized, rule-based and predictable systems. These collaborative approaches encompass therapy games based on joint exercises, imitation and interaction between robots and children. This paper’s aim was to implement an algorithm to automatically recognize small and similar gestures within a humanoid-robot therapy called IOGIOCO for ASD children. IOGIOCO is a multi-level HRI therapy meant to teach 19 meaningful gestures in a semantic framework based on a feedback interaction. Gestures were tracked as 3D coordinates of body keypoints captured by a Kinect. A Residual Neural Network was implemented and trained on a segmented Dataset acquired within this study to generate the offline model which was then exploited in a real-time classification using a sliding window. Feedback as sound stimuli from NAO robot was provided based on the automatic evaluation of each performance. Clinical acquisitions were carried out on 4 ASD children within the IOGIOCO therapy. Offline recognition was successful: exploiting Artificial Neural Networks we reached 95% of test accuracy for 19 gestures. A real-time recognition on healthy subjects reached 94% accuracy. Clinical applications were evaluated through the F1 score that achieved 79% value. These outcomes were encouraging considering the wide gesture set and all the challenges the therapy raises. This kind of automatic algorithm was able to decrease the therapist workload and increase the robustness of the therapy and engagement of the child.

Virtual Interface With Kinect 3D Sensor for Interaction With Bedridden People

The human-machine interaction has evolved significantly in the last years, allowing a new range of opportunities for developing solutions for people with physical limitations. Natural user interfaces (NUI) allow bedridden and/or physically disabled people to perform a set of actions trough gestures thus increasing their quality of life and autonomy. This paper presents a solution based on image processing and computer vision using the Kinect 3D sensor for development of applications that recognize gestures made by the human hand. The gestures are then identified by a software application that triggers a set of actions of upmost importance for the bedridden person, for example, trigger the emergency, switch on/off the TV or control the bed slope. It was used a shape matching technique for six gestures recognition, being the final actions activated by the Arduino platform. The results show a success rate of 96%. This system can improve the quality of life and autonomy of bedridden people, being able to be adapted for the specific necessities of an individual subject.

Evaluating the Performance of Machine Learning Algorithms in Gaze Gesture Recognition Systems

Despite the utility of gaze gestures as an input method, there is a lack of guidelines available regarding how to design gaze gestures, what algorithms to use for gaze gesture recognition, and how these algorithms compare in terms of performance. To facilitate the development of applications that leverage gaze gestures, we have evaluated the performance of a combination of template-based and data-driven algorithms on two custom gesture sets that can map to user actions. Template-based algorithms had consistently high accuracies but the slowest runtimes, making them best for small gesture sets or accuracy-critical applications. Data-driven algorithms run much faster and scale better to larger gesture sets, but require more training data to achieve the accuracy of the template-based methods. The main takeaways for gesture set design are 1) gestures should have distinct forms even when performed imprecisely and 2) gestures should have clear key-points for the eyes to fixate onto.

Hand Gesture Recognition With Flexible Capacitive Wristband Using Triplet Network in Inter-Day Applications.

Human-machine interfaces for hand gesture recognition across multiple sessions and days of doffing and re-donning while maintaining acceptable recognition accuracy are still challenging. In this paper, a flexible wristband, which was integrated with a highly sensitive capacitive pressure sensing array, was used for inter-day hand gesture recognition. The performance of the entire system was further improved by utilizing a triplet network for deep feature embedding. Seven hand gestures were included into the gesture set, and inter-day experiments which lasted for five consecutive days with three sessions on each day were conducted. Five healthy subjects participated in the experiment. Between each session, the wristband was doffed, and re-donned before the next session. The triplet network achieved an average recognition accuracy of 91.98% across all the sessions of all the subjects, and yielded a higher classification result (p < 0.05) over the convolutional neural network trained with softmax-cross-entropy loss (with an average accuracy of 84.65%). Furthermore, we also found that the capacitive array size had an evident influence on the inter-day classification result. The array with the full size (thirty-two channels) achieved a higher average recognition accuracy over all the down-sampled arrays. This work demonstrated the feasibility of improving the hand gesture recognition performance over days of usage by fabricating a wearable, flexible multi-channel capacitive wristband and implementing the triplet network.

Gesture recognition of wrist motion based on wearables sensors

This paper aims to evaluate two machine learning techniques using low-frequency photoplethysmography (PPG) associated with motion sensors, accelerometers and gyroscopes, from wearable devices such as smartwatches in gesture recognition from the wrist and fingers motion. The applied method consider the application of second-order gradient calculation on PPG to identify motion artifacts and then create segments that will be used gesture classification, the classification process used a support vector machine and random forests trained using statistical features extracted from PPG and motion sensor signals. Preliminary evaluations show that frequencies of 25 Hz are suitable for the recognition process, achieving an F1-score of 0.819 for seven gesture sets.

Hand Gestures Identification for Fine-Grained Human Activity Recognition in Smart Homes

Most developed countries are facing important demographic issues related to ageing populations. Maintaining elders at home while ensuring their safety and well-being often constitutes the main goal of these countries. An interesting solution to this challenge is to develop a smart home, able to monitor the routines of the resident, to recognize the on-going activities, and to provide support when required. In the literature, most works focus on monitoring high-level behaviors such as eating, sleeping, etc. However, to provide live guidance, the system needs to have a far more detailed recognition process able to identify the specific steps of the on-going task and the erroneous executions. In this paper, we propose an algorithmic approach for hand gesture recognition designed to be used as the core component of a fine-grained activity recognition model. It is based on the processing of inertial data collected from a wristband equipped with triaxial accelerometer and gyroscope, and machine learning techniques. A simple set of gestures for cooking activities as been defined, enabling characterizing high-level cooking tasks. To do that, we constructed a labelled dataset of atomic gestures performed by participants that we made available to the scientific community. We obtained an average accuracy of 0.83 in recognizing the gestures with the leave-one-subject-out strategy.

A Novel Myoelectric Control Scheme Supporting Synchronous Gesture Recognition and Muscle Force Estimation.

Aiming to provide feasible solutions for the realization of the robust and natural myoelectric control systems, a novel myoelectric control scheme supporting gesture recognition and muscle force estimation is proposed in this study. Eleven grasping gestures abstracted from daily life are selected as the target gesture set. The high-density surface electromyography (HD-sEMG) of the forearm flexor and the grasping force signal are collected simultaneously. The synchronous prediction of gesture category and instantaneous force is realized by the multi-task learning (MTL) technique. Especially, a post-processing algorithm based on threshold method is conducted to overcome the influence of force variation on the accuracy of gesture recognition. The experimental results show that the proposed post-processing method can decrease the classification error significantly. Specifically, the overall gesture classification error is reduced by 27 ~ 30 percent compared with not using the post-processing method; and 16 ~ 24 percent compared with using classical post-processing methods. The whole scheme can realize the synchronous gesture recognition and force estimation with 9.35 ± 11.48% gesture classification error and 0.1479 ± 0.0436 root-mean-square deviation force estimation accuracy. Meanwhile, it is feasible in different number of electrodes and well meets the real-time requirement of the EMG control system in response time delay (about 28.22 ~ 113.16ms on average). The proposed framework provides the possibility for myoelectric control supporting synchronous gesture recognition and force estimation, which can be extended and applied in the fields of myoelectric prosthesis and exoskeleton devices.

Eyelid gestures for people with motor impairments

Although eye-based interactions can be beneficial for people with motor impairments, they often rely on clunky or specialized equipment (e.g., stationary eye-trackers) and focus primarily on gaze and blinks. However, two eyelids can open and close in different orders and for different duration to form rich eyelid gestures. We take a first step to design, detect, and evaluate a set of eyelid gestures for people with motor impairments on mobile devices. We present an algorithm to detect nine eyelid gestures on smartphones in real time and evaluate it with 12 able-bodied people and 4 people with severe motor impairments in two studies. The results of the study with people with motor-impairments show that the algorithm can detect the gestures with .76 and .69 overall accuracy in user-dependent and user-independent evaluations. Furthermore, we design and evaluate a gesture mapping scheme for people with motor impairments to navigate mobile applications only using eyelid gestures. Finally, we discuss considerations for designing and using eyelid gestures for people with motor impairments.

‘Address and command’: Two-handed mid-air interactions with multiple home devices

Mid-air gestural interaction with multiple targets (or devices) in the physical space presents several challenges: gesture design gets more complex in the attempt to avoid gesture conflicts and accidental activations; the potential commands are many but the gesture set must be kept small to enable memorability; and, from a design-process perspective, there are not readily available technologies for implementation and prototyping. In this paper, we present the Address and Command (A&C) interaction model that enables two-handed mid-air interactions with multiple remote devices, in a research approach that includes gesture elicitation, prototyping, and evaluation of the user experience. The A&C model requires that users employ the non-dominant hand to address a device (address gestures) and the dominant hand to provide a command to it (command gestures). This approach affords interactions that simultaneously address and command multiple devices in ubiquitous environments that respond to mid-air input. A&C interactions also afford the design of a single gesture for the same command when applicable for multiple devices, thus decreasing the total number of gestures to be memorized and promoting end-user learning of the gesture set. We have (a) conducted an elicitation study (n = 18) to define A&C gestures for seven smart home devices and twelve commands; (b) developed a spatial augmented reality prototype that responds to mid-air gesture commands exploiting MS Kinect sensor and Visual Gesture Builder, Unity 3D and projection-mapping technologies onto foam mockups; (c) tested the approach and the gesture vocabulary in terms of usability, memorability and user experience at a scenario of 36 tasks among devices (n = 17). We found that A&C interactions are feasible, fast, error-free, easy to learn and remember, and are highly valued in terms of user experience.

OctoPocus in VR: Using a Dynamic Guide for 3D Mid-Air Gestures in Virtual Reality.

Bau and Mackays OctoPocus dynamic guide helps novices learn, execute, and remember 2D surface gestures. We adapt OctoPocus to 3D mid-air gestures in Virtual Reality (VR) using an optimization-based recognizer, and by introducing an optional exploration mode to help visualize the spatial complexity of guides in a 3D gesture set. A replication of the original experiment protocol is used to compare OctoPocus in VR with a VR implementation of a crib-sheet. Results show that despite requiring 0.9s more reaction time than crib-sheet, OctoPocus enables participants to execute gestures 1.8s faster with 13.8 percent more accuracy during training, while remembering a comparable number of gestures. Subjective ratings support these results, 75 percent of participants found OctoPocus easier to learn and 83 percent found it more accurate. We contribute an implementation and empirical evidence demonstrating that an adaptation of the OctoPocus guide to VR is feasible and beneficial.

Mapping Directional Mid-Air Unistroke Gestures to Interaction Commands: A User Elicitation and Evaluation Study

A stroke is the basic limb movement that both humans and animals naturally and repetitiously perform. Having been introduced into gestural interaction, mid-air stroke gestures saw a wide application range and quite intuitive use. In this paper, we present an approach for building command-to-gesture mapping that exploits the semantic association between interactive commands and the directions of mid-air unistroke gestures. Directional unistroke gestures make use of the symmetry of the semantics of commands, which makes a more systematic gesture set for users’ cognition and reduces the number of gestures users need to learn. However, the learnability of the directional unistroke gestures is varying with different commands. Through a user elicitation study, a gesture set containing eight directional mid-air unistroke gestures was selected by subjective ratings of the direction in respect to its association degree with the corresponding command. We evaluated this gesture set in a following study to investigate the learnability issue, and the directional mid-air unistroke gestures and user-preferred freehand gestures were compared. Our findings can offer preliminary evidence that “return”, “save”, “turn-off” and “mute” are the interaction commands more applicable to using directional mid-air unistrokes, which may have implication for the design of mid-air gestures in human–computer interaction.

Gesture-controlled image system positioning for minimally invasive interventions

Abstract This work examines how a touchless interaction concept contributes to an efficient, direct, and sterile interaction workflow during CT-guided interventions. Two hand gesture sets were designed specifically under consideration of the clinical workflow and the hardware capabilities. These were used to change the position of an X-Ray tube and detector of a CT scanner without breaking sterility and are compared regarding usability and performance in a user study with 10 users. The user study revealed that it is possible to change the angle of the gantry within 10 seconds average in an experimental setup. A straight hand gesture showed higher acceptance than a pistol motivated gesture. Furthermore, the sequences were not optimal and confused the users. It turned out that it feels more natural to activate and confirm the system with the same gesture.

Drone affect: Folded points of view as a co-affection method for empathy and care

Drone usage in fieldwork and participatory processes entails direct sensations that fold the pilot/planner and audience into one. Footage from drone filming entails direct kinesthetic and synesthetic effects that occur while one edits the film material, giving rise to affective responses. Through one experimental drone postproduction film, this article discusses how the interplay of vision, motion, and sound works as a set of gestures that comprise sensation as a (self- and co-) affection modus of immediation when one engages with moving images on-screen during drone filming, editing, and screening. The article builds upon Merleau-Ponty's film writings and Puig de la Bellacasa's touch/care perspectives. Specifically, it asks what constitutive aspects characterize the folding of the drone's/pilot's point of view (POV) in relation to the perceiver's POV during postediting, and to the conceiver's POV during subsequent viewings. The article's original contribution is to show how affective experiences are transposed as haptic touch via the same images and folded POVs. It also addresses a research methodological perspective on filmic mediations and written accounts. The article concludes by speculating about how the co-affective potential of the folding of POVs through drone imagery might inform fieldwork and participatory processes from perspectives of care and empathy.

PolyRec Gesture Design Tool: A tool for fast prototyping of gesture‐based mobile applications

AbstractIn this article, we present PolyRec Gesture Design Tool (PolyRec GDT), a tool that allows the definition of gesture sets for mobile applications. It supports the entire process of designing gesture sets, from collecting gestures to analyzing and exporting the set created for use in a mobile application. It is equipped with the PolyRec gesture recognizer, a unistroke recognizer based on template matching with good accuracy and performances while still allowing support for other recognizers. The main features of PolyRec GDT include the ability to collect gestures directly on mobile devices, to detect similarities between gestures to help the designer to detect ambiguities in the design phase, and the ability to automatically select the most representative gestures for each gesture class by using a clustering technique to increase recognition speed without affecting accuracy. Experimental results show that this latter feature allows an increase in accuracy compared with when the same operation is performed manually by developers or randomly by software. Finally, a user study was carried out to evaluate PolyRec GDT, in which participants were asked to use it to add gesture functionality to a mobile application. The results showed that with the support of PolyRec GDT this operation took only a few minutes for participants with mobile development experience, whereas it would have taken much longer without the support of PolyRec GDT.

XGest: Enabling Cross-Label Gesture Recognition with RF Signals

Extensive efforts have been devoted to human gesture recognition with radio frequency (RF) signals. However, their performance degrades when applied to novel gesture classes that have never been seen in the training set. To handle unseen gestures, extra efforts are inevitable in terms of data collection and model retraining. In this article, we present XGest, a cross-label gesture recognition system that can accurately recognize gestures outside of the predefined gesture set with zero extra training effort. The key insight of XGest is to build a knowledge transfer framework between different gesture datasets. Specifically, we design a novel deep neural network to embed gestures into a high-dimensional Euclidean space. Several techniques are designed to tackle the spatial resolution limits imposed by RF hardware and the specular reflection effect of RF signals in this model. We implement XGest on a commodity mmWave device, and extensive experiments have demonstrated the significant recognition performance.

Home Automation Using Gestures

The proposed system “Home Automation models using gestures” is designed for providing ease of control of home appliances to the people, especially elderly or those who are physically unable to efficiently perform day-to-day activities. Initially the images of the gestures are captured in real time using a camera placed in the room and are processed to be compared with the existing set of gestures. Based on the gesture shown, the image is then processed to attain the actual meaning of the gesture. So if the gestures match, then the particular appliance in that room is switched on or off by which it helps the user to stay at his place in the room and doesn't need to actually move to the switchboard to turn on the appliances.