Target Gestures Research Articles

Vehicle Control on an Uninstrumented Surface With an Off-the-Shelf Smartwatch

Advances in wearable and machine learning technologies have caused smartwatches to emerge as promising input devices across disciplines. In this study, we propose a user interface, designed for an automotive environment for recognizing input commands. As the use of extra elements to extend the scope of interactions may visually distract users during driving, we focus on using body parts as an interaction space. Specifically, we utilized the lap as an interaction hyperplane due to its near-flat surface, which provides users interaction analogous to a touchpad. To substantiate the proposed approach, we collected motion signals using an off-the-shelf smartwatch and trained them in supervised learning settings. As target gestures, we defined ten different hand gestures, such as tapping and folding/spreading all fingers on the lap. The experimental results exhibited a test accuracy of 94%, thereby validating the feasibility of the proposed approach.

Interaction with a Hand Rehabilitation Exoskeleton in EMG-Driven Bilateral Therapy: Influence of Visual Biofeedback on the Users' Performance.

The effectiveness of EMG biofeedback with neurorehabilitation robotic platforms has not been previously addressed. The present work evaluates the influence of an EMG-based visual biofeedback on the user performance when performing EMG-driven bilateral exercises with a robotic hand exoskeleton. Eighteen healthy subjects were asked to perform 1-min randomly generated sequences of hand gestures (rest, open and close) in four different conditions resulting from the combination of using or not (1) EMG-based visual biofeedback and (2) kinesthetic feedback from the exoskeleton movement. The user performance in each test was measured by computing similarity between the target gestures and the recognized user gestures using the L2 distance. Statistically significant differences in the subject performance were found in the type of provided feedback (p-value 0.0124). Pairwise comparisons showed that the L2 distance was statistically significantly lower when only EMG-based visual feedback was present (2.89 ± 0.71) than with the presence of the kinesthetic feedback alone (3.43 ± 0.75, p-value = 0.0412) or the combination of both (3.39 ± 0.70, p-value = 0.0497). Hence, EMG-based visual feedback enables subjects to increase their control over the movement of the robotic platform by assessing their muscle activation in real time. This type of feedback could benefit patients in learning more quickly how to activate robot functions, increasing their motivation towards rehabilitation.

What affects gestural learning in children with and without Autism? The role of prior knowledge and imitation

The present study examined whether prior knowledge to the learning target and imitation during learning affected learning outcomes in preschool children with autism spectrum disorder (ASD, N = 22) compared to their typically developing (TD, N = 15) peers. Children’s gestural skills in recognizing and producing the target gestures before and after the training, as well as their imitative behavior during the training were coded. Results showed that consistent prior knowledge benefited gestural learning in both groups. Besides, only children with ASD were hindered by inconsistent prior knowledge. Notably, the effect of imitation was not significant in the ASD group. In conclusion, the learning process in children with ASD may differ from those with typical development, suggesting special-designed interventions are required.

Pragmatics of L2 gestures in beginners’ classes of Italian

This article examines the responses that two groups of beginners of Italian in Canada (n=30) and in Italy (n=30) contributed to a multimodal exercise, where six culturally-based Italian gestures were presented alone and in combination with mini-dialogues. The main goal of this study was to encourage Italian L2 pedagogy to become more oriented towards the teaching of the pragmatics of body movements occurring during interactions. This work advocates that it is important to promote awareness of the pragmatic roles of the body from the early stages of language development to better prepare L2 learners in the recognition of the nonverbal contents communicated in a target language. One of the main results of this study suggests that combining target gestures with speech is a necessary pedagogical approach to enable the understanding of the semantics and the pragmatics of multimodal messages in L2 communication.

A Hand Gesture Recognition Algorithm Based on Multi-scale Hybrid Features

A human hand gesture recognition algorithm based on multi-scale hybrid features is pro-posed. A large number of gesture data images are organized and annotated by automatic im-age annotation algorithm. Based on dense sampling, the multi-scale Gaussian pyramid is taken into account for enhancing the image details; it can realize the recognition of both large and small target gestures. Meanwhile, according to the aggregation and distribution of gestures, statistical analysis is used to optimize the size and aspect ratio of anchors, which can further improve the accuracy of recognition. The experiment results show that the mean average precisions of this algorithm can be improved to 99.7% and 93.3% for the simple gesture and complex gesture, respectively, which are higher than that obtained by these traditional methods, such as faster region-convolution neural network (Faster R-CNN), You Only Look Once (YOLO), and SSD.

Wrist-Worn Hand Gesture Recognition While Walking via Transfer Learning.

Walking, one of the most common daily activities, causes unwanted movement artifacts which can significantly deteriorate hand gesture recognition accuracy. However, traditional hand gesture recognition algorithms are typically developed and validated with wrist-worn devices only during static human poses, neglecting the critical importance of dynamic effects on gesture accuracy. Thus, we developed and validated a signal decomposition approach via empirical mode decomposition to accurately segment target gestures from coupled raw signals during dynamic walking and a transfer learning method based on distribution adaptation to enable gesture recognition through domain transfer between dynamic walking and static standing scenarios. Ten healthy subjects performed seven hand gestures during both walking and standing experiments while wearing an IMU wrist-worn device. Experimental results showed that the signal decomposition approach reduced the gesture detection error by 83.8%, and the transfer learning approach (20% transfer rate) improved hand gesture recognition accuracy by 15.1%. This ground-breaking work demonstrates the feasibility of hand gesture recognition while walking via wrist-worn sensing. These findings serve to inform real-life and ubiquitous adoption of wrist-worn hand gesture recognition for intuitive human-machine interaction in dynamic walking situations.

Multichannel Biphasic Muscle Stimulation System for Post Stroke Rehabilitation

We present biphasic stimulator electronics developed for a wearable functional electrical stimulation system. The reported stimulator electronics consist of a twenty four channel biphasic stimulator. The stimulator circuitry is physically smaller per channel and offers a greater degree of control over stimulation parameters than existing functional electrical stimulator systems. The design achieves this by using, off the shelf multichannel high voltage switch integrated circuits combined with discrete current limiting and dc blocking circuitry for the frontend, and field programmable gate array based logic to manage pulse timing. The system has been tested on both healthy adults and those with reduced upper limb function following a stroke. Initial testing on healthy users has shown the stimulator can reliably generate specific target gestures such as palm opening or pointing with an average accuracy of better than 4 degrees across all gestures. Tests on stroke survivors produced some movement but this was limited by the mechanical movement available in those users’ hands.

Wearable EMG-Based Gesture Recognition Systems During Activities of Daily Living: An Exploratory Study.

Recent advancements in wearable technologies have increased the potential for practical gesture recognition systems using electromyogram (EMG) signals. However, despite the high classification accuracies reported in many studies (> 90%), there is a gap between academic results and industrial success. This is in part because state-of-the-art EMG-based gesture recognition systems are commonly evaluated in highly-controlled laboratory environments, where users are assumed to be resting and performing one of a closed set of target gestures. In real world conditions, however, a variety of non-target gestures are performed during activities of daily living (ADLs), resulting in many false positive activations. In this study, the effect of ADLs on the performance of EMG-based gesture recognition using a wearable EMG device was investigated. EMG data for 14 hand and finger gestures, as well as continuous activity during uncontrolled ADLs (>10 hours in total) were collected and analyzed. Results showed that (1) the cluster separability of 14 different gestures during ADLs was 171 times worse than during rest; (2) the probability distributions of EMG features extracted from different ADLs were significantly different (p <; 0.05). (3) of the 14 target gestures, a right angle gesture (extension of the thumb and index finger) was least often inadvertently activated during ADLs. These results suggest that ADLs and other non-trained gestures must be taken into consideration when designing EMG-based gesture recognition systems.

Advanced Hand Gesture Prediction Robust to Electrode Shift with an Arbitrary Angle.

Recent advances in myoelectric controlled techniques have made the surface electromyogram (sEMG)-based sensing armband a promising candidate for acquiring bioelectric signals in a simple and convenient way. However, inevitable electrode shift as a non-negligible defect commonly causes a trained classifier requiring continuous recalibrations. In this study, a novel hand gesture prediction is firstly proposed; it is robust to electrode shift with arbitrary angle. Unlike real-time recognition which outputs target gestures only after the termination of hand motions, our proposed advanced prediction can provide the same results, even before the completion of signal collection. Moreover, by combining interpolated peak location and preset synchronous gesture, the developed simplified rapid electrode shift detection and correction at random rather than previous fixed angles are realized. Experimental results demonstrate that it is possible to achieve both electrode shift detection with high precision and gesture prediction with high accuracy. This study provides a new insight into electrode shift robustness which brings gesture prediction a step closer to practical applications.

Finger Gesture Spotting from Long Sequences Based on Multi-Stream Recurrent Neural Networks.

Gesture spotting is an essential task for recognizing finger gestures used to control in-car touchless interfaces. Automated methods to achieve this task require to detect video segments where gestures are observed, to discard natural behaviors of users’ hands that may look as target gestures, and be able to work online. In this paper, we address these challenges with a recurrent neural architecture for online finger gesture spotting. We propose a multi-stream network merging hand and hand-location features, which help to discriminate target gestures from natural movements of the hand, since these may not happen in the same 3D spatial location. Our multi-stream recurrent neural network (RNN) recurrently learns semantic information, allowing to spot gestures online in long untrimmed video sequences. In order to validate our method, we collect a finger gesture dataset in an in-vehicle scenario of an autonomous car. 226 videos with more than 2100 continuous instances were captured with a depth sensor. On this dataset, our gesture spotting approach outperforms state-of-the-art methods with an improvement of about 10% and 15% of recall and precision, respectively. Furthermore, we demonstrated that by combining with an existing gesture classifier (a 3D Convolutional Neural Network), our proposal achieves better performance than previous hand gesture recognition methods.

Device-Free Localization Scheme With Time-Varying Gestures Using Block Compressive Sensing

In Device-Free Localization (DFL) schemes, most researches assume that the targets are stationary with fixed gestures and motions. However, objects always change their gestures in practice, undoubtedly influencing the localization accuracy. To solve the issue, we propose a new algorithm named Ges-DFL under the Compressive Sensing (CS) framework, which considers the time-varying target gestures in the DFL scheme. Firstly, leaning that the matrixes referring to different target gestures cannot be achieved, we transfer them to a fixed sensing matrix by the transferring method. Secondly, we build the localization scheme as an MMV recovery issue and exploit the block sparsity property of transferred location vectors to improve the localization accuracy. Thirdly, the new algorithm Ges-DFL scheme is designed under the framework of the variational Bayesian inference to reconstruct transferred location vectors. Simulations show that the proposed algorithm performs well both in localization accuracy and robustness.

Do all aspects of learning benefit from iconicity? Evidence from motion capture

abstractRecent work suggests that not all aspects of learning benefit from an iconicity advantage (Ortega, 2017). We present the results of an artificial sign language learning experiment testing the hypothesis that iconicity may help learners to learn mappings between forms and meanings, whilst having a negative impact on learning specific features of the form. We used a 3D camera (Microsoft Kinect) to capture participants’ gestures and quantify the accuracy with which they reproduce the target gestures in two conditions. In the iconic condition, participants were shown an artificial sign language consisting of congruent gesture–meaning pairs. In the arbitrary condition, the language consisted of non-congruent gesture–meaning pairs. We quantified the accuracy of participants’ gestures using dynamic time warping (Celebi et. al., 2013). Our results show that participants in the iconic condition learn mappings more successfully than participants in the arbitrary condition, but there is no difference in the accuracy with which participants reproduce the forms. While our work confirms that iconicity helps to establish form–meaning mappings, our study did not give conclusive evidence about the effect of iconicity on production; we suggest that iconicity may only have an impact on learning forms when these are complex.

Applying information theory to investigate music performance with continuous electronic sensors

A wide variety of electronic sensors can be used for designing new digital musical instruments and other human-computer interfaces. However, presently human abilities for continuously controlling such sensors are not well quantified. The field of information theory suggests that a human together with a user interface can be modeled as a communication channel. Previously, Fitts' Law used a discrete communications channel to model information conveyed by a human pointing at discrete targets. In contrast, the present work employs a continuous communications channel to model a human continuously controlling an analog-valued sensor. The Shannon-Hartley theorem implies that the channel capacity (e.g., HCI throughput) can be estimated by asking human subjects to perform gestures that match idealized, bandlimited Gaussian “target gestures” across a range of bandwidths. Then, the signal-to-noise ratio of the recorded gestures determines the channel capacity (e.g., HCI throughput). This approach is tested on human users alternately operating simple analog sensors. Suggestions are made for creating knowledge about user interfaces that could potentially transmit an enhanced amount of information to a computer.

Development of a generic method to generate upper-body emotional expressions for different social robots

To investigate the effect of gestures in human–robot interaction, a number of social robots capable of gesturing have been designed. Gestures are often preprogrammed off-line or generated by mapping motion capture data to the robot. Since these gestures are dependent on the robot’s joint configuration, they cannot be used for other robots. Therefore, when using a new robot platform with a different morphology, new joint trajectories to reach the desired postures need to be implemented. This method aims to minimize the workload when implementing gestures on a new robot platform and facilitate the sharing of gestures between different robots. The innovative aspect of this method is that it is constructed independently of any robot configuration, and therefore it can be used to generate gestures for different robot platforms. To calculate a gesture for a certain configuration, the developed method uses a set of target gestures listed in a database and maps them to that specific configuration. The database currently consists of a set of emotional expressions. The method was validated on the virtual model of different robots and an online survey was performed to evaluate the user’s perception of the output of the method. The results of this survey showed that the calculated gestures for a certain robot configuration well resemble the target gestures, and thus that our developed method to map gestures to different robot morphologies gives good results.

Audiovisual Temporal Integration of Speech and Gesture

The relation between speech and gestures is a topic of vigorous research in language comprehension and production. Recent studies indicate that speech and gestures are integrated in language comprehension (Kelly, Özyürek, & Maris, 2010) with mutual and obligatory interactions (Kita & Özyürek, 2003). This hypothesis is also supported by neurophysiological findings on the integration of speech and gestures (Willems, Özyürek, & Hagoort, 2007). In the present study, using two behavioral tasks, we further investigated the integration of speech and gesture by focusing on the ‘unity effect’ (the perception of two or more sensory signals with spatiotemporal contiguity and semantic/informational relatedness is unified into a multisensory whole; e.g., Vatakis & Spence, 2007) from the multisensory literature. In particular, we manipulated the levels of semantic relatedness between speech and gestures. In Experiment 1, we presented a series of dynamic iconic gestures (i.e., “cut”, “saw”, “write”) with the respective auditory speech. We manipulated semantic congruency through three conditions: congruent, weakly incongruent (e.g., “cut” with “saw”), and strongly incongruent (e.g., “cut” or “saw” with “write”). Participants performed a speeded discrimination task of unimodal (audio or visual) and multimodal conditions by detection of two target gestures (e.g., respond when you see, hear, or both “cut” and “saw”; all stimulus combinations where tested in a blocked format). Analysis of the reaction times (RTs) revealed shorter RTs, as expected, for the multimodal congruent cases as compared to the unimodal and incongruent conditions presented. In Experiment 2, participants performed an unspeeded temporal order judgment task (TOJ). For the TOJ task the above mentioned conditions were presented with a set of stimulus onset ansynchronies (±541, ±333, ±208, ±125, and 0ms). Analysis of the just noticeable difference (JND) showed higher JNDs and, thus, worse performance for the congruent as compared to the incongruent condition. Both experiments, therefore, confirmed, for the first time, the unity effect in speech-gesture pairings, as well as the facilitation effect of semantic relatedness on the integration of speech and gestures. Overall, the present study confirms the hypothesis of integration between speech and iconic gestures, opening new pathways to research on multisensory and temporal relationships between the two communicative signals.

THE IMPACT OF BODY‐PART‐NAMING TRAINING ON THE ACCURACY OF IMITATIVE PERFORMANCES IN 2‐ TO 3‐YEAR‐OLD CHILDREN

A series of three experiments explored the relationship between 3-year-old children's ability to name target body parts and their untrained matching of target hand-to-body touches. Nine participants, 3 per experiment, were presented with repeated generalized imitation tests in a multiple-baseline procedure, interspersed with step-by-step training that enabled them to (i) tact the target locations on their own and the experimenter's bodies or (ii) respond accurately as listeners to the experimenter's tacts of the target locations. Prompts for on-task naming of target body parts were also provided later in the procedure. In Experiment 1, only tact training followed by listener probes were conducted; in Experiment 2, tacting was trained first and listener behavior second, whereas in Experiment 3 listener training preceded tact training. Both tact and listener training resulted in emergence of naming together with significant and large improvements in the children's matching performances; this was true for each child and across most target gestures. The present series of experiments provides evidence that naming--the most basic form of self-instructional behavior--may be one means of establishing untrained matching as measured in generalized imitation tests. This demonstration has a bearing on our interpretation of imitation reported in the behavior analytic, cognitive developmental, and comparative literature.

Vision-based arm gesture recognition for a long-range human–robot interaction

This paper proposes a vision-based human arm gesture recognition method for human---robot interaction, particularly at a long distance where speech information is not available. We define four meaningful arm gestures for a long-range interaction. The proposed method is capable of recognizing the defined gestures only with 320×240 pixel-sized low-resolution input images captured from a single camera at a long distance, approximately five meters from the camera. In addition, the system differentiates the target gestures from the users' normal actions that occur in daily life without any constraints. For human detection at a long distance, the proposed approach combines results from mean-shift color tracking, short- and long-range face detection, and omega shape detection. The system then detects arm blocks using a background subtraction method with a background updating module and recognizes the target gestures based on information about the region, periodical motion, and shape of the arm blocks. From experiments using a large realistic database, a recognition rate of 97.235% is achieved, which is a sufficiently practical level for various pervasive and ubiquitous applications based on human gestures.

Determinants of imitation of hand-to-body gestures in 2- and 3-year-old children.

Twenty children, ten 2-year-olds and ten 3-year-olds, participated in an AB procedure. In the baseline phase, each child was trained the same four matching relations to criterion under intermittent reinforcement. During the subsequent imitation test, the experimenter modeled a total of 20 target gestures (six trials each) interspersed with intermittently reinforced baseline trials. In each session, target gestures were selected in a pre-randomized sequence from: Set 1--ear touches; Set 2--shoulder touches; Set 3--midarm touches; and Set 4--wrist touches; subjects' responses to targets were not reinforced. In each target set, half the gestures featured in nursery matching games and were termed common targets whereas the remainder, which were topographically similar but did not feature in the games, served as uncommon targets. The children produced significantly more matching responses to common target models than to uncommon ones. Common responses were also produced as mismatches to uncommon target models more often than vice versa. Response accuracy did not improve over trials, suggesting that "parity" did not serve as a conditioned reinforcer. All children showed a strong bias for "mirroring"--responding in the same hemispace as the modeler. The 2-year-olds produced more matching errors than the 3-year-olds and most children showed a bias for responding with their right hands. The strong effects of training environment (nursery matching games) are consistent with a Skinnerian account, but not a cognitive goal theory account, of imitation in young children.

Mother versus stranger: a triadic situation of imitation at the end of the first year of life

AbstractThis study presents two experiments investigating 8‐ and 12‐month‐old infants' imitative behaviour. Seventy‐two 8‐month‐olds and seventy‐two 12‐month‐olds were observed in a triadic situation which included their mother and a stranger. Depending on the condition, either the mother or the stranger acted as the demonstrator and either stayed close or withdrew after the demonstration, during the response period. In addition to imitative acts, visual exploration and smiles addressed, respectively, to each partner were computed. Results showed that at both ages, neither the familiarity nor the position of the partner has an effect on the number of target gestures that are imitated. At 12 months, infants looked and smiled more at the stranger when he demonstrated target actions but no difference was found when the mother acted as demonstrator. Moreover, 12‐month‐old infants looked more at the demonstrating partner immediately after their first imitation. At 8 months, infants paid more attention to the stranger in all conditions except when the mother performed the target actions and moved away, a pattern that suggests a referencing to the mother. Results from the gaze and smile variables suggest that with age different motivations (social contact, exploration of objects) induce imitation. Copyright © 2003 John Wiley &amp; Sons, Ltd.

Neonatal imitation in a mother–infant setting

Imitation of tongue protrusions and mouth openings was investigated in 18 newborn infants using a procedure differing in three ways from previous laboratory studies: the mother presented the target gestures, she was permitted to adjust to the infant's reactions, and the infant's baseline response rate was assessed during face-to-face interaction. This procedure was assumed to facilitate infant imitation. Compared with baseline responses in face-to-face interaction, both tongue protrusions and mouth openings were imitated in the two conditions. No imitation, however, was found when the responses in the two imitation conditions were compared. These results are less convincing than those found in strictly controlled laboratory experiments. A less restrictive procedure did not, therefore, promote imitation. The mixed results could be explained by too short an exposure to the target gesture. If repeated and insistent gesturing is a crucial condition, neonatal imitation might be understood to be the result of instruction and teaching, rather than a means of mutual communication. © 1998 John Wiley & Sons, Ltd.