Stroke Gestures Research Articles

Empirical Myoelectric Feature Extraction and Pattern Recognition in Hemiplegic Distal Movement Decoding.

In myoelectrical pattern recognition (PR), the feature extraction methods for stroke-oriented applications are challenging and remain discordant due to a lack of hemiplegic data and limited knowledge of skeletomuscular function. Additionally, technical and clinical barriers create the need for robust, subject-independent feature generation while using supervised learning (SL). To the best of our knowledge, we are the first study to investigate the brute-force analysis of individual and combinational feature vectors for acute stroke gesture recognition using surface electromyography (EMG) of 19 patients. Moreover, post-brute-force singular vectors were concatenated via a Fibonacci-like spiral net ranking as a novel, broadly applicable concept for feature selection. This semi-brute-force navigated amalgamation in linkage (SNAiL) of EMG features revealed an explicit classification rate performance advantage of 10-17% compared to canonical feature sets, which can drastically extend PR capabilities in biosignal processing.

Electrophysiological evidence for the enhancement of gesture-speech integration by linguistic predictability during multimodal discourse comprehension

In face-to-face discourse, listeners exploit cues in the input to generate predictions about upcoming words. Moreover, in addition to speech, speakers produce a multitude of visual signals, such as iconic gestures, which listeners readily integrate with incoming words. Previous studies have shown that processing of target words is facilitated when these are embedded in predictable compared to non-predictable discourses and when accompanied by iconic compared to meaningless gestures. In the present study, we investigated the interaction of both factors. We recorded electroencephalogram from 60 Dutch adults while they were watching videos of an actress producing short discourses. The stimuli consisted of an introductory and a target sentence; the latter contained a target noun. Depending on the preceding discourse, the target noun was either predictable or not. Each target noun was paired with an iconic gesture and a gesture that did not convey meaning. In both conditions, gesture presentation in the video was timed such that the gesture stroke slightly preceded the onset of the spoken target by 130 ms. Our ERP analyses revealed independent facilitatory effects for predictable discourses and iconic gestures. However, the interactive effect of both factors demonstrated that target processing (i.e., gesture-speech integration) was facilitated most when targets were part of predictable discourses and accompanied by an iconic gesture. Our results thus suggest a strong intertwinement of linguistic predictability and non-verbal gesture processing where listeners exploit predictive discourse cues to pre-activate verbal and non-verbal representations of upcoming target words.

Studies and guidelines for two concurrent stroke gestures

This paper investigates thumb–index interaction on touch input devices, and more precisely the potential of two concurrent stroke gestures, i.e. gestures in which two fingers of the same hand concurrently draw one stroke each. We present two fundamental studies, one using such gestures for two-dimensional control, by precisely drawing figures, and the other for command activation, by roughly sketching figures. Results give a first analysis of user performance on 35 gestures with a varying complexity based on numbers of turns and symmetries. All 35 gestures, were grouped into six families. From these results we classify these families and propose new guidelines for designing future mobile interfaces. For instance, favoring anchored gestures (forefinger drawing while the thumb remains still on the surface) to increase input bandwidth when forefinger precision is required.

Attentive Sequence-to-Sequence Modeling of Stroke Gestures Articulation Performance

Production time of stroke gestures is a fundamental measure of user performance with graphical user interfaces. However, production time represents an overall quantification of the user's gesture articulation process and therefore provides an incomplete picture of such process. Moreover, previous approaches assumed stroke gestures as synchronous point sequences, when most gesture-driven applications have to deal with asynchronous point sequences. Furthermore, deep generative models of human handwriting ignore the temporal information, thereby missing a key component of the user's gesture articulation process. To solve these issues, we introduce Ditto , a sequence-to-sequence deep learning model that estimates the velocity profile of any stroke gesture using spatial information only, providing thus a fine-grained estimation of the moment-by-moment behavior of the user's articulation performance. We show that this unique capability makes Ditto remarkably accurate while handling gestures of any type: unistrokes, multistrokes, and multitouch gestures. Our model, code, and associated web application are available as open source software.

Mapping Directional Mid-Air Unistroke Gestures to Interaction Commands: A User Elicitation and Evaluation Study

A stroke is the basic limb movement that both humans and animals naturally and repetitiously perform. Having been introduced into gestural interaction, mid-air stroke gestures saw a wide application range and quite intuitive use. In this paper, we present an approach for building command-to-gesture mapping that exploits the semantic association between interactive commands and the directions of mid-air unistroke gestures. Directional unistroke gestures make use of the symmetry of the semantics of commands, which makes a more systematic gesture set for users’ cognition and reduces the number of gestures users need to learn. However, the learnability of the directional unistroke gestures is varying with different commands. Through a user elicitation study, a gesture set containing eight directional mid-air unistroke gestures was selected by subjective ratings of the direction in respect to its association degree with the corresponding command. We evaluated this gesture set in a following study to investigate the learnability issue, and the directional mid-air unistroke gestures and user-preferred freehand gestures were compared. Our findings can offer preliminary evidence that “return”, “save”, “turn-off” and “mute” are the interaction commands more applicable to using directional mid-air unistrokes, which may have implication for the design of mid-air gestures in human–computer interaction.

Two-dimensional Stroke Gesture Recognition

The expansion of touch-sensitive technologies, ranging from smartwatches to wall screens, triggered a wider use of gesture-based user interfaces and encouraged researchers to invent recognizers that are fast and accurate for end-users while being simple enough for practitioners. Since the pioneering work on two-dimensional (2D) stroke gesture recognition based on feature extraction and classification, numerous approaches and techniques have been introduced to classify uni- and multi-stroke gestures, satisfying various properties of articulation-, rotation-, scale-, and translation-invariance. As the domain abounds in different recognizers, it becomes difficult for the practitioner to choose the right recognizer, depending on the application and for the researcher to understand the state-of-the-art. To address these needs, a targeted literature review identified 16 significant 2D stroke gesture recognizers that were submitted to a descriptive analysis discussing their algorithm, performance, and properties, and a comparative analysis discussing their similarities and differences. Finally, some opportunities for expanding 2D stroke gesture recognition are drawn from these analyses.

Recognizing 3D Trajectories as 2D Multi-stroke Gestures

While end users can acquire full 3D gestures with many input devices, they often capture only 3D trajectories, which are 3D uni-path, uni-stroke single-point gestures performed in thin air. Such trajectories with their $(x,y,z)$ coordinates could be interpreted as three 2D stroke gestures projected on three planes,\ie, $XY$, $YZ$, and $ZX$, thus making them admissible for established 2D stroke gesture recognizers. To investigate whether 3D trajectories could be effectively and efficiently recognized, four 2D stroke gesture recognizers, \ie, \$P, \$P+, \$Q, and Rubine, are extended to the third dimension: $\$P^3$, $\$P+^3$, $\$Q^3$, and Rubine-Sheng, an extension of Rubine for 3D with more features. Two new variations are also introduced: $\F for flexible cloud matching and FreeHandUni for uni-path recognition. Rubine3D, another extension of Rubine for 3D which projects the 3D gesture on three orthogonal planes, is also included. These seven recognizers are compared against three challenging datasets containing 3D trajectories, \ie, SHREC2019 and 3DTCGS, in a user-independent scenario, and 3DMadLabSD with its four domains, in both user-dependent and user-independent scenarios, with varying number of templates and sampling. Individual recognition rates and execution times per dataset and aggregated ones on all datasets show a highly significant difference of $\$P+^3$ over its competitors. The potential effects of the dataset, the number of templates, and the sampling are also studied.

A Survey on Applying Automated Recognition of Touchscreen Stroke Gestures to Children’s Input

Abstract Gesture recognition algorithms help designers create intelligent user interfaces for a number of application areas. However, these recognition algorithms are usually designed to recognize the gestures of adults, not children, and as such they generally do not perform as well for children as adults. Recognition of younger children’s gestures is particularly poor when compared to recognition of older children’s and adults’ gestures. Researchers have begun to examine the aspects of children’s gesture articulation patterns that make recognition difficult. This paper extends the initial work examining child-specific recognition approaches by considering general-purpose approaches and how they might apply to the problem of recognizing children’s touchscreen gestures. This paper presents a survey of existing recognition and analysis techniques for gestures of both adults and children from a human-centered perspective, highlighting ways in which improved recognition can lead to a better experience for children using touchscreen gestures in a variety of contexts.

DG3: Exploiting Gesture Declarative Models for Sample Generation and Online Recognition

In this paper, we introduce DG3, an end-to-end method for exploiting gesture interaction in user interfaces. The method allows to declaratively model stroke gestures and their sub-parts, generating the training samples for the recognition algorithm. In addition, we extend the algorithms of the $-family for supporting the online (i.e., real-time ) stroke recognition and their parts, as declared in the models. Finally, we show that the method outperforms existing approaches for online recognition and has comparable accuracy with offline methods after a few gesture segments.

Omnis Prædictio: Estimating the full spectrum of human performance with stroke gestures

Designing effective, usable, and widely adoptable stroke gesture commands for graphical user interfaces is a challenging task that traditionally involves multiple iterative rounds of prototyping, implementation, and follow-up user studies and controlled experiments for evaluation, verification, and validation. An alternative approach is to employ theoretical models of human performance, which can deliver practitioners with insightful information right from the earliest stages of user interface design. However, very few aspects of the large spectrum of human performance with stroke gesture input have been investigated and modeled so far, leaving researchers and practitioners of gesture-based user interface design with a very narrow range of predictable measures of human performance, mostly focused on estimating production time, of which extremely few cases delivered accompanying software tools to assist modeling. We address this problem by introducing "Omnis Praedictio" (Omnis for short), a generic technique and companion web tool that provides accurate user-independent estimations of any numerical stroke gesture feature, including custom features specified in code. Our experimental results on three public datasets show that our model estimations correlate on average r > .9 with groundtruth data. Omnis also enables researchers and practitioners to understand human performance with stroke gestures on many levels and, consequently, raises the bar for human performance models and estimation techniques for stroke gesture input.

Algorithmic Efficiency of Stroke Gesture Recognizers: a Comparative Analysis

Gesture interaction is today recognized as a natural, intuitive way to execute commands of an interactive system. For this purpose, several stroke gesture recognizers become more efficient in recognizing end-user gestures from a training set. Although the rate algorithms propose their rates of return there is a deficiency in knowing which is the most recommended algorithm for its use. In the same way, the experiments known by the most successful algorithms have been carried out under different conditions, resulting in non-comparable results. To better understand their respective algorithmic efficiency, this paper compares the recognition rate, the error rate, and the recognition time of five reference stroke gesture recognition algorithms, i.e., $1, $P, $Q, !FTL, and Penny Pincher, on three diverse gesture sets, i.e., NicIcon, HHReco, and Utopiano Alphabet, in a user-independent scenario. Similar conditions were applied to all algorithms, to be executed under the same characteristics. For the algorithms studied, the method agreed to evaluate the error rate and performance rate, as well as the execution time of each of these algorithms. A software testing environment was developed in JavaScript to perform the comparative analysis. The results of this analysis help recommending a recognizer where it turns out to be the most efficient. !FTL (NLSD) is the best recognition rate and the most efficient algorithm for the HHreco and NicIcon datasets. However, Penny Pincher was the faster algorithm for HHreco datasets. Finally, $1 obtained the best recognition rate for the Utopiano Alphabet dataset.

The multimodal marking of aspect: The case of five periphrastic auxiliary constructions in North American English

AbstractCognitive linguistics (CL) has, in recent years, seen an increase in appeals to include multiple modalities in language analyses. While individual studies have incorporated gesture, gaze, facial expression, and prosody, among other modalities, CL has yet to completely embrace the systematic analysis of face-to-face interaction. Here, I present an investigation of five aspect-marking periphrastic constructions in North American English. Using naturalistic interactional data (n=250) from the Red Hen archive, this study establishes a multimodal profile for auxiliary constructions headed by one of five highly aspectualized verbs: continue, keep, start, stop, and quit, as in The jackpot continued to grow and He quit smoking. Results show that gesture timing, the structure of the gesture stroke, and gesture movement type, are variables that iconically and differentially represent distinctive aspectual conceptualizations. This study enhances our understanding of aspectual representation in co-speech gesture and informs the ongoing debate within CL and construction grammar circles of what constitutes conventionalization, or what constitutes a construction (mono- or multimodal).

DEICTIC: A compositional and declarative gesture description based on hidden markov models

• Declarative composition of HMMs for recognizing stroke gestures. • Support for sub-gestures identification and prediction. • Supports the implementation of feedback and feedforward with an effert comparable to heurist approaches, together with a definition procedure and accuracy comparable to machine learning approaches. • High accuracy in recognising composite gestures training only the primitives. The consumer-level devices that track the user’s gestures eased the design and the implementation of interactive applications relying on body movements as input. Gesture recognition based on computer vision and machine-learning focus mainly on accuracy and robustness. The resulting classifiers label precisely gestures after their performance, but they do not provide intermediate information during the execution. Human-Computer Interaction research focused instead on providing an easy and effective guidance for performing and discovering interactive gestures. The compositional approaches developed for solving such problem provide information on both the whole gesture and on its sub-parts, but they exploit heuristic techniques that have a low recognition accuracy. In this paper, we introduce DEICTIC, a compositional and declarative description for stroke gestures, which uses basic Hidden Markov Models (HMMs) to recognise meaningful predefined primitives (gesture sub-parts) and it composes them to recognise complex gestures. It provides information for supporting gesture guidance and it reaches an accuracy comparable with state-of-the-art approaches, evaluated on two datasets from the literature. Through a developer evaluation, we show that the implementation of a guidance system with DEICTIC requires an effort comparable to compositional approaches, while the definition procedure and the perceived recognition accuracy is comparable to machine learning.

G-Gene

The large availability of touch-sensitive screens fostered the research in gesture recognition. The Machine Learning community focused mainly on accuracy and robustness to noise, creating classifiers that precisely recognize gestures after their performance. Instead, the User Interface Engineering community developed compositional gesture descriptions that model gestures and their sub-parts. They are suitable for building guidance systems, but they lack a robust and accurate recognition support. In this paper, we establish a compromise between the accuracy and the provided information introducing G-Gene, a method for transforming compositional stroke gesture definitions into profile Hidden Markov Models (HMMs), able to provide both a good accuracy and information on gesture sub-parts. It supports online recognition without using any global feature, and it updates the information while receiving the input stream, with an accuracy useful for prototyping the interaction. We evaluated the approach in a user interface development task, showing that it requires less time and effort for creating guidance systems with respect to common gesture classification approaches.

Speech Based Intelligent System to Assist Blind/Low Vision Users Virtual Eyes

Smartphones are evolving day by day but the requirements of a blind person is yet to be fulfilled. Accessibility features are also introduced aiming at the visually impaired users, but a major proportion of the mobile applications are targeted for sighted users. This paper brings in the concept of complete integrated speech based intelligent system for both low vision and blind users. The application includes speech guides and speech confirmations at each and every point that enables the user to interact with his smart phone like a normal person does. Covering security and general features, it becomes a complete solution for the low vision and blind users all over the globe. They could interact with the smart phone in a smart way by using their other senses and get all their jobs done quickly and easily. This includes speech driven features combined with easy one stroke gestures t o make the functionality less complex.

The Impact of Low Vision on Touch-Gesture Articulation on Mobile Devices

A comprehensive study of stroke gestures on mobile device touchscreens by users with low vision reveals that some aspects of gesture articulation, such as gesture length or size, are minimally impacted by the impairment, while other aspects, such as production times, reveal suboptimal visuomotor routines of eye-hand coordination. Informed by the studys findings, the authors outline design guidelines for effective stroke-gesture input on mobile devices under low vision conditions.

A new iterative synthetic data generation method for CNN based stroke gesture recognition

Training a stroke gesture classifier by using the state-of-the-art Convolutional Neural Network method requires a large sample size to achieve good performance. This becomes a serious problem when users want to add new gestures to the system because adding so many samples is time-consuming and expensive. In this paper, we propose an iterative synthetic data generation method to solve this problem. The method takes in one user-input template gesture which is modeled by Bezier curve and can generate thousands of samples for training. We propose two different modeling approaches so the method can be applied to both mono and multi-stroke gestures. By applying perturbation to the control points, we can obtain enough samples for training. The generation process is carried out in an iterative way, so the variability in different categories of stroke gestures can be balanced. The variability is measured by the dynamic time wrapping method. The proposed method is tested on our own dataset and two published datasets. Our method outperforms methods with fixed generation process and reaches high recognition accuracy.

Human robot interaction for manipulation tasks based on stroke gesture recognition

PurposeThe purpose of this paper is to extend the usage of stroke gestures in manipulation tasks to make the interaction between human and robot more efficient.Design/methodology/approachIn this paper, a set of stroke gestures is designed for typical manipulation tasks. A gesture recognition and parameter extraction system is proposed to exploit the information in stroke gestures drawn by the users.FindingsThe results show that the designed gesture recognition subsystem can reach a recognition accuracy of 99.00 per cent. The parameter extraction subsystem can successfully extract parameters needed for typical manipulation tasks with a success rate about 86.30 per cent. The system shows an acceptable performance in the experiments.Practical implicationsUsing stroke gesture in manipulation tasks can make the transmission of human intentions to the robots more efficient. The proposed gesture recognition subsystem is based on convolutional neural network which is robust to different input. The parameter extraction subsystem can extract the spatial information encoded in stroke gestures.Originality/valueThe author designs stroke gestures for manipulation tasks which is an extension of the usage of stroke gestures. The proposed gesture recognition and parameter extraction system can make use of stroke gestures to get the type of the task and important parameters for the task simultaneously.

Identification of Single Vector Gesture Stroke

Identification of Single Vector Gesture Stroke

GESMO- MOtion GESture based Mobile Application

has been number of android mobile apps available for clients to solve their daily problems. This paper presents a Motion Gesture based mobile (GESMO) app in Android which allows user to draw gestures in the air resulting opening of a desired app assigned to corresponding gesture. This gesture may be the first letter of an app name which is to be launched. In order to start the motion, user must press and hold any point on the screen until the motion is completed. Once motion completes, it is released. For sensing the motion, we used two built-in sensors from mobile devices: accelerometer and gyroscope. The paper proposes a two-stage approach for spotting and recognition of generated stroke gesture. The spotting stage uses a Support Vector Machine (SVM) to identify data fragments containing one stroke gesture. The recognition stage uses Hidden Markov Models (HMM) to generate the text representation from the motion sensor data. With this technique, you can successfully receive 70-80% accuracy in detecting an air gesture for mobile devices.