Set Of Gestures Research Articles

American sign language recognition using HAAR type classifier

In this paper, we introduce a hand gesture recognition system to recognize real time gesture in unconstrained environment. We consider fixed set of manual commands, and develop a simple, yet affective procedure for gesture recognition. The system consists of three modules: real time hand tracking, gesture set training and gesture recognition. We have used Viola Jones Object detection framework which employs a variant of the learning algorithm AdaBoost to both select the best features and to train classifiers that use them. We have tested our system vocabulary of four gestures and results effectiveness of this approach using principal component analysis and Eigen vectors.

Multisensor Serious Game-Based Therapy Environment for Hemiplegic Patients

We present an e-therapy framework that collects live therapeutic context by analyzing body joint data in a noninvasive way. Using our proposed framework, a therapist can model complex gestures by mapping them to a set of primitive gesture sequences and generate high-level serious game-based therapies. As a proof of concept, we have developed scenarios to express a hemiplegic patient's behavior into a set of trackable primitive gestures. We have used Second Life and Map browsing based serious game environment for immersive experience. The initial feedback from the therapists who have tested our developed framework is encouraging. Finally, we share the implementation details and analysis of test results.

Gestures for Manually Controlling a Helping Hand Robot

Helping hand robots have been the focus of a number of studies and have high potential in modern manufacturing processes and for use in daily living. As helping hand robots interact closely with users, it is important to find natural and intuitive user interfaces for interacting with the robots in various situations. This study describes a set of gestures for interacting with and controlling helping hand robots in situations in which users need to manually control the robot but both hands are not available, for example, when users are holding tools or objects in their hands. The gestures are derived from an experimental study that asked participants for gestures suitable for controlling primitive robot motions. The selected gestures can be used to control translation and orientation of an end effector of a helping hand robot when one or both hands are engaged with tasks. As an example for validating the proposed gestures, we implemented a helping hand robot system to perform a soldering task.

A Scale and Translation Invariant Approach for Early Classification of Spatio-Temporal Patterns Using Spiking Neural Networks

This paper addresses the problem of encoding and classifying spatio-temporal patterns, which are typical for human actions or gestures. The proposed method has the following main contributions: (i) it requires a very small number of training examples, (ii) it accepts variable sized input patterns, (iii) it is invariant to scale and translation, and (iv) it enables early recognition, from only partial information of the pattern. The underlying representation employed is a spiking neural network with axonal conductance delay. We designed a novel approach for mapping spatio-temporal patterns to spike trains, which are used to stimulate the network. The pattern features emerge in the network as a result of this stimulation in the form of polychronous neuronal groups, which are used for classification. The proposed method is validated on a set of gestures representing the digits from $$0$$0 to $$9$$9, extracted from video data of a human drawing the corresponding digits. The paper presents a comparison with several other standard pattern recognition approaches. The results show that the proposed approach significantly outperforms these methods, it is invariant to scale and translation, and it has the ability to recognize patterns from only partial information.

LOCALIZATION AND RECOGNITION OF DYNAMIC HAND GESTURES BASED ON HIERARCHY OF MANIFOLD CLASSIFIERS

Abstract. Generally, the dynamic hand gestures are captured in continuous video sequences, and a gesture recognition system ought to extract the robust features automatically. This task involves the highly challenging spatio-temporal variations of dynamic hand gestures. The proposed method is based on two-level manifold classifiers including the trajectory classifiers in any time instants and the posture classifiers of sub-gestures in selected time instants. The trajectory classifiers contain skin detector, normalized skeleton representation of one or two hands, and motion history representing by motion vectors normalized through predetermined directions (8 and 16 in our case). Each dynamic gesture is separated into a set of sub-gestures in order to predict a trajectory and remove those samples of gestures, which do not satisfy to current trajectory. The posture classifiers involve the normalized skeleton representation of palm and fingers and relative finger positions using fingertips. The min-max criterion is used for trajectory recognition, and the decision tree technique was applied for posture recognition of sub-gestures. For experiments, a dataset “Multi-modal Gesture Recognition Challenge 2013: Dataset and Results” including 393 dynamic hand-gestures was chosen. The proposed method yielded 84–91% recognition accuracy, in average, for restricted set of dynamic gestures.

A gesture-controlled projection display for CT-guided interventions.

The interaction with interventional imaging systems within a sterile environment is a challenging task for physicians. Direct physician-machine interaction during an intervention is rather limited because of sterility and workspace restrictions. We present a gesture-controlled projection display that enables a direct and natural physician-machine interaction during computed tomography (CT)-based interventions. Therefore, a graphical user interface is projected on a radiation shield located in front of the physician. Hand gestures in front of this display are captured and classified using a leap motion controller. We propose a gesture set to control basic functions of intervention software such as gestures for 2D image exploration, 3D object manipulation and selection. Our methods were evaluated in a clinically oriented user study with 12 participants. The results of the performed user study confirm that the display and the underlying interaction concept are accepted by clinical users. The recognition of the gestures is robust, although there is potential for improvements. The gesture training times are less than 10 min, but vary heavily between the participants of the study. The developed gestures are connected logically to the intervention software and intuitive to use. The proposed gesture-controlled projection display counters current thinking, namely it gives the radiologist complete control of the intervention software. It opens new possibilities for direct physician-machine interaction during CT-based interventions and is well suited to become an integral part of future interventional suites.

Development of a generic method to generate upper-body emotional expressions for different social robots

To investigate the effect of gestures in human–robot interaction, a number of social robots capable of gesturing have been designed. Gestures are often preprogrammed off-line or generated by mapping motion capture data to the robot. Since these gestures are dependent on the robot’s joint configuration, they cannot be used for other robots. Therefore, when using a new robot platform with a different morphology, new joint trajectories to reach the desired postures need to be implemented. This method aims to minimize the workload when implementing gestures on a new robot platform and facilitate the sharing of gestures between different robots. The innovative aspect of this method is that it is constructed independently of any robot configuration, and therefore it can be used to generate gestures for different robot platforms. To calculate a gesture for a certain configuration, the developed method uses a set of target gestures listed in a database and maps them to that specific configuration. The database currently consists of a set of emotional expressions. The method was validated on the virtual model of different robots and an online survey was performed to evaluate the user’s perception of the output of the method. The results of this survey showed that the calculated gestures for a certain robot configuration well resemble the target gestures, and thus that our developed method to map gestures to different robot morphologies gives good results.

Adaptive and Personalized Gesture Recognition Using Textile Capacitive Sensor Arrays

Upper extremity mobility impairment is a common sequel of Spinal Cord Injury (SCI), brain injury, strokes, and degenerative diseases such as Guillain-Barre and ALS. Existing assistive technology solutions that provide access as user input devices are intrusive and expensive, and require physical contact that can have deleterious effects such as skin friction injury for paralyzed users who have reduced skin sensitivity. To address this problem, in this paper, we present the design, implementation, and evaluation of a non-contact proximity gesture recognition system using fabric capacitive sensor arrays. The fabric sensors are lightweight, flexible, and can be easily integrated into items of quotidian use such as clothing, bed sheets, and pillow covers. Our gesture recognition algorithm builds on two known classification techniques, Hidden Markov Model and Dynamic Time Warping to convert raw capacitance values to alphanumeric gestures. Our system is personalized to the user, allowing personalized selection of gesture sets and definition of gesture patterns in accordance with their capabilities. Our system adapts to changes in sensor configuration and orientation with minimal user training and intervention. We have evaluated our system in the context of a gesture-driven home automation system on six subjects that includes an individual who has a C6 Spinal Cord injury. We show that our system can recognize gestures of varying complexity with an average accuracy of 99 percent with minimal training.

Dynamic Time Warping 기반의 특징 강조형 제스처 인식 모델

스마트 디바이스가 보편화되면서 이에 내장된 가속도 센서를 사용한 제스처의 인식에 관한 연구가 주목받고 있다. 최근 가속도 센서 데이터 시컨스를 통한 제스처 인식에 Dynamic Time Warping(DTW) 기법이 사용되는데, 본 논문에서는 DTW 사용 시 제스처의 인식률을 높이기 위한 특징 강조형 제스처 인식(FsGr) 모델을 제안한다. FsGr 모델은 잘못 인식될 가능성이 높은 유사 제스처들의 집합에 대해 특징이 강조되는 데이터 시컨스의 부분들을 정의하고 이들에 대해 추가적인 DTW를 실행하여 인식률을 높인다. FsGr 모델의 훈련 과정에서는 유사 제스처들의 집합들을 정의하고 유사 제스처들의 특징들을 분석한다. 인식 과정에서는 DTW를 사용한 1차 인식 시도의 결과 제스처가 유사 제스처 집합에 속한 경우, 특징 분석 결과를 기반으로 한 추가적인 인식을 시도하여 인식률을 높인다. 알파베트 소문자에 대한 인식 실험을 통해 FsGr 모델의 성능 평가 결과를 보인다. As smart devices get popular, research on gesture recognition using their embedded-accelerometer draw attention. As Dynamic Time Warping(DTW), recently, has been used to perform gesture recognition on data sequence from accelerometer, in this paper we propose Feature-Strengthened Gesture Recognition(FsGr) Model which can improve the recognition success rate when DTW is used. FsGr model defines feature-strengthened parts of data sequences to similar gestures which might produce unsuccessful recognition, and performs additional DTW on them to improve the recognition rate. In training phase, FsGr model identifies sets of similar gestures, and analyze features of gestures per each set. During recognition phase, it makes additional recognition attempt based on the result of feature analysis to improve the recognition success rate, when the result of first recognition attempt belongs to a set of similar gestures. We present the performance result of FsGr model, by experimenting the recognition of lower case alphabets.

A new approach for touch gesture recognition: Conversive Hidden non-Markovian Models

With the current boom of touch devices the recognition of touch gestures is becoming an important field of research. Performing such gestures can be seen as a stochastic process, as there can be many little differences between different executions. Therefore stochastic models like Hidden Markov Models have already been applied to gesture recognition. Although the modelling possibilities of Hidden Markov Models are limited, they achieve an acceptable recognition quality. But they have never been tested with gestures that only differ in execution speed.We propose the use of Conversive Hidden non-Markovian Models for touch gesture recognition. This extension of Hidden Markov Models enhances the modelling possibilities and adds timing features. In this paper, two touch gesture recognition systems were developed and implemented based on these two model types. Experiments with a set of similar gestures show that the proposed model class is a good and competitive alternative to Hidden Markov Models.

Exploring the Benefits of Context in 3D Gesture Recognition for Game-Based Virtual Environments

We present a systematic exploration of how to utilize video game context (e.g., player and environmental state) to modify and augment existing 3D gesture recognizers to improve accuracy for large gesture sets. Specifically, our work develops and evaluates three strategies for incorporating context into 3D gesture recognizers. These strategies include modifying the well-known Rubine linear classifier to handle unsegmented input streams and per-frame retraining using contextual information (CA-Linear); a GPU implementation of dynamic time warping (DTW) that reduces the overhead of traditional DTW by utilizing context to evaluate only relevant time sequences inside of a multithreaded kernel (CA-DTW); and a multiclass SVM with per-class probability estimation that is combined with a contextually based prior probability distribution (CA-SVM). We evaluate each strategy using a Kinect-based third-person perspective VE game prototype that combines parkour-style navigation with hand-to-hand combat. Using a simple gesture collection application to collect a set of 57 gestures and the game prototype that implements 37 of these gestures, we conduct three experiments. In the first experiment, we evaluate the effectiveness of several established classifiers on our gesture set and demonstrate state-of-the-art results using our proposed method. In our second experiment, we generate 500 random scenarios having between 5 and 19 of the 57 gestures in context. We show that the contextually aware classifiers CA-Linear, CA-DTW, and CA-SVM significantly outperform their non--contextually aware counterparts by 37.74%, 36.04%, and 20.81%, respectively. On the basis of the results of the second experiment, we derive upper-bound expectations for in-game performance for the three CA classifiers: 96.61%, 86.79%, and 96.86%, respectively. Finally, our third experiment is an in-game evaluation of the three CA classifiers with and without context. Our results show that through the use of context, we are able to achieve an average in-game recognition accuracy of 89.67% with CA-Linear compared to 65.10% without context, 79.04% for CA-DTW compared to 58.1% without context, and 90.85% with CA-SVM compared to 75.2% without context.

Designing concurrent full-body gestures for intense gameplay

Full body gestures provide alternative input to video games that are more natural and intuitive. However, full-body game gestures designed by developers may not always be the most suitable gestures available. A key challenge in full-body game gestural interfaces lies in how to design gestures such that they accommodate the intensive, dynamic nature of video games, e.g., several gestures may need to be executed simultaneously using different body parts. This paper investigates suitable simultaneous full-body game gestures, with the aim of accommodating high interactivity during intense gameplay. Three user studies were conducted: first, to determine user preferences, a user-elicitation study was conducted where participants were asked to define gestures for common game actions/commands; second, to identify suitable and alternative body parts, participants were asked to rate the suitability of each body part (one and two hands, one and two legs, head, eyes, and torso) for common game actions/commands; third, to explore the consensus of suitable simultaneous gestures, we proposed a novel choice-based elicitation approach where participants were asked to mix and match gestures from a predefined list to produce their preferred simultaneous gestures. Our key findings include (i) user preferences of game gestures, (ii) a set of suitable and alternative body parts for common game actions/commands, (iii) a consensus set of simultaneous full-body game gestures that assist interaction in different interactive game situations, and (iv) generalized design guidelines for future full-body game interfaces. These results can assist designers and practitioners to develop more effective full-body game gestural interfaces or other highly interactive full-body gestural interfaces.

User-defined gestures for controlling primitive motions of an end effector

In designing and developing a gesture recognition system, it is crucial to know the characteristics of a gesture selected to control, for example, an end effector of a robot arm. We conducted an experiment to collect a set of user-defined gestures and investigate characteristics of the gestures for controlling primitive motions of an end effector in human–robot collaboration. We recorded 152 gestures from 19 volunteers by presenting virtual robotic arm movements to the participants, and then asked the participants to think about and perform gestures that would cause the motions. It was found that the hands were the parts of the body used most often for gesture articulation even when the participants were holding tools and objects with both hands: a number of participants used one- and two-handed gestures interchangeably, gestures were consistently performed by the participants across all pairs of reversible gestures, and the participants expected better recognition performance for gestures that were easy to think of and perform. These findings are expected to be useful as guidelines in creating a gesture set for controlling robotic arms according to natural user behaviors.

A User-Developed 3-D Hand Gesture Set for Human–Computer Interaction

The purpose of this study was to develop a lexicon for 3-D hand gestures for common human-computer interaction (HCI) tasks by considering usability and effort ratings. Recent technologies create an opportunity for developing a free-form 3-D hand gesture lexicon for HCI. Subjects (N = 30) with prior experience using 2-D gestures on touch screens performed 3-D gestures of their choice for 34 common HCI tasks and rated their gestures on preference, match, ease, and effort. Videos of the 1,300 generated gestures were analyzed for gesture popularity, order, and response times. Gesture hand postures were rated by the authors on biomechanical risk and fatigue. A final task gesture set is proposed based primarily on subjective ratings and hand posture risk. The different dimensions used for evaluating task gestures were not highly correlated and, therefore, measured different properties of the task-gesture match. A method is proposed for generating a user-developed 3-D gesture lexicon for common HCIs that involves subjective ratings and a posture risk rating for minimizing arm and hand fatigue.

Potential of multimodal and multiuser interaction with virtual holography

Virtual holography is a disruptive technology that can inspire innovations in variety of fields through blending the physical world with sensory-rich virtual world. The technology enables users to naturally and intuitively manipulate objects and navigate in 3D space. The zSpace virtual holography platform is described. The platform provides a 3D display and head-tracking technology that transforms PCs into virtual-holographic computing facility using a stereoscopic user interface with an interactive stylus. The major components of the platform and its potential benefits, along with some of the current applications are briefly described.The use of the zSpace platform to explore and manipulate a number of space system models is outlined. The models considered include Titan Saturn system, a joint NASA/ESA mission; Mars Science Lab concept; James Webb Telescope; and a Lunar Lander concept. In each of the applications considered, the platform is enhanced by using multimodal interactions, and providing support for multiuser collaboration. The multimodal interactions, which enable more engaging, enhanced accessibility, are achieved by fusing information from a set of stylus input, 3D gestures, and neural input. Simultaneous and collaborative multiuser interactions are described, which support both local and distributed teams, using variety of displays.Emerging and future enhancements of the zSpace platform are outlined, along with novel future applications.

I-Therapy: a non-invasive multimedia authoring framework for context-aware therapy design

In this paper, we present an e-Therapy framework, named i-Therapy, which collects live therapeutic context by analyzing multi-sensory body joint data in a non-invasive way. Using our proposed framework, a therapist can model complex gestures by mapping them to a set of primitive gesture sequences and generate high-level therapies. As a proof of concept, we have developed scenarios to express a Hemiplegic patient's behavior into a set of track-able primitive gestures. The initial feedback from the therapists who have tested our developed framework is encouraging. Finally, we share the implementation details and analysis of test results.

User-designed movement interactions. An exploratory study for natural interactions

The context of this paper is movement and gesture-based interaction for natural user interaction (NUI) design. Naturalness is not an intrinsic but a perceived quality; and although movement seems an ideal candidate for NUIs, it bears its own limitations. Although experience can be drawn from other interaction styles (e.g. desktop graphical interfaces, touch), we advocate that movement - based natural interaction should also be tackled from a fresh perspective. Many existing works are technology - driven; our approach focuses on users’ definitions of interactions, leaving the technical issues for later works. In this paper, we present an experiment that identifies a set of user-defined gestures for a set of core tasks in freehand movement interaction using a large screen display. The experiment is divided into two steps, for exploration and validation. The exploratory step uses a bottom-up approach. We use the context of a large screen television. Subjects were asked to spontaneously perform gestures to achieve a set of typical tasks (e.g. pointing, selecting, engaging/disengaging) in a wizard-of-Oz setup. We isolated the favored interactions for such tasks. The validation step uses a top-down approach. We used a motion capture device to produce skeleton videos of the gestures retained from step one. An online questionnaire allowed validating the first findings and assessing how the suggested interactions can be perceived. From those experiment we drew five natural, typical interactions corresponding to five tasks, that are heavily preferred by subjects and which form good candidates as controls in future interaction designs.

Assessing Sailor and Civilian Gestural Optimal Relationships for Multi-touch Gestures and Functions in Computer Applications

With touchscreen-enabled devices becoming more prevalent throughout military applications, it is important to create a standardized set of touchscreen gestures for military purposes. This standardized set must easily transfer among the devices military personnel use for operations. The authors describe two experiments targeting Department of Defense (DoD) civilians and active-duty Navy sailors to determine optimal gesture-to-function mapping of common gestures. Results indicate the two populations sampled had significant differences in their experiences with several categories of touchscreen (TS) and non-TS enabled devices. Results for gestural agreement between the military and civilian populations continued, showing substantial differences in their expectations for gestural functions. Through this research, the authors provide a beneficial first step in evaluating the set of gestures currently under development and suggest further research with military personnel as a necessary next step for developing a standardized gesture set for military purposes.

Designing Motion Gesture Interfaces in Mobile Phones for Blind People

Despite the existence of advanced functions in smartphones, most blind people are still using old-fashioned phones with familiar layouts and dependence on tactile buttons. Smartphones support accessibility features including vibration, speech and sound feedback, and screen readers. However, these features are only intended to provide feedback to user commands or input. It is still a challenge for blind people to discover functions on the screen and to input the commands. Although voice commands are supported in smartphones, these commands are difficult for a system to recognize in noisy environments. At the same time, smartphones are integrated with sophisticated motion sensors, and motion gestures with device tilt have been gaining attention for eyes-free input. We believe that these motion gesture interactions offer more efficient access to smartphone functions for blind people. However, most blind people are not smartphone users and they are aware of neither the affordances available in smartphones nor the potential for interaction through motion gestures. To investigate the most usable gestures for blind people, we conducted a user-defined study with 13 blind participants. Using the gesture set and design heuristics from the user study, we implemented motion gesture based interfaces with speech and vibration feedback for browsing phone books and making a call. We then conducted a second study to investigate the usability of the motion gesture interface and user experiences using the system. The findings indicated that motion gesture interfaces are more efficient than traditional button interfaces. Through the study results, we provided implications for designing smartphone interfaces.

Gestures that people can understand and use

Recent advances in computing devices push researchers to envision new interaction modalities that go beyond traditional mouse and keyboard input. Typical examples are large displays for which researchers hope to create more “natural” means of interaction by using human gestures and body movements as input. In this article, we reflect about this goal of designing gestures that people can easily understand and use and how designers of gestural interaction can capitalize on the experience of 30 years of research on visual languages to achieve it. Concretely, we argue that gestures can be regarded as “visual expressions to convey meaning” and thus are a visual language. Based on what we have learned from visual language research in the past, we then explain why the design of a generic gesture set or language that spans many applications and devices is likely to fail. We also discuss why we recommend using gestural manipulations that enable users to directly manipulate on-screen objects instead of issuing commands with symbolic gestures whose meaning varies among different users, contexts, and cultures.