Arm Gesture Recognition Research Articles

Real-Time Recognition of the User's Arm Gestures in 2D Space with a Smart Camera

Real-Time Recognition of the User's Arm Gestures in 2D Space with a Smart Camera

Lightweight Midrange Arm-Gesture Recognition System From mmWave Radar Point Clouds

This article considers arm-gesture recognition in the form of a point cloud in mid-range scenarios. A 2-D steer vector is established according to the structure of the radar array antenna, and the average power of the array is added to the point cloud image as a feature. In the data processing stage, the proposed system aggregates multiple measurements on all frames, overcoming the issues of the spatial sparsity and ambiguous temporal variation of the point cloud image of the millimeter-wave (mmWave) radar, as a new arm-gesture point cloud image. In the arm-gesture recognition stage, a lightweight PointNet-based classifier is custom-designed to recognize and classify arm-gestures point cloud images. Extensive experiments are carried out on custom-built radar gesture datasets with eight human subjects performing 19 arm-gestures. The experimental results demonstrate that the proposed arm-gesture recognition system can accurately distinguish 19 different arm-gestures with an average accuracy of 99.69% in two indoor environments and outperform five related arm-gesture classifiers. We further analyze the effect of the scenarios and distance on recognition performance.

U-WeAr: User Recognition on Wearable Devices through Arm Gesture

The use of wearable devices equipped with inertial sensors has become increasingly pervasive. It has been widely demonstrated in the literature that inertial signals acquired by these sensors can be used by machine learning algorithms to predict actions performed and/or to recognize the identities of the person wearing the sensors. In this article, we present a hardware/software system for arm gesture recognition, identity recognition, and verification of a person based on inertial sensors. The hardware part is a custom wristband that consists of a computing unit, a wireless communication unit, and an inertial sensor. The software part is an algorithm based on recurrent neural networks that is able to process the signals coming from the sensor and to return a prediction. To validate the system, a dataset consisting of 25 symbols drawn with the arm is collected. These symbols are performed by 33 subjects. We conduct two evaluations: 1) performance evaluation for arm gesture recognition, user recognition and verification; and 2) usability assessment of the system. The performance of the three recognition tasks indicate that this system can be reliably applied in real environments with an accuracy above 96% for gesture recognition, an accuracy of about 85% for user identification, and an equal error rate of about 13% for user verification. The outcome of the usability test proves a great satisfaction from the users in terms of high simplicity in the use of the wristband and goodness of the machine learning predictions.

Real-time Arm Gesture Recognition in Smart Home Scenarios via Millimeter Wave Sensing

"In air" gesture recognition using millimeter wave (mmWave) radar and its applications in natural human-computer-interaction for smart home has shown its potential. However, the state-of-the-art works still fall short in terms of limited gesture space, vulnerable to surrounding interference, and off-line recognition. In this paper, we propose mHomeGes, a real-time mmWave arm gesture recognition system for practical smart home-usage. To this end, we first distill arm gesture's position and dynamic variation, and then custom-design a lightweight convolution neural network to recognize fine-grained gestures. Next, we propose a user discovery method to focus on the target human gesture, thus eliminating the adverse impact of surrounding interference. Finally, we design a hidden Markov model-based voting mechanism to handle continuous gesture signals at run-time, which leads to continuous gesture recognition in real-time. We implement mHomeGes on a commodity mmWave radar and also perform a user study, which demonstrates that mHomeGes achieves high recognition accuracy above 95.30% in real-time across various smart home scenarios, regardless of the impact of surrounding movements, concurrent gestures, human physiological conditions, and outer packing materials. Moreover, we have also publicly archived a mmWave gesture data-set collected during developing mHomeGes, which consists of about 22,000 instances from 25 persons and may have an independent value of facilitating future research.

Arm Motion Classification Using Time-Series Analysis of the Spectrogram Frequency Envelopes

Hand and arm gesture recognition using radio frequency (RF) sensing modality proves valuable in man–machine interfaces and smart environments. In this paper, we use the time-series analysis method to accurately measure the similarity of the micro-Doppler (MD) signatures between the training and test data, thus providing improved gesture classification. We characterize the MD signatures by the maximum instantaneous Doppler frequencies depicted in the spectrograms. In particular, we apply two machine learning (ML) techniques, namely, the dynamic time warping (DTW) method and the long short-term memory (LSTM) network. Both methods take into account the values as well as the temporal evolution and characteristics of the time-series data. It is shown that the DTW method achieves high gesture classification rates and is robust to time misalignment.

Analysis of attribute domain for geometrical gesture performed by arm movements

<span>Hand gesture recognition commonly uses a camera to track hand movements and transformed into gesture database by using various computational approaches. Motion tracking utilized to map coordinate point of the subject movement, either in skeletal model or marker tracing. Data from motion trackers usually contains massive coordinate sequences of marker movement. A reliable method is required to select best features and analyze these data. However, the current issue whether the selected features and data presentation are significant for the research or not. This research brings the concept of ontology design for arm gesture recognition systems by utilizing the motion capture system. Ontology is the conceptual structure mainly used to retrieve information by establishing relation in complex data model. The proposed ontology framework is divided into three domains which are knowledge domain, attribute domain and process domain. Knowledge domain holds pre-processed gestural data from motion capture. The attribute domain is that the level where all the attribute elements were presented. This paper shows the analysis of the datasets in attribute domain. The analysis is divided into two parts which is precision measure and ANOVA test. Both analyses are to prove the reliability of datasets in attribute domain. The precision measure is used to remove all the common data for all gesture. A statistical analysis of p-value is lower than 0.01 which means the gestural data are statistically significant to be used for the similarity measure.</span>

Continuous Arm Gesture Recognition Based on Natural Features and Logistic Regression

In this paper, we propose a new gesture modeling method by combining natural features with logistic regression to recognize continuous natural gestures, which is considered as a challenge. Instead of manually modeling gestures with criteria, gesture models can be derived from a few gesture instances. Arm gestures are modeled with three types of intuitive features, and logistic regression is employed to obtain the optimal weights to linearly combine all the features, which can deal with the differences among instances with the same gesture automatically and facilitate distinguishing different gestures, and as a result, improve the recognition accuracy. With these features, numbers of natural arm gestures can be modeled with good comprehensibility. To evaluate our method, we defined twelve natural gestures, including three static gestures and nine dynamic gestures. A continuous gesture database which includes 154024 data frames (102 minutes) and 1628 gesture instances was collected from 10 subjects with a wearable multi-IMU sensing system. Both subject-dependent and subject-mixed recognition tests were conducted with the database. In the subject-dependent test, models were trained with about 30% data, and the average accuracy over all subjects was 91.54%. In the subject-mixed test, only about 10% data was used for training, and the accuracy is 88.75%. The recognition accuracy is highly competitive with previous gesture recognition approaches.

DeepGesture: Deep learning-based gesture recognition scheme using motion sensors

Recent advancement in smart phones and sensor technology has promoted research in gesture recognition. This has made designing of efficient gesture interface easy. However, human activity recognition (HAR) through gestures is not trivial since each person may pose the same gesture differently. In this paper, we propose deepGesture algorithm, a new arm gesture recognition method based on gyroscope and accelerometer sensors using deep convolution and recurrent neural networks. This method uses four deep convolution layers to automate feature learning in raw sensor data. The features of the convolution layers are used as input of the gated recurrent unit (GRU) which is based on the state-of-the-art recurrent neural network (RNN) structure to capture long-term dependency and model sequential data. The input data of the proposed algorithm is obtained through motion sequence data extracted using a wrist-type smart band device equipped with gyroscope and accelerometer sensors. The data is initially segmented in fixed length segments. The segmented data is labeled and we construct the database. Then the labeled data is used in our learning algorithm. To verify the applicability of the algorithm, several experiments have been performed to measure the accuracy of gesture classification. Compared to the human activity recognition method, our experimental results show that the proposed deepGesture algorithm can increase the average F1-score for recognition of nine defined arm gestures by 6%.

Dynamic Arm Gesture Recognition Using Spherical Angle Features and Hidden Markov Models

We introduce a vision-based arm gesture recognition (AGR) system using Kinect. The AGR system learns the discrete Hidden Markov Model (HMM), an effective probabilistic graph model for gesture recognition, from the dynamic pose of the arm joints provided by the Kinect API. Because Kinect’s viewpoint and the subject’s arm length can substantially affect the estimated 3D pose of each joint, it is difficult to recognize gestures reliably with these features. The proposed system performs the feature transformation that changes the 3D Cartesian coordinates of each joint into the 2D spherical angles of the corresponding arm part to obtain view-invariant and more discriminative features. We confirmed high recognition performance of the proposed AGR system through experiments with two different datasets.

FUNCTIONAL PRINCIPAL COMPONENT ANALYSIS FOR RECOGNITION OF ARM GESTURES AND HUMANOID IMITATION

This paper investigates the use of functional principal component analysis (FPCA) for automatic recognition of dynamic human arm gestures and robot imitation. FPCA is a statistical technique of functional data analysis that generalizes standard multivariate principal component analysis. Functional data analysis signals (e.g., gestures) are functions that are considered as observations of a random variable on a functional space. In particular, FPCA reduces the dimensionality of the input data by projecting them onto a finite-dimensional space spanned by a few prominent eigenfunctions. The main contribution of this work is the proposal of a novel technique for unsupervised clustering of training data and dynamic gesture recognition based on FPCA. FPCA has not been considered in previous studies on humanoid learning. The proposed approach has been evaluated in two experimental settings for motion capture. In the first setup single arm gestures are recognized from inertial sensors attached to the arm of the user. In the second setup the method is extended to two-arm gestures acquired from a range sensor. Recognized gestures are reproduced by a small humanoid robot. The FPCA method has also been compared to a high performance algorithm for gesture classification based on dynamic time warping (DTW). The FPCA algorithm achieves comparable results in both recognition rate and robustness to missing data, while it outperforms DTW in terms of efficiency in execution time.

특징 변환과 은닉 마코프 모델을 이용한 팔 제스처 인식 시스템의 설계

본 논문에서는 Kinect 센서를 이용한 팔 제스처 인식 시스템의 설계에 대해 소개한다. 제스처 인식을 위한 기존의 연구들에서는 동적 시간 왜곡(DTW)에서 은닉 마코프 모델(HMM)에 이르기까지 다양한 방법들이 적용되어 왔다. 본 논문에서 제안하는 제스처 인식 시스템은 Kinect 센서를 통해 얻을 수 있는 순차적인 팔 관절 위치 데이터로부터 각 제스처 별 고유한 은닉 마코프 모델을 학습한다. 동일한 제스처를 수행하더라도 Kinect 센서에 포착되는 각 관절의 위치 좌표 값들은 팔의 길이와 방향에 따라 크게 달라질 수 있다는 문제점이 있다. 본 논문에서 제안하는 시스템에서는 다양한 환경 조건에서도 높은 제스처 인식 성능을 얻기 위해, 팔 관절들의 좌표 값으로 구성된 특징 벡터를 팔 관절들 간의 각도 값으로 변환하는 특징 변환 과정을 수행한다. 또한, 본 시스템에서는 은닉 마코프 모델의 학습과 적용의 효율성을 높이기 위해, 고차원 실수 관측 벡터들에 k-평균 군집화를 적용하여 이산 은닉 마코프 모델들을 위한 1차원 정수 시퀀스들을 구한다. 이와 같은 차원 축소와 이산화를 통해, 실시간 환경에서도 은닉 마코프 모델들을 효율적으로 제스처 인식에 이용할 수 있다. 끝으로, 서로 다른 두 가지 데이터 집합을 이용한 실험을 통해, 본 논문에서 제안한 시스템의 높은 인식 성능을 입증해 보인다. This paper presents the design of an arm gesture recognition system using Kinect sensor. A variety of methods have been proposed for gesture recognition, ranging from the use of Dynamic Time Warping(DTW) to Hidden Markov Models(HMM). Our system learns a unique HMM corresponding to each arm gesture from a set of sequential skeleton data. Whenever the same gesture is performed, the trajectory of each joint captured by Kinect sensor may much differ from the previous, depending on the length and/or the orientation of the subject's arm. In order to obtain the robust performance independent of these conditions, the proposed system executes the feature transformation, in which the feature vectors of joint positions are transformed into those of angles between joints. To improve the computational efficiency for learning and using HMMs, our system also performs the k-means clustering to get one-dimensional integer sequences as inputs for discrete HMMs from high-dimensional real-number observation vectors. The dimension reduction and discretization can help our system use HMMs efficiently to recognize gestures in real-time environments. Finally, we demonstrate the recognition performance of our system through some experiments using two different datasets.

Vision-based arm gesture recognition for a long-range human–robot interaction

This paper proposes a vision-based human arm gesture recognition method for human---robot interaction, particularly at a long distance where speech information is not available. We define four meaningful arm gestures for a long-range interaction. The proposed method is capable of recognizing the defined gestures only with 320×240 pixel-sized low-resolution input images captured from a single camera at a long distance, approximately five meters from the camera. In addition, the system differentiates the target gestures from the users' normal actions that occur in daily life without any constraints. For human detection at a long distance, the proposed approach combines results from mean-shift color tracking, short- and long-range face detection, and omega shape detection. The system then detects arm blocks using a background subtraction method with a background updating module and recognizes the target gestures based on information about the region, periodical motion, and shape of the arm blocks. From experiments using a large realistic database, a recognition rate of 97.235% is achieved, which is a sufficiently practical level for various pervasive and ubiquitous applications based on human gestures.

Visual recognition of 3D emblematic gestures in an HMM framework

In human-machine interaction, gestures play an important role as input modality for natural and intuitive interfaces. While the usage of sign languages or crafted command gestures typically requires special user training, the class of gestural actions called “emblems” represents more intuitive yet expressive signs that seem well suited for the task. Following this, an approach for the visual recognition of 3D emblematic arm gestures in a realistic smart room scenario is presented. Hand and head positions are extracted in multiple unsynchronized monocular camera streams, combined to spatiotemporal 3D gesture trajectories and classified in an Hidden Markov Model (HMM) classification and detection framework. The contributions within this article are threefold: Firstly, a solution for the 3D combination of trajectories obtained from unsynchronized cameras and with varying frame rates is proposed. Secondly, the suitability of different alternative feature representations derived from a hand trajectory is assessed, and it is shown that intuitive gestures can be represented by projection on their principal plane of motion. Thirdly, it is demonstrated that a rejection model for gesture spotting and segmentation can be constructed using out-of-domain data. The approach is evaluated on a challenging realistic data set.

A Measurement System for the Recognition of Arm Gestures Using Accelerometers

This paper describes a strategy to measure arm movements using accelerometers for the computer recognition of arm gestures. Gesture recognition is being investigated as an alternative method of computer input for people with severe speech and motor impairment; the emphasis is on the needs of people with athetoid cerebral palsy who have difficulties with existing computer input devices. An initial model-based approach to estimate the kinematic motion of the arm from acceleration measurements is given, followed by the chosen measurement scheme. The current system considers the forearm as a rigid body and uses two data streams derived from four linear accelerometers. By treating these signals as outputs from postulated mechanical models, the data are reduced to two series of step inputs that are appropriate for pattern classification and recognition. Although this does not in any way model the arm, the concepts used are based on the type of driving signals expected in the control of arm gestures. Initial experimental results show that the information content of gestures is preserved by this data parameterization.