Hand Keypoints Research Articles

On attaining user-friendly hand gesture interfaces to control existing GUIs

BackgroundHand gesture interfaces are dedicated programs that principally perform hand tracking and hand gesture prediction to provide alternative controls and interaction methods. They take advantage of one of the most natural ways of interaction and communication, proposing novel input and showing great potential in the field of the human-computer interaction. Developing a flexible and rich hand gesture interface is known to be a time-consuming and arduous task. Previously published studies have demonstrated the significance of the finite-state-machine (FSM) approach when mapping detected gestures to GUI actions. MethodsIn our hand gesture interface, we broadened the FSM approach by utilizing gesture-specific attributes, such as distance between hands, distance from the camera, and time of occurrences, to enable users to perform unique GUI actions. These attributes are obtained from hand gestures detected by the RealSense SDK employed in our hand gesture interface. By means of these gesture-specific attributes, users can activate static gestures and perform them as dynamic gestures. We also provided supplementary features to enhance the efficiency, convenience, and user-friendliness of our hand gesture interface. Moreover, we developed a complementary application for recording hand gestures by capturing hand keypoints in depth and color images to facilitate the generation of hand gesture datasets. ResultsWe conducted a small-scale user survey with fifteen subjects to test and evaluate our hand gesture interface. Anonymous feedback obtained from the users indicates that our hand gesture interface is adequately facile and self-explanatory to use. In addition, we received constructive feedback about minor flaws regarding the responsiveness of the interface. ConclusionsWe proposed a hand gesture interface along with key concepts to attain user-friendliness and effectiveness in the control of existing GUIs.

Hand Gesture Recognition in Complex Background Based on Convolutional Pose Machine and Fuzzy Gaussian Mixture Models

Hand gesture is one of the most intuitive and natural ways for human to communicate with computers, and it has been widely adopted in many human–computer interaction applications. However, it is still a challenging problem when confronted with complex background, illumination variation and occlusion in real-world scenarios. In this paper, a two-stage hand gesture recognition method is proposed to tackle these problems. At the first stage, hand pose estimation is developed to locate the hand keypoints using the convolutional pose machine, which can effectively localize hand keypoints even in a complex background. At the second stage, the Fuzzy Gaussian mixture models (FGMMs) are tailored to reject the nongesture patterns and classify the gestures based on the estimated hand keypoints. Extensive experiments are conducted to evaluate the performance of the proposed method, and the result demonstrates that the proposed algorithm is effective, robust, and satisfactory in real-time scenarios.

Hand sign language recognition using multi-view hand skeleton

Hand sign language recognition from video is a challenging research area in computer vision, which performance is affected by hand occlusion, fast hand movement, illumination changes, or background complexity, just to mention a few. In recent years, deep learning approaches have achieved state-of-the-art results in the field, though previous challenges are not completely solved. In this work, we propose a novel deep learning-based pipeline architecture for efficient automatic hand sign language recognition using Single Shot Detector (SSD), 2D Convolutional Neural Network (2DCNN), 3D Convolutional Neural Network (3DCNN), and Long Short-Term Memory (LSTM) from RGB input videos. We use a CNN-based model which estimates the 3D hand keypoints from 2D input frames. After that, we connect these estimated keypoints to build the hand skeleton by using midpoint algorithm. In order to obtain a more discriminative representation of hands, we project 3D hand skeleton into three views surface images. We further employ the heatmap image of detected keypoints as input for refinement in a stacked fashion. We apply 3DCNNs on the stacked features of hand, including pixel level, multi-view hand skeleton, and heatmap features, to extract discriminant local spatio-temporal features from these stacked inputs. The outputs of the 3DCNNs are fused and fed to a LSTM to model long-term dynamics of hand sign gestures. Analyzing 2DCNN vs. 3DCNN using different number of stacked inputs into the network, we demonstrate that 3DCNN better capture spatio-temporal dynamics of hands. To the best of our knowledge, this is the first time that this multi-modal and multi-view set of hand skeleton features are applied for hand sign language recognition. Furthermore, we present a new large-scale hand sign language dataset, namely RKS-PERSIANSIGN, including 10′000 RGB videos of 100 Persian sign words. Evaluation results of the proposed model on three datasets, NYU, First-Person, and RKS-PERSIANSIGN, indicate that our model outperforms state-of-the-art models in hand sign language recognition, hand pose estimation, and hand action recognition.

SRHandNet: Real-time 2D Hand Pose Estimation with Simultaneous Region Localization.

This paper introduces a novel method for real-time 2D hand pose estimation from monocular color images, which is named as SRHandNet. Existing methods can not time efficiently obtain appropriate results for small hand. Our key idea is to simultaneously regress the hand region of interests (RoIs) and hand keypoints for a given color image, and iteratively take the hand RoIs as feedback information for boosting the performance of hand keypoints estimation with a single encoder-decoder network architecture. Different from previous region proposal network (RPN), a new lightweight bounding box representation, which is called region map, is proposed. The proposed bounding box representation map together with hand keypoints heatmaps are combined into the unified multi-channel feature maps, which can be easily acquired with only one forward network inference and thus improve the runtime efficiency of the network. Our proposed SRHandNet can run at 40fps for hand bounding box detection and up to 30fps accurate hand keypoints estimation under the desktop environment without implementation optimization. Experiments demonstrate the effectiveness of the proposed method. State-of-the-art results are also achieved out competing all recent methods.

Pose Anchor: A Single-Stage Hand Keypoint Detection Network

This paper presents an effective single network for hand keypoint detection, instead of relying on the frequently-used two-stage pipeline consisting of localizing the hand and detecting the key points. Our method trains a fully convolutional neural network in an end-to-end manner, based on a novelly proposed pose anchor network, which can be deemed as an extension of the region proposal network (RPN) in Faster Region-based convolutional network. Moreover, we generate our pose anchor in a data-driven way, i.e., a K-means cluster algorithm based on object keypoint similarity (OKS), instead of manually design. In this way, we can obtain multiple representative pose anchors with various gestures, angles, and scales. By introducing the pose anchor, we are capable of utilizing the prior knowledge of the hand structure, mitigating the problem of occlusion to some extent. We demonstrate the feasibility and effectiveness of our method with extensive experiments on the challenging large-scale multiview 3D hand pose dataset (LSM-HPD) and New Zealand Sign Language Dataset (NZSL).

Feature covariance matrix-based dynamic hand gesture recognition

Over the past 2 decades, vision-based dynamic hand gesture recognition (HGR) has made significant progresses and been widely adopted in many practical applications. Although the advent of RGB-D cameras and deep learning-based methods provides more feasible solutions for HGR, it is still very challenging to satisfy the requirements of both high efficiency and accuracy for real-world HGR systems. In this paper, we propose a novel method using the feature covariance matrix for effective and efficient dynamic HGR. We extract a set of local feature vectors that represent local motion patterns to construct the feature covariance matrix efficiently, which also provides a compact representation of a dynamic hand gesture. By tracking hand keypoints in three successive frames and calculating their motion features, our method can be extended to both 2D dynamic HGR and 3D dynamic HGR. To evaluate the effectiveness of the proposed framework, we perform extensive experiments on three publicly available datasets (one 2D dataset and two 3D datasets). The experimental results demonstrate the effectiveness of our proposed method.

Technology of Operations on Devices in a Virtual Aircraft Cabin

This paper focuses on the manual operations on the virtual cabin devices of a flight simulator developed by virtual reality technology. The system consists of the space positioning system, the data glove, the three-dimensional head-mounted display, the left and right eye stereoscopic imaging computers, the space positioning computer, and the main simulation computer. The hand position, attitude, and shape data are measured through the space positioning system and the data glove while the aircraft position and attitude are calculated through the aircraft simulation system. The hand model is represented by the hand position, attitude, rotation angles of fingers, the key points in the hand and the movement directions of these points .The virtual devices are modeled with object-oriented method encapsulating device shapes, operation status, the hand key points and their movement directions that are affective on the device shapes and operation status. The aircraft attitude is overlaid with the hand attitude in order to regard the moving cabin as static. Thus the coordinate transformations are confined to only the hand model and the device models that operation status changed to avoid the large amount of coordinate transformations and reduce the system time consumption. The application result shows that the virtual operation algorithm has a high efficiency, and the operation precision meets the need of system performance.

A two stage neural network-based personal identification system using handprint

With the increasing emphasis on security, automated personal identification based on biometrics has been receiving extensive attention over the past decades. Handprint identification, as an emerging biometric identification technology, is receiving more and more attention in both research and practical applications as time goes by. In this paper, a novel approach for handprint identification is proposed. Firstly, region of interest is segmented through hand's key points localization, then the Gabor filtering and Zernike moments methods are used to extract the palmprint features. A two stage neural network structure is employed to measure the degree of similarity in the identification stage. The experimental results demonstrate that the proposed approach is effective and feasible.

P56 The clinical trials group's computerized data checking system

With the increasing emphasis on security, automated personal identification based on biometrics has been receiving extensive attention over the past decades. Handprint identification, as an emerging biometric identification technology, is receiving more and more attention in both research and practical applications as time goes by. In this paper, a novel approach for handprint identification is proposed. Firstly, region of interest is segmented through hand's key points localization, then the Gabor filtering and Zernike moments methods are used to extract the palmprint features. A two stage neural network structure is employed to measure the degree of similarity in the identification stage. The experimental results demonstrate that the proposed approach is effective and feasible.