Hand Pose Recognition Research Articles

Dual Leap Motion Controller 2: A Robust Dataset for Multi-view Hand Pose Recognition

This paper presents Multi-view Leap2 Hand Pose Dataset (ML2HP Dataset), a new dataset for hand pose recognition, captured using a multi-view recording setup with two Leap Motion Controller 2 devices. This dataset encompasses a diverse range of hand poses, recorded from different angles to ensure comprehensive coverage. The dataset includes real images with the associated precise and automatic hand properties, such as landmark coordinates, velocities, orientations, and finger widths. This dataset has been meticulously designed and curated to maintain a balance in terms of subjects, hand poses, and the usage of right or left hand, ensuring fairness and parity. The content includes 714,000 instances from 21 subjects of 17 different hand poses (including real images and 247 associated hand properties). The multi-view setup is necessary to mitigate hand occlusion phenomena, ensuring continuous tracking and pose estimation required in real human-computer interaction applications. This dataset contributes to advancing the field of multimodal hand pose recognition by providing a valuable resource for developing advanced artificial intelligence human computer interfaces.

A Computational Approach to Hand Pose Recognition in Early Modern Paintings.

Hands represent an important aspect of pictorial narration but have rarely been addressed as an object of study in art history and digital humanities. Although hand gestures play a significant role in conveying emotions, narratives, and cultural symbolism in the context of visual art, a comprehensive terminology for the classification of depicted hand poses is still lacking. In this article, we present the process of creating a new annotated dataset of pictorial hand poses. The dataset is based on a collection of European early modern paintings, from which hands are extracted using human pose estimation (HPE) methods. The hand images are then manually annotated based on art historical categorization schemes. From this categorization, we introduce a new classification task and perform a series of experiments using different types of features, including our newly introduced 2D hand keypoint features, as well as existing neural network-based features. This classification task represents a new and complex challenge due to the subtle and contextually dependent differences between depicted hands. The presented computational approach to hand pose recognition in paintings represents an initial attempt to tackle this challenge, which could potentially advance the use of HPE methods on paintings, as well as foster new research on the understanding of hand gestures in art.

A Novel Heteromorphic Ensemble Algorithm for Hand Pose Recognition

Imagining recognition of behaviors from video sequences for a machine is full of challenges but meaningful. This work aims to predict students’ behavior in an experimental class, which relies on the symmetry idea from reality to annotated reality centered on the feature space. A heteromorphic ensemble algorithm is proposed to make the obtained features more aggregated and reduce the computational burden. Namely, the deep learning models are improved to obtain feature vectors representing gestures from video frames and the classification algorithm is optimized for behavior recognition. So, the symmetric idea is realized by decomposing the task into three schemas including hand detection and cropping, hand joints feature extraction, and gesture classification. Firstly, a new detector method named YOLOv4-specific tiny detection (STD) is proposed by reconstituting the YOLOv4-tiny model, which could produce two outputs with some attention mechanism leveraging context information. Secondly, the efficient pyramid squeeze attention (EPSA) net is integrated into EvoNorm-S0 and the spatial pyramid pool (SPP) layer to obtain the hand joint position information. Lastly, the D–S theory is used to fuse two classifiers, support vector machine (SVM) and random forest (RF), to produce a mixed classifier named S–R. Eventually, the synergetic effects of our algorithm are shown by experiments on self-created datasets with a high average recognition accuracy of 89.6%.

Design of a Hand Pose Recognition System for Mobile and Embedded Devices

Today, smart devices such smart watches and smart cell phones are becoming ever-present in all fields that influence the quality of life of the modern people. These on-board systems have revolutionized the behavior of human beings and especially their way of communicating. In this context and to improve the experience of using these devices, we aim to develop a system that recognizes hand poses in the air by a smart device. In this work, the system is based on Histogram of Oriented Gradient (HOG) features and Support Vector Machine (SVM) classifier. The impact of using HOG and SVM on mobile devices is studied. To carry out this study, we used an improved version of the "NUS I" dataset and obtained a recognition rate of approximately 94%. In addition, we conducted run speed experiments on various mobile devices to study the impact of this task on this embedded platform. The main contribution of this work is to test the impact of using the HOG descriptor and the SVM classifier in terms of recognition rate and execution time on low-end smartphones.Today, smart devices such smart watches and smart cell phones are becoming ever-present in all fields that influence the quality of life of the modern people. These on-board systems have revolutionized the behavior of human beings and especially their way of communicating. In this context and to improve the experience of using these devices, we aim to develop a system that recognizes hand poses in the air by a smart device. In this work, the system is based on Histogram of Oriented Gradient (HOG) features and Support Vector Machine (SVM) classifier. The impact of using HOG and SVM on mobile devices is studied. To carry out this study, we used an improved version of the "NUS I" dataset and obtained a recognition rate of approximately 94%. In addition, we conducted run speed experiments on various mobile devices to study the impact of this task on this embedded platform. The main contribution of this work is to test the impact of using the HOG descriptor and the SVM classifier in terms of recognition rate and execution time on low-end smartphones.

Towards pen‐holding hand pose recognition: A new benchmark and a coarse‐to‐fine PHHP recognition network

Hand pose recognition has been one of the most fundamental tasks in computer vision and pattern recognition, and substantial effort has been devoted to this field. However, owing to lack of public large-scale benchmark dataset, there is little literature to specially study pen-holding hand pose (PHHP) recognition. As an attempt to fill this gap, in this paper, a PHHP image dataset, consisting of 18,000 PHHP samples is established. To the best of the authors’ knowledge, this is the largest vision-based PHHP dataset ever collected. Furthermore, the authors design a coarse-to-fine PHHP recognition network consisting of a coarse multi-feature learning network and a fine pen-grasping-specific feature learning network, where the coarse learning network aims to extensively exploit the multiple discriminative features by sharing a hand-shape-based spatial attention information, and the fine learning network further learns the pen-grasping-specific features by embedding a couple of convolutional block attention modules into three convolution blocks models. Experimental results show that the authors’ proposed method can achieve a very competitive PHHP recognition performance when compared with the baseline recognition models.

Adaptive sensor fusion labeling framework for hand pose recognition in robot teleoperation

Purpose The purpose of this paper is to mainly center on the touchless interaction between humans and robots in the real world. The accuracy of hand pose identification and stable operation in a non-stationary environment is the main challenge, especially in multiple sensors conditions. To guarantee the human-machine interaction system’s performance with a high recognition rate and lower computational time, an adaptive sensor fusion labeling framework should be considered in surgery robot teleoperation. Design/methodology/approach In this paper, a hand pose estimation model is proposed consisting of automatic labeling and classified based on a deep convolutional neural networks (DCNN) structure. Subsequently, an adaptive sensor fusion methodology is proposed for hand pose estimation with two leap motions. The sensor fusion system is implemented to process depth data and electromyography signals capturing from Myo Armband and leap motion, respectively. The developed adaptive methodology can perform stable and continuous hand position estimation even when a single sensor is unable to detect a hand. Findings The proposed adaptive sensor fusion method is verified with various experiments in six degrees of freedom in space. The results showed that the clustering model acquires the highest clustering accuracy (96.31%) than other methods, which can be regarded as real gestures. Moreover, the DCNN classifier gets the highest performance (88.47% accuracy and lowest computational time) than other methods. Originality/value This study can provide theoretical and engineering guidance for hand pose recognition in surgery robot teleoperation and design a new deep learning model for accuracy enhancement.

A Novel Method for CNN Training Using Existing Color Datasets for Classifying Hand Postures in Bayer Images

Deep learning has revolutionized the computer vision by achieving human level performance on classification and detection of the objects in the images. These algorithms are often trained using supervised learning methods which require a labeled training data set. Since the labeling requires human intervention, the human interpretable tri-color images are needed for this purpose even though the image sensors capture subsampled images in Bayer format. For inference also, the Bayer format images are converted to tri-color images which require sophisticated image processing pipeline. This conversion seems unnecessary given the proven ability of the deep networks to learn in multiple different scenarios. Moreover, the image processing pipeline can be simplified by directly using the Bayer images for computer vision tasks. Availability of the dataset for training the deep networks remains the challenge for building such systems; so, a methodology is needed to use the color images for training DNNs which can classify the Bayer images for a given application. The existing tri-color images from training datasets can be subsampled in pseudo-Bayer pattern for training such deep neural networks. This paper presents the methodology and results of one such method adopted for hand pose recognition application using SqueezeNet with encouraging performance. Using this methodology, we achieved 100% accuracy in nine out of ten hand postures with pseudo-Bayer images and in six out of ten hand postures with real Bayer images.

DenseAttentionSeg: Segment hands from interacted objects using depth input

Hand segmentation is an important task in computer vision, which is usually the foundation of hand pose recognition, hand tracking, and reconstruction. For hand segmentation, it is more challenging when the hand is interacting with objects, but handling interacting motions is more important for applications like HCI and VR. In this paper, we propose a real-time DNN-based technique to segment hand and object in interacting motions from a single depth input. Our model is called DenseAttentionSeg, which contains a dense attention mechanism which effectively fuses information in different scales and improves the quality of result with skip-connections. Besides, we introduce a contour loss in model training, which helps to generate accurate hand and object boundaries. Finally, we propose our InterSegHands dataset, a fine-scale hand segmentation dataset containing about 52k depth maps of hand-object interactions, with the ground truth segmentation masks. Our experiments evaluate the effectiveness of our techniques and datasets, and indicate that our method outperforms the current state-of-the-art deep segmentation methods in handling hand-object interactions.

DeepDynamicHand: A Deep Neural Architecture for Labeling Hand Manipulation Strategies in Video Sources Exploiting Temporal Information.

Humans are capable of complex manipulation interactions with the environment, relying on the intrinsic adaptability and compliance of their hands. Recently, soft robotic manipulation has attempted to reproduce such an extraordinary behavior, through the design of deformable yet robust end-effectors. To this goal, the investigation of human behavior has become crucial to correctly inform technological developments of robotic hands that can successfully exploit environmental constraint as humans actually do. Among the different tools robotics can leverage on to achieve this objective, deep learning has emerged as a promising approach for the study and then the implementation of neuro-scientific observations on the artificial side. However, current approaches tend to neglect the dynamic nature of hand pose recognition problems, limiting the effectiveness of these techniques in identifying sequences of manipulation primitives underpinning action generation, e.g., during purposeful interaction with the environment. In this work, we propose a vision-based supervised Hand Pose Recognition method which, for the first time, takes into account temporal information to identify meaningful sequences of actions in grasping and manipulation tasks. More specifically, we apply Deep Neural Networks to automatically learn features from hand posture images that consist of frames extracted from grasping and manipulation task videos with objects and external environmental constraints. For training purposes, videos are divided into intervals, each associated to a specific action by a human supervisor. The proposed algorithm combines a Convolutional Neural Network to detect the hand within each video frame and a Recurrent Neural Network to predict the hand action in the current frame, while taking into consideration the history of actions performed in the previous frames. Experimental validation has been performed on two datasets of dynamic hand-centric strategies, where subjects regularly interact with objects and environment. Proposed architecture achieved a very good classification accuracy on both datasets, reaching performance up to 94%, and outperforming state of the art techniques. The outcomes of this study can be successfully applied to robotics, e.g., for planning and control of soft anthropomorphic manipulators.

A Study on Real-time Hand Pose Recognition and its Classification

Hand gesture recognition is a problem. Many computer and human computing have had an interactive community for years, add-on components or existing solutions to users, causing additional computational time requirements. These limitations are mainly human hands due to the camera’s high performance and limited field of view. This work uses a novel method to identify hand-folding metric shape descriptors to generate a 3D cylindrical map of hand-folding with depth information and size and rotation-invariant hand kinematics, and we achieved robust pose recognition in real time. In this paper, new hand force geometry features are proposed for hand pose recognition using the approach. A skeleton arm there was a model built transmits and analyze abduction Finger movements and these variables were added to transform the dimensional probability distribution. A skeleton model was concretized and these parameters were updated to change the dimensional probability distribution. The proposed algorithm very Strong irregular arm division and lateral movement Test results of fingers Show that the proposed method optimal human computing related applications interaction Virtual reality, autonomous driving systems, human machine hand gesture recognition augmented robotics, interfaces and other new, emerging technologies have become an interesting research topic in recent years. Although there are many approaches to a strong authentication System, gesture recognition is based on visual perception, sensors or devices like electronic gloves have many advantages. 10 Visualization to Recognize Hand Positions in an Embedded Computer This article describes a basic recognition implementation. Hand detection is a light curve achieved using a tracking algorithm and neural network. The results show Low power consumption and real-time response accuracy. From this, the proposed system can use to greater extent.

From Linear Classifier to Convolutional Neural Network for Hand Pose Recognition

Recently gathered image datasets and new capabilities of high performance computing systems allowed developing new artificial neural network models and training algorithms. Using the new machine learning models, computer vision tasks can be accomplished based on the raw values of image pixels, instead of specific features. The principle of operation of deep artificial neural networks is more and more resembling of what we believe to be happening in the human visual cortex. In this paper we build up an understanding of convolutional neural networks through investigating supervised machine learning methods suchas K-Nearest Neighbors, linear classifiers and fully connected neural networks. We provide examples and accuracy results based on our implementation aimed for the problem of hand pose recognition.

Enhanced hand part classification from a single depth image using random decision forests

Hand pose recognition has received increasing attention in an area of human–computer interaction. With the recent spread of many low‐cost three‐dimensional (3D) cameras, research into understanding more natural gestures has increased. In this study, the authors present a method for hand part classification and joint estimation from a single depth image. They apply random decision forests (RDFs) for hand part classification. Foreground pixels in the hand image are estimated by RDF. Then hand joints are estimated based on the classified hand parts. They suggest a robust feature extraction method for per‐pixel classification, which enhances the accuracy of hand part classification. They also propose a tree selection algorithm using legacy trained RDF to classify unseen test data. Selecting trees using the proposed method show better performance than using all the trees as in conventional method. Depth images and label images synthesised by 3D hand mesh model were used for training forests and algorithm verification. The authors’ experiments show that the enhanced algorithm outperforms the state‐of‐the‐art method in accuracy.

Towards Naturally Grabbing and Moving Objects in VR

Many VR applications require picking and moving of virtual objects. Many gesture based solutions do not work reliably, whereas controller based methods are not as natural as hand pose recognition. Tethered controllers add the issue of a cable being in the user’s way. We developed a gesture interface using a Leap Motion finger tracker attached to an Oculus Rift DK2 and implemented three ways of interacting with objects: innate pinching, magnetic force, and a physical button attached to the index finger. We built a virtual reality test scenario, in which the user needs to move virtual objects between shelves to sort them. Initial testing shows that grabbing with the button works better than the other two more natural methods. Besides the user interaction techniques, we also report on our practical experiences using Oculus Rift, Leap Motion and the button with the Unity 3D development platform.

Hand pose recognition from monocular images by geometrical and texture analysis

One challenging research problem of hand pose recognition is the accurate detection of finger abduction and flexion with a single camera. The detection of flexion movements from a 2D image is difficult, because it involves estimation of finger movements along the optical axis of the camera (z direction). In this paper, a novel approach to hand pose recognition is proposed. We use the concept of object-based video abstraction for segmenting the frames into video object planes (VOPs), as used in MPEG-4, with each VOP corresponding to one semantically meaningful hand position. Subsequently, a particular hand pose is recognized by analyzing the key geometrical features and the textures of the hand. The abduction and adduction movements of the fingers are analyzed by considering a skeletal model. Probabilistic distributions of the geometric features are considered for modeling intra-class abduction and adduction variations. Additionally, gestures differing in flexion positions of the fingers are classified by texture analysis using homogeneous texture descriptors (HTD). Finally, hand poses are classified based on proximity measurement by considering the intra-class abduction and adduction and/or inter-class flexion variations. Experimental results show the efficacy of our proposed hand pose recognition system. The system achieved a 99% recognition rate for one-hand poses and a 97% recognition rate for two-hand poses.

Spherical blurred shape model for 3-D object and pose recognition: quantitative analysis and HCI applications in smart environments.

The use of depth maps is of increasing interest after the advent of cheap multisensor devices based on structured light, such as Kinect. In this context, there is a strong need of powerful 3-D shape descriptors able to generate rich object representations. Although several 3-D descriptors have been already proposed in the literature, the research of discriminative and computationally efficient descriptors is still an open issue. In this paper, we propose a novel point cloud descriptor called spherical blurred shape model (SBSM) that successfully encodes the structure density and local variabilities of an object based on shape voxel distances and a neighborhood propagation strategy. The proposed SBSM is proven to be rotation and scale invariant, robust to noise and occlusions, highly discriminative for multiple categories of complex objects like the human hand, and computationally efficient since the SBSM complexity is linear to the number of object voxels. Experimental evaluation in public depth multiclass object data, 3-D facial expressions data, and a novel hand poses data sets show significant performance improvements in relation to state-of-the-art approaches. Moreover, the effectiveness of the proposal is also proved for object spotting in 3-D scenes and for real-time automatic hand pose recognition in human computer interaction scenarios.

Hand shape classification using depth data for unconstrained 3D interaction

In this paper, we introduce a novel method for view-independent hand pose recognition from depth data. The proposed approach, which does not rely on color information, provides an estimation of the shape and orientation of the user's hand without constraining him/her to maintain a fixed position in the 3D space. We use principal component analysis to estimate the hand orientation in space, Flusser moment invariants as image features and two SVM-RBF classifiers for visual recognition. Moreover, we describe a novel weighting method that takes advantage of the orientation and velocity of the user's hand to assign a score to each hand shape hypothesis. The complete processing chain is described and evaluated in terms of real-time performance and classification accuracy. As a case study, it has also been integrated into a touchless interface for 3D medical visualization, which allows users to manipulate 3D anatomical parts with up to six degrees of freedom. Furthermore, the paper discusses the results of a user study aimed at assessing if using hand velocity as an indicator of the user's intentionality in changing hand posture results in an overall gain in the classification accuracy. The experimental results show that, especially in the presence of out-of-plane rotations of the hand, the introduction of the velocity-based weighting method produces a significant increase in the pose recognition accuracy.

A Real-Time Hand Pose Recognition Method with Hidden Finger Prediction

ABSTRACT In this paper, we present a real-time hand pose recognition method to provide an intuitive user interface through hand poses or movements without a keyboard and a mouse. For this, the areas of right and left hands are segmented from the depth camera image, and noise removal is performed. Then, the rotation angle and the centroid point of each hand area are calculated. Subsequently, a circle is expanded at regular intervals from a centroid point of the hand to detect joint points and end points of the finger by obtaining the midway points of the hand boundary crossing. Lastly, the matching between the hand information calculated previously and the hand model of previous frame is performed, and the hand model is recognized to update the hand model for the next frame. This method enables users to predict the hidden fingers through the hand model information of the previous frame using temporal coherence in consecutive frames. As a result of the experiment on various hand poses with the hidden fingers using both hands, the accuracy showed over 95% and the performance indicated over 32 fps. The proposed method can be used as a contactless input interface in presentation, advertisement, education, and game applications. Keywords : Hand Pose Recognition, Hand-based User Interface, Hand Model

은닉된 손가락 예측이 가능한 실시간 손 포즈 인식 방법

본 논문에서는 키보드나 마우스를 이용하지 않고 손 포즈나 동작으로 직관적인 사용자 인터 페이스를 제공하기 위한 실시간 손 포즈 인식 방법을 제안한다. 먼저 깊이 카메라 입력영상에서 왼손과 오른손의 영역을 분할 및 잡음 보정 후 각 손 영역에 대하여 손 회전각과 손 중심점을 계산한다. 그리고 손 중심점에서 일정간격으로 원을 확장해 나가면서 손 경계 교차점의 중간 지점을 구해 손가락 관절점과 끝점을 검출한다. 마지막으로 앞서 구한 손 정보와 이전 프레임의 손 모델간의 매칭을 수행하여 손 포즈를 인식한 후 다음 프레임을 위하여 손 모델을 갱신한다. 본 방법은 연속된 프레임간의 시간 일관성을 이용하여 이전 프레임의 손 모델 정보를 통하여 은닉된 손가락의 예측이 가능하다. 양손을 사용하여 은닉된 손가락을 가진 다양한 손 포즈에 대해 실험한 결과 제안 방법은 평균 95% 이상의 정확도로 32 fps 이상의 성능을 보였다. 제안 방법은 프리젠테이션, 광고, 교육, 게임 등의 응용분야에서 비접촉식 입력 인터페이스로 사용될 수 있다.

A Framework for Gesture Interface Design

Gesture interfaces have long been pursued in the context of portable computing and immersive environments. However, such interfaces have been difficult to build, in part due to a lack of frameworks for their design and implementation. This paper presents a framework for automatically producing a gesture interface based on a simple interface description. Rather than defining hand poses in a low level high-dimensional joint angle space, we describe and recognize gestures in a “lexical” space, in which each hand pose is decomposed into elements in a finger-pose alphabet. The alphabet and underlying rules are defined as a gesture notating system called GeLex. By implementing a generic hand pose recognition algorithm, and a mechanism to adapt it to a specific application based on an interface description, developing a gesture interface becomes straightforward.

Visual gesture interfaces for virtual environments

Virtual environments provide a whole new way of viewing and manipulating 3D data. Current technology moves the images out of desktop monitors and into the space immediately surrounding the user. Users can literally put their hands on the virtual objects. Unfortunately, techniques for interacting with such environments are yet to mature. Gloves and sensor-based trackers are unwieldy, constraining and uncomfortable to use. A natural, more intuitive method of interaction would be to allow the user to grasp objects with their hands and manipulate them as if they were real objects. We are investigating the use of computer vision in implementing a natural interface based on hand gestures. A framework for a gesture recognition system is introduced along with results of experiments in colour segmentation, feature extraction and template matching for finger and hand tracking, and simple hand pose recognition. Implementation of a gesture interface for navigation and object manipulation in virtual environments is presented.