Recognizing Hand Postures Research Articles

A Deep Convolutional Neural Network Approach to Sign Alphabet Recognition

In this paper, we present an application that has been developed to be used as a tool for the purposes of learning sign language for beginners that utilizes hand detection as part of the process. It uses a skin-color modelling technique, such as explicit thresholding in the skin-color space, which is based on modeling skin-color spaces. This predetermined range of skin-colors is used to determine how pixels (hand) will be extracted from non-pixels (background). To classify the images, convolutional neural networks (CNN) were fed the images for the creation of the classifier. The training of the images was done using Keras. A uniform background and proper lighting conditions enabled the system to achieve a test accuracy of 93.67%, of which 90.04% was attributed to ASL alphabet recognition, 93.44% for number recognition and 97.52% recognition of static words, surpassing other studies of the type. An approach which is based on this technique is used for fast computation as well as real-time processing. Deaf-dumb people face a number of social challenges as the communication barrier prevents them from accessing basic and essential services of the life that they are entitled to as members of the hearing community. In spite of the fact that a number of factors have been incorporated into the innovations in the automatic recognition of sign language, an adequate solution has yet to be reached because of a number of challenges. As far as I know, the vast majority of existing works focus on developing vision based recognizers by deriving complex feature descriptors from captured images of the gestures and applying a classical pattern analysis technique. Although utilizing these methods can be effective when dealing with small sign vocabulary captures with a complex and uncontrolled background, they are very limited when dealing with large sign vocabulary. This paper proposes a method for analyzing and representing hand gestures, which acts as the core component of the vocabulary for signing languages, using a deep convolutional neural networks (CNN) architecture. On two publicly accessible datasets (the NUS hand posture dataset and the American fingerspelling A dataset), the method was demonstrated to be more accurate in recognizing hand postures.

Robustify Hand Tracking by Fusing Generative and Discriminative Methods

With the development of virtual reality (VR) technology and its applications in many fields, creating simulated hands in the virtual environment is an e ective way to replace the controller as well as to enhance user experience in interactive processes. Therefore, hand tracking problem is gaining a lot of research attention, making an important contribution in recognizing hand postures as well as tracking hand motions for VR’s input or human machine interaction applications. In order to create a markerless real-time hand tracking system suitable for natural human machine interaction, we propose a new method that combines generative and discriminative methods to solve the hand tracking problem using a single RGBD camera. Our system removes the requirement of the user having to wear to color wrist band and robustifies the hand localization even in di cult tracking scenarios.
 KeywordsHand tracking, generative method, discriminative method, human performance capture
 References[1] Malik, A. Elhayek, F. Nunnari, K. Varanasi, Tamaddon, A. Heloir, D. Stricker, Deephps: End-to-end estimation of 3d hand pose and shape by learning from synthetic depth, CoRR abs/1808.09208, 2018.
 URL http://arxiv.org/abs/1808.09208.
 [2] Glauser, S. Wu, D. Panozzo, O. Hilliges, Sorkine-Hornung, Interactive hand pose estimation using a stretch-sensing soft glove, ACM Trans, Graph. 38(4) (2019) 1-15.[3] Jiang, H. Xia, C. Guo, A model-based system for real-time articulated hand tracking using a simple data glove and a depth camera, Sensors 19 (2019) 4680. https://doi.org/10.3390/s19214680.[4] Cao, G. Hidalgo, T. Simon, S. Wei, Y. Sheikh, Openpose: Realtime multi-person 2d pose estimation using part a nity fields, CoRR abs/1812.08008, 2018.[5] Tagliasacchi, M. Schroder, A. Tkach, S. Bouaziz, M. Botsch, M. Pauly, Robust articulated-icp for real-time hand tracking, Computer Graphics Forum 34, 2015.[6] Qian, X. Sun, Y. Wei, X. Tang, J. Sun, Realtime and robust hand tracking from depth, in: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014.[7] Tomasi, Petrov, Sastry, 3d tracking = classification + interpolation, in: Proceedings Ninth IEEE International Conference on Computer Vision 2 (2003) 1441-1448.[8] Sharp, C. Keskin, D. Robertson, J. Taylor, J. Shotton, D. Kim, C. Rhemann, I. Leichter, A. Vinnikov, Y. Wei, et al., Accurate, robust, and flexible real-time hand tracking, in: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, 2015, pp. 3633-3642.[9] Sridhar, F. Mueller, A. Oulasvirta, C. Theobalt, Fast and robust hand tracking using detection-guided optimization, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.[10] Oikonomidis, N. Kyriazis, A.A. Argyros, Tracking the articulated motion of two strongly interacting hands, in: 2012 IEEE Conference on Computer Vision and Pattern Recognition, 2012, pp. 1862-1869.[11] Melax, L. Keselman, S. Orsten, Dynamics based 3d skeletal hand tracking, CoRR abs/1705.07640, 2017.[12] Wang, S. Paris, J. Popovic, 6d hands: Markerless hand tracking for computer aided design, 2011, pp. 549-558.
 https://doi.org/10.1145/2047196.2047269.[13]Tang, T. Yu, T. Kim, Real-time articulated hand pose estimation using semi-supervised transductive regression forests, in: 2013 IEEE International Conference on Computer Vision, 2013, pp. 3224-3231.[14] Oberweger, P. Wohlhart, V. Lepetit, Generalized feedback loop for joint hand-object pose estimation, 2019, CoRR abs/1903.10883. URL http://arxiv.org/abs/1903.10883.[15] Malik, A. Elhayek, F. Nunnari, K. Varanasi, K. Tamaddon, A. Heloir,´ D. Stricker, Deephps: End-to-end estimation of 3d hand pose and shape by learning from synthetic depth, 2018, pp. 110-119. https://doi.org/10.1109/3DV.2018.00023.[16] A. Mohammed, J.L.M. Islam, A deep learning-based end-to-end composite system for hand detection and gesture recognition, Sensors 19 (2019) 5282. https://doi.org/10.3390/s19235282.

Real-time hand posture recognition using hand geometric features and Fisher Vector

Hand posture recognition (HPR), one of the most effective and intuitive human computer interfaces, has been extensively studied and widely adopted in various multimedia applications. Shape descriptors extracted from a hand contour or silhouette have been proved effective in representing a hand posture. However, it is difficult for these shape-based methods to achieve good balance between accuracy and efficiency. To this end, in this paper, we propose a novel hand shape descriptor based on a set of geometric features (SoGF) and Fisher Vector (FV), for effective and efficient HPR. Three types of geometric features, including distances, angles and curvatures, are extracted from a hand silhouette to form a discriminative local descriptor, and FV is adopted to encode the set of local descriptors for compact hand shape representation. To recognize hand postures, we construct a classifier using a multi-class Support Vector Machine (SVM) with FVs as input. The experimental results on four public HPR datasets show that the proposed method can achieve the mean accuracy of the state-of-the-art methods in real time.

View-invariant representation of hand postures in the human lateral occipitotemporal cortex

Understanding other people's actions and mental states includes the interpretation of body postures and movements. In particular, hand postures are an important channel to signal both action and communicative intentions. Recognizing hand postures is computationally challenging because hand postures often differ only in the subtle configuration of relative finger positions and because visual characteristics of hand postures change across viewpoints. To allow for accurate interpretation, the brain needs to represent hand postures in a view-invariant but posture-specific manner. Here we test for such representations in hand-, body-, and object-selective regions of the lateral occipitotemporal cortex (LOTC). We used multivariate pattern analysis of fMRI data to test for view-specific and view-invariant representations of individual hand postures, separately for two domains: action-related postures (e.g., a precision grasp) and communicative postures (e.g., thumbs up). Results showed that hand-selective LOTC, but not nearby body- and object-selective LOTC, represented hand postures in a view-invariant manner, with relatively similar activity patterns to the same hand posture seen from different viewpoints. View invariance was equally strong for action and communicative postures. By contrast, object-selective cortex represented hand postures in a view-specific manner. These results indicate a role for hand-selective LOTC in solving the view-invariance problem for individual hand postures. View-invariant representations of hand postures in this region may then be accessed and further interpreted by multiple downstream systems to inform high-level judgments related to action understanding, emotion recognition, and non-verbal communication.

A bionic hand controlled by hand gesture recognition based on surface EMG signals: A preliminary study

A bionic hand with fine motor ability could be a favorable option for replacing the human hand when performing various operations. Myoelectric control has been widely used to recognize hand movements in recent years. However, most of the previous studies have focused on whole-hand movements, with only a few investigating subtler motions. The aim of this study was to construct a prototype system for recognizing hand postures with the aim of controlling a bionic hand by analyzing sEMG signals measured at the flexor digitorum superficialis and extensor digitorum muscles. We adopted multiple features commonly used in previous studies—mean absolute value, zero crossing, slope sign change, and waveform length—in the algorithm for extracting hand-posture features, and the k-nearest-neighbors (KNN) algorithm as the classifier to perform hand-posture recognition. The bionic hand was controlled by an Arduino microprocessor, which converted the signals received from the classification process that were fed to the servo motors controlling the bionic fingers. We constructed a two-channel sEMG pattern-recognition system that can identify human hand postures and control a homemade bionic hand to perform corresponding hand postures. The KNN approach was able to recognize four different hand postures with a classification accuracy of 94% in the online experiment by using the channel combination. Moreover, the experimental tests show that the bionic hand could faithfully imitate the hand postures of the human hand. This study has bridged the gap between the features of sEMG signals of fingers and the postures of a bionic hand.

Hand Gesture Recognition for Thai Sign Language in Complex Background Using Fusion of Depth and Color Video

Hand detection and gesture recognition are the active research area in the computer vision. The main purpose to develop the sign language recognition and Human Computer Interaction (HCI). This article investigates and develops the technique to recognize hand posture of Thai sign language in a complex background using fusion of depth and color video. The new technology of sensors, such as the Microsoft Kinect, recently provides the depth video which helps researchers to find the hand position in the scene. This advantage is used to segment the hand sign in the color video without the environment interference such as skin color background. The histograms of oriented gradients are used to extract the image features of hand sign. These features are then pass to the artificial neural network for training and recognition. The result showed that the proposed method is robust to detect the hand gestures in the complex background. It provides the accuracy recognition for the Thai fingerspelling of 84.05%

Depth based Sign Language Recognition System Using SVM

In this paper, we propose a sign language recognition system using an SVM (Support Vector Machine) and a depth camera. In particular, we focus on the Korean sign language. For the sign language system, we suggest two methods, one for the hand feature extraction stage and the other for the recognition stage. Hand features consist of the number of fingers, finger length, palm radius, and hand direction. To extract hand features, we use Distance Transform and a hand skeleton. This method is more accurate and faster than a traditional method that uses contours. To recognize hand posture, we develop a decision tree with hand features. For more accuracy, we use SVM to determine the threshold value in the decision tree. In the experiment results, we show that the suggested method is more accurate and faster when extracting hand features and recognizing hand postures.

깊이 카메라와 SVM을 이용한 수화 인식 시스템

In this paper, we propose a sign language recognition system using SVM and depth camera. Especially, we focus on the Korean sign language. For the sign language system, we suggest two methods, one in hand feature extraction stage and the other in recognition stage. Hand features are consisted of the number of fingers, finger length, radius of palm, and direction of the hand. To extract hand features, we use Distance Transform and make hand skeleton. This method is more accurate than a traditional method which uses contours. To recognize hand posture, we develop the decision tree with the hand features. For ∙제1저자 : 김기상 ∙교신저자 : 최형일 ∙투고일 : 2014. 9. 19, 심사일 : 2014. 9. 29, 게재확정일 : 2014. 10 28 * 숭실대학교 미디어학과(School of Media, Soongsil University) ※ 본 연구는 서울시 산학연 협력사업(SS110013)과 2013년도 정부(미래창조과학부)의 재원으로 한국연구재단의 지원을 받아 수행된 기초연구사업임 (No.2013R1A1A2012012) 64 Journal of The Korea Society of Computer and Information November 2014 more accuracy, we use SVM to determine the threshold value in the decision tree. In the experimental results, we show that the suggested method is more accurate and faster when extracting hand features a recognizing hand postures. ▸

피부색 및 깊이정보를 이용한 영역채움 기반 손 분리 기법

Extracting hand region from images is the first part in the process to recognize hand posture and gesture interaction. Therefore, a good segmenting method is important because it determines the overall performance of hand recognition systems. Conventional hand segmentation researches were prone to changing illumination conditions or limited to the ability to detect multiple people. In this paper, we propose a robust technique based on the fusion of skin-color data and depth information for hand segmentation process. The proposed algorithm uses skin-color data to localize accurate seed location for region-growing from a complicated background. Based on the seed location, our algorithm adjusts each detected blob to fill up the hole region. A region-growing algorithm is applied to the adjusted blob boundary at the detected depth image to obtain a robust hand region against illumination effects. Also, the resulting hand region is used to train our skin-model adaptively which further reduces the effects of changing illumination. We conducted experiments to compare our results with conventional techniques which validates the robustness of the proposed algorithm and in addition we show our method works well even in a counter light condition.

건구동식 로봇 의수용 착용형 인터페이스

This paper proposes a wearable interface for a tendon-driven robotic hand prosthesis. The proposed interface is composed of a dataglove to measure finger and wrist joint angle, and a micro-control board with a wireless RF module. The interface is used for posture control of the robotic hand prosthesis. The measured joint angles by the dataglove are transferred to the main controller via the wireless module. The controller works for directly controlling the joint angle of the hand or for recognizing hand postures using a pattern recognition method such as LDA and k-NN. The recognized hand postures in this study are the paper, the rock, the scissors, the precision grasp, and the tip grasp. In experiments, we show the performances of the wearable interface including the pattern recognition method.

A person independent system for recognition of hand postures used in sign language

We present a novel user independent framework for representing and recognizing hand postures used in sign language. We propose a novel hand posture feature, an eigenspace Size Function, which is robust to classifying hand postures independent of the person performing them. An analysis of the discriminatory properties of our proposed eigenspace Size Function shows a significant improvement in performance when compared to the original unmodified Size Function. We describe our support vector machine based recognition framework which uses a combination of our eigenspace Size Function and Hu moments features to classify different hand postures. Experiments, based on two different hand posture data sets, show that our method is robust at recognizing hand postures independent of the person performing them. Our method also performs well compared to other user independent hand posture recognition systems.

Feature alignment approach for hand posture recognition based on curvature scale space

One of the most important aspects of gesture recognition is recognizing hand postures. Much research has been devoted to extracting reliable features for hand posture recognition. In this paper, a new feature alignment approach for hand posture recognition based on curvature scale space (CSS) is presented. The basis point for alignment is based on the two-dimensional distribution of a coordinate-peak set of the CSS image instead of on the coordinate with the maximal peak. A convolution operation is performed with the sequence of a coordinate-peak set and a predefined function. The coordinate with the maximal convolution value is designated as a basis point for aligning the CSS features of the hand posture. Results show that the proposed approach performs well in recognizing hand postures. Furthermore, the proposed approach is more accurate than previous methods based on conventional features.

Towards Sociable Virtual Humans: Multimodal Recognition of Human Input and Behavior

One of the biggest obstacles for constructing effective sociable virtual humans lies in the failure of machines to recognize the desires, feelings and intentions of the human user. Virtual humans lack the ability to fully understand and decode the communication signals human users emit when communicating with each other. This article describes our research in overcoming this problem by developing senses for the virtual humans which enables them to hear and understand human speech, localize the human user in front of the display system, recognize hand postures and to recognize the emotional state of the human user by classifying facial expression. We report on the methods needed to perform these tasks in real-time and conclude with an outlook on promising research issues of the future.

Cognitive representations of hand posture in ideomotor apraxia

Ideomotor apraxia (IM) is a disorder of skilled action characterized by spatiotemporal errors in pantomiming object use and in using objects. Recent evidence suggests that at least some patients with IM may exhibit particular deficits in forming hand configurations appropriate for object use. Sirigu et al. [Cortex 31 (1995) 41] reported an apraxic who positioned her hand inappropriately when attempting to use objects in accordance with stored knowledge of object-specific manipulation, but in reaching tasks could grasp the same objects appropriately in response to their structure. To this point, however, apraxics’ ability to respond to functional and structural attributes of objects has not been empirically assessed. We investigated the hypothesis that patients with IM ( n=9) due to left inferior parietal damage would be impaired in producing and recognizing hand postures associated with familiar objects, indicating deficient memorial representations for object-specific hand postures. In contrast, we predicted relatively unimpaired ability to produce and recognize appropriate hand postures with novel objects, indicating integrity of “on line” spatiomotor procedures coding hand position in response to object structure. Apraxics’ performance was contrasted with 10 healthy controls and 8 brain-lesioned non-apraxics. Consistent with our predictions, the apraxics responded abnormally with familiar objects but normally in recognizing hand postures appropriate for novel objects. In addition, performance with objects calling for a prehensile response (pinch or clench) was superior to that with objects evoking a non-prehensile response (palm or poke). These data suggest degradation of inferior parietally-mediated representations of the precise hand postures associated with familiar objects. Furthermore, they are consistent with possible over-reliance upon dorsal system procedures for calculating precision and power grip in response to object structure.