Hand Segmentation Research Articles

Left/right hand segmentation in egocentric videos

Wearable cameras allow people to record their daily activities from a user-centered (First Person Vision) perspective. Due to their favorable location, wearable cameras frequently capture the hands of the user, and may thus represent a promising user-machine interaction tool for different applications. Existent First Person Vision methods handle hand segmentation as a background-foreground problem, ignoring two important facts: i) hands are not a single “skin-like” moving element, but a pair of interacting cooperative entities, ii) close hand interactions may lead to hand-to-hand occlusions and, as a consequence, create a single hand-like segment. These facts complicate a proper understanding of hand movements and interactions. Our approach extends traditional background-foreground strategies, by including a hand-identification step (left-right) based on a Maxwell distribution of angle and position. Hand-to-hand occlusions are addressed by exploiting temporal superpixels. The experimental results show that, in addition to a reliable left/right hand-segmentation, our approach considerably improves the traditional background-foreground hand-segmentation.

A new method for recognizing hand configurations of Brazilian gesture language.

This paper describes a new method for recognizing hand configurations of the Brazilian Gesture Language - LIBRAS - using depth maps obtained with a Kinect® camera. The proposed method comprised three phases: hand segmentation, feature extraction, and classification. The segmentation phase is independent from the background and depends only on pixel depth information. Using geometric operations and numerical normalization, the feature extraction process was done independent from rotation and translation. The features are extracted employing two techniques: (2D)2LDA and (2D)2PCA. The classification is made with a novelty classifier. A robust database was constructed for classifier evaluation, with 12,200 images of LIBRAS and 200 gestures of each hand configuration. The best accuracy obtained was 95.41%, which was greater than previous values obtained in the literature.

Movement Epenthesis Detection for Continuous Sign Language Recognition

AbstractAutomatic sign language recognition (SLR) is a current area of research as this is meant to serve as a substitute for sign language interpreters. In this paper, we present the design of a continuous SLR system that can extract out the meaningful signs and consequently recognize them. Here, we have used height of the hand trajectory as a salient feature for separating out the meaningful signs from the movement epenthesis patterns. Further, we have incorporated a unique set of spatial and temporal features for efficient recognition of the signs encapsulated within the continuous sequence. The implementation of an efficient hand segmentation and hand tracking technique makes our system robust to complex background as well as background with multiple signers. Experiments have established that our proposed system can identify signs from a continuous sign stream with a 92.8% spotting rate.

Structural evolution of poly(l-lactide) block upon heating of the glassy ABA triblock copolymers containing poly(l-lactide) A blocks

Abstract Two triblock ABA copolymers poly( l -lactide- b -dimethylsiloxane- b - l -lactide) (PLLA- b -PDMS- b -PLLA) and poly( l -lactide- b -ethylene glycol- b - l -lactide) (PLLA- b -PEG- b -PLLA) containing poly( l -lactide) were synthesized by ring opening polymerization of l -lactide using bis(hydroxyalkyl) terminated PDMS and hydroxyl end-capped polyethylene glycol as macroinitiators, respectively. Both the triblock copolymers were melt-quenched to −40 °C to prepare the amorphous samples. The melt morphology of the triblock copolymers was preserved upon cooling the melt to −40 °C. PLLA and PDMS blocks are immiscible in the melt, and the amorphous triblock copolymer shows two distinct glass transition temperatures ( T g ) on heating. On the other hand, PLLA and PEG segments are miscible and the triblock copolymer shows the single T g . Structural evolution of PLLA during heating of the amorphous ABA triblock copolymers has been investigated by measuring the variable temperature small-angle and wide-angle X-ray scattering (SAXS and WAXS) and FTIR spectra. In the case of PLLA- b -PDMS- b -PLLA triblock copolymer (immiscible system), the mesophase of PLLA was found to appear just above the T g of PLLA block (∼45 °C), and on further heating the mesophase changed to the regular α form at around 90 °C. On the other hand, in PLLA- b -PEG- b -PLLA (miscible system), the mesophase of PLLA was found to appear at lower temperature i.e. −20 °C, because of the enhanced molecular mobility of PLLA chains in the presence of PEG. However, irrespective of the mesophase formation temperature, the ordered α form has appeared at around 90 °C. In this way, during heating of the amorphous triblock copolymers, the PLLA block was found to crystallize into the ordered α form always through the mesophase just above the T g of PLLA. These results are helpful in understanding the regularization process of semicrystalline polymers in different environments.

A survey on manual and non-manual sign language recognition for isolated and continuous sign

Sign language recognition is an important area of human computer interaction (HCI). The last decade witnessed a good number of publications in this field. Furthermore, several surveys can be found in the literature but none of them addresses an overall review in this field. In this paper, we have specifically highlighted the Indian sign language (ISL). The works under the complex and moving background, integration of non-manual signals, large vocabulary and signer independent have got a very little attention in the past. In this paper, we have discussed hand segmentation and tracking, feature extraction and classification methods exist in the literature. Within these methods, we examine the various issues such as signer dependence/independence, manual/non-manual, glove/device-based, vocabulary size, constraints in hand segmentation, and isolated/continuous sign. The purpose of this paper is to provide a complete progress in the field of SLR, specifically in ISL.

Albanian Sign Language (AlbSL) Number Recognition from Both Hand’s Gestures Acquired by Kinect Sensors

Albanian Sign Language (AlbSL) is relatively new and until now there doesn't exist a system that is able to recognize Albanian signs by using natural user interfaces (NUI). The aim of this paper is to present a real-time gesture recognition system that is able to automatically recognize number signs for Albanian Sign Language, captured from signer's both hands. Kinect device is used to obtain data streams. Every pixel generated from Kinect device contains depth data information which is used to construct a depth map. Hands segmentation process is performed by applying a threshold constant to depth map. In order to differentiate signer's hands a K-means clustering algorithm is applied to partition pixels into two groups corresponding to each signer's hands. Centroid distance function is calculated in each hand after extracting hand's contour pixels. Fourier descriptors, derived form centroid distance is used as a hand shape representation. For each number gesture there are 15 Fourier descriptors coefficients generated which represent uniquely that gesture. Every input data is compared against training data set by calculating Euclidean distance, using Fourier coefficients. Sign with the lowest Euclidean distance is considered as a match. The system is able to recognize number signs captured from one hand or both hands. When both signer's hands are used, some of the methodology processes are executed in parallel in order to improve the overall performance. The proposed system achieves an accuracy of 91% and is able to process 55 frames per second.

Hybrid Deep Network and Polar Transformation Features for Static Hand Gesture Recognition in Depth Data

Static hand gesture recognition is an interesting and challenging problem in computer vision. It is considered a significant component of Human Computer Interaction and it has attracted many research efforts from the computer vision community in recent decades for its high potential applications, such as game interaction and sign language recognition. With the recent advent of the cost-effective Kinect, depth cameras have received a great deal of attention from researchers. It promoted interest within the vision and robotics community for its broad applications. In this paper, we propose the effective hand segmentation from the full depth image that is important step before extracting the features to represent for hand gesture. We also represent the novel hand descriptor explicitly encodes the shape and appearance information from depth maps that are significant characteristics for static hand gestures. We propose hand descriptor based on Polar Transformation coordinate is called Histogram of Polar Transformation (HPT) in order to capture both shape and appearance. Beside a robust hand descriptor, a robust classification model also plays a very important role in the hand recognition model. In order to have a high performance in recognition rate, we propose hybrid model for classification based on Sparse Auto-encoder and Deep Neural Network. We demonstrate large improvements over the state-of-the-art methods on two challenging benchmark datasets are NTU Hand Digits and ASL Finger Spelling and achieve the overall accuracy as 97.7% and 84.58%, respectively. Our experiments show that the proposed method significantly outperforms state-of-the-art techniques.

Handwriting Recognition in Free Space Using WIMU-Based Hand Motion Analysis

We present a wireless-inertial-measurement-unit- (WIMU-) based hand motion analysis technique for handwriting recognition in three-dimensional (3D) space. The proposed handwriting recognition system is not bounded by any limitations or constraints; users have the freedom and flexibility to write characters in free space. It uses hand motion analysis to segment hand motion data from a WIMU device that incorporates magnetic, angular rate, and gravity sensors (MARG) and a sensor fusion algorithm to automatically distinguish segments that represent handwriting from nonhandwriting data in continuous hand motion data. Dynamic time warping (DTW) recognition algorithm is used to recognize handwriting in real-time. We demonstrate that a user can freely write in air using an intuitive WIMU as an input and hand motion analysis device to recognize the handwriting in 3D space. The experimental results for recognizing handwriting in free space show that the proposed method is effective and efficient for other natural interaction techniques, such as in computer games and real-time hand gesture recognition applications.

A combination of user-guide scheme and kernel descriptor on RGB-D data for robust and realtime hand posture recognition

This paper presents a robust and real-time hand posture recognition system. To obtain this, key elements of the proposed system contain an user-guide scheme and a kernel-based hand posture representation. We firstly describe a three-stage scheme to train an end-user. This scheme aims to adapt environmental conditions (e.g., background images, distance from device to hand/human body) as well as to learn appearance-based features such as hand-skin color. Thanks to the proposed user-guide scheme, we could precisely estimate heuristic parameters which play an important role for detecting and segmenting hand regions. Based on the segmented hand regions, we utilize a kernel-based hand representation in which three levels of feature are extracted. Whereas pixel-level and patch-level are conventional extractions, we construct image-level which presents a hand pyramid structure. These representations contribute to a Multi-class support vector machine classifier. We evaluate the proposed system in term of the learning time versus the robustness and real time performances. Averagely, the proposed system requires 14s in advanced to guide an end-user. However, the hand posture recognition rate obtains 91.2% accuracy. Performance of the proposed system is comparable with state-of-the-art methods (e.g. Pisharady et al., 2012) but it is a real time system. To recognize a posture, its computational cost is only 0.15s. This is significantly faster than works in Pisharady et al. (2012), which required approximately 2min. The proposed methods therefore are feasible to embed into smart devices, particularly, consumer electronics in domain of home-automation such as televisions, game consoles, or lighting systems.

Lending A Hand: Detecting Hands and Recognizing Activities in Complex Egocentric Interactions.

Hands appear very often in egocentric video, and their appearance and pose give important cues about what people are doing and what they are paying attention to. But existing work in hand detection has made strong assumptions that work well in only simple scenarios, such as with limited interaction with other people or in lab settings. We develop methods to locate and distinguish between hands in egocentric video using strong appearance models with Convolutional Neural Networks, and introduce a simple candidate region generation approach that outperforms existing techniques at a fraction of the computational cost. We show how these high-quality bounding boxes can be used to create accurate pixelwise hand regions, and as an application, we investigate the extent to which hand segmentation alone can distinguish between different activities. We evaluate these techniques on a new dataset of 48 first-person videos of people interacting in realistic environments, with pixel-level ground truth for over 15,000 hand instances.

A Real-Time Hand Detection System during Hand over Face Occlusion

This paper devises a vision-based hand detection system which can handle the situation of the hand and face occlusion. Firstly, human face is coarsely located by skin color detection and ellipse template matching and further determined by using the grayscales and positions of human eyes. Secondly, when the hand occludes the partial region of the face, a novel hand detection method based on Camshift tracking algorithm, force field method and Sobel edge extraction is developed to segment the hand. Finally, the positions of segmented hands are sent to the computer for controlling the cursor movement. In order to reduce the cursor jitter caused by wrong hand detection and generate a smooth trajectory of the cursor, the coordinates of hand positions are corrected by combining the least squares fitting method of orthogonal polynomial and an adaptive Catmull-Rom interpolation algorithm. Experimental results showed that the proposed method could detect hands accurately with the run time of 0.08s per frame and demonstrated a significant improvement in performance when the hand enters the face region. Moreover, our system accomplished smooth movements of the cursor by the vision-based hand detection.

An Integrative Framework of Human Hand Gesture Segmentation for Human–Robot Interaction

This paper proposes a novel framework to segment hand gestures in RGB-depth (RGB-D) images captured by Kinect using humanlike approaches for human–robot interaction. The goal is to reduce the error of Kinect sensing and, consequently, to improve the precision of hand gesture segmentation for robot NAO. The proposed framework consists of two main novel approaches. First, the depth map and RGB image are aligned by using the genetic algorithm to estimate key points, and the alignment is robust to uncertainties of the extracted point numbers. Then, a novel approach is proposed to refine the edge of the tracked hand gestures in RGB images by applying a modified expectation–maximization (EM) algorithm based on Bayesian networks. The experimental results demonstrate that the proposed alignment method is capable of precisely matching the depth maps with RGB images, and the EM algorithm further effectively adjusts the RGB edges of the segmented hand gestures. The proposed framework has been integrated and validated in a system of human–robot interaction to improve NAO robot's performance of understanding and interpretation.

Using automatic alignment on child speech: Directions for improvement

Phonetic analysis is labor intensive, limiting the amount of data that can be considered. Automated techniques (e.g., forced alignment based on Automatic Speech Recognition, ASR) have recently emerged allowing for larger-scale analysis. While forced alignment can be accurate for adult speech (e.g., Yuan & Liberman, 2009), ASR techniques remain a challenge for child speech (Benzeghiba et al., 2007). We used a trainable forced aligner (Gorman et al., 2011) to examine the effect of four factors on alignment accuracy with child speech: (1) Datasets CHILDES (McWhinney, 2000):—Spontaneous speech (single child)—Picture naming (multiple children, Paidologos data); (2) Phonetic Transcription—Manual—Automatic—CMU dictionary (Weide, 1998); (3) Training data—Adult lab data—one dataset of child data—All child data—Child & adult lab data; (4) Segment—voiceless stops—voiceless sibilants—vowels Automatically generated alignments were compared to hand segmentations. While there were limits on accuracy, in general, better results were obtained with (1) picture naming, (2) manual phonetic transcription, (3) training data including child speech, and (4) voiceless stops. These four factors increase the utility of analyzing children’s speech production using forced alignment, potentially allowing researchers to conduct larger-scale studies that would not otherwise be feasible.

New Methodology for Translation of Static Sign Symbol to Words in Kannada Language

is the mean for data transfer over a medium to interact their expressions with each other. The common mode of communication is vocal speech conversation. The main modes of communication are constraint to vocally disabled individuals. For the communication of such individuals various means of communication is suggested, which are called as sign language. Aim: The aim of sign language alphabets recognition is to provide an easy, efficient and accurate mechanism for automatic translation of static sign (determined by a certain configuration of hand) to textual version in kannada language. Problem Statement: The work presented in this paper goal to develop a system for automatic translation of static gestures of alphabets in kannada sign language. It maps letters, words and expression of a certain language to a set of hand gestures enabling an in individual to communicate by using hands gestures rather than by speaking. The system capable of recognizing sign language symbols can be used as a means of communication with hard of hearing people. Sign of the deaf individual can be captured, recognized and translated to words in kannada language for the benefit of blind people. Approach: It has been divided into two phases firstly, feature extraction phase which in turn uses histogram technique, Hough and Segmentation to extract hand from the static sign. Secondly classification phase uses neural network for training samples. Extreme points were extracted from the segmented hand using star skeletonization and recognition was performed by distance signature. Results: The proposed method was tested on the dataset captured in the closed environment with the assumption that the user should be in the field of view. This study was performed for five different datasets in varying lighting conditions. Conclusion: The developed system is focused with objective of reducing the communication gap between normal people and vocally disabled.

American Sign Language Alphabet Translator Android Application: Hand Shapes Into Text

American Sign Language is used by the deaf-mute community in expressing their thoughts. Many applications have been developed to assist the deaf-mute person, but most of the applications integrate on a computer or on desktop. These leads to create a mobile application that will recognize hand pose into text. American Sign Language Alphabet Translator Android Application: Hand Shapes into Text is a mobile application that translates American Sign Language Alphabet into text. These applications translate the ASL alphabet to a person who is not capable of understanding the language of a deaf-mute person. The application used the process of hand segmentation and feature extraction to get the information needed to extract the hand and data set that used to match the hand pose to display the equivalent letters. The study is successful in recognizing most of the alphabet handshape as long as it is well detected.

Advanced Marathi Sign Language Recognition using Computer Vision

Sign language is a natural language that uses different means of expression for communication in everyday life. As compare to other sign language ISL interpretation has got less attention by researcher. This paper presents an Automatic translation system for gesture of manual alphabets in Marathi sign language. It deals with images of bare hands, which allows the user to interact with the system in a natural way. System provides an opportunity for deaf persons to communicate with normal people without the need of an interpreter. We are going to build a systems and methods for the automatic recognition of Marathi sign language. The first step of this system is to create a database of Marathi Sign Language. Hand segmentation is the most crucial step in every hand gesture recognition system since if we get better segmented output, better recognition rates can be achieved. The proposed system also includes efficient and robust hand segmentation and tracking algorithm to achieve better recognition rates. A large set of samples has been used to recognize 43 isolated words from the Standard Marathi sign language. In proposed system, we intend to recognize some very basic elements of sign language and to translate them to text and vice versa in Marathi language. General Terms Image Capturing, Pre-processing, Feature Extraction, Classification, Pattern Recognition/Matching.

An Integrated Approach of Real-time Hand Gesture Recognition Based on Feature Points

Hand Gesture recognition systems enable people to interact with digital systems naturally. Due to the spread of body motion capture device, depth information is available for getting more delicate and effective gesture recognition results. However, due to the limitation of devices such as Microsoft Kinect, it is still very difficult to obtain hand gesture information in rela-time. This paper proposes an integrated approach of real-time hand gesture recognition based on feature points. It explains our solutions for hand segmentation and feature points abstraction based on real-time motion captured images. Having been tested with a series of applications, our method is proved to be robust and effective, and suitable for further extension in real-time hand gesture recognition systems.

Generic System for Human-Computer Gesture Interaction: Applications on Sign Language Recognition and Robotic Soccer Refereeing

Hand gestures are a powerful way for human communication, with lots of potential applications in the area of human computer interaction. Vision-based hand gesture recognition techniques have many advantages compared with traditional devices, giving users a simpler and more natural way to communicate with electronic devices. This work proposes a generic system architecture based in computer vision and machine learning, able to be used with any interface for real-time human-machine interaction. Its novelty is the integration of different tools for gesture spotting and the proposed solution is mainly composed of three modules: a pre-processing and hand segmentation module, a static gesture interface module and a dynamic gesture interface module. The experiments showed that the core of vision-based interaction systems could be the same for all applications and thus facilitate the implementation. For hand posture recognition, a SVM (Support Vector Machine) model was trained with a centroid distance dataset composed of 2170 records, able to achieve a final accuracy of 99.4 %. For dynamic gestures, an HMM (Hidden Markov Model) model was trained for each one of the defined gestures that the system should recognize with a final average accuracy of 93.7 %. The datasets were built from four different users with a total of 25 gestures per user, totalling 1100 records for model construction. The proposed solution has the advantage of being generic enough with the trained models able to work in real-time, allowing its application in a wide range of human-machine applications. To validate the proposed framework two applications were implemented. The first one is a real-time system able to interpret the Portuguese Sign Language. The second one is an online system able to help a robotic soccer game referee judge a game in real-time.

Prediction of Stature from Somatometry of the Left Hand in Igbos, Nigeria

The identification of a person is based on some biological profiles. Stature is one of such; its prediction plays important role in forensic investigation. An attempt was made to establish the stature of the Igbos using left finger lengths, hand length and hand breadth. The correlation between stature and hand dimensions was studied. Which of the hand segments is the best to predict the stature of the Igbos? 88 males and 123 females were included in this study after obtaining informed consent. Measurements of hand segments were taken using Sliding Caliper and the height was recorded using Anthropometer. The data were subjected for statistical analysis. Significant correlation between stature and the entire hand dimensions was established except hand breadth (HB) in the females. Independent linear regression equations to calculate the stature were obtained for hand segments in both genders put together, in males and females separately. Hand length had the best prediction power in both genders put together as well as in the females. In males, the best prediction power was observed in fourth digit length followed by hand length. The multiple linear regression model generated for both genders put together performed better than the simple linear regression models for the males or females.

An Active Learning Framework for Human Hand Sign Gestures and Handling Movement Epenthesis Using Enhanced Level Building Approach

Abstract Human hand detection and segmentation plays an important role in sign language recognition and human machine interaction. In this paper, a novel approach for learning a vision-based hand detection system is introduced. The main contribution of this paper includes robust boosting-based framework for real-time detection of a hand in unconstrained and heterogeneous environments. The proposed system makes use of efficient representative features which allows fast computation while changing the hand appearances and background. Moreover, this proposed strategy efficiently improves the performance while reducing the effort of hand labeling. Experimental results show that the proposed method is practically more flattering as it meets the requirements of real-time performance, accuracy and robustness. This system has been proved to work well with a reasonable amount of training samples and was computationally found to be more effective and efficient.