Hand Shape Recognition Research Articles

Selection of hand features based on Random Forest algorithm and hand shape recognition

To obtain effective features applicable to hand morphology recognition, the method of obtaining effective feature indicators based on the Random Forest (RF) algorithm to downscale hand morphology parameters is proposed. Firstly, 232 female university students collected three-dimensional hand information, constructed auxiliary point, line, and surface standardised measurement methods, obtained 33 characteristic parts of human dimensions, and used k-means clustering for hand morphology subdivision. Hand morphology can be divided into three categories: short, slender and broad. The RF algorithm is used for feature index importance assessment and hand shape recognition model. The accuracy of the feature metrics determined by the RF algorithm, PCA, and VC method applied to hand shape recognition is compared and analysed to verify the effectiveness of the dimensionality reduction of the RF algorithm. The results showed that the feature indexes used for hand shape recognition were five items: hand length, thickness at the metacarpal, thenar width, the distance between the thumb and index finger, and distance from the root of the little finger to the centre of the wrist. Using the RF algorithm to reduce the dimensionality is more effective; the average recognition accuracy of the four hand shape recognition models is 93.78% on average, compared with PCA and VC reduction methods, the average accuracy of hand shape recognition models is increased by 19.17%, and 14.86% respectively. The study's results can provide methodological references for the objective selection of characteristic indicators and morphological recognition of human body parts.

Give Me a Sign: Using Data Gloves for Static Hand-Shape Recognition.

Human-to-human communication via the computer is mainly carried out using a keyboard or microphone. In the field of virtual reality (VR), where the most immersive experience possible is desired, the use of a keyboard contradicts this goal, while the use of a microphone is not always desirable (e.g., silent commands during task-force training) or simply not possible (e.g., if the user has hearing loss). Data gloves help to increase immersion within VR, as they correspond to our natural interaction. At the same time, they offer the possibility of accurately capturing hand shapes, such as those used in non-verbal communication (e.g., thumbs up, okay gesture, …) and in sign language. In this paper, we present a hand-shape recognition system using Manus Prime X data gloves, including data acquisition, data preprocessing, and data classification to enable nonverbal communication within VR. We investigate the impact on accuracy and classification time of using an outlier detection and a feature selection approach in our data preprocessing. To obtain a more generalized approach, we also studied the impact of artificial data augmentation, i.e., we created new artificial data from the recorded and filtered data to augment the training data set. With our approach, 56 different hand shapes could be distinguished with an accuracy of up to 93.28%. With a reduced number of 27 hand shapes, an accuracy of up to 95.55% could be achieved. The voting meta-classifier (VL2) proved to be the most accurate, albeit slowest, classifier. A good alternative is random forest (RF), which was even able to achieve better accuracy values in a few cases and was generally somewhat faster. outlier detection was proven to be an effective approach, especially in improving the classification time. Overall, we have shown that our hand-shape recognition system using data gloves is suitable for communication within VR.

Sign Language Recognition Using the Electromyographic Signal: A Systematic Literature Review.

The analysis and recognition of sign languages are currently active fields of research focused on sign recognition. Various approaches differ in terms of analysis methods and the devices used for sign acquisition. Traditional methods rely on video analysis or spatial positioning data calculated using motion capture tools. In contrast to these conventional recognition and classification approaches, electromyogram (EMG) signals, which measure muscle electrical activity, offer potential technology for detecting gestures. These EMG-based approaches have recently gained attention due to their advantages. This prompted us to conduct a comprehensive study on the methods, approaches, and projects utilizing EMG sensors for sign language handshape recognition. In this paper, we provided an overview of the sign language recognition field through a literature review, with the objective of offering an in-depth review of the most significant techniques. These techniques were categorized in this article based on their respective methodologies. The survey discussed the progress and challenges in sign language recognition systems based on surface electromyography (sEMG) signals. These systems have shown promise but face issues like sEMG data variability and sensor placement. Multiple sensors enhance reliability and accuracy. Machine learning, including deep learning, is used to address these challenges. Common classifiers in sEMG-based sign language recognition include SVM, ANN, CNN, KNN, HMM, and LSTM. While SVM and ANN are widely used, random forest and KNN have shown better performance in some cases. A multilayer perceptron neural network achieved perfect accuracy in one study. CNN, often paired with LSTM, ranks as the third most popular classifier and can achieve exceptional accuracy, reaching up to 99.6% when utilizing both EMG and IMU data. LSTM is highly regarded for handling sequential dependencies in EMG signals, making it a critical component of sign language recognition systems. In summary, the survey highlights the prevalence of SVM and ANN classifiers but also suggests the effectiveness of alternative classifiers like random forests and KNNs. LSTM emerges as the most suitable algorithm for capturing sequential dependencies and improving gesture recognition in EMG-based sign language recognition systems.

Saliency-Driven Hand Gesture Recognition Incorporating Histogram of Oriented Gradients (HOG) and Deep Learning.

Hand gesture recognition is a vital means of communication to convey information between humans and machines. We propose a novel model for hand gesture recognition based on computer vision methods and compare results based on images with complex scenes. While extracting skin color information is an efficient method to determine hand regions, complicated image backgrounds adversely affect recognizing the exact area of the hand shape. Some valuable features like saliency maps, histogram of oriented gradients (HOG), Canny edge detection, and skin color help us maximize the accuracy of hand shape recognition. Considering these features, we proposed an efficient hand posture detection model that improves the test accuracy results to over 99% on the NUS Hand Posture Dataset II and more than 97% on the hand gesture dataset with different challenging backgrounds. In addition, we added noise to around 60% of our datasets. Replicating our experiment, we achieved more than 98% and nearly 97% accuracy on NUS and hand gesture datasets, respectively. Experiments illustrate that the saliency method with HOG has stable performance for a wide range of images with complex backgrounds having varied hand colors and sizes.

Comparison of small sample methods for Handshape Recognition

Automatic Sign Language Translation (SLT) systems can be a great asset to improve the communication with and within deaf communities. Currently, the main issue preventing effective translation models lays in the low availability of labelled data, which hinders the use of modern deep learning models.
 SLT is a complex problem that involves many subtasks, of which handshape recognition is the most important. We compare a series of models specially tailored for small datasets to improve their performance on handshape recognition tasks. We evaluate Wide-DenseNet and few-shot Prototypical Network models with and without transfer learning, and also using Model-Agnostic Meta-Learning (MAML).
 Our findings indicate that Wide-DenseNet without transfer learning and Prototipical Networks with transfer learning provide the best results. Prototypical networks, particularly, are vastly superior when using less than 30 samples, while Wide-DenseNet achieves the best results with more samples. On the other hand, MAML does not improve performance in any scenario. These results can help to design better SLT models.

Grassmannian learning mutual subspace method for image set recognition

This paper addresses the problem of object recognition given a set of images as input (e.g., multiple camera sources and video frames). Convolutional neural network (CNN)-based frameworks do not exploit these sets effectively, processing a pattern as observed, not capturing the underlying feature distribution as it does not consider the variance of images in the set. To address this issue, we propose the Grassmannian learning mutual subspace method (G-LMSM), a NN layer embedded on top of CNNs that can process image sets more effectively and can be trained in an end-to-end manner. The image set is first represented by a low-dimensional input subspace and then this input subspace is matched with dictionary subspaces by a similarity of their canonical angles, an interpretable and easy to compute metric. The key idea of G-LMSM is that the dictionary subspaces are learned as points on the Grassmann manifold, optimized with Riemannian stochastic gradient descent. This learning is stable, efficient and theoretically well-grounded. We demonstrate the effectiveness of our proposed method on hand shape recognition, face identification, and facial emotion recognition.

Self-designed hierarchical network based hand shape intelligent recognition and evaluation for Chinese zither performing

A proper hand shape is the foundation for professional musical instrument performance. In this paper, the image recognition technology based on artificial intelligence is introduced into the hand shape recognition of performing Chinese zither, which is referred to as Zheng in Chinese, for the first time to realize the function of hand shape intelligent evaluation and the self-designed hierarchical network is proposed to recognize and evaluate correct hand shape for Zheng performing. The intra-class difference is larger than the inter-class difference for Zheng performing hand shape image, which belongs to fine-grained image. Therefore, we use the first layer network to determine four classes of images acquired from different viewpoints. Meanwhile, the feature maps from different convolutional blocks of this layer are concatenated as the input of the second layer, which performs fine classification of Zheng performing hand shape images. Consequently, the learning ability of the network can be improved and the complexity of the network can be reduced at the same time. We design an experimental paradigm for instrumentalist hand shape performance evaluation, formulate a Zheng hand shape evaluation merit based on image recognition, and construct a Chinese zither hand shape Dataset (CZ-dataset V3) for the real scene. The experiments show that the method proposed in this paper can effectively improve the recognition accuracy of fine-grained hand shape images and the result is consistent with the evaluation of professional advisors, which realizes the perfect combination of the intelligent image recognition and the hand shape evaluation for Chinese traditional instrument performing.

A Wearable Multi-Joint Wrist Contour Measuring Device for Hand Shape Recognition

Recently, various types of hand shape recognition systems have been developed for human–machine interfaces. However, most wearable recognition systems cannot robustly handle the variations in attachment positions of the devices. Thus, we propose a hand shape recognition system using a wearable multi-joint wrist contour measuring device to realize robust and effective recognition of hand shape, regardless of attachment position variations. In particular, this device can measure the wrist contour and band flexion data to recognize hand shape. The wrist contour data are measured using photo-reflectors mounted inside the device, and the band flexion data are measured using photo-interrupters installed at the device joints. Additionally, the attachment position information is extracted from the wrist contour or band flexion data using dimensional compression or attachment position recognition to achieve robustness against the position variations. Subsequently, the extracted information is incorporated into the hand shape recognizer. The results of the recognition experiment demonstrate that the attachment position information extracted from the band flexion data using dimensional compression could effectively realize robust hand shape recognition, considering variations in the device attachment position.

Seeing Signs: Language Experience and Handshape Perception

Thousands of times every day you see signing around you. But what does it mean to "see" a sign? In this article we will discuss the difference between "seeing" and "perceiving". You see with your eyes, but you perceive with your mind. In this special issue on "Science and the Senses" we address what happens in our mind when we "see" signs, and how our experiences growing up around sign language shape the way our minds make sense of visual experience. We report the results of an experiment investigating whether deaf and hearing signers who learned to sign at different ages perceive handshapes differently using synthetic signs produced via animation. All signers tested categorized the handshapes in a similar manner, but signers differed in discrimination performance. Early learners of ASL tend to ignore meaningless phonetic variation in handshape tokens more than individuals who learned to sign later in life, potentially allowing faster and more efficient recognition of handshapes.

Development of a hand shape recognition system for handover task of assistant robots

Development of a hand shape recognition system for handover task of assistant robots

Weakly Supervised Learning with Multi-Stream CNN-LSTM-HMMs to Discover Sequential Parallelism in Sign Language Videos.

In this work we present a new approach to the field of weakly supervised learning in the video domain. Our method is relevant to sequence learning problems which can be split up into sub-problems that occur in parallel. Here, we experiment with sign language data. The approach exploits sequence constraints within each independent stream and combines them by explicitly imposing synchronisation points to make use of parallelism that all sub-problems share. We do this with multi-stream HMMs while adding intermediate synchronisation constraints among the streams. We embed powerful CNN-LSTM models in each HMM stream following the hybrid approach. This allows the discovery of attributes which on their own lack sufficient discriminative power to be identified. We apply the approach to the domain of sign language recognition exploiting the sequential parallelism to learn sign language, mouth shape and hand shape classifiers. We evaluate the classifiers on three publicly available benchmark data sets featuring challenging real-life sign language with over 1,000 classes, full sentence based lip-reading and articulated hand shape recognition on a fine-grained hand shape taxonomy featuring over 60 different hand shapes. We clearly outperform the state-of-the-art on all data sets and observe significantly faster convergence using the parallel alignment approach.

Handshape Recognition Using Correlation Filter and Euclidean Distance

Handshape Recognition Using Correlation Filter and Euclidean Distance

Principal component pyramids for manifold learning in hand shape recognition

Abstract This paper presents two algorithms using data pyramids for hand shape recognition in Irish Sign Language. Principal Component Analysis (PCA) is used as a feature extraction and dimensionality reduction method. Originally, the problem is nonlinear and it is hard for PCA to extract the underlying structure of the data. The proposed PCA pyramids provide an alternative to nonlinear PCA as they depend on dividing the space into subspaces which are approximately linear using the appropriate eigenspace in each level. They are used to accelerate the search process to approximate the nearest neighbour search problem. The first algorithm uses unsupervised multidimensional grids to cluster the space into cells of similar objects. The second algorithm is based on training a set of simple architecture multilayer neural networks. Experimental results are given to measure the accuracy and performance of the proposed algorithms in comparison with the exhaustive search scenario. The proposed algorithms are applicable for real time applications with high accuracy measures.

A Comparative Study of Shape and Texture Features for Finger Spelling Recognition in Big Data Applications

With a wide variety of big data applications, Sign Language Recognition has become one of the most important research areas in the field of human-computer interaction. Despite recent progresses, the task of classifying finger spelling is still very challenging in Sign Language Recognition. The visually similarity of some signs, the invisibility of the thumb and the large amount of variation by different signers are all make the hand shape recognition very challenging. The work presented in this paper aims to evaluate the performance of some state-of-the-art features for static finger spelling of alphabets in sign language recognition. The comparison experiments were implemented and tested using two popular data sets. Based on the experimental results, analysis and recommendations are given on the efficiency and capabilities of the compared features.

Bimodal biometric system for hand shape and palmprint recognition based on SIFT sparse representation

Biometric-based hand modality is considered as one of the most popular biometric technologies especially in forensic applications. In this paper, a bimodal hand identification system was proposed based on Scale Invariant Feature Transform (SIFT) descriptors, extracted from hand shape and palmprint modalities. A local sparse representation method was adopted in order to represent images with high discrimination. Moreover, fusion was performed at feature and decision levels using a cascade fusion in order to generate the final identification rate of our bimodal system. Our experiments were applied on two hand databases: Indian Institute of Technology of Delhi (IITD) hand database and Bosphorus hand database containing, respectively, 230 and 615 subjects. The results show that the proposed method offers high accuracies compared to other popular bimodal hand biometric methods over the two hand databases. The correct identification rate reaches 99.57 % which is competitive compared to systems existing in the literature.

Design and deployment of a contactless hand-shape identification system for smart spaces

This paper explores the feasibility of a contactless identification system based on hand features. The identification solution has been designed to be integrated with smart space applications and relies on a commercial 3D sensor (i.e., Leap Motion) for palm features capture. The first part of the paper is devoted to evaluate the significance of the different hand features and the performance of a set of classification algorithms. 21 users have contributed to build a testing dataset; for each user, the morphology of each hand has been gathered from 52 features, which include bones length and width, palm characteristics and relative distances among fingers, palm center and wrist. In order to get consistent samples and guarantee the best performance for Leap Motion device, the data collection system provides sweet spot control. This functionality guides the user to situate the hand in the best position and orientation with respect to the device. The selected classification strategies—nearest neighbor (NN), supported vector machine, multilayer perceptron, logistic regression and tree algorithms—have been evaluated on Weka. We have found that relative hand-pose distances are more significant than pure morphological features. On this feature set, the highest correct classified instances (CCI) rate is reached through the multilayer perceptron algorithm (>96 %), although all the evaluated classifiers provide a CCI rate above 90 %. The analysis also gather how these algorithms perform with a variable number of users in the database, and what the sensitivity to the number of training samples is for each algorithm. Results show that there are different alternatives that are accurate enough for non-critical, immediate response applications. The second part of the paper focuses on the implementation of application examples that are integrated with a real-time hand-based identification system using NN. In particular, the applications enable customization and gesture-based control of the smart space. A five-user study has provided insight into the system performance and user experience. Results confirm the viability of using in-air hand shape recognition in smart space settings, although also it is still needed to deal with aspects that hinder faster and more accurate recognition.

A hand shape instruction recognition and learning system using growing SOM with asymmetric neighborhood function

For Human–Machine Interaction systems, it is a convenient method to send user׳s instructions to robots, TV sets, and other electronic equipments by showing different shapes of a hand of user. In our previous works, we proposed to use improved Kohonen׳s Self-Organizing Maps (SOMs), i.e., Transient-SOM (T-SOM) and Parameterless Growing SOM (PL-G-SOM) to recognize different patterns of hand shapes given by different bendings of five fingers of a hand. Recently, an asymmetric neighborhood function was proposed and introduced into the conventional SOM to improve the learning performance by Aoki and Aoyagi. In this paper, we propose to employ their asymmetric neighborhood function into Growing SOM (GSOM), which is an improved SOM to deal with additional online learning for input data. Furthermore, the improved GSOM is applied to a hand shape recognition and instruction learning system, and the results of experiments with eight kinds of instructions showed the effectiveness of the proposed system.

Synthesis of large scale hand-shape databases for biometric applications

This work proposes and analyzes a novel methodology for hand-shape image synthesis. The hand-shape is a popular biometric trait with a high convenience of use and non-intrusive acquisition. The proposed algorithm allows to generate realistic images with natural intra-person and inter-person variability. The method is based on the Active Shape Model algorithm which has been modified in order to add the biometric information typical of new synthetic identities. The generated images are evaluated using three public databases and two hand-shape recognition systems. The results show the suitability of the synthetic data for biometric recognition works. In addition, two novel applications have been proposed to provide new insights in hand-shape biometric recognition including: improvement of machine learning classification based on synthetic training sets and scalability analysis of hand-shape biometrics when the population of the database is increased by two orders of magnitude with respect to existing databases.

A Novel Phonology- and Radical-Coded Chinese Sign Language Recognition Framework Using Accelerometer and Surface Electromyography Sensors.

Sign language recognition (SLR) is an important communication tool between the deaf and the external world. It is highly necessary to develop a worldwide continuous and large-vocabulary-scale SLR system for practical usage. In this paper, we propose a novel phonology- and radical-coded Chinese SLR framework to demonstrate the feasibility of continuous SLR using accelerometer (ACC) and surface electromyography (sEMG) sensors. The continuous Chinese characters, consisting of coded sign gestures, are first segmented into active segments using EMG signals by means of moving average algorithm. Then, features of each component are extracted from both ACC and sEMG signals of active segments (i.e., palm orientation represented by the mean and variance of ACC signals, hand movement represented by the fixed-point ACC sequence, and hand shape represented by both the mean absolute value (MAV) and autoregressive model coefficients (ARs)). Afterwards, palm orientation is first classified, distinguishing “Palm Downward” sign gestures from “Palm Inward” ones. Only the “Palm Inward” gestures are sent for further hand movement and hand shape recognition by dynamic time warping (DTW) algorithm and hidden Markov models (HMM) respectively. Finally, component recognition results are integrated to identify one certain coded gesture. Experimental results demonstrate that the proposed SLR framework with a vocabulary scale of 223 characters can achieve an averaged recognition accuracy of 96.01% ± 0.83% for coded gesture recognition tasks and 92.73% ± 1.47% for character recognition tasks. Besides, it demonstrats that sEMG signals are rather consistent for a given hand shape independent of hand movements. Hence, the number of training samples will not be significantly increased when the vocabulary scale increases, since not only the number of the completely new proposed coded gestures is constant and limited, but also the transition movement which connects successive signs needs no training samples to model even though the same coded gesture performed in different characters. This work opens up a possible new way to realize a practical Chinese SLR system.

Hand shape classification in various pronation angles using a wearable wrist contour sensor

Hand gestures are potentially useful for communications between humans and between a human and a machine. However, existing methods entail several problems for practical use. We have proposed an approach to hand shape recognition based on wrist contour measurement. Especially in this paper, two assignments are addressed. First is the development of a new sensing device in which all elements are installed in a wrist-watch-type device. Second is the development of a new hand shape classifier that can accommodate pronation angle changes. The developed sensing device enables wrist contour data collection under conditions in which the pronation angle varies. The classifier recognizes the hand shape based on statistics produced through data forming and statistics conversion processes. The most important result is that no large difference exists between classification rates that include or those that exclude the independent (preliminary) pronation estimation process using inertia measurement units. This result shows two possible insights: (1) the wrist contour has some features that depend on the hand shape but which do not depend on the pronation angle, or (2) the wrist contour potentially includes pronation angle variation information. These insights indicate the possibility that hand shape can be recognized solely from the wrist contour, even while changing the pronation angle.