Hand Posture Recognition Research Articles

Edge impulse-based convolutional neural network for Hand Posture Recognition

Hand Posture Recognition (HPR) plays a crucial role in enabling effective human-computer interaction, particularly for individuals with hearing disabilities. The study compares five models, including MobileNetV2 96x96 0.35, MobileNetV1 96x96 0.25, MobileNetV1 96x96 0.1, self-designed Network 1, and self-designed Network 2, based on the Sbastien Marcel Static Hand Posture Database. Evaluation metrics - infserencing time, peak RAM usage, flash usage, and accuracy - are used to analyze the performance. The experiment workflow for each model comprises five major steps. Firstly, a random selection of 120 images from the Sbastien Marcel Static Hand Posture Database is converted to JPG format. Then, the images are divided into 80% training data and 20% testing data. Subsequently, the original images are normalized, and features are extracted for further processing. Subsequently, the models are individually trained using the preprocessed data, optimizing their parameters. Finally, the trained models are evaluated using the testing data set to assess their performance in hand posture recognition. The results indicate that MobileNetV2 96x96 0.35 achieves the highest accuracy of 96.69% while consuming fewer hardware resources compared to other models. MobileNetV1 96x96 0.1 demonstrates the lowest inferencing time and peak RAM usage, making it suitable for real-time applications. Furthermore, self-designed Model 1 exhibits the lowest flash usage, making it a viable option for resource-constrained devices. This study provides valuable insights into the selection of CNN architectures for HPR, offering guidance for practitioners to choose models based on specific application requirements.

A Four-Stage Mahalanobis-Distance-Based Method for Hand Posture Recognition

Automatic recognition of hand postures is an important research topic with many applications, e.g., communication support for deaf people. In this paper, we present a novel four-stage, Mahalanobis-distance-based method for hand posture recognition using skeletal data. The proposed method is based on a two-stage classification algorithm with two additional stages related to joint preprocessing (normalization) and a rule-based system, specific to hand shapes that the algorithm is meant to classify. The method achieves superior effectiveness on two benchmark datasets, the first of which was created by us for the purpose of this work, while the second is a well-known and publicly available dataset. The method’s recognition rate measured by leave-one-subject-out cross-validation tests is 94.69% on the first dataset and 97.44% on the second. Experiments, including comparison with other state-of-the-art methods and ablation studies related to classification accuracy and time, confirm the effectiveness of our approach.

Control Mouse using Hand Gesture and Voice

Abstract: In Human Computer Interaction (HCI), the finest invention is the mouse. Even though a wireless mouse or the Bluetooth mouse is in demand today, it still lacks in many fields like cost and power. This paper, therefore, puts forward a modern approach to regulate the mouse movements using a real-time camera with the help of the hand. Our System aims to improve the recognition of human hand postures in a Human Computer Interaction application, reduce the time spent computing and improve user comfort related to human hand postures. We have developed an application for computer mouse control. Based on the proposed algorithm and selected hand feature, the application has good time-based performance. The user finds it easier to operate the system due to the proposed hand postures in combination with the voice assistant. Voice assistants are the great innovation in the field of AI that can change the way of living of the people in a different manner. Many devices are becoming smarter in their own way to interact with human in an easy language. The Desktop based voice assistant are the programs that can recognize human voices and can respond via integrated voice system.

HAND RECOGNITION SYSTEM BASED ON ELECTROMYOGRAPHY FOR REAL-TIME HAND PROSTHETIC CONTROL

Upper limb amputation is a significant limitation for achieving routine activities. Myoelectric signals detected by electrodes well-known as Electromyography (EMG) have been targeted to control upper limb prostheses of such lost limbs. Unfortunately, the acquisition, processing and use of such myoelectric signals are sophisticated. Furthermore, it necessarily requires complex computation to fulfil accuracy, robustness, and time-consumption execution for the real-time prosthesis application. Thus, machine learning schemes for pattern recognition are a potential approach to improve the traditional control for hand prostheses due to the movement of users and muscle contraction. This paper presents real-time hand posture recognition based on three hand postures using surface EMG (sEMG) signals. sEMG signals are acquired by the electrode channel and simultaneously collected while making a hand posture. Performance evaluation relies on classification accuracy and time consumption. The performance of six real-time recognition models is evaluated which combine two projection techniques and three classifiers. Results indicate that EMG-based pattern recognition (EMG-PR) control outperforms the traditional control for hand prostheses in real-time application. The highest classification accuracy is approximately 96%, whereas the lowest time consumption is 4 ms. In addition, the accuracy is dropped when the number of electrodes decreases nearly to 3%. These outcomes can apply to real-time hand prostheses to alleviate the limited prostheses available.

Gesture Controlled Mouse and Voice Assistant

Abstract: This project advocates a Human Computer Interaction method where cursor movement can be controlled via real-time camera by making use of human hand postures recognition. This method is an alternative to current techniques, which include manually pressing buttons or using physical computer mouse. Instead, it uses camera and computer vision software to manage different mouse events and can carry out any action that a conventional computer mouse can. The Virtual Mouse color recognition application will continuously gather real-time photos, which will then go through several conversions and filters. Once everything is converted, the application will use image processing to extract coordinates of specified color position from converted frames. The process then compares current color schemes within the frames to a list of color combinations, where each combination corresponds to a particular set of mouse operations. If the current color scheme matches, the application will perform the mouse command, which will be converted into a real mouse command on the user's computer. In addition to that authors have also developed a voice assistant to improve user productivity by managing routine tasks of the user and by providing information from online sources to the user because voice assistant is effortless to use.

SEMG-based hand posture recognition and visual-feedback training

sEMG-based hand posture recognition and visual-feedback training

SEMG-Based Hand Posture Recognition and Visual Feedback Training for the Forearm Amputee.

sEMG-based gesture recognition is useful for human–computer interactions, especially for technology supporting rehabilitation training and the control of electric prostheses. However, high variability in the sEMG signals of untrained users degrades the performance of gesture recognition algorithms. In this study, the hand posture recognition algorithm and radar plot-based visual feedback training were developed using multichannel sEMG sensors. Ten healthy adults and one bilateral forearm amputee participated by repeating twelve hand postures ten times. The visual feedback training was performed for two days and five days in healthy adults and a forearm amputee, respectively. Artificial neural network classifiers were trained with two types of feature vectors: a single feature vector and a combination of feature vectors. The classification accuracy of the forearm amputee increased significantly after three days of hand posture training. These results indicate that the visual feedback training efficiently improved the performance of sEMG-based hand posture recognition by reducing variability in the sEMG signal. Furthermore, a bilateral forearm amputee was able to participate in the rehabilitation training by using a radar plot, and the radar plot-based visual feedback training would help the amputees to control various electric prostheses.

Interactive Application of Data Glove Based on Emotion Recognition and Judgment System.

In this paper, the interactive application of data gloves based on emotion recognition and judgment system is investigated. A system of emotion recognition and judgment is established based on the set of optimal features of physiological signals, and then a data glove with multi-channel data transmission based on the recognition of hand posture and emotion is constructed. Finally, the system of virtual hand control and a manipulator driven by emotion is built. Five subjects were selected for the test of the above systems. The test results show that the virtual hand and manipulator can be simultaneously controlled by the data glove. In the case that the subjects do not make any hand gesture change, the system can directly control the gesture of the virtual hand by reading the physiological signal of the subject, at which point the gesture control and emotion control can be carried out at the same time. In the test of the manipulator driven by emotion, only the results driven by two emotional trends achieve the desired purpose.

End-to-end multiple modals deep learning system for hand posture recognition

Multi-modal or multi-view dataset that was captured from various resources (e.g. RGB and Depth) of a subject at the same time. Combination between different cues has still faced to many challenges as unique data and complementary in-formation. In adition, the proposed method for multiple modalities recognition consists of discrete blocks, such as: extract features for separative data flows, combine of features, and classify gestures. To address the challenges, we pro-posed two novel end-to-end hand posture recognition frameworks, which are integrated all steps into a convolution neuronal network (CNN) system from capturing various types of cues (RGB and Depth images) to classify hand ges-ture labels. Both frameworks use the Resnet50 backbone that was pretrained by ImageNet dataset. We proposed a novel end-to-end multi-modal frameworks, which are named attention convolution module (ACM) and gated concatenation module (GCM). Both of them are deployed, evaluated and compared on vari-ous multi-modalities hand posture datasets. Experimental results show that our proposed method outperforms with others state-of-the-art techniques (SOTA) methods.

Applying machine learning for hand posture recognition from a series of depth images

Image recognition one of the most difficult issue, is becoming an applied research in recent times. The possibility of allowing computers to recognize human’s hand posture in real-time is gradually becoming popular and involved in many fields. It is also integrated into so many products. Thus, this study proposes an approach for faster and more accurate image recognition. With the development of technology, this camera device has had many significant changes; the camera can capture both RGB images and images in depth. This makes identification in low-light conditions more accurate, which improves recognition performance. The Random Forest algorithm is evaluated to be more efficient, faster, and more accurate than previous recognition algorithms used for model testing. We collected about 10,000 hand image data samples, then trained and evaluated the model with nearly 80% accuracy results. In addition, the study presents some results of gesture recognition of human hands in real-time using information obtained from Kinect sensor; the model has the ability to train quite large data and the recognition speed is faster. Research results can be used in control applications serving in life and industrial production.

SEMG-Based Hand Posture Recognition Considering Electrode Shift, Feature Vectors, and Posture Groups.

Surface electromyography (sEMG)-based gesture recognition systems provide the intuitive and accurate recognition of various gestures in human-computer interaction. In this study, an sEMG-based hand posture recognition algorithm was developed, considering three main problems: electrode shift, feature vectors, and posture groups. The sEMG signal was measured using an armband sensor with the electrode shift. An artificial neural network classifier was trained using 21 feature vectors for seven different posture groups. The inter-session and inter-feature Pearson correlation coefficients (PCCs) were calculated. The results indicate that the classification performance improved with the number of training sessions of the electrode shift. The number of sessions necessary for efficient training was four, and the feature vectors with a high inter-session PCC (r > 0.7) exhibited high classification accuracy. Similarities between postures in a posture group decreased the classification accuracy. Our results indicate that the classification accuracy could be improved with the addition of more electrode shift training sessions and that the PCC is useful for selecting the feature vector. Furthermore, hand posture selection was as important as feature vector selection. These findings will help in optimizing the sEMG-based pattern recognition algorithm more easily and quickly.

An implementation of sign language alphabet hand posture recognition using geometrical features through artificial neural network (part 2)

In the sign language alphabet, several hand signs are in use. Automatic recognition of performed hand signs can facilitate the communication between hearing and none hearing people. This framework proposes hand posture recognition of the American Sign Language alphabet based on a neural network (NN) which works on geometrical feature extraction of the hand. The user’s hand is captured by a 3D depth-based sensor camera. Consequently, the hand is segmented according to the depth features. The proposed system is called ‘Depth-based Geometrical Sign Language Recognition’ (DGSLR). The DGSLR adopted an easier hand segmentation approach, which is further used in other segmentation applications. The proposed geometrical feature extraction framework improves the accuracy of recognition due to unchangeable features against hand orientation or rotation compared to Discrete Cosine Transform (DCT) and Moment Invariant. As a support vector machine (SVM) is a type of artificial neural network (ANN), it is used to drive desired outcomes. Since there are 26 different signs in the Sign Language alphabet, a multi-class SVM versus a single SVM classifier with 26 classes by an RBF kernel was used to validate each class. The proposed framework is proficient to hand posture recognition and provides an accuracy of up to 96.78%. The findings of the iterations demonstrated that the combination of the extracted features resulted in a better accuracy rate in the recognition process in the classification step.

Approach to hand posture recognition based on hand shape features for human–robot interaction

Hand segmentation is the initial step for hand posture recognition. To reduce the effect of variable illumination in hand segmentation step, a new CbCr-I component Gaussian mixture model (GMM) is proposed to detect the skin region. The hand region is selected as a region of interest from the image using the skin detection technique based on the presented CbCr-I component GMM and a new adaptive threshold. A new hand shape distribution feature described in polar coordinates is proposed to extract hand contour features to solve the false recognition problem in some shape-based methods and effectively recognize the hand posture in cases when different hand postures have the same number of outstretched fingers. A multiclass support vector machine classifier is utilized to recognize the hand posture. Experiments were carried out on our data set to verify the feasibility of the proposed method. The results showed the effectiveness of the proposed approach compared with other methods.

Fine-grained Hand-posture Recognition for Natural User-interface Technologies

Fine-grained Hand-posture Recognition for Natural User-interface Technologies

A Knitted Sensing Glove for Human Hand Postures Pattern Recognition.

In recent years, flexible sensors for data gloves have been developed that aim to achieve excellent wearability, but they are associated with difficulties due to the complicated manufacturing and embedding into the glove. This study proposes a knitted glove integrated with strain sensors for pattern recognition of hand postures. The proposed sensing glove is fabricated at all once by a knitting technique without sewing and bonding, which is composed of strain sensors knitted with conductive yarn and a glove body with non-conductive yarn. To verify the performance of the developed glove, electrical resistance variations were measured according to the flexed angle and speed. These data showed different values depending on the speed or angle of movements. We carried out experiments on hand postures pattern recognition for the practicability verification of the knitted sensing glove. For this purpose, 10 able-bodied subjects participated in the recognition experiments on 10 target hand postures. The average classification accuracy of 10 subjects reached 94.17% when their own data were used. The accuracy of up to 97.1% was achieved in the case of grasp posture among 10 target postures. When all mixed data from 10 subjects were utilized for pattern recognition, the average classification expressed by the confusion matrix arrived at 89.5%. Therefore, the comprehensive experimental results demonstrated the effectiveness of the knitted sensing gloves. In addition, it is expected to reduce the cost through a simple manufacturing process of the knitted sensing glove.

3D PostureNet: A unified framework for skeleton-based posture recognition

Abstract Image-based posture recognition is a very challenging problem as it is difficult to acquire rich 3D information from postures in 2D images. Existing methods founded on 3D skeleton cues could alleviate this issue, but they are not particularly efficient due to the application of handcrafted features and traditional classifiers. This paper presents a novel and unified framework for skeleton-based posture recognition, applying powerful 3D Convolutional Neural Network (CNN) to this issue. Technically, bounding-box-based normalization for the raw skeleton data is proposed to eliminate the coordinate differences caused by diverse recording environments and posture displacements. Moreover, Gaussian voxelization for the skeleton is employed to expressively represent the posture configuration. Thereby, an end-to-end framework based on 3D CNN, called 3D PostureNet, is developed for robust posture recognition. To verify its effectiveness, a large-scale writing posture dataset is created and released in this work, including 113,400 samples of 30 subjects with 15 postures. Extensive experiments on the public MSRA hand gesture dataset, body pose dataset and the proposed writing posture dataset demonstrate that 3D PostureNet achieves significantly superior performance on both skeleton-based human posture and hand posture recognition tasks.

Human Action Recognition Using Bone Pair Descriptor and Distance Descriptor

The paper presents a method for the recognition of human actions based on skeletal data. A novel Bone Pair Descriptor is proposed, which encodes the angular relations between pairs of bones. Its features are combined with Distance Descriptor, previously used for hand posture recognition, which describes relationships between distances of skeletal joints. Five different time series classification methods are tested. The selection of features, input joints, and bones is performed. The experiments are conducted using person-independent validation tests and a challenging, publicly available dataset of human actions. The proposed method is compared with other approaches found in the literature achieving relatively good results.

A new framework for sign language alphabet hand posture recognition using geometrical features through artificial neural network (part 1)

Hand pose tracking is essential in sign languages. An automatic recognition of performed hand signs facilitates a number of applications, especially for people with speech impairment to communication with normal people. This framework which is called ASLNN proposes a new hand posture recognition technique for the American sign language alphabet based on the neural network which works on the geometrical feature extraction of hands. A user’s hand is captured by a three-dimensional depth-based sensor camera; consequently, the hand is segmented according to the depth analysis features. The proposed system is called depth-based geometrical sign language recognition as named DGSLR. The DGSLR adopted in easier hand segmentation approach, which is further used in segmentation applications. The proposed geometrical feature extraction framework improves the accuracy of recognition due to unchangeable features against hand orientation compared to discrete cosine transform and moment invariant. The findings of the iterations demonstrate the combination of the extracted features resulted to improved accuracy rates. Then, an artificial neural network is used to drive desired outcomes. ASLNN is proficient to hand posture recognition and provides accuracy up to 96.78% which will be discussed on the additional paper of this authors in this journal.

In-Vehicle Device Control System by Hand Posture Recognition with Movement Detection Using Infrared Array Sensor

Nowadays, with the development of automotive driving technologies, more and more functions and devices with control systems based on tactile, optical, and acoustic sensors are assembled into cars. ...

Estimation of Hand Skeletal Postures by Using Deep Convolutional Neural Networks

Hand posture estimation attracts researchers because of its many applications. Hand posture recognition systems simulate the hand postures by using mathematical algorithms. Convolutional neural networks have provided the best results in the hand posture recognition so far. In this paper, we propose a new method to estimate the hand skeletal posture by using deep convolutional neural networks. To simplify the proposed method and to be more functional, the depth factor is ignored. So only the simple color images of hands are used as inputs of the system. The proposed method is evaluated by using two datasets with high-diversity named Mixamo and RWTH, which include 43,986 and 1160 color images, respectively, where 74% of these images are selected as a training set and, 26% of the rest images are selected as the evaluation set. The experiments show that the proposed method provides better results in both hand posture recognition and detection of sign languages compared to state-of-the-art methods.