Vision-based Gesture Recognition System Research Articles

A Review of the Hand Gesture Recognition System: Current Progress and Future Directions

This paper reviewed the sign language research in the vision-based hand gesture recognition system from 2014 to 2020. Its objective is to identify the progress and what needs more attention. We have extracted a total of 98 articles from well-known online databases using selected keywords. The review shows that the vision-based hand gesture recognition research is an active field of research, with many studies conducted, resulting in dozens of articles published annually in journals and conference proceedings. Most of the articles focus on three critical aspects of the vision-based hand gesture recognition system, namely: data acquisition, data environment, and hand gesture representation. We have also reviewed the performance of the vision-based hand gesture recognition system in terms of recognition accuracy. For the signer dependent, the recognition accuracy ranges from 69% to 98%, with an average of 88.8% among the selected studies. On the other hand, the signer independent’s recognition accuracy reported in the selected studies ranges from 48% to 97%, with an average recognition accuracy of 78.2%. The lack in the progress of continuous gesture recognition could indicate that more work is needed towards a practical vision-based gesture recognition system.

A survey on hand gesture recognition for mobile devices

As smartphones and mobile devices become ubiquitous in the daily life of human beings, facilitating interaction with them will have a positive impact on their use. This paper presents an emerging and important area for research on human-machine interaction, involving interaction with mobile devices. Hand gesture recognition is already a quite mature field and its integration on advanced technologies like smartphones makes a survey an interesting and needed contribution. The design of interactive mobile applications should differ from traditional desktop applications. For this purpose, the paper discusses some concepts that help to understand new challenges and introduces techniques that may be useful to properly exploit these devices. This paper gives also an overview of recent works done on vision-based and contact-based hand gesture recognition systems for mobile devices. Used techniques and approaches are also discussed. At the end, we conclude with some reflections.

A Signer Independent Sign Language Recognition with Co-articulation Elimination from Live Videos: An Indian Scenario

Due to the high population of hearing impaired and vocal disabled people in India, a sign language interpretation system is becoming highly important for minimizing their isolation in society. This paper proposes a signer independent novel vision-based gesture recognition system which is capable of recognizing single handed static and dynamic gestures, double-handed static gestures and finger spelling words of Indian Sign Language (ISL) from live video. The use of Zernike moments for key frame extraction reduces the computation speed to a large extent. It also proposes an improved method for co-articulation elimination in fingerspelling alphabets. The gesture recognition module comprises mainly three steps – Preprocessing, Feature Extraction, and Classification. In the preprocessing phase, the signs are extracted from a real-time video using skin color segmentation. An appropriate feature vector is extracted from the gesture sequence after co-articulation elimination phase. The obtained features are then used for classification using Support Vector Machine(SVM). The system successfully recognized finger spelling alphabets with 91% accuracy and single-handed dynamic words with 89% accuracy. The experimental results show that the system has a better recognition rate compared to some of the existing methods.

2D Hand Detection Using Multi-Feature Skin Model Supervised Cascaded CNN

Hand gesture recognition is one of the most popular Human Computer Interface. The first step in most vision-based gesture recognition system is the hand detection and segmentation. Since hands are involved in a variety of daily tasks, the detection work suffers from both extreme illumination changes and the intrinsic variability of hand appearance. To overcome these problems, we propose a new method for 2D hand detection which can be considered as a combination of Multi-Feature based hand proposal generation and cascaded convolutional neural network (CCNN) classification. Considered various luminance, we choose color, Gabor, HOG and SIFT feature to discriminate skin region and generate hand proposal. Also, we propose a cascaded CNN that keeps the deep context information to detect hand among the proposals. The proposed Multi-Feature Supervised Cascaded CNN (MFS-CCNN) method is tested on a combination of several datasets including Oxford Hands Dataset, VIVA hand detection and Egohands Dataset as positive sample and ImageNet 2012, FDDB dataset as negative sample. The proposed method achieves competitive results.

Computer Vision-Based Real-Time 3D Gesture Recognition Using Depth Image

Gesture is one of the fundamental ways of human machine natural interaction. To understand gesture, the system should be able to interpret 3D movements of human. This paper presents a computer vision-based real-time 3D gesture recognition system using depth image which tracks 3D joint position of head, neck, shoulder, arms, hands and legs. This tracking is done by Kinect motion sensor with OpenNI API and 3D motion gesture is recognized using the movement trajectory of those joints. User to Kinect sensor distance is adapted using proposed center of gravity (COG) correction method and 3D joint position is normalized using proposed joint position normalization method. For gesture learning and recognition, data mining classification algorithms such as Naive Bayes and neural network is used. The system is trained to recognize 12 gestures used by umpires in a cricket match. It is trained and tested using about 2000 training instances for 12 gesture of 15 persons. The system is tested using 5-fold cross validation method and achieved 98.11% accuracy with neural network and 88.84% accuracy with Naive Bayes classification method.

Dynamic Arm Gesture Recognition Using Spherical Angle Features and Hidden Markov Models

We introduce a vision-based arm gesture recognition (AGR) system using Kinect. The AGR system learns the discrete Hidden Markov Model (HMM), an effective probabilistic graph model for gesture recognition, from the dynamic pose of the arm joints provided by the Kinect API. Because Kinect’s viewpoint and the subject’s arm length can substantially affect the estimated 3D pose of each joint, it is difficult to recognize gestures reliably with these features. The proposed system performs the feature transformation that changes the 3D Cartesian coordinates of each joint into the 2D spherical angles of the corresponding arm part to obtain view-invariant and more discriminative features. We confirmed high recognition performance of the proposed AGR system through experiments with two different datasets.

Real time hand detection in a complex background

Hand gesture recognition has gained the interest of many researchers in recent years, as it has become one of the most popular Human Computer Interfaces. The first step in most vision-based gesture recognition systems is the hand region detection and segmentation. This segmentation can be a particularly challenging task when it comes to complex backgrounds and varying illumination. In such environments, most hand detection techniques fail to obtain the exact region of the hand shape, especially in cases of dynamic gestures. Meeting these requirements becomes even more difficult, due to real-time operation demand. To overcome these problems, in this paper, we propose a new method for real-time hand detection in a complex background. We employ a combination of existing techniques, based on motion detection and introduce a novel skin color classifier to improve segmentation accuracy. Motion detection is based on image differencing and background subtraction. Skin color detection is accomplished via a color classification technique that employs online color training, so that the system can dynamically adapt to the variety of lighting conditions and the user׳s skin color as well as possible. Morphological features of the detected hand in previous frames are employed to estimate the probability of a pixel belonging to the hand section in the current frame. Finally, the derived motion, color and morphological information are combined to detect the hand region. Experimental results show significant improvement in hand region detection, compared to existing methods with an average accuracy of 98.75%.

A vision based system for communicating in virtual reality environments by recognizing human hand gestures

Nowadays, with the emerging of the new applications like virtual reality using image processing and machine vision algorithms, it is necessary to have more modern interfaces for interaction with robots and computers. To cope with this problem, vision based gesture recognition systems play a significant role. This paper implements a vision based system for human hand gesture recognition. We propose three different methods for human hand gesture recognition, including, 1-a new method based on spatio-temporal volumes, 2-hand gesture recognition based on Motion History Image (MHI) and 3-method based on eigen space features. In these algorithms, after applying necessary pre-processing on video frames, some features are extracted for each gesture. These features are then analyzed and finally, a classification algorithm is applied for the hand gesture recognition. We tested the proposed three algorithms with the collected dataset and the results showed that proposed method based on the spatio-temporal volumes results in the correct recognition rate of 99.58 % for noiseless and 97.92 % for noisy data. Comparing the results of the proposed method based on the spatio-temporal volumes with two other methods shows that the recognition rate is improved up to 5.83 % and 8.75 % respectively.

Poster 134 Developing a Vision Based Gesture Recognition System to Control Assistive Technology in Neuro-Disability

Poster 134 Developing a Vision Based Gesture Recognition System to Control Assistive Technology in Neuro-Disability

The Effect of Visual Feedback on One-hand Gesture Performance in Vision-based Gesture Recognition System

Objective: This study presents the effect of visual feedback on one-hand gesture performance in vision-based gesture recognition system when people use gestures to control a screen device remotely. Backgroud: gesture interaction receives growing attention because it uses advanced sensor technology and it allows users natural interaction using their own body motion. In generating motion, visual feedback has been to considered critical factor affect speed and accuracy. Method: three types of visual feedback(arrow, star, and animation) were selected and 20 gestures were listed. 12 participants perform each 20 gestures while given 3 types of visual feedback in turn. Results: People made longer hand trace and take longer time to make a gesture when they were given arrow shape feedback than star-shape feedback. The animation type feedback was most preferred. Conclusion: The type of visual feedback showed statistically significant effect on the length of hand trace, elapsed time, and speed of motion in performing a gesture. Application: This study could be applied to any device that needs visual feedback for device control. A big feedback generate shorter length of motion trace, less time, faster than smaller one when people performs gestures to control a device. So the big size of visual feedback would be recommended for a situation requiring fast actions. On the other hand, the smaller visual feedback would be recommended for a situation requiring elaborated actions.

The Effect of Gesture-Command Pairing Condition on Learnability when Interacting with TV

Objective: The aim of this study is to investigate learnability of gestures-commands pair when people use gestures to control a device. Background: In vision-based gesture recognition system, selecting gesture-command pairing is critical for its usability in learning. Subjective preference and its agreement score, used in previous study(Lim et al., 2012) was used to group four gesture-command pairings. To quantify the learnability, two learning models, average time model and marginal time model, were used. Method: Two sets of eight gestures, total sixteen gestures were listed by agreement score and preference data. Fourteen participants divided into two groups, memorized each set of gesture-command pair and performed gesture. For a given command, time to recall the paired gesture was collected. Results: The average recall time for initial trials were differed by preference and agreement score as well as the learning rate R driven by the two learning models. Conclusion: Preference rate agreement score showed influence on learning of gesture-command pairs. Application: This study could be applied to any device considered to adopt gesture interaction system for device control.

Extraction of spatial-temporal features for vision-based gesture recognition

One of the key problems in a vision-based gesture recognition system is the extraction of spatial-temporal features of gesturing. In this paper an approach of motion-based segmentation is proposed to realize this task. The direct method cooperated with the robust M-estimator to estimate the affine parameters of gesturing motion is used, and based on the dominant motion model the gesturing region is extracted, i.e., the dominant object. So the spatial-temporal features of gestures can be extracted. Finally, the dynamic time warping (DTW) method is directly used to perform matching of 12 control gestures (6 for “translation” orders, 6 for “rotation” orders). A small demonstration system has been set up to verify the method, in which a panorama image viewer can be controlled (set by mosaicing a sequence of standard “Garden” images) with recognized gestures instead of the 3-D mouse tool.

Artificial retina chips as on-chip image processors and gesture-oriented interfaces

Players of a video game may sometimes find the use of conventional interfaces inappropriate. In such cases, we think that interfaces realized with a vision-based gesture recognition system may find favor. The artificial retina (AR) chip is a versatile image sensor whose use ranges from normal image acquisition to on-chip image processing, including on-chip image convolution. In this paper, we describe a gesture- input video game system, with the AR module including the AR chip, and motion-based gesture recognition algorithms. We showed that the algorithms can be accelerated by projection data, the direct output from the AR chip. To show its performance, we have applied our system to two commercially available video games.