Gesture Detection Research Articles

CAMERA-BASED GESTURE TRACKING FOR 3D INTERACTION BEHIND MOBILE DEVICES

Number of mobile devices such as Smartphones or Tablet PCs has been dramatically increased over the recent years. New mobile devices are equipped with integrated cameras and large displays that make the interaction with the device easier and more efficient. Although most of the previous works on interaction between humans and mobile devices are based on 2D touch-screen displays, camera-based interaction opens a new way to manipulate in 3D space behind the device in the camera's field of view. In this paper, our gestural interaction relies on particular patterns from local orientation of the image called rotational symmetries. This approach is based on finding the most suitable pattern from a large set of rotational symmetries of different orders that ensures a reliable detector for fingertips and user's gesture. Consequently, gesture detection and tracking can be used as an efficient tool for 3D manipulation in various virtual/augmented reality applications.

A Vision-Based Method to Find Fingertips in a Closed Hand

Hand gesture recognition is an important area of research in the field of Human Computer Interaction (HCI). The geometric attributes of the hand play an important role in hand shape reconstruction and gesture recognition. That said, fingertips are one of the important attributes for the detection of hand gestures and can provide valuable information from hand images. Many methods are available in scientific literature for fingertips detection with an open hand but very poor results are available for fingertips detection when the hand is closed. This paper presents a new method for the detection of fingertips in a closed hand using the corner detection method and an advanced edge detection algorithm. It is important to note that the skin color segmentation methodology did not work for fingertips detection in a closed hand. Thus the proposed method applied Gabor filter techniques for the detection of edges and then applied the corner detection algorithm for the detection of fingertips through the edges. To check the accuracy of the method, this method was tested on a vast number of images taken with a webcam. The method resulted in a higher accuracy rate of detections from the images. The method was further implemented on video for testing its validity on real time image capturing. These closed hand fingertips detection would help in controlling an electro-mechanical robotic hand via hand gesture in a natural way.

Project Mudra: Personalization of Computers using Natural Interface

In recent years, the popularity for natural interfaces has experienced tremendous growth. Natural interfaces provide the computers or the so called machines with a sense of humanity. The basic knowledge required to understand and utilize a computer system is reduced to a minimal extent. There have been recent commercial developments in the area of natural interfaces, including the Microsoft XBOX Kinect and Nintendo WII. However, the capital cost of such a system remains very high. The primary intention of this paper is to design a technique to redesign every aspect of our interaction with the computer. It explores and develops an interface that helps simple computer systems to support gestural interface. We could utilize the low cost cameras available with the laptop or desktop to support detection of human gestures and multiple touch experiences, which would ultimately lead to the direct interaction of computers with humans. In such a system, the need for external input devices like mouse and keyboard could be eliminated. The paper discusses how such a system can be implemented, along with a brief description of the approach used.

Tracking, recognition, and distance detection of hand gestures for a 3‐D interactive display

Abstract— This study proposes an interactive system for displays, the technologies of which consists of three main parts: hand‐gesture tracking, recognition, and depth measurement. The proposed interactive system can be applied to a general 3‐D display. In this interactive system, for hand‐gesture tracking, Haar‐like features are employed to detect a specific hand gesture to start tracking, while the mean‐shift algorithm and Kalman filter are adopted for fast tracking. First, for recognizing hand gestures, a principal component analysis (PCA) algorithm is used to localize colored areas of skin, and then hand gestures are identified by comparison with a prepared database. Second, a simple optical system is set up with an infrared laser source and a grid mask in order to project a proposed horizontal stripe pattern. Third, the projected patterns are deciphered to extract the depth information using the Hough‐transform algorithm. The system containing hand‐gesture localization, recognition, and associated depth detection (the distance between the display and the hand), was included in a prototype of an interactive display. Demonstration of rotation recognition of a finger‐pointing hand gesture was successful by using the algorithm of radar‐like scanning.

Real-Time Gesture Recognition for Controlling a Virtual Hand

Object tracking in three dimensional environments is an area of research that has attracted a lot of attention lately, for its potential regarding the interaction between man and machine. Hand gesture detection and recognition, in real time, from video stream, plays a significant role in the human-computer interaction and, on the current digital image processing applications, this represent a difficult task. This paper aims to present a new method for human hand control in virtual environments, by eliminating the need of an external device currently used for hand motion capture and digitization. A first step in this direction would be the detection of human hand, followed by the detection of gestures and their use to control a virtual hand in a virtual environment.

Real Time Gesture Recognition System for Interaction in Dynamic Environment

Human Computer Interaction techniques have become a bottleneck in the effective utilization of the available information flow. The development of user interfaces influences the changes in the Human-Computer Interaction (HCI). Human hand gestures have been a mode of non verbal interaction widely used. Naturalistic and intuitiveness of the hand gesture has been a great motivating factor for the researchers in the area of HCI to put their efforts to research and develop the more promising means of interaction between human and computers. This paper designs a system for gestural interaction between a user and a computer in dynamic environment. The gesture recognition system uses image processing techniques for detection, segmentation, tracking and recognition of hand gestures for converting it to a meaningful command. The interface being proposed here can be substantially applied towards different applications like image browser, games etc.

A low cost embedded solution for detection and analysis of three-dimensional gestures through wireless communication

This paper describes the concept of a non-optical acceleration-based gesture recognition system for a low cost and small size remote control for mobile robots which is capable of recognizing more than 5 gestures based on the input signals from a MEMS 3-axes accelerometer and a MEMS 2 axes-gyroscope. The system is designed like a wristwatch and based on a microcontroller module which detects the accelerations and the movements of the user's arm and transmits the results wirelessly to a receiver. The two main topics in this paper are the description of the hardware and the newly developed algorithm used to analyze and classify the detected movements in comparison to the traditional algorithms Hidden Markov Model (HMM) and Dynamic Time Warping (DTW).

Robust classification of face and head gestures in video

Automatic analysis of head gestures and facial expressions is a challenging research area and it has significant applications in human-computer interfaces. We develop a face and head gesture detector in video streams. The detector is based on face landmark paradigm in that appearance and configuration information of landmarks are used. First we detect and track accurately facial landmarks using adaptive templates, Kalman predictor and subspace regularization. Then the trajectories (time series) of facial landmark positions during the course of the head gesture or facial expression are converted in various discriminative features. Features can be landmark coordinate time series, facial geometric features or patches on expressive regions of the face. We use comparatively, two feature sequence classifiers, that is, Hidden Markov Models (HMM) and Hidden Conditional Random Fields (HCRF), and various feature subspace classifiers, that is, ICA (Independent Component Analysis) and NMF (Non-negative Matrix Factorization) on the spatiotemporal data. We achieve 87.3% correct gesture classification on a seven-gesture test database, and the performance reaches 98.2% correct detection under a fusion scheme. Promising and competitive results are also achieved on classification of naturally occurring gesture clips of LIlir TwoTalk Corpus.

2P2-M09 Depth-based Scope Switching and Image Skeletonization for Gesture Recognition(Sense, Motion and Measurement)

We describe a software framework from which gesture detection components are parametrically defined and applied to depth images with a human in the field of view. Using depth data, the framework can quickly switch between a wide view and a narrow view, depending on whether or not the person in view is performing the gesture in the xy-plane of his or her body or in front of his or her body. Gesture recognition is based on extracting local maxima and local minima of border pixels of the region of interest. Gesture detection examples include finger-counting, waving, "come here", and "It's hot!", which can be applied to robot systems that require human-robot interaction via robot vision.

Detection of Tongue Protrusion Gestures from Video

SUMMARY We propose a system that, using video information, segments the mouth region from a face image and then detects the protrusion of the tongue from inside the oral cavity. Initially, under the assumption that the mouth is closed, we detect both mouth corners. We use a set of specifically oriented Gabor filters for enhancing horizontal features corresponding to the shadow existing between the upper and lower lips. After applying the Hough line detector, the extremes of the line that was found are regarded as the mouth corners. Detection rate for mouth corner localization is 85.33%. These points are then input to a mouth appearance model which fits a mouth contour to the image. By segmenting its bounding box we obtain a mouth template. Next, considering the symmetric nature of the mouth, we divide the template into right and left halves. Thus, our system makes use of three templates. We track the mouth in the following frames using normalized correlation for mouth template matching. Changes happening in the mouth region are directly described by the correlation value, i.e., the appearance of the tongue in the surface of the mouth will cause a decrease in the correlation coefficient through time. These coefficients are used for detecting the tongue protrusion. The right and left tongue protrusion positions will be detected by analyzing similarity changes between the right and left half-mouth templates and the currently tracked ones. Detection rates under the default parameters of our system are 90:20% for the tongue protrusion regardless of the position, and 84:78% for the right and left tongue protrusion positions. Our results demonstrate the feasibility of real-time tongue protrusion detection in vision-based systems and motivates further investigating the usage of this new modality in human

Head Gesture Analysis using Matrix Group Displacement Algorithm

A novel algorithm for head gestures interpretation is designed and tested. The designed system carries out gesture detection and recognitio n using the MGDA algorithm, which implements random sampling and importance sampling, such technique can track head poses and estimate head positions. Problem statement: Head position is an important indicator of a perso n's focus of attention, which can be used as a key for multi-view face anal ysis assuming that face recognition and identification to be viewed dependently. This will help in selecting the best view model. Also, in the past few years face detection and person identifica tion became important issues due to security concerns, leading to head gesture algorithm develop ment and implementation. Approach: The captured image was allocated a map after which a fi le conversion process is carried out, allowing the next stage of image data conversion of head poses t o be applied. This results in a specific number of matrices per pose that hold the necessary informati on. Such information was then allocated sequences representing head gesture poses which is combined f or classification and correlation purposes to regenerate a predicted time reconstructed continuou s movements. Results: A reliable, fast and robust approach for static head gesture recognition was ac hieved and presented. Conclusion: This very successful approach to head pose detection and gest ure classification is strongly supported by its ability to correlate different signal input technol ogies as the devised algorithm can accommodate different inputs.

Skin Color Detection Model Using Neural Networks and its Performance Evaluation

Problem statement: Skin color detection is used as a preliminary step in numerous computer vision applications like face detection, nudity recognition, hand gesture detection and person identification. In this study we present a pixel based skin color classification approach, for detecting skin pixels and non skin pixels in color images, using a novel neural network symmetric classifier. The neural classifiers used in the literature either uses a symmetric model with single neuron in the output layer or uses two separate neural networks (asymmetric model) for each of the skin and non-skin classes. The novelty of our approach is that it has two output layer neurons; one each for skin and non-skin class, instead of using two separate classifiers. Thus by using a single neural network classifier we have improved the separability between these two classes, eliminating additional time complexity that is needed in asymmetric classifier. Approach: Skin samples from web images of people from different ethnic groups were collected and used for training. Ground truth skin segmented images were obtained by using semiautomatic skin segmentation tool developed by the authors. The ground truth database of skin segmented images, thus obtained was used to evaluate the performance of our NN based classifier. Results: With proper selection of optimum classification threshold that varies from image to image the classifier gave the detection rate of more than 90% with 7% false positives on an average, Conclusion/Recommendations: It is observed that the neural network is capable of detecting skin in complex lighting and background environments. The classifier has the ability to classify the skin pixels belonging to people from different ethnic groups even when they are present simultaneously in an image. The proper choice of optimum classification threshold that varies from image to image is an issue here. Automatic computation of this optimum threshold for each image is desired in practical skin detection applications. This issue can be taken up as a future study, which will enable us to perform fully automatic skin segmentation with reduced false positives.

Commanding a humanoid to move objects in a multimodal language

This article describes a study on a humanoid robot that moves objects at the request of its users. The robot understands commands in a multimodal language which combines spoken messages and two types of hand gesture. All of ten novice users directed the robot using gestures when they were asked to spontaneously direct the robot to move objects after learning the language for a short period of time. The success rate of multimodal commands was over 90%, and the users completed their tasks without trouble. They thought that gestures were preferable to, and as easy as, verbal phrases to inform the robot of action parameters such as direction, angle, step, width, and height. The results of the study show that the language is fairly easy for nonexperts to learn, and can be made more effective for directing humanoids to move objects by making the language more sophisticated and improving our gesture detector.

Simulating a Smartboard by Real-Time Gesture Detection in Lecture Videos

Gesture plays an important role for recognizing lecture activities in video content analysis. In this paper, we propose a real-time gesture detection algorithm by integrating cues from visual, speech and electronic slides. In contrast to the conventional ldquocomplete gesturerdquo recognition, we emphasize detection by the prediction from ldquoincomplete gesturerdquo. Specifically, intentional gestures are predicted by the modified hidden Markov model (HMM) which can recognize incomplete gestures before the whole gesture paths are observed. The multimodal correspondence between speech and gesture is exploited to increase the accuracy and responsiveness of gesture detection. In lecture presentation, this algorithm enables the on-the-fly editing of lecture slides by simulating appropriate camera motion to highlight the intention and flow of lecturing. We develop a real-time application, namely simulated smartboard, and demonstrate the feasibility of our prediction algorithm using hand gesture and laser pen with simple setup without involving expensive hardware.

Gesture recognition in virtual reality

In this paper, our purpose is to investigate the system architectures and thresholds to detect gestures for a gesture-based system. We developed two gesture recognition systems: DESigning In virtual Reality (DesIRe) and DRiving for disabled (DRive). DesIRe allows real-time dynamic interaction with an object to perform a number of design operations, controlling 1 LED and wearing Datagloves. DRive allows a quadriplegic person to control a car interface taking input from 2 LEDs and shoulder movements only, wearing a garment with 2 LEDs. We developed an interface using a stereoscopic display (VISOR) and various motion capture systems integrating Vizard Virtual Reality Toolkit, an optical tracking system, datagloves, and a sensor-based jacket. In this paper, we describe a generic system architecture for gesture detection and present an approach to represent the finger states using binary digits. We have found this approach is sufficient enough and have not detected any specific delays in the detection of gestures using different input systems.

View invariant gesture recognition using the CSEM SwissRanger SR-2 camera

This paper introduces the use of range information acquired by a CSEM SwissRanger SR-2 camera for view invariant recognition of one and two arms gestures. The range data enables motion detection and 3D representation of gestures. Motion is detected by double difference range images and filtered by a hysteresis bandpass filter. Gestures are represented by concatenating harmonic shape contexts over time. This representation allows for a view invariant matching of the gestures. The system is trained on gestures from one viewpoint and evaluated on gestures from other viewpoints. The results show a recognition rate of 93.75%.

Head gestures for perceptual interfaces: The role of context in improving recognition

Head pose and gesture offer several conversational grounding cues and are used extensively in face-to-face interaction among people. To accurately recognize visual feedback, humans often use contextual knowledge from previous and current events to anticipate when feedback is most likely to occur. In this paper we describe how contextual information can be used to predict visual feedback and improve recognition of head gestures in human–computer interfaces. Lexical, prosodic, timing, and gesture features can be used to predict a user's visual feedback during conversational dialog with a robotic or virtual agent. In non-conversational interfaces, context features based on user–interface system events can improve detection of head gestures for dialog box confirmation or document browsing. Our user study with prototype gesture-based components indicate quantitative and qualitative benefits of gesture-based confirmation over conventional alternatives. Using a discriminative approach to contextual prediction and multi-modal integration, performance of head gesture detection was improved with context features even when the topic of the test set was significantly different than the training set.

Orientation sensing for gesture-based interaction with smart artifacts

Orientation sensing is considered an important means to implement embedded technology enhanced artifacts (often referred to as ‘smart artifacts’), exhibiting embodied means of interaction based on their position, orientation, and the respective dynamics. Considering artifacts subject to manual (or ‘by-hand’) manipulation by the user, we identify hand worn, hand carried and (hand) graspable real world objects as exhibiting different artifact orientation dynamics, justifying an analysis along these three categories. We refer to orientation dynamics as ‘gestures’ in an abstract sense, and present a general framework for orientation sensor based gesture recognition. The framework specification is independent of sensor technology and classification methods, and elaborates an application-independent set of gestures. It enables multi sensor interoperability and it accommodates a variable number of sensors. A core component of the framework is a gesture library that contains gestures from three categories: hand gestures, gestures of artifact held permanently and gestures of artifact that are detached from the hand and are manipulated occasionally. An inertial orientation sensing based gesture detection and recognition system is developed and composed into a gesture-based interaction development framework. The use of this framework is demonstrated with the development of tangible remote controls for a media player, both in hardware and in software.

Gesture Recognition: The Gesture Segmentation Problem

The gesture segmentation problem is introduced as the first step towards visual gesture recognition i.e. with the detection, analysis and recognition of gestures from sequences of real images. Our gesture segmentation scheme is composed of two steps: accurate gesture contour tracking in space domain, and continuous tracking in time domain. Experimental results and implementations issues are presented.

Driving the virtual orchestra with a conductor gesture interpreter

The interactive multichannel computer music system known as the ‘‘virtual orchestra’’ (VO) has been used several times in professional opera and theater as an alternative to a live pit orchestra. In each case, the VO has been operated by a ‘‘tapper’’ who follows the conductor and operates the computer which controls the VO. Recent research in noncontact, video-based conductor gesture interpretation has shown interesting possibilities in tempo and limited musical dynamics management when the gesture interpreter is directly connected to the VO controller. Two different approaches to conductor gesture detection will be discussed and the virtual orchestra will be demonstrated.