Facial Feature Tracking Research Articles

3D face pose estimation by a robust real time tracking of facial features

In this paper, we present a new 3D head pose estimating approach based on real-time facial feature tracking and facial feature recovering method which copes with the surrounding light variation and various occlusion. The major facial features are obtained by Haar-like feature detection along with AdaBoost learning from an input video image. The detected facial features are robustly tracked by optical flow with a template matching scheme which continuously compensates for losing track of the initially detected features in a sequence of input images. The head pose of an input face image can be obtained by evaluating 3D information of facial features from the detected 2D eye-points, nose and lip. From the experiments, the proposed method shows effectiveness in tracking and recovering facial features and produces reliable result in head pose estimation.

Explicit and implicit emotional expression in bulimia nervosa in the acute state and after recovery.

Expression of emotional state is considered to be a core facet of an individual's emotional competence. Emotional processing in BN has not been often studied and has not been considered from a broad perspective. This study aimed at examining the implicit and explicit emotional expression in BN patients, in the acute state and after recovery. Sixty-three female participants were included: 22 BN, 22 recovered BN (R-BN), and 19 healthy controls (HC). The clinical cases were drawn from consecutive admissions and diagnosed according to DSM-IV-TR diagnostic criteria. Self reported (explicit) emotional expression was measured with State-Trait Anger Expression Inventory-2, State-Trait Anxiety Inventory, and Symptom Check List-90 items-Revised. Emotional facial expression (implicit) was recorded by means of an integrated camera (by detecting Facial Feature Tracking), during a 20 minutes therapeutic video game. In the acute illness explicit emotional expression [anxiety (p<0.001) and anger (p<0.05)] was increased. In the recovered group this was decreased to an intermediate level between the acute illness and healthy controls [anxiety (p<0.001) and anger (p<0.05)]. In the implicit measurement of emotional expression patients with acute BN expressed more joy (p<0.001) and less anger (p<0.001) than both healthy controls and those in the recovered group. These findings suggest that there are differences in the implicit and explicit emotional processing in BN, which is significantly reduced after recovery, suggesting an improvement in emotional regulation.

Conditional Gabor phase–based disparity estimation applied to facial tracking for person–specific facial action recognition: a preliminary study

Within the affective computing research field, researchers are still facing a big challenge to establish automated systems to recognize human emotions from video sequences. Performances are quite dependent on facial feature localization and tracking. In this paper, we present a method based on a coarse---to---fine paradigm to characterize a set of facial fiducial points using a bank of Gabor filters. When the first face image is captured, the coarsest level is used to estimate a rough position for each facial feature. Afterward, a coarse---to---fine displacement refinement on an image pyramid is performed. The positions are then tracked over the subsequent frames using a modification of a fast Gabor---phase based technique. This includes a redefinition of the confidence measure and introduces an iterative conditional disparity estimation procedure. We used the proposed tracking process to implement a personalized feature---based facial action recognition framework, motivated by the fact that the same facial expression may vary differently across humans. Experimental results show that the facial feature points can be localized with high accuracy and tracked with sufficient precision leading to a better facial action recognition performance.

AUTOMATIC EYE TEMPLATE GENERATION AND TRACKING

Eye blinking is a physiological necessity for humans. This method automatically locates the user’s eye by detecting eye blinks. A system is the improvement of driver carefulness and accident reduction. The driver’s face is tracked while he is driving and he is warned if there seems to be an alerting fact that can result in an accident such as sleepy eyes, or looking out of the road. Furthermore, with a facial feature tracker, it becomes possible to play a synthesized avatar so that it imitates the expressions of the performer. For a user who is incapable of using her hands, a facial expression controller may be a solution to send limited commands to a computer. Eye blinking is one of the prominent areas to solve many real world problems. The process of blink detection consists of two phases. These are eye tracking followed by detection of blink. The work that has been carried out for eye tracking only is not suitable for eye blink detection. Therefore some approaches had been proposed for eye tracking along with eyes blink detection. In this thesis, real time implementation is done to count number of eye blinks in an image sequence. At last after analyzing all these approaches some of the parameters we obtained on which better performance of eye blink detection algorithm depend. This project focuses on automatic eye blink detection in real time. The aim of this thesis is to count the number of eye blinks in a video. This project will be performed on a video database of the facial expressions.

Utility of optical facial feature and arm movement tracking systems to enable text communication in critically ill patients who cannot otherwise communicate

PurposePatients recovering from critical illness especially those with critical illness related neuropathy, myopathy, or burns to face, arms and hands are often unable to communicate by writing, speech (due to tracheostomy) or lip reading. This may frustrate both patient and staff.Two low cost movement tracking systems based around a laptop webcam and a laser/optical gaming system sensor were utilised as control inputs for on-screen text creation software and both were evaluated as communication tools in volunteers. MethodsTwo methods were used to control an on-screen cursor to create short sentences via an on-screen keyboard: (i) webcam-based facial feature tracking, (ii) arm movement tracking by laser/camera gaming sensor and modified software. 16 volunteers with simulated tracheostomy and bandaged arms to simulate communication via gross movements of a burned limb, communicated 3 standard messages using each system (total 48 per system) in random sequence. ResultsTen and 13 minor typographical errors occurred with each system respectively, however all messages were comprehensible. Speed of sentence formation ranged from 58 to 120s with the facial feature tracking system, and 60–160s with the arm movement tracking system. The average speed of sentence formation was 81s (range 58–120) and 104s (range 60–160) for facial feature and arm tracking systems respectively, (P<0.001, 2-tailed independent sample t-test). ConclusionBoth devices may be potentially useful communication aids in patients in general and burns critical care units who cannot communicate by conventional means, due to the nature of their injuries.

Robust Real-Time Facial Expressions Tracking and Recognition

Enabling computer systems to track and recognize facial expressions and then infer emotions from about real time video is a challenging research topic. In this work, a real time approach to emotion recognition through facial expression in live video is introduced. Several automatic methods for face localization, facial feature tracker, and facial expression recognition are employed. A robust tracking is achieved by using a face mask to resolve mismatches that could be generated during the tracking process. Action units (AUs) are then built to recognize the facial expression in each frame. The main objective of this work is to provide a prediction ability of a human behavior such as a crime, angry or for being nervous.

Facial Feature Tracking Using Efficient Particle Filter and Active Appearance Model

For natural human-robot interaction, the location and shape of facial features in a real environment must be identified. One robust method to track facial features is by using a particle filter and the active appearance model. However, the processing speed of this method is too slow for utilization in practice. In order to improve the efficiency of the method, we propose two ideas: (1) changing the number of particles situationally, and (2) switching the prediction model depending upon the degree of the importance of each particle using a combination strategy and a clustering strategy. Experimental results show that the proposed method is about four times faster than the conventional method using a particle filter and the active appearance model, without any loss of performance.

안정적인 실시간 얼굴 특징점 추적과 감정인식 응용

Facial feature extraction and tracking are essential steps in human-robot-interaction (HRI) field such as face recognition, gaze estimation, and emotion recognition. Active shape model (ASM) is one of the successful generative models that extract the facial features. However, applying only ASM is not adequate for modeling a face in actual applications, because positions of facial features are unstably extracted due to limitation of the number of iterations in the ASM fitting algorithm. The unaccurate positions of facial features decrease the performance of the emotion recognition. In this paper, we propose real-time facial feature extraction and tracking framework using ASM and LK optical flow for emotion recognition. LK optical flow is desirable to estimate time-varying geometric parameters in sequential face images. In addition, we introduce a straightforward method to avoid tracking failure caused by partial occlusions that can be a serious problem for tracking based algorithm. Emotion recognition experiments with k-NN and SVM classifier shows over 95% classification accuracy for three emotions: joy, anger, and disgust.

Realtime facial animation with on-the-fly correctives

We introduce a real-time and calibration-free facial performance capture framework based on a sensor with video and depth input. In this framework, we develop an adaptive PCA model using shape correctives that adjust on-the-fly to the actor's expressions through incremental PCA-based learning. Since the fitting of the adaptive model progressively improves during the performance, we do not require an extra capture or training session to build this model. As a result, the system is highly deployable and easy to use: it can faithfully track any individual, starting from just a single face scan of the subject in a neutral pose. Like many real-time methods, we use a linear subspace to cope with incomplete input data and fast motion. To boost the training of our tracking model with reliable samples, we use a well-trained 2D facial feature tracker on the input video and an efficient mesh deformation algorithm to snap the result of the previous step to high frequency details in visible depth map regions. We show that the combination of dense depth maps and texture features around eyes and lips is essential in capturing natural dialogues and nuanced actor-specific emotions. We demonstrate that using an adaptive PCA model not only improves the fitting accuracy for tracking but also increases the expressiveness of the retargeted character.

Using kernels for a video-based mouse-replacement interface

Some people cannot use their hands to control a computer mouse due to conditions such as cerebral palsy or multiple sclerosis. For these individuals, there are various mouse-replacement solutions. One approach is to enable them to control the mouse pointer using head motions captured with a web camera. One such system, the Camera Mouse, uses an optical flow approach to track a manually-selected small patch of the subject’s face, such as the nostril or the edge of the eyebrow. The optical flow tracker may lose the facial feature when the tracked image patch drifts away from the initially-selected feature or when a user makes a rapid head movement. To address the problem of feature loss, we developed and incorporated the Kernel-Subset-Tracker into the Camera Mouse. The Kernel-Subset-Tracker is an exemplar-based method that uses a training set of representative images to produce online templates for positional tracking. We designed the augmented Camera Mouse so that it can compute these templates in real time, employing kernel techniques traditionally used for classification. We propose three versions of the Kernel-Subset-Tracker, each using a different kernel, and compared their performance to the optical-flow tracker under five different experimental conditions. Our experiments with test subjects show that augmenting the Camera Mouse with the Kernel-Subset-Tracker improves communication bandwidth statistically significantly. Tracking of facial features was accurate, without feature drift, even during rapid head movements and extreme head orientations. We conclude by describing how the Camera Mouse augmented with the Kernel-Subset-Tracker enabled a stroke-victim with severe motion impairments to communicate via an on-screen keyboard.

Visual tracking of hands, faces and facial features of multiple persons

This paper presents an integrated approach for tracking hands, faces and specific facial features (eyes, nose, and mouth) in image sequences. For hand and face tracking, we employ a state-of-the-art blob tracker which is specifically trained to track skin-colored regions. In this paper we extend the skin color tracker by proposing an incremental probabilistic classifier, which can be used to maintain and continuously update the belief about the class of each tracked blob, which can be left-hand, right hand or face as well as to associate hand blobs with their corresponding faces. An additional contribution of this paper is related to the employment of a novel method for the detection and tracking of specific facial features within each detected facial blob which consists of an appearance-based detector and a feature-based tracker. The proposed approach is intended to provide input for the analysis of hand gestures and facial expressions that humans utilize while engaged in various conversational states with robots that operate autonomously in public places. It has been integrated into a system which runs in real time on a conventional personal computer which is located on a mobile robot. Experimental results confirm its effectiveness for the specific task at hand.

Facial Features Tracking Using Auxiliary Particle Filtering and Observation Model Based on Bhattacharyya Distance

Recently, particle filtering has become an effective algorithm for facial feature tracking. One problem with particle filtering is that as the dimensionality of the state space increases, a large number of particles that are propagated from the previous time are wasted in area where they have low observation probability, hence a very large number of particles are necessary to track the state and, as a result the complexity increases and the speed of algorithm reduces. In this research, auxiliary particle filter with factorized likelihoods is used in order to overcome this problem. In a tracking approach, the estimated state is updated by incorporating the new observations. Therefore an observation model is needed. In this paper a novel color-based observation model that is invariant to changes in illumination intensity is proposed. The proposed observation model employs the Bhattacharyya distance to update a prior distribution calculated by the particle filter. In this paper experimentally is showed that the proposed algorithm clearly outperforms multiple independent template tracking.

Human-centric design personalization of 3D glasses frame in markerless augmented reality

This paper presents a virtual try-on system based on augmented reality for design personalization of facial accessory products. The system offers several novel functions that support real-time evaluation and modification of eyeglasses frame. 3D glasses model is embedded within video stream of the person who is wearing the glasses. Machine learning algorithms are developed for instantaneous tracking of facial features without use of markers. The tracking result enables continuously positioning of the glasses model on the user’s face while it is moving during the try-on process. In addition to color and texture, the user can instantly modify the glasses shape through simple semantic parameters. These functions not only facilitate evaluating products highly interactive with human users, but also engage them in the design process. This work has thus implemented the concept of human-centric design personalization.

Robust Facial Feature Tracking Using Shape-Constrained Multiresolution-Selected Linear Predictors

This paper proposes a learned data-driven approach for accurate, real-time tracking of facial features using only intensity information. The task of automatic facial feature tracking is nontrivial since the face is a highly deformable object with large textural variations and motion in certain regions. Existing works attempt to address these problems by either limiting themselves to tracking feature points with strong and unique visual cues (e.g., mouth and eye corners) or by incorporating a priori information that needs to be manually designed (e.g., selecting points for a shape model). The framework proposed here largely avoids the need for such restrictions by automatically identifying the optimal visual support required for tracking a single facial feature point. This automatic identification of the visual context required for tracking allows the proposed method to potentially track any point on the face. Tracking is achieved via linear predictors which provide a fast and effective method for mapping pixel intensities into tracked feature position displacements. Building upon the simplicity and strengths of linear predictors, a more robust biased linear predictor is introduced. Multiple linear predictors are then grouped into a rigid flock to further increase robustness. To improve tracking accuracy, a novel probabilistic selection method is used to identify relevant visual areas for tracking a feature point. These selected flocks are then combined into a hierarchical multiresolution LP model. Finally, we also exploit a simple shape constraint for correcting the occasional tracking failure of a minority of feature points. Experimental results show that this method performs more robustly and accurately than AAMs, with minimal training examples on example sequences that range from SD quality to Youtube quality. Additionally, an analysis of the visual support consistency across different subjects is also provided.

A Robust Facial Feature Tracking Method Based on Optical Flow and Prior Measurement

Cognitive informatics (CI) is a research area including some interdisciplinary topics. Visual tracking is not only an important topic in CI, but also a hot topic in computer vision and facial expression recognition. In this paper, a novel and robust facial feature tracking method is proposed, in which Kanade-Lucas-Tomasi (KLT) optical flow is taken as basis. The prior method of measurement consisting of pupils detecting features restriction and errors and is used to improve the predictions. Simulation experiment results show that the proposed method is superior to the traditional optical flow tracking. Furthermore, the proposed method is used in a real time emotion recognition system and good recognition result is achieved.

Assessing Outcomes in Facial Reanimation: Evaluation and Validation of the SMILE System for Measuring Lip Excursion During Smiling

Assessing Outcomes in Facial Reanimation: Evaluation and Validation of the SMILE System for Measuring Lip Excursion During Smiling

Real-time hands, face and facial features detection and tracking: Application to cognitive rehabilitation tests monitoring

In this paper, a marker-free computer vision system for cognitive rehabilitation tests monitoring is presented. The system monitors and analyzes the correct and incorrect realization of a set of psicomotricity exercises in which a hand has to touch a facial feature. The monitoring requires different human body parts detection and tracking. Detection of face, eyes, nose, and hands is achieved with a set of classifiers built independently based on the AdaBoost algorithm. Comparisons with other detection approaches, regarding performance and applicability to the monitoring system, are presented. Face and hands tracking is accomplished through the CAMShift algorithm with independent and adaptive two-dimensional histograms of the chromaticity components of the TSL color space for the pixels inside these three regions. The TSL color space was selected after a study of five color spaces regarding skin color characterization. The system is easily implemented with a consumer-grade computer and a camera, unconstrained background and illumination and runs at more than 23 frames per second. The system was tested and achieved a successful monitoring percentage of 97.62%. The automation of the human body parts motion monitoring, its analysis in relation to the psicomotricity exercise indicated to the patient and the storage of the result of the realization of a set of exercises free the rehabilitation experts of doing such demanding tasks. The vision-based system is potentially applicable to other human–computer interface tasks with minor changes.

Facial Feature Tracking via Evolutionary Multiobjective Optimization

Facial feature tracking for model–based coding has evolved over the past decades. Of particular interest is its application in very low bit rate coding in which optimization is used to analyze head and shoulder sequences. We present the results of a computational experiment in which we apply a combination of non-dominated sorting genetic algorithm and a deterministic search to find optimal facial animation parameters at many bandwidths simultaneously. As objective functions are concerned, peak signal-to-noise ratio is maximized while the total number of facial animation parameters is minimized. Particularly, the algorithm is tested for efficiency and reliability. The results show that the overall methodology works effectively, but that a better error assessment function is needed for future study.

Welfare interface implementation using multiple facial features tracking for the disabled people

This paper proposes a multiple facial feature interface that allows disabled users with various disabilities to implement different mouse operations. Using a regular PC camera, the proposed system detects the user’s eye and mouth movements, and then interprets the communication intent to control the computer. Here, mouse movements are implemented based on the user’s eye movements, while clicking events are implemented based on the user’s mouth shapes, such as opening/closing. The proposed system is composed of three modules: facial feature detector, facial feature tracker, and mouse controller. The facial region is initially identified using a skin-color model and connected-component (CC) analysis. Thereafter, the eye regions are localized using a neural network (NN)-based texture classifier that discriminates the facial region into eye class and non-eye class, then the mouth region is localized using an edge detector. Once the eye and mouth regions are localized, they are continuously and accurately tracking using a mean-shift algorithm and template matching, respectively. Based on the tracking results, the mouse movements and clicks are then implemented. To assess the validity of the proposed method, it was applied to three applications: a web browser, ‘spelling board’, and the game ‘catching-a-bird’. The two test groups involved 34 users, and the results showed that the proposed system could be efficiently and effectively applied as a user-friendly and convenient communication device.

A robust multimodal approach for emotion recognition

Emotion recognition is one of the latest challenges in intelligent human/computer communication. Most of previous work on emotion recognition focused on extracting emotions from visual or audio information separately. A novel approach is presented in this paper, including both visual and audio from video clips, to recognize the human emotion. The Facial Animation Parameters (FAPs) compliant facial feature tracking based on GASM (GPU based Active Shape Model) is performed on the video to generate two vector streams which represent the expression feature and the visual speech one. To extract effective speech features, based on geodesic distance estimation, we develop an enhanced Lipschitz embedding to embed high dimensional acoustic features into low dimensional space. Combined with the visual vectors, the audio vector is extracted in terms of low dimensional features. Then, a tripled Hidden Markov Model is introduced to perform the recognition which allows the state asynchrony of the audio and visual observation sequences while preserving their natural correlation over time. The experimental results show that this approach outperforms the conventional approaches for emotion recognition.