Facial Feature Points Research Articles

3D 스캔 데이터를 이용한 얼굴 애니메이션 시스템

In this paper, we describe the development of a system for generating a 3-dimensional human face using 3D scanned facial data and photo images, and morphing animation. The system comprises a facial feature input tool, a 3-dimensional texture mapping interface, and a 3-dimensional facial morphing interface. The facial feature input tool supports texture mapping and morphing animation - facial morphing areas between two facial models are defined by inputting facial feature points interactively. The texture mapping is done first by means of three photo images - a front and two side images - of a face model. The morphing interface allows for the generation of a morphing animation between corresponding areas of two facial models after texture mapping. This system allows users to interactively generate morphing animations between two facial models, without programming, using 3D scanned facial data and photo images.

A graphical model based solution to the facial feature point tracking problem

In this paper a facial feature point tracker that is motivated by applications such as human–computer interfaces and facial expression analysis systems is proposed. The proposed tracker is based on a graphical model framework. The facial features are tracked through video streams by incorporating statistical relations in time as well as spatial relations between feature points. By exploiting the spatial relationships between feature points, the proposed method provides robustness in real-world conditions such as arbitrary head movements and occlusions. A Gabor feature-based occlusion detector is developed and used to handle occlusions. The performance of the proposed tracker has been evaluated on real video data under various conditions including occluded facial gestures and head movements. It is also compared to two popular methods, one based on Kalman filtering exploiting temporal relations, and the other based on active appearance models (AAM). Improvements provided by the proposed approach are demonstrated through both visual displays and quantitative analysis.

Robust Facial Feature Tracking Using Shape-Constrained Multiresolution-Selected Linear Predictors

This paper proposes a learned data-driven approach for accurate, real-time tracking of facial features using only intensity information. The task of automatic facial feature tracking is nontrivial since the face is a highly deformable object with large textural variations and motion in certain regions. Existing works attempt to address these problems by either limiting themselves to tracking feature points with strong and unique visual cues (e.g., mouth and eye corners) or by incorporating a priori information that needs to be manually designed (e.g., selecting points for a shape model). The framework proposed here largely avoids the need for such restrictions by automatically identifying the optimal visual support required for tracking a single facial feature point. This automatic identification of the visual context required for tracking allows the proposed method to potentially track any point on the face. Tracking is achieved via linear predictors which provide a fast and effective method for mapping pixel intensities into tracked feature position displacements. Building upon the simplicity and strengths of linear predictors, a more robust biased linear predictor is introduced. Multiple linear predictors are then grouped into a rigid flock to further increase robustness. To improve tracking accuracy, a novel probabilistic selection method is used to identify relevant visual areas for tracking a feature point. These selected flocks are then combined into a hierarchical multiresolution LP model. Finally, we also exploit a simple shape constraint for correcting the occasional tracking failure of a minority of feature points. Experimental results show that this method performs more robustly and accurately than AAMs, with minimal training examples on example sequences that range from SD quality to Youtube quality. Additionally, an analysis of the visual support consistency across different subjects is also provided.

In and out vision-based driver-interactive assistance system

The overall driving environment consists of the Traffic environment, vehicle and driver states (TVD). advanced driver assistance Systems (ADAS) must consider not only information on each of the TVD states but also their context. Recent research has focused on making more efficient and effective assistance systems by fusing all the information from the TVD states. Based on this research trend, this paper focuses on decision-level fusion to estimate the level of danger of a warning by using visual information of the traffic environment and the driver state. The driver state consists of the gazing region and the facial feature points that are obtained using the active appearance model (AAM). The traffic environment state consists of time to collision (TTC), time to lane Crossing (TLC), and lane color information, which are obtained from the environment in front of the vehicle, i.e., position of lanes and other vehicles. Warnings against lane-off, collision, and driver inattention are generated by fusing this vision-based information from inside and outside the vehicle. The experimental results prove that our vision-based interactive driver assistance system reduces most useless warnings.

Survey on Various Gesture Recognition Techniques for Interfacing Machines Based on Ambient Intelligence

Gesture recognition is mainly apprehensive on analyzing the functionality of human wits. The main goal of gesture recognition is to create a system which can recognize specific human gestures and use them to convey information or for device control. Hand gestures provide a separate complementary modality to speech for expressing ones ideas. Information associated with hand gestures in a conversation is degree,discourse structure, spatial and temporal structure. The approaches present can be mainly divided into Data-Glove Based and Vision Based approaches. An important face feature point is the nose tip. Since nose is the highest protruding point from the face. Besides that, it is not affected by facial expressions.Another important function of the nose is that it is able to indicate the head pose. Knowledge of the nose location will enable us to align an unknown 3D face with those in a face database. Eye detection is divided into eye position detection and eye contour detection. Existing works in eye detection can be classified into two major categories: traditional image-based passive approaches and the active IR based approaches. The former uses intensity and shape of eyes for detection and the latter works on the assumption that eyes have a reflection under near IR illumination and produce bright/dark pupil effect. The traditional methods can be broadly classified into three categories: template based methods,appearance based methods and feature based methods. The purpose of this paper is to compare various human Gesture recognition systems for interfacing machines directly to human wits without any corporeal media in an ambient environment.

A SfM-based 3D face reconstruction method robust to self-occlusion by using a shape conversion matrix

This paper presents a 3D face reconstruction method using multiple 2D face images. Structure from motion (SfM) methods, which have been widely used to reconstruct 3D faces, are vulnerable to point correspondence errors caused by self-occlusion. In order to solve this problem, we propose a shape conversion matrix (SCM) which estimates the ground-truth 2D facial feature points (FFPs) from the observed 2D FFPs corrupted by self-occlusion errors. To make the SCM, the training observed 2D FFPs and ground-truth 2D FFPs are collected by using 3D face scans. An observed shape model and a ground-truth shape model are then built to represent the observed 2D FFPs and the ground-truth 2D FFPs, respectively. Finally, the observed shape model parameter is converted to the ground truth shape model parameter via the SCM. By using the SCM, the true locations of the self-occluded FFPs are estimated exactly with simple matrix multiplications. As a result, SfM-based 3D face reconstruction methods combined with the proposed SCM become more robust against point correspondence errors caused by self-occlusion, and the computational cost is significantly reduced. In experiments, the reconstructed 3D facial shape is quantitatively compared with the 3D facial shape obtained from a 3D scanner, and the results show that SfM-based 3D face reconstruction methods with the proposed SCM show a higher accuracy and a faster processing time than SfM-based 3D face reconstruction methods without the SCM.

Support vector machine–based facial-expression recognition method combining shape and appearance

Facial expression recognition can be widely used for various applications, such as emotion-based human-machine interaction, intelligent robot interfaces, face recognition robust to expression variation, etc. Previous studies have been classified as either shape- or appearance-based recognition. The shape-based method has the disadvantage that the individual variance of facial feature points exists irrespective of similar expressions, which can cause a reduction of the recognition accuracy. The appearance-based method has a limitation in that the textural information of the face is very sensitive to variations in illumination. To overcome these problems, a new facial-expression recognition method is proposed, which combines both shape and appearance information, based on the support vector machine (SVM). This research is novel in the following three ways as compared to previous works. First, the facial feature points are automatically detected by using an active appearance model. From these, the shape-based recognition is performed by using the ratios between the facial feature points based on the facial-action coding system. Second, the SVM, which is trained to recognize the same and different expression classes, is proposed to combine two matching scores obtained from the shape- and appearance-based recognitions. Finally, a single SVM is trained to discriminate four different expressions, such as neutral, a smile, anger, and a scream. By determining the expression of the input facial image whose SVM output is at a minimum, the accuracy of the expression recognition is much enhanced. The experimental results showed that the recognition accuracy of the proposed method was better than previous researches and other fusion methods.

Augmented DT-CWT feature based classification using Regularized Neighborhood Projection Discriminant Analysis for face recognition

This paper presents a novel face recognition method which integrates the Augmented Dual-Tree Complex Wavelet Transform (ADT-CWT) representation of face images and Regularized Neighborhood Projection Discriminant Analysis (RNPDA) method. ADT-CWT first derives desirable facial features characterized by spatial frequency, spatial locality, and orientation selectivity to cope with the variations due to illumination and facial expression changes. Different from DT-CWT, which does not consider the structural characteristics of the face images, our representation method not only considers the statistical property of the input features but also adopts an Eigenmask to emphasize those important facial feature points. The dimensionality of the derivation of ADT-CWT feature is further reduced by using RNPDA, which directly obtain a set of optimal eigenvectors with a simple regression framework and thus can overcome the small sample size problem of NPDA. Extensive experiments have been made to compare the recognition performance of the proposed method with some popular dimensionality reduction methods on the FERET database, the extended YALEB database and the CMU PIE database. The results verify the effectiveness of the proposed method.

Robust facial feature points extraction in color images

A method of facial feature points extraction based on improved active appearance model (AAM) with Gabor wavelet features was presented in the paper. After the pre-processing of a standard face detector and lighting compensation, the paper proposed a hybrid AAM by combining the local skin similarity with the original local grey-level appearance model. Moreover, the feature points by the hybrid AAM and their neighbors were considered by a classification problem to further refine the results. Namely, the Gabor feature around the feature points was extracted, trained by linear discriminant analysis (LDA) and classified by K Nearest Neighbor (KNN) to give the precise location of the feature points. Experimental results indicated that facial feature points can be located robustly and precisely by the proposed method.

ASM 기반의 얼굴 특징 영역 추출 및 변형된 알파 블렌딩을 이용한 가상 메이크업 프로그램

본 논문은 사용자 사진에서 ASM(Active Shape Model)을 이용하여 얼굴의 각 특징 점을 추출하고, 추출 된 특징점을 이용하여 화장할 부분의 영역을 생성 한다. 기존의 가상 메이크업 프로그램에서는 사용자가 수동적으로 몇 개의 특징 점을 정확히 선택해야 하는데서 불편함을 초래했다. 본 논문에서 제안하는 가상 메이크업 프로그램에서는 ASM을 이용하여 사용자의 입력을 필요로 하지 않는다. 자연스러운 화장 효과를 표현하기 위해서 기본적인 알파 블렌딩을 각각 화장품의 특징에 맞게 변형하여 사용자 피부색과 화장품의 색을 혼합한다. 얼굴 윤곽, 눈, 눈썹, 입술, 볼의 영역을 생성하고, Foundation, Blush, Lip Stick, Lip Liner, Eye Pencil, Eye Liner, Eye Shadow 종류의 화장을 할 수 있게 구현하였다. In this paper, facial feature areas in user picture are created by facial feature points extracted by ASM(Active Shape Model). In a existing virtual make-up application, users manually select a few features that are exactly. Users are uncomfortable with this method. We propose a virtual makeup application using ASM that does not require user input. In order to express a natural makeup, the modified alpha blendings for each cosmetic are used to blend skin color with cosmetic color. The Virtual makeup application was implemented to apply Foundation, Blush, Lip Stick, Lip Liner, Eye Pencil, Eye Liner and Eye Shadow.

Face feature-points location based on 3D transform shape search

Through three-dimension(3D)transformation shape search on two-dimension(2D)face image,the variation of the face gesture can be simulated correctly,and the nonlinear variations of face on Active Shape Model(ASM)are solved effectively.Based on standard face 3D model,the three-dimension coordinates were gotten by 2D initial shape;and 10 gesture-parameters of 3D transformation were calculated through the iterating process of 3D transformation,projecting to track the object shape.The test data show the 3D transformation shape search produces better results on searching multi-gesture face images that are not included in ASM training set.

Online subspace learning using a misalignment detection for tracking a talking face in video sequence

Facial appearance changes in video sequences represent a nonstationary data problem, because of factors such as variation in pose, illumination and facial expressions. While most algorithm, that employ fixed appearance models of the target object, are not robust to track objects in uncontrolled environments. Existing Adaptive Appearance Models ( AAMs) approaches solve this problem to an extent. However, they do not adequately track facial feature points, such as those relating to the eyes or mouth in the presence of significant expression changes. In this paper, we propose a method to combine an online and an offline learning for robust tracking of facial feature points. Our method firstly estimates facial feature points globally with a stochastic approach which allows to escape from local minimum. We then refine the feature points with a deterministic approach. The tracked results are filtered by offline learning approach to ensure rejection of poorly aligned targets. This allows the proposed tracker to significantly improves robustness against appearance changes and occlusions. Experiment results on tracking facial feature points in long video sequences with a wide range of facial expressions in head movement demonstrate the effectiveness and robustness of our tracker.

Face Description with Local Invariant Features: Application to Face Recognition

Automatic face recognition system based on local feature detection and feature extraction techniques is presented. The method works on color face images and performs face localization initially. It then detects and selects important fiducial facial points and characterizes them by bank of Gabor filters (jets). A well known PCA technique is used to reduce the dimensionality of jets and recognition is realized by measuring the similarity between different jets in eigenspace. A complete investigation on the proposed system is conducted, which covers face recognition under pose, illumination and expression variations of the subjects. The performance of the proposed system is compared with standard methods and it shows the superiority of the proposed system. The proposed approach provides excellent performance by using only 30 fiducial facial feature points and suggests that L1 distance metric is most suitable to measure image similarity in eigenspace. It is an attractive finding that proposed local feature based method is most suitable for machine face recognition application because of its robustness to all types of image variations and computational efficiency.

Dynamics Analysis of Facial Expression Changes for Person Identification

We propose a new method for analyzing dynamics of facial expression changes to identify persons. Several methods have been proposed to identify persons using facial images. In most methods, variations in facial expressions are one trouble factor because an input face image does not always contain the same facial expression as training images. However, the dynamics of facial expression changes are one measure of personal characteristics. In the proposed method, facial feature points are extracted using Active Appearance Models (AAMs) in the first frame of each video. They are then tracked using the Lucas-Kanade (LK) based feature point tracking method. Next, the starting and ending frames of facial expression changes are extracted by differences in the facial feature points' position between two successive frames. Finally, a feature vector is obtained as the sequence of the 2D coordinate variations of facial feature points. In the identification phase, an input feature vector is classified by calculating the distance between the input vector and the training vectors using dynamic programming matching (DP matching). We show the effectiveness of the proposed method using facial expression videos of the Facial Expressions and Emotions Database from Technical University of Munich (FEEDTUM database).

Optimal feature selection for 3D facial expression recognition using coarse-to-fine classification

Automatic facial expression recognition for novel individuals from 3D face data is a challenging task in pattern analysis. This paper describes a feature selection process for pose-invariant 3D facial expression recognition. The process provides a lower dimensional subspace representation, which is optimized to improve the classification accuracy, retrieved from geometrical localization of facial feature points to classify facial expressions. Fisher criterion-based approach is adopted to provide a basis for the optimal selection of features. Two-stage probabilistic neural network architecture is employed as a classifier to recognize the facial expressions. In the first stage, which can be regarded as the coarse classification, the facial expressions are classified into one of the three expression groups formed using seven basic facial expressions. In the fine classification stage, final expression is determined by using within group classification. Facial expressions such as Neutral, Anger, Disgust, Fear, Happiness, Sadness, and Surprise are successfully recognized with an average recognition rate of 93.72%.

Automatic Location of Facial Feature Points and Synthesis of Facial Sketches Using Direct Combined Model

Automatically locating multiple feature points (i.e., the shape) in a facial image and then synthesizing the corresponding facial sketch are highly challenging since facial images typically exhibit a wide range of poses, expressions, and scales, and have differing degrees of illumination and/or occlusion. When the facial sketches are to be synthesized in the unique sketching style of a particular artist, the problem becomes even more complex. To resolve these problems, this paper develops an automatic facial sketch synthesis system based on a novel direct combined model (DCM) algorithm. The proposed system executes three cascaded procedures, namely, 1) synthesis of the facial shape from the input texture information (i.e., the facial image); 2) synthesis of the exaggerated facial shape from the synthesized facial shape; and 3) synthesis of a sketch from the original input image and the synthesized exaggerated shape. Previous proposals for reconstructing facial shapes and synthesizing the corresponding facial sketches are heavily reliant on the quality of the texture reconstruction results, which, in turn, are highly sensitive to occlusion and lighting effects in the input image. However, the DCM approach proposed in this paper accurately reconstructs the facial shape and then produces lifelike synthesized facial sketches without the need to recover occluded feature points or to restore the texture information lost as a result of unfavorable lighting conditions. Moreover, the DCM approach is capable of synthesizing facial sketches from input images with a wide variety of facial poses, gaze directions, and facial expressions even when such images are not included within the original training data set.

Image Ratio Features for Facial Expression Recognition Application

Video-based facial expression recognition is a challenging problem in computer vision and human-computer interaction. To target this problem, texture features have been extracted and widely used, because they can capture image intensity changes raised by skin deformation. However, existing texture features encounter problems with albedo and lighting variations. To solve both problems, we propose a new texture feature called image ratio features. Compared with previously proposed texture features, e.g., high gradient component features, image ratio features are more robust to albedo and lighting variations. In addition, to further improve facial expression recognition accuracy based on image ratio features, we combine image ratio features with facial animation parameters (FAPs), which describe the geometric motions of facial feature points. The performance evaluation is based on the Carnegie Mellon University Cohn-Kanade database, our own database, and the Japanese Female Facial Expression database. Experimental results show that the proposed image ratio feature is more robust to albedo and lighting variations, and the combination of image ratio features and FAPs outperforms each feature alone. In addition, we study asymmetric facial expressions based on our own facial expression database and demonstrate the superior performance of our combined expression recognition system.

Facial feature localization using weighted vector concentration approach

We propose an efficient and generic facial feature localization method based on a weighted vector concentration approach. Our method does not require any specific priors on facial shape but implicitly learns its structural information from a training data. Unlike previous work, facial feature points are globally estimated by the concentration of directional vectors from sampling points on a face region, and those vectors are weighted by using local likelihood patterns which discriminate the appropriate position of the feature points. The directional vectors and local likelihood patterns are provided through nearest neighbor search between local patterns around the sampling points and a trained codebook of extended templates. The combination of the global vector concentration and the verification with the local likelihood patterns achieves robust facial feature point detection. We demonstrate that our method outperforms state-of-the-art method based on the Active Shape Models in our evaluation.

AAM 기반 얼굴 표정 인식을 위한 입술 특징점 검출 성능 향상 연구

AAM(Active Appearance Model)은 PCA(Principal Component Analysis)를 기반으로 객체의 형태(shape)와 질감(texture) 정보에 대한 통계적 모델을 통해 얼굴의 특징점을 검출하는 알고리즘으로 얼굴인식, 얼굴 모델링, 표정인식과 같은 응용에 널리 사용되고 있다. 하지만, AAM알고리즘은 초기 값에 민감하고 입력영상이 학습 데이터 영상과의 차이가 클 경우에는 검출 에러가 증가되는 문제가 있다. 특히, 입을 다문 입력얼굴 영상의 경우에는 비교적 높은 검출 정확도를 나타내지만, 사용자의 표정에 따라 입을 벌리거나 입의 모양이 변형된 얼굴 입력 영상의 경우에는 입술에 대한 검출 오류가 매우 증가되는 문제점이 있다. 이러한 문제점을 해결하기 위해 본 논문에서는 입술 특징점 검출을 통해 정확한 입술 영역을 검출한 후에 이 정보를 이용하여 AAM을 수행함으로써 얼굴 특징점 검출 정확성을 향상시키는 방법을 제안한다. 본 논문에서는 AAM으로 검출한 얼굴 특징점 정보를 기반으로 초기 입술 탐색 영역을 설정하고, 탐색 영역 내에서 Canny 경계 검출 및 히스토그램 프로젝션 방법을 이용하여 입술의 양 끝점을 추출한 후, 입술의 양 끝점을 기반으로 재설정된 탐색영역 내에서 입술의 칼라 정보와 에지 정보를 함께 결합함으로써 입술 검출의 정확도 및 처리속도를 향상시켰다. 실험결과, AAM 알고리즘을 단독으로 사용할 때보다, 제안한 방법을 사용하였을 경우 입술 특징점 검출 RMS(Root Mean Square) 에러가 4.21픽셀만큼 감소하였다. AAM(Active Appearance Model) is an algorithm to extract face feature points with statistical models of shape and texture information based on PCA(Principal Component Analysis). This method is widely used for face recognition, face modeling and expression recognition. However, the detection performance of AAM algorithm is sensitive to initial value and the AAM method has the problem that detection error is increased when an input image is quite different from training data. Especially, the algorithm shows high accuracy in case of closed lips but the detection error is increased in case of opened lips and deformed lips according to the facial expression of user. To solve these problems, we propose the improved AAM algorithm using lip feature points which is extracted based on a new lip detection algorithm. In this paper, we select a searching region based on the face feature points which are detected by AAM algorithm. And lip corner points are extracted by using Canny edge detection and histogram projection method in the selected searching region. Then, lip region is accurately detected by combining color and edge information of lip in the searching region which is adjusted based on the position of the detected lip corners. Based on that, the accuracy and processing speed of lip detection are improved. Experimental results showed that the RMS(Root Mean Square) error of the proposed method was reduced as much as 4.21 pixels compared to that only using AAM algorithm.

Face Recognition by Extending Elastic Bunch Graph Matching with Particle Swarm Optimization

Elastic Bunch Graph Matching is one of the well known methods proposed for face recognition. In this work, we propose several extensions to Elastic Bunch Graph Matching and its recent variant Landmark Model Matching. We used data from the FERET database for experimentations and to compare the proposed methods. We apply Particle Swarm Optimization to improve the face graph matching procedure in Elastic Bunch Graph Matching method and demonstrate its usefulness. Landmark Model Matching depends solely on Gabor wavelets for feature extraction to locate the landmarks (facial feature points). We show that improvements can be made by combining gray-level profiles with Gabor wavelet features for feature extraction. Furthermore, we achieve improved recognition rates by hybridizing Gabor wavelet with eigenface features found by Principal Component Analysis, which would provide information contained in the overall appearance of a face. We use Particle Swarm Optimization to fine tune the hybridization weights. Results of both fully automatic and partially automatic versions of all methods are presented. The best-performing method improves the recognition rate up to 22.6% and speeds up the processing time by 8 times over the Elastic Bunch Graph Matching for the fully automatic case.