Active Appearance Model Research Articles

ACTIVE APPEARANCE BASED POSE AND ILLUMINATION VARIANT FACE RECOGNITION FROM VIDEO SEQUENCES

Surveillance video provides security by monitoring the people entering and leaving the premises. Face recognition in video may act as a tool to record the information with minimal manual intervention. The major difficulty is that the faces are captured in different pose and illumination in the video. The recognition of these faces from the video sequences is complex. In this paper, we have addressed the face recognition problem in different pose and illumination.We have created a set of ten video sequences for ten people with different pose and illumination and extracted the faces at different angles and illumination from these video sequences to create a face database. Active Appearance Model (AAM) is used to create a feature vector for each face that exists in the database. Artificial Neural Network (ANN) is trained to recognize the face from the appearance feature vectors. The proposed method reduces the computational complexity involved in recognizing the faces in different pose and illumination. We have achieved a recognition rate of 96.1% on the test set of the face database.

Learning to Represent Handwritten Shapes and Words for Matching and Recognition

Writing is one of the most important forms of communication and for centuries, handwriting had been the most reliable way to preserve knowledge. However, despite the recent development of printing houses and electronic devices, handwriting is still broadly used for taking notes, doing annotations, or sketching ideas. In order to be easily accessed, there is a huge amount of handwritten documents, some of them with uncountable cultural value, that have been recently digitized. This has made necessary the development of methods able to extract information from these document images.Transferring the ability of understanding handwritten text or recognizing handwritten shapes to computers has been the goal of many researches due to its huge importance for many different fields. However, designing good representations to deal with handwritten shapes, e.g. symbols or words, is a very challenging problem due to the large variability of these kinds of shapes. One of the consequences of working with handwritten shapes is that we need representations to be, i.e., able to adapt to large intra-class variability. We need representations to be discriminative, i.e., able to learn what are the differences between classes. And, we need representations to be efficient, i.e., able to be rapidly computed and compared. Unfortunately, current techniques of handwritten shape representation for matching and recognition do not fulfill some or all of these requirements.Through this thesis we focus on the problem of learning to represent handwritten shapes aimed at retrieval and recognition tasks. Concretely, on the first part of the thesis, we focus on the general problem of representing any kind of handwritten shape. We first present a novel shape descriptor based on a deformable grid that deals with large deformations by adapting to the shape and where the cells of the grid can be used to extract different features. Then, we propose to use this descriptor to learn statistical models, based on the Active Appearance Model, that jointly learns the variability in structure and texture of a given class. Then, on the second part, we focus on a concrete application, the problem of representing handwritten words, for the tasks of word spotting, where the goal is to find all instances of a query word in a dataset of images, and recognition. First, we address the segmentation-free problem and propose an unsupervised, sliding-window-based approach that achieves state-of-the-art results in two public datasets.Second, we address the more challenging multi-writer problem, where the variability in words exponentially increases. We describe an approach in which both word images and text strings are embedded in a common vectorial subspace, and where those that represent the same word are close together. This is achieved by a combination of label embedding and attributes learning, and a common subspace regression. This leads to a low-dimensional, unified representation of word images and strings, resulting in a method that allows one to perform either image and text searches, as well as image transcription, in a unified framework. We evaluate our methods on different public datasets of both handwritten documents and natural images showing results comparable or better than the state-of-the-art on spotting and recognition tasks.

MAPPING 2D TO 3D FORENSIC FACIAL RECOGNITION VIA BIO- INSPIRED ACTIVE APPEARANCE MODEL

This research done is to solve the problems faced by digital forensic analysts in identifying a suspect captured on their CCTV. Identifying the suspect through the CCTV video footage is a very challenging task for them as it involves tedious rounds of processes to match the facial information in the video footage to a set of suspect’s images. The biggest problem faced by digital forensic analysis is modeling 2D model extracted from CCTV video as the model does not provide enough information to carry out the identification process. Problems occur when a suspect in the video is not facing the camera, the image extracted is the side image of the suspect and it is difficult to make a matching with portrait image in the database. There are also many factors that contribute to the process of extracting facial information from a video to be difficult, such as low-quality video. Through 2D to 3D image model mapping, any partial face information that is incomplete can be matched more efficiently with 3D data by rotating it to matched position. The first methodology in this research is data collection; any data obtained through video recorder. Then, the video will be converted into an image. Images are used to develop the Active Appearance Model (the 2D face model is AAM) 2D and AAM 3D. AAM is used as an input for learning and testing process involving three classifiers, which are Random Forest, Support Vector Machine (SVM), and Neural Networks classifier. The experimental results show that the 3D model is more suitable for use in face recognition as the percentage of the recognition is higher compared with the 2D model.

Robust Face Recognition from Video based on Extensive Feature Set and Fuzzy_Bat Algorithm

Background/Objectives: This paper proposes a novel method of enhancing the face recognition process from video sequence with various pose and occlusion using an extensive feature set called Pose and Occlusion Invariant Feature set (POIF) and unsupervised learning technique. Methods/Statistical Analysis: Here an extensive feature set, POIF is created using local invariant feature namely Speeded Up Robust Feature (SURF), appearance features and weighted holo-entropy to find out the uniqueness of the face image. The Active Appearance Model (AAM) has been used to find the appearance based features in the face image. The proposed feature set, POIF is used to select the key frames in the video sequence and the key frame selection is optimised using unsupervised learning method namely, Fuzzy Clustering using Bat algorithm (FC-Bat). A dictionary of keyframes is then created, using which the faces from the test video is recognized. Findings: Experimental evaluation is done in MATLAB using McGill Real-World Unconstrained Face Video Database and Honda UCSD Dataset 1. The proposed system using FC_Bat algorithm is compared with Fuzzy optimized POIF feature set and Fuzzy cmeans optimized POIF feature set and it is found that POIF with FC_Bat algorithm performs better with an accuracy of 97.5%. Applications/Improvements: The computational complexity of the proposed face recognition system is less as it uses unsupervised learning of features and best suits applications involving unlabelled data.Keywords: Face Recognition, Fuzzy_Bat Algorithm, Occlusion, Pose, Unsupervised Learning

Fully automatic face normalization and single sample face recognition in unconstrained environments

Single sample face recognition have become an important problem because of the limitations on the availability of gallery images. In many real-world applications such as passport or driver license identification, there is only a single facial image per subject available. The variations between the single gallery face image and the probe face images, captured in unconstrained environments, make the single sample face recognition even more difficult. In this paper, we present a fully automatic face recognition system robust to most common face variations in unconstrained environments. Our proposed system is capable of recognizing faces from non-frontal views and under different illumination conditions using only a single gallery sample for each subject. It normalizes the face images for both in-plane and out-of-plane pose variations using an enhanced technique based on active appearance models (AAMs). We improve the performance of AAM fitting, not only by training it with in-the-wild images and using a powerful optimization technique, but also by initializing the AAM with estimates of the locations of the facial landmarks obtained by a method based on flexible mixture of parts. The proposed initialization technique results in significant improvement of AAM fitting to non-frontal poses and makes the normalization process robust, fast and reliable. Owing to the proper alignment of the face images, made possible by this approach, we can use local feature descriptors, such as Histograms of Oriented Gradients (HOG), for matching. The use of HOG features makes the system robust against illumination variations. In order to improve the discriminating information content of the feature vectors, we also extract Gabor features from the normalized face images and fuse them with HOG features using Canonical Correlation Analysis (CCA). Experimental results performed on various databases outperform the state-of-the-art methods and show the effectiveness of our proposed method in normalization and recognition of face images obtained in unconstrained environments.

상반된 감성에 따른 안면 움직임 차이에 대한 분석

본 논문에서는 Kinect 카메라를 통해 촬영된 영상 처리를 통해 상반된 감성 자극 관점에서 안면 움직임의 차이를 분석하는 연구를 진행하였다. Russell의 2차원 감성 모델에서 원점 대칭 위치에 존재하는 두 상반된 감성인 "Sad - Excitement", "Contentment - Angry" 감성을 유발하기 위해 피험자에게 시각자극과 청각자극을 동시에 제공하였다. Kinect Face Tracking SDK에서 제공되는 121개 특징점으로 구성된 안면 active appearance model에서 안면 움직임을 잘 표현하는 31개의 주요 특징점 주변의 화소 변화를 측정하였다. 안면 근육의 비선형적 움직임 문제를 해결하기 위해 지역 이동 기반 최소거리 결정 방법(local minimum shift matching)을 사용하였다. 분석 결과, sad 감성에서는 우측 안면 움직임이 많이 나타났고, excitement 감성에서는 좌측 안면 움직임이 많이 나타남으로써 두 상반된 감성 자극에 대한 안면 움직임의 위치 또한 상반된 결과를 보였다. 또한 "Contentment" 감성에서는 좌측 안면 움직임이 많이 나타났고, "Angry" 감성에서는 안면의 좌우 구분 없이 움직임이 나타남으로써, 두 상반된 감성 자극에 대해서는 우측 안면에서 차이를 확인할 수 있었다. In this paper, a study on facial movements are analyzed in terms of opposite emotion stimuli by image processing of Kinect facial image. To induce two opposite emotion pairs such as "Sad - Excitement"and "Contentment - Angry" which are oppositely positioned onto Russell's 2D emotion model, both visual and auditory stimuli are given to subjects. Firstly, 31 main points are chosen among 121 facial feature points of active appearance model obtained from Kinect Face Tracking SDK. Then, pixel changes around 31 main points are analyzed. In here, local minimum shift matching method is used in order to solve a problem of non-linear facial movement. At results, right and left side facial movements were occurred in cases of "Sad" and "Excitement" emotions, respectively. Left side facial movement was comparatively more occurred in case of "Contentment" emotion. In contrast, both left and right side movements were occurred in case of "Angry" emotion.

Longitudinal validation of periarticular bone area and 3D shape as biomarkers for knee OA progression? Data from the FNIH OA Biomarkers Consortium

ObjectiveTo perform a longitudinal validation study of imaging bone biomarkers of knee osteoarthritis (OA) progression.MethodsWe undertook a nested case–control study within the Osteoarthritis Initiative in knees (one knee per subject)...

Efficient Unwrap Representation of Faces for Video Editing

Unwrap mosaic is a method for decomposing a video into a 2D texture and a dense mapping that enable the reconstruction of the video from the texture. This representation is useful in some frameworks because we can edit videos by simply retouching 2D textures. However, the complexity of conventional approaches is too high to be adopted in time-critical applications (it takes up to several hours). In this letter, we focus on face-related applications such as face editing and replacement, and propose a face unwrap approach for these applications. To be precise, we adopt the view-based active appearance model (AAM) trackers and estimate dense mappings from the AAM results. The AAM also provides pose information which is also exploited in building the texture map. Experimental results show that our method is very efficient compared with the conventional unwrap mosaic approach. Moreover, based on the proposed system, we develop face-related applications.

Automatic Tongue Tracking in X‐Ray Images

X-ray imaging is an effective technique toobtain the continuous motions of the vocal tract duringspeech, and Active appearance model (AAM) is a usefultool to analyze the X-ray images. However, for the task oftongue tracking in X-ray images, the accuracy of AAM fitting is insufficient. AAM aims to minimize the residual error between the model appearance and the input image. Itoften fails to accurately converge to the true landmarks. Toimprove the tracking accuracy, we propose a fitting methodby combining Constrained local model (CLM) into AAM.In our method, we first combine the objective functionsof AAM and CLM into a single objective function. Then,we project out the texture variation and derive a gradient based method to optimize the objective function. Ourmethod effectively incorporates not only the shape priorand global texture, but also local texture around each landmark. Experiments demonstrate that the proposed methodsignificantly reduces the fitting error. We also show that realistic 3D tongue animation can be created by using tonguetracking results of the X-ray images.

Automatic brain matter segmentation of computed tomography images using a statistical model: A tool to gain working time!

Brain computed tomography (CT) is useful diagnostic tool for the evaluation of several neurological disorders due to its accuracy, reliability, safety and wide availability. In this field, a potentially interesting research topic is the automatic segmentation and recognition of medical regions of interest (ROIs). Herein, we propose a novel automated method, based on the use of the active appearance model (AAM) for the segmentation of brain matter in CT images to assist radiologists in the evaluation of the images. The method described, that was applied to 54 CT images coming from a sample of outpatients affected by cognitive impairment, enabled us to obtain the generation of a model overlapping with the original image with quite good precision. Since CT neuroimaging is in widespread use for detecting neurological disease, including neurodegenerative conditions, the development of automated tools enabling technicians and physicians to reduce working time and reach a more accurate diagnosis is needed.

A deep bidirectional LSTM approach for video-realistic talking head

This paper proposes a deep bidirectional long short-term memory approach in modeling the long contextual, nonlinear mapping between audio and visual streams for video-realistic talking head. In training stage, an audio-visual stereo database is firstly recorded as a subject talking to a camera. The audio streams are converted into acoustic feature, i.e. Mel-Frequency Cepstrum Coefficients (MFCCs), and their textual labels are also extracted. The visual streams, in particular, the lower face region, are compactly represented by active appearance model (AAM) parameters by which the shape and texture variations can be jointly modeled. Given pairs of the audio and visual parameter sequence, a DBLSTM model is trained to learn the sequence mapping from audio to visual space. For any unseen speech audio, whether it is original recorded or synthesized by text-to-speech (TTS), the trained DBLSTM model can predict a convincing AAM parameter trajectory for the lower face animation. To further improve the realism of the proposed talking head, the trajectory tiling method is adopted to use the DBLSTM predicted AAM trajectory as a guide to select a smooth real sample image sequence from the recorded database. We then stitch the selected lower face image sequence back to a background face video of the same subject, resulting in a video-realistic talking head. Experimental results show that the proposed DBLSTM approach outperforms the existing HMM-based approach in both objective and subjective evaluations.

Facial expression recognition based on AAM–SIFT and adaptive regional weighting

The active appearance model (AAM), one of the most effective facial feature localization methods, is widely used in frontal facial expression recognition. However, non‐frontal facial expression recognition is important in many scenarios. Thus, we propose a new method for facial expression recognition based on AAM‐SIFT and adaptive regional weighting. First, multi‐pose AAM templates are used for pose estimation and feature point location of the facial expression image. For effective and efficient description of these feature points, a hybrid representation, which integrates gradient direction histograms based on the descriptors of scale‐invariant feature transform (SIFT) and AAM, is utilized to form AAM‐SIFT features. Meanwhile, according to different expression regions, AAM‐SIFT features are divided into different groups and the obtained adaptive weights by means of a regional weighted method based on the fuzzy C‐means (FCM) clustering algorithm. Finally, the membership degree computed by FCM, which represents the possibility for each class, is regarded as the input feature vector for support vector machine (SVM) classifier. Extensive experiments on BU‐3DFE database with six facial expressions and seven poses demonstrate the effectiveness of different types of weighting strategies and the influence of different features. Comparison with other state‐of‐art methods illustrates that the proposed method not only improves the recognition rates of the frontal face but also has better robustness for non‐frontal facial expressions. © 2015 Institute of Electrical Engineers of Japan. Published by John Wiley & Sons, Inc.

AAM Based Facial Feature Tracking with Kinect

Abstract Facial features tracking is widely used in face recognition, gesture, expression analysis, etc. AAM (Active Appearance Model) is one of the powerful methods for objects feature localization. Nevertheless, AAM still suffers from a few drawbacks, such as the view angle change problem. We present a method to solve it by using the depth data acquired from Kinect. We use the depth data to get the head pose information and RGB data to match the AAM result. We establish an approximate facial 3D gird model and then initialize the subsequent frames with this model and head pose information. To avoid the local extremum, we divide the model into several parts by the poses and match the facial features with the closest model. The experimental results show improvement of AAM performance when rotating the head.

New interface for musical instruments using lip reading

As smart audio-visual multimedia devices are developed for various applications, there has been a growing interest in effective human–computer interaction (HCI) interfaces for specific environments. There have also been great efforts to implement HCI interfaces into musical instruments, in which it would be possible to take intuitions, comfort and expressiveness into the musical instruments. However, most of the traditional HCI interfaces are not applicable because both hands are likely to be occupied while playing a musical instrument. In this environment, a lip reading method can be used. A lip reading method is a HCI method that analyses lip motion to recognise spoken words. In this study, a lip reading method is proposed with its application. As a specific example of the interface for musical instruments, the guitar effector application is presented. The proposed lip reading method uses a constrained local model instead of the conventional active appearance model for effective facial feature tracking. The proposed method also uses a dynamic time warping-based classifier for word recognition which is effective for simple real-time implementation of lip reading. The proposed lip reading method shows 85.0% word recognition accuracy on OuluVS database and is effectively applied to the proposed guitar effector application.

Multiple camera in car audio–visual speech recognition using phonetic and visemic information

This paper presents a phonetic and visemic information-based audio–visual speech recognizer (AVSR). Active appearance model (AAM) is used to extract the visual features as it finely represents the shape and appearance information extracted from jaw and lip region. Consideration of visual features along with traditional acoustic feature has been found to be promising in the complex auditory environment. However, most of the existing AVSR systems rarely faced the visual domain problems. In this work, a real world multiple camera corpus audio visual in car (AVICAR) is used for the speech recognition experiment. Texas Instruments and Massachusetts Institute of Technology (TIMIT) corpus sentence portion is used to study the performance of bimodal audio–visual speech recognizer. To consider “Mc-Guruk” effect, acoustic and visual models are trained according to phonetic and visemic information, respectively. Phonetic–visemic AVSR system shows significant improvement over phonetic AVSR system.

Hybrid facial image feature extraction and recognition for non-invasive chronic fatigue syndrome diagnosis

Due to an absence of reliable biochemical markers, the diagnosis of chronic fatigue syndrome (CFS) mainly relies on the clinical symptoms, and the experience and skill of the doctors currently. To improve objectivity and reduce work intensity, a hybrid facial feature is proposed. First, several kinds of appearance features are identified in different facial regions according to clinical observations of traditional Chinese medicine experts, including vertical striped wrinkles on the forehead, puffiness of the lower eyelid, the skin colour of the cheeks, nose and lips, and the shape of the mouth corner. Afterwards, such features are extracted and systematically combined to form a hybrid feature. We divide the face into several regions based on twelve active appearance model (AAM) feature points, and ten straight lines across them. Then, Gabor wavelet filtering, CIELab color components, threshold-based segmentation and curve fitting are applied to extract features, and Gabor features are reduced by a manifold preserving projection method. Finally, an AdaBoost based score level fusion of multi-modal features is performed after classification of each feature. Despite that the subjects involved in this trial are exclusively Chinese, the method achieves an average accuracy of 89.04% on the training set and 88.32% on the testing set based on the K-fold cross-validation. In addition, the method also possesses desirable sensitivity and specificity on CFS prediction.

Facial expression recognition from image sequences using twofold random forest classifier

We propose a novel framework for facial expressions analysis by recognizing AUs from image sequences using twofold random forest classifier in this paper. The measurement of facial motion is through tracking of Active Appearance Model (AAM) facial feature points using Lucas–Kanade (LK) optical flow tracker by estimating the displacements of the feature points. The displacement vectors between the neutral expression frame and the peak expression frame are used as motion features of facial expression. They will then be transformed to the first level random forest to determine the Action Units (AUs) of the corresponding expression sequences. Finally, the detected AUs are inputed into the second level random forest for facial expressions classification. The experiments on Extended Cohn–Kanade(CK+) database demonstrate that the proposed method can achieve higher performance than several other approaches on both AUs and facial expression recognition. We attain an average recognition rate of AUs and facial expression of 100% and 96.38% respectively.

Feature-Based Lucas–Kanade and Active Appearance Models

Lucas-Kanade and active appearance models are among the most commonly used methods for image alignment and facial fitting, respectively. They both utilize nonlinear gradient descent, which is usually applied on intensity values. In this paper, we propose the employment of highly descriptive, densely sampled image features for both problems. We show that the strategy of warping the multichannel dense feature image at each iteration is more beneficial than extracting features after warping the intensity image at each iteration. Motivated by this observation, we demonstrate robust and accurate alignment and fitting performance using a variety of powerful feature descriptors. Especially with the employment of histograms of oriented gradient and scale-invariant feature transform features, our method significantly outperforms the current state-of-the-art results on in-the-wild databases.

Recognizing multiple emotion from ambiguous facial expressions on mobile platforms

Extracting and understanding of emotion is of high importance for the interaction between human and machine communication systems. The most expressive way to display the human's emotion is through facial expression analysis. This paper proposes a multiple emotion recognition system that can recognize combinations of up to a maximum of three different emotions using an active appearance model (AAM), the proposed classification standard, and a k-nearest neighbor (k-NN) classifier in mobile environments. AAM can take the expression of variations that are calculated by the proposed classification standard according to changes in human expressions in real time. The proposed k-NN can classify basic emotions (normal, happy, sad, angry, surprise) as well as more ambiguous emotions by combining the basic emotions in real time, and each recognized emotion that can be subdivided has strength. Whereas most previous methods of emotion recognition recognize various kind of a single emotion, this paper recognizes various emotions with a combination of the five basic emotions. To be easily understood, the recognized result is presented in three ways on a mobile camera screen. The result of the experiment was an average 85 % recognition rate and a 40 % performance showed optimized emotions. The implemented system can be represented by one of the example for augmented reality on displaying combination of real face video and virtual animation with user's avatar.

Face model fitting on video sequences based on incremental virtual active appearance model

It has been proved that the performance of a person-specific active appearance model (AAM) built to model the appearance variation of a single person across pose, illumination, and expression is substantially better than the performance of a generic AAM built to model the appearance variation of many faces. However, it is not practical to build a personal AAM before tracking an unseen subject. A virtual person-specific AAM is proposed to tackle the problem. The AAM is constructed from a set of virtual personal images with different poses and expressions which are synthesized from the annotated first frame via regressions. To preserve personal facial details on the virtual images, a poison fusion strategy is designed and applied to the virtual facial images generated via bilinear kernel ridge regression. Furthermore, the AAM subspace is sequentially updated during tracking based on sequential Karhunen---Loeve algorithm, which helps the AAM adaptive to the facial context variation. Experiments show the proposed virtual personal AAM is robust to facial context changes during tracking, and outperforms other state-of-the-art AAM on facial feature tracking accuracy and computation cost.