3D Facial Expression Research Articles

Facial expression recognition based on spatio-temporal interest points for depth sequences

Of the numerous facial expression recognition methods previously proposed, most are based on texture frames or sequences. Recently, the development of depth sensors has raised new possibilities of dealing with 3D data. The proposed method extracts histograms of oriented gradient (HOG) and optical flow (HOF) for STIPs directly from depth sequences rather than involving registration/deformation techniques to find correspondence in 3D scan data. Mutual information score (MIS) and weighted matching score (WMS) are, respectively, calculated on the basis of naïve-Bayes mutual information maximization and constrained matching pairs. Finally, the MIS and WMS results are concatenated into feature vectors which are then fed into a support vector machine for facial expression classification. The proposed method is applied to the public BU-4DFE (Binghamton University 4D Facial Expression) database for six different facial expressions: anger, disgust, fear, happy, sad and surprise. Experimental results confirm that the proposed method is simple but effective.

Realtime 3D eye gaze animation using a single RGB camera

This paper presents the first realtime 3D eye gaze capture method that simultaneously captures the coordinated movement of 3D eye gaze, head poses and facial expression deformation using a single RGB camera. Our key idea is to complement a realtime 3D facial performance capture system with an efficient 3D eye gaze tracker. We start the process by automatically detecting important 2D facial features for each frame. The detected facial features are then used to reconstruct 3D head poses and large-scale facial deformation using multi-linear expression deformation models. Next, we introduce a novel user-independent classification method for extracting iris and pupil pixels in each frame. We formulate the 3D eye gaze tracker in the Maximum A Posterior (MAP) framework, which sequentially infers the most probable state of 3D eye gaze at each frame. The eye gaze tracker could fail when eye blinking occurs. We further introduce an efficient eye close detector to improve the robustness and accuracy of the eye gaze tracker. We have tested our system on both live video streams and the Internet videos, demonstrating its accuracy and robustness under a variety of uncontrolled lighting conditions and overcoming significant differences of races, genders, shapes, poses and expressions across individuals.

A spatio-temporal wavelet-based descriptor for dynamic 3D facial expression retrieval and recognition

Human emotions are often expressed by facial expressions and are generated by facial muscle movements. In recent years, the analysis of facial expressions has emerged as an active research area due to its various applications such as human---computer interaction, human behavior understanding, biometrics, emotion recognition, computer graphics, driver fatigue detection, and psychology. A novel analysis of dynamic 3D facial expressions using the positional information of automatically detected facial landmarks and the wavelet transformation is presented, which results in the proposed spatio-temporal descriptor. This descriptor is employed within the current paper in a retrieval scheme for dynamic 3D facial expression datasets and is thoroughly evaluated. Experiments have been conducted using the six prototypical expressions of the publicly available BU-4DFE dataset as well as the eight expressions included in the newly released publicly available BP4D-Spontaneous dataset. The obtained retrieval results outperform the retrieval results of the state-of-the-art methodologies. Furthermore, the retrieval results are exploited to achieve unsupervised dynamic 3D facial expression recognition. The aforementioned unsupervised procedure achieves better recognition accuracy compared to supervised dynamic 3D facial expression recognition state-of-the-art techniques.

Exploring generalized shape analysis by topological representations

One of the most common properties of various data in pattern recognition is the shape, and the shape matters. However, the shape can appear with uncertain appearances, e.g., the shapes of a person in different poses. We realize that the most fundamental feature of any shape is the number of connected components, the number of holes and its higher dimensional counterparts. These are what we call topological invariants. This is the place where topology comes into play for pattern recognition. Persistent homology, one of the most powerful tools in algebraic topology, is proposed to compute these topological invariants at different resolutions. The proposed method, by firstly transferring the given data into a topological graph representation, i.e., the simplicial complex, can assemble discrete points into a global structure. Then by integrating with multiple filtrations and metric learning, both the global structure and different local parts can be taken into account at the same time. We test the proposed method in 2D shape classification, 2.5D gait identification and 3D facial expression recognition. Experimental results demonstrate the effectiveness of this generalized shape analysis method and show its potentials in different applications. Moreover, we provide a new insight for the generalized shape analysis.

Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-Related Applications.

Facial expressions are an important way through which humans interact socially. Building a system capable of automatically recognizing facial expressions from images and video has been an intense field of study in recent years. Interpreting such expressions remains challenging and much research is needed about the way they relate to human affect. This paper presents a general overview of automatic RGB, 3D, thermal and multimodal facial expression analysis. We define a new taxonomy for the field, encompassing all steps from face detection to facial expression recognition, and describe and classify the state of the art methods accordingly. We also present the important datasets and the bench-marking of most influential methods. We conclude with a general discussion about trends, important questions and future lines of research.

Creative tools for producing realistic 3D facial expressions and animation

Creative exploration of realistic 3D facial animation is a popular but very challenging task due to the high level knowledge and skills required. This forms a barrier for creative individuals who have limited technical skills but wish to explore their creativity in this area. This paper proposes a new technique that facilitates users' creative exploration by hiding the technical complexities of producing facial expressions and animation. The proposed technique draws on research from psychology, anatomy and employs Autodesk Maya as a use case by developing a creative tool, which extends Maya's Blend Shape Editor. User testing revealed that novice users in the creative media, employing the proposed tool can produce rich and realistic facial expressions that portray new interesting emotions. It reduced production time by 25% when compared to Maya and by 40% when compared to 3DS Max equivalent tools.

Recognizing Surgically Altered Face Images and 3D Facial Expression Recognition

Altering Facial appearances using surgical procedures are common now days. But it raised challenges for face recognition algorithms. Plastic surgery introduces non linear variations. Because of these variations it is difficult to be modeled by the existing face recognition system. Here presents a multi objective evolutionary granular algorithm. It operates on several granules extracted from a face images at multiple level of granularity. This granular information is unified in an evolutionary manner using multi objective genetic approach. Then identify the facial expression from the face images. For that 3D facial shapes are considering here. A novel automatic feature selection method is proposed based on maximizing the average relative entropy of marginalized class-conditional feature distributions and apply it to a complete pool of candidate features composed of normalized Euclidian distances between 83 facial feature points in the 3D space. A regularized multi-class AdaBoost classification algorithm is used here to get the highest average recognition rate.

3D HMM-based Facial Expression Recognition using Histogram of Oriented Optical Flow

In this paper, we propose a 3D HMM (Three-dimensional Hidden Markov Models) approach to recognizing human facial expressions and associated emotions. Human emotion is usually classified by psychologists into six categories: Happiness, Sadness, Anger, Fear, Disgust and Surprise. Further, psychologists categorize facial movements based on the muscles that produce those movements using a Facial Action Coding System (FACS). We look beyond pure muscle movements and investigate facial features – brow, mouth, nose, eye height and facial shape – as a means of determining associated emotions. Histogram of Optical Flow is used as the descriptor for extracting and describing the key features, while training and testing are performed on 3D Hidden Markov Models. Experiments on datasets show our approach is promising and robust.

3D Character Animation: A Brief Review

The Augmented Reality (AR) and Virtual Reality (VR) tech -nologies provide many contents that we can barely experience in reality. Further, the developments in performance of comput-er and GPU are based on putting virtual environment into our reality efficiently. The core technologies in developing contents of AR and VR are 3D character modeling and 3D character ani -mation. These are used in various areas such as 3D film, game industry, Physics, Medical simulation and Educational contents. In this review, we discuss the method of animating contents in the virtual reality. There are two topics of 3D character animation, namely, Fa-cial animation and Body animation. Facial animation implies the representation of natural facial expressions. There are two methods of producing this type of animation. One is the Geom-etry-based approach which is based on control point in 3D char-acter face mesh and the other is Blend shape model-based ap-proach which utilizes 3D facial expression template blending. The tree-structured skeleton is usually used for animating body motion. In addition, to achieve more realistic character move-ments, not only physics but also non-rigid deformation of skin or clothes are adopted. In this paper, we discuss Geometry- and Blend shape model-based of facial animation as well as Skeleton- and Physics-based of body animation.

2D facial expression recognition via 3D reconstruction and feature fusion

In this paper, a novel feature extraction method is proposed for facial expression recognition by extracting the feature from facial depth and 3D mesh alongside texture. Accordingly, the 3D Facial Expression Generic Elastic Model (3D FE-GEM) method is used to reconstruct an expression-invariant 3D model from the human face. Then, the texture, depth and mesh are extracted from the reconstructed face model. Afterwards, the Local Binary Pattern (LBP), proposed 3D High-Low Local Binary Pattern (3DH-LLBP) and Local Normal Binary Patterns (LNBPs) are applied to texture, depth and mesh of the face, respectively, to extract the feature from 2D images. Finally, the final feature vectors are generated through feature fusion and are classified by the Support Vector Machine (SVM). Convincing results are acquired for facial expression recognition on the CK+, CK, JAFFE and Bosphorus image databases compared to several state-of-the-art methods.

Rotation update on manifold in probabilistic NRSFM for robust 3D face modeling

This paper focuses on recovering the 3D structure and motion of human faces from a sequence of 2D images. Based on a probabilistic model, we extensively studied the rotation constraints of the problem. Instead of imposing numerical optimizations, the inherent geometric properties of the rotation matrices are taken into account. The conventional Newton’s method for optimization problems was generalized on the rotation manifold, which ultimately resolves the constraints into unconstrained optimization on the manifold. Furthermore, we also extended the algorithm to model within-individual and between-individual shape variances separately. Evaluation results give evidence to the improvement over the state-of-the-art algorithms on the Mocap-Face dataset with additive noise, as well as on the Binghamton University A 3D Facial Expression (BU-3DFE) dataset. Robustness in handling noisy data and modeling multiple subjects shows the capability of our system to deal with real-world image tracks.

Video-audio driven real-time facial animation

We present a real-time facial tracking and animation system based on a Kinect sensor with video and audio input. Our method requires no user-specific training and is robust to occlusions, large head rotations, and background noise. Given the color, depth and speech audio frames captured from an actor, our system first reconstructs 3D facial expressions and 3D mouth shapes from color and depth input with a multi-linear model. Concurrently a speaker-independent DNN acoustic model is applied to extract phoneme state posterior probabilities (PSPP) from the audio frames. After that, a lip motion regressor refines the 3D mouth shape based on both PSPP and expression weights of the 3D mouth shapes, as well as their confidences. Finally, the refined 3D mouth shape is combined with other parts of the 3D face to generate the final result. The whole process is fully automatic and executed in real time. The key component of our system is a data-driven regresor for modeling the correlation between speech data and mouth shapes. Based on a precaptured database of accurate 3D mouth shapes and associated speech audio from one speaker, the regressor jointly uses the input speech and visual features to refine the mouth shape of a new actor. We also present an improved DNN acoustic model. It not only preserves accuracy but also achieves real-time performance. Our method efficiently fuses visual and acoustic information for 3D facial performance capture. It generates more accurate 3D mouth motions than other approaches that are based on audio or video input only. It also supports video or audio only input for real-time facial animation. We evaluate the performance of our system with speech and facial expressions captured from different actors. Results demonstrate the efficiency and robustness of our method.

An effective methodology for dynamic 3D facial expression retrieval

The problem of facial expression recognition in dynamic sequences of 3D face scans has received a significant amount of attention in the recent past whereas the problem of retrieval in this type of data has not. A novel retrieval methodology for such data is introduced in this paper. The proposed methodology automatically detects specific facial landmarks and uses them to create a descriptor. This descriptor is the concatenation of three sub-descriptors which capture topological as well as geometric information of the 3D face scans. The motivation behind the proposed hybrid facial expression descriptor is the fact that some facial expressions, like happiness and surprise, are characterized by obvious changes in the mouth topology while others, like anger, fear and sadness, produce geometric but no significant topological changes. The proposed retrieval scheme exploits the Dynamic Time Warping technique in order to compare descriptors corresponding to different 3D facial sequences. A detailed evaluation of the introduced retrieval scheme is presented showing that it outperforms previous state-of-the-art retrieval schemes. Experiments have been conducted using the six prototypical expressions of the standard dataset BU-4DFE and the eight prototypical expressions of the recently available dataset BP4D-Spontaneous. Finally, a majority voting scheme based on the retrieval results is used to achieve unsupervised dynamic 3D facial expression recognition. The achieved classification accuracy is comparable to the state-of-the-art supervised dynamic 3D facial expression recognition techniques.

Análisis y Comparación del Descriptor Cone Curvature frente al Reconocimiento de Expresiones Faciales

Se presenta el resultado de analizar el comportamiento del descriptor de forma Cone Curvature (CC) en la tarea de reconocimiento de expresiones faciales en imágenes 3D. El descriptor CC es una representación del modelo 3D que se calcula a partir de un conjunto de ondas de modelado para cada vértice de una malla poligonal. Se empleó la base de datos de rostros 3D (BU-3DFE), la cual contiene imágenes con 6 expresiones faciales. Con el uso del descriptor CC, las expresiones fueron reconocidas en un porcentaje promedio del 76.67% con una red neuronal, y del 78.88% con un clasificador bayesiano. Al realizar una combinación del descriptor CC con otros descriptores como DESIRE y Spherical Spin Image, se logr´o un porcentaje promedio de reconocimiento de gestos del 90.27% y del 97.2 %, usando los mismos clasificadores mencionados previamente.

3D dynamic facial expression recognition using low-resolution videos

In this paper, we focus on the problem of 3D dynamic (4D) facial expression recognition. While traditional methods rely on building deformation models on high-resolution 3D meshes, our approach works directly on low-resolution RGB-D sequences; this feature allows us to apply our algorithm to videos retrieved by widespread and standard low-resolution RGB-D sensors, such as Kinect. After preprocessing both RGB and depth image sequences, sparse features are learned from spatio-temporal local cuboids. Conditional Random Fields classifier is then employed for training and classification. The proposed system is fully-automatic and achieves superior results on three low-resolution datasets built from the 4D facial expression recognition dataset – BU-4DFE. Extensive evaluations of our approach and comparisons with state-of-the-art methods are presented.

A robust spatio-temporal scheme for dynamic 3D facial expression retrieval

The problem of facial expression recognition in dynamic sequences of 3D face scans has received a significant amount of attention in the recent past whereas the problem of retrieval in this type of data has not. A novel retrieval scheme for such data is introduced in this paper. It is the first spatio-temporal retrieval scheme ever used for retrieval in dynamic sequences of 3D face scans. The proposed scheme automatically detects specific facial landmarks and uses them to create a spatio-temporal descriptor. At first, geometric as well as topological information of the 3D face scans is captured by using the detected landmarks. In the sequel, the aforementioned spatial information is filtered by using wavelet transformation, resulting to our final spatio-temporal descriptor. Our descriptor is invariant to the number of the 3D face scans of a facial expression sequence. The proposed retrieval scheme exploits the Square of Euclidean distance in order to compare descriptors corresponding to different 3D facial sequences. A detailed evaluation of the introduced retrieval scheme is presented showing that it outperforms previous state-of-the-art retrieval schemes. Experiments have been conducted using the six prototypical expressions of the standard data set $$\textit{BU}-4\textit{DFE}$$BU-4DFE. Finally, a majority voting methodology based on the retrieval results is used to achieve unsupervised dynamic 3D facial expression recognition. The achieved classification accuracy outperforms the state-of-the-art supervised dynamic 3D facial expression recognition techniques.

Real-World and Rapid Face Recognition Toward Pose and Expression Variations via Feature Library Matrix

In this paper, a novel method for face recognition under pose and expression variations is proposed from only a single image in the gallery. A 3D probabilistic facial expression recognition generic elastic model is proposed to reconstruct a 3D model from real-world human face using only a single 2D frontal image with/without facial expressions. Then, a feature library matrix (FLM) is generated for each subject in the gallery from all face poses by rotating the 3D reconstructed models and extracting features in the rotated face pose. Therefore, each FLM is subsequently rendered for each subject in the gallery based on triplet angles of face poses. In addition, before matching the FLM, an initial estimate of triplet angles is obtained from the face pose in probe images using an automatic head pose estimation approach. Then, an array of the FLM is selected for each subject based on the estimated triplet angles. Finally, the selected arrays from FLMs are compared with extracted features from the probe image by iterative scoring classification using the support vector machine. Convincing results are acquired to handle pose and expression changes on the Bosphorus, Face Recognition Technology (FERET), Carnegie Mellon University-Pose, Illumination, and Expression (CMU-PIE), and Labeled Faces in the Wild (LFW) face databases compared with several state-of-the-art methods in pose-invariant face recognition. The proposed method not only demonstrates an excellent performance by obtaining high accuracy on all four databases but also outperforms other approaches realistically.

Three Dimensional Binary Edge Feature Representation for Pain Expression Analysis.

Automatic pain expression recognition is a challenging task for pain assessment and diagnosis. Conventional 2D-based approaches to automatic pain detection lack robustness to the moderate to large head pose variation and changes in illumination that are common in real-world settings and with few exceptions omit potentially informative temporal information. In this paper, we propose an innovative 3D binary edge feature (3D-BE) to represent high-resolution 3D dynamic facial expression. To exploit temporal information, we apply a latent-dynamic conditional random field approach with the 3D-BE. The resulting pain expression detection system proves that 3D-BE represents the pain facial features well, and illustrates the potential of noncontact pain detection from 3D facial expression data.

Using Kinect for real-time emotion recognition via facial expressions

Emotion recognition via facial expressions (ERFE) has attracted a great deal of interest with recent advances in artificial intelligence and pattern recognition. Most studies are based on 2D images, and their performance is usually computationally expensive. In this paper, we propose a real-time emotion recognition approach based on both 2D and 3D facial expression features captured by Kinect sensors. To capture the deformation of the 3D mesh during facial expression, we combine the features of animation units (AUs) and feature point positions (FPPs) tracked by Kinect. A fusion algorithm based on improved emotional profiles (IEPs) and maximum confidence is proposed to recognize emotions with these real-time facial expression features. Experiments on both an emotion dataset and a real-time video show the superior performance of our method.

Multiple strategies to enhance automatic 3D facial expression recognition

The research on 3D facial expression recognition has attracted numbers of interests due to its superiority to 2D data and it has been greatly promoted in recent years. However, its performance needs to be further improved and its data structure needs to be further analyzed to keep its automation well as the mesh structure of 3D face models cannot be applied directly to algebraic operations. This paper addresses these problems with multiple strategies, so that 3D facial expression recognition can be automatically implemented and its performance is subsequently enhanced. Firstly, an image-like-structure is proposed to represent the 3D face models, so that algebraic operations can be directly applied to analyze 3D data. Based on this image-like-structure, the strategies of irregular division schemes and the entropy weighted blocks are employed to improve the recognition accuracy. The former aims to keep the integrity of local structure; the latter is employed to emphasize the contribution of different facial regions. Both of them can be separately or jointly, utilized to facial feature descriptors. With the remarkable experimental results based on LBP and LTP, we can conclude that these strategies are available to promote the performance of automatic 3D facial expression recognition, which draws a promising direction for automatic 3D facial expression recognition.