Video-based Facial Expression Recognition Research Articles

A Local Feature-Based Robust Approach for Facial Expression Recognition from Depth Video

Facial expression recognition (FER) plays a very significant role in computer vision, pattern recognition, and image processing applications such as human computer interaction as it provides sufficient information about emotions of people. For video-based facial expression recognition, depth cameras can be better candidates over RGB cameras as a person’s face cannot be easily recognized from distance-based depth videos hence depth cameras also resolve some privacy issues that can arise using RGB faces. A good FER system is very much reliant on the extraction of robust features as well as recognition engine. In this work, an efficient novel approach is proposed to recognize some facial expressions from time-sequential depth videos. First of all, efficient Local Binary Pattern (LBP) features are obtained from the time-sequential depth faces that are further classified by Generalized Discriminant Analysis (GDA) to make the features more robust and finally, the LBP-GDA features are fed into Hidden Markov Models (HMMs) to train and recognize different facial expressions successfully. The depth information-based proposed facial expression recognition approach is compared to the conventional approaches such as Principal Component Analysis (PCA), Independent Component Analysis (ICA), and Linear Discriminant Analysis (LDA) where the proposed one outperforms others by obtaining better recognition rates.

Video-based facial expression recognition using learned spatiotemporal pyramid sparse coding features

Recently, hand-designed local descriptors like spatiotemporal Gabor filters and VLBP have been successfully applied in video-based facial expression recognition. One major drawback of these methods is that they are hard to generalize to different problems. In this paper, we propose a new video-based facial expression recognition method by automatically learning features from video data. Specifically, we use sparse coding algorithm to learn spatiotemporal features from unlabeled facial expression videos. For modeling spatiotemporal layout information embedded in facial expressions to improve recognition performance, we extend the idea of spatial pyramid matching (SPM) into video case, and perform spatiotemporal pyramid feature pooling following sparse coding feature extraction. Experimental results on widely used Cohn–Kanade database show that the classification performance can be improved effectively by considering spatiotemporal layout of facial expressions, and our method outperforms popular methods using hand-designed features.

Video‐based facial expression recognition by removing the style variations

This study examines the performance of person-independent facial expression recognition improved by adapting the system to a given person. The proposed method transfers the style of particular subjects to the semi-style-free space. There is no need to change the person-independent classifier in order to improve the performance. The style transfer mapping (STM) has been proposed in image-based classification. The challenges of employing this technique in video-based facial expression recognition are: estimating STM from image sequences of each subject (adaptation data) and projecting new sequential data of each subject in semi-style-free space. A mixture of ‘binary support vector machines’ and ‘hidden Markov models’ were employed to overcome these challenges. Moreover, virtual samples generated by using the person's neutral samples were used to estimate STM. Experimental results on the CK+ database confirm the efficiency of the proposed method in recognition rate improvement.

A depth video-based facial expression recognition system utilizing generalized local directional deviation-based binary pattern feature discriminant analysis

Facial expression recognition from video data is considered to be a very challenging task in the research areas of computer vision, image processing, and pattern recognition. A novel approach is pr...

An Automatic Framework for Textured 3D Video-Based Facial Expression Recognition

Most of the existing research on 3D facial expression recognition has been done using static 3D meshes. 3D videos of a face are believed to contain more information in terms of the facial dynamics which are very critical for expression recognition. This paper presents a fully automatic framework which exploits the dynamics of textured 3D videos for recognition of six discrete facial expressions. Local video-patches of variable lengths are extracted from numerous locations of the training videos and represented as points on the Grassmannian manifold. An efficient graph-based spectral clustering algorithm is used to separately cluster these points for every expression class. Using a valid Grassmannian kernel function, the resulting cluster centers are embedded into a Reproducing Kernel Hilbert Space (RKHS) where six binary SVM models are learnt. Given a query video, we extract video-patches from it, represent them as points on the manifold and match these points with the learnt SVM models followed by a voting based strategy to decide about the class of the query video. The proposed framework is also implemented in parallel on 2D videos and a score level fusion of 2D & 3D videos is performed for performance improvement of the system. The experimental results on BU4DFE data set show that the system achieves a very high classification accuracy for facial expression recognition from 3D videos.

BAUM-2: a multilingual audio-visual affective face database

Access to audio-visual databases, which contain enough variety and are richly annotated is essential to assess the performance of algorithms in affective computing applications, which require emotion recognition from face and/or speech data. Most databases available today have been recorded under tightly controlled environments, are mostly acted and do not contain speech data. We first present a semi-automatic method that can extract audio-visual facial video clips from movies and TV programs in any language. The method is based on automatic detection and tracking of faces in a movie until the face is occluded or a scene cut occurs. We also created a video-based database, named as BAUM-2, which consists of annotated audio-visual facial clips in several languages. The collected clips simulate real-world conditions by containing various head poses, illumination conditions, accessories, temporary occlusions and subjects with a wide range of ages. The proposed semi-automatic affective clip extraction method can easily be used to extend the database to contain clips in other languages. We also created an image based facial expression database from the peak frames of the video clips, which is named as BAUM-2i. Baseline image and video-based facial expression recognition results using state-of-the art features and classifiers indicate that facial expression recognition under tough and close-to-natural conditions is quite challenging.

Facial Expression Recognition Using 3D Convolutional Neural Network

This paper is concerned with video-based facial expression recognition frequently used in conjunction with HRI (Human-Robot Interaction) that can naturally interact between human and robot. For this purpose, we design a 3D-CNN(3D Convolutional Neural Networks) by augmenting dimensionality reduction methods such as PCA(Principal Component Analysis) and TMPCA(Tensor-based Multilinear Principal Component Analysis) to recognize simultaneously the successive frames with facial expression images obtained through video camera. The 3D-CNN can achieve some degree of shift and deformation invariance using local receptive fields and spatial subsampling through dimensionality reduction of redundant CNN’s output. The experimental results on video-based facial expression database reveal that the presented method shows a good performance in comparison to the conventional methods such as PCA and TMPCA.

A depth video-based facial expression recognition system using radon transform, generalized discriminant analysis, and hidden Markov model

In this paper, a depth camera-based novel method is proposed to recognize several facial expressions from depth video. At first, Radon Transformation (RT) is done to extract features from the time-sequential depth faces that are further improved by Generalized Discriminant Analysis (GDA) to generate more robust features and then, Hidden Markov Models (HMMs) are applied to train and recognize different facial expressions successfully. Performance of the proposed facial expression recognition shows the superiority over conventional RGB camera-based approaches.

Understanding Discrete Facial Expressions in Video Using an Emotion Avatar Image

Existing video-based facial expression recognition techniques analyze the geometry-based and appearance-based information in every frame as well as explore the temporal relation among frames. On the contrary, we present a new image-based representation and an associated reference image called the emotion avatar image (EAI), and the avatar reference, respectively. This representation leverages the out-of-plane head rotation. It is not only robust to outliers but also provides a method to aggregate dynamic information from expressions with various lengths. The approach to facial expression analysis consists of the following steps: 1) face detection; 2) face registration of video frames with the avatar reference to form the EAI representation; 3) computation of features from EAIs using both local binary patterns and local phase quantization; and 4) the classification of the feature as one of the emotion type by using a linear support vector machine classifier. Our system is tested on the Facial Expression Recognition and Analysis Challenge (FERA2011) data, i.e., the Geneva Multimodal Emotion Portrayal-Facial Expression Recognition and Analysis Challenge (GEMEP-FERA) data set. The experimental results demonstrate that the information captured in an EAI for a facial expression is a very strong cue for emotion inference. Moreover, our method suppresses the person-specific information for emotion and performs well on unseen data.

Image Ratio Features for Facial Expression Recognition Application

Video-based facial expression recognition is a challenging problem in computer vision and human-computer interaction. To target this problem, texture features have been extracted and widely used, because they can capture image intensity changes raised by skin deformation. However, existing texture features encounter problems with albedo and lighting variations. To solve both problems, we propose a new texture feature called image ratio features. Compared with previously proposed texture features, e.g., high gradient component features, image ratio features are more robust to albedo and lighting variations. In addition, to further improve facial expression recognition accuracy based on image ratio features, we combine image ratio features with facial animation parameters (FAPs), which describe the geometric motions of facial feature points. The performance evaluation is based on the Carnegie Mellon University Cohn-Kanade database, our own database, and the Japanese Female Facial Expression database. Experimental results show that the proposed image ratio feature is more robust to albedo and lighting variations, and the combination of image ratio features and FAPs outperforms each feature alone. In addition, we study asymmetric facial expressions based on our own facial expression database and demonstrate the superior performance of our combined expression recognition system.