Spectral embedding based facial expression recognition with multiple features

Abstract

Many approaches to facial expression recognition utilize only one type of features at a time. It can be difficult for a single type of features to characterize in a best possible way the variations and complexity of realistic facial expressions. In this paper, we propose a spectral embedding based multi-view dimension reduction method to fuse multiple features for facial expression recognition. Facial expression features extracted from one type of expressions can be assumed to form a manifold embedded in a high dimensional feature space. We construct a neighborhood graph that encodes the structure of the manifold locally. A graph Laplacian matrix is constructed whose spectral decompositions reveal the low dimensional structure of the manifold. In order to obtain discriminative features for classification, we propose to build a neighborhood graph in a supervised manner by utilizing the label information of training data. As a result, multiple features are able to be transformed into a unified low dimensional feature space by combining the Laplacian matrix of each view with the multiview spectral embedding algorithm. A linearization method is utilized to map unseen data to the learned unified subspace. Experiments are conducted on a set of established real-world and benchmark datasets. The experimental results provide a strong support to the effectiveness of the proposed feature fusion framework on realistic facial expressions.