Facial expression recognition using iterative fusion of MO-HOG and deep features

Abstract

Facial expression recognition is a challenging problem in computer vision. Due to the limited feature extraction capability of a single feature descriptor, this paper proposes a facial expression recognition method that iteratively fuses classifiers based on multi-orientation gradient calculated HOG (MO-HOG) features and deep-learned features. Diagonal orientation gradient calculated HOG (D-HOG) is a complementary part to the histogram of oriented gradient (HOG), which is proposed to obtain the diagonal gradient information and combines HOG to form a novel feature descriptor MO-HOG. Our method extracts MO-HOG features from whole facial images and expression-rich local facial images. Meanwhile, deep-learned features are not reliable enough on small databases but contain high-level semantic information, so the deep network is designed to extract effective deep-learned features. In addition, a classifier fusion method based on an optimization algorithm is proposed, and the best-fused classifier is obtained through iteration. The experiments are evaluated on the public databases (CK+ and JAFFE). The proposed method shows the effectiveness of facial expression recognition and outperforms the state-of-the-art methods. The recognition accuracy is 97.70% on the CK+ database and 97.64% on the JAFFE database.