Facial appearance and texture feature-based robust facial expression recognition framework for sentiment knowledge discovery

Abstract

Facial sentiment analysis has been an enthusiastic research area for the last two decades. A fair amount of work has been done by researchers in this field due to its utility in numerous applications such as facial expression driven knowledge discovery. However, developing an accurate and efficient facial expression recognition system is still a challenging problem. Although many efficient recognition systems have been introduced in the past, the recognition rate is not satisfactory in general due to inherent limitations including light, pose variations, noise, and occlusion. In this paper, a hybrid approach of facial expression based sentiment analysis has been presented combining local and global features. Feature extraction is performed fusing the histogram of oriented gradients (HOG) descriptor with the uniform local ternary pattern (U-LTP) descriptor. These features are extracted from the entire face image rather than from individual components of faces like eyes, nose, and mouth. The most suitable set of HOG parameters are selected after analyzing them experimentally along with the ULTP descriptor, boosting performance of the proposed technique over face images containing noise and occlusions. Face sentiments are analyzed classifying them into seven universal emotional expressions: Happy, Angry, Fear, Disgust, Sad, Surprise, and Neutral. Extracted features via HOG and ULTP are fused into a single feature vector and this feature vector is fed into a Multi-class Support Vector Machine classifier for emotion classification. Three types of experiments are conducted over three public facial image databases including JAFFE, MMI, and CK+ to evaluate the recognition rate of the proposed technique during experimental evaluation; recognition accuracy in percent, i.e., 95.71, 98.20, and 99.68 are achieved for JAFFE, MMI, and CK+, respectively.