Face Expression Recognition in Video Using Hybrid Feature Extractor and CNN-LSTM

Abstract

AbstractFace expression recognition in videos is a challenging task as it involves extracting and combining high level temporal and spatial features from the video sequences to automatically identify human expression categories. In this paper, we propose a Hybrid feature extractor, analyze various deep learning classifiers and their performance. Addressing illumination invariance and scale invariance, this method initially pre-processes the input video to remove noise and uses Viola jones algorithm to detect face region; then the high significant features, such as temporal features, spatial domain features, and the frequency domain features are extracted. Further, dimensionality issues are handled effectively using the Principle Component Analysis (PCA). Finally, the fused features are fed to train CNN-LSTM, a hybrid deep learning classifier for identifying face expressions. Experiment was conducted on SAVEE, RAVDESS and AFEW Datasets. The proposed method obtained an overall accuracy, precision, and recall of 90.5%, 92.39%, 88.18%, respectively.KeywordsFace expression recognitionViola jones algorithmPrinciple component analysisLocal binary patternAnd hybrid deep learning classifier