A facial expression recognition system using robust face features from depth videos and deep learning

Abstract

This work proposes a depth camera-based robust facial expression recognition (FER) system that can be adopted for better human machine interaction. Although video-based facial expression analysis has been focused on by many researchers, there are still various problems to be solved in this regard such as noise due to illumination variations over time. Depth video data in the helps to make an FER system person-independent as pixel values in depth images are distributed based on distances from a depth camera. Besides, depth images should resolve some privacy issues as real identity of a user can be hidden. The accuracy of an FER system is much dependent on the extraction of robust features. Here, we propose a novel method to extract salient features from depth faces that are further combined with deep learning for efficient training and recognition. Eight directional strengths are obtained for each pixel in a depth image where signs of some top strengths are arranged to represent unique as well as robust face features, which can be denoted as Modified Local Directional Patterns (MLDP). The MLDP features are further processed by Generalized Discriminant Analysis (GDA) for better face feature extraction. GDA is an efficient tool that helps distinguishing MLDP features of different facial expressions by clustering the features from the same expression as close as possible and separating the features from different expressions as much as possible in a non-linear space. Then, MLDP-GDA features are applied with Deep Belief Network (DBN) for training different facial expressions. Finally, the trained DBN is used to recognize facial expressions in a depth video for testing. The proposed approach was compared with other traditional approaches in a standalone system where the proposed one showed its superiority by achieving mean recognition rate of 96.25% where the other approaches could make 91.67% at the best. The deep learning-based training and recognition of the facial expression features can also be undertaken with cloud computing to support many users and make the system faster than a standalone system.