Abstract
Emotion recognition based on facial expressions is very important for effective interaction of humans with artificial intelligence (AI) systems such as social robots. On the other hand, in real environment, it is much harder to recognize facial micro-expressions (FMEs) than facial general-expressions having rich emotions. In this paper, we propose a two-dimensional (2D) landmark feature map for effectively recognizing such FMEs. The proposed 2D landmark feature map (LFM) is obtained by transforming conventional coordinate-based landmark information into 2D image information. LFM is designed to have an advantageous property independent of the intensity of facial expression change. Also, we propose an LFM-based emotion recognition method that is an integrated framework of convolutional neural network (CNN) and long short-term memory (LSTM). Experimental results show that the proposed method achieves about 71% and 74% in the well-known micro-expression datasets, i.e., SMIC and CASME II, respectively, which outperforms the conventional methods. The performance of the proposed method was also verified through experiments on composite micro-expression dataset, which consists of SMIC, CAMSE II and SAMM, and cross-dataset validation using SMIC and CAMSE II. In addition, we prove that the proposed method is independent of facial expression intensity through an experiment on CK+ dataset. Finally, we demonstrate that the proposed method is valid even for the MAHNOB-HCI and MEVIEW datasets that are produced to monitor actual and wild emotional responses.
Highlights
Emotion recognition technology through facial expression, action, and voice have been actively studied for advanced human-robot interaction (HRI)
In order to reduce the network size without performance degradation, this paper presents a more compact landmark feature map (LFM) (CLFM) which is generated by appropriately sampling landmark points
leave-one-subject-out cross validation (LOSO) validation was applied to all techniques. 16-fold validation was used for SMIC and 26-fold validation for CASME II as in conventional techniques
Summary
Emotion recognition technology through facial expression, action, and voice have been actively studied for advanced human-robot interaction (HRI). Studies on facial expression-based emotion recognition (FER) are most active [1]–[6], [8]–[12], [33]–[35]. Conventional FER methods are focusing on facial macro-expressions as shown in Fig. 1 (a). Recognizing emotions corresponding to such macro-expressions is not a big deal. In actual situations (see Fig. 1 (b)), people may rarely express their emotions on the faces. In terms of simple facial expression metric (SFEM), i.e., one of the facial expression intensity metrics (see Section IV), the intensities of facial expression of Fig. 1 (a) and Fig. 1 (b) are 1.83 and 0.20, respectively, which indicate a big difference of about
Sign-in/Register to view highlights & summary Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
