Unconstrained Facial Expression Recognition With No-Reference De-Elements Learning

Abstract

Most unconstrained facial expression recognition (FER) methods take original facial images as inputs to learn discriminative features by well-designed loss functions, which cannot reflect important visual information in faces. Although existing methods have explored the visual information of constrained facial expressions, there is no explicit modeling of what visual information is important for unconstrained FER. To find out valuable information of unconstrained facial expressions, we pose a new problem of no-reference de-elements learning: we decompose any unconstrained facial image into the facial expression element and a neutral face without the reference of corresponding neutral faces. Importantly, the element provides visualization results to understand important facial expression information and improves the discriminative power of features. Moreover, we propose a simple yet effective <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">D</b> e- <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">E</b> lements <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Net</b> work (DENet) to learn the element and introduce appropriate constraints to overcome no ground truth of corresponding neutral faces during the de-elements learning. We extensively evaluate the proposed method on in-the-wild FER datasets including RAF-DB, AffectNet, SFEW and FERPlus. The comparable results show that our method is promising to improve classification performance and achieves equivalent performance compared with state-of-the-art methods. Also, we demonstrate the strong generalization performance on realistic occlusion and pose variation datasets and the cross-dataset evaluation.