Quaternion Capsule Neural Network With Region Attention for Facial Expression Recognition in Color Images

Abstract

Facial expression recognition (FER) is an essential subject of computer vision and human-computer interaction. It has been reported that many factors are closely related to the FER performance such as the pose, facial muscle variations, and the ignored color information in facial images. In this study, we propose a quaternion capsule neural network (Q-CapsNet) with region attention mechanism for FER in color images. The proposed Q-CapsNet is an end-to-end deep learning framework, which adopts the concept of quaternion theory to the Capsule Neural Network and further uses an attention mechanism to focus on the facial region of interests (ROIs). The Q-CapsNet addresses the FER problems in the following aspects. First, the internal dependencies between color channels are captured by the quaternion technique, which is of great importance in color image processing. Second, the Q-CapsNet could disentangle latent geometry information from facial images by quaternion capsules and quaternion routing algorithm. Third, inspired by the fact that facial expressions are mainly determined by several vital facial regions, the region attention mechanism is introduced to extract emotional features from the ROIs of the face. The proposed Q-CapsNet is evaluated on four public color FER datasets including MMI, Oulu-CASIA, RAF-DB, and SFEW. The comprehensive experiments and visualization results demonstrate that the proposed network outperforms comparative models and the state-of-the-art FER methods.