3-D Facial Expression Recognition via Attention-Based Multichannel Data Fusion Network

Abstract

Facial expression has long been recognized as containing meaningful nonverbal affective cues for decoding human emotions. Recently, multimodal 2-D + 3-D fusion method has shown significant potential in facial expression recognition (FER) due to its fine-grained face descriptions in various spatial channels. However, current work mainly relies on feature- or even score-level fusion to find emotion cues spread in different channels and may miss key information due to lack of focus. To this end, we propose an attention-based multichannel data fusion network (AMDFN) to better preserve and find such key facial cues. More specifically, we first map a 3-D face scan into multichannel images and then fuse them in a ResNet18 backbone to get layered emotion features. Second, we leverage a layer attention model to explore the dependencies between features of different layers to learn discriminative affective cues for effective emotion recognition. Our comprehensive experiments on two widely used datasets (i.e., Facescape and Bosphorus) have verified the performance of our approach compared to several state-of-the-art rivals.