Enhanced discriminative global-local feature learning with priority for facial expression recognition

Abstract

Facial expression recognition (FER) with disturbances is challenging because of contamination of global features and inappropriate patch cropping of local features. Although methods based on global and local features have been combined, their performance is typically determined by feature fusion. This study proposes enhanced discriminative global-local feature learning with priority (EDGL-FLP), which focuses on feature extraction without auxiliary information and feature fusion based on priority. The proposed uniform mixing multiscale module traverses all subset combinations for global features to solve the insufficient information mixing problem in the existing multiscale module. A novel saliency and statistics-aware attention module is proposed for local features to extract the channel and spatial attention of saliency and statistics in two subbranches. It avoids information confusion caused by the early channel attention fusion in the convolutional block attention module. The global features are enhanced with core face area features and fused with half-face features for additional discriminative guidance because faces are symmetrical and their center is crucial. EDGL-FLP achieves state-of-the-art performance on the FER benchmarks RAF-DB, SFEW, AffectNet, FED-RO, and MMI with accuracies of 89.63%, 62.31%, 61.09%, 71.42%, and 86.47%, respectively. Thus, EDGL-FLP is robust for both in-the-wild and in-the-lab FER datasets.