Fine-grained facial expression recognition via relational reasoning and hierarchical relation optimization

Abstract

Facial expression recognition (FER), aiming to recognize the type of facial expressions, has achieved significant progress. However, most of the existing FER approaches ignore the influence of structure relations among image set and the semantic associations between labels. Recently, some studies turn to explore fine-grained FER which includes hierarchical label structure, but they merely explore the influence of hierarchical relations of labels. Inspired by this, in this paper, we propose a relational reasoning and hierarchical relation optimization network (R3HO-Net) that explores the above three kinds of relations simultaneously. Concretely, we first construct two sub-graphs, i.e., intra-image graph (IIG) and intra-label graph (ILG). Meanwhile, we propose an entropy-based relation adaptive initialization strategy to construct a heterogeneous inter-graph (HIG). Then fine-grained stream of R3HO-Net, including a relational update GCN module, updates the graphs simultaneously in an iterative way and outputs the mapping probabilities between heterogeneous node pairs to infer the final mapping results. Moreover, we also propose a hierarchical label optimization module and a hierarchical optimization loss to optimize fine-grained prediction results. Extensive experiments on serveral benchmarks demonstrate the superiority of the proposed approach.