Expression-assisted facial action unit detection through an attention mechanism and smooth class-weighted Loss

Abstract

The performance of facial action unit (AU) detection is often limited owing to the lack of annotated AU data and data imbalance. The data scarcity problem can be partially mitigated by abundant expression data, as AU detection and facial expression recognition (FER) are closely related. Accordingly, in this study, FER and AU detection are trained jointly in a multi-task learning framework, in which FER serves as an auxiliary task for AU detection by providing supplementary information. Meanwhile, we propose to use an attention gate unit between the two tasks to flexibly select valuable information from each other. To address the model bias issue caused by AU data imbalance, a smooth class-weighted loss is adopted to alleviate the dominance of negative AU classes. The best average F1-score obtained using our approach on the BP4D dataset is 63.5%, which is very close to the state-of-the-art performance and exceeds the single task baseline by 3.2%.