Sequence-level affective level estimation based on pyramidal facial expression features

Abstract

People tend to focus on changes in a certain complex human affect in the majority of practical applications of affective computing. Facial expression classification models are unable to represent all human affects through a limited number of expression categories. In this backdrop, this paper studies the Sequence-level affective level estimation (S-ALE), which is more relevant to real scenarios and can depict individual affective level in continuous manner. A spatio-temporal framework applied to S-ALE is proposed, which consists of a Facial Expression Features Pyramid Network (FEFPN) and a Temporal Transformer Encoder (TTE). FEFPN is capable of extracting pyramidal facial expression features, while TTE can effectively capture coarse-grained and fine-grained temporal variations of facial sequences. The proposed model is evaluated on six public datasets across three typical S-ALE tasks (engagement prediction, fatigue detection, and pain assessment), and experimental results show that our method is comparable to or outperforms the state-of-the-art algorithms.