Imbalanced regression for intensity series of pain expression from videos by regularizing spatio-temporal face nets

Abstract

• Addressing limited data with expression intensity labels. • Pushing the pain assessment performance by a large margin. • Proposing to add a center-loss-based regularizer. • Proposing a more sensible evaluation metric . Obtaining accurate patient-reported pain intensity is essential for effective pain management. An automatic pain recognition system can simplify the pain reporting process and reduce the strain on manual efforts. Limited and imbalanced labeled data are available for the research of estimating the intensity of pain based on facial expressions. However, the ability to train deep networks for automated pain assessment is limited by small datasets with imbalanced labels of patient-reported pain levels. Fortunately, fine-tuning from a data-extensive pre-trained domain, such as face verification or recognition, can alleviate this problem to some extent. In this paper, we propose a network which fine-tunes a face verification or recognition network using a regularized regression loss and additional data with pain-intensity labels. The expression intensity regression task can benefit from the rich feature representations trained on a large number of data for face analysis tasks. In order to explore the temporal information between frames, we combine CNN with LSTM to obtain a better prediction result of each frame in videos. A weighted evaluation metric and re-sampling technique are also proposed to address the imbalance issue of different pain levels. The proposed regularized deep regressor is applied to estimate the pain expression intensity and verified on the widely-used UNBC-McMaster Shoulder-Pain dataset and BioVid Heat Pain dataset, achieving the state-of-the-art performance. As pain detection is a form of micro facial expression recognition, we also apply the transferred deep regressor to estimate the intensity of facial action units, obtaining high quality performance.