HybNet: a hybrid network structure for pain intensity estimation

Abstract

Automatic pain intensity estimation has great potential in current rehabilitation medicine, and patients’ health status information can be obtained through the analysis of facial images. At present, deep convolutional neural networks (CNNs) have made great progress in many fields, including natural language processing, image classification and action recognition. Motivated by the current achievements, a novel end-to-end hybrid network is proposed to extract multidimensional features from image sequences, which is composed of 3D convolution, 2D convolution and 1D convolution. Specifically, the 3D convolutional neural network (3D CNN) is designed to capture the spatiotemporal features, and the 2D convolutional neural network (2D CNN) is designed to capture the spatial features, while the 1D convolutional neural network (1D CNN) is mainly used to capture the geometric information from facial landmarks. Finally, the features obtained by the three different networks are fused together for regression. The proposed HybNet is evaluated on UNBC-McMaster Shoulder Pain Expression Archive Database, and the experimental results show that it can effectively extract the discriminative high-level features and can achieve competitive performance with the state-of-the-art methods.