Multi-scale features based interpersonal relation recognition using higher-order graph neural network

Abstract

Interpersonal relation plays an essential role to gain understandings on how people interact with each other. In computer vision, interpersonal relations provide vital information to interpret people’s behaviors. However, the existing research has either omitted the interaction information between subjects or the structural information in the images. In this paper, we propose a new architecture to reason interpersonal relations based on higher-order graph networks and multi-scale features. First, we extract features of the whole images, the facial features, and the union region of face pairs. Apart from the pixel-wise features, we also consider the positional features of face-to-face pairs and the spatial scene cues. Higher-order Graph Neural Networks (GNNs) were employed to map out the interpersonal relations based on the feature extracted. Experimental results show that the proposed Higher-order Graph Neural Networks with multi-scale features can effectively recognize the social relations in images with over 5% improvement in absolute balanced accuracy compared with the state-of-the-art work.