Joint Prediction of Group-Level Emotion and Cohesiveness with Multi-Task Loss

Abstract

This paper presents a hybrid deep learning network for the prediction of group-level emotion and cohesiveness. In this work, we first train deep models individually on face, pose, whole image, as well as fusion of them on Group Affect Dataset to predict group-level emotion, then feed the classification results into additional regression layer to regress group cohesiveness. Thus, our model combines group emotion and group cohesiveness and achieves better results. The best result we obtained on the test set is an ensemble of best models we trained on the validation set, and this model achieve a MSE of 0.4849. In order to further improve the performance, a multi-task loss model which combines classification of group emotion with regression of cohesiveness is adopted. Prior work on group cohesiveness usually fulfill the task of cohesiveness regression based on the output of emotion classification network. However, the two characteristics are believed to be correlated but one cannot necessarily predict the other. Hence, both information sources are important. Thus, the proposed multi-task loss setting combines the classification and regression tasks. The results prove that estimation of group emotion and cohesiveness is correlated and can be benefited by joint training of the two tasks.