Hybrid feature-based analysis of video’s affective content using protagonist detection

Abstract

Extracting an effective representation to analyze affection in videos is an inherent challenge. To address this problem, we present a novel emotion recognition system that can intelligently analyze and automatically recognize video emotions. We observe that most of the emotions in a video are closely related to the roles, especially those of the protagonist, which motivates us to explore how to utilize the protagonist’s information to help recognize video emotions. By analyzing the traits of the protagonists, we suggest a new solution to detect the protagonists to adapt not only to the whole video but also to a video clip. Moreover, a new keyframe selection strategy based on the protagonist is designed to select a set of representative frames from video clips. Furthermore, the scale invariant feature transform (SIFT) features matrix, built from each keyframe, is fed into a convolutional neural network (CNN) to learn the discriminative representations, which makes the CNN and local features complement each other. Considering that emotions are usually continuous, we introduce temporal information into the CNN by using optical flow images. Additionally, we extract some handcrafted visual and audio features as a supplement. Finally, all the features, including the features learned from the CNN and the handcrafted features, are fused and input into a support vector machine (SVM) and a support vector regression (SVR) for video emotion recognition. The proposed system is validated on a public dataset (LIRIS-ACCEDE) and a new dataset (PMSZU). The experimental results demonstrate the promising performance of the proposed system, which achieves better performance than the compared methods. Our designed system for recognizing video emotions could also facilitate the development of similar video expert systems, for applications such as video recommendations, video classification and video retrieval.