Recognizing Social Relationships in Long Videos via Multimodal Character Interaction

Abstract

Social relationships between characters in multiple clips of a long video may necessitate multi-step reasoning. However, most existing long video understanding approaches fail to capture the relational dependencies between characters in different clips. To solve this problem, we propose a Multimodal and Multi-granularity Relation Recognition (MMRR) framework to extract social relationships from long videos. First, we design a novel Multimodal Heterogeneous Graph (MHG) that learns the relational interactions between characters by propagating information between multimodal character nodes and multimodal clip nodes. Second, to better incorporate the contextual information from multiple character feature representations, we build a Multi-granularity Character Representation Module (MCRM) that learns global character representations over the entire long video, as well as person-specific local character representations via attention mechanism. Experimental results on two real-world datasets demonstrate the superiority of our method.