Learning Key Actors and Their Interactions for Group Activity Recognition

Abstract

Group activity recognition is a challenging task. Group activities usually involves many actors, while only some key actors play a decisive role in group activity recognition. Therefore, extraction of key actors and their interactions is an important problem in group activity recognition. To tackle this problem, we propose a new method based on graph convolutional layers and self-attention mechanisms to extract the Subgraphs of the Actor Relation Graph (SARG). SARG is a scene representation that only contains key actors and their interactions, which is used to enhance the importance of key actors in each group activity. First, the actor relation graph was generated via the appearance and location information of the actors; it was further analyzed by using a graph convolutional layer. Second, we use the graph convolutional layer to generate the self-attention features for each participant and extract the actor relation subgraphs that can ascertain group activities. Finally, we fuse actor relation subgraphs, actor relation graphs and original features to recognize group activities. We evaluate our model over two datasets: the collective activity dataset and the volleyball dataset. SARG has an average improvement of \(4\%\), compared to 8 benchmarks.