Group Activity Recognition Research Articles

Contextual motion-aware for group activity recognition

Contextual motion-aware for group activity recognition

Group activity recognition using unreliable tracked pose

Group activity recognition using unreliable tracked pose

Multi-dimensional convolution transformer for group activity recognition

Multi-dimensional convolution transformer for group activity recognition

Coarse-Fine Nested Network for Weakly Supervised Group Activity Recognition.

Weakly supervised group activity recognition (WSGAR) aims at identifying the overall behavior of multiple persons without any fine-grained supervision information (including individual position and action label). Traditional methods usually adopt a person-to-whole way: detect persons via off-the-shelf detectors, obtain person-level features, and integrate into the group-level features for training the classifier. However, these methods are unflexible due to serious reliance on the quality of detectors. To get rid of the detector, recent works learn several prototype tokens from noisy grid features with learnable weights directly, which treat all the local visual information equally and bring in redundant and ambiguous information to some extent. To this end, we propose a novel coarse-fine nested network (CFNN) to coarsely localize the key visual patches of activity and further finely learn the local features, as well as the global features. Specifically, we design a nested interactor (NI) to progressively model the spatiotemporal interactions of the learnable global token. According to the cue of spatial interaction in NI, we localize several key visual patches via a new coarse-grained spatial localizer (CSL). Then, we finally encode these localized visual patches with the help of global spatiotemporal dependency via a new fine-grained spatiotemporal selector (FSS). Extensive experiments on Volleyball and NBA datasets demonstrate the effectiveness of the proposed CFNN compared with the existing competitive methods. Code is available at: https://github.com/gexiaojingshelby/CFNN.

MLP-AIR: An effective MLP-based module for actor interaction relation learning in group activity recognition

Modeling actor interaction relations is crucial for group activity recognition. Previous approaches often adopt a fixed paradigm that involves calculating an affinity matrix to model these interaction relations, yielding significant performance. On the one hand, the affinity matrix introduces an inductive bias that actor interaction relations should be dynamically computed based on the input actor features. On the other hand, MLPs with static parameterization, in which parameters are fixed after training, can represent arbitrary functions. Therefore, it is an open question whether inductive bias is necessary for modeling actor interaction relations. To explore the impact of this inductive bias, we propose an affinity matrix-free paradigm that directly uses the MLP with static parameterization to model actor interaction relations. We term this approach MLP-AIR. This paradigm overcomes the limitations of the inductive bias and enhances the capture of implicit actor interaction relations. Specifically, MLP-AIR consists of two sub-modules: the MLP-based Interaction relation modeling module (MLP-I) and the MLP-based Relation refining module (MLP-R). MLP-I is used to model the spatial–temporal interaction relations by emphasizing cross-actor and cross-frame feature learning. Meanwhile, MLP-R is used to refine the relation between different channels of each relation feature, thereby enhancing the expression ability of the features. MLP-AIR is a plug-and-play module. To evaluate our module, we applied MLP-AIR to replicate three representative methods. We conducted extensive experiments on two widely used benchmarks—the Volleyball and Collective Activity datasets. The experiments demonstrate that MLP-AIR achieves favorable results. The code is available at https://github.com/Xuguoliang12/MLP-AIR.

Exploring global context and position-aware representation for group activity recognition

This paper explores the context and position information in the scene for group activity understanding. Firstly, previous group activity recognition methods strive to reason on individual features without considering the information in the scene. Besides correlations among actors, we argue that integrating the scene context simultaneously can afford us more useful and supplementary cues. Therefore, we propose a new network, termed Contextual Transformer Network (CTN), to incorporate global contextual information into individual representations. In addition, the position of individuals also plays a vital role in group activity understanding. Unlike previous methods that explore correlations among individuals semantically, we propose Clustered Position Embedding (CPE) to integrate the spatial structure of actors and produce position-aware representations. Experimental results on two widely used datasets for sports video and social activity (i.e., Volleyball and Collective Activity datasets) show that the proposed method outperforms state-of-the-art approaches. Especially, when using ResNet-18 as the backbone, our method achieves 93.6/93.9% MCA/MPCA on the Volleyball dataset and 95.4/96.3% MCA/MPCA on the Collective Activity dataset.

React: recognize every action everywhere all at once

In the realm of computer vision, Group Activity Recognition (GAR) plays a vital role, finding applications in sports video analysis, surveillance, and social scene understanding. This paper introduces Recognize Every Action Everywhere All At Once (REACT), a novel architecture designed to model complex contextual relationships within videos. REACT leverages advanced transformer-based models for encoding intricate contextual relationships, enhancing understanding of group dynamics. Integrated Vision-Language Encoding facilitates efficient capture of spatiotemporal interactions and multi-modal information, enabling comprehensive scene understanding. The model’s precise action localization refines joint understanding of text and video data, enabling precise bounding box retrieval and enhancing semantic links between textual descriptions and visual reality. Actor-Specific Fusion strikes a balance between actor-specific details and contextual information, improving model specificity and robustness in recognizing group activities. Experimental results demonstrate REACT’s superiority over state-of-the-art GAR approaches, achieving higher accuracy in recognizing and understanding group activities across diverse datasets. This work significantly advances group activity recognition, offering a robust framework for nuanced scene comprehension.

Spatiotemporal information complementary modeling and group relationship reasoning for group activity recognition

Spatiotemporal information complementary modeling and group relationship reasoning for group activity recognition

FIFAWC: a dataset with detailed annotation and rich semantics for group activity recognition

FIFAWC: a dataset with detailed annotation and rich semantics for group activity recognition

Rethinking group activity recognition under the open set condition

Rethinking group activity recognition under the open set condition

Design and Analysis of Efficient Attention in Transformers for Social Group Activity Recognition

Design and Analysis of Efficient Attention in Transformers for Social Group Activity Recognition

Unveiling group activity recognition: Leveraging Local–Global Context-Aware Graph Reasoning for enhanced actor–scene interactions

Group activity recognition aims to recognize holistic activity in multi-person scene, necessitating consideration of the interactions between actors and their surroundings. It has various applications, such as public surveillance, video analysis. Nonetheless, existing works merely extract scene features as a supplementary component of activity features, failing to adequately explore the interplay between the scene and actors. To address this limitation, this paper proposes a Local–Global Context-Aware Graph Reasoning Model (LG-CAGR), which leverages and reasons through local and global context features to gain deeper insights into group activity within the scene. In particular, we present an innovative feature extraction strategy to harness local location features and global scene attributes, effectively complementing actor features by capturing spatial group topology and determining relative positions between actors. Subsequently, we delve into these features by devising a local–global group reasoning module that deduces pair-wise interactions between actors and scenes within Graph Convolutional Network, comprehensively elucidating correlations between overall scene and local individuals to construct group-level features. Multi-graphs are constructed considering actor’s features, scene features as nodes, and interactions as edges. A self-attention graph pooling network is introduced to automatically integrate key actor features and form rich group-level features to recognize group activity. The results on Collective Activity Dataset, Collective Activity Extended Dataset, Volleyball Dataset and Public Life in Public Space dataset have reached 94.0%, 97.7%, 92.7% and 56.1%. Compared with existing methods using the same backbone, we exceeded 1%, 2.1%, 0.3%, and 14.9% respectively, affirming the superiority of the proposed method compared with state-of-the-art methods.

Temporal Contrastive and Spatial Enhancement Coarse Grained Network for Weakly Supervised Group Activity Recognition

Group activity recognition (GAR) is an increasingly popular topic in the field of computer vision. Numerous researchers have proposed a range of methods to achieve outstanding recognition performance. However, these methods invariably require fine-grained personal feature extraction and a large network architecture to aggregate individual features or reason person relationships. To mitigate the need for a bloated portfolio of annotations and high training costs, weak supervision has emerged as a promising approach. Under the weak supervision paradigm, only coarse-grained labels are used during network training. Nevertheless, this method poses two key challenges. Firstly, it is limited in its ability to model temporal relationships among individual persons, and secondly, it tends to focus on less relevant information, thereby leading to suboptimal network parameter optimization. Both of these challenges result in erroneous temporal information judgment and training inefficiencies. To address these challenges within the weak supervision paradigm, we propose a novel Temporal Contrastive and Spatial Enhancement Coarse-Grained Network (TCSE-CGN) to solve the GAR problem. TCSE-CGN comprises two simple yet effective streams, namely the Spatial Enhancement Stream and the Temporal Contrastive Stream. After extracting features using only several RGB frames, half of the extracted feature is sent to the Spatial Enhancement Stream for enhancement using an attention mechanism. Consequently, the network automatically learns more representative information. The remaining feature is sent to the Temporal Contrastive Stream, which uses contrastive learning to model temporal relationships among all RGB frame-level features. Specifically, the network is guided to learn the hidden semantic temporal information about inter-frame sequences. Network parameters are optimized using a combination of universe cross-entropy loss and a novel temporal contrastive loss. Comprehensive experiments are conducted on two widely used datasets, namely the Volleyball dataset and the Collective dataset, to demonstrate the effectiveness of TCSE-CGN. Results show that TCSE-CGN performs competitively with other works that require more supervision and a larger architecture.

Dynamical Attention Hypergraph Convolutional Network for Group Activity Recognition.

Recently, group activity recognition (GAR) has drawn growing interests in video analysis and computer vision communities. The current models of GAR tasks are often impractical in that they suppose that all interactions between actors are pairwise, which only models and leverages part of the information in real entire interactions. Motivated by this, we design a distinct dynamical attention hypergraph convolutional network framework, referred to as DAHGCN, for precise GAR, modeling the entire interactions and capturing the high-order relationships among involved actors in a real-life scenario. Specifically, to learn complementary feature representations for fine-grained GAR, a multilevel feature descriptor (MLFD) module is proposed. Furthermore, for learning higher order interaction relationships, we construct a DAHGCN to accommodate complex group interactions, which can dynamically change the topology of the hypergraph and learn these key representations by virtue of the "similarity-based shared nearest-neighbor (SSNN) clustering" and "attention mechanisms" on hypergraph. Finally, a multiscale temporal convolution (MSTC) module is utilized to explore various long-range temporal dynamic correlations across different frames. In addition, comprehensive experiments on three commonly used GAR datasets clearly demonstrate that, when compared with the state-of-the-art methods, our proposed method can achieve the most optimal performance.

GCVC: Graph Convolution Vector Distribution Calibration for Fish Group Activity Recognition

Only a few key fish individuals can play a dominant role in actual fish group, therefore, it is reasonable to infer group activities from the relationship between individual actions. However, the complex underwater environment, rapid and similar fish individual movements are likely to cause the indistinct action characteristics, as well as adhesion of data distribution, and it is difficult to infer the relationship between individual actions directly by using graph convolutional network (GCN). Therefore, this paper proposes a graph convolution vector calibration (GCVC) network for fish group activity recognition through individual action relationship reasoning. By improving reasoning ability of GCN, an activity feature vector calibration module is designed to solve the data adhesion and mismatch between the estimated and true distribution. The idea is to first count the distribution of the original data, and make each dimension of its active feature vector follow the Gaussian distribution, so as to generate a better similar category distribution. In addition, we also produced a fish activity dataset to verify the performance of the proposed algorithm. The experimental results show that the GCVC achieves a group activity recognition accuracy of 93.33 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\%$</tex-math></inline-formula> , and the Macro-F1 is 93.25 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\%$</tex-math></inline-formula> , which is 19.21 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\%$</tex-math></inline-formula> and 24.2 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\%$</tex-math></inline-formula> higher than before, respectively. By using GCVC, the corrected activity feature vector distribution is more consistent, and the data adhesion is reduced, the model can achieve more fully supervised learning. The fish group activity dataset is available on Github: <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/crazysboy/GCVC/tree/master</uri> .

Active Factor Graph Network for Group Activity Recognition.

Group activity recognition aims to identify a consistent group activity from different actions performed by respective individuals. Most existing methods focus on learning the interaction between each two individuals (i.e., second-order interaction). In this work, we argue that the second-order interactive relation is insufficient to address this task. We propose a third-order active factor graph network, which models the third-order interaction in each pair of three active individuals. At first, to alleviate the noisy individual actions, we select active individuals by measuring each individual's influence. The individuals with the top-k largest influence weights are selected as active individuals. Then, for each three-individuals pair, we build a new factor node and contact the factor node with these individual nodes. In other words, we extend the base second-order interactive graph to a new third-order interactive graph, which is defined as factor graph. Next, we design a two-branch factor graph network, in which one branch is to consider all individuals (denoted as full factor graph) and the other one takes the active individuals into consideration (denoted as active factor graph). We leverage both the active and full factor graphs comprehensively for group activity recognition. Besides, to enforce group consistency, a consistency-aware reasoning module is designed with two penalty terms, which describe the inconsistency between individual actions and group activity respectively. Extensive experiments demonstrate that our method achieves state-of-the-art performance on four benchmark datasets, i.e., Volleyball, Collective Activity, Collective Activity Extended, and SoccerNet-v3 datasets. Visualization results further validate the interpretability of our method.

Masked Autoencoders for Spatial–Temporal Relationship in Video-Based Group Activity Recognition

Masked Autoencoders for Spatial–Temporal Relationship in Video-Based Group Activity Recognition

Learning Label Semantics for Weakly Supervised Group Activity Recognition

Learning Label Semantics for Weakly Supervised Group Activity Recognition

Corrections to “Attention Relational Network for Skeleton-Based Group Activity Recognition”

Corrections to “Attention Relational Network for Skeleton-Based Group Activity Recognition”

Multi-level neural prompt for zero-shot weakly supervised group activity recognition

Group activity recognition aims at understanding the overall activity involved multi-person interaction with each other as a whole in the complex variations in both spatial and temporal transition in untrimmed video. While remarkably significance in applications, existing methods are often impractical for the following two reasons: (1) they rely on heavy annotations and off-the-shelf object detectors even in test stage; (2) they are trained to predict a closed-set predetermined group categories, which limits their generality to wider unseen categories. Motivated by this, we propose a novel zero-shot weakly supervised group activity recognition neural model, ZSTGroupCLIP, which is not only free from the restriction of heavy annotations, but also captures the generalizable elements that are vital for open-set exploration. Specifically, our model based on visual–textual joint representation learning seeks discriminative visual cues for aligning both the group vision and category text branches by multi-level neural prompt. Moreover, to further learn group-related contextual representations under weakly supervision, we learn layer-wise prompts across early layer of group encoder for progressive context modeling. Through extensive experimental validations and comparisons to several baseline methods on two benchmarks, Volleyball and NBA datasets, our method achieves outstanding performance.