Skeleton-based Action Recognition Research Articles

Attention-Guided and Topology-Enhanced Shift Graph Convolutional Network for Skeleton-Based Action Recognition

Graph Convolutional Networks (GCNs) have emerged as a game-changer in skeleton-based action recognition. However, most previous works are resource-heavy, with large FLoating-number OPerations (FLOPs) limiting the model’s potential. A recent work involving shift operators to GCN (Shift-GCN) has successfully introduced a lightweight GCN, but there is still a performance gap compared to previous results. Inspired by Shift-GCN, we propose an innovative and novel model named attention-guided and topology-enhanced shift graph convolutional network (AT-Shift-GCN), which continues the lightweight benchmark and provides a more powerful performance. We employ a topological transfer operation to aggregate the information flow of different channels and extract spatial information. In addition, to extract temporal information across scales, we apply attention to interacting with shift convolution kernels of different lengths. Furthermore, we integrate an ultralight spatiotemporal attention module to fuse spatiotemporal details and provide robust neighborhood representation. In summary, AT-Shift-GCN is a breakthrough in skeleton-based action recognition that provides a lightweight model with enhanced performance on three datasets.

Multi-scale motion contrastive learning for self-supervised skeleton-based action recognition

Multi-scale motion contrastive learning for self-supervised skeleton-based action recognition

MAFormer: A cross-channel spatio-temporal feature aggregation method for human action recognition

Human action recognition has been widely used in fields such as human–computer interaction and virtual reality. Despite significant progress, existing approaches still struggle with effectively integrating hierarchical information and processing data beyond a certain frame count. To address these challenges, we introduce the Multi-AxisFormer (MAFormer) model, which is organized in terms of spatial, temporal, and channel dimensions of the action sequence, thereby enhancing the model’s understanding of correlations and intricate structures among and within features. Drawing on the Transformer architecture, we propose the Cross-channel Spatio-temporal Aggregation (CSA) structure for more refined feature extraction and the Multi-Axis Attention (MAA) module for more comprehensive feature aggregation. Moreover, the integration of Rotary Position Embedding (RoPE) boosts the model’s extrapolation and generalization abilities. MAFormer surpasses the known state-of-the-art on multiple skeleton-based action recognition benchmarks with the accuracy of 93.2% on NTU RGB+D 60 cross-subject split, 89.9% on NTU RGB+D 120 cross-subject split, and 97.2% on N-UCLA, offering a novel paradigm for hierarchical modeling in human action recognition.

Prompt-supervised dynamic attention graph convolutional network for skeleton-based action recognition

Skeleton-based action recognition is a core task in the field of video understanding. Skeleton sequences are characterized by high information density, low redundancy, and clear structural information, thereby facilitating the analysis of complex relationships among human behaviors more readily than other modalities. Although existing studies have encoded skeleton data and achieved positive outcomes, they have often overlooked the precise high-level semantic information inherent in the action descriptions. To address this issue, this paper proposes a prompt-supervised dynamic attention graph convolutional network (PDA-GCN). Specifically, the PDA-GCN incorporates a prompt supervision (PS) module that leverages a pre-trained large-scale language model (LLM) as a knowledge engine and retains the generated text features as prompts to provide additional supervision during model training, enhancing the model’s ability to discern analogous actions with negligible computational cost. In addition, for the purpose of bolstering the learning of discriminative features, a dynamic attention graph convolution (DA-GC) module is presented. This module utilizes self-attention mechanism to adaptively infer intrinsic relationships between joints and integrates dynamic convolution to strengthen the emphasis on local information. This dual focus on both global context and local details further amplifies the efficiency and effectiveness of the model. Extensive experiments, conducted on the widely-used skeleton-based action recognition datasets NTU RGB+D 60 and NTU RGB+D 120, demonstrate that the PDA-GCN surpasses known state-of-the-art methods, achieving accuracies of 93.4% on the NTU RGB+D 60 cross-subject split and 90.7% on the NTU RGB+D 120 cross-subject split.

Improved semantic-guided network for skeleton-based action recognition

A fundamental issue in skeleton-based action recognition is the extraction of useful features from skeleton joints. Unfortunately, the current state-of-the-art models for this task have a tendency to be overly complex and parameterized, which results in low model training and inference time efficiency for large-scale datasets. In this work, we develop a simple but yet an efficient baseline for skeleton-based Human Action Recognition (HAR). The architecture is based on adaptive GCNs (Graph Convolutional Networks) to capture the complex interconnections within skeletal structures automatically without the need of a predefined topology. The GCNs are followed and empowered with an attention mechanism to learn more informative representations. This paper reports interesting accuracy on a large-scale dataset NTU-RGB+D 60, 89.7% and 95.0% on respectively Cross-Subject, and Cross-View benchmarks. On NTU-RGB+D 120, 84.6% and 85.8% over Cross-Subject and Cross-Setup settings, respectively. This work provides an improvement of the existing model SGN (Semantic-Guided Neural Networks) when extracting more discriminant spatial and temporal features.

Modeling the skeleton-language uncertainty for 3D action recognition

Human 3D skeleton-based action recognition has received increasing interest in recent years. Inspired by the excellent ability of the multi-modal model, some pioneer attempts to employ diverse modalities, i.e., skeleton and language, to construct the skeleton-language model and have shown compelling results. Yet, these attempts model the data representation as deterministic point estimation, ignoring a key issue that descriptions of similar motions are uncertain and ambiguous, which brings about restricted comprehension of complex concept hierarchies and impoverished cross-modal alignment reliability. To tackle this challenge, this paper proposes a new Uncertain Skeleton-Language Learning Framework (USLLF) to capture the semantic ambiguity among diverse modalities in a probabilistic manner for the first time. USLLF consists of both inter- and intra-modal uncertainties. Specifically, first, we integrate the language (text) generated by ChatGPT with the generic skeleton-based network and develop a deterministic multi-modal baseline, which can be easily achieved via any off-the-shelf skeleton and text encoders. Then, based on this baseline, we explicitly model the intra-modal (skeleton/language) uncertainties as the Gaussian distributions using the new uncertainty networks capable of learning the distributional embeddings of modalities. Following this, these embeddings are aligned and formulated as inter-modal (skeleton-language) uncertainty using both the contrastive and negative log-likelihood objectives to alleviate the cross-modal alignment error. Experimental results on NTU RGB+D, NTU RGB+D 120, and NW-UCLA datasets show that our approach outperforms the proposed baseline and achieves comparable performance with a high inference efficiency compared to the state-of-the-art methods. Besides, we also deliver insightful analyses on how learned uncertainty reduces the impact of uncertain and ambiguous data on model performance.

Variation-aware directed graph convolutional networks for skeleton-based action recognition

Directed Graph convolutional networks (DGCNs) have been indeed gaining attention and being applied in skeleton-based action recognition tasks to capture the hierarchical relationships of skeleton via directed graph topology. However, they typically pay the same attention to regions with variations and regions with relative statics of the skeleton, which leads to low accuracy when recognizing fine-grained actions. To this end, we design an innovative variation-aware directed graph convolutional network (VA-DGCN) to focus on the regions where the variation takes place. VA-DGCN comprises a variation-aware directed spatial convolution (VDSC) module and a multi-scale contrastive temporal convolution (MCTC) module. Specifically, in order to capture subtle variations in fine-grained actions, VDSC introduce the average posture of the action sequence as the static anchor. In VDSC, the channel-specific topology branch is designed to model the different kinematic properties of different channels to extract global features, to which global-attention graph convolution is added for unnaturally connected joints. Subsequently, we identify the regions with variations by comparing global features acquired from action sequence and average posture. In temporal, MCTC comprises multiple branches for extracting temporal features at different scales. Moreover, to maximize the mutual information between branches, we introduce contrastive learning to drive the module to learn more meaningful action representations. We conduct extensive experiments on three public datasets to validate the feasibility and efficacy of our proposed VA-DGCN.

Spatiotemporal Decouple-and-Squeeze Contrastive Learning for Semisupervised Skeleton-Based Action Recognition.

Contrastive learning has been successfully leveraged to learn action representations for addressing the problem of semisupervised skeleton-based action recognition. However, most contrastive learning-based methods only contrast global features mixing spatiotemporal information, which confuses the spatial- and temporal-specific information reflecting different semantic at the frame level and joint level. Thus, we propose a novel spatiotemporal decouple-and-squeeze contrastive learning (SDS-CL) framework to comprehensively learn more abundant representations of skeleton-based actions by jointly contrasting spatial-squeezing features, temporal-squeezing features, and global features. In SDS-CL, we design a new spatiotemporal-decoupling intra-inter attention (SIIA) mechanism to obtain the spatiotemporal-decoupling attentive features for capturing spatiotemporal specific information by calculating spatial- and temporal-decoupling intra-attention maps among joint/motion features, as well as spatial- and temporal-decoupling inter-attention maps between joint and motion features. Moreover, we present a new spatial-squeezing temporal-contrasting loss (STL), a new temporal-squeezing spatial-contrasting loss (TSL), and the global-contrasting loss (GL) to contrast the spatial-squeezing joint and motion features at the frame level, temporal-squeezing joint and motion features at the joint level, as well as global joint and motion features at the skeleton level. Extensive experimental results on four public datasets show that the proposed SDS-CL achieves performance gains compared with other competitive methods.

SkelVIT: consensus of vision transformers for a lightweight skeleton-based action recognition system

SkelVIT: consensus of vision transformers for a lightweight skeleton-based action recognition system

Glimpse and Zoom: Spatio-Temporal Focused Dynamic Network for Skeleton-Based Action Recognition

Glimpse and Zoom: Spatio-Temporal Focused Dynamic Network for Skeleton-Based Action Recognition

Joint-Partition Group Attention for skeleton-based action recognition

Skeleton-based action recognition aims to recognize human actions from the coordinates of human joints. By encoding coordinates as joint tokens, previous methods have successfully utilized the self-attention (SA) mechanism to capture the relationship of each pair of joints. However, the attention map generated from the joint Query and joint Key in SA only captures joint-to-joint correlations at a single granularity, which is obviously insufficient for human actions that express semantics in terms of body parts. In this paper, we argue that SA should have a more comprehensive mechanism to capture correlations in joint-to-joint and joint-to-partition patterns for a higher semantic representation of skeleton-based actions. Therefore, we propose Joint-Partition Group Attention (JPGA) to simultaneously capture correlations between joints and body parts of different granularity sizes. Specifically, JPGA integrates the joint tokens according to the joint’s human body partition attributes and produces different body parts tokens (partition-tokens) with different granularities. Then the attention map of JPGA is computed from joint-token and partition-token of different granularity sizes to represent the relationship between joints and body parts. To adaptively partition human body parts at different granularities, we apply the reparameterization trick to adaptively learn the multi-granularity partitioning matrix. Based on JPGA, we construct our Joint-Partition Former (JPFormer), and conduct extensive experiments on NTU-RGB+D, NTU-RRB+D 120, and Northwestern UCLA datasets and achieve state-of-the-art results, which highlights the effectiveness of our design and practice.

Multi-scale sampling attention graph convolutional networks for skeleton-based action recognition

Skeleton-based action recognition has attracted increasing interest in recent years. With the flexibility of modeling long-range dependency of joints, the self-attention module has served as the basic component in skeleton-based action recognition. However, the global receptive field of self-attention is not conducive to the modeling of skeleton locality, and the self-attention model is imbued with less inductive bias, which leads to overfitting. In this paper, we propose an attention graph convolutional network (AGCN) with multi-scale sampling to effectively model the local and global features of the skeleton. Firstly, we propose two extreme sampling strategies for generating and ordering neighboring nodes of root nodes. A local-first sampling method is introduced to construct local graph windows, and a global-first sampling method is proposed to assemble long-range joints for constructing global graph windows. The local-first sampling and global-first sampling introduce more skeleton-specific inductive biases to regularize the model capacity. Secondly, the AGCN combines the self-attention mechanism with graph convolution operation, which alleviates the over-smoothing of graph convolution and preserves the translation invariant. Based on the multi-scale sampling strategy, the AGCN can effectively model the locality and non-locality of the skeleton. Finally, by coupling the aforementioned proposals, we develop a two-pathway model for multi-scale feature fusion. Extensive experiments demonstrate that our model could achieve comparable performance with state-of-the-art works on the NTU RGB+D 60, NTU RGB+D 120, the UAV-HUMAN and NW-UCLA datasets.

A puzzle questions form training for self-supervised skeleton-based action recognition

This paper proposed a novel pretext task to address the skeleton-based video representation for self-supervised action recognition tasks. Instead of exploiting only the whole body, various levels of the skeleton structure (e.g., upper body, lower body, left arm, left leg, right arm, right leg, and torso) are employed to extract essential coarser-grained characteristics. This involves computing statistical representations like motion, orientation, trajectory, and magnitude shift from unlabeled skeleton configurations. Then a learning model is built and trained to yield these statistical representations given the sequence configuration as the input. Our approach is question-driven, where each question acts as a puzzle piece contributing to a deeper understanding of the skeleton joint configuration. It's inspired by the ability of the cognitive system observed in individuals to hypothesize unseen actions. This is accomplished by posing pertinent questions and envisioning plausible scenarios to recognize the actions taking place. The answers to these devised questions are derived from the statistical representation of skeleton configurations. To this end, we made 44 questions designed to encompass the broadest overview to the finest detail. Our experiments on the NTU RGB-D, NW-UCLA, and PKU-MMD datasets demonstrate outstanding results in action recognition, proving the superiority of our approach in learning discriminative characteristics.

Priori separation graph convolution with long-short term temporal modeling for skeleton-based action recognition

Priori separation graph convolution with long-short term temporal modeling for skeleton-based action recognition

Exploring High-Order Skeleton Correlations with Physical and Non-Physical Connection for Action Recognition

Hypergraphs have received widespread attention in modeling complex data correlations due to their superior performance. In recent years, some researchers have used hypergraph structures to characterize complex non-pairwise joints in the human skeleton and model higher-order correlations of the human skeleton. However, traditional methods of constructing hypergraphs based on physical connections ignore the dependencies among non-physically connected joints or bones, and it is difficult to model the correlation among joints or bones that are highly correlated in human action but are physically connected at long distances. To address these issues, we propose a skeleton-based action recognition method for hypergraph learning based on skeleton correlation, which explores the effects of physically and non-physically connected skeleton information on accurate action recognition. Specifically, in this paper, spatio-temporal correlation modeling is performed on the natural connections inherent in humans (physical connections) and the joints or bones that are more dependent but not directly connected (non-physical connection) during human actions. In order to better learn the hypergraph structure, we construct a spatio-temporal hypergraph neural network to extract the higher-order correlations of the human skeleton. In addition, we use an attentional mechanism to compute the attentional weights among different hypergraph features, and adaptively fuse the rich feature information in different hypergraphs. Extensive experiments are conducted on two datasets, NTU-RGB+D 60 and Kinetics-Skeleton, and the results show that compared with the state-of-the-art skeleton-based methods, our proposed method can achieve an optimal level of performance with significant advantages, providing a more accurate environmental perception and action analysis for the development of embodied intelligence.

Static graph convolution with learned temporal and channel-wise graph topology generation for skeleton-based action recognition

Graph convolutional networks (GCNs) are widely used in skeleton-based action recognition. It is known that the graph topology is a vital part in GCNs, and different kinds of graph topologies have been proposed for skeleton-based action recognition, mostly based on a predefined topology and a dynamically learned one. The predefined topology is based on the human intuition for skeleton (the connectivity of joints) and has not been investigated whether it is optimal. In this paper, we focus on investigating this static graph topology and propose to generate a learned static graph topology for skeleton. To be specific, a temporal frame-wise and channel-wise topology-based GCNs (TC-GCNs) are developed, where, instead of using a predefined topology by human, a topology is learned for skeleton-based action recognition. The TC-GCNs consist of generating a temporal frame-wise topology and a channel-wise topology to formulate the relationship of skeleton joints in the temporal dimension and channel dimension, respectively. The proposed method can be integrated with the conventional dynamic topology by replacing the predefined graph topology with our generated one. Experimental results show that our method with learned static graph achieves better performance than the predefined graph and dynamic graph on three widely used benchmarks, namely the NTU-RGB+D, NTU-RGB+D 120 and UAV-Human.

Fusing Higher-Order Features in Graph Neural Networks for Skeleton-Based Action Recognition.

Skeleton sequences are lightweight and compact and thus are ideal candidates for action recognition on edge devices. Recent skeleton-based action recognition methods extract features from 3-D joint coordinates as spatial-temporal cues, using these representations in a graph neural network for feature fusion to boost recognition performance. The use of first- and second-order features, that is, joint and bone representations, has led to high accuracy. Nonetheless, many models are still confused by actions that have similar motion trajectories. To address these issues, we propose fusing higher-order features in the form of angular encoding (AGE) into modern architectures to robustly capture the relationships between joints and body parts. This simple fusion with popular spatial-temporal graph neural networks achieves new state-of-the-art accuracy in two large benchmarks, including NTU60 and NTU120, while employing fewer parameters and reduced run time. Our source code is publicly available at: https://github.com/ZhenyueQin/Angular-Skeleton-Encoding.

Spatial-temporal graph neural ODE networks for skeleton-based action recognition.

In the field of skeleton-based action recognition, accurately recognizing human actions is crucial for applications such as virtual reality and motion analysis. However, this task faces challenges such intraindividual action differences and long-term temporal dependencies. To address these challenges, we propose an innovative model called spatial-temporal graph neural ordinary differential equations (STG-NODE). First, in the data preprocessing stage, the dynamic time warping (DTW) algorithm is used to normalize and calculate 3D skeleton data to facilitate the derivation of customized adjacency matrices for improving the influence of intraindividual action differences. Secondly, a custom ordinary differential equation (ODE) integrator is applied based on the initial conditions of the temporal features, producing a solution function that simulates the dynamic evolution trend of the events of interest. Finally, the outstanding ODE solver is used to numerically solve the time features based on the solution function to increase the influence of long-term dependencies on the recognition accuracy of the model and provide it with a more powerful temporal modeling ability. Through extensive experiments conducted on the NTU RGB+D 60 and Kinetics Skeleton 400 benchmark datasets, we demonstrate the superior performance of STG-NODE in the action recognition domain. The success of the STG-NODE model also provides new ideas and methods for the future development of the action recognition field.

Enhancing action recognition from low-quality skeleton data via part-level knowledge distillation

Skeleton-based action recognition is vital for comprehending human-centric videos and has applications in diverse domains. One of the challenges of skeleton-based action recognition is dealing with low-quality data, such as skeletons that have missing or inaccurate joints. This paper addresses the issue of enhancing action recognition using low-quality skeletons through a general knowledge distillation framework. The proposed framework employs a teacher-student model setup, where a teacher model trained on high-quality skeletons guides the learning of a student model that handles low-quality skeletons. To bridge the gap between heterogeneous high-quality and low-quality skeletons, we present a novel part-based skeleton matching strategy, which exploits shared body parts to facilitate local action pattern learning. An action-specific part matrix is developed to emphasize critical parts for different actions, enabling the student model to distill discriminative part-level knowledge. A novel part-level multi-sample contrastive loss achieves knowledge transfer from multiple high-quality skeletons to low-quality ones, which enables the proposed knowledge distillation framework to include training low-quality skeletons that lack corresponding high-quality matches. Comprehensive experiments conducted on the NTU-RGB+D, Penn Action, and SYSU 3D HOI datasets demonstrate the effectiveness of the proposed knowledge distillation framework.

An overview of Graph Convolutional Networks in skeleton-based action recognition

The field of research is currently focused on human activity recognition. Hence, numerous pertinent literature reviews have expounded upon the multifaceted nature of data, the process of selecting feature vectors, and the advantages and disadvantages of classification networks. Graph Convolutional Networks (GCNs) have demonstrated significant efficacy in the domain of human action recognition. In recent years, with the rapid development of 3D skeleton data collection, a plethora of studies in action recognition based on skeleton data have emerged. Skeleton data consists of three-dimensional coordinates of multiple spatiotemporal skeletal joints, making it an effective representation of kinematics. It can be easily acquired through low-cost depth sensors and also directly extracted from two-dimensional images using video-based pose estimation algorithms, attracting widespread attention. As relational networks continue to evolve, GCNs have been applied to various fields, including human action recognition. GCNs have demonstrated significant advantages in feature extraction from skeleton data. However, using GCNs alone may have various limitations. Therefore, in recent years, many enhancement measures for GCNs have emerged. This review aims to summarize the research achievements of Graph Convolutional Network improvements in the field of human action recognition in recent years. It intends to assist future researchers in quickly organizing their research ideas, facilitating the generation of new results.