Skeleton-based Action Recognition Research Articles

Skeleton-based action recognition through dual-granularity feature fusion with self-adapting graph convolution and multi-scale temporal convolution

Skeleton-based action recognition through dual-granularity feature fusion with self-adapting graph convolution and multi-scale temporal convolution

Positive Sample Mining: Fuzzy Threshold-Based Contrastive Learning for Enhanced Unsupervised Skeleton-Based Action Recognition

Positive Sample Mining: Fuzzy Threshold-Based Contrastive Learning for Enhanced Unsupervised Skeleton-Based Action Recognition

Selectable receptive-field graph convolution networks for skeleton-based action recognition

Selectable receptive-field graph convolution networks for skeleton-based action recognition

From Forest to Zoo: Great Ape Behavior Recognition with ChimpBehave

Abstract This paper addresses the significant challenge of recognizing behaviors in non-human primates, specifically focusing on chimpanzees. Automated behavior recognition is crucial for both conservation efforts and the advancement of behavioral research. However, it is often hindered by the labor-intensive process of manual video annotation. Despite the availability of large-scale animal behavior datasets, effectively applying machine learning models across varied environmental settings remains a critical challenge due to the variability in data collection contexts and the specificity of annotations. In this paper, we introduce ChimpBehave, a novel dataset comprising over 2 h and 20 min of video (approximately 215,000 frames) of zoo-housed chimpanzees, annotated with bounding boxes and fine-grained locomotive behavior labels. Uniquely, ChimpBehave aligns its behavior classes with those in PanAf, an existing dataset collected in distinct visual environments, enabling the study of cross-dataset generalization - where models are trained on one dataset and tested on another with differing data distributions. We benchmark ChimpBehave using state-of-the-art video-based and skeleton-based action recognition models, establishing performance baselines for both within-dataset and cross-dataset evaluations. Our results highlight the strengths and limitations of different model architectures, providing insights into the application of automated behavior recognition across diverse visual settings. The dataset, models, and code can be accessed at: https://github.com/MitchFuchs/ChimpBehave

SHoTGCN: Spatial high-order temporal GCN for skeleton-based action recognition

SHoTGCN: Spatial high-order temporal GCN for skeleton-based action recognition

SG-CLR: Semantic representation-guided contrastive learning for self-supervised skeleton-based action recognition

SG-CLR: Semantic representation-guided contrastive learning for self-supervised skeleton-based action recognition

Local and Global Spatial–Temporal Transformer for skeleton-based action recognition

Local and Global Spatial–Temporal Transformer for skeleton-based action recognition

Asymmetric Context-Guided Adaptive Alignment Network for Skeleton-Based Action Recognition

Asymmetric Context-Guided Adaptive Alignment Network for Skeleton-Based Action Recognition

Action recognition using part and attention enhanced feature fusion

In various studies on skeleton-based action recognition, graph convolution has been widely used to extract crucial skeleton information. However, existing methods often face challenges in effectively capturing the motion relationships between distinct body parts, such as the coordination between the torso and limbs. To address these limitations, this paper constructs a novel Part and Attention enhanced Feature Fusion Network (PAFFNet). First, we construct center-of-gravity dynamics within semantic body parts to enhance motion representation and capture dynamic trends. Additionally, we design Joint-based Feature Fusion (JFF) stream and Bone-based Feature Fusion (BFF) stream to extract and fuse complementary information between constructed local parted and global skeleton features for enhanced motion representation. Furthermore, an Improved Adaptive Graph Convolution (IAGC) block with denoising and attention mechanisms is proposed to prioritize critical semantic features. Experiments conducted on the NTU RGB+D and Kinetics-Skeleton datasets demonstrate the effectiveness of PAFFNet in improving skeleton-based action recognition.

Dual-path spatio-temporal Mamba for skeleton-based action recognition

Dual-path spatio-temporal Mamba for skeleton-based action recognition

Spatial-Temporal Heatmap Masked Autoencoder for Skeleton-Based Action Recognition.

Skeleton representation learning offers substantial advantages for action recognition by encoding intricate motion details and spatial-temporal dependencies among joints. However, fully supervised approaches necessitate large amounts of annotated data, which are often labor-intensive and costly to acquire. In this work, we propose the Spatial-Temporal Heatmap Masked Autoencoder (STH-MAE), a novel self-supervised framework tailored for skeleton-based action recognition. Unlike coordinate-based methods, STH-MAE adopts heatmap volumes as its primary representation, mitigating noise inherent in pose estimation while capitalizing on advances in Vision Transformers. The framework constructs a spatial-temporal heatmap (STH) by aggregating 2D joint heatmaps across both spatial and temporal axes. This STH is partitioned into non-overlapping patches to facilitate local feature learning, with a masking strategy applied to randomly conceal portions of the input. During pre-training, a Vision Transformer-based autoencoder equipped with a lightweight prediction head reconstructs the masked regions, fostering the extraction of robust and transferable skeletal representations. Comprehensive experiments on the NTU RGB+D 60 and NTU RGB+D 120 benchmarks demonstrate the superiority of STH-MAE, achieving state-of-the-art performance under multiple evaluation protocols.

LLMs Encounter Critical Elements Prompts: Semantically Guided Partial Supervision Skeleton-Based Action Recognition

LLMs Encounter Critical Elements Prompts: Semantically Guided Partial Supervision Skeleton-Based Action Recognition

Fast distance-enhanced graph convolutional network for skeleton-based action recognition

Fast distance-enhanced graph convolutional network for skeleton-based action recognition

Dynamic Graph Attention Network for Skeleton-Based Action Recognition

Skeleton-based human action recognition has garnered significant attention for its robustness to background noise and illumination variations. However, existing methods relying on Graph Convolutional Networks (GCNs) and Transformers exhibit inherent limitations: GCNs struggle to model interactions between non-adjacent joints due to predefined skeletal topology, while Transformers accumulate noise through unrestricted global dependency modeling. To address these challenges, we propose a Dynamic Graph Attention Network (DGAN) that dynamically integrates local structural features and global spatiotemporal dependencies. DGAN employs a masked attention mechanism to adaptively adjust node connectivity, forming a dynamic adjacency matrix that extends beyond physical skeletal constraints by selectively incorporating highly correlated joints. Additionally, a node-partition bias strategy is introduced to prioritize attention on collaboratively moving body parts, thereby enhancing discriminative feature extraction. Extensive experiments on the NTU RGB+D 60 and NTU RGB+D 120 datasets validate the effectiveness of DGAN, which outperforms state-of-the-art methods by achieving a balance between local topology preservation and global interaction modeling. Our approach provides a robust framework for skeleton-driven action recognition, demonstrating superior generalization across diverse scenarios.

Efficient human skeleton-based action recognition using a two-stream LSTM framework with limited datasets

ABSTRACT This study proposes a novel human skeleton-based action recognition model (HSARM) designed to achieve high classification accuracy with limited training data, relying solely on image-based inputs. Current human posture recognition (HPR) methods, especially those based on machine learning, often involve extensive preprocessing and feature engineering, which limit their scalability and adaptability. Although deep learning models have shown performance improvements, they typically require large datasets for effective training. To address the challenge of limited data, this study introduces a deep learning framework that combines the Skeleton View-Invariant Transform (SVIT) with a two-stream Long Short-Term Memory (LSTM) architecture, capturing both spatial and temporal features of human motion. The key contribution is the optimization of action recognition with minimal data, achieving high accuracy without wearable sensors or large datasets. Experimental results demonstrate that the proposed model delivers comparable recognition accuracy to existing methods while using significantly fewer training samples. This approach not only advances human posture recognition but also has potential applications in other domains, such as action recognition and gesture tracking, offering a flexible and scalable solution for real-time motion analysis.

GCN-Former: A Method for Action Recognition Using Graph Convolutional Networks and Transformer

Skeleton-based action recognition, which aims to classify human actions through the coordinates of body joints and their connectivity, is a significant research area in computer vision with broad application potential. Although Graph Convolutional Networks (GCNs) have made significant progress in processing skeleton data represented as graphs, their performance is constrained by local receptive fields and fixed joint connection patterns. Recently, researchers have introduced Transformer-based methods to overcome these limitations and better capture long-range dependencies. However, these methods face significant computational resource challenges when attempting to capture the correlations between all joints across all frames. This paper proposes an innovative Spatio-Temporal Graph Convolutional Network: GCN-Former, which aims to enhance model performance in skeleton-based action recognition tasks. The model integrates the Transformer architecture with traditional GCNs, leveraging the Transformer’s powerful capability for handling long-sequence data and the effective capture of spatial dependencies by GCNs. Specifically, this study designs a Transformer Block temporal encoder based on the self-attention mechanism to model long-sequence temporal actions. The temporal encoder can effectively capture long-range dependencies in action sequences while retaining global contextual information in the temporal dimension. In addition, in order to achieve a smooth transition from graph convolutional networks (GCNs) to Transformers, we further develop a contextual temporal attention (CTA) module. These components are aimed at enhancing the understanding of temporal and spatial information within action sequences. Experimental validation on multiple benchmark datasets demonstrates that our approach not only surpasses existing techniques in prediction accuracy, but also has significant performance advantages in handling action recognition tasks involving long time sequences and can more effectively capture and understand long-range dependencies in complex action patterns.

Skeleton-based Action Recognition with Non-linear Dependency Modeling and Hilbert-Schmidt Independence Criterion

Human skeleton-based action recognition has long been an indispensable aspect of artificial intelligence. Current state-of-the-art methods tend to consider only the dependencies between connected skeletal joints, limiting their ability to capture non-linear dependencies between physically distant joints. Moreover, most existing approaches distinguish action classes by estimating the probability density of motion representations, yet the high-dimensional nature of human motions invokes inherent difficulties in accomplishing such measurements. In this paper, we seek to tackle these challenges from two directions: (1) We propose a novel dependency refinement approach that explicitly models dependencies between any pair of joints, effectively transcending the limitations imposed by joint distance. (2) We further propose a framework that utilizes the Hilbert-Schmidt Independence Criterion to differentiate action classes without being affected by data dimensionality, and mathematically derive learning objectives guaranteeing precise recognition. Empirically, our approach sets the state-of-the-art performance on NTU RGB+D, NTU RGB+D 120, and Northwestern-UCLA datasets.

Cross-Scale Spatial Refinement Graph Convolutional Network for Skeleton-Based Action Recognition

In skeleton-based action recognition, abstracting the human body to skeletal representations often results in the loss of crucial information, which may result in misclassification of similar actions. To address this issue, we propose a Cross-scale Spatial Refinement Graph Convolutional Network (CSR-GCN), which aims to improve action recognition accuracy by effectively capturing fine-grained features of skeleton sequences. In detail, we introduce an Attention-based Graph Pooling (AGP) module and a Cross-scale Feature Aggregation (CFA) module. The AGP module uses graph pooling to construct multi-scale skeletal sub-graphs, capturing implicit joint relationships and preserving crucial motion details. It retains global motion information while emphasizing local joint interactions, which enables a better understanding of dynamic changes in complex actions. Furthermore, the CFA module selectively integrates features from different spatial scales, enhancing feature distinctiveness while balancing global motion and local details. This multi-scale refinement of skeletal sequence representations, thereby capturing subtle dynamic changes in actions more precisely and enhancing the ability of the model to recognize and classify complex movement patterns. Finally, we validate the effectiveness of our method on three large-scale datasets, achieving superior accuracy compared to other state-of-the-art methods.

Multidimensional Refinement Graph Convolutional Network With Robust Decouple Loss for Fine-Grained Skeleton-Based Action Recognition.

Graph convolutional networks (GCNs) have been widely used in skeleton-based action recognition. However, existing approaches are limited in fine-grained action recognition due to the similarity of interclass data. Moreover, the noisy data from pose extraction increase the challenge of fine-grained recognition. In this work, we propose a flexible attention block called channel-variable spatial-temporal attention (CVSTA) to enhance the discriminative power of spatial-temporal joints and obtain a more compact intraclass feature distribution. Based on CVSTA, we construct a multidimensional refinement GCN (MDR-GCN) that can improve the discrimination among channel-, joint-, and frame-level features for fine-grained actions. Furthermore, we propose a robust decouple loss (RDL) that significantly boosts the effect of the CVSTA and reduces the impact of noise. The proposed method combining MDR-GCN with RDL outperforms the known state-of-the-art skeleton-based approaches on fine-grained datasets, FineGym99 and FSD-10, and also on the coarse NTU-RGB + D 120 dataset and NTU-RGB + D X-view version. Our code is publicly available at https://github.com/dingyn-Reno/MDR-GCN.

Selective directed graph convolutional network for skeleton-based action recognition

Selective directed graph convolutional network for skeleton-based action recognition