Network For Action Recognition Research Articles

Multi-stream P&U adaptive graph convolutional networks for skeleton-based action recognition

Multi-stream P&U adaptive graph convolutional networks for skeleton-based action recognition

A Multi-Scale Video Longformer Network for Action Recognition

Action recognition has found extensive applications in fields such as video classification and security monitoring. However, existing action recognition methods, such as those based on 3D convolutional neural networks, often struggle to capture comprehensive global information. Meanwhile, transformer-based approaches face challenges associated with excessively high computational complexity. We introduce a Multi-Scale Video Longformer network (MSVL), built upon the 3D Longformer architecture featuring a “local attention + global features” attention mechanism, enabling us to reduce computational complexity while preserving global modeling capabilities. Specifically, MSVL gradually reduces the video feature resolution and increases the feature dimensions across four stages. In the lower layers of the network (stage 1, stage 2), we leverage local window attention to alleviate local redundancy and computational demands. Concurrently, global tokens are employed to retain global features. In the higher layers of the network (stage 3, stage 4), this local window attention evolves into a dense computation mechanism, enhancing overall performance. Finally, extensive experiments are conducted on UCF101 (97.6%), HMDB51 (72.9%), and the assembly action dataset (100.0%), demonstrating the effectiveness and efficiency of the MSVL.

Graph Neural Networks for Skeleton-based action recognition

One of the important directions of the application of artificial intelligence based on human bone behavior recognition is also a research hotspot in the field of computer vision in recent years. Human image video not only contains complex backgrounds, but also uncertain factors such as changes in illumination and changes in the appearance of the human body, which makes behavior recognition based on image videos have certain limitations. Compared with image video, human skeleton video can well overcome the influence of these uncertain factors, so be- havior recognition based on human skeleton has received more and more attention. The human skeleton sequence not only contains the tempo- ral features, but also the spatial structure features of the human body. How to effectively extract the discriminative spatial and temporal fea- tures from the human skeleton sequence is a problem to be solved. In recent years, many methods have been applied to bone-based behavior recognition, such as Recurrent Neural Network (RNN), Convolutional Neural Network (CNN) and Graph Neural Network (GCN). This article will introduce the content and characteristics of these three methods one by one. , And conduct a comparative analysis on it.

Fourier analysis on robustness of graph convolutional neural networks for skeleton-based action recognition

Using Fourier analysis, we explore the robustness and vulnerability of graph convolutional neural networks (GCNs) for skeleton-based action recognition. We adopt a joint Fourier transform (JFT), a combination of the graph Fourier transform (GFT) and the discrete Fourier transform (DFT), to examine the robustness of adversarially-trained GCNs against adversarial attacks and common corruptions. Experimental results with the NTU RGB+D dataset reveal that adversarial training does not introduce a robustness trade-off between adversarial attacks and low-frequency perturbations, which typically occurs during image classification based on convolutional neural networks. This finding indicates that adversarial training is a practical approach to enhancing robustness against adversarial attacks and common corruptions in skeleton-based action recognition. Furthermore, we find that the Fourier approach cannot explain vulnerability against skeletal part occlusion corruption, which highlights its limitations. These findings extend our understanding of the robustness of GCNs, potentially guiding the development of more robust learning methods for skeleton-based action recognition.

Formula omitted]-NN attention-based video vision transformer for action recognition

Action Recognition aims to understand human behavior and predict a label for each action. Recently, Vision Transformer (ViT) has achieved remarkable performance on action recognition, which models the long sequences token over spatial and temporal index in a video. The fully-connected self-attention layer is the fundamental key in the vanilla Transformer. However, the redundant architecture of the vision Transformer model ignores the locality of video frame patches, which involves non-informative tokens and potentially leads to increased computational complexity. To solve this problem, we propose a k-NN attention-based Video Vision Transformer (k-ViViT) network for action recognition. We adopt k-NN attention to Video Vision Transformer (ViViT) instead of original self-attention, which can optimize the training process and neglect the irrelevant or noisy tokens in the input sequence. We conduct experiments on the UCF101 and HMDB51 datasets to verify the effectiveness of our model. The experimental results illustrate that the proposed k-ViViT achieves superior accuracy compared to several state-of-the-art models on these action recognition datasets.

Action Detection for Wildlife Monitoring with Camera Traps Based on Segmentation with Filtering of Tracklets (SWIFT) and Mask-Guided Action Recognition (MAROON)

Behavioral analysis of animals in the wild plays an important role for ecological research and conservation and has been mostly performed by researchers. We introduce an action detection approach that automates this process by detecting animals and performing action recognition on the detected animals in camera trap videos. Our action detection approach is based on SWIFT (segmentation with filtering of tracklets), which we have already shown to successfully detect and track animals in wildlife videos, and MAROON (mask-guided action recognition), an action recognition network that we are introducing here. The basic ideas of MAROON are the exploitation of the instance masks detected by SWIFT and a triple-stream network. The instance masks enable more accurate action recognition, especially if multiple animals appear in a video at the same time. The triple-stream approach extracts features for the motion and appearance of the animal. We evaluate the quality of our action recognition on two self-generated datasets, from an animal enclosure and from the wild. These datasets contain videos of red deer, fallow deer and roe deer, recorded both during the day and night. MAROON improves the action recognition accuracy compared to other state-of-the-art approaches by an average of 10 percentage points on all analyzed datasets and achieves an accuracy of 69.16% on the Rolandseck Daylight dataset, in which 11 different action classes occur. Our action detection system makes it possible todrasticallyreduce the manual work of ecologists and at the same time gain new insights through standardized results.

DFN: A deep fusion network for flexible single and multi-modal action recognition

Multi-modal action recognition methods can be generally classified into two categories: (1) fusing multi-modal features with simple concatenation or fusing the classification scores of individual modalities without considering the interaction among the multi-modalities; (2) using one of the modalities as privileged information in training to boost the recognition on the other modalities in inference. The former approach usually is not able to deal with the cases where one of the modalities is missing. In the latter, the trained classifier does not work on the privileged modality. To address these shortcomings, this paper presents a novel end-to-end trainable deep fusion network (DFN) that is able to improve the performance not only in the cases where all modalities are available and also in the cases where there is a missing modality. The DFN is simple yet effective with the capability of retrieving an estimation of one modality by using another modality through a Multilayer Perceptron (MLP). In order to better preserve structure information, the DFN first maps the individual modality features to a high dimensional Kronecker-product space and subsequently learns a low-dimensional discriminative space for classification. The effectiveness of the proposed DFN has been verified on three benchmark datasets: the large NTU RGB+D, UTD-MHAD, and SYSU-3D datasets and it has achieved state-of-the-art results.

Channel Attention-Based Approach with Autoencoder Network for Human Action Recognition in Low-Resolution Frames

Action recognition (AR) has many applications, including surveillance, health/disabilities care, man-machine interactions, video-content-based monitoring, and activity recognition. Because human action videos contain a large number of frames, implemented models must minimize computation by reducing the number, size, and resolution of frames. We propose an improved method for detecting human actions in low-size and low-resolution videos by employing convolutional neural networks (CNNs) with channel attention mechanisms (CAMs) and autoencoders (AEs). By enhancing blocks with more representative features, convolutional layers extract discriminating features from various networks. Additionally, we use random sampling of frames before main processing to improve accuracy while employing less data. The goal is to increase performance while overcoming challenges such as overfitting, computational complexity, and uncertainty by utilizing CNN-CAM and AE. Identifying patterns and features associated with selective high-level performance is the next step. To validate the method, low-resolution and low-size video frames were used in the UCF50, UCF101, and HMDB51 datasets. Additionally, the algorithm has relatively minimal computational complexity. Consequently, the proposed method performs satisfactorily compared to other similar methods. It has accuracy estimates of 77.29, 98.87, and 97.16%, respectively, for HMDB51, UCF50, and UCF101 datasets. These results indicate that the method can effectively classify human actions. Furthermore, the proposed method can be used as a processing model for low-resolution and low-size video frames.

Spatio-temporal-based multi-level aggregation network for physical action recognition

This paper introduces spatio-temporal-based multi-level aggregation network (ST-MANet) for action recognition. It utilizes the correlations between different spatial positions and the correlations between different temporal positions on the feature map to explore long-range spatial and temporal dependencies, respectively, generating the spatial and temporal attention map that assigns different weights to features at different spatial and temporal locations. Additionally, a multi-scale approach is introduced, proposing a multi-scale behavior recognition framework that models various visual rhythms while capturing multi-scale spatiotemporal information. A spatial diversity constraint is then proposed, encouraging spatial attention maps at different scales to focus on distinct areas. This ensures a greater emphasis on spatial information unique to each scale, thereby incorporating more diverse spatial information into multi-scale features. Finally, ST-MANet is compared with existing approaches, demonstrating high accuracy on the three datasets.

Attention-Based AdaptSepCX Network for Effective Student Action Recognition in Online Learning

In the realm of online learning and distance education, the issue of inadequate supervision looms large, posing a significant obstacle. This paper delves into the challenges posed by the lack of supervision in online learning environments and proposes an innovative solution to understand and recognize students’ behaviors. This study's primary objective is to detect and recognize students’ actions in images captured through webcam. This task distinguishes itself from the well-established video-based student action recognition domain, which relies on temporal cues. Recognizing student actions from images intensifies the complexity of the problem. To meet this challenge, a novel deep learning model named AdaptSepCX Attention, specifically designed for student action recognition in online learning environments, is introduced. The proposed method exhibits exceptional performance with 92.73% validation accuracy on the Student Online Action Image dataset (SOAId), a carefully curated collection comprising 2029 student-centric images. The proposed model outperforms well-established models such as DenseNet121, NASNet Mobile, Con-vXNet, DELVS1 and MobileNetV2 in student action recognition. Action recognition for students has broader implications beyond the online classroom. It has the potential to revolutionize educational technology, making online learning more interactive and engaging. Enabling machines to understand and respond to student actions enhances education, personalizes learning, and supports students’ academic success and well-being. This research enhances the understanding of student involvement in online learning and offers an effective solution for recognizing actions from images.

Self-Adaptive Graph With Nonlocal Attention Network for Skeleton-Based Action Recognition.

Graph convolutional networks (GCNs) have achieved encouraging progress in modeling human body skeletons as spatial-temporal graphs. However, existing methods still suffer from two inherent drawbacks. Firstly, these models process the input data based on the physical structure of the human body, which leads to some latent correlations among joints being ignored. Furthermore, the key temporal relationships between nonadjacent frames are overlooked, preventing to fully learn the changes of the body joints along the temporal dimension. To address these issues, we propose an innovative spatial-temporal model by introducing a self-adaptive GCN (SAGCN) with global attention network, collectively termed SAGGAN. Specifically, the SAGCN module is proposed to construct two additional dynamic topological graphs to learn the common characteristics of all data and represent a unique pattern for each sample, respectively. Meanwhile, the global attention module (spatial attention (SA) and temporal attention (TA) modules) is designed to extract the global connections between different joints in a single frame and model temporal relationships between adjacent and nonadjacent frames in temporal sequences. In this manner, our network can capture richer features of actions for accurate action recognition and overcome the defect of the standard graph convolution. Extensive experiments on three benchmark datasets (NTU-60, NTU-120, and Kinetics) have demonstrated the superiority of our proposed method.

Multi-Scale Adaptive Graph Convolution Network for Skeleton-Based Action Recognition

Multi-Scale Adaptive Graph Convolution Network for Skeleton-Based Action Recognition

Cross-Modal Contrastive Learning Network for Few-Shot Action Recognition.

Few-shot action recognition aims to recognize new unseen categories with only a few labeled samples of each class. However, it still suffers from the limitation of inadequate data, which easily leads to the overfitting and low-generalization problems. Therefore, we propose a cross-modal contrastive learning network (CCLN), consisting of an adversarial branch and a contrastive branch, to perform effective few-shot action recognition. In the adversarial branch, we elaborately design a prototypical generative adversarial network (PGAN) to obtain synthesized samples for increasing training samples, which can mitigate the data scarcity problem and thereby alleviate the overfitting problem. When the training samples are limited, the obtained visual features are usually suboptimal for video understanding as they lack discriminative information. To address this issue, in the contrastive branch, we propose a cross-modal contrastive learning module (CCLM) to obtain discriminative feature representations of samples with the help of semantic information, which can enable the network to enhance the feature learning ability at the class-level. Moreover, since videos contain crucial sequences and ordering information, thus we introduce a spatial-temporal enhancement module (SEM) to model the spatial context within video frames and the temporal context across video frames. The experimental results show that the proposed CCLN outperforms the state-of-the-art few-shot action recognition methods on four challenging benchmarks, including Kinetics, UCF101, HMDB51 and SSv2.

Spatial-Temporal Adaptive Metric Learning Network for One-Shot Skeleton-Based Action Recognition

Spatial-Temporal Adaptive Metric Learning Network for One-Shot Skeleton-Based Action Recognition

Hierarchical Aggregated Graph Neural Network for Skeleton-based Action Recognition

Hierarchical Aggregated Graph Neural Network for Skeleton-based Action Recognition

Adversarial Attention Networks for Early Action Recognition

Adversarial Attention Networks for Early Action Recognition

Design of innovative teaching methods for classical dance based on group intelligence

Abstract Compared with image records, 3D dance scores can realize the demonstration and practice of classical dance teaching more accurately and efficiently. In this paper, the Li group skeletal representation model is used for Li algebra space mapping processing to learn action feature data, classify them, and complete classical dance action feature extraction. Then, the residual network model is used as the backbone network to construct a two-stream spatiotemporal residual network for action recognition by using two different temporal spans and fusing the results of the two-channel networks through side connections. Finally, the group intelligence algorithm group intelligence algorithm ACOR-SGD is designed to optimize the parameters of the convolutional neural network, and then the dance score generation model is integrated into the design of the innovative teaching method of classical dance for the application of the professional teaching of classical dance in colleges and universities. There is basically no significant difference between the pre-test scores of the experimental and control classes. After one semester of applying the innovative teaching method based on the generation of dance scores, the scores of the experimental class in the five items of basic pace mastery, movement fluency, stage presence, technical quality, and coordination of the music rhythm were improved by 4.6, 4, 4.28, 3.63, and 4.72 respectively, and the total score of 81.54 was much higher than that of the control class of 60.57 points posttest scores, and all were significant at the 0.01 level. This study explores innovatively and provides an effective path for integrating dance teaching with cutting-edge information technology, thus improving the quality and efficiency of dance teaching.

Temporal Shift Module-based Vision Transformer network for action recognition

Temporal Shift Module-based Vision Transformer network for action recognition

Converting Artificial Neural Networks to Ultra-Low-Latency Spiking Neural Networks for Action Recognition

Converting Artificial Neural Networks to Ultra-Low-Latency Spiking Neural Networks for Action Recognition

Multi-View Time-Series Hypergraph Neural Network for Action Recognition.

Recently, action recognition has attracted considerable attention in the field of computer vision. In dynamic circumstances and complicated backgrounds, there are some problems, such as object occlusion, insufficient light, and weak correlation of human body joints, resulting in skeleton-based human action recognition accuracy being very low. To address this issue, we propose a Multi-View Time-Series Hypergraph Neural Network (MV-TSHGNN) method. The framework is composed of two main parts: the construction of a multi-view time-series hypergraph structure and the learning process of multi-view time-series hypergraph convolutions. Specifically, given the multi-view video sequence frames, we first extract the joint features of actions from different views. Then, limb components and adjacent joints spatial hypergraphs based on the joints of different views at the same time are constructed respectively, temporal hypergraphs are constructed joints of the same view at continuous times, which are established high-order semantic relationships and cooperatively generate complementary action features. After that, we design a multi-view time-series hypergraph neural network to efficiently learn the features of spatial and temporal hypergraphs, and effectively improve the accuracy of skeleton-based action recognition. To evaluate the effectiveness and efficiency of MV-TSHGNN, we conduct experiments on NTU RGB+D, NTU RGB+D 120 and imitating traffic police gestures datasets. The experimental results indicate that our proposed method model achieves the new state-of-the-art performance.