Action Recognition Method Research Articles

Advancing Incremental Few-Shot Video Action Recognition with Cluster Compression and Generative Separation

Few-Shot Class Incremental Learning (FSCIL) is a trending topic in deep learning, addressing the need for models to incrementally learn novel classes, particularly in real-world scenarios where continuously emerging classes come with limited labeled samples. However, the majority of FSCIL research has been dedicated to image classification and object recognition tasks, with limited attention given to video action classification. In this paper, we present a new Cluster Compression and Generative Separation (CCGS) method for Incremental Few-Shot Video Action Recognition (iFSVAR), which introduces contrastive learning to boost the degree of class separation in the base session. Simultaneously, it creates numerous fine-grained classes with diverse semantics, effectively filling the unallocated representation space. Experimental results on UCF101, Kinetics, and Something-Something-V2[Formula: see text]demonstrate the effectiveness of the framework.

Depth Video-Based Secondary Action Recognition in Vehicles via Convolutional Neural Network and Bidirectional Long Short-Term Memory with Spatial Enhanced Attention Mechanism

Secondary actions in vehicles are activities that drivers engage in while driving that are not directly related to the primary task of operating the vehicle. Secondary Action Recognition (SAR) in drivers is vital for enhancing road safety and minimizing accidents related to distracted driving. It also plays an important part in modern car driving systems such as Advanced Driving Assistance Systems (ADASs), as it helps identify distractions and predict the driver’s intent. Traditional methods of action recognition in vehicles mostly rely on RGB videos, which can be significantly impacted by external conditions such as low light levels. In this research, we introduce a novel method for SAR. Our approach utilizes depth-video data obtained from a depth sensor located in a vehicle. Our methodology leverages the Convolutional Neural Network (CNN), which is enhanced by the Spatial Enhanced Attention Mechanism (SEAM) and combined with Bidirectional Long Short-Term Memory (Bi-LSTM) networks. This method significantly enhances action recognition ability in depth videos by improving both the spatial and temporal aspects. We conduct experiments using K-fold cross validation, and the experimental results show that on the public benchmark dataset Drive&Act, our proposed method shows significant improvement in SAR compared to the state-of-the-art methods, reaching an accuracy of about 84% in SAR in depth videos.

Wearable Activity Recognition for Advancing Elderly Care with Modified Transformer Model

Technological devices such as smartphones can be utilized in tracking various human activities or movements through built-in accelerometers and gyroscopes. Data obtained from these inertial sensors can be utilized in various applications to assist people, including healthcare, human-computer interaction, and sports. As a result, further developments in the effective classification of time series data through machine and deep learning is highly valued and actively pursued. In this study, the transformer model, a deep learning architecture designed for sequential data such as natural language processing (NLP), has been utilized for analysis of time-series motion readings from wearable accelerometers. The transformer model in this study has been refined by incorporating a Long Short-Term Memory (LSTM) recurrent neural network (RNN) architecture. By leveraging the HAR70+ dataset with a wide range of activities, the modified transformer model in this study obtained a best accuracy of 95.85%, demonstrating that it can match the performance of state-of-the-art wearable activity recognition methods using Deep Neural Networks (DNN) and LSTM. Hence, the findings presented in this study suggest the future relevance of improved transformer or deep learning models to enhance the quality of life for seniors.

The Application of Deep Learning in Sports Training

In response to the problems of overfitting, susceptibility to interference information, and insufficient feature expression ability in existing deep learning methods for sports action recognition, the author proposes a deep learning sports action recognition method that integrates attention mechanism. This method proposes a video data augmentation algorithm in data preprocessing to reduce the risk of model overfitting. Then, during the video frame sampling process, the existing sampling algorithms are improved to effectively suppress the influence of interference information. In the special section, the network residual consolidation is proposed to improve the feature extraction capacity of the structure. The Long Term Time Transform (LSTM) network is used to solve the problem of the global correlation of the spatial correlation, and the classification algorithm is achieved by Softmax. and the classification algorithm is proposed. The experimental results show that the recognition rates of this method on UCFYouTube, KTH, and HMDB-51 data are 96.73%, 98.07%, and 64.82%, respectively.

Identifying Truck Activity from Global Positioning System Data Based on Satellite Imagery

Road freight transportation plays a crucial role in fostering economic growth and enhancing logistics efficiency. Existing research primarily focuses on analyzing freight truck activity characteristics based on global positioning system (GPS) data of freight truck trajectories. Those studies proposed a variety of identification methods from different perspectives, but they mainly rely on rule-based methods and subjectively defined key threshold parameters, ignoring the complexity of truck freight activities and relying on long sequences of trajectory data. To bridge this gap, we propose the satellite image augmented truck activity recognition method, a structured framework that utilizes satellite imagery to identify truck activities from GPS data. Firstly, we categorize truck activities into three types: loading/unloading, resting, and driving. We extract geographic entities from the area of interest data to create the satellite image dataset of truck activity. Secondly, we evaluate the recognition accuracy of benchmark image classification models on our dataset, and ResNet50 achieves 98.04% accuracy on the test set. Thirdly, we apply the image classification model to identify truck activity from GPS trajectory data and use the recognition results of neighboring points to improve the identification results. The framework delineated in this paper entails the extraction of truck activities from GPS data at the satellite image level. Our approach holds promise in facilitating logistics operation fleets and traffic management authorities to maintain near real-time surveillance of the truck status, thereby fostering enhancements in both the efficiency and safety of freight transport.

A novel rehabilitation action recognition approach using AF-LiteFormer from millimeter wave radar

Abstract Rehabilitation action recognition is a hot research topic in the medical field, which serves as the foundation for achieving remote healthcare, unsupervised exercise, intelligent home healthcare, and possesses extensive application value. Currently, vision-based action recognition methods are susceptible to limitations imposed by factors such as range of motion and environmental lighting during human motion capture. Due to its ability to effectively protect patient privacy and its immunity to lighting conditions, this paper proposed a millimeter-wave radar-based rehabilitation action recognition system, AF-LiteFormer. Firstly, EfficientFormerV2 is employed as the baseline, then a Lite-MSLA-FFN block-and-layer is designed to replace MHSA in EfficientFormerV2, which improves the diversity of attention and reduces computational complexity. Meanwhile, a Lite-Subsample block-and-layer is designed to replace the dual-path downsampling part that composed of attention downsampling and stride attention to realize global modeling and multi-scale learning.
Secondly, the iterative attention feature fusion (iAFF) mechanism is introduced to improve the recognition accuracy of rehabilitation actions. Finally, the effectiveness of the AF-LiteFormer model is validated on a self-collected rehabilitation action dataset and a publicly available micro-Doppler dataset. Experimental results show that the overall performance of the AF-LiteFormer model is better than the State-of-the-Art model (SOTA), the recognition accuracy of rehabilitation actions is as high as 99.7%, and it has strong generalization ability.

Tennis action recognition and evaluation with inertial measurement unit and SVM

Action recognition in tennis plays a crucial role for athletes and coaches, aiding in understanding and evaluating the players' skill levels to formulate more effective training plans and tactical strategies. To enhance the recognition and grading of tennis player actions, this study introduces the use of inertial measurement units and flexible resistive sensors for data collection. An improved Support Vector Machine is employed for data classification to achieve efficient action recognition. The results demonstrated that the proposed classification algorithm achieved an average accuracy of 95.35 % in recognizing actions of elite athletes, with the highest accuracy (96.38 %) observed in forehand strokes. In the case of sub-elite athletes, the algorithm achieved an impressive average accuracy of 97.67 %. For amateur enthusiasts, the algorithm exhibited an average accuracy of 94.08 %. Furthermore, elite athletes exhibited larger peak values in the three-axis acceleration waveform during ball striking. Specifically, the absolute peak value of acceleration in the Y-axis for elite athletes reached 78 m/s², representing an increase of 39 m/s² and 8 m/s² compared to the other two levels of athletes, respectively. Additionally, on the X and Z axes, elite athletes' acceleration peak values reached 59 m/s² and 78 m/s², significantly higher than those of sub-elite athletes and amateur enthusiasts. Moreover, the acceleration curves of elite athletes demonstrated a higher overall regularity. These findings indicate that the proposed action recognition method has a significant impact on recognition and evaluation, providing valuable insights for action recognition and assessment across various domains and advancing the application of artificial intelligence technology in the field of sports.

MAFormer: A cross-channel spatio-temporal feature aggregation method for human action recognition

Human action recognition has been widely used in fields such as human–computer interaction and virtual reality. Despite significant progress, existing approaches still struggle with effectively integrating hierarchical information and processing data beyond a certain frame count. To address these challenges, we introduce the Multi-AxisFormer (MAFormer) model, which is organized in terms of spatial, temporal, and channel dimensions of the action sequence, thereby enhancing the model’s understanding of correlations and intricate structures among and within features. Drawing on the Transformer architecture, we propose the Cross-channel Spatio-temporal Aggregation (CSA) structure for more refined feature extraction and the Multi-Axis Attention (MAA) module for more comprehensive feature aggregation. Moreover, the integration of Rotary Position Embedding (RoPE) boosts the model’s extrapolation and generalization abilities. MAFormer surpasses the known state-of-the-art on multiple skeleton-based action recognition benchmarks with the accuracy of 93.2% on NTU RGB+D 60 cross-subject split, 89.9% on NTU RGB+D 120 cross-subject split, and 97.2% on N-UCLA, offering a novel paradigm for hierarchical modeling in human action recognition.

ABNet: AI-Empowered Abnormal Action Recognition Method for Laboratory Mouse Behavior.

The automatic recognition and quantitative analysis of abnormal behavior in mice play a crucial role in behavioral observation experiments in neuroscience, pharmacology, and toxicology. Due to the challenging definition of abnormal behavior and difficulty in collecting training samples, directly applying behavior recognition methods to identify abnormal behavior is often infeasible. This paper proposes ABNet, an AI-empowered abnormal action recognition approach for mice. ABNet utilizes an enhanced Spatio-Temporal Graph Convolutional Network (ST-GCN) as an encoder; ST-GCN combines graph convolution and temporal convolution to efficiently capture and analyze spatio-temporal dynamic features in graph-structured data, making it suitable for complex tasks such as action recognition and traffic prediction. ABNet trains the encoding network with normal behavior samples, then employs unsupervised clustering to identify abnormal behavior in mice. Compared to the original ST-GCN network, the method significantly enhances the capabilities of feature extraction and encoding. We conduct comprehensive experiments on the Kinetics-Skeleton dataset and the mouse behavior dataset to evaluate and validate the performance of ABNet in behavior recognition and abnormal motion detection. In the behavior recognition experiments conducted on the Kinetics-Skeleton dataset, ABNet achieves an accuracy of 32.7% for the top one and 55.2% for the top five. Moreover, in the abnormal behavior analysis experiments conducted on the mouse behavior dataset, ABNet achieves an average accuracy of 83.1%.

A Novel Active Learning Framework for Cross-Subject Human Activity Recognition from Surface Electromyography.

Wearable sensor-based human activity recognition (HAR) methods hold considerable promise for upper-level control in exoskeleton systems. However, such methods tend to overlook the critical role of data quality and still encounter challenges in cross-subject adaptation. To address this, we propose an active learning framework that integrates the relation network architecture with data sampling techniques. Initially, target data are used to fine tune two auxiliary classifiers of the pre-trained model, thereby establishing subject-specific classification boundaries. Subsequently, we assess the significance of the target data based on classifier discrepancy and partition the data into sample and template sets. Finally, the sampled data and a category clustering algorithm are employed to tune model parameters and optimize template data distribution, respectively. This approach facilitates the adaptation of the model to the target subject, enhancing both accuracy and generalizability. To evaluate the effectiveness of the proposed adaptation framework, we conducted evaluation experiments on a public dataset and a self-constructed electromyography (EMG) dataset. Experimental results demonstrate that our method outperforms the compared methods across all three statistical metrics. Furthermore, ablation experiments highlight the necessity of data screening. Our work underscores the practical feasibility of implementing user-independent HAR methods in exoskeleton control systems.

LG-AKD: Application of a lightweight GCN model based on adversarial knowledge distillation to skeleton action recognition

Human action recognition, a pivotal topic in computer vision, is a highly complex and challenging task. It requires the analysis of not only spatial dependencies of targets but also temporal changes in these targets. In recent decades, the advancement of deep learning has led to the development of numerous action recognition methods based on deep neural networks. Given that the skeleton points of the human body can be treated as a graph structure, graph neural networks (GNNs) have emerged as an effective tool for modeling such data, garnering significant interest from researchers. This paper aims to address the issue of low test speed caused by over-complicated deep graph convolutional models. To achieve this, we compress the network structure using knowledge distillation from a teacher-student architecture, leading to a compact and lightweight student GNN. To enhance the model’s robustness and generalization capabilities, we introduce a data augmentation mechanism that generates diverse action sequences while maintaining consistent behavior labels, thereby providing a more comprehensive learning basis for the model. The proposed model integrates three distinct knowledge learning paths: teacher networks, original datasets, and derived data. The fusion of knowledge distillation and data augmentation enables lightweight student networks to outperform their teacher networks in terms of both performance and efficiency. Experimental results demonstrate the efficacy of our approach in the context of skeleton-based human action recognition, highlighting its potential to simplify state-of-the-art models while enhancing their performance.

Language-guided temporal primitive modeling for skeleton-based action recognition

Human actions describe the complex body dynamics, which are defined by an ordered set of sub-action primitives. Existing methods in skeleton-based action recognition primarily focus on designing various networks to learn the entire action representation, in alignment with the action label. However, training the heterogeneous action as a whole increases the burden on the network and is not conducive to learning the temporal structure of the action. In this paper, the traditional action label is extended into the ordered primitive language description by employing a large language model, which not only unveils the temporal structure of the action but also enriches its connotation. Based on this, we propose a language-guided skeleton representation learning network (LGS-Net) that leverages the language description to guide the skeleton feature learning from the ordered primitive and global action levels. Especially, the primitive guidance aligns the features of skeleton clips with those of the ordered primitive descriptions while preserving temporal order, which promotes the network to model the temporal primitive and capture the internal structure within the action from skeletons. To enhance the cross-modal alignment performance, we develop an innovative temporal alignment loss supplemented with diversity and sparsity regularization terms to generate the discriminative multi-modal representation. Evaluated on three benchmark datasets, NTU-60, NTU-120 and N-UCLA, the proposed LGS-Net achieves comparable results to state-of-the-art methods, which proves the effectiveness of the language-guided learning mechanism.

FineTea: A Novel Fine-Grained Action Recognition Video Dataset for Tea Ceremony Actions.

Methods based on deep learning have achieved great success in the field of video action recognition. When these methods are applied to real-world scenarios that require fine-grained analysis of actions, such as being tested on a tea ceremony, limitations may arise. To promote the development of fine-grained action recognition, a fine-grained video action dataset is constructed by collecting videos of tea ceremony actions. This dataset includes 2745 video clips. By using a hierarchical fine-grained action classification approach, these clips are divided into 9 basic action classes and 31 fine-grained action subclasses. To better establish a fine-grained temporal model for tea ceremony actions, a method named TSM-ConvNeXt is proposed that integrates a TSM into the high-performance convolutional neural network ConvNeXt. Compared to a baseline method using ResNet50, the experimental performance of TSM-ConvNeXt is improved by 7.31%. Furthermore, compared with the state-of-the-art methods for action recognition on the FineTea and Diving48 datasets, the proposed approach achieves the best experimental results. The FineTea dataset is publicly available.

A skeleton-based assembly action recognition method with feature fusion for human-robot collaborative assembly

Human-robot collaborative assembly (HRCA) is one of the current trends of intelligent manufacturing, and assembly action recognition is the basis of and the key to HRCA. A multi-scale and multi-stream graph convolutional network (2MSGCN) for assembly action recognition is proposed in this paper. 2MSGCN takes the temporal skeleton sample as input and outputs the class of the assembly action to which the sample belongs. RGBD images of the operator performing the assembly actions are captured by three RGBD cameras mounted at different viewpoints and pre-processed to generate the complete human skeleton. A multi-scale and multi-stream (2MS) mechanism and a feature fusion mechanism are proposed to improve the recognition accuracy of 2MSGCN. The 2MS mechanism is designed to input the skeleton data to 2MSGCN in the form of a joint stream, a bone stream and a motion stream, while the joint stream further generates two sets of input with rough scales to represent features in higher dimensional human skeleton, which obtains information of different scales and streams in temporal skeleton samples. And the feature fusion mechanism enables the fused feature to retain the information of the sub-feature while incorporating union information between the sub-features. Also, the improved convolution operation based on Ghost module is introduced to the 2MSGCN to reduce the number of the parameters and floating-point operations (FLOPs) and improve the real-time performance. Considering that there will be transitional actions when the operator switches between assembly actions in the continuous assembly process, a transitional action classification (TAC) method is proposed to distinguish the transitional actions from the assembly actions. Experiments on the public dataset NTU RGB+D 60 (NTU 60) and a self-built assembly action dataset indicate that the proposed 2MSGCN outperforms the mainstream models in recognition accuracy and real-time performance.

Data augmentation aided excavator activity recognition using deep convolutional conditional generative adversarial networks

Excavator activity recognition is a crucial task with significant practical applications such as assessing production efficiency, injury prevention, and enabling intelligent control. However, implementing deep learning-based activity recognition methods presents considerable challenges due to the requirement of large training datasets. And sensor-based activity recognition demands more convenient and reliable sensors. This paper proposes a generalized data augmentation framework for excavator activity recognition. Main pump pressure and flow signals are generated using Deep Convolutional Conditional Generative Adversarial Networks (DC-CGAN). The proposed framework integrates Long Short-Term Memory (LSTM) and Convolutional Neural Network (CNN) architectures for activity recognition. Results indicate that the framework outperforms traditional data augmentation methods regarding the quality of generated data and shallow and single deep networks regarding model accuracy and generalization. Additionally, the framework was implemented on three sizes of excavators and a wheel loader, demonstrating excellent versatility.

Explore human parsing modality for action recognition

AbstractMultimodal‐based action recognition methods have achieved high success using pose and RGB modality. However, skeletons sequences lack appearance depiction and RGB images suffer irrelevant noise due to modality limitations. To address this, the authors introduce human parsing feature map as a novel modality, since it can selectively retain effective semantic features of the body parts while filtering out most irrelevant noise. The authors propose a new dual‐branch framework called ensemble human parsing and pose network (EPP‐Net), which is the first to leverage both skeletons and human parsing modalities for action recognition. The first human pose branch feeds robust skeletons in the graph convolutional network to model pose features, while the second human parsing branch also leverages depictive parsing feature maps to model parsing features via convolutional backbones. The two high‐level features will be effectively combined through a late fusion strategy for better action recognition. Extensive experiments on NTU RGB + D and NTU RGB + D 120 benchmarks consistently verify the effectiveness of our proposed EPP‐Net, which outperforms the existing action recognition methods. Our code is available at https://github.com/liujf69/EPP‐Net‐Action.

Exposing Data Leakage in Wi-Fi CSI-Based Human Action Recognition: A Critical Analysis

Wi-Fi channel state information (CSI)-based human action recognition systems have garnered significant interest for their non-intrusive monitoring capabilities. However, the integrity of these systems can be compromised by data leakage, particularly when improper dataset partitioning strategies are employed. This paper investigates the presence and impact of data leakage in three published Wi-Fi CSI-based human action recognition methods that utilize deep learning techniques. The original studies achieve precision rates of 95% or higher, attributed to the lack of human-based dataset splitting. By re-evaluating these systems with proper subject-based partitioning, our analysis reveals a substantial decline in performance, underscoring the prevalence of data leakage. This study highlights the critical need for rigorous dataset management and evaluation protocols to ensure the development of robust and reliable human action recognition systems. Our findings advocate for standardized practices in dataset partitioning to mitigate data leakage and enhance the generalizability of Wi-Fi CSI-based models.

Action recognition method based on multi-stream attention-enhanced recursive graph convolution

Action recognition method based on multi-stream attention-enhanced recursive graph convolution

Cross-view motion consistent self-supervised video inter-intra contrastive for action representation understanding

Self-supervised contrastive learning draws on power representational models to acquire generic semantic features from unlabeled data, and the key to training such models lies in how accurately to track motion features. Previous video contrastive learning methods have extensively used spatially or temporally augmentation as similar instances, resulting in models that are more likely to learn static backgrounds than motion features. To alleviate the background shortcuts, in this paper, we propose a cross-view motion consistent (CVMC) self-supervised video inter-intra contrastive model to focus on the learning of local details and long-term temporal relationships. Specifically, we first extract the dynamic features of consecutive video snippets and then align these features based on multi-view motion consistency. Meanwhile, we compare the optimized dynamic features for instance comparison of different videos and local spatial fine-grained with temporal order in the same video, respectively. Ultimately, the joint optimization of spatio-temporal alignment and motion discrimination effectively fills the challenges of the missing components of instance recognition, spatial compactness, and temporal perception in self-supervised learning. Experimental results show that our proposed self-supervised model can effectively learn visual representation information and achieve highly competitive performance compared to other state-of-the-art methods in both action recognition and video retrieval tasks.

Comparative Study of Deep Learning Based Methods for Human Activity Recognition

Comparative Study of Deep Learning Based Methods for Human Activity Recognition