Human Action Recognition Method Research Articles

MAFormer: A cross-channel spatio-temporal feature aggregation method for human action recognition

Human action recognition has been widely used in fields such as human–computer interaction and virtual reality. Despite significant progress, existing approaches still struggle with effectively integrating hierarchical information and processing data beyond a certain frame count. To address these challenges, we introduce the Multi-AxisFormer (MAFormer) model, which is organized in terms of spatial, temporal, and channel dimensions of the action sequence, thereby enhancing the model’s understanding of correlations and intricate structures among and within features. Drawing on the Transformer architecture, we propose the Cross-channel Spatio-temporal Aggregation (CSA) structure for more refined feature extraction and the Multi-Axis Attention (MAA) module for more comprehensive feature aggregation. Moreover, the integration of Rotary Position Embedding (RoPE) boosts the model’s extrapolation and generalization abilities. MAFormer surpasses the known state-of-the-art on multiple skeleton-based action recognition benchmarks with the accuracy of 93.2% on NTU RGB+D 60 cross-subject split, 89.9% on NTU RGB+D 120 cross-subject split, and 97.2% on N-UCLA, offering a novel paradigm for hierarchical modeling in human action recognition.

A Novel Active Learning Framework for Cross-Subject Human Activity Recognition from Surface Electromyography.

Wearable sensor-based human activity recognition (HAR) methods hold considerable promise for upper-level control in exoskeleton systems. However, such methods tend to overlook the critical role of data quality and still encounter challenges in cross-subject adaptation. To address this, we propose an active learning framework that integrates the relation network architecture with data sampling techniques. Initially, target data are used to fine tune two auxiliary classifiers of the pre-trained model, thereby establishing subject-specific classification boundaries. Subsequently, we assess the significance of the target data based on classifier discrepancy and partition the data into sample and template sets. Finally, the sampled data and a category clustering algorithm are employed to tune model parameters and optimize template data distribution, respectively. This approach facilitates the adaptation of the model to the target subject, enhancing both accuracy and generalizability. To evaluate the effectiveness of the proposed adaptation framework, we conducted evaluation experiments on a public dataset and a self-constructed electromyography (EMG) dataset. Experimental results demonstrate that our method outperforms the compared methods across all three statistical metrics. Furthermore, ablation experiments highlight the necessity of data screening. Our work underscores the practical feasibility of implementing user-independent HAR methods in exoskeleton control systems.

Exposing Data Leakage in Wi-Fi CSI-Based Human Action Recognition: A Critical Analysis

Wi-Fi channel state information (CSI)-based human action recognition systems have garnered significant interest for their non-intrusive monitoring capabilities. However, the integrity of these systems can be compromised by data leakage, particularly when improper dataset partitioning strategies are employed. This paper investigates the presence and impact of data leakage in three published Wi-Fi CSI-based human action recognition methods that utilize deep learning techniques. The original studies achieve precision rates of 95% or higher, attributed to the lack of human-based dataset splitting. By re-evaluating these systems with proper subject-based partitioning, our analysis reveals a substantial decline in performance, underscoring the prevalence of data leakage. This study highlights the critical need for rigorous dataset management and evaluation protocols to ensure the development of robust and reliable human action recognition systems. Our findings advocate for standardized practices in dataset partitioning to mitigate data leakage and enhance the generalizability of Wi-Fi CSI-based models.

Comparative Study of Deep Learning Based Methods for Human Activity Recognition

Comparative Study of Deep Learning Based Methods for Human Activity Recognition

Auxiliary audio–textual modalities for better action recognition on vision-specific annotated videos

Most current audio–visual datasets are class-relevant, where audio and visual modalities are annotated. Thus, current audio–visual recognition methods apply cross-modality attention or modality fusion. However, leveraging the audio modality effectively in vision-specific videos for human activity recognition is of particular challenge. We address this challenge by proposing a novel audio–visual recognition framework that effectively leverages audio modality in any vision-specific annotated dataset. The proposed framework employs language models (e.g., GPT-3, CPT-text, BERT) for building a semantic audio–video label dictionary (SAVLD) that serves as a bridge between audio and video datasets by mapping each video label to its most K-relevant audio labels. Then, SAVLD along with a pre-trained audio multi-label model are used to estimate the audio–visual modality relevance. Accordingly, we propose a novel learnable irrelevant modality dropout (IMD) to completely drop the irrelevant audio modality and fuse only the relevant modalities. Finally, for the efficiency of the proposed multimodal framework, we present an efficient two-stream video Transformer to process the visual modalities (i.e., RGB frames and optical flow). The final predictions are re-ranked with GPT-3 recommendations of the human activity classes. GPT-3 provides high-level recommendations using the labels of the detected visual objects and the audio predictions of the input video. Our framework demonstrated a remarkable performance on the vision-specific annotated datasets Kinetics400 and UCF-101 by outperforming most relevant human activity recognition methods.

FMCW Radar Human Action Recognition Based on Asymmetric Convolutional Residual Blocks.

Human action recognition based on optical and infrared video data is greatly affected by the environment, and feature extraction in traditional machine learning classification methods is complex; therefore, this paper proposes a method for human action recognition using Frequency Modulated Continuous Wave (FMCW) radar based on an asymmetric convolutional residual network. First, the radar echo data are analyzed and processed to extract the micro-Doppler time domain spectrograms of different actions. Second, a strategy combining asymmetric convolution and the Mish activation function is adopted in the residual block of the ResNet18 network to address the limitations of linear and nonlinear transformations in the residual block for micro-Doppler spectrum recognition. This approach aims to enhance the network's ability to learn features effectively. Finally, the Improved Convolutional Block Attention Module (ICBAM) is integrated into the residual block to enhance the model's attention and comprehension of input data. The experimental results demonstrate that the proposed method achieves a high accuracy of 98.28% in action recognition and classification within complex scenes, surpassing classic deep learning approaches. Moreover, this method significantly improves the recognition accuracy for actions with similar micro-Doppler features and demonstrates excellent anti-noise recognition performance.

2D human skeleton action recognition with spatial constraints

AbstractHuman actions are predominantly presented in 2D format in video surveillance scenarios, which hinders the accurate determination of action details not apparent in 2D data. Depth estimation can aid human action recognition tasks, enhancing accuracy with neural networks. However, reliance on images for depth estimation requires extensive computational resources and cannot utilise the connectivity between human body structures. Besides, the depth information may not accurately reflect actual depth ranges, necessitating improved reliability. Therefore, a 2D human skeleton action recognition method with spatial constraints (2D‐SCHAR) is introduced. 2D‐SCHAR employs graph convolution networks to process graph‐structured human action skeleton data comprising three parts: depth estimation, spatial transformation, and action recognition. The initial two components, which infer 3D information from 2D human skeleton actions and generate spatial transformation parameters to correct abnormal deviations in action data, support the latter in the model to enhance the accuracy of action recognition. The model is designed in an end‐to‐end, multitasking manner, allowing parameter sharing among these three components to boost performance. The experimental results validate the model's effectiveness and superiority in human skeleton action recognition.

Human movement science-informed multi-task spatio temporal graph convolutional networks for fitness action recognition and evaluation

In recent years, with the rise of health consciousness, people’s demand for fitness has steadily increased. Utilizing automated human action recognition technology to monitor users’ movements during exercise continuously would help prevent situations where incorrect movements lead to injuries while working out. Skeleton-based human action recognition methods can overcome the susceptibility of past color-based and depth-based methods to various external backgrounds and noise, becoming a more successful solution in recent years. In this study, the auto-fitness advisor system we propose not only identifies the category of the action but also assesses the quality of the action and provides suggestions. We integrate human movement science, such as the Five Primary Kinetic Chains (5PKC), which defines the primary physiological principles in human movement, to enhance the accuracy of fitness action recognition by providing a more precise relationship between the human skeleton and muscles. For assessing the quality of movements and providing suggestions, we have designed a multi-task objective function within our model. Overall, the proposed model is a multi-task model based on Spatio Temporal Graph Convolutional Networks (ST-GCN), which employs the Five Primary Kinetic Chains (5PKC) as a partitioning strategy for skeletal information. In our experiments, we not only collected a certain amount of datasets in gyms to validate the performance of our model but also compared it with other current methods using existing public datasets.

A human activity recognition method based on Vision Transformer

Human activity recognition has a wide range of applications in various fields, such as video surveillance, virtual reality and human–computer intelligent interaction. It has emerged as a significant research area in computer vision. GCN (Graph Convolutional networks) have recently been widely used in these fields and have made great performance. However, there are still some challenges including over-smoothing problem caused by stack graph convolutions and deficient semantics correlation to capture the large movements between time sequences. Vision Transformer (ViT) is utilized in many 2D and 3D image fields and has surprised results. In our work, we propose a novel human activity recognition method based on ViT (HAR-ViT). We integrate enhanced AGCL (eAGCL) in 2s-AGCN to ViT to make it process spatio-temporal data (3D skeleton) and make full use of spatial features. The position encoder module orders the non-sequenced information while the transformer encoder efficiently compresses sequence data features to enhance calculation speed. Human activity recognition is accomplished through multi-layer perceptron (MLP) classifier. Experimental results demonstrate that the proposed method achieves SOTA performance on three extensively used datasets, NTU RGB+D 60, NTU RGB+D 120 and Kinetics-Skeleton 400.

An unsupervised statistical representation learning method for human activity recognition

An unsupervised statistical representation learning method for human activity recognition

A Spatio-temporal Graph Transformer driven model for recognizing fine-grained data human activity

Human activity recognition(HAR) is an important research focus in ubiquitous computing. It has been widely applied in various domains, such as smart homes, healthcare assistance, and sports training. Accurate human activity recognition directly impacts the performance of downstream tasks. Existing methods for human activity recognition primarily rely on extracting temporal or spatial features from the data. These features are used for the task of human activity recognition. The existing methods mainly use CNN or RNN models to model the Euclidean correlations among spatially adjacent sensors or channels. However, non-Euclidean pairwise correlations among all sensors or channels are even critical for accurate classification, which has been ignored by the existing methods. In this paper, we incorporate fine-grained spatial structural data into the model to overcome these limitations. A novel deep learning model for human activity recognition is proposed, which is called the fine-grained data-oriented Spatial-Temporal Graph Transformer network (STGT). The introduction of the STGT model can eliminate the limitations of existing spatial feature extraction methods by leveraging a novel data organization approach proposed in this study. This model enhances the spatial features within the time feature extraction module for effective spatio-temporal feature extraction. We conducted experiments on four large-scale real-world HAR datasets to evaluate its performance. The experimental results demonstrate the superiority of our method over state-of-the-art approaches.

Position-Aware Indoor Human Activity Recognition Using Multisensors Embedded in Smartphones.

Composite indoor human activity recognition is very important in elderly health monitoring and is more difficult than identifying individual human movements. This article proposes a sensor-based human indoor activity recognition method that integrates indoor positioning. Convolutional neural networks are used to extract spatial information contained in geomagnetic sensors and ambient light sensors, while transform encoders are used to extract temporal motion features collected by gyroscopes and accelerometers. We established an indoor activity recognition model with a multimodal feature fusion structure. In order to explore the possibility of using only smartphones to complete the above tasks, we collected and established a multisensor indoor activity dataset. Extensive experiments verified the effectiveness of the proposed method. Compared with algorithms that do not consider the location information, our method has a 13.65% improvement in recognition accuracy.

Data-efficient multimodal human action recognition for proactive human–robot collaborative assembly: A cross-domain few-shot learning approach

With the recent vision of Industry 5.0, the cognitive capability of robots plays a crucial role in advancing proactive human–robot collaborative assembly. As a basis of the mutual empathy, the understanding of a human operator’s intention has been primarily studied through the technique of human action recognition. Existing deep learning-based methods demonstrate remarkable efficacy in handling information-rich data such as physiological measurements and videos, where the latter category represents a more natural perception input. However, deploying these methods in new unseen assembly scenarios requires first collecting abundant case-specific data. This leads to significant manual effort and poor flexibility. To deal with the issue, this paper proposes a novel cross-domain few-shot learning method for data-efficient multimodal human action recognition. A hierarchical data fusion mechanism is designed to jointly leverage the skeletons, RGB images and depth maps with complementary information. Then a temporal CrossTransformer is developed to enable the action recognition with very limited amount of data. Lightweight domain adapters are integrated to further improve the generalization with fast finetuning. Extensive experiments on a real car engine assembly case show the superior performance of proposed method over state-of-the-art regarding both accuracy and finetuning efficiency. Real-time demonstrations and ablation study further indicate the potential of early recognition, which is beneficial for the robot procedures generation in practical applications. In summary, this paper contributes to the rarely explored realm of data-efficient human action recognition for proactive human–robot collaboration.

Lightweight human activity recognition method based on the MobileHARC model

In recent years, Human activity recognition (HAR) based on wearable devices has been widely applied in health applications and other fields. Currently, most HAR models are based on the Convolutional Neural Network (CNN), Long Short-Term Memory (LSTM), or their combination. Recently, there have been proposals based on Transformer and its variant models. However, due to the fact that these models have sequential network structures and are unable to simultaneously focus on local and global features, thus, resulting in a reduction in recognition performance. In addition, along with the substantial computational resources required by Transformers, they are not suitable for resource-constrained devices. In this paper, the primary distinction of our proposed model from other hybrid models that combine CNN and Transformer is that our model adopts a completely new parallel network architecture and primarily focuses on lightweight design. Particularly, We proposed the Mobile Human Activity Recognition Conformer (MobileHARC), which adopts the parallel structure with a lightweight Transformer and CNN as the backbone networks. Furthermore, we proposed the Inverted Residual Lightweight Convolution Block and Multiscale Lightweight Multi-Head Self-Attention Mechanism. We systematically evaluate the proposed models on four public datasets. Experimental results show that MobileHARC achieves superior recognition performance, and uses fewer Floating-Point Operations per Second (FLOPs) and parameters compared to current models.

Research on Human Action Recognition Method Based on Machine Learning

Human action recognition (HAR) is an important research direction in the field of computer vision, and human action recognition technology provides an important foundation for computers to understand and simulate human behavior. It has been widely used in many practical applications, including human-computer interaction, gesture recognition, motion analysis, motion capture, virtual reality and augmented reality. Early methods were mainly based on the characteristics of manual design and traditional machine learning methods, such as support vector machine (SVM) and Random Forest. These methods usually rely on manually extracted features, such as edge, texture and color information, but they do not perform well in complex scenes and changing environments. With the rise of deep learning, especially the successful application of Convolutional Neural Network (CNN), great breakthroughs have been made in human action recognition. Using CNN, we can learn the feature representation directly from the original image data, which avoids the trouble of manually designing features. This paper expounds the research progress and future development trend of human action recognition methods based on machine learning.

Deep learning method for human activity recognition using heaped LSTM and image pattern of activity

Deep learning method for human activity recognition using heaped LSTM and image pattern of activity

A Novel Method for Cross-Subject Human Activity Recognition with Wearable Sensors

A Novel Method for Cross-Subject Human Activity Recognition with Wearable Sensors

A profile similarity-based personalized federated learning method for wearable sensor-based human activity recognition

Wearable sensor-based human activity recognition (HAR) utilizes artificial intelligence models to analyze real-time data like accelerometer data to recognize daily human activities. While it greatly benefits the life of senior citizens and postoperative patients, it conventionally requires the collected data to be uploaded to a central server to train AI models, raising critical security and privacy concerns. Though Federated learning (FL) emerges as a viable way to cope with these problems, it is confronted by the data heterogeneity problem, where the varying activity patterns of different individuals result in non-identically distributed local data. Some FL models have been proposed to solve the data heterogeneity problem by leveraging the similarity between individuals to create a personalized global model for each individual. However, they are still limited by increased computation or unreliable relationships in the similarity computation. This study proposes a novel profile similarity-based personalized federated learning for wearable sensor-based HAR where the similarity between individuals can be reflected in their profile, such as age, gender, height, and weight. When personalizing a model for an individual, we compute the weighted sum of all clients’ local models, where the weight is determined by the similarity value computed from the profile. In this way, the local models from individuals who have higher similarity values will contribute more towards personalizing a model for a targeted individual than those who are less similar. Experiment results demonstrate that the proposed model outperformed the baseline FL and centralized learning on both RealWorld and SisFall datasets. We also discuss the tradeoff between privacy and personalization and FL's advantages over centralized learning.

CSI-GLSTN: A Location-Independent CSI Human Activity Recognition Method Based on Spatio-Temporal and Channel Feature Fusion

CSI-GLSTN: A Location-Independent CSI Human Activity Recognition Method Based on Spatio-Temporal and Channel Feature Fusion

Human Activity Recognition Method Based on Scattering Separation Using Multi-Frequency Radar Data

Human Activity Recognition Method Based on Scattering Separation Using Multi-Frequency Radar Data