Human Action Recognition Method Research Articles

Intelligent Video Analytics for Human Action Recognition: The State of Knowledge.

The paper presents a comprehensive overview of intelligent video analytics and human action recognition methods. The article provides an overview of the current state of knowledge in the field of human activity recognition, including various techniques such as pose-based, tracking-based, spatio-temporal, and deep learning-based approaches, including visual transformers. We also discuss the challenges and limitations of these techniques and the potential of modern edge AI architectures to enable real-time human action recognition in resource-constrained environments.

A 3DCNN-Based Knowledge Distillation Framework for Human Activity Recognition.

Human action recognition has been actively explored over the past two decades to further advancements in video analytics domain. Numerous research studies have been conducted to investigate the complex sequential patterns of human actions in video streams. In this paper, we propose a knowledge distillation framework, which distills spatio-temporal knowledge from a large teacher model to a lightweight student model using an offline knowledge distillation technique. The proposed offline knowledge distillation framework takes two models: a large pre-trained 3DCNN (three-dimensional convolutional neural network) teacher model and a lightweight 3DCNN student model (i.e., the teacher model is pre-trained on the same dataset on which the student model is to be trained on). During offline knowledge distillation training, the distillation algorithm trains only the student model to help enable the student model to achieve the same level of prediction accuracy as the teacher model. To evaluate the performance of the proposed method, we conduct extensive experiments on four benchmark human action datasets. The obtained quantitative results verify the efficiency and robustness of the proposed method over the state-of-the-art human action recognition methods by obtaining up to 35% improvement in accuracy over existing methods. Furthermore, we evaluate the inference time of the proposed method and compare the obtained results with the inference time of the state-of-the-art methods. Experimental results reveal that the proposed method attains an improvement of up to 50× in terms of frames per seconds (FPS) over the state-of-the-art methods. The short inference time and high accuracy make our proposed framework suitable for human activity recognition in real-time applications.

Determination of workers' compliance to safety regulations using a spatio-temporal graph convolution network

The safety of workers in construction remains a critical issue despite the automation of several tasks with fewer workers on site. As fatal accidents of workers account for a significant number of construction accidents, considerable effort has been made to monitor workers’ safety behaviors with additional personnel for supervising workers. With the advancement of data analytics, recent research has reported various human activity recognition methods based on image data to perform automated worker monitoring without additional labor. Nevertheless, unlike existing approaches based on a single image, a method that can capture a series of actions from sequential images is required to monitor workers’ compliance with safety behavior. To this end, an approach based on OpenPose and a spatio-temporal graph convolutional network is proposed in this study to evaluate workers’ compliance with safety regulations using sequential videos. The two primary functions of the developed method include 1) classifying each safety behavior among five representative behaviors stipulated in construction, and 2) determining the compliance of workers with each safety regulation. The results indicate that the developed approach can capture momentary safety behaviors and workers’ compliance with feasible accuracy of an average F1 score greater than 0.8. Furthermore, the proposed method can be extended to safety intervention policies with behavior-based feedback to inform workers of their non-compliance with safety behaviors. Therefore, this study contributes to proactive safety management by focusing on workers’ behavioral levels rather than on accident rate-based management.

A New Deep-Learning Method for Human Activity Recognition.

Currently, three-dimensional convolutional neural networks (3DCNNs) are a popular approach in the field of human activity recognition. However, due to the variety of methods used for human activity recognition, we propose a new deep-learning model in this paper. The main objective of our work is to optimize the traditional 3DCNN and propose a new model that combines 3DCNN with Convolutional Long Short-Term Memory (ConvLSTM) layers. Our experimental results, which were obtained using the LoDVP Abnormal Activities dataset, UCF50 dataset, and MOD20 dataset, demonstrate the superiority of the 3DCNN + ConvLSTM combination for recognizing human activities. Furthermore, our proposed model is well-suited for real-time human activity recognition applications and can be further enhanced by incorporating additional sensor data. To provide a comprehensive comparison of our proposed 3DCNN + ConvLSTM architecture, we compared our experimental results on these datasets. We achieved a precision of 89.12% when using the LoDVP Abnormal Activities dataset. Meanwhile, the precision we obtained using the modified UCF50 dataset (UCF50mini) and MOD20 dataset was 83.89% and 87.76%, respectively. Overall, our work demonstrates that the combination of 3DCNN and ConvLSTM layers can improve the accuracy of human activity recognition tasks, and our proposed model shows promise for real-time applications.

A lightweight fast human activity recognition method using hybrid unsupervised-supervised feature

A lightweight fast human activity recognition method using hybrid unsupervised-supervised feature

Privacy-Preserving Cross-Environment Human Activity Recognition.

Recent studies have demonstrated the success of using the channel state information (CSI) from the WiFi signal to analyze human activities in a fixed and well-controlled environment. Those systems usually degrade when being deployed in new environments. A straightforward solution to solve this limitation is to collect and annotate data samples from different environments with advanced learning strategies. Although workable as reported, those methods are often privacy sensitive because the training algorithms need to access the data from different environments, which may be owned by different organizations. We present a practical method for the WiFi-based privacy-preserving cross-environment human activity recognition (HAR). It collects and shares information from different environments, while maintaining the privacy of individual person being involved. At the core of our approach is the utilization of the Johnson-Lindenstrauss transform, which is theoretically shown to be differentially private. Based on that, we further design an adversarial learning strategy to generate environment-invariant representations for HAR. →doWe demonstrate the effectiveness of the proposed method with different data modalities from two real-life environments. More specifically, on the raw CSI dataset, it shows 2.18% and 1.24% improvements over challenging baselines for two environments, respectively. Moreover, with the discrete wavelet transform features, it further yields 5.71% and 1.55% improvements, respectively.

Human Action Recognition for Dynamic Scenes of Emergency Rescue Based on Spatial-Temporal Fusion Network

Targeting the problems of the insufficient utilization of temporal and spatial information in videos and a lower accuracy rate, this paper proposes a human action recognition method for dynamic videos of emergency rescue based on a spatial-temporal fusion network. A time domain segmentation strategy based on random sampling maintains the overall time domain structure of the video. Considering the spatial-temporal asynchronous relationship, multiple asynchronous motion sequences are increased as input of the temporal convolutional network. spatial-temporal features are fused in convolutional layers to reduce feature loss. Because time series information is crucial for human action recognition, the acquired mid-layer spatial-temporal fusion features are sent into Bidirectional Long Short-Term Memory (Bi-LSTM) to obtain the human movement features in the whole video temporal dimension. Experiment results show the proposed method fully fuses spatial and temporal dimension information and improves the accuracy of human action recognition in dynamic scenes. It is also faster than traditional methods.

A human activity recognition method using wearable sensors based on convtransformer model

A human activity recognition method using wearable sensors based on convtransformer model

Skeleton-Based Mutually Assisted Interacted Object Localization and Human Action Recognition

Skeleton data carries valuable motion information and is widely explored in human action recognition. However, not only the motion information but also the interaction with the environment provides discriminative cues to recognize the action of persons. In this paper, we propose a joint learning framework for mutually assisted ''interacted object localization'' and ''human action recognition'' based on skeleton data. The two tasks are serialized together and collaborate to promote each other, where preliminary action type derived from skeleton alone helps improve interacted object localization, which in turn provides valuable cues for the final human action recognition. Besides, we explore the temporal consistency of interacted object as constraint to better localize the interacted object with the absence of ground-truth labels. Extensive experiments on the datasets of SYSU-3D, NTU60 RGB+D, Northwestern-UCLA and UAV-Human show that our method achieves the best or competitive performance with the state-of-the-art methods for human action recognition. Visualization results show that our method can also provide reasonable interacted object localization results.

A PSL-based Approach to Human Activity Recognition in Smart Home Environments

Human activity recognition is widely used in smart cities, public safety and other fields, especially in smart home systems where it has a pivotal role. The study addresses the shortcomings of Markov logic networks for human activity recognition and proposes a human activity recognition method in smart home scenarios - an activity recognition framework based on Probabilistic Soft Logic (PSL). The framework is able to deal with logical uncertainty problems and provides expression and inference mechanisms for data uncertainty problems on this basis. The framework utilizes Deng entropy evidence theory to provide an evaluation method for sensor event uncertainty, and combines event calculus for activity modeling. Comparing the PSL method with three other common recognition methods, Ontology, Hidden Markov Model (HMM), and Markov logic network, on a public dataset, it was found that the PSL method has a much better ability to handle data uncertainty than the other three algorithms. The average recognition rates on the ADL and ADL-E sub datasets were 82.87% and 80.33%, respectively. In experiments to verify the ability of PSL to handle temporal complexity, PSL showed the least significant decrease in the average recognition rate and maintained an average recognition rate of 81.02% in the presence of concurrent and alternating activities. The human activity recognition method based on PSL has a better performance in handling both data uncertainty and temporal complexity.

Research on Human Action Analysis and Recognition Methods Based on Deep Learning

Research on Human Action Analysis and Recognition Methods Based on Deep Learning

STGL-GCN: Spatial–Temporal Mixing of Global and Local Self-Attention Graph Convolutional Networks for Human Action Recognition

Human action recognition methods based on skeleton data have been widely studied owing to their strong robustness to illumination and complex backgrounds. Existing methods have achieved good recognition results; however, they have certain challenges, such as the fixed topological structure of the graph, the omission of nonphysical joint correlation, and the inability to extract local spatial–temporal features. Herein, we propose spatial–temporal mixing of global and local self-attention graph convolutional networks (STGL-GCN) using skeleton data. The global self-attention matrix captures the potential dependencies of nonphysical correlations between joints, and the local self-attention matrix determines the connection strength of the physical edges of joints. The matrices are updated together with the convolution parameters in each network layer as the model is trained for optimal graph structure to achieve accurate action expressions Experiments on the NTU-RGBD dataset demonstrate that our model accurately recognizes actions.

Graph Diffusion Convolutional Network for Skeleton Based Semantic Recognition of Two-Person Actions.

Graph Convolutional Networks (GCNs) have successfully boosted skeleton-based human action recognition. However, existing GCN-based methods mostly cast the problem as separated person's action recognition while ignoring the interaction between the action initiator and the action responder, especially for the fundamental two-person interactive action recognition. It is still challenging to effectively take into account the intrinsic local-global clues of the two-person activity. Additionally, message passing in GCN depends on adjacency matrix, but skeleton-based human action recognition methods tend to calculate the adjacency matrix with the fixed natural skeleton connectivity. It means that messages can only travel along a fixed path at different layers of the network or in different actions, which greatly reduces the flexibility of the network. To this end, we propose a novel graph diffusion convolutional network for skeleton based semantic recognition of two-person actions by embedding the graph diffusion into GCNs. At technical fronts, we dynamically construct the adjacency matrix based on practical action information, so that we can guide the message propagation in a more meaningful way. Simultaneously, we introduce the frame importance calculation module to conduct dynamic convolution, so that we can avoid the negative effect caused by the traditional convolution, wherein the shared weights may fail to capture key frames or be affected by noisy frames. Besides, we comprehensively leverage the multidimensional features related to joints' local visual appearances, global spatial relationship and temporal coherency, and for different features, different metrics are designed to measure the similarity underlying the corresponding real physical law of the motions. Moreover, extensive experiments and comprehensive evaluations on four public large-scale datasets (NTU-RGB+D 60, NTU-RGB+D 120, Kinetics-Skeleton 400, and SBU-Interaction) demonstrate that our method outperforms the state-of-the-art methods.

Assessing the Soldier Survivability Tradespace Using a Single IMU

Soldier burden is influenced by the environment, metabolic demands, equipment properties, and psychological stressors; however, much of our knowledge of soldier burden is in the context of body-borne load mass in controlled laboratory environments. Thus, to further our understanding of how all aspects of soldier burden affect the survivability tradespace (i.e., performance, health, and susceptibility to enemy action), field-based motion capture methods are needed. We developed a human activity recognition method using the deep convolutional long short-term memory neural network architecture, trained using a single inertial measurement unit on the upper back, to identify eleven tactical movement patterns commonly performed by soldiers. Using a two-step logical algorithm, real-world constraints are forced, and class labels are expanded to 19 movements. Presented are three models based on Indoor, Section Attack (outdoors), and a General approach. Across all three approaches, we obtained an average accuracy of 90.0%. Further, we used these predictions to calculate meaningful tradespace metrics, which had an excellent agreement with calculations using the true labels. Military leaders and defence scientists can use this approach to quantify tradespace metrics in the field, as a preprocessing tool to supplement other technology, and make data-driven decisions that can help improve performance, decrease susceptibility, and increase overall mission success.

STC-NLSTMNet: An Improved Human Activity Recognition Method Using Convolutional Neural Network with NLSTM from WiFi CSI

Human activity recognition (HAR) has emerged as a significant area of research due to its numerous possible applications, including ambient assisted living, healthcare, abnormal behaviour detection, etc. Recently, HAR using WiFi channel state information (CSI) has become a predominant and unique approach in indoor environments compared to others (i.e., sensor and vision) due to its privacy-preserving qualities, thereby eliminating the need to carry additional devices and providing flexibility of capture motions in both line-of-sight (LOS) and non-line-of-sight (NLOS) settings. Existing deep learning (DL)-based HAR approaches usually extract either temporal or spatial features and lack adequate means to integrate and utilize the two simultaneously, making it challenging to recognize different activities accurately. Motivated by this, we propose a novel DL-based model named spatio-temporal convolution with nested long short-term memory (STC-NLSTMNet), with the ability to extract spatial and temporal features concurrently and automatically recognize human activity with very high accuracy. The proposed STC-NLSTMNet model is mainly comprised of depthwise separable convolution (DS-Conv) blocks, feature attention module (FAM) and NLSTM. The DS-Conv blocks extract the spatial features from the CSI signal and add feature attention modules (FAM) to draw attention to the most essential features. These robust features are fed into NLSTM as inputs to explore the hidden intrinsic temporal features in CSI signals. The proposed STC-NLSTMNet model is evaluated using two publicly available datasets: Multi-environment and StanWiFi. The experimental results revealed that the STC-NLSTMNet model achieved activity recognition accuracies of 98.20% and 99.88% on Multi-environment and StanWiFi datasets, respectively. Its activity recognition performance is also compared with other existing approaches and our proposed STC-NLSTMNet model significantly improves the activity recognition accuracies by 4% and 1.88%, respectively, compared to the best existing method.

First-Person Hand Action Recognition Using Multimodal Data

Extensive studies have been conducted on human action recognition, whereas relatively few methods have been proposed for hand action recognition. Although it is very natural and straightforward to apply a human action recognition method to hand action recognition, this approach cannot always lead to state-of-the-art performance. One of the important reasons is that both the between-class difference and the within-class difference in hand actions are much smaller than those in human actions. In this article, we study first-person hand action recognition from RGB-D sequences. To explore whether pretrained networks substantially influence accuracy, eight classic pretrained networks and one pretrained network designed by us are introduced for extracting RGB-D features. A Lie group is introduced for hand pose representation. Ablation studies are conducted to compare the discriminative power of the RGB modality, depth modality, pose modality, and their combinations. In our method, a fixed number of frames are randomly sampled to represent an action. This temporal modeling strategy is simple but is proven more effective than both the graph convolutional network (GCN) and the recurrent neural network (RNN), which are widely adopted by conventional methods. Evaluation experiments on two public data sets demonstrate that our method markedly outperforms recent baselines.

Human activity recognition and Alert system

Monitoring and taking care of people has been a necessary factor. Since it is very vital to monitor the movements of the people left alone in our homes and people in unattended scenarios, this project focuses on Human Activity Recognition (HAR). This project showcases a smart human action recognition method to automatically identify the human activities from skeletal joint motions and combines the competencies. This is a low-cost method and has high accuracy. An independent mobile application is also developed to monitor the activities of the people and their surroundings when they are alone. A mobile notification is sent during abnormal activities. Thus, our project provides a way to help senior citizens from any kind of mishaps.

Convolutional Neural Network-Based Human Activity Recognition for Edge Fitness and Context-Aware Health Monitoring Devices

The vital signs can vary significantly depending on the daily physical activities, which may not be due to defects of the organs. Under remote human health-monitoring applications, for reliable disease diagnosis, recorded biomedical signals such as an electrocardiogram (ECG) and a photoplethysmogram (PPG) must be indexed with physical activity information, as it is unknown to the physicians and also to the computer-aided diagnostic system. Since deep-learning (DL) networks were explored for various vital signs extraction and disease classification, we present an effective convolutional neural network (CNN)-based human activity recognition (HAR) method by exploring suitable hyperparameters. CNN-based methods are evaluated using the acceleration signals taken from the standard HAR benchmark database, University of California, Irvine (UCI) with accuracy (ACC), model size (in kB), and processing time (PT), and also implemented on the Raspberry Pi 4 (R-Pi-4) to study real-time feasibility. Evaluation results showed that higher HAR accuracy can be achieved with the activation function of the exponential linear unit (ELU) for the 2-layer CNN with a segment size of 2.5 s (ACC of 89.05% and PT of 0.142 ms), the 4-layer CNN with a segment size of 1 s (ACC of 91.66% and PT of 0.541 ms), and the 6-layer CNN with a segment size of 2 s (ACC of 91.18% and PT of 1.672 ms). Results demonstrated that the selection of an optimal number of layers and hyperparameters plays a major role in achieving higher accuracy with lower computational time on both personal computer central processing unit (PC-CPU) and R-Pi computing platforms, which were not addressed in past studies.

Human action recognition method based on Motion Excitation and Temporal Aggregation module

Aiming at the problem of low modeling efficiency and feature loss of temporal modeling in human action recognition, we propose a human action recognition method based on Motion Excitation and Temporal Aggregation module (META). The method can capture multi-state and multi-scale temporal information to achieve effective motion excitation. Firstly, temporal relational sampling is performed on video frames; Secondly, META is proposed to capture multi-state and multi-scale temporal information. META is composed of Multi-scale Motion Excitation module (MME) and Squeeze and Excitation Temporal Aggregation module (SETA). MME captures the feature level temporal difference by transforming the features into the temporal channel, which directly establishes the relationship between features and temporal channel, and solves the problem of low modeling efficiency. SETA transforms the local convolution into a set of sub-convolutions. Multiple sub-convolutions form hierarchies to extract features together and share the results of the upper convolutional layer, which increases the final temporal receptive field and solves the problem of feature loss. Moreover, the optical flow features are extracted through Cross modality pre-training to improve the utilization of temporal information. Finally, the result of human action recognition is carried out by combining spatiotemporal two stream features. Experimental results show that the accuracy of this method in UCF101 and HMDB-51 is 96.0% and 71.2% respectively, which is higher than other studies in the same period.

Performance Analysis of Deep Learning based Human Activity Recognition Methods

Human Activity Recognition (HAR) is one of the most important branches of human-centered research activities. Along with the development of artificial intelligence, deep learning techniques have gained remarkable success in computer vision. In recent years, there is a growing interest in Human Activity Recognition systems applied in healthcare, security surveillance, and human motion-based activities. A HAR system is essentially made of a wearable device equipped with a set of sensors (like accelerometers, gyroscopes, magnetometers, heart-rate sensors, etc.). Different methods are being applied for improving the accuracy and performance of the HAR system. In this paper, we implement Artificial Neural Network (ANN), and Convolutional Neural Network (CNN) in combination with Long Short-term Memory (LSTM) methods with different layers and compare their outputs towards the accuracy in the HAR system. We compare the accuracy of different HAR methods and observed that the performance of our proposed model of CNN 2 layers with LSTM 1 layer is the best.