Human Action Recognition Algorithm Research Articles

Self-supervised learning for human activity recognition using 700,000 person-days of wearable data

Accurate physical activity monitoring is essential to understand the impact of physical activity on one’s physical health and overall well-being. However, advances in human activity recognition algorithms have been constrained by the limited availability of large labelled datasets. This study aims to leverage recent advances in self-supervised learning to exploit the large-scale UK Biobank accelerometer dataset—a 700,000 person-days unlabelled dataset—in order to build models with vastly improved generalisability and accuracy. Our resulting models consistently outperform strong baselines across eight benchmark datasets, with an F1 relative improvement of 2.5–130.9% (median 24.4%). More importantly, in contrast to previous reports, our results generalise across external datasets, cohorts, living environments, and sensor devices. Our open-sourced pre-trained models will be valuable in domains with limited labelled data or where good sampling coverage (across devices, populations, and activities) is hard to achieve.

Research on human behavior recognition algorithm based on HRNet and LSTM with model lightweight

This paper proposes a lightweight human action recognition algorithm for pedestrian behavior recognition. First, the skeleton feature information is input into the HRNet network model. In order to selectively enhance more details containing the target features and suppress irrelevant or weak features, an external attention mechanism is added to the HRN child model. Secondly, in order to extract the temporal characteristics of the target feature vector and ensure the continuity of actions in human behavior recognition, a dual-stream network based on HRNet and Long Short-Term Memory (LSTM) is constructed; finally, due to the huge model, it cannot be well transplanted to embedded. Therefore, this paper uses depthwise separable convolution to lightweight the network model. The experimental results show that in terms of human behavior recognition, the method in this paper has better recognition accuracy than Two-stream, Multi-streamCNN, Cov3DJ, ConvNets, JTM, ASM-3, RF+SW, hd-CNN and TPSMMs.

A micro-Doppler spectrogram denoising algorithm for radar human activity recognition

Radar signal recognition based on micro-Doppler spectrogram has been widely used in human action recognition tasks. However, in practical application scenarios, radar signals inevitably have noise, which leads to different degrees of deformation of the spectrogram graph structure and affects the accuracy of subsequent recognition algorithms. In this paper, we present “ACFL”, a novel algorithm for micro-Doppler spectrogram denoising, which aims to reduce the impact of noise on human action recognition. ACFL employs amplitude–frequency two-dimensional clustering and fuzzy logic clustering selection mechanism to remove noise elements from the spectrogram. Moreover, to address the issue of noise leakage or target missing under time-varying noise and action conditions, ACFL adopts spectrogram segmentation based on short-term Rényi entropy. By dividing the spectrogram into intervals with different time–frequency distributions, the dynamic spectrogram denoise over time is achieved. Simulation and measured data experiments demonstrate that the proposed algorithm not only achieves a higher-quality denoised spectrogram but also significantly improves the accuracy of human action recognition under noisy conditions.

HARDVS: Revisiting Human Activity Recognition with Dynamic Vision Sensors

The main streams of human activity recognition (HAR) algorithms are developed based on RGB cameras which usually suffer from illumination, fast motion, privacy preservation, and large energy consumption. Meanwhile, the biologically inspired event cameras attracted great interest due to their unique features, such as high dynamic range, dense temporal but sparse spatial resolution, low latency, low power, etc. As it is a newly arising sensor, even there is no realistic large-scale dataset for HAR. Considering its great practical value, in this paper, we propose a large-scale benchmark dataset to bridge this gap, termed HARDVS, which contains 300 categories and more than 100K event sequences. We evaluate and report the performance of multiple popular HAR algorithms, which provide extensive baselines for future works to compare. More importantly, we propose a novel spatial-temporal feature learning and fusion framework, termed ESTF, for event stream based human activity recognition. It first projects the event streams into spatial and temporal embeddings using StemNet, then, encodes and fuses the dual-view representations using Transformer networks. Finally, the dual features are concatenated and fed into a classification head for activity prediction. Extensive experiments on multiple datasets fully validated the effectiveness of our model. Both the dataset and source code will be released at https://github.com/Event-AHU/HARDVS.

Human Activity Recognition Based On Video Summarization And Deep Convolutional Neural Network

Abstract In this technological era, human activity recognition (HAR) plays a significant role in several applications like surveillance, health services, Internet of Things, etc. Recent advancements in deep learning and video summarization have motivated us to integrate these techniques for HAR. This paper introduces a computationally efficient HAR technique based on a deep learning framework, which works well in realistic and multi-view environments. Deep convolutional neural networks (DCNNs) normally suffer from different constraints, including data size dependencies, computational complexity, overfitting, training challenges and vanishing gradients. Additionally, with the use of advanced mobile vision devices, the demand for computationally efficient HAR algorithms with the requirement of limited computational resources is high. To address these issues, we used integration of DCNN with video summarization using keyframes. The proposed technique offers a solution that enhances performance with efficient resource utilization. For this, first, we designed a lightweight and computationally efficient deep learning architecture based on the concept of identity skip connections (features reusability), which preserves the gradient loss attenuation and can handle the enormous complexity of activity classes. Subsequently, we employed an efficient keyframe extraction technique to minimize redundancy and succinctly encapsulate the entire video content in a lesser number of frames. To evaluate the efficacy of the proposed method, we performed the experimentation on several publicly available datasets. The performance of the proposed method is measured in terms of evaluation parameters Precision, Recall, F-Measure and Classification Accuracy. The experimental results demonstrated the superiority of the presented algorithm over other existing state-of-the-art methods.

Video Abnormal Action Recognition Based on Multimodal Heterogeneous Transfer Learning

Human abnormal action recognition is crucial for video understanding and intelligent surveillance. However, the scarcity of labeled data for abnormal human actions often hinders the development of high-performance models. Inspired by the multimodal approach, this paper proposes a novel approach that leverages text descriptions associated with abnormal human action videos. Our method exploits the correlation between the text domain and the video domain in the semantic feature space and introduces a multimodal heterogeneous transfer learning framework from the text domain to the video domain. The text of the videos is used for feature encoding and knowledge extraction, and knowledge transfer and sharing are realized in the feature space, which is used to assist in the training of the abnormal action recognition model. The proposed method reduces the reliance on labeled video data, improves the performance of the abnormal human action recognition algorithm, and outperforms the popular video-based models, particularly in scenarios with sparse data. Moreover, our framework contributes to the advancement of automatic video analysis and abnormal action recognition, providing insights for the application of multimodal methods in a broader context.

Self-attention deep ConvLSTM with sparse-learned channel dependencies for wearable sensor-based human activity recognition

In this study, we propose a novel deep-learning architecture with sparse learning for human activity recognition. The proposed model contains 1D CNNs and LSTM layers with a self-attention mechanism to enhance a substantial number of time points in time-series data for human activity recognition systems. Based on the recent success of squeeze-and-excite (SE) networks, the proposed deep learning model utilizes the SE module to enhance channel-wise interdependencies, which in turn leads to a boost in performance. In addition, we utilized sparse learning to retrain only weak nodes and freeze stronger nodes in a fully connected layer prior to classification layer. Furthermore, we utilized an entropy-inspired formula to find sparsely located weaker nodes and validated our model on various datasets, including Opportunity, UCI-HAR, and WISDM. Herein, we present an extensive analysis and survey of state-of-the-art studies, in addition to our proposed research. For a fair comparison, we evaluated our deep learning architecture using various performance metrics and achieved better results; the proposed model outperformed state-of-the-art algorithms for human activity recognition.

Design and Development of an Imitation Detection System for Human Action Recognition Using Deep Learning.

Human action recognition (HAR) is a rapidly growing field with numerous applications in various domains. HAR involves the development of algorithms and techniques to automatically identify and classify human actions from video data. Accurate recognition of human actions has significant implications in fields such as surveillance and sports analysis and in the health care domain. This paper presents a study on the design and development of an imitation detection system using an HAR algorithm based on deep learning. This study explores the use of deep learning models, such as a single-frame convolutional neural network (CNN) and pretrained VGG-16, for the accurate classification of human actions. The proposed models were evaluated using a benchmark dataset, KTH. The performance of these models was compared with that of classical classifiers, including K-Nearest Neighbors, Support Vector Machine, and Random Forest. The results showed that the VGG-16 model achieved higher accuracy than the single-frame CNN, with a 98% accuracy rate.

A novel algorithm for human action recognition in compressed domain using attention-guided approach

A novel algorithm for human action recognition in compressed domain using attention-guided approach

Spatio-temporal features for fast early warning of unplanned self-extubation in ICU

Patients’ behaviors in the Intensive Care Units (ICU) have garnered research attention, particularly regarding the impact of Unplanned Extubation (UEX). However, there is currently no existing report on methods for early warning of UEX action in RGB video. Applying traditional human action recognition algorithms to UEX in the complex ICU environment proves challenging. To address the above issue, we propose a novel feature for early warning of UEX action in patients using RGB videos. Firstly, we employ the YOLOv3 detection method to extract the region of interest (ROI), which corresponds to the region where the patient is located. Subsequently, we develop a spatio-temporal (ST) feature for human action tracking by using the L-K optical flow algorithm. This ST feature encompasses optical flow corner number, trajectory distance, and wavelet transform features. Finally, we utilize support vector machine (SVM) for patient action classification and early warning. Experimental results on the ICU monitoring dataset demonstrate the superior performance of the proposed feature in UEX prediction.

ENGA: Elastic Net-Based Genetic Algorithm for human action recognition

ENGA: Elastic Net-Based Genetic Algorithm for human action recognition

A Novel Multiperson Activity Recognition Algorithm Based on Point Clouds Measured by Millimeter-Wave MIMO Radar

Radar is widely used in human activity recognition (HAR). The radar-based HAR algorithms can be roughly categorized into the feature map-based and the point cloud-based. Between the two, the point cloud-based algorithms are more suitable for multi-person activity recognition (MPAR) tasks due to the spatial properties of point clouds, and are therefore increasingly attracting attentions. However, most existing point cloud-based algorithms make decisions by utilizing either Doppler or coordinate features of point clouds, which are not sufficient to characterize the activity; the remaining algorithms have a large randomness in extracting both features. Neither of these algorithms can fully exploit the advantage of feature fusion, which is the key to recognition performance. To address this shortcoming, this paper proposes a novel MPAR algorithm. First, a feature mapping approach is proposed and defined in equation form, the Doppler, range, azimuth, and elevation features of the point clouds are calculated and accumulated sequentially, so we can obtain the four time-domain feature maps. Second, with the four feature maps as inputs, a four-channel CNN classification model with channel attention is trained for MPAR tasks. Datasets of multi-person activities are collected respectively under the indoor circumstance and aquatic circumstance using the millimeter-wave multiple-input-multiple-output (MIMO) radar platform. The dataset-based evaluation performance shows that the proposed algorithm achieves accuracy results of 96.09% for the indoor-MPAR task and 93.97% for the aquatic-MPAR task, and outperforms three conventional point cloud-based algorithms in terms of the overall MPAR accuracy, the generalization ability to the aquatic activity recognition, and the robustness of distance and headcount.

Human activity recognition algorithm in video sequences based on the fusion of multiple features for realistic and multi-view environment

Human activity recognition algorithm in video sequences based on the fusion of multiple features for realistic and multi-view environment

A Prior Knowledge-Guided Graph Convolutional Neural Network for Human Action Recognition in Solar Panel Installation Process

Human action recognition algorithms have garnered significant research interest due to their vast potential for applications. Existing human behavior recognition algorithms primarily focus on recognizing general behaviors using a large number of datasets. However, in industrial applications, there are typically constraints such as limited sample sizes and high accuracy requirements, necessitating algorithmic improvements. This article proposes a graph convolution neural network model that combines prior knowledge supervision and attention mechanisms, designed to fulfill the specific action recognition requirements for workers installing solar panels. The model extracts prior knowledge from training data, improving the training effectiveness of action recognition models and enhancing the recognition reliability of special actions. The experimental results demonstrate that the method proposed in this paper surpasses traditional models in terms of recognizing solar panel installation actions accurately. The proposed method satisfies the need for highly accurate recognition of designated person behavior in industrial applications, showing promising application prospects.

From Lab to Real World: Assessing the Effectiveness of Human Activity Recognition and Optimization through Personalization.

Human activity recognition (HAR) algorithms today are designed and evaluated on data collected in controlled settings, providing limited insights into their performance in real-world situations with noisy and missing sensor data and natural human activities. We present a real-world HAR open dataset compiled from a wristband equipped with a triaxial accelerometer. During data collection, participants had autonomy in their daily life activities, and the process remained unobserved and uncontrolled. A general convolutional neural network model was trained on this dataset, achieving a mean balanced accuracy (MBA) of 80%. Personalizing the general model through transfer learning can yield comparable and even superior results using fewer data, with the MBA improving to 85%. To emphasize the issue of insufficient real-world training data, we conducted training of the model using the public MHEALTH dataset, resulting in 100% MBA. However, upon evaluating the MHEALTH-trained model on our real-world dataset, the MBA drops to 62%. After personalizing the model with real-world data, an improvement of 17% in the MBA is achieved. This paper showcases the potential of transfer learning to make HAR models trained in different contexts (lab vs. real-world) and on different participants perform well for new individuals with limited real-world labeled data available.

Device Orientation Independent Human Activity Recognition Model for Patient Monitoring Based on Triaxial Acceleration

Tracking a person’s activities is relevant in a variety of contexts, from health and group-specific assessments, such as elderly care, to fitness tracking and human–computer interaction. In a clinical context, sensor-based activity tracking could help monitor patients’ progress or deterioration during their hospitalization time. However, during routine hospital care, devices could face displacements in their position and orientation caused by incorrect device application, patients’ physical peculiarities, or patients’ day-to-day free movement. These aspects can significantly reduce algorithms’ performances. In this work, we investigated how shifts in orientation could impact Human Activity Recognition (HAR) classification. To reach this purpose, we propose an HAR model based on a single three-axis accelerometer that can be located anywhere on the participant’s trunk, capable of recognizing activities from multiple movement patterns, and, thanks to data augmentation, can deal with device displacement. Developed models were trained and validated using acceleration measurements acquired in fifteen participants, and tested on twenty-four participants, of which twenty were from a different study protocol for external validation. The obtained results highlight the impact of changes in device orientation on a HAR algorithm and the potential of simple wearable sensor data augmentation for tackling this challenge. When applying small rotations (<20 degrees), the error of the baseline non-augmented model steeply increased. On the contrary, even when considering rotations ranging from 0 to 180 along the frontal axis, our model reached a f1-score of 0.85±0.11 against a baseline model f1-score equal to 0.49±0.12.

XGBoost-Based Human Activity Recognition Algorithm using Wearable Smart Devices

With the popularity of wearable smart devices, human activity recognition (HAR) based on smart sensors has been widely applied in daily life and medical health fields. In order to balance the accuracy of HAR and the complexity of the algorithm, this paper proposes an HAR algorithm based on the extreme gradient boosting (XGBoost) method. The original data collected by sensors contains noise, thus denoising process is firstly performed. Then multi-dimensional features are extracted from the data because of the limited dimensional features, which cannot be directly utilized in the training process. After that, principal component analysis (PCA) is used to reduce the dimensionality of the data in order to alleviate the input complexity of the training model. Finally, the human activity is recognized using the XGBoost method, and the ultimate goal is to obtain the tradeoff between speed and accuracy of the HAR algorithm.

DeepHAR: a deep feed-forward neural network algorithm for smart insole-based human activity recognition

Health monitoring, rehabilitation, and fitness are just a few domains where human activity recognition can be applied. In this study, a deep learning approach has been proposed to recognise ambulation and fitness activities from data collected by five participants using smart insoles. Smart insoles, consisting of pressure and inertial sensors, allowed for seamless data collection while minimising user discomfort, laying the baseline for the development of a monitoring and/or rehabilitation system for everyday life. The key objective has been to enhance the deep learning model performance through several techniques, including data segmentation with overlapping technique (2 s with 50% overlap), signal down-sampling by averaging contiguous samples, and a cost-sensitive re-weighting strategy for the loss function for handling the imbalanced dataset. The proposed solution achieved an Accuracy and F1-Score of 98.56% and 98.57%, respectively. The Sitting activities obtained the highest degree of recognition, closely followed by the Spinning Bike class, but fitness activities were recognised at a higher rate than ambulation activities. A comparative analysis was carried out both to determine the impact that pre-processing had on the proposed core architecture and to compare the proposed solution with existing state-of-the-art solutions. The results, in addition to demonstrating how deep learning solutions outperformed those of shallow machine learning, showed that in our solution the use of data pre-processing increased performance by about 2%, optimising the handling of the imbalanced dataset and allowing a relatively simple network to outperform more complex networks, reducing the computational impact required for such applications.

Activity recognition using a combination of high gain observer and deep learning computer vision algorithms

Inertial sensors have become increasingly popular in human activity classification due to their ease of use and affordability. This paper proposes a novel algorithm for human activity recognition that is a combination of a high-gain observer and deep learning computer vision classification algorithms. The nonlinear high-gain observer designed using Lyapunov analysis accurately estimates the attitude of the chest of a human subject using measurements from a single Inertial Measurement Unit (IMU). The signals processed by the observer are then converted into spectrograms to obtain “images” of the frequency response of the signals. The images for activities from a dataset of 7 human subjects are annotated and used for training/ fine-tuning of several well-known deep learning algorithms for image processing. The results from the best combination of our algorithms shows an exceptional accuracy of 98% for activity recognition. Using deep learning computer vision algorithms, this paper shows how to perform transfer learning from networks pre-trained on millions of images, thus showing how we can train a powerful deep learning network for activity recognition even with just small datasets. The algorithm that uses the high gain observer is shown to perform significantly better than an algorithm based on raw accelerometer and gyro signals.

Multi-View Human Action Recognition Using Skeleton Based-FineKNN with Extraneous Frame Scrapping Technique.

Human action recognition (HAR) is one of the most active research topics in the field of computer vision. Even though this area is well-researched, HAR algorithms such as 3D Convolution Neural Networks (CNN), Two-stream Networks, and CNN-LSTM (Long Short-Term Memory) suffer from highly complex models. These algorithms involve a huge number of weights adjustments during the training phase, and as a consequence, require high-end configuration machines for real-time HAR applications. Therefore, this paper presents an extraneous frame scrapping technique that employs 2D skeleton features with a Fine-KNN classifier-based HAR system to overcome the dimensionality problems.To illustrate the efficacy of our proposed method, two contemporary datasets i.e., Multi-Camera Action Dataset (MCAD) and INRIA Xmas Motion Acquisition Sequences (IXMAS) dataset was used in experiment. We used the OpenPose technique to extract the 2D information, The proposed method was compared with CNN-LSTM, and other State of the art methods. Results obtained confirm the potential of our technique. The proposed OpenPose-FineKNN with Extraneous Frame Scrapping Technique achieved an accuracy of 89.75% on MCAD dataset and 90.97% on IXMAS dataset better than existing technique.