Human Action Recognition Algorithm Research Articles

A Lightweight Deep Human Activity Recognition Algorithm Using Multiknowledge Distillation

A Lightweight Deep Human Activity Recognition Algorithm Using Multiknowledge Distillation

Revolutionizing health monitoring: Integrating transformer models with multi-head attention for precise human activity recognition using wearable devices.

A daily activity routine is vital for overall health and well-being, supporting physical and mental fitness. Consistent physical activity is linked to a multitude of benefits for the body, mind, and emotions, playing a key role in raising a healthy lifestyle. The use of wearable devices has become essential in the realm of health and fitness, facilitating the monitoring of daily activities. While convolutional neural networks (CNN) have proven effective, challenges remain in quickly adapting to a variety of activities. This study aimed to develop a model for precise recognition of human activities to revolutionize health monitoring by integrating transformer models with multi-head attention for precise human activity recognition using wearable devices. The Human Activity Recognition (HAR) algorithm uses deep learning to classify human activities using spectrogram data. It uses a pretrained convolution neural network (CNN) with a MobileNetV2 model to extract features, a dense residual transformer network (DRTN), and a multi-head multi-level attention architecture (MH-MLA) to capture time-related patterns. The model then blends information from both layers through an adaptive attention mechanism and uses a SoftMax function to provide classification probabilities for various human activities. The integrated approach, combining pretrained CNN with transformer models to create a thorough and effective system for recognizing human activities from spectrogram data, outperformed these methods in various datasets - HARTH, KU-HAR, and HuGaDB produced accuracies of 92.81%, 97.98%, and 95.32%, respectively. This suggests that the integration of diverse methodologies yields good results in capturing nuanced human activities across different activities. The comparison analysis showed that the integrated system consistently performs better for dynamic human activity recognition datasets. In conclusion, maintaining a routine of daily activities is crucial for overall health and well-being. Regular physical activity contributes substantially to a healthy lifestyle, benefiting both the body and the mind. The integration of wearable devices has simplified the monitoring of daily routines. This research introduces an innovative approach to human activity recognition, combining the CNN model with a dense residual transformer network (DRTN) with multi-head multi-level attention (MH-MLA) within the transformer architecture to enhance its capability.

IoT-FAR: A multi-sensor fusion approach for IoT-based firefighting activity recognition

Inadequate training poses a significant risk of injury among young firefighters. Although Human Activity Recognition (HAR) algorithms have shown potential in monitoring and evaluating performance, most existing studies focus on daily activities and have difficulty distinguishing complex firefighting tasks. This study introduces the Internet of things (IoT)-based wearable firefighting activity recognition (IoT-FAR) system which employs a multi-modal sensor fusion approach to achieve comprehensive activity recognition during firefighting training. The IoT-FAR comprises five wearable body sensor nodes and a coordinator node. This study explores the significance of features extracted from the surface electromyography, heart rate, and inertial measurement units in firefighting training activity recognition. A hybrid machine learning (HML)-based network is proposed, which integrates three models: one trained with all features (MA), another with upper body features (MU), and a third with lower body features (ML). The proposed HML-SVM-RBF1-RF2 network achieves superior performance, with a mean recall of 93.94%, mean precision of 90.94%, and a mean accuracy rate of 98.29%. Additionally, the study introduces the specialized firefighting training associated activities (SFTAA) dataset, which includes endurance training activities involving self-contained breathing apparatus (SCBA) conducted by eighteen firefighters. This dataset represents preliminary work towards building a comprehensive dataset covering various events and scenarios for tracking firefighter activities. The IoT-FAR system also demonstrates the potential use of misclassified activities as evaluation metrics for assessing firefighter training performance.

MF-Match: A Semi-Supervised Model for Human Action Recognition.

Human action recognition (HAR) technology based on radar signals has garnered significant attention from both industry and academia due to its exceptional privacy-preserving capabilities, noncontact sensing characteristics, and insensitivity to lighting conditions. However, the scarcity of accurately labeled human radar data poses a significant challenge in meeting the demand for large-scale training datasets required by deep model-based HAR technology, thus substantially impeding technological advancements in this field. To address this issue, a semi-supervised learning algorithm, MF-Match, is proposed in this paper. This algorithm computes pseudo-labels for larger-scale unsupervised radar data, enabling the model to extract embedded human behavioral information and enhance the accuracy of HAR algorithms. Furthermore, the method incorporates contrastive learning principles to improve the quality of model-generated pseudo-labels and mitigate the impact of mislabeled pseudo-labels on recognition performance. Experimental results demonstrate that this method achieves action recognition accuracies of 86.69% and 91.48% on two widely used radar spectrum datasets, respectively, utilizing only 10% labeled data, thereby validating the effectiveness of the proposed approach.

Unsupervised deep domain adaptation algorithm for video based human activity recognition via recurrent neural networks

The integration of cyber-physical systems and artificial intelligent human activity recognition (HAR) applications enables intelligent interactions within a physical environment. In real-world HAR applications, the domain shift between training (source) and testing (target) images captured in different scenarios leads to low classification accuracy. Existing unsupervised domain adaptation (UDA) methods often require some labeled target samples for model adaptation, which limits their practicality. This study proposes a novel unsupervised deep domain adaptation algorithm (UDDAA) for HAR using recurrent neural networks. UDDAA introduces a maximum mean class discrepancy (MMCD) metric, accounting for both inter-class and intra-class differences within each domain. MMCD extends the maximum mean discrepancy to measure the class-level distribution discrepancy across source and target domains, aligning these distributions to enhance domain adaptation performance. Without relying on labeled target data, UDDAA predicts pseudo-labels for the unlabeled target dataset, combining these with labeled source data to train the model on domain-invariant representations. This approach makes UDDAA highly practical for scenarios where labeled target data is difficult or expensive to obtain, enabling human-computer interaction (HCI) systems to function effectively across varied environments and user behaviors. Extensive experiments on benchmark datasets demonstrate UDDAA's superior classification accuracy over existing baselines. Notably, UDDAA achieved 92% and 99% accuracy for University of Central Florida database (UCF) to Human Motion Database (HMDB) and HMDB to UCF transfers, respectively. Additionally, on personal recorded videos with complex backgrounds, it achieved high classification accuracies of 95% for basketball and 90% for football activities, underscoring its generalization ability, robustness, and effectiveness.

Human Activity Recognition Algorithm with Physiological and Inertial Signals Fusion: Photoplethysmography, Electrodermal Activity, and Accelerometry.

Inertial signals are the most widely used signals in human activity recognition (HAR) applications, and extensive research has been performed on developing HAR classifiers using accelerometer and gyroscope data. This study aimed to investigate the potential enhancement of HAR models through the fusion of biological signals with inertial signals. The classification of eight common low-, medium-, and high-intensity activities was assessed using machine learning (ML) algorithms, trained on accelerometer (ACC), blood volume pulse (BVP), and electrodermal activity (EDA) data obtained from a wrist-worn sensor. Two types of ML algorithms were employed: a random forest (RF) trained on features; and a pre-trained deep learning (DL) network (ResNet-18) trained on spectrogram images. Evaluation was conducted on both individual activities and more generalized activity groups, based on similar intensity. Results indicated that RF classifiers outperformed corresponding DL classifiers at both individual and grouped levels. However, the fusion of EDA and BVP signals with ACC data improved DL classifier performance compared to a baseline DL model with ACC-only data. The best performance was achieved by a classifier trained on a combination of ACC, EDA, and BVP images, yielding F1-scores of 69 and 87 for individual and grouped activity classifications, respectively. For DL models trained with additional biological signals, almost all individual activity classifications showed improvement (p-value < 0.05). In grouped activity classifications, DL model performance was enhanced for low- and medium-intensity activities. Exploring the classification of two specific activities, ascending/descending stairs and cycling, revealed significantly improved results using a DL model trained on combined ACC, BVP, and EDA spectrogram images (p-value < 0.05).

Self-supervised learning for human activity recognition using 700,000 person-days of wearable data

Accurate physical activity monitoring is essential to understand the impact of physical activity on one’s physical health and overall well-being. However, advances in human activity recognition algorithms have been constrained by the limited availability of large labelled datasets. This study aims to leverage recent advances in self-supervised learning to exploit the large-scale UK Biobank accelerometer dataset—a 700,000 person-days unlabelled dataset—in order to build models with vastly improved generalisability and accuracy. Our resulting models consistently outperform strong baselines across eight benchmark datasets, with an F1 relative improvement of 2.5–130.9% (median 24.4%). More importantly, in contrast to previous reports, our results generalise across external datasets, cohorts, living environments, and sensor devices. Our open-sourced pre-trained models will be valuable in domains with limited labelled data or where good sampling coverage (across devices, populations, and activities) is hard to achieve.

Research on human behavior recognition algorithm based on HRNet and LSTM with model lightweight

This paper proposes a lightweight human action recognition algorithm for pedestrian behavior recognition. First, the skeleton feature information is input into the HRNet network model. In order to selectively enhance more details containing the target features and suppress irrelevant or weak features, an external attention mechanism is added to the HRN child model. Secondly, in order to extract the temporal characteristics of the target feature vector and ensure the continuity of actions in human behavior recognition, a dual-stream network based on HRNet and Long Short-Term Memory (LSTM) is constructed; finally, due to the huge model, it cannot be well transplanted to embedded. Therefore, this paper uses depthwise separable convolution to lightweight the network model. The experimental results show that in terms of human behavior recognition, the method in this paper has better recognition accuracy than Two-stream, Multi-streamCNN, Cov3DJ, ConvNets, JTM, ASM-3, RF+SW, hd-CNN and TPSMMs.

A micro-Doppler spectrogram denoising algorithm for radar human activity recognition

Radar signal recognition based on micro-Doppler spectrogram has been widely used in human action recognition tasks. However, in practical application scenarios, radar signals inevitably have noise, which leads to different degrees of deformation of the spectrogram graph structure and affects the accuracy of subsequent recognition algorithms. In this paper, we present “ACFL”, a novel algorithm for micro-Doppler spectrogram denoising, which aims to reduce the impact of noise on human action recognition. ACFL employs amplitude–frequency two-dimensional clustering and fuzzy logic clustering selection mechanism to remove noise elements from the spectrogram. Moreover, to address the issue of noise leakage or target missing under time-varying noise and action conditions, ACFL adopts spectrogram segmentation based on short-term Rényi entropy. By dividing the spectrogram into intervals with different time–frequency distributions, the dynamic spectrogram denoise over time is achieved. Simulation and measured data experiments demonstrate that the proposed algorithm not only achieves a higher-quality denoised spectrogram but also significantly improves the accuracy of human action recognition under noisy conditions.

HARDVS: Revisiting Human Activity Recognition with Dynamic Vision Sensors

The main streams of human activity recognition (HAR) algorithms are developed based on RGB cameras which usually suffer from illumination, fast motion, privacy preservation, and large energy consumption. Meanwhile, the biologically inspired event cameras attracted great interest due to their unique features, such as high dynamic range, dense temporal but sparse spatial resolution, low latency, low power, etc. As it is a newly arising sensor, even there is no realistic large-scale dataset for HAR. Considering its great practical value, in this paper, we propose a large-scale benchmark dataset to bridge this gap, termed HARDVS, which contains 300 categories and more than 100K event sequences. We evaluate and report the performance of multiple popular HAR algorithms, which provide extensive baselines for future works to compare. More importantly, we propose a novel spatial-temporal feature learning and fusion framework, termed ESTF, for event stream based human activity recognition. It first projects the event streams into spatial and temporal embeddings using StemNet, then, encodes and fuses the dual-view representations using Transformer networks. Finally, the dual features are concatenated and fed into a classification head for activity prediction. Extensive experiments on multiple datasets fully validated the effectiveness of our model. Both the dataset and source code will be released at https://github.com/Event-AHU/HARDVS.

Human Activity Recognition Based On Video Summarization And Deep Convolutional Neural Network

Abstract In this technological era, human activity recognition (HAR) plays a significant role in several applications like surveillance, health services, Internet of Things, etc. Recent advancements in deep learning and video summarization have motivated us to integrate these techniques for HAR. This paper introduces a computationally efficient HAR technique based on a deep learning framework, which works well in realistic and multi-view environments. Deep convolutional neural networks (DCNNs) normally suffer from different constraints, including data size dependencies, computational complexity, overfitting, training challenges and vanishing gradients. Additionally, with the use of advanced mobile vision devices, the demand for computationally efficient HAR algorithms with the requirement of limited computational resources is high. To address these issues, we used integration of DCNN with video summarization using keyframes. The proposed technique offers a solution that enhances performance with efficient resource utilization. For this, first, we designed a lightweight and computationally efficient deep learning architecture based on the concept of identity skip connections (features reusability), which preserves the gradient loss attenuation and can handle the enormous complexity of activity classes. Subsequently, we employed an efficient keyframe extraction technique to minimize redundancy and succinctly encapsulate the entire video content in a lesser number of frames. To evaluate the efficacy of the proposed method, we performed the experimentation on several publicly available datasets. The performance of the proposed method is measured in terms of evaluation parameters Precision, Recall, F-Measure and Classification Accuracy. The experimental results demonstrated the superiority of the presented algorithm over other existing state-of-the-art methods.

Outdoor activity classification using smartphone based inertial sensor measurements

Human Activity Recognition (HAR) deals with the automatic recognition of physical activities and plays a crucial role in healthcare and sports where wearable sensors and intelligent computational techniques are used. We propose a HAR algorithm that uses the smartphones accelerometer data for human activity recognition. In particular, we present a recurrent convolutional neural network-based HAR algorithm that combines a Convolutional Neural Network (CNN) to extract temporal features from the sensor data, a Fuzzy C-Means (FCM) clustering algorithm to cluster the features extracted by the CNN, and a Long Short-Term Memory (LSTM) network to learn the temporal dependencies between the features. We evaluate the proposed methodology on two distinct datasets: the MotionSense dataset and the WISDM dataset. We evaluate the proposed CNN-FCM-LSTM model on the publicly available MotionSense dataset to classify ten activity types: 1) walking upstairs, 2) walking downstairs, 3) jogging, 4) sitting, 5) standing, 6) level ground walking, 7) jumping jacks, 8) brushing teeth, 9) writing, and 10) eating. Next, we evaluate the model’s performance on the WISDM dataset to assess its ability to generalize to unseen data. On the MotionSense test dataset, CNN-FCM-LSTM achieves a classification accuracy of 99.69%, a sensitivity of 99.62%, a specificity of 99.63%, and a false positive rate per hour (FPR/h) of 0.37%. Meanwhile, it achieves a classification accuracy of 97.27% on the WISDM dataset. The CNN-FCM-LSTM model’s capability to classify a diverse range of activities within a single architecture is noteworthy. The results suggest that the proposed CNN-FCM-LSTM model using smartphone inputs is more accurate, reliable, and robust in detecting and classifying activities than the state-of-the-art models. It should be noted that activity recognition technology has the potential to aid in studying the underpinnings of physical activity, designing more effective training regimens, and simulating the rigors of competition in sports.

Video Abnormal Action Recognition Based on Multimodal Heterogeneous Transfer Learning

Human abnormal action recognition is crucial for video understanding and intelligent surveillance. However, the scarcity of labeled data for abnormal human actions often hinders the development of high-performance models. Inspired by the multimodal approach, this paper proposes a novel approach that leverages text descriptions associated with abnormal human action videos. Our method exploits the correlation between the text domain and the video domain in the semantic feature space and introduces a multimodal heterogeneous transfer learning framework from the text domain to the video domain. The text of the videos is used for feature encoding and knowledge extraction, and knowledge transfer and sharing are realized in the feature space, which is used to assist in the training of the abnormal action recognition model. The proposed method reduces the reliance on labeled video data, improves the performance of the abnormal human action recognition algorithm, and outperforms the popular video-based models, particularly in scenarios with sparse data. Moreover, our framework contributes to the advancement of automatic video analysis and abnormal action recognition, providing insights for the application of multimodal methods in a broader context.

A Projected OMP-Hybridized Discriminative K-SVD based Dictionary Learning Algorithm for Human Activity Recognition from Accelerometer Signals

A Projected OMP-Hybridized Discriminative K-SVD based Dictionary Learning Algorithm for Human Activity Recognition from Accelerometer Signals

Self-attention deep ConvLSTM with sparse-learned channel dependencies for wearable sensor-based human activity recognition

In this study, we propose a novel deep-learning architecture with sparse learning for human activity recognition. The proposed model contains 1D CNNs and LSTM layers with a self-attention mechanism to enhance a substantial number of time points in time-series data for human activity recognition systems. Based on the recent success of squeeze-and-excite (SE) networks, the proposed deep learning model utilizes the SE module to enhance channel-wise interdependencies, which in turn leads to a boost in performance. In addition, we utilized sparse learning to retrain only weak nodes and freeze stronger nodes in a fully connected layer prior to classification layer. Furthermore, we utilized an entropy-inspired formula to find sparsely located weaker nodes and validated our model on various datasets, including Opportunity, UCI-HAR, and WISDM. Herein, we present an extensive analysis and survey of state-of-the-art studies, in addition to our proposed research. For a fair comparison, we evaluated our deep learning architecture using various performance metrics and achieved better results; the proposed model outperformed state-of-the-art algorithms for human activity recognition.

Design and Development of an Imitation Detection System for Human Action Recognition Using Deep Learning.

Human action recognition (HAR) is a rapidly growing field with numerous applications in various domains. HAR involves the development of algorithms and techniques to automatically identify and classify human actions from video data. Accurate recognition of human actions has significant implications in fields such as surveillance and sports analysis and in the health care domain. This paper presents a study on the design and development of an imitation detection system using an HAR algorithm based on deep learning. This study explores the use of deep learning models, such as a single-frame convolutional neural network (CNN) and pretrained VGG-16, for the accurate classification of human actions. The proposed models were evaluated using a benchmark dataset, KTH. The performance of these models was compared with that of classical classifiers, including K-Nearest Neighbors, Support Vector Machine, and Random Forest. The results showed that the VGG-16 model achieved higher accuracy than the single-frame CNN, with a 98% accuracy rate.

A novel algorithm for human action recognition in compressed domain using attention-guided approach

A novel algorithm for human action recognition in compressed domain using attention-guided approach

Spatio-temporal features for fast early warning of unplanned self-extubation in ICU

Patients’ behaviors in the Intensive Care Units (ICU) have garnered research attention, particularly regarding the impact of Unplanned Extubation (UEX). However, there is currently no existing report on methods for early warning of UEX action in RGB video. Applying traditional human action recognition algorithms to UEX in the complex ICU environment proves challenging. To address the above issue, we propose a novel feature for early warning of UEX action in patients using RGB videos. Firstly, we employ the YOLOv3 detection method to extract the region of interest (ROI), which corresponds to the region where the patient is located. Subsequently, we develop a spatio-temporal (ST) feature for human action tracking by using the L-K optical flow algorithm. This ST feature encompasses optical flow corner number, trajectory distance, and wavelet transform features. Finally, we utilize support vector machine (SVM) for patient action classification and early warning. Experimental results on the ICU monitoring dataset demonstrate the superior performance of the proposed feature in UEX prediction.

ENGA: Elastic Net-Based Genetic Algorithm for human action recognition

ENGA: Elastic Net-Based Genetic Algorithm for human action recognition

A Novel Multiperson Activity Recognition Algorithm Based on Point Clouds Measured by Millimeter-Wave MIMO Radar

Radar is widely used in human activity recognition (HAR). The radar-based HAR algorithms can be roughly categorized into the feature map-based and the point cloud-based. Between the two, the point cloud-based algorithms are more suitable for multi-person activity recognition (MPAR) tasks due to the spatial properties of point clouds, and are therefore increasingly attracting attentions. However, most existing point cloud-based algorithms make decisions by utilizing either Doppler or coordinate features of point clouds, which are not sufficient to characterize the activity; the remaining algorithms have a large randomness in extracting both features. Neither of these algorithms can fully exploit the advantage of feature fusion, which is the key to recognition performance. To address this shortcoming, this paper proposes a novel MPAR algorithm. First, a feature mapping approach is proposed and defined in equation form, the Doppler, range, azimuth, and elevation features of the point clouds are calculated and accumulated sequentially, so we can obtain the four time-domain feature maps. Second, with the four feature maps as inputs, a four-channel CNN classification model with channel attention is trained for MPAR tasks. Datasets of multi-person activities are collected respectively under the indoor circumstance and aquatic circumstance using the millimeter-wave multiple-input-multiple-output (MIMO) radar platform. The dataset-based evaluation performance shows that the proposed algorithm achieves accuracy results of 96.09% for the indoor-MPAR task and 93.97% for the aquatic-MPAR task, and outperforms three conventional point cloud-based algorithms in terms of the overall MPAR accuracy, the generalization ability to the aquatic activity recognition, and the robustness of distance and headcount.