Human Activity Recognition Model Research Articles

Human activity recognition for analyzing stress behavior based on Bi-LSTM.

Stress is one of the critical health factors that could be detected by Human Activity Recognition (HAR) which consists of physical and mental health. HAR can raise awareness of self-care and prevent critical situations. Recently, HAR used non-invasive wearable physiological sensors. Moreover, deep learning techniques are becoming a significant tool for analyzing health data. In this paper, we propose a human lifelog monitoring model for stress behavior recognition based on deep learning, which analyses stress levels during activity. The proposed approach considers activity and physiological data for recognizing physical activity and stress levels. To tackle these issues, we proposed a model that utilizes hand-crafted feature generation techniques compatible with a Bidirectional Long Short-Term Memory (Bi-LSTM) based method for physical activity and stress level recognition. We have used a dataset called WESAD, collected using wearable sensors for model evaluation. This dataset presented four levels of stress emotion, including baseline, amusement, stress, and meditation. The following results are from the hand-crafted feature approaches compatible with the bidirectional LSTM model. The proposed model achieves an accuracy of 95.6% and an F1-score of 96.6%. The proposed HAR model efficiently recognizes stress levels and contributes to maintaining physical and mental well-being.

Multivariate CNN Model for Human Locomotion Activity Recognition with a Wearable Exoskeleton Robot.

This study introduces a novel convolutional neural network (CNN) architecture, encompassing both single and multi-head designs, developed to identify a user's locomotion activity while using a wearable lower limb robot. Our research involved 500 healthy adult participants in an activities of daily living (ADL) space, conducted from 1 September to 30 November 2022. We collected prospective data to identify five locomotion activities (level ground walking, stair ascent/descent, and ramp ascent/descent) across three terrains: flat ground, staircase, and ramp. To evaluate the predictive capabilities of the proposed CNN architectures, we compared its performance with three other models: one CNN and two hybrid models (CNN-LSTM and LSTM-CNN). Experiments were conducted using multivariate signals of various types obtained from electromyograms (EMGs) and the wearable robot. Our results reveal that the deeper CNN architecture significantly surpasses the performance of the three competing models. The proposed model, leveraging encoder data such as hip angles and velocities, along with postural signals such as roll, pitch, and yaw from the wearable lower limb robot, achieved superior performance with an inference speed of 1.14 s. Specifically, the F-measure performance of the proposed model reached 96.17%, compared to 90.68% for DDLMI, 94.41% for DeepConvLSTM, and 95.57% for LSTM-CNN, respectively.

Hybrid Learning Models for IMU-Based HAR with Feature Analysis and Data Correction.

This paper proposes a novel approach to tackle the human activity recognition (HAR) problem. Four classes of body movement datasets, namely stand-up, sit-down, run, and walk, are applied to perform HAR. Instead of using vision-based solutions, we address the HAR challenge by implementing a real-time HAR system architecture with a wearable inertial measurement unit (IMU) sensor, which aims to achieve networked sensing and data sampling of human activity, data pre-processing and feature analysis, data generation and correction, and activity classification using hybrid learning models. Referring to the experimental results, the proposed system selects the pre-trained eXtreme Gradient Boosting (XGBoost) model and the Convolutional Variational Autoencoder (CVAE) model as the classifier and generator, respectively, with 96.03% classification accuracy.

Deep learning models for real-life human activity recognition from smartphone sensor data

Nowadays, the field of human activity recognition (HAR) is a remarkably hot topic within the scientific community. Given the low cost, ease of use and high accuracy of the sensors from different wearable devices and smartphones, more and more researchers are opting to do their bit in this area. However, until very recently, all the work carried out in this field was done in laboratory conditions, with very few similarities with our daily lives. This paper will focus on this new trend of integrating all the knowledge acquired so far into a real-life environment. Thus, a dataset already published following this philosophy was used. In this way, this work aims to be able to identify the different actions studied there. In order to perform this classification, this paper explores new designs and architectures for models inspired by the ones which have yielded the best results in the literature. More specifically, different configurations of Convolutional Neural Networks (CNN) and Long-Short Term Memory (LSTM) have been tested, but on real-life conditions instead of laboratory ones. It is worth mentioning that the hybrid models formed from these techniques yielded the best results, with a peak accuracy of 94.80% on the dataset used.

An Explainable EEG-Based Human Activity Recognition Model Using Machine-Learning Approach and LIME.

Electroencephalography (EEG) is a non-invasive method employed to discern human behaviors by monitoring the neurological responses during cognitive and motor tasks. Machine learning (ML) represents a promising tool for the recognition of human activities (HAR), and eXplainable artificial intelligence (XAI) can elucidate the role of EEG features in ML-based HAR models. The primary objective of this investigation is to investigate the feasibility of an EEG-based ML model for categorizing everyday activities, such as resting, motor, and cognitive tasks, and interpreting models clinically through XAI techniques to explicate the EEG features that contribute the most to different HAR states. The study involved an examination of 75 healthy individuals with no prior diagnosis of neurological disorders. EEG recordings were obtained during the resting state, as well as two motor control states (walking and working tasks), and a cognition state (reading task). Electrodes were placed in specific regions of the brain, including the frontal, central, temporal, and occipital lobes (Fz, C1, C2, T7, T8, Oz). Several ML models were trained using EEG data for activity recognition and LIME (Local Interpretable Model-Agnostic Explanations) was employed for interpreting clinically the most influential EEG spectral features in HAR models. The classification results of the HAR models, particularly the Random Forest and Gradient Boosting models, demonstrated outstanding performances in distinguishing the analyzed human activities. The ML models exhibited alignment with EEG spectral bands in the recognition of human activity, a finding supported by the XAI explanations. To sum up, incorporating eXplainable Artificial Intelligence (XAI) into Human Activity Recognition (HAR) studies may improve activity monitoring for patient recovery, motor imagery, the healthcare metaverse, and clinical virtual reality settings.

Conformer-Based Human Activity Recognition Using Inertial Measurement Units.

Human activity recognition (HAR) using inertial measurement units (IMUs) is gaining popularity due to its ease of use, accurate and reliable measurements of motion and orientation, and its suitability for real-time IoT applications such as healthcare monitoring, sports and fitness tracking, video surveillance and security, smart homes and assistive technologies, human-computer interaction, workplace safety, and rehabilitation and physical therapy. IMUs are widely used as they provide precise and consistent measurements of motion and orientation, making them an ideal choice for HAR. This paper proposes a Conformer-based HAR model that employs attention mechanisms to better capture the temporal dynamics of human movement and improve the recognition accuracy. The proposed model consists of convolutional layers, multiple Conformer blocks with self-attention and residual connections, and classification layers. Experimental results show that the proposed model outperforms existing models such as CNN, LSTM, and GRU. The attention mechanisms in the Conformer blocks have residual connections, which can prevent vanishing gradients and improve convergence. The model was evaluated using two publicly available datasets, WISDM and USCHAD, and achieved accuracy of 98.1% and 96%, respectively. These results suggest that Conformer-based models can offer a promising approach for HAR using IMU.

Exploration of Deep Learning Models for Video Based Multiple Human Activity Recognition

Human Activity Recognition (HAR) with Deep Learning is a challenging and a highly demanding classification task. Complexity of the activity detection and the number of subjects are the main issues. Data mining approaches improved decision-making performance. This work presents one such model for Human activity recognition for multiple subjects carrying out multiple activities. Involving real time datasets, the work developed a rapid algorithm for minimizing the problems of neural networks classifier. An optimal feature extraction happens and develops a multi-modal classification technique and predicts solutions with better accuracy when compared to other traditional methods. This paper discussing on HAR prediction in four phases namely (i) Depthwise Separable Convolution with BiLSTM (DSC-BLSTM); (ii) Enhanced Bidirectional Grated Recurrent Unit with Long Short Term Memory (BGRU-LSTM); (iii) Enhanced TimeSformer Model with Multi-Layer Perceptron Neural Networks classification and (iv) Filtering Single Activity Recognition are described.In comparison to previous efforts like the DSC-BLSTM and BGRU-LSTM classifications, the experimental result of the ETMLP classification attained 98.90%, which was more efficient. The end outcome revealed that the new model performed better in terms of accuracy than the other models.

Multi-Camera-Based Human Activity Recognition for Human-Robot Collaboration in Construction.

As the use of construction robots continues to increase, ensuring safety and productivity while working alongside human workers becomes crucial. To prevent collisions, robots must recognize human behavior in close proximity. However, single, or RGB-depth cameras have limitations, such as detection failure, sensor malfunction, occlusions, unconstrained lighting, and motion blur. Therefore, this study proposes a multiple-camera approach for human activity recognition during human-robot collaborative activities in construction. The proposed approach employs a particle filter, to estimate the 3D human pose by fusing 2D joint locations extracted from multiple cameras and applies long short-term memory network (LSTM) to recognize ten activities associated with human and robot collaboration tasks in construction. The study compared the performance of human activity recognition models using one, two, three, and four cameras. Results showed that using multiple cameras enhances recognition performance, providing a more accurate and reliable means of identifying and differentiating between various activities. The results of this study are expected to contribute to the advancement of human activity recognition and utilization in human-robot collaboration in construction.

A Framework for Personalized Human Activity Recognition

In today’s world, Human Activity Recognition (HAR) through video streams is actively used in every aspect of our life, such as automated surveillance systems and sports statistics are computed according to the videos with the help of HAR. Activity detection is not a new subject, and several methods are available. However, the most recent and most promising techniques rely on Convolutional Neural Networks (CNNs). CNNs primary usage is based on a single image frame to perform logical or categorical identification of an object, scene, or activity. We exploit this feature to adapt CNN on video streams to achieve HAR. In this study, we present a Personalized HAR (PHAR) framework that increases activity recognition accuracy with Object Detection (OD). First, we demonstrate the state-of-the-art HAR and OD methods in the literature. Then we illustrate our framework with two new Single Person Human Activity Recognition models. Finally, the performance of the new framework is evaluated with the well-known activity detection methods. Results show that our new PHAR model with 95% accuracy ratio outperforms the CNN-LSTM-based reference model (90%). Moreover, a new metric Average Accuracy Score (AAS) is described in this study, PHAR models approximately have 94% AAS, which is better than the reference model with 89% AAS.

YOLOv3-based human detection and heuristically modified-LSTM for abnormal human activities detection in ATM machine

In existing works, accurately analyzing human activities is a complicated problem in public places. Consequently, the detection of human activities becomes challenging the computer vision technology. The major scope of the research is to develop an abnormal Human Activity Recognition (HAR) model using deep structured architectures for detecting the suspicious activities of humans in the ATM using the video surveillance system. The classification phase utilizes the enhanced deep learning approach named improved Long Short-Term Memory (LSTM) by optimizing certain parameters in LSTM by hybrid optimization algorithm for accurately classifying the normal and abnormal activities of humans. This hybrid optimization algorithm is developed and termed Hybrid Spider Monkey-Chicken Swarm Optimization (HSM-CSO) for achieving the effective performance of the deep learning-based classification. Hence, the designed HAR model in ATM is proven that it helps to improve the system performance and also give relief from prohibited activities or crimes and false alarms for humans.

HAR-DeepConvLG: Hybrid deep learning-based model for human activity recognition in IoT applications

Smartphones and wearable devices have built-in sensors that can collect multivariant time-series data that can be used to recognize human activities. Research on human activity recognition (HAR) has gained significant attention in recent years due to its growing demand in various application domains. As wearable sensor-aided devices and the Internet of Things (IoT) became more common, great attention has been paid to the HAR ubiquitous computing and mobile computing. To infer human activity data from a massive amount of multivariant data generated by different wearable devices, in this study an innovative deep learning–based model named HAR-DeepConvLG is proposed. It includes three convolution layers and a squeezing and excitation (SE) block, which are employed to precisely learn and extract the spatial representation data from the collected raw sensor data. The extracted features are used as input of three parallel paths, each of which includes a long short-term memory (LSTM) layer connected in sequence with a gated recurrent unit (GRU) layer to learn temporal representation. The three paths are connected in parallel to avoid the vanishing gradient problem. Finally, to evaluate the effectiveness of the proposed model, experiments were conducted on four widely utilized HAR datasets. Additionally, the model’s performance was compared to several state-of-the-art deep learning models, which further validated its effectiveness. The experimental results show that the proposed HAR-DeepConvLG model performs better than the existing HAR deep learning–based models, achieving a competitive classification accuracy.

A study on light weight of radar-based human activity recognition model using self-attention mechanism

A study on light weight of radar-based human activity recognition model using self-attention mechanism

SWL-Adapt: An Unsupervised Domain Adaptation Model with Sample Weight Learning for Cross-User Wearable Human Activity Recognition

In practice, Wearable Human Activity Recognition (WHAR) models usually face performance degradation on the new user due to user variance. Unsupervised domain adaptation (UDA) becomes the natural solution to cross-user WHAR under annotation scarcity. Existing UDA models usually align samples across domains without differentiation, which ignores the difference among samples. In this paper, we propose an unsupervised domain adaptation model with sample weight learning (SWL-Adapt) for cross-user WHAR. SWL-Adapt calculates sample weights according to the classification loss and domain discrimination loss of each sample with a parameterized network. We introduce the meta-optimization based update rule to learn this network end-to-end, which is guided by meta-classification loss on the selected pseudo-labeled target samples. Therefore, this network can fit a weighting function according to the cross-user WHAR task at hand, which is superior to existing sample differentiation rules fixed for special scenarios. Extensive experiments on three public WHAR datasets demonstrate that SWL-Adapt achieves the state-of-the-art performance on the cross-user WHAR task, outperforming the best baseline by an average of 3.1% and 5.3% in accuracy and macro F1 score, respectively.

ConvBoost

Human activity recognition (HAR) is one of the core research themes in ubiquitous and wearable computing. With the shift to deep learning (DL) based analysis approaches, it has become possible to extract high-level features and perform classification in an end-to-end manner. Despite their promising overall capabilities, DL-based HAR may suffer from overfitting due to the notoriously small, often inadequate, amounts of labeled sample data that are available for typical HAR applications. In response to such challenges, we propose ConvBoost -- a novel, three-layer, structured model architecture and boosting framework for convolutional network based HAR. Our framework generates additional training data from three different perspectives for improved HAR, aiming to alleviate the shortness of labeled training data in the field. Specifically, with the introduction of three conceptual layers--Sampling Layer, Data Augmentation Layer, and Resilient Layer--we develop three "boosters"--R-Frame, Mix-up, and C-Drop--to enrich the per-epoch training data by dense-sampling, synthesizing, and simulating, respectively. These new conceptual layers and boosters, that are universally applicable for any kind of convolutional network, have been designed based on the characteristics of the sensor data and the concept of frame-wise HAR. In our experimental evaluation on three standard benchmarks (Opportunity, PAMAP2, GOTOV) we demonstrate the effectiveness of our ConvBoost framework for HAR applications based on variants of convolutional networks: vanilla CNN, ConvLSTM, and Attention Models. We achieved substantial performance gains for all of them, which suggests that the proposed approach is generic and can serve as a practical solution for boosting the performance of existing ConvNet-based HAR models. This is an open-source project, and the code can be found at https://github.com/sshao2013/ConvBoost

Online continual learning for human activity recognition

Sensor-based human activity recognition (HAR), with the ability to recognise human activities from wearable or embedded sensors, has been playing an important role in many applications including personal health monitoring, smart home, and manufacturing. The real-world, long-term deployment of these HAR systems drives a critical research question: how to evolve the HAR model automatically over time to accommodate changes in an environment or activity patterns. This paper presents an online continual learning (OCL) scenario for HAR, where sensor data arrives in a streaming manner which contains unlabelled samples from already learnt activities or new activities. We propose a technique, OCL-HAR, making a real-time prediction on the streaming sensor data while at the same time discovering and learning new activities. We have empirically evaluated OCL-HAR on four third-party, publicly available HAR datasets. Our results have shown that this OCL scenario is challenging to state-of-the-art continual learning techniques that have significantly underperformed. Our technique OCL-HAR has consistently outperformed them in all experiment setups, leading up to 0.17 and 0.23 improvements in micro and macro F1 scores.

Human Action Recognition using Deep Learning

Abstract: The aim of this project is to recognize human actions for monitoring and security purposes. This project is mainly focused on building a system that is helpful for doctors to monitor patients .Human Action Recognition is required to recognize a set of human activities by training a supervised learning model and displaying the activity/action result as per the input action received. It has wide range of applications such as patient monitoring system, ATM/ Bank security system, etc. Human Action Recognition model can be mainly used for security and monitoring purposes. We can use various machine learning and deep learning algorithms for this project . One of the best approach is using CNN .

Implementation of Convolutional Neural Network and Long Short-Term Memory Algorithms in Human Activity Recognition Based on Visual Processing Video

Human Activity Recognition (HAR) is an interesting research topic, especially in identifying human movement actions focusing on video-based security surveillance. Symptom of an illness from a movement. The use of HAR in this research is the key to better understanding the various semantics contained in the video to find out the pattern of a human movement, especially in sports movements. In this study, a combination of the CNN and LSTM method algorithms was applied by using several variations of the model parameter values on the dropout layer and batch size to convert the pattern in the video into image form to produce a HAR model. Data processing at the convolution layer is used to extract spatial features in the frame. The extraction results are fed to the LSTM layer on each network for modeling the temporal sequence of human movement. In this way, the network on the model will learn spatiotemporal features directly in end-to-end data training tests to produce a robust model. The test data used are 10 sports activities obtained from related research from the University of Central Florida (UCF). The results showed that the performance was quite good, although there were still errors in the classification of sports activities because they had similarities in the movements of the activities carried out. The classification results show a loss value of 0.4 and an accuracy of 0.94. In further research, what needs to be corrected is the loss value which is still high so that several times the test results show an error in the classification of sports activities that have similarities in the movements of the activities.

From Lab to Real World: Assessing the Effectiveness of Human Activity Recognition and Optimization through Personalization.

Human activity recognition (HAR) algorithms today are designed and evaluated on data collected in controlled settings, providing limited insights into their performance in real-world situations with noisy and missing sensor data and natural human activities. We present a real-world HAR open dataset compiled from a wristband equipped with a triaxial accelerometer. During data collection, participants had autonomy in their daily life activities, and the process remained unobserved and uncontrolled. A general convolutional neural network model was trained on this dataset, achieving a mean balanced accuracy (MBA) of 80%. Personalizing the general model through transfer learning can yield comparable and even superior results using fewer data, with the MBA improving to 85%. To emphasize the issue of insufficient real-world training data, we conducted training of the model using the public MHEALTH dataset, resulting in 100% MBA. However, upon evaluating the MHEALTH-trained model on our real-world dataset, the MBA drops to 62%. After personalizing the model with real-world data, an improvement of 17% in the MBA is achieved. This paper showcases the potential of transfer learning to make HAR models trained in different contexts (lab vs. real-world) and on different participants perform well for new individuals with limited real-world labeled data available.

A Lightweight Attentional Shift Graph Convolutional Network for Skeleton-Based Action Recognition

In the field of skeleton-based human behavior recognition, graph convolutional neural networks have made remarkable achievements. However, high precision networks are often accompanied by numerous parameters and computational cost, and their application in mobile devices has considerable limitations. Aiming at the problem of excessive spatiotemporal complexity of high-accuracy methods, this paper further analyzes the lightweight human action recognition model and proposes a lightweight architecture attentional shift graph convolutional network. There are three main improvements in this model. Firstly, shift convolution is a lightweight convolution method that can be combined with graph convolution to effectively reduce its complexity. At the same time, a shallow architecture for multi-stream early fusion is designed to reduce the network scale by merging multi-stream networks and reducing the number of network layers. In addition, the efficient channel attention module is introduced into the model to capture the underlying characteristic information in the channel domain. Experiments are conducted on the three existing skeleton datasets, NTU RGB+D, NTU-120 RGB+D, and Northwestern-UCLA. Results demonstrate that the proposed model is not only competitive in accuracy, but also outperforms current mainstream methods in parameter count and computational cost, and supports running in some devices with limited computing and storage resources.

Enhanced Prediction Model for Human Activity Using an End-to-End Approach

Human activity recognition (HAR) based on ambient sensors aims to recognize a conducted activity. A large number of deep learning models (DLMs) for HAR have been proposed. In contrast, human activity prediction (HAP) aims to early predict an activity. Compared to HAR, the advantage of HAP is to prevent a person from being exposed to unexpected cases by early predicting the activity. However, few DLMs for HAP have been proposed. They predict the next activity via a non-end-to-end fashion, e.g., they take a sequence of consecutive activities where the activities were classified from the sensor information. Thus, the information that which sensors are activated is not used in the prediction. In this study, we propose an end-to-end HAP model to predict the next activity from a sequence of consecutive events. The model has an encoder, a classifier, and a regressor. The encoder gives an encoded vector by encoding events. The regressor learns temporal dependency from a sequence of encoded vectors to predict the next encoded vector. The classifier predicts the next activity using the next encoded vector. We use the Milan and Aruba datasets to study a prediction accuracy of the model. We compare our model with a non-end-to-end model based on long-term memory, taking a sequence of past activities. We show that our model achieves the better prediction accuracy than the non-end-to-end model by up to 4.73% and 7.39% for Milan and Aruba, respectively, meaning that the information related to events can be used in the prediction.