Human Activity Recognition Model Research Articles

X-CHAR: A Concept-based Explainable Complex Human Activity Recognition Model.

End-to-end deep learning models are increasingly applied to safety-critical human activity recognition (HAR) applications, e.g., healthcare monitoring and smart home control, to reduce developer burden and increase the performance and robustness of prediction models. However, integrating HAR models in safety-critical applications requires trust, and recent approaches have aimed to balance the performance of deep learning models with explainable decision-making for complex activity recognition. Prior works have exploited the compositionality of complex HAR (i.e., higher-level activities composed of lower-level activities) to form models with symbolic interfaces, such as concept-bottleneck architectures, that facilitate inherently interpretable models. However, feature engineering for symbolic concepts-as well as the relationship between the concepts-requires precise annotation of lower-level activities by domain experts, usually with fixed time windows, all of which induce a heavy and error-prone workload on the domain expert. In this paper, we introduce X-CHAR , an eXplainable Complex Human Activity Recognition model that doesn't require precise annotation of low-level activities, offers explanations in the form of human-understandable, high-level concepts, while maintaining the robust performance of end-to-end deep learning models for time series data. X-CHAR learns to model complex activity recognition in the form of a sequence of concepts. For each classification, X-CHAR outputs a sequence of concepts and a counterfactual example as the explanation. We show that the sequence information of the concepts can be modeled using Connectionist Temporal Classification (CTC) loss without having accurate start and end times of low-level annotations in the training dataset-significantly reducing developer burden. We evaluate our model on several complex activity datasets and demonstrate that our model offers explanations without compromising the prediction accuracy in comparison to baseline models. Finally, we conducted a mechanical Turk study to show that the explanations provided by our model are more understandable than the explanations from existing methods for complex activity recognition.

Device Orientation Independent Human Activity Recognition Model for Patient Monitoring Based on Triaxial Acceleration

Tracking a person’s activities is relevant in a variety of contexts, from health and group-specific assessments, such as elderly care, to fitness tracking and human–computer interaction. In a clinical context, sensor-based activity tracking could help monitor patients’ progress or deterioration during their hospitalization time. However, during routine hospital care, devices could face displacements in their position and orientation caused by incorrect device application, patients’ physical peculiarities, or patients’ day-to-day free movement. These aspects can significantly reduce algorithms’ performances. In this work, we investigated how shifts in orientation could impact Human Activity Recognition (HAR) classification. To reach this purpose, we propose an HAR model based on a single three-axis accelerometer that can be located anywhere on the participant’s trunk, capable of recognizing activities from multiple movement patterns, and, thanks to data augmentation, can deal with device displacement. Developed models were trained and validated using acceleration measurements acquired in fifteen participants, and tested on twenty-four participants, of which twenty were from a different study protocol for external validation. The obtained results highlight the impact of changes in device orientation on a HAR algorithm and the potential of simple wearable sensor data augmentation for tackling this challenge. When applying small rotations (<20 degrees), the error of the baseline non-augmented model steeply increased. On the contrary, even when considering rotations ranging from 0 to 180 along the frontal axis, our model reached a f1-score of 0.85±0.11 against a baseline model f1-score equal to 0.49±0.12.

A Novel Human Activity Recognition Model for Smartphone Authentication

A Novel Human Activity Recognition Model for Smartphone Authentication

Development of a Novel Lightweight CNN Model for Classification of Human Actions in UAV-Captured Videos

There has been increased attention paid to autonomous unmanned aerial vehicles (UAVs) recently because of their usage in several fields. Human action recognition (HAR) in UAV videos plays an important role in various real-life applications. Although HAR using UAV frames has not received much attention from researchers to date, it is still a significant area that needs further study because of its relevance for the development of efficient algorithms for autonomous drone surveillance. Current deep-learning models for HAR have limitations, such as large weight parameters and slow inference speeds, which make them unsuitable for practical applications that require fast and accurate detection of unusual human actions. In response to this problem, this paper presents a new deep-learning model based on depthwise separable convolutions that has been designed to be lightweight. Other parts of the HarNet model comprised convolutional, rectified linear unit, dropout, pooling, padding, and dense blocks. The effectiveness of the model has been tested using the publicly available UCF-ARG dataset. The proposed model, called HarNet, has enhanced the rate of successful classification. Each unit of frame data was pre-processed one by one by different computer vision methods before it was incorporated into the HarNet model. The proposed model, which has a compact architecture with just 2.2 million parameters, obtained a 96.15% success rate in classification, outperforming the MobileNet, Xception, DenseNet201, Inception-ResNetV2, VGG-16, and VGG-19 models on the same dataset. The proposed model had numerous key advantages, including low complexity, a small number of parameters, and high classification performance. The outcomes of this paper showed that the model’s performance was superior to that of other models that used the UCF-ARG dataset.

Single Accelerometer to Recognize Human Activities Using Neural Networks.

Exoskeletons have decreased physical effort and increased comfort in activities of daily living (ADL) such as walking, squatting, and running. However, this assistance is often activity specific and does not accommodate a wide variety of different activities. To overcome this limitation and increase the scope of exoskeleton application, an automatic human activity recognition (HAR) system is necessary. We developed two deep-learning models for HAR using one-dimensional-convolutional neural network (CNN) and a hybrid model using CNNs and long-short term memory (LSTM). We trained both models using the data collected from a single three-axis accelerometer placed on the chest of ten subjects. We were able to classify five different activities, standing, walking on level ground, walking on an incline, running, and squatting, with an accuracy of 98.1% and 97.8%, respectively. A two subject real-time validation trial was also conducted to validate the real-time applicability of the system. The real-time accuracy was measured at 96.6% and 97.2% for the CNN and the hybrid model, respectively. The high classification accuracy in the test and real-time evaluation suggests that a single sensor could distinguish human activities using machine-learning-based models.

Framework for deep learning based model for human activity recognition (HAR) using adapted PSRA6 dataset

Framework for deep learning based model for human activity recognition (HAR) using adapted PSRA6 dataset

Wrapper-based deep feature optimization for activity recognition in the wearable sensor networks of healthcare systems

The Human Activity Recognition (HAR) problem leverages pattern recognition to classify physical human activities as they are captured by several sensor modalities. Remote monitoring of an individual’s activities has gained importance due to the reduction in travel and physical activities during the pandemic. Research on HAR enables one person to either remotely monitor or recognize another person’s activity via the ubiquitous mobile device or by using sensor-based Internet of Things (IoT). Our proposed work focuses on the accurate classification of daily human activities from both accelerometer and gyroscope sensor data after converting into spectrogram images. The feature extraction process follows by leveraging the pre-trained weights of two popular and efficient transfer learning convolutional neural network models. Finally, a wrapper-based feature selection method has been employed for selecting the optimal feature subset that both reduces the training time and improves the final classification performance. The proposed HAR model has been tested on the three benchmark datasets namely, HARTH, KU-HAR and HuGaDB and has achieved 88.89%, 97.97% and 93.82% respectively on these datasets. It is to be noted that the proposed HAR model achieves an improvement of about 21%, 20% and 6% in the overall classification accuracies while utilizing only 52%, 45% and 60% of the original feature set for HuGaDB, KU-HAR and HARTH datasets respectively. This proves the effectiveness of our proposed wrapper-based feature selection HAR methodology.

Human Activity Recognition for the Identification of Bullying and Cyberbullying Using Smartphone Sensors

The smartphone is an excellent source of data; it is possible to extrapolate smartphone sensor values and, through Machine Learning approaches, perform anomaly detection analysis characterized by human behavior. This work exploits Human Activity Recognition (HAR) models and techniques to identify human activity performed while filling out a questionnaire via a smartphone application, which aims to classify users as Bullying, Cyberbullying, Victims of Bullying, and Victims of Cyberbullying. The purpose of the work is to discuss a new smartphone methodology that combines the final label elicited from the cyberbullying/bullying questionnaire (Bully, Cyberbully, Bullying Victim, and Cyberbullying Victim) and the human activity performed (Human Activity Recognition) while the individual fills out the questionnaire. The paper starts with a state-of-the-art analysis of HAR to arrive at the design of a model that could recognize everyday life actions and discriminate them from actions resulting from alleged bullying activities. Five activities were considered for recognition: Walking, Jumping, Sitting, Running and Falling. The best HAR activity identification model then is applied to the Dataset derived from the “Smartphone Questionnaire Application” experiment to perform the analysis previously described.

HARDeep: design and evaluation of a deep ensemble model for human activity recognition

HARDeep: design and evaluation of a deep ensemble model for human activity recognition

HARDeep: Design and Evaluation of a Deep Ensemble Model for Human Activity Recognition

HARDeep: Design and Evaluation of a Deep Ensemble Model for Human Activity Recognition

A Comparative Study on Deep Learning and Machine Learning Models for Human Action Recognition in Aerial Videos

Unmanned Aerial Vehicle )UAV( finds its significant application in video surveillance due to its low cost, high portability and fast-mobility. In this paper, the proposed approach focuses on recognizing the human activity in aerial video sequences through various keypoints detected on the human body via OpenPose. The detected keypoints are passed onto machine learning and deep learning classifiers for classifying the human actions. Experimental results demonstrate that multilayer perceptron and SVM outperformed all the other classifiers by reporting an accuracy of 87.80% and 87.77% respectively whereas LSTM did not produce very good results as compared to other classifiers. Stacked Long Short-Term Memory networks (LSTM( produced an accuracy of 71.30% and Bidirectional LSTM yielded an accuracy of 76.04%. The results also indicate that machine learning models performed better than deep learning models. The major reason for this finding is the lesser availability of data and the deep learning models being data hungry models require a large amount of data to work upon. The paper also analyses the failure cases of OpenPose by testing the system on aerial videos captured by a drone flying at a higher altitude. This work provides a baseline for validating machine learning classifiers and deep learning classifiers against recognition of human action from aerial videos.

B2C-AFM: Bi-directional Co-Temporal and Cross-Spatial Attention Fusion Model for Human Action Recognition.

Human Action Recognition plays a driving engine of many human-computer interaction applications. Most current researches focus on improving the model generalization by integrating multiple homogeneous modalities, including RGB images, human poses, and optical flows. Furthermore, contextual interactions and out-of-context sign languages have been validated to depend on scene category and human per se. Those attempts to integrate appearance features and human poses have shown positive results. However, with human poses' spatial errors and temporal ambiguities, existing methods are subject to poor scalability, limited robustness, and sub-optimal models. In this paper, inspired by the assumption that different modalities may maintain temporal consistency and spatial complementarity, we present a novel Bi-directional Co-temporal and Cross-spatial Attention Fusion Model (B2C-AFM). Our model is characterized by the asynchronous fusion strategy of multi-modal features along temporal and spatial dimensions. Besides, the novel explicit motion-oriented pose representations called Limb Flow Fields (Lff) are explored to alleviate the temporal ambiguity regarding human poses. Experiments on publicly available datasets validate our contributions. Abundant ablation studies experimentally show that B2C-AFM achieves robust performance across seen and unseen human actions. The codes are available here1.

Modified Wild Horse Optimization with Deep Learning Enabled Symmetric Human Activity Recognition Model

Traditional indoor human activity recognition (HAR) is a time-series data classification problem and needs feature extraction. Presently, considerable attention has been given to the domain of HAR due to the enormous amount of its real-time uses in real-time applications, namely surveillance by authorities, biometric user identification, and health monitoring of older people. The extensive usage of the Internet of Things (IoT) and wearable sensor devices has made the topic of HAR a vital subject in ubiquitous and mobile computing. The more commonly utilized inference and problem-solving technique in the HAR system have recently been deep learning (DL). The study develops a Modified Wild Horse Optimization with DL Aided Symmetric Human Activity Recognition (MWHODL-SHAR) model. The major intention of the MWHODL-SHAR model lies in recognition of symmetric activities, namely jogging, walking, standing, sitting, etc. In the presented MWHODL-SHAR technique, the human activities data is pre-processed in various stages to make it compatible for further processing. A convolution neural network with an attention-based long short-term memory (CNN-ALSTM) model is applied for activity recognition. The MWHO algorithm is utilized as a hyperparameter tuning strategy to improve the detection rate of the CNN-ALSTM algorithm. The experimental validation of the MWHODL-SHAR technique is simulated using a benchmark dataset. An extensive comparison study revealed the betterment of the MWHODL-SHAR technique over other recent approaches.

SSRT: A Sequential Skeleton RGB Transformer to Recognize Fine-Grained Human-Object Interactions and Action Recognition

Combining skeleton and RGB modalities in human action recognition (HAR) has garnered attention due to their ability to complement each other. However, previous studies did not address the challenge of recognizing fine-grained human-object interaction (HOI). To tackle this problem, this study introduces a new transformer-based architecture called Sequential Skeleton RGB Transformer (SSRT), which fuses skeleton and RGB modalities. First, SSRT leverages the strength of Long Short-Term Memory (LSTM) and a multi-head attention mechanism to extract high-level features from both modalities. Subsequently, SSRT employs a two-stage fusion method, including transformer cross-attention fusion and softmax layer late score fusion, to effectively integrate the multimodal features. Aside from evaluating the proposed method on fine-grained HOI, this study also assesses its performance on two other action recognition tasks: general HAR and cross-dataset HAR. Furthermore, this study conducts a performance comparison between a HAR model using single-modality features (RGB and skeleton) alongside multimodality features on all three action recognition tasks. To ensure a fair comparison, comparable state-of-the-art transformer architectures are employed for both the single-modality HAR model and SSRT. In terms of modality, SSRT outperforms the best-performing single-modality HAR model on all three tasks, with accuracy improved by 9.92% on fine-grained HOI recognition, 6.73% on general HAR, and 11.08% on cross-dataset HAR. Additionally, the proposed fusion model surpasses state-of-the-art multimodal fusion techniques like Transformer Early Concatenation, with an accuracy improved by 6.32% on fine-grained HOI recognition, 4.04% on general HAR, and 6.56% on cross-dataset.

A Hybrid Accuracy- and Energy-Aware Human Activity Recognition Model in IoT Environment

Personalised health and fitness provide users with information regarding their wellbeing and an opportunity to inform healthcare services for better patient outcomes. Underpinning this industry sector is the need to establish human activity recognition (HAR) in a ubiquitous manner. For example, through the use of smartwatches and/or mobile phones gathering information such as heart rates, movement, and steps of a user. The engineering challenge is providing accurate, informative, and timely data without rapidly depleting the mobile device's battery life. This problem is compounded as a number of algorithms used to process such data require substantial, cloud-based resources, to achieve higher accuracy. Therefore, a balance is required between battery depletion, accuracy of data, and timely delivery of results through a mixture of cloud and local algorithmic execution. In this paper, we propose <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">AE-HAR (Accuracy and Energy Aware-HAR)</i> model that delivers engineered solutions which approach optimal combinations in the consideration of energy consumption, accuracy, and timeliness of results. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">AE-HAR</i> introduces a “light-weight” <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">machine learning</i> on-device component identifying the probabilistic accuracy of data together with energy consumption identification requirements. A heuristic is then adopted to determine if cloud-enabled calculations are required while including possible performance costs related to the analysis of networking infrastructures. Our model is validated in a real-world environment through experimentation that demonstrates accuracy in excess of 93% and energy consumption savings in excess of 94%.

Contrastive Accelerometer–Gyroscope Embedding Model for Human Activity Recognition

Recently, the widespread use of smart devices has invoked more interest in sensor-based applications. Human activity recognition (HAR) with body-worn sensors is an underlying task that aims to recognize a person’s physical activity from on-body sensor readings. In this article, we address the HAR problem that utilizes accelerometers and gyroscopes. While deep-learning-based feature extraction methods are advancing, recent HAR systems with multimodal sensors mostly use data-level fusion, aggregating different sensor signals into one multichannel input. However, they neglect the fact that different sensors capture different physical properties and produce distinct patterns. In this article, we propose a two-stream convolutional neural network (CNN) model that processes the accelerometer and gyroscope signals separately. The modality-specific features are fused in feature-level and jointly used for the recognition task. Furthermore, we introduce a self-supervised learning (SSL) task that pairs the accelerometer and the gyroscope embeddings acquired from the same activity instance. This auxiliary objective allows the feature extractors of our model to communicate during training and exploit complementary information, achieving better representations for HAR. We name our end-to-end multimodal HAR system the Contrastive Accelerometer–Gyroscope Embedding (CAGE) model. CAGE outperforms preceding HAR models in four publicly available benchmarks. The code is available at github.com/quotation2520/CAGE4HAR.

RepHAR: Decoupling Networks With Accuracy-Speed Tradeoff for Sensor-Based Human Activity Recognition

Sensor-based human activity recognition (HAR) can provide users with more convenience and security than machine vision-based methods. The ideal HAR application would achieve high-accuracy and low-latency performance with minimal hardware consumption. However, so far, the speed-accuracy tradeoff in mobile device-based HAR has rarely been systematically analyzed and discussed. In contrast to the static networks of existing deep HAR models, this article aims to design a resource-efficient dynamic network, called RepHAR, for low-cost hardware-constrained HAR tasks. This article first combines multibranch (MB) topologies and structured reparameterization techniques for HAR. Specifically, RepHAR decouples the model into a training time and inference time architecture using the idea of structural reparameterization. That is to say, the model has a MB topology at training time, but it no longer has the same model structure at inference time and is transformed into a plain convolutional neural network (CNN) structure. As a result, RepHAR achieves a trade-off between speed and accuracy by simultaneously obtaining the accuracy gain of a MB network and the high-speed inference of a plain CNN. On four publicly available datasets (including UCI-HAR, PAMAP2, UNIMIB-SHAR, and OPPORTUNITY), RepHAR can achieve 0.83%–2.18% accuracy improvement and 12%–44% parameter reduction, which demonstrates the effectiveness of the proposed method. Finally, on the embedded Raspberry Pi, the RepHAR runs 72% faster than the MB CNN model, demonstrating its usefulness and practicality. Our model and codes will be released soon.

DMSTL: A Deep Multi-Scale Transfer Learning Framework for Unsupervised Cross-Position Human Activity Recognition

Human activity recognition (HAR) based on wearable sensors has been a prosperous research topic in recent years. Considering that the sensor may be worn at diverse body positions, obtaining enough labeled human activity data for each body-worn position is usually expensive and labor-intensive. Furthermore, the variability and diversity of data distribution induced by different body-worn positions make the HAR model trained on data collected from one body position perform poorly for other body-worn positions. To achieve accurate HAR with low labeling cost, which we call unsupervised cross-position HAR, in this article, we propose a deep multiscale transfer learning (DMSTL) model. In the model, we first introduce an unsupervised source selection method to select the most similar source domain for transferring domain knowledge. Then, we develop a multiscale spatial–temporal Net (MSSTNet) to learn comprehensive multimodal representations from multiple feature subspaces. Finally, we design the category-level adaptation module and the domain-level adversarial module for learning domain-invariant features. We conduct extensive experiments on three public HAR data sets and demonstrate the reasonable generalization performance of the DMSTL, which remarkably outperforms other state-of-the-art baselines.

Human Activity Recognition Based on an Efficient Neural Architecture Search Framework Using Evolutionary Multi-Objective Surrogate-Assisted Algorithms

Human activity recognition (HAR) is a popular and challenging research topic driven by various applications. Deep learning methods have been used to improve HAR models’ accuracy and efficiency. However, this kind of method has a lot of manually adjusted parameters, which cost researchers a lot of time to train and test. So, it is challenging to design a suitable model. In this paper, we propose HARNAS, an efficient approach for automatic architecture search for HAR. Inspired by the popular multi-objective evolutionary algorithm, which has a strong capability in solving problems with multiple conflicting objectives, we set weighted f1-score, flops, and the number of parameters as objects. Furthermore, we use a surrogate model to select models with a high score from the large candidate set. Moreover, the chosen models are added to the training set of the surrogate model, which makes the surrogate model update along the search process. Our method avoids manually designing the network structure, and the experiment results demonstrate that it can reduce 40% training costs on both time and computing resources on the OPPORTUNITY dataset and 75% on the UniMiB-SHAR dataset. Additionally, we also prove the portability of the trained surrogate model and HAR model by transferring them from the training dataset to a new dataset.

PrISM-Tracker

A user often needs training and guidance while performing several daily life procedures, e.g., cooking, setting up a new appliance, or doing a COVID test. Watch-based human activity recognition (HAR) can track users' actions during these procedures. However, out of the box, state-of-the-art HAR struggles from noisy data and less-expressive actions that are often part of daily life tasks. This paper proposes PrISM-Tracker, a procedure-tracking framework that augments existing HAR models with (1) graph-based procedure representation and (2) a user-interaction module to handle model uncertainty. Specifically, PrISM-Tracker extends a Viterbi algorithm to update state probabilities based on time-series HAR outputs by leveraging the graph representation that embeds time information as prior. Moreover, the model identifies moments or classes of uncertainty and asks the user for guidance to improve tracking accuracy. We tested PrISM-Tracker in two procedures: latte-making in an engineering lab study and wound care for skin cancer patients at a clinic. The results showed the effectiveness of the proposed algorithm utilizing transition graphs in tracking steps and the efficacy of using simulated human input to enhance performance. This work is the first step toward human-in-the-loop intelligent systems for guiding users while performing new and complicated procedural tasks.