Human Activity Recognition Dataset Research Articles

Action class relation detection and classification across multiple video datasets

The Meta Video Dataset (MetaVD) provides annotated relations between action classes in major datasets for human action recognition in videos. Although these annotated relations enable dataset augmentation, it is only applicable to those covered by MetaVD. For an external dataset to enjoy the same benefit, the relations between its action classes and those in MetaVD need to be determined. To address this issue, we consider two new machine learning tasks: action class relation detection and classification. We propose a unified model to predict relations between action classes, using language and visual information associated with classes. Experimental results show that (i) pre-trained recent neural network models for texts and videos contribute to high predictive performance, (ii) the relation prediction based on action label texts is more accurate than based on videos, and (iii) a blending approach that combines predictions by both modalities can further improve the predictive performance in some cases.

Optimal Deep Recurrent Neural Networks for IoT-enabled Human Activity Recognition in Elderly and Disabled Persons

Aging is related to a decrease in the ability to execute activities of day-to-day routine and decay in physical exercise, which affect mental and physical health. Elderly patients or people can depend on a human activity recognition (HAR) system, which monitors the activity interventions and patterns if any critical event or behavioral changes occur. A HAR system incorporated with the Internet of Things (IoT) environment might allow these people to live independently. While the number of groups of activities and sensor measurements is enormous, the HAR problem could not be resolved deterministically. Hence, machine learning (ML) algorithm was broadly applied for the advancement of the HAR system to find the patterns of human activity from the sensor data. Therefore, this study presents an Optimal Deep Recurrent Neural Networks for Human Activity Recognition (ODRNN-HAR) on Elderly and Disabled Persons technique in the IoT platform. The intension of the ODRNN-HAR approach lies in the recognition and classification of various kinds of human activities in the IoT environment. Primarily, the ODRNN-HAR technique enables IoT devices to collect human activity data and employs Z-score normalization as a preprocessing step. For effectual recognition of human activities, the ODRNN-HAR technique uses the DRNN model. At the final stage, the optimal hyperparameter adjustment of the DRNN model takes place using the mayfly optimization (MFO) algorithm. The result analysis of the ODRNN-HAR algorithm takes place on benchmark HAR dataset, and the outcomes are examined. The comprehensive simulation outcomes highlighted the improved recognition results of the ODRNN-HAR approach in terms of different measures.

ProtoHAR: Prototype Guided Personalized Federated Learning for Human Activity Recognition.

Federated Learning (FL) has recently attracted great interest in sensor-based human activity recognition (HAR) tasks. However, in real-world environment, sensor data on devices is non-independently and identically distributed (Non-IID), e.g., activity data recorded by most devices is sparse, and sensor data distribution for each client may be inconsistent. As a result, the traditional FL methods in the heterogeneous environment may incur a drifted global model that causes slow convergence and a heavy communication burden. Although some FL methods are gradually being applied to HAR, they are designed for overly ideal scenarios and do not address such Non-IID problem in the real-world setting. It is still a question whether they can be applied to cross-device FL. To tackle this challenge, we propose ProtoHAR, a prototype-guided FL framework for HAR, which aims to decouple the representation and classifier in the heterogeneous FL setting efficiently. It leverages the global prototype to correct the activity feature representation to make the prototype knowledge flow among clients without leaking privacy while solving a better classifier to avoid excessive drift of the local model in personalized training. Extensive experiments are conducted on four publicly available datasets: USC-HAD, UNIMIB-SHAR, PAMAP2, and HARBOX, which are collected in both controlled environments and real-world scenarios. The results show that compared with the state-of-the-art FL algorithms, ProtoHAR achieves the best performance and faster convergence speed in HAR datasets.

A Human Activity Recognition Method Based on Lightweight Feature Extraction Combined With Pruned and Quantized CNN for Wearable Device

Human Activity Recognition (HAR) is becoming an essential part of human life care. Existing HAR methods are usually developed using a two-level approach, wherein a first-level Machine Learning (ML) classifier is employed to distinguish the static and dynamic activities, followed by a second-level classifier to identify the specific activity. These approaches are not suitable for wearable devices, due to the high computational and memory consumption. Our rigorous analysis of various HAR datasets opens up a new possibility that static or dynamic activities can be discriminated against through a simple statistical technique. Therefore, we propose to utilize a statistical feature extraction technique to replace the first-level ML classifier, thus achieving more lightweight computation. Next, we employ Random Forest (RF) and Convolutional Neural Networks (CNN) to classify the specific activities, achieving higher accuracy compared to the state-of-the-art results. We further reduce the computation and memory consumption of the above combined approach by applying pruning and quantizing techniques to CNN (PQ-CNN). Experimental results show the proposed lightweight HAR method achieved an F1 score of 0.9417 and 0.9438 for unbalanced and balanced datasets, respectively. On top of lightweight and accuracy, the proposed HAR method is practical for wearable devices by using a single accelerometer.

Temporal-channel convolution with self-attention network for human activity recognition using wearable sensors

Human activity recognition (HAR) is an essential task in many applications such as health monitoring, rehabilitation, and sports training. Sensor-based HAR has received increasing attention due to the widespread availability of sensors in daily life. In this paper, we propose two novel architectures, the convolution with self-attention network (CSNet) and the temporal-channel convolution with self-attention network (TCCSNet), for classifying sequences of human activity data from different sensors. CSNet leverages both convolution and self-attention to capture both local and global dependencies in the input data, while TCCSNet exploits both temporal and inter-channel dependencies through two branches of convolutions and self-attentions for extracting time-wise and channel-wise information. The proposed methods are evaluated on seven different sensor-based HAR datasets, namely: MHEALTH, PAMAP2, UTD1, UTD2, WHARF, USC-HAD, and WISDM, using the leave-one-subject-out cross-validation protocol. Our experiments show that the proposed models outperform other modern approaches, such as Transformers and long short-term memory (LSTM) based models.

Human Action Representation Learning Using an Attention-Driven Residual 3DCNN Network

The recognition of human activities using vision-based techniques has become a crucial research field in video analytics. Over the last decade, there have been numerous advancements in deep learning algorithms aimed at accurately detecting complex human actions in video streams. While these algorithms have demonstrated impressive performance in activity recognition, they often exhibit a bias towards either model performance or computational efficiency. This biased trade-off between robustness and efficiency poses challenges when addressing complex human activity recognition problems. To address this issue, this paper presents a computationally efficient yet robust approach, exploiting saliency-aware spatial and temporal features for human action recognition in videos. To achieve effective representation of human actions, we propose an efficient approach called the dual-attentional Residual 3D Convolutional Neural Network (DA-R3DCNN). Our proposed method utilizes a unified channel-spatial attention mechanism, allowing it to efficiently extract significant human-centric features from video frames. By combining dual channel-spatial attention layers with residual 3D convolution layers, the network becomes more discerning in capturing spatial receptive fields containing objects within the feature maps. To assess the effectiveness and robustness of our proposed method, we have conducted extensive experiments on four well-established benchmark datasets for human action recognition. The quantitative results obtained validate the efficiency of our method, showcasing significant improvements in accuracy of up to 11% as compared to state-of-the-art human action recognition methods. Additionally, our evaluation of inference time reveals that the proposed method achieves up to a 74× improvement in frames per second (FPS) compared to existing approaches, thus showing the suitability and effectiveness of the proposed DA-R3DCNN for real-time human activity recognition.

Sharing leaky-integrate-and-fire neurons for memory-efficient spiking neural networks.

Spiking Neural Networks (SNNs) have gained increasing attention as energy-efficient neural networks owing to their binary and asynchronous computation. However, their non-linear activation, that is Leaky-Integrate-and-Fire (LIF) neuron, requires additional memory to store a membrane voltage to capture the temporal dynamics of spikes. Although the required memory cost for LIF neurons significantly increases as the input dimension goes larger, a technique to reduce memory for LIF neurons has not been explored so far. To address this, we propose a simple and effective solution, EfficientLIF-Net, which shares the LIF neurons across different layers and channels. Our EfficientLIF-Net achieves comparable accuracy with the standard SNNs while bringing up to ~4.3× forward memory efficiency and ~21.9× backward memory efficiency for LIF neurons. We conduct experiments on various datasets including CIFAR10, CIFAR100, TinyImageNet, ImageNet-100, and N-Caltech101. Furthermore, we show that our approach also offers advantages on Human Activity Recognition (HAR) datasets, which heavily rely on temporal information. The code has been released at https://github.com/Intelligent-Computing-Lab-Yale/EfficientLIF-Net.

HAR-DeepConvLG: Hybrid deep learning-based model for human activity recognition in IoT applications

Smartphones and wearable devices have built-in sensors that can collect multivariant time-series data that can be used to recognize human activities. Research on human activity recognition (HAR) has gained significant attention in recent years due to its growing demand in various application domains. As wearable sensor-aided devices and the Internet of Things (IoT) became more common, great attention has been paid to the HAR ubiquitous computing and mobile computing. To infer human activity data from a massive amount of multivariant data generated by different wearable devices, in this study an innovative deep learning–based model named HAR-DeepConvLG is proposed. It includes three convolution layers and a squeezing and excitation (SE) block, which are employed to precisely learn and extract the spatial representation data from the collected raw sensor data. The extracted features are used as input of three parallel paths, each of which includes a long short-term memory (LSTM) layer connected in sequence with a gated recurrent unit (GRU) layer to learn temporal representation. The three paths are connected in parallel to avoid the vanishing gradient problem. Finally, to evaluate the effectiveness of the proposed model, experiments were conducted on four widely utilized HAR datasets. Additionally, the model’s performance was compared to several state-of-the-art deep learning models, which further validated its effectiveness. The experimental results show that the proposed HAR-DeepConvLG model performs better than the existing HAR deep learning–based models, achieving a competitive classification accuracy.

EatSense: Human centric, action recognition and localization dataset for understanding eating behaviors and quality of motion assessment

Current datasets for computer vision-based action recognition and localization cover a wide range of classes and challenging scenarios. However, these datasets don't cater to healthcare applications that involve long-term monitoring, tracking minor changes in movements over time for healthcare purposes, or completely modeling a specific human behavior that includes multiple sub-actions. Specifically, there are no existing datasets for research on either health monitoring on atomic-action-based eating behavior or for a full range of eating sub-actions that fully segment the main action. Addressing these gaps is valuable for extending research on the health monitoring of elderly people and is needed for creating richer and more complete descriptions of actions. This paper introduces a new benchmark dataset named EatSense that targets both the computer vision and healthcare communities and fills in the aforementioned gaps. EatSense is recorded while a person eats in an uncontrolled dining setting. The key features of EatSense are the introduction of challenging atomic actions for action recognition, the significantly diverse durations of actions that make it difficult for current temporal action localization frameworks to localize, the capability to model comprehensive eating behavior in terms of a sequence of action-based behaviors, and the simulation of minor variations in motion or performance. We conduct extensive experiments on EatSense with baseline deep learning-based approaches for benchmarking and hand-crafted feature-based approaches for explainable applications. We believe this dataset will benefit future researchers in building robust temporal action localization networks, behavior recognition, and performance assessment models for eating.

Spatial Hard Attention Modeling via Deep Reinforcement Learning for Skeleton-Based Human Activity Recognition

Deep learning-based algorithms have been very successful in skeleton-based human activity recognition. Skeleton data contains 2-D or 3-D coordinates of human body joints. The main focus of most of the existing skeleton-based activity recognition methods is on designing new deep architectures to learn discriminative features, where all body joints are considered equally important in recognition. However, the importance of joints varies as an activity proceeds within a video and across different activities. In this work, we hypothesize that selecting relevant joints, prior to recognition, can enhance performance of the existing deep learning-based recognition models. We propose a spatial hard attention finding method that aims to remove the uninformative and/or misleading joints at each frame. We formulate the joint selection problem as a Markov decision process and employ deep reinforcement learning to train the proposed spatial-attention-aware agent. No extra labels are needed for the agent’s training. The agent takes a sequence of features extracted from skeleton video as input and outputs a sequence of probabilities for joints. The proposed method can be considered as a general framework that can be integrated with the existing skeleton-based activity recognition methods for performance improvement purposes. We obtain very competitive activity recognition results on three commonly used human activity recognition datasets.

Deep features fusion for user authentication based on human activity

AbstractThe exponential growth in the use of smartphones means that users must constantly be concerned about the security and privacy of mobile data because the loss of a mobile device could compromise personal information. To address this issue, continuous authentication systems have been proposed, in which users are monitored transparently after initial access to the smartphone. In this study, the authors address the problem of user authentication by considering human activities as behavioural biometric information. The authors convert the behavioural biometric data (considered as time series) into a 2D colour image. This transformation process keeps all the characteristics of the behavioural signal. Time series does not receive any filtering operation with this transformation, and the method is reversible. This signal‐to‐image transformation allows us to use the 2D convolutional networks to build efficient deep feature vectors. This allows them to compare these feature vectors to the reference template vectors to compute the performance metric. The authors evaluate the performance of the authentication system in terms of Equal Error Rate on a benchmark University of Californy, Irvine Human Activity Recognition dataset, and they show the efficiency of the approach.

Enhancing lifestyle and health monitoring of elderly populations using CSA-TkELM classifier

Nowadays, there is a growing interest among researchers in health monitoring using body sensor data due to its usage in diverse applications. Human Activity Recognition (HAR) utilizes smart wearable devices, particularly in healthcare, to help the elderly population improve their lifestyle by preventing forthcoming exceptional events such as health risks and falls. Although various types of sensors are available to recognize human behaviors, wearable sensors provide consistent information about the lifestyle and functionality of subjects. However, accurate recognition requires intelligent algorithms to extract temporal features of time series data from raw sensor readings. In order to recognize human activities more accurately from time series data, this paper proposes a crossover Aquila-optimized Tri-kernel Extreme Learning Machine (CSA-TkELM) classifier. The sensor data utilized to evaluate the effectiveness of the proposed CSA-TkELM classifier is obtained from the USC Human Activity Dataset (USC-HAD), the Public domain dataset for Real-life human activity recognition using smartphone Sensors (PDS-SS), and the OPPORTUNITY dataset. The direct application of raw time series data to the classifier affects its recognition performance. Therefore, preprocessing steps such as noise removal, handling missing values, standardization, and segmentation are executed. The proposed CSA-TkELM classifier extracts significant temporal features from the time series information and recognizes activities. Several performance metrics, such as recall, precision, accuracy, and F1 score, are measured to investigate the results of the proposed CSA-TkELM classifier. The proposed CSA-TkELM classifier achieves a recall of 98.1%, precision of 97.6%, accuracy of 97.9%, and F1 score of 97.3%, which are higher than those of other compared existing techniques.

A large-scale fMRI dataset for human action recognition

Human action recognition is a critical capability for our survival, allowing us to interact easily with the environment and others in everyday life. Although the neural basis of action recognition has been widely studied using a few action categories from simple contexts as stimuli, how the human brain recognizes diverse human actions in real-world environments still needs to be explored. Here, we present the Human Action Dataset (HAD), a large-scale functional magnetic resonance imaging (fMRI) dataset for human action recognition. HAD contains fMRI responses to 21,600 video clips from 30 participants. The video clips encompass 180 human action categories and offer a comprehensive coverage of complex activities in daily life. We demonstrate that the data are reliable within and across participants and, notably, capture rich representation information of the observed human actions. This extensive dataset, with its vast number of action categories and exemplars, has the potential to deepen our understanding of human action recognition in natural environments.

Defending Black-Box Skeleton-Based Human Activity Classifiers

Skeletal motions have been heavily relied upon for human activity recognition (HAR). Recently, a universal vulnerability of skeleton-based HAR has been identified across a variety of classifiers and data, calling for mitigation. To this end, we propose the first black-box defense method for skeleton-based HAR to our best knowledge. Our method is featured by full Bayesian treatments of the clean data, the adversaries and the classifier, leading to (1) a new Bayesian Energy-based formulation of robust discriminative classifiers, (2) a new adversary sampling scheme based on natural motion manifolds, and (3) a new post-train Bayesian strategy for black-box defense. We name our framework Bayesian Energy-based Adversarial Training or BEAT. BEAT is straightforward but elegant, which turns vulnerable black-box classifiers into robust ones without sacrificing accuracy. It demonstrates surprising and universal effectiveness across a wide range of skeletal HAR classifiers and datasets, under various attacks. Appendix and code are available.

Edge-Fog-Cloud Data Analysis for eHealth-IoT

Thanks to advancements in artificial intelligence and the Internet of Things (IoT), eHealth is becoming an increasingly attractive area for researchers. However, different challenges arise when sensor-generated information is stored and analyzed using cloud computing. Latency, response time, and security are critical concerns that require attention. Fog and Edge Computing technologies have emerged in response to the requirement for resources near the network edge where data is collected, to minimize cloud challenges. This paper aims to assess the effectiveness of Machine Learning (ML) and Deep Learning (DL) techniques when executed in Edge or Fog nodes within the eHealth data. We compared the most efficient baseline techniques from the state-of-the-art on three eHealth datasets: Human Activity Recognition (HAR), University of Milano Bicocca Smartphone-based Human Activity Recognition (UniMiB SHAR), and MIT-BIH Arrhythmia. The experiment showed that for the HAR dataset, the Support Vector Machines (SVM) model was the best performer among the ML techniques, with low processing time and an accuracy of 96%. In comparison, the K-Nearest Neighbors (KNN) performed 94.43, and 96%, respectively, for SHAR and MIT-BIH datasets. Among the DL techniques, the Convolutional Neural Network with Fourier (CNNF) model performed the best, with accuracies of 94.49% and 98.72% for HAR and MIT-BIH. In comparison, CNN achieved 96.90% for the SHAR dataset.

Semi-Supervised Learning for Wearable-based Momentary Stress Detection in the Wild

Physiological and behavioral data collected from wearable or mobile sensors have been used to estimate self-reported stress levels. Since stress annotation usually relies on self-reports during the study, a limited amount of labeled data can be an obstacle to developing accurate and generalized stress-predicting models. On the other hand, the sensors can continuously capture signals without annotations. This work investigates leveraging unlabeled wearable sensor data for stress detection in the wild. We propose a two-stage semi-supervised learning framework that leverages wearable sensor data to help with stress detection. The proposed structure consists of an auto-encoder pre-training method for learning information from unlabeled data and the consistency regularization approach to enhance the robustness of the model. Besides, we propose a novel active sampling method for selecting unlabeled samples to avoid introducing redundant information to the model. We validate these methods using two datasets with physiological signals and stress labels collected in the wild, as well as four human activity recognition (HAR) datasets to evaluate the generality of the proposed method. Our approach demonstrated competitive results for stress detection, improving stress classification performance by approximately 7% to 10% on the stress detection datasets compared to the baseline supervised learning models. Furthermore, the ablation study we conducted for the HAR tasks supported the effectiveness of our methods. Our approach showed comparable performance to state-of-the-art semi-supervised learning methods for both stress detection and HAR tasks.

Online continual learning for human activity recognition

Sensor-based human activity recognition (HAR), with the ability to recognise human activities from wearable or embedded sensors, has been playing an important role in many applications including personal health monitoring, smart home, and manufacturing. The real-world, long-term deployment of these HAR systems drives a critical research question: how to evolve the HAR model automatically over time to accommodate changes in an environment or activity patterns. This paper presents an online continual learning (OCL) scenario for HAR, where sensor data arrives in a streaming manner which contains unlabelled samples from already learnt activities or new activities. We propose a technique, OCL-HAR, making a real-time prediction on the streaming sensor data while at the same time discovering and learning new activities. We have empirically evaluated OCL-HAR on four third-party, publicly available HAR datasets. Our results have shown that this OCL scenario is challenging to state-of-the-art continual learning techniques that have significantly underperformed. Our technique OCL-HAR has consistently outperformed them in all experiment setups, leading up to 0.17 and 0.23 improvements in micro and macro F1 scores.

Low-light aware framework for human activity recognition via optimized dual stream parallel network

Human Activity Recognition (HAR) plays a crucial role in communication and the Internet of Things (IoT), by enabling vision sensors to understand and respond to human behavior more intelligently and efficiently. Existing deep learning models are complex to deal with the low illumination, diverse viewpoints, and cluttered backgrounds, which require substantial computing resources and are not appropriate for edge devices. Furthermore, without an effective video analysis technique it processes entire frames, resulting inadequate performance. To address these key challenges, a cloud-assisted IoT computing framework is proposed for HAR in uncertain low-lighting environments, which is mainly composed of two tiers: edge and cloud computing. Initially, a lightweight Convolutional Neural Network (CNN) model is developed which is responsible to enhance the low-light frames, followed by the human detection algorithm to process the selective frames, thus enabling efficient resource utilization. Next, these refined frames are then transmitted to the cloud for accurate HAR, where dual stream CNN and transformer fusion network extract both short-and long-range spatiotemporal discriminative features followed by proposed Optimized Parallel Sequential Temporal Network (OPSTN) with squeeze and excitation attention to efficiently learn HAR in complex scenarios. Finally, extensive experiments are conducted over three challenging HAR datasets to deeply examine the proposed framework from various perspectives such as complex activity recognition, lowlighting, etc., where the results are outperformed compared with the state-of-art methods.

Semisupervised Generative Adversarial Networks With Temporal Convolutions for Human Activity Recognition

Many potential applications of human activity recognition (HAR) can be found in health, surveillance, manufacturing, sports, etc. For instance, HAR can be exploited in ambient assisted living (AAL) systems to provide users with assistance services intend to improve their well-being, safety, and autonomy. Data annotation in HAR is a complex and time-consuming process that limits the availability of labeled samples. Furthermore, the classification performance of supervised deep neural networks depends on the availability of large, annotated training data. This paper proposes an alternative framework for semi-supervised generative adversarial networks (GANs) using temporal convolutions for semi-supervised action recognition to address several problems related to conventional approaches in the HAR context, such as high-dimensionality, scarcity of annotated data, scalability, and robustness. The proposed framework employs a single architecture on different datasets and its effectiveness is tested under four conditions reflecting real-world semi-supervised scenarios to investigate the impact of intersubject training, amount of labeled data, number of classes, and IMU positions on model performance. The evaluations are performed on the PAMAP2, Opportunity-locomotion, and LISSI HAR datasets to evaluate the generalizability of the framework. The results show the proposed framework’s high classification performance and generalization ability compared to baseline methods, achieving up to 25% improvement when only a small amount of annotated data is available. A comparison to the previous work using the same datasets has also validated the performance of the framework.

Counting Activities Using Weakly Labeled Raw Acceleration Data: A Variable-Length Sequence Approach with Deep Learning to Maintain Event Duration Flexibility.

This paper presents a novel approach for counting hand-performed activities using deep learning and inertial measurement units (IMUs). The particular challenge in this task is finding the correct window size for capturing activities with different durations. Traditionally, fixed window sizes have been used, which occasionally result in incorrectly represented activities. To address this limitation, we propose segmenting the time series data into variable-length sequences using ragged tensors to store and process the data. Additionally, our approach utilizes weakly labeled data to simplify the annotation process and reduce the time to prepare annotated data for machine learning algorithms. Thus, the model receives only partial information about the performed activity. Therefore, we propose an LSTM-based architecture, which takes into account both the ragged tensors and the weak labels. To the best of our knowledge, no prior studies attempted counting utilizing variable-size IMU acceleration data with relatively low computational requirements using the number of completed repetitions of hand-performed activities as a label. Hence, we present the data segmentation method we employed and the model architecture that we implemented to show the effectiveness of our approach. Our results are evaluated using the Skoda public dataset for Human activity recognition (HAR) and demonstrate a repetition error of ±1 even in the most challenging cases. The findings of this study have applications and can be beneficial for various fields, including healthcare, sports and fitness, human-computer interaction, robotics, and the manufacturing industry.