Human Activity Recognition Model Research Articles

Exploring Cutout and Mixup for Robust Human Activity Recognition on Sensor and Skeleton Data

Human Activity Recognition (HAR) is an essential area of research in Artificial Intelligence and Machine Learning, with numerous applications in healthcare, sports science, and smart environments. While several advancements in the field, such as attention-based models and Graph Neural Networks, have made great strides, this work focuses on data augmentation methods that tackle issues like data scarcity and task variability in HAR. In this work, we investigate and expand the use of mixup and cutout data augmentation methods to sensor-based and skeleton-based HAR datasets. These methods were first widely used in Computer Vision and Natural Language Processing. We use both augmentation techniques, customized for time-series and skeletal data, to improve the robustness and performance of HAR models by diversifying the data and overcoming the drawbacks of having limited training data. Specifically, we customize mixup data augmentation for sensor-based datasets and cutout data augmentation for skeleton-based datasets with the goal of improving model accuracy without adding more data. Our results show that using mixup and cutout techniques improves the accuracy and generalization of activity recognition models on both sensor-based and skeleton-based human activity datasets. This work showcases the potential of data augmentation techniques on transformers and Graph Neural Networks by offering a novel method for enhancing time series and skeletal HAR tasks.

Belief-Rule-Based System with Self-organizing and Multi-temporal Modeling for Sensor-based Human Activity Recognition.

Smart environment is an efficient and cost- effective way to afford intelligent supports for the elderly people. Human activity recognition (HAR) is a crucial aspect of the research field of smart environments, and it has attracted widespread attention lately. The goal of this study is to develop an effective sensor-based HAR model based on the belief-rule-based system (BRBS), which is one of representative rule-based expert systems. Specially, a new belief rule base (BRB) modeling approach is proposed by taking into account the self- organizing rule generation method and the multi-temporal rule representation scheme, in order to address the problem of combination explosion that existed in the traditional BRB modelling procedure and the time correlation found in continuous sensor data in chronological order. The new BRB modeling approach is so called self-organizing and multi-temporal BRB (SOMT-BRB) modeling procedure. A case study is further deducted to validate the effectiveness of the SOMT-BRB modeling procedure. By comparing with some conventional BRBSs and classical activity recognition models, the results show a significant improvement of the BRBS in terms of the number of belief rules, modelling efficiency, and activity recognition accuracy.

BodyFlow: An Open-Source Library for Multimodal Human Activity Recognition

Human activity recognition is a critical task for various applications across healthcare, sports, security, gaming, and other fields. This paper presents BodyFlow, a comprehensive library that seamlessly integrates human pose estimation and multiple-person estimation and tracking, along with activity recognition modules. BodyFlow enables users to effortlessly identify common activities and 2D/3D body joints from input sources such as videos, image sets, or webcams. Additionally, the library can simultaneously process inertial sensor data, offering users the flexibility to choose their preferred input, thus facilitating multimodal human activity recognition. BodyFlow incorporates state-of-the-art algorithms for 2D and 3D pose estimation and three distinct models for human activity recognition.

Lightweight Multi-Domain Fusion Model for Through-Wall Human Activity Recognition Using IR-UWB Radar

Impulse radio ultra-wideband (IR-UWB) radar, operating in the low-frequency band, can penetrate walls and utilize its high range resolution to recognize different human activities. Complex deep neural networks have demonstrated significant performance advantages in classifying radar spectrograms of various actions, but at the cost of a substantial computational overhead. In response, this paper proposes a lightweight model named TG2-CAFNet. First, clutter suppression and time–frequency analysis are used to obtain range–time and micro-Doppler feature maps of human activities. Then, leveraging GhostV2 convolution, a lightweight feature extraction module, TG2, suitable for radar spectrograms is constructed. Using a parallel structure, the features of the two spectrograms are extracted separately. Finally, to further explore the correlation between the two spectrograms and enhance the feature representation capabilities, an improved nonlinear fusion method called coordinate attention fusion (CAF) is proposed based on attention feature fusion (AFF). This method extends the adaptive weighting fusion of AFF to a spatial distribution, effectively capturing the subtle spatial relationships between the two radar spectrograms. Experiments showed that the proposed method achieved a high degree of model lightweightness, while also achieving a recognition accuracy of 99.1%.

CAPTURE-24: A large dataset of wrist-worn activity tracker data collected in the wild for human activity recognition

Existing activity tracker datasets for human activity recognition are typically obtained by having participants perform predefined activities in an enclosed environment under supervision. This results in small datasets with a limited number of activities and heterogeneity, lacking the mixed and nuanced movements normally found in free-living scenarios. As such, models trained on laboratory-style datasets may not generalise out of sample. To address this problem, we introduce a new dataset involving wrist-worn accelerometers, wearable cameras, and sleep diaries, enabling data collection for over 24 hours in a free-living setting. The result is CAPTURE-24, a large activity tracker dataset collected in the wild from 151 participants, amounting to 3883 hours of accelerometer data, of which 2562 hours are annotated. CAPTURE-24 is two to three orders of magnitude larger than existing publicly available datasets, which is critical to developing accurate human activity recognition models.

Detection of sensors used for adversarial examples against machine learning models

Machine learning (ML) systems that utilize sensor inputs have become integral in various applications; however, they remain vulnerable to sensor-based adversarial example (AE) attacks, where compromised sensors can be exploited to manipulate system outputs. This study addresses the critical issue of safeguarding these systems by identifying and mitigating compromised sensors, thereby enhancing their resilience. This study introduces a novel detection method using a feature-removable model (FRM), which allows the selective removal of features to identify inconsistencies in the model's outputs when different sensor features are altered. The methodology was validated on a human activity recognition (HAR) model utilizing sensors placed on the chest, wrist, and ankle, with a focus on identifying attacker-compromised sensors. The results demonstrated the efficacy of the method, achieving an average Recall of detected sensors of 0.92 and an average Precision of detected sensors of 0.72, highlighting the ability of the approach to accurately detect and identify compromised sensors. This study significantly contributes to advancing the security and robustness of ML systems against sensor-based AE attacks.

Validation of two novel human activity recognition models for typically developing children and children with Cerebral Palsy.

Human Activity Recognition models have potential to contribute to valuable and detailed knowledge of habitual physical activity for typically developing children and children with Cerebral Palsy. The main objective of the present study was to develop and validate two Human Activity Recognition models. One trained on data from typically developing children (n = 63), the second also including data from children with Cerebral Palsy (n = 16), engaging in standardised activities and free play. Our data was collected using accelerometers and ground truth was established with video annotations. Additionally, we aimed to investigate the influence of window settings on model performance. Utilizing the Extreme gradient boost (XGBoost) classifier, twelve sub-models were created, with 1-,3- and 5-seconds windows, with and without overlap. Both Human Activity Recognition models demonstrated excellent predictive capabilities (>92%) for standardised activities for both typically developing and Cerebral Palsy. From all window sizes, the 1-second window performed best for all test groups. Accuracy was slightly lower (>75%) for the Cerebral Palsy test group performing free play activities. The impact of window size and overlap varied depending on activity. In summary both Human Activity Recognition models effectively predict standardised activities, surpassing prior models for typically developing and children with Cerebral Palsy. Notably, the model trained on combined typically developing children and Cerebral Palsy data performed exemplary across all test groups. Researchers should select window settings aligned with their specific research objectives.

A novel hybrid deep learning approach with GWO–WOA optimization technique for human activity recognition

The effectiveness of Human Activity Recognition (HAR) models can be largely attributed to the components derived from domain expertise. The classification system swiftly and effectively categorizes human physical activity by utilizing a comprehensive collection of variables. To construct HAR models and categorize different activities, deep learning algorithms have recently seen increased application in academic research for autonomously extracting features from raw sensory data. The primary goal of this study is to develop a strong and accurate activity classification model by first extracting features from an input dataset and then using deep learning models. In this research, we look at how well the model does in the field of HAR. Analyzing time-series data from smartphones and wearable sensors, algorithms based on deep learning (DL) have been used to forecast various human actions. Applying DL-based methods to time-series data presents a variety of challenges, notwithstanding their usage for activity identification. If the suggested approach is put into practice, the concerns mentioned before may be alleviated. To construct HAR classification techniques using data collected from wearable sensors, this study presents two Hybrid Learning Algorithms (HLA). This research employs Convolutional Neural Network (CNN) and Recurrent Neural Network (RNN) methods to teach the model sequential and spatial patterns. The process of selecting characteristics has been enhanced through the use of optimization techniques. Together, the Whale Optimization Algorithm and the Grey Wolf Optimizer form a Hybrid Optimization approach, which is introduced in this study. By combining and capitalizing the benefits of both approaches, the hybrid technique enhances the optimization process as a whole. This research may also assist individuals who have undergone amputations or experienced strokes in selecting an appropriate model and developing a personalized gait reference trajectory.

Activity recognition in children with CP: Development and validation of a human activity recognition model

Activity recognition in children with CP: Development and validation of a human activity recognition model

Achieving More with Less: A Lightweight Deep Learning Solution for Advanced Human Activity Recognition (HAR).

Human activity recognition (HAR) is a crucial task in various applications, including healthcare, fitness, and the military. Deep learning models have revolutionized HAR, however, their computational complexity, particularly those involving BiLSTMs, poses significant challenges for deployment on resource-constrained devices like smartphones. While BiLSTMs effectively capture long-term dependencies by processing inputs bidirectionally, their high parameter count and computational demands hinder practical applications in real-time HAR. This study investigates the approximation of the computationally intensive BiLSTM component in a HAR model by using a combination of alternative model components and data flipping augmentation. The proposed modifications to an existing hybrid model architecture replace the BiLSTM with standard and residual LSTM, along with convolutional networks, supplemented by data flipping augmentation to replicate the context awareness typically provided by BiLSTM networks. The results demonstrate that the residual LSTM (ResLSTM) model achieves superior performance while maintaining a lower computational complexity compared to the traditional BiLSTM model. Specifically, on the UCI-HAR dataset, the ResLSTM model attains an accuracy of 96.34% with 576,702 parameters, outperforming the BiLSTM model's accuracy of 95.22% with 849,534 parameters. On the WISDM dataset, the ResLSTM achieves an accuracy of 97.20% with 192,238 parameters, compared to the BiLSTM's 97.23% accuracy with 283,182 parameters, demonstrating a more efficient architecture with minimal performance trade-off. For the KU-HAR dataset, the ResLSTM model achieves an accuracy of 97.05% with 386,038 parameters, showing comparable performance to the BiLSTM model's 98.63% accuracy with 569,462 parameters, but with significantly fewer parameters.

TriFusion hybrid model for human activity recognition

TriFusion hybrid model for human activity recognition

Large Language Models for Wearable Sensor-Based Human Activity Recognition, Health Monitoring, and Behavioral Modeling: A Survey of Early Trends, Datasets, and Challenges.

The proliferation of wearable technology enables the generation of vast amounts of sensor data, offering significant opportunities for advancements in health monitoring, activity recognition, and personalized medicine. However, the complexity and volume of these data present substantial challenges in data modeling and analysis, which have been addressed with approaches spanning time series modeling to deep learning techniques. The latest frontier in this domain is the adoption of large language models (LLMs), such as GPT-4 and Llama, for data analysis, modeling, understanding, and human behavior monitoring through the lens of wearable sensor data. This survey explores the current trends and challenges in applying LLMs for sensor-based human activity recognition and behavior modeling. We discuss the nature of wearable sensor data, the capabilities and limitations of LLMs in modeling them, and their integration with traditional machine learning techniques. We also identify key challenges, including data quality, computational requirements, interpretability, and privacy concerns. By examining case studies and successful applications, we highlight the potential of LLMs in enhancing the analysis and interpretation of wearable sensor data. Finally, we propose future directions for research, emphasizing the need for improved preprocessing techniques, more efficient and scalable models, and interdisciplinary collaboration. This survey aims to provide a comprehensive overview of the intersection between wearable sensor data and LLMs, offering insights into the current state and future prospects of this emerging field.

A Human Activity Recognition model based on CNN and Transformer

Abstract This study aims to utilize data from built-in sensors in smartphones for human activity recognition. By analyzing the three-dimensional accelerometer and gyroscope data in user behavior, accurate classification of eight common activity states is achieved, including walking, standing, sitting, squatting, going up stairs, going down stairs, climbing ladders, and descending ladders. To enhance the model’s generalization capability, a method combining Transformer neural networks with one-dimensional Convolutional Neural Networks (CNNs) is employed, along with data sample augmentation. Experimental results demonstrate a significant improvement in recognition accuracy compared to traditional models, indicating the potential for real-time application on smartphones and other devices. This approach provides essential technical support for predictive human-computer interaction on smart devices and holds extensive application prospects.

MF-Match: A Semi-Supervised Model for Human Action Recognition.

Human action recognition (HAR) technology based on radar signals has garnered significant attention from both industry and academia due to its exceptional privacy-preserving capabilities, noncontact sensing characteristics, and insensitivity to lighting conditions. However, the scarcity of accurately labeled human radar data poses a significant challenge in meeting the demand for large-scale training datasets required by deep model-based HAR technology, thus substantially impeding technological advancements in this field. To address this issue, a semi-supervised learning algorithm, MF-Match, is proposed in this paper. This algorithm computes pseudo-labels for larger-scale unsupervised radar data, enabling the model to extract embedded human behavioral information and enhance the accuracy of HAR algorithms. Furthermore, the method incorporates contrastive learning principles to improve the quality of model-generated pseudo-labels and mitigate the impact of mislabeled pseudo-labels on recognition performance. Experimental results demonstrate that this method achieves action recognition accuracies of 86.69% and 91.48% on two widely used radar spectrum datasets, respectively, utilizing only 10% labeled data, thereby validating the effectiveness of the proposed approach.

Wearable sensors based on artificial intelligence models for human activity recognition.

Human motion detection technology holds significant potential in medicine, health care, and physical exercise. This study introduces a novel approach to human activity recognition (HAR) using convolutional neural networks (CNNs) designed for individual sensor types to enhance the accuracy and address the challenge of diverse data shapes from accelerometers, gyroscopes, and barometers. Specific CNN models are constructed for each sensor type, enabling them to capture the characteristics of their respective sensors. These adapted CNNs are designed to effectively process varying data shapes and sensor-specific characteristics to accurately classify a wide range of human activities. The late-fusion technique is employed to combine predictions from various models to obtain comprehensive estimates of human activity. The proposed CNN-based approach is compared to a standard support vector machine (SVM) classifier using the one-vs-rest methodology. The late-fusion CNN model showed significantly improved performance, with validation and final test accuracies of 99.35 and 94.83% compared to the conventional SVM classifier at 87.07 and 83.10%, respectively. These findings provide strong evidence that combining multiple sensors and a barometer and utilizing an additional filter algorithm greatly improves the accuracy of identifying different human movement patterns.

A Spatio-temporal Graph Transformer driven model for recognizing fine-grained data human activity

Human activity recognition(HAR) is an important research focus in ubiquitous computing. It has been widely applied in various domains, such as smart homes, healthcare assistance, and sports training. Accurate human activity recognition directly impacts the performance of downstream tasks. Existing methods for human activity recognition primarily rely on extracting temporal or spatial features from the data. These features are used for the task of human activity recognition. The existing methods mainly use CNN or RNN models to model the Euclidean correlations among spatially adjacent sensors or channels. However, non-Euclidean pairwise correlations among all sensors or channels are even critical for accurate classification, which has been ignored by the existing methods. In this paper, we incorporate fine-grained spatial structural data into the model to overcome these limitations. A novel deep learning model for human activity recognition is proposed, which is called the fine-grained data-oriented Spatial-Temporal Graph Transformer network (STGT). The introduction of the STGT model can eliminate the limitations of existing spatial feature extraction methods by leveraging a novel data organization approach proposed in this study. This model enhances the spatial features within the time feature extraction module for effective spatio-temporal feature extraction. We conducted experiments on four large-scale real-world HAR datasets to evaluate its performance. The experimental results demonstrate the superiority of our method over state-of-the-art approaches.

Weighted voting ensemble of hybrid CNN-LSTM Models for vision-based human activity recognition

Weighted voting ensemble of hybrid CNN-LSTM Models for vision-based human activity recognition

Machine learning and deep learning models for human activity recognition in security and surveillance: a review

Machine learning and deep learning models for human activity recognition in security and surveillance: a review

CrossHAR: Generalizing Cross-dataset Human Activity Recognition via Hierarchical Self-Supervised Pretraining

The increasing availability of low-cost wearable devices and smartphones has significantly advanced the field of sensor-based human activity recognition (HAR), attracting considerable research interest. One of the major challenges in HAR is the domain shift problem in cross-dataset activity recognition, which occurs due to variations in users, device types, and sensor placements between the source dataset and the target dataset. Although domain adaptation methods have shown promise, they typically require access to the target dataset during the training process, which might not be practical in some scenarios. To address these issues, we introduce CrossHAR, a new HAR model designed to improve model performance on unseen target datasets. CrossHAR involves three main steps: (i) CrossHAR explores the sensor data generation principle to diversify the data distribution and augment the raw sensor data. (ii) CrossHAR then employs a hierarchical self-supervised pretraining approach with the augmented data to develop a generalizable representation. (iii) Finally, CrossHAR fine-tunes the pretrained model with a small set of labeled data in the source dataset, enhancing its performance in cross-dataset HAR. Our extensive experiments across multiple real-world HAR datasets demonstrate that CrossHAR outperforms current state-of-the-art methods by 10.83% in accuracy, demonstrating its effectiveness in generalizing to unseen target datasets.

AutoDLAR: A Semi-supervised Cross-modal Contact-free Human Activity Recognition System

WiFi-based human activity recognition (HAR) plays an essential role in various applications such as security surveillance, health monitoring, and smart home. Existing HAR methods, though yielding promising performance in indoor scenarios, highly depend on a massive labeled dataset for training which is extremely difficult to acquire in practical applications. In this paper, we present an automatic data labeling and HAR system, termed AutoDLAR. Taking a semi-supervised cross-modal learning framework with a hybrid loss function as the core, AutoDLAR transfers rich visual information to automatically label WiFi signals for WiFi-based HAR. Specifically, we devise a lightweight and multi-view WiFi sensing model with a parallel feature embedding method to accurately identify activities and accelerate recognition speed. Then, we exploit the video data to fine-tune a well-established visual HAR model, generating effective pseudo-labels for guiding the WiFi model’s training. We also build a synchronized Video-WiFi dataset with seven types of human activities under different scenarios to enable training and validating the semi-supervised HAR system. Extensive experiments on our collected activity dataset and the emotion recognition benchmark demonstrate that AutoDLAR attains an average accuracy of over 95.89% without manual labeling and only spends the inference time of 3.35 ms, outperforming the state-of-the-art (SOTA) methods.