Human Activity Recognition Research Articles

IoT-FAR: A multi-sensor fusion approach for IoT-based firefighting activity recognition

Inadequate training poses a significant risk of injury among young firefighters. Although Human Activity Recognition (HAR) algorithms have shown potential in monitoring and evaluating performance, most existing studies focus on daily activities and have difficulty distinguishing complex firefighting tasks. This study introduces the Internet of things (IoT)-based wearable firefighting activity recognition (IoT-FAR) system which employs a multi-modal sensor fusion approach to achieve comprehensive activity recognition during firefighting training. The IoT-FAR comprises five wearable body sensor nodes and a coordinator node. This study explores the significance of features extracted from the surface electromyography, heart rate, and inertial measurement units in firefighting training activity recognition. A hybrid machine learning (HML)-based network is proposed, which integrates three models: one trained with all features (MA), another with upper body features (MU), and a third with lower body features (ML). The proposed HML-SVM-RBF1-RF2 network achieves superior performance, with a mean recall of 93.94%, mean precision of 90.94%, and a mean accuracy rate of 98.29%. Additionally, the study introduces the specialized firefighting training associated activities (SFTAA) dataset, which includes endurance training activities involving self-contained breathing apparatus (SCBA) conducted by eighteen firefighters. This dataset represents preliminary work towards building a comprehensive dataset covering various events and scenarios for tracking firefighter activities. The IoT-FAR system also demonstrates the potential use of misclassified activities as evaluation metrics for assessing firefighter training performance.

AS-APF: Encoding time series as images for human activity recognition with SK-based convolutional networks

AS-APF: Encoding time series as images for human activity recognition with SK-based convolutional networks

Achieving More with Less: A Lightweight Deep Learning Solution for Advanced Human Activity Recognition (HAR).

Human activity recognition (HAR) is a crucial task in various applications, including healthcare, fitness, and the military. Deep learning models have revolutionized HAR, however, their computational complexity, particularly those involving BiLSTMs, poses significant challenges for deployment on resource-constrained devices like smartphones. While BiLSTMs effectively capture long-term dependencies by processing inputs bidirectionally, their high parameter count and computational demands hinder practical applications in real-time HAR. This study investigates the approximation of the computationally intensive BiLSTM component in a HAR model by using a combination of alternative model components and data flipping augmentation. The proposed modifications to an existing hybrid model architecture replace the BiLSTM with standard and residual LSTM, along with convolutional networks, supplemented by data flipping augmentation to replicate the context awareness typically provided by BiLSTM networks. The results demonstrate that the residual LSTM (ResLSTM) model achieves superior performance while maintaining a lower computational complexity compared to the traditional BiLSTM model. Specifically, on the UCI-HAR dataset, the ResLSTM model attains an accuracy of 96.34% with 576,702 parameters, outperforming the BiLSTM model's accuracy of 95.22% with 849,534 parameters. On the WISDM dataset, the ResLSTM achieves an accuracy of 97.20% with 192,238 parameters, compared to the BiLSTM's 97.23% accuracy with 283,182 parameters, demonstrating a more efficient architecture with minimal performance trade-off. For the KU-HAR dataset, the ResLSTM model achieves an accuracy of 97.05% with 386,038 parameters, showing comparable performance to the BiLSTM model's 98.63% accuracy with 569,462 parameters, but with significantly fewer parameters.

WISNet: A deep neural network based human activity recognition system

Nowadays, Human Activity Recognition (HAR) is a key research area with many ubiquitous innovative solutions, where both accelerometer and gyroscope data provide information about an observed person’s physical activity. HAR offers a diverse variety of important applications, including healthcare, burglary detection, workplace monitoring, and emergency identification. Traditional recognition approaches rely on extracting handmade features from the obtained data to identify the typeof human action. Additionally, the efficacy of these works is dependent upon the specific customized features that are chosen. One potential approach to tackle this issue is to utilize Convolutional Neural Networks (CNN) to automatically learn the relevant features. In this paper, we propose a deep learning model, WISNet, a custom 1D-CNN approach to recognize six complex human activities: Jogging, Walking Downstairs, Sitting, Standing, Walking and Climbing Upstairs. The model includes a Convolved Normalized Pooled (CNPM) Block to generate significant features from the initial layers. An Identity and Basic (IDBN) Block is incorporated to extract residual progressive features for capturing complex sequential data dependencies. Channel and Spatial attention (CASb) Block is integrated with the network to prioritize or minimize essential features based on relative weights. The proposed WISNet model achieved an enhanced accuracy and F1-score of 96.41 % and 0.95 for the HAR dataset by surpassing the existing transfer learning architectures such as Gated Recurrent Units (GRU), Long Short-Term Memory (LSTM), and Recurrent Neural Network (SimpleRNN). By strategically integrating the CNPM, IDBN, and CASb blocks, this study aims to tackle distinct challenges encountered in the classification process by enhancing the discernment of features essential for the precise identification of multi-class human activity recognition. The seamless integration of these blocks within the model plays a pivotal role in elevating the overall performance of the WISNet architecture. The work also validates WISNet with similar open-source datasets (UCI-HAR and KU-HAR) and dissimilar open-source datasets (Sleep state detection, Fall detection, and ECG Heartbeat).

IMUGPT 2.0: Language-Based Cross Modality Transfer for Sensor-Based Human Activity Recognition

One of the primary challenges in the field of human activity recognition (HAR) is the lack of large labeled datasets. This hinders the development of robust and generalizable models. Recently, cross modality transfer approaches have been explored that can alleviate the problem of data scarcity. These approaches convert existing datasets from a source modality, such as video, to a target modality, such as inertial measurement units (IMUs). With the emergence of generative AI models such as large language models (LLMs) and text-driven motion synthesis models, language has become a promising source data modality as well - as shown in proof of concepts such as IMUGPT. In this work, we conduct a large-scale evaluation of language-based cross modality transfer to determine their effectiveness for HAR. Based on this study, we introduce two new extensions for IMUGPT that enhance its use for practical HAR application scenarios: a motion filter capable of filtering out irrelevant motion sequences to ensure the relevance of the generated virtual IMU data, and a set of metrics that measure the diversity of the generated data facilitating the determination of when to stop generating virtual IMU data for both effective and efficient processing. We demonstrate that our diversity metrics can reduce the effort needed for the generation of virtual IMU data by at least 50%, which opens up IMUGPT for practical use cases beyond a mere proof of concept.

Experience: A Comparative Analysis of Multivariate Time-Series Generative Models: A Case Study on Human Activity Data

Human activity recognition (HAR) is an active research field that has seen great success in recent years due to advances in sensory data collection methods and activity recognition systems. Deep artificial intelligence (AI) models have contributed to the success of HAR systems lately although still suffering from limitations such as data scarcity, the high costs of labelling data instances and datasets’ imbalance and bias. The temporal nature of human activity data, represented as time series data, impose an additional challenge to using AI models in HAR because most state-of-the-art models do not account for the time component of the data instances. These limitations have inspired the time-series research community to design generative models for sequential data but very little work has been done to evaluate the quality of such models. In this work, we conduct a comparative quality analysis of three generative models for time-series data, using a case study in which we aim to generate sensory human activity data from a seed public dataset. Additionally, we adapt and clearly explain four evaluation methods of synthetic time-series data from the literature and apply them to assess the quality of the synthetic activity data we generate. We show experimentally that high quality human activity data can be generated using deep generative models, and the synthetic data can thus be used in HAR systems to augment real activity data. We also demonstrate that the chosen evaluation methods effectively ensure that the generated data meets the essential quality benchmarks of realism, diversity, coherence and utility. Our findings suggest that using deep generative models to produce synthetic human activity data can potentially address challenges related to data scarcity, biases, and expensive labeling. This holds promise for enhancing the efficiency and reliability of HAR systems.

Modeling the skeleton-language uncertainty for 3D action recognition

Human 3D skeleton-based action recognition has received increasing interest in recent years. Inspired by the excellent ability of the multi-modal model, some pioneer attempts to employ diverse modalities, i.e., skeleton and language, to construct the skeleton-language model and have shown compelling results. Yet, these attempts model the data representation as deterministic point estimation, ignoring a key issue that descriptions of similar motions are uncertain and ambiguous, which brings about restricted comprehension of complex concept hierarchies and impoverished cross-modal alignment reliability. To tackle this challenge, this paper proposes a new Uncertain Skeleton-Language Learning Framework (USLLF) to capture the semantic ambiguity among diverse modalities in a probabilistic manner for the first time. USLLF consists of both inter- and intra-modal uncertainties. Specifically, first, we integrate the language (text) generated by ChatGPT with the generic skeleton-based network and develop a deterministic multi-modal baseline, which can be easily achieved via any off-the-shelf skeleton and text encoders. Then, based on this baseline, we explicitly model the intra-modal (skeleton/language) uncertainties as the Gaussian distributions using the new uncertainty networks capable of learning the distributional embeddings of modalities. Following this, these embeddings are aligned and formulated as inter-modal (skeleton-language) uncertainty using both the contrastive and negative log-likelihood objectives to alleviate the cross-modal alignment error. Experimental results on NTU RGB+D, NTU RGB+D 120, and NW-UCLA datasets show that our approach outperforms the proposed baseline and achieves comparable performance with a high inference efficiency compared to the state-of-the-art methods. Besides, we also deliver insightful analyses on how learned uncertainty reduces the impact of uncertain and ambiguous data on model performance.

Interpretability in Video-based Human Action Recognition: Saliency Maps and GradCAM in 3D Convolutional Neural Networks

Interpretability plays a vital role in understanding complex deep learning models by providing transparency and insights. It addresses the black-box nature of these models, aids in human-in-the-loop systems, enhances model development, and supports education. However, existing interpretability algorithms in computer vision primarily target images, leaving a gap for video applications. In this article, we emphasize the importance of interpretability in video-based Human Action Recognition (HAR). We extend existing 2D interpretability techniques to the 3D domain, specifically focusing on saliency maps and gradient-weighted class activation maps. The proposed interpretability system is then employed in the analysis of a well-known HAR dataset to better understand action recognition in videos.

Multi-Person Action Recognition Based on Millimeter-Wave Radar Point Cloud

Human action recognition has many application prospects in human-computer interactions, innovative furniture, healthcare, and other fields. The traditional human motion recognition methods have limitations in privacy protection, complex environments, and multi-person scenarios. Millimeter-wave radar has attracted attention due to its ultra-high resolution and all-weather operation. Many existing studies have discussed the application of millimeter-wave radar in single-person scenarios, but only some have addressed the problem of action recognition in multi-person scenarios. This paper uses a commercial millimeter-wave radar device for human action recognition in multi-person scenarios. In order to solve the problems of severe interference and complex target segmentation in multiplayer scenarios, we propose a filtering method based on millimeter-wave inter-frame differences to filter the collected human point cloud data. We then use the DBSCAN algorithm and the Hungarian algorithm to segment the target, and finally input the data into a neural network for classification. The classification accuracy of the system proposed in this paper reaches 92.2% in multi-person scenarios through experimental tests with the five actions we set.

Exposing Data Leakage in Wi-Fi CSI-Based Human Action Recognition: A Critical Analysis

Wi-Fi channel state information (CSI)-based human action recognition systems have garnered significant interest for their non-intrusive monitoring capabilities. However, the integrity of these systems can be compromised by data leakage, particularly when improper dataset partitioning strategies are employed. This paper investigates the presence and impact of data leakage in three published Wi-Fi CSI-based human action recognition methods that utilize deep learning techniques. The original studies achieve precision rates of 95% or higher, attributed to the lack of human-based dataset splitting. By re-evaluating these systems with proper subject-based partitioning, our analysis reveals a substantial decline in performance, underscoring the prevalence of data leakage. This study highlights the critical need for rigorous dataset management and evaluation protocols to ensure the development of robust and reliable human action recognition systems. Our findings advocate for standardized practices in dataset partitioning to mitigate data leakage and enhance the generalizability of Wi-Fi CSI-based models.

TriFusion hybrid model for human activity recognition

TriFusion hybrid model for human activity recognition

WiSigPro: Transformer for elevating CSI-based human activity recognition through attention mechanisms

The utilization of Channel State Information (CSI) in Wi-Fi-based passive sensing has become popular due to its cost-effectiveness and broad applicability. This technology is highly valued for its ability to gather data without requiring active user interaction, making it versatile in various contexts. However, existing methods face significant challenges, including low sensing accuracy, high processing requirements, and system instability. To address these issues, we introduce the WiSigPro Transformer, an advanced CSI-based Wi-Fi passive sensing model specifically designed for Human Activity Recognition. This model employs multi-head attention mechanisms and positional encoding to effectively capture complex spatiotemporal patterns, thereby enhancing both robustness and accuracy. Our approach integrates advanced signal processing techniques to improve signal quality and feature extraction. These techniques include wavelet denoising to reduce noise, median filtering to smooth the signal, and Power Spectral Density analysis using Welch’s method to capture frequency domain features. Additionally, we use normalization to standardize amplitude and phase data, and feature engineering methods to extract comprehensive signal characteristics, such as skewness and kurtosis. To address data imbalance, we apply the Synthetic Minority Over-sampling Technique and data augmentation strategies, ensuring balanced representation and improved model generalization. Through comprehensive simulations, the WiSigPro Transformer demonstrates superior performance across key metrics, including recognition accuracy, precision, recall, and F1-score. Achieving an impressive 98% accuracy, it outperforms conventional neural networks such as CNN, RNN, BiLSTM, LSTM, and ABLSTM. These results underscore the transformative potential of the WiSigPro Transformer in Wi-Fi-based passive sensing and activity recognition applications, making it a powerful tool for accurately capturing and analyzing spatiotemporal data.

Human activity recognition using smart phone for physical activity analysis

Human Activity Recognition (HAR) is an active research area that is useful in many applications, such as elderly people monitoring, health parameter analysis, fall detection, video surveillance, etc. There are many challenges in HAR, such as generalized feature extraction, varying background and illumination, etc. The sensor based and image based are two major approaches in HAR. Image based is complex and can be used in the monitoring area only.The sensor based appraches are not user friendly as an individuals need to wear the sensors on the body which is not comfortable. The use of smart phones for HAR is increasing due to advancements in machine learning algorithms. We have proposed a convolutional neural network model for the classification of activities of daily leaving (ADLs) performed by 30 individuals carrying a smartphone. The model consists of two and three dense layers, respectively. We trained the model using 7352 unique samples from the database. We obtained the highest accuracy of 95.02% when testing the result on 2947 samples with 15 epochs. The accuracy and loss plotted is shown in fig 5 a to d which indicates the proper traing of the model. The model is neither overfitted nor underfitted. The hyper-parameter tuning helped in increasing the accuracy. The classification of the physical activities can be and useful inferences can be obtained by analyzing the activities and its time. This will useful in generating the health related alerts to an individual for the activities.

Deep Learning-based Human Action Recognition in Videos

Deep Learning-based Human Action Recognition in Videos

Edge human activity recognition using federated learning on constrained devices

Human Activity Recognition (HAR) using wearable Internet of Things (IoT) devices represents a well investigated researched field encompassing various application domains. Many current approaches rely on cloud-based methodologies for gathering data from diverse users, resulting in the creation of extensive training datasets. Although this strategy facilitates the application of powerful Machine Learning (ML) techniques, it raises significant privacy concerns, which can become particularly severe given the sensitivity of HAR data. Moreover, the labeling process can be extremely time-consuming and even more challenging for IoT wearable devices due to the absence of efficient input systems. In this paper, we address both aforementioned challenges by designing, implementing, and validating edge-based Human Activity Recognition (HAR) systems that operate on resource-constrained IoT devices, which relies on the utilization of Self-Organizing Maps (SOM) for activity detection. We incorporate a feature selection process before training to reduce data dimensionality and, consequently, the SOM size, aligning with the resource limitations of wearable IoT devices. Additionally, we explore the application of Federated Learning (FL) techniques for HAR tasks, enabling new users to leverage SOM models trained by others on their respective datasets. Our federated Extreme Edge (EE)-aware HAR system is implemented on a wearable IoT device and rigorously tested against state-of-the-art and experimental datasets. The results demonstrate that our C++-based SOM implementation achieves a consistent reduction in model size compared to state-of-the-art approaches. Furthermore, our findings highlight the effectiveness of the FL-based approach in overcoming personalized training challenges, particularly in onboarding scenarios.

Healthcare System from Multisensor Collaboration and Human Action Recognition

Healthcare System from Multisensor Collaboration and Human Action Recognition

Recognizing salat activity using deep learning models via smartwatch sensors

In this study, we focus on human activity recognition, particularly aiming to distinguish the activity of praying (salat) from other daily activities. To achieve this goal, we have created a new dataset named HAR-P (Human activity recognition for Praying), which includes eight different activities: walking, running, sitting, standing, walking upstairs, walking downstairs, typing with a keyboard, and praying (salat). The HAR-P dataset was collected from 50 male individuals, who wore smartwatches on their dominant wrists. We compare the activity classification performance using three state-of-the-art algorithms from the literature: Long Short-Term Memory, Convolutional Long Short-Term Memory, and Convolutional Neural Network—Long Short-Term Memory. To assess the influence of sensors, data from accelerometer, gyroscope, linear acceleration sensor, and magnetic field sensor were utilized. The impact of individual sensor data as well as combinations thereof was investigated. The highest classification accuracy within single sensor groups, reaching 95.7%, was achieved using the accelerometer data with the Convolutional Long Short-Term Memory method. Combining two sensor groups resulted in an increase in accuracy of up to 9%. The highest accuracy of 96.4% was obtained by utilizing three sensor groups together with the Convolutional Neural Network—Long Short-Term Memory method. Furthermore, the evaluation of sensor and model performance was conducted using the stratified k-fold cross-validation method with 5-folds. These findings contribute significantly to evaluating the performance of sensor combinations and different algorithms in activity classification. This study may provide an effective foundation for the automatic recognition and tracking of human activities and offer an applicable model, particularly for the recognition of religious practices such as praying.

Human Visual Pathways for Action Recognition Versus Deep Convolutional Neural Networks: Representation Correspondence in Late But Not Early Layers.

Deep convolutional neural networks (DCNNs) have attained human-level performance for object categorization and exhibited representation alignment between network layers and brain regions. Does such representation alignment naturally extend to other visual tasks beyond recognizing objects in static images? In this study, we expanded the exploration to the recognition of human actions from videos and assessed the representation capabilities and alignment of two-stream DCNNs in comparison with brain regions situated along ventral and dorsal pathways. Using decoding analysis and representational similarity analysis, we show that DCNN models do not show hierarchical representation alignment to human brain across visual regions when processing action videos. Instead, later layers of DCNN models demonstrate greater representation similarities to the human visual cortex. These findings were revealed for two display formats: photorealistic avatars with full-body information and simplified stimuli in the point-light display. The discrepancies in representation alignment suggest fundamental differences in how DCNNs and the human brain represent dynamic visual information related to actions.

Multi-granular spatial-temporal synchronous graph convolutional network for robust action recognition

Graph Convolutional Networks (GCNs) have shown great potential in skeleton-based human action recognition. However, due to the diversity and complexity, modeling human actions as general graphs and capturing discriminative spatial–temporal motion patterns is challenging. Besides, the inevitable interference, especially occlusion, impairs the robustness of existing methods that depend on complete skeletons. To solve these problems, we propose a Multi-Granular Spatial-Temporal Synchronous Graph Convolutional Network (MSS-GCN). Firstly, we investigate three partition strategies: attribute, activity, and mixed partition strategy to optimize the weight-sharing mechanism of GCNs, which facilitates the novel Extended Adaptive Graph Convolution (EAGC) module. Secondly, we elaborate on a Multi-sliced Spatial–temporal Graph (MSTG) for multi-granular action modeling. Thirdly, we present a Synchronized Slice Encoder (Syn-STE) to simultaneously embed spatial and temporal action patterns. Then, we design Multi-granular Spatial–temporal Encoders (Multi-STE) with multi-branch Syn-STE to generate multi-granular context. The extensive experiments verified that MSS-GCN is more robust and outperforms benchmarks on NTU-RGB+D, NTU-RGB+D 120, and NW-UCLA datasets.

Large Language Models for Wearable Sensor-Based Human Activity Recognition, Health Monitoring, and Behavioral Modeling: A Survey of Early Trends, Datasets, and Challenges.

The proliferation of wearable technology enables the generation of vast amounts of sensor data, offering significant opportunities for advancements in health monitoring, activity recognition, and personalized medicine. However, the complexity and volume of these data present substantial challenges in data modeling and analysis, which have been addressed with approaches spanning time series modeling to deep learning techniques. The latest frontier in this domain is the adoption of large language models (LLMs), such as GPT-4 and Llama, for data analysis, modeling, understanding, and human behavior monitoring through the lens of wearable sensor data. This survey explores the current trends and challenges in applying LLMs for sensor-based human activity recognition and behavior modeling. We discuss the nature of wearable sensor data, the capabilities and limitations of LLMs in modeling them, and their integration with traditional machine learning techniques. We also identify key challenges, including data quality, computational requirements, interpretability, and privacy concerns. By examining case studies and successful applications, we highlight the potential of LLMs in enhancing the analysis and interpretation of wearable sensor data. Finally, we propose future directions for research, emphasizing the need for improved preprocessing techniques, more efficient and scalable models, and interdisciplinary collaboration. This survey aims to provide a comprehensive overview of the intersection between wearable sensor data and LLMs, offering insights into the current state and future prospects of this emerging field.