Human Activity Recognition Framework Research Articles

Human action recognition with transformer based on convolutional features

As one of the key research directions in the field of computer vision, human action recognition has a wide range of practical application values and prospects. In the fields of video surveillance, human-computer interaction, sports analysis, and healthcare, human action recognition technology shows a broad application prospect and potential. However, the diversity and complexity of human actions bring many challenges, such as handling complex actions, distinguishing similar actions, coping with changes in viewing angle, and overcoming occlusion problems. To address the challenges, this paper proposes an innovative framework for human action recognition. The framework combines the latest pose estimation algorithms, pre-trained CNN models, and a Vision Transformer to build an efficient system. The first step involves utilizing the latest pose estimation algorithm to accurately extract human pose information from real RGB image frames. Then, a pre-trained CNN model is used to perform feature extraction on the extracted pose information. Finally, the Vision Transformer model is applied for fusion and classification operations on the extracted features. Experimental validation is conducted on two benchmark datasets, UCF 50 and UCF 101, to demonstrate the effectiveness and efficiency of the proposed framework. The applicability and limitations of the framework in different scenarios are further explored through quantitative and qualitative experiments, providing valuable insights and inspiration for future research.

Multi-head CNN-based activity recognition and its application on chest-mounted sensor-belt

In recent years, a great deal of research has been done on the identification, monitoring, and classification of human activities. Human activity recognition (HAR) is a term commonly used to describe the automatic identification of physical activities. For activity recognition, there are primarily vision-based and sensor-based methods available. The computer vision-based method is generally effective in lab settings, but because of clutter, fluctuating light levels, and contrast, it may not perform well in real-world scenarios. Continuous monitoring and analysis of physiological signals obtained from heterogeneous sensors attached to an individual’s body is required to realise sensor-based HAR systems. Most of the previous research in human activity recognition (HAR) is biased along with feature engineering and pre-processing which requires a good amount of domain knowledge. Application-specific modelling and time-taking methods are involved in these approaches. In this work, the multi-head convolutional neural network-based human activity recognition framework is proposed where automatic feature extraction and classification are involved in the form of an end-to-end classification approach. Experiments of this approach are performed by taking raw wearable sensor data with few pre-processing steps and without the involvement of a handcrafted feature extraction technique. 99.23% and 93.55% accuracy are obtained on the WISDM and UCI-HAR datasets which denoted the much improvement in the assessment of HAR over other similar approaches. The model is also tested on locally collected data from a chest mounted belt with fabric sensors and an accuracy of 87.14% has been achieved on that data.

Multi-modal lifelog data fusion for improved human activity recognition: A hybrid approach

The rapid growth of lifelog data, collected through smartphones and wearable devices, has driven the need for better Human Activity Recognition (HAR) solutions. However, lifelog data is complex and challenging to analyze due to its diverse sources of information. In response, we introduce an innovative hybrid data fusion framework for HAR. This framework comprises three key elements: a hybrid fusion mechanism, an attention-based classifier, and an ensemble-based recognition approach. Our hybrid fusion mechanism expertly combines the advantages of late and intermediate fusion, enhancing classification performance and improving the network’s ability to learn connections between different data modalities. Additionally, our solution incorporates an attention-based classifier and an ensemble approach, ensuring robust and consistent performance in real-world scenarios. We evaluated our method across multiple public lifelog datasets, demonstrating that our hybrid fusion approach consistently surpasses existing fusion strategies in HAR, promising significant advancements in activity recognition.

WIFI based human activity recognition using multi-head adaptive attention mechanism

Human Activity Recognition (HAR) utilizing Channel State Information (CSI) extracted from WiFi signals has garnered substantial interest across various domains and applications. This field’s potential paths and applications extend beyond CSI-based HAR and include smart homes, assisted living, security, gaming, surveillance, and context-aware computing. The ability of deep learning algorithms to effectively process and interpret CSI data opens up new possibilities for accurate and robust human activity recognition in real-world scenarios. However, traditional Recurrent Neural Networks (RNN) models, such as Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU), rely solely on their internal memory cells to maintain information over time. Important details might be diluted or lost within the memory cells in complex CSI sequences. To address this limitation, we propose a lightweight approach that incorporates a multi-head adaptive attention weight mechanism MHAAM into the HAR framework. The multi-head attention mechanism allows the model to attend to different informative patterns within the CSI data simultaneously, capturing fine-grained temporal dependencies and improving the model’s ability to recognize complex activities. The implemented models effectively filter out noise and irrelevant information by assigning higher weights to informative CSI features, further enhancing activity classification accuracy. Experimental evaluations and comparative analyses of HAR for seven activities demonstrate that attention-based RNN models with multi-head attention consistently outperform traditional RNN models. The multi-head attention mechanism achieves improved generalization and testing for seven common human activities and environments, leading to a higher complex human activity classification accuracy of up to 98.5%.

AP-TransNet: a polarized transformer based aerial human action recognition framework

AP-TransNet: a polarized transformer based aerial human action recognition framework

Shots segmentation-based optimized dual-stream framework for robust human activity recognition in surveillance video

Nowadays, for controlling crime, surveillance cameras are typically installed in all public places to ensure urban safety and security. However, automating Human Activity Recognition (HAR) using computer vision techniques faces several challenges such as lowlighting, complex spatiotemporal features, clutter backgrounds, and inefficient utilization of surveillance system resources. Existing attempts in HAR designed straightforward networks by analyzing either spatial or motion patterns resulting in limited performance while the dual streams methods are entirely based on Convolutional Neural Networks (CNN) that are inadequate to learning the long-range temporal information for HAR. To overcome the above-mentioned challenges, this paper proposes an optimized dual stream framework for HAR which mainly consists of three steps. First, a shots segmentation module is introduced in the proposed framework to efficiently utilize the surveillance system resources by enhancing the lowlight video stream and then it detects salient video frames that consist of human. This module is trained on our own challenging Lowlight Human Surveillance Dataset (LHSD) which consists of both normal and different levels of lowlighting data to recognize humans in complex uncertain environments. Next, to learn HAR from both contextual and motion information, a dual stream approach is used in the feature extraction. In the first stream, it freezes the learned weights of the backbone Vision Transformer (ViT) B-16 model to select the discriminative contextual information. In the second stream, ViT features are then fused with the intermediate encoder layers of FlowNet2 model for optical flow to extract a robust motion feature vector. Finally, a two stream Parallel Bidirectional Long Short-Term Memory (PBiLSTM) is proposed for sequence learning to capture the global semantics of activities, followed by Dual Stream Multi-Head Attention (DSMHA) with a late fusion strategy to optimize the huge features vector for accurate HAR. To assess the strength of the proposed framework, extensive empirical results are conducted on real-world surveillance scenarios and various benchmark HAR datasets that achieve 78.6285%, 96.0151%, and 98.875% accuracies on HMDB51, UCF101, and YouTube Action, respectively. Our results show that the proposed strategy outperforms State-of-the-Art (SOTA) methods. The proposed framework gives superior performance in HAR, providing accurate and reliable recognition of human activities in surveillance systems.

A Self-Supervised Human Activity Recognition Approach via Body Sensor Networks in Smart City

In smart cities, pervasive sensing and wearable computing techniques are increasingly being employed to monitor and recognize human activities through body sensor networks, which have been widely used in urban safety, healthcare, and manufacturing. However, most researchers regard human activity recognition (HAR) as a high-cost research task requiring a large amount of labeled data, which is often unrealistic in real-world applications. To address this problem, we propose a self-supervised learning framework for HAR (SS-HAR). SS-HAR initially takes the unlabeled data generated by data augmentation as the input of the network, mines the supervised information of the unlabeled data under the effect of self-supervised learning, and uses the obtained backbone network as a feature extractor to extract activity features for subsequent classification. After that, we use part of the labeled data as the training set and extract the activity features using the backbone network for training and fitting the classifier. Then we utilize the rest of the data to verify the feasibility and effectiveness of the proposed self-supervised learning method. We have conducted multiple experiments on three publicly available datasets and one self-collected basketball activity dataset SZU_HAD_Basketball. The experimental results show that the SS-HAR method is able to achieve higher classification accuracy and stability than supervised and semi-supervised methods. Specifically, on the UCI dataset, SS-HAR achieves better classification performance compared to other approaches which improve the classification accuracy by 1% over the supervised method and by 5%-6% over the semi-supervised method respectively.

Residual deep gated recurrent unit-based attention framework for human activity recognition by exploiting dilated features

Residual deep gated recurrent unit-based attention framework for human activity recognition by exploiting dilated features

A human activity recognition framework in videos using segmented human subject focus

A human activity recognition framework in videos using segmented human subject focus

Subsampled Randomized Hadamard Transformation-based Ensemble Extreme Learning Machine for Human Activity Recognition

Extreme Learning Machine (ELM) is becoming a popular learning algorithm due to its diverse applications, including Human Activity Recognition (HAR). In ELM, the hidden node parameters are generated at random, and the output weights are computed analytically. However, even with a large number of hidden nodes, feature learning using ELM may not be efficient for natural signals due to its shallow architecture. Due to noisy signals of the smartphone sensors and high dimensional data, substantial feature engineering is required to obtain discriminant features and address the “curse-of-dimensionality”. In traditional ML approaches, dimensionality reduction and classification are two separate and independent tasks, increasing the system’s computational complexity. This research proposes a new ELM-based ensemble learning framework for human activity recognition to overcome this problem. The proposed architecture consists of two key parts: (1) Self-taught dimensionality reduction followed by classification. (2) they are bridged by “Subsampled Randomized Hadamard Transformation” (SRHT). Two different HAR datasets are used to establish the feasibility of the proposed framework. The experimental results clearly demonstrate the superiority of our method over the current state-of-the-art methods.

DiamondNet: A Neural-Network-Based Heterogeneous Sensor Attentive Fusion for Human Activity Recognition.

With the proliferation of intelligent sensors integrated into mobile devices, fine-grained human activity recognition (HAR) based on lightweight sensors has emerged as a useful tool for personalized applications. Although shallow and deep learning algorithms have been proposed for HAR problems in the past decades, these methods have limited capability to exploit semantic features from multiple sensor types. To address this limitation, we propose a novel HAR framework, DiamondNet, which can create heterogeneous multisensor modalities, denoise, extract, and fuse features from a fresh perspective. In DiamondNet, we leverage multiple 1-D convolutional denoising autoencoders (1-D-CDAEs) to extract robust encoder features. We further introduce an attention-based graph convolutional network to construct new heterogeneous multisensor modalities, which adaptively exploit the potential relationship between different sensors. Moreover, the proposed attentive fusion subnet, which jointly employs a global-attention mechanism and shallow features, effectively calibrates different-level features of multiple sensor modalities. This approach amplifies informative features and provides a comprehensive and robust perception for HAR. The efficacy of the DiamondNet framework is validated on three public datasets. The experimental results demonstrate that our proposed DiamondNet outperforms other state-of-the-art baselines, achieving remarkable and consistent accuracy improvements. Overall, our work introduces a new perspective on HAR, leveraging the power of multiple sensor modalities and attention mechanisms to significantly improve the performance.

Optimization-Free Test-Time Adaptation for Cross-Person Activity Recognition

Human Activity Recognition (HAR) models often suffer from performance degradation in real-world applications due to distribution shifts in activity patterns across individuals. Test-Time Adaptation (TTA) is an emerging learning paradigm that aims to utilize the test stream to adjust predictions in real-time inference, which has not been explored in HAR before. However, the high computational cost of optimization-based TTA algorithms makes it intractable to run on resource-constrained edge devices. In this paper, we propose an Optimization-Free Test-Time Adaptation (OFTTA) framework for sensor-based HAR. OFTTA adjusts the feature extractor and linear classifier simultaneously in an optimization-free manner. For the feature extractor, we propose Exponential Decay Test-time Normalization (EDTN) to replace the conventional batch normalization (CBN) layers. EDTN combines CBN and Test-time batch Normalization (TBN) to extract reliable features against domain shifts with TBN's influence decreasing exponentially in deeper layers. For the classifier, we adjust the prediction by computing the distance between the feature and the prototype, which is calculated by a maintained support set. In addition, the update of the support set is based on the pseudo label, which can benefit from reliable features extracted by EDTN. Extensive experiments on three public cross-person HAR datasets and two different TTA settings demonstrate that OFTTA outperforms the state-of-the-art TTA approaches in both classification performance and computational efficiency. Finally, we verify the superiority of our proposed OFTTA on edge devices, indicating possible deployment in real applications. Our code is available at https://github.com/Claydon-Wang/OFTTA.

Praxis: a framework for AI-driven human action recognition in assembly

Praxis: a framework for AI-driven human action recognition in assembly

A Privacy and Energy-Aware Federated Framework for Human Activity Recognition

Human activity recognition (HAR) using wearable sensors enables continuous monitoring for healthcare applications. However, the conventional centralised training of deep learning models on sensor data poses challenges related to privacy, communication costs, and on-device efficiency. This paper proposes a federated learning framework integrating spiking neural networks (SNNs) with long short-term memory (LSTM) networks for energy-efficient and privacy-preserving HAR. The hybrid spiking-LSTM (S-LSTM) model synergistically combines the event-driven efficiency of SNNs and the sequence modelling capability of LSTMs. The model is trained using surrogate gradient learning and backpropagation through time, enabling fully supervised end-to-end learning. Extensive evaluations of two public datasets demonstrate that the proposed approach outperforms LSTM, CNN, and S-CNN models in accuracy and energy efficiency. For instance, the proposed S-LSTM achieved an accuracy of 97.36% and 89.69% for indoor and outdoor scenarios, respectively. Furthermore, the results also showed a significant improvement in energy efficiency of 32.30%, compared to simple LSTM. Additionally, we highlight the significance of personalisation in HAR, where fine-tuning with local data enhances model accuracy by up to 9% for individual users.

Lightweight accurate trigger to reduce power consumption in sensor-based continuous human activity recognition

Wearable devices have become increasingly popular in recent years, and they offer a great opportunity for sensor-based continuous human activity recognition in real-world scenarios. However, one of the major challenges is their limited battery life. In this study, we propose an energy-aware human activity recognition framework for wearable devices based on a lightweight accurate trigger. The trigger acts as a binary classifier capable of recognizing, with maximum accuracy, the presence or absence of one of the interesting activities in the real-time input signal and it is responsible for starting the energy-intensive classification procedure only when needed. The measurement results conducted on a real wearable device show that the proposed approach can reduce energy consumption by up to 95% in realistic case studies, with a cost of performance deterioration of at most 1% or 2% compared to the traditional energy-intensive classification strategy.

An Adaptive Batch Size-Based-CNN-LSTM Framework for Human Activity Recognition in Uncontrolled Environment

Human Activity Recognition (HAR) is a process of identifying the daily living activities of an individual using a set of sensors and appropriate learning algorithms. Most of the works on HAR are done using a mix of sensor data that is collected in a simulated environment, and due to that, the real-time recognition suffers. This paper proposes an efficient adaptive batch size based-CNN-LSTM model for recognising different human activities in an uncontrolled environment. It uses adaptive batch sizes from 128 to 1024 for iterative model training and validation. The proposed model can handle imbalanced classes and un-normalised data efficiently. A state-of-art HAR dataset is also generated in an open environment to get the activity data of an uncontrolled scenario. With minimal data pre-processing and data augmentation, the model is tested, and the proposed model managed to get the highest accuracy of 99.29% with an average loss of 0.08 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\pm$</tex-math></inline-formula> 0.136%. The presented model is also tested with two public datasets named- mHealth and MotionSense and achieved an accuracy of 99.5% and 99.8%, respectively. The proposed model outperforms all the previous benchmarks and approaches by a good margin.

A general skeleton-based action and gesture recognition framework for human–robot collaboration

Recognizing human actions is crucial for an effective and safe collaboration between humans and robots. For example, in a collaborative assembly task, human workers can use gestures to communicate with the robot, and the robot can use the recognized actions to anticipate the next steps in the assembly process, leading to improved safety and productivity. In this work, we propose a general framework for human action recognition based on 3D pose estimation and ensemble techniques, which allows to recognize both body actions and hand gestures. The framework relies on OpenPose and 2D to 3D lifting methods to estimate 3D joints for the human body and the hands, feeding then these joints into a set of graph convolutional networks based on the Shift-GCN architecture. The output scores of all networks are combined using an ensemble approach to predict the final human action. The proposed framework was evaluated on a custom dataset designed for human–robot collaboration tasks, named IAS-Lab Collaborative HAR dataset. The results showed that using an ensemble of action recognition models improves the accuracy and robustness of the overall system; moreover, the proposed framework can be easily specialized on different scenarios and achieve state-of-the-art results on the HRI30 dataset when coupled with an object detector or classifier.

ImgFi: A High Accuracy and Lightweight Human Activity Recognition Framework Using CSI Image

ImgFi: A High Accuracy and Lightweight Human Activity Recognition Framework Using CSI Image

ProtoHAR: Prototype Guided Personalized Federated Learning for Human Activity Recognition.

Federated Learning (FL) has recently attracted great interest in sensor-based human activity recognition (HAR) tasks. However, in real-world environment, sensor data on devices is non-independently and identically distributed (Non-IID), e.g., activity data recorded by most devices is sparse, and sensor data distribution for each client may be inconsistent. As a result, the traditional FL methods in the heterogeneous environment may incur a drifted global model that causes slow convergence and a heavy communication burden. Although some FL methods are gradually being applied to HAR, they are designed for overly ideal scenarios and do not address such Non-IID problem in the real-world setting. It is still a question whether they can be applied to cross-device FL. To tackle this challenge, we propose ProtoHAR, a prototype-guided FL framework for HAR, which aims to decouple the representation and classifier in the heterogeneous FL setting efficiently. It leverages the global prototype to correct the activity feature representation to make the prototype knowledge flow among clients without leaking privacy while solving a better classifier to avoid excessive drift of the local model in personalized training. Extensive experiments are conducted on four publicly available datasets: USC-HAD, UNIMIB-SHAR, PAMAP2, and HARBOX, which are collected in both controlled environments and real-world scenarios. The results show that compared with the state-of-the-art FL algorithms, ProtoHAR achieves the best performance and faster convergence speed in HAR datasets.

A Framework for Personalized Human Activity Recognition

In today’s world, Human Activity Recognition (HAR) through video streams is actively used in every aspect of our life, such as automated surveillance systems and sports statistics are computed according to the videos with the help of HAR. Activity detection is not a new subject, and several methods are available. However, the most recent and most promising techniques rely on Convolutional Neural Networks (CNNs). CNNs primary usage is based on a single image frame to perform logical or categorical identification of an object, scene, or activity. We exploit this feature to adapt CNN on video streams to achieve HAR. In this study, we present a Personalized HAR (PHAR) framework that increases activity recognition accuracy with Object Detection (OD). First, we demonstrate the state-of-the-art HAR and OD methods in the literature. Then we illustrate our framework with two new Single Person Human Activity Recognition models. Finally, the performance of the new framework is evaluated with the well-known activity detection methods. Results show that our new PHAR model with 95% accuracy ratio outperforms the CNN-LSTM-based reference model (90%). Moreover, a new metric Average Accuracy Score (AAS) is described in this study, PHAR models approximately have 94% AAS, which is better than the reference model with 89% AAS.