Human Activity Recognition Research Articles

DiamondNet: A Neural-Network-Based Heterogeneous Sensor Attentive Fusion for Human Activity Recognition.

With the proliferation of intelligent sensors integrated into mobile devices, fine-grained human activity recognition (HAR) based on lightweight sensors has emerged as a useful tool for personalized applications. Although shallow and deep learning algorithms have been proposed for HAR problems in the past decades, these methods have limited capability to exploit semantic features from multiple sensor types. To address this limitation, we propose a novel HAR framework, DiamondNet, which can create heterogeneous multisensor modalities, denoise, extract, and fuse features from a fresh perspective. In DiamondNet, we leverage multiple 1-D convolutional denoising autoencoders (1-D-CDAEs) to extract robust encoder features. We further introduce an attention-based graph convolutional network to construct new heterogeneous multisensor modalities, which adaptively exploit the potential relationship between different sensors. Moreover, the proposed attentive fusion subnet, which jointly employs a global-attention mechanism and shallow features, effectively calibrates different-level features of multiple sensor modalities. This approach amplifies informative features and provides a comprehensive and robust perception for HAR. The efficacy of the DiamondNet framework is validated on three public datasets. The experimental results demonstrate that our proposed DiamondNet outperforms other state-of-the-art baselines, achieving remarkable and consistent accuracy improvements. Overall, our work introduces a new perspective on HAR, leveraging the power of multiple sensor modalities and attention mechanisms to significantly improve the performance.

LHAR: Lightweight Human Activity Recognition on Knowledge Distillation.

Sensor-based Human Activity Recognition (HAR) is widely used in daily life and is the basic-level bridge to virtual healthcare in the metaverse. The current challenge is the low recognition accuracy for personalized users on smart wearable devices. The limited resource cannot support large deep learning models updated locally. Besides, integrating and transmitting sensor data to the cloud would reduce the efficiency. Considering the tradeoff between performance and complexity, we propose a Lightweight Human Activity Recognition (LHAR) framework. In LHAR, we combine the cross-people HAR task with the lightweight model task. LHAR framework is designed on the teacher-student architecture and the student network consists of multiple depthwise separable convolution layers to achieve fewer parameters. The dark knowledge distilled from the complex teacher model enhances the generalization ability of LHAR. To achieve effective knowledge distillation, we propose two optimization methods. Firstly, we train the teacher model by ensemble learning to promote teacher performance. Secondly, a multi-channel data augmentation method is proposed for the diversity of the dataset, which is a plug-in operation for the ensemble teacher model. In the experiments, we compare LHAR with state-of-art models in comparison evaluation, ablation study and the hyperparameter analysis, which proves the better performance of LHAR in efficiency and effectiveness.

OTM-HC: Enhanced Skeleton-Based Action Representation via One-to-Many Hierarchical Contrastive Learning

Human action recognition has become crucial in computer vision, with growing applications in surveillance, human–computer interaction, and healthcare. Traditional approaches often use broad feature representations, which may miss subtle variations in timing and movement within action sequences. Our proposed One-to-Many Hierarchical Contrastive Learning (OTM-HC) framework maps the input into multi-layered feature vectors, creating a hierarchical contrast representation that captures various granularities within a human skeleton sequence temporal and spatial domains. Using sequence-to-sequence (Seq2Seq) transformer encoders and downsampling modules, OTM-HC can distinguish between multiple levels of action representations, such as instance, domain, clip, and part levels. Each level contributes significantly to a comprehensive understanding of action representations. The OTM-HC model design is adaptable, ensuring smooth integration with advanced Seq2Seq encoders. We tested the OTM-HC framework across four datasets, demonstrating improved performance over state-of-the-art models. Specifically, OTM-HC achieved improvements of 0.9% and 0.6% on NTU60, 0.4% and 0.7% on NTU120, and 0.7% and 0.3% on PKU-MMD I and II, respectively, surpassing previous leading approaches across these datasets. These results showcase the robustness and adaptability of our model for various skeleton-based action recognition tasks.

Toward Lightweight End-to-End Semantic Learning of Real-Time Human Activity Recognition for Enabling Ambient Intelligence

Toward Lightweight End-to-End Semantic Learning of Real-Time Human Activity Recognition for Enabling Ambient Intelligence

KGAN-Based Semisupervised Domain Adapted Human Activity Recognition

KGAN-Based Semisupervised Domain Adapted Human Activity Recognition

EGTCN: An Efficient Graph and Temporal Convolution Network for Sensor-Based Human Activity Recognition in Federated Learning

EGTCN: An Efficient Graph and Temporal Convolution Network for Sensor-Based Human Activity Recognition in Federated Learning

MHCNLS-HAR: Multiheaded CNN-LSTM-Based Human Activity Recognition Leveraging a Novel Wearable Edge Device for Elderly Health Care

MHCNLS-HAR: Multiheaded CNN-LSTM-Based Human Activity Recognition Leveraging a Novel Wearable Edge Device for Elderly Health Care

Enhancing inertial sensor-based sports activity recognition through reduction of the signals and deep learning

The increasing demand for sport training and health monitoring aligns with contemporary lifestyle trends. Developing a system to support sport training and verify exercise correctness can significantly enhance the acquisition of detailed, subject-specific data. This study aims to evaluate the accuracy of sport exercise recognition while minimizing the number of sensors required. The dataset includes 8,968 samples of exercises such as bar dips, squats, dips, lunges, pull-ups, sit-ups, and push-ups. Data were collected from 60 signals using mobile sensors positioned at the chest, right hand, and right foot of 21 subjects. The objective is to identify the most efficient method for recognizing these activities. Our methodology involved experiments with 21 participants in a custom-built setup and the application of deep learning techniques. Initially, a novel algorithm identified the optimal set of signals. We then tested two scenarios: training a Convolutional Neural Network (CNN) on raw signals and on pre-processed signals to reduce noise. The findings indicate that the magnetic field (MF) signal is crucial for recognizing exercises in both filtered and unfiltered data sets. The CNN’s accuracy was 2–3% higher with unfiltered data and remained robust at 93.7% for training and 90.0% for testing, despite a reduction in model complexity. This method’s practical implications are significant, enhancing sports training systems by reducing the number of sensors needed, thereby improving user comfort. Additionally, it contributes valuable insights to the field of Human Activity Recognition.

A unified framework for unsupervised action learning via global-to-local motion transformer

Human action recognition remains challenging due to the inherent complexity arising from the combination of diverse granularity of semantics, ranging from the local motion of body joints to high-level relationships across multiple people. To learn this multi-level characteristic of human action in an unsupervised manner, we propose a novel pretraining strategy along with a transformer-based model architecture named GL-Transformer++. Prior methods in unsupervised action recognition or unsupervised group activity recognition (GAR) have shown limitations, often focusing solely on capturing a partial scope of the action, such as the local movements of each individual or the broader context of the overall motion. To tackle this problem, we introduce a novel pretraining strategy named multi-interval pose displacement prediction (MPDP) that enables the model to learn the diverse extents of the action. In the architectural aspect, we incorporate the global and local attention (GLA) mechanism within the transformer blocks to learn local dynamics between joints, global context of each individual, as well as high-level interpersonal relationships in both spatial and temporal manner. In fact, the proposed method is a unified approach that demonstrates efficacy in both action recognition and GAR. Particularly, our method presents a new and strong baseline, surpassing the current SOTA GAR method by significant margins: 29.6% in Volleyball and 60.3% and 59.9% on the xsub and xset settings of the Mutual NTU dataset, respectively.

Dual-path aggregation transformer network for super-resolution with images occlusions and variability

While Transformer-based approaches have recently achieved notable success in super-resolution, their extensive computational requirements impede widespread practical adoption. High-resolution meteorological satellite cloud imagery is essential for weather analysis and forecasting. Enhancing image resolution through super-resolution techniques facilitates the accurate identification and localization of geographic features by meteorological systems. However, current super-resolution methods fail to restore the intricacies of cloud formations and complex regions fully. This research introduces a novel dual-path aggregation Transformer network (DPAT) tailored to enhance the super-resolution of meteorological satellite cloud images. The DPAT network adeptly captures cloud imagery's subtle details and textures, effectively addressing occlusions and the variability inherent in satellite imagery. It bolsters the model's ability to manage the complex attributes of cloud images through the introduction of the Dual-path Aggregation Self-Attention (DASA) mechanism and the Multi-scale Feature Aggregation Block (MFAB), thereby enhancing performance in processing intricate cloud features. The DASA mechanism synthesizes features across spatial, depth, and channel dimensions via a dual-path approach, thoroughly exploiting feature correlations. The MFAB, designed to supplant the multilayer perceptron, incorporates shift convolution and a multi-scale interaction block to augment feature information, compensating for the deficiency in local information absorption due to fixed receptive fields. Experimental outcomes indicate that DPAT delivers superior super-resolution outcomes. With a parameter count of only 32% of the Enhanced Deep Residual Network (EDSR) or 77% of the Image Restoration using Shift Window Transformer (SwinIR), DPAT matches SwinIR's performance on the satellite cloud dataset. Moreover, DPAT balances accuracy and parameter economy across various datasets. This technology is expected to improve image super-resolution capabilities in multiple fields such as human action recognition and industrial recognition, and indirectly improve the accuracy of image perception tasks.

Action recognition using attention-based spatio-temporal VLAD networks and adaptive video sequences optimization

In the field of human action recognition, it is a long-standing challenge to characterize the video-level spatio-temporal features effectively. This is attributable in part to the inability of CNN to model long-range temporal information, especially for actions that consist of multiple staged behaviors. In this paper, a novel attention-based spatio-temporal VLAD network (AST-VLAD) with self-attention model is developed to aggregate the informative deep features across the video according to the adaptive deep feature selected. Moreover, an overall automatic approach to adaptive video sequences optimization (AVSO) is proposed through shot segmentation and dynamic weighted sampling, the AVSO increase in the proportion of action-related frames and eliminate the redundant intervals. Then, based on the optimized video, a self-attention model is introduced in AST-VLAD to modeling the intrinsic spatio-temporal relationship of deep features instead of solving the frame-level features in an average or max pooling manner. Extensive experiments are conducted on two public benchmarks-HMDB51 and UCF101 for evaluation. As compared to the existing frameworks, results show that the proposed approach performs better or as well in the accuracy of classification on both HMDB51 (73.1% ) and UCF101 (96.0%) datasets.

Fusing human action recognition and object detection for human-robot collaborative assembly

Fusing human action recognition and object detection for human-robot collaborative assembly

Extended multi-stream temporal-attention module for skeleton-based human action recognition (HAR)

Graph convolutional networks (GCNs) are an effective skeleton-based human action recognition (HAR) technique. GCNs enable the specification of CNNs to a non-Euclidean frame that is more flexible. The previous GCN-based models still have a lot of issues: (I) The graph structure is the same for all model layers and input data. GCN model's hierarchical structure and human action recognition input diversity make this a problematic approach; (II) Bone length and orientation are understudied due to their significance and variance in HAR. For this purpose, we introduce an Extended Multi-stream Temporal-attention Adaptive GCN (EMS-TAGCN). By training the network topology of the proposed model either consistently or independently according to the input data, this data-based technique makes graphs more flexible and faster to adapt to a new dataset. A spatial, temporal, and channel attention module helps the adaptive graph convolutional layer focus on joints, frames, and features. Hence, a multi-stream framework representing bones, joints, and their motion enhances recognition accuracy. Our proposed model outperforms the NTU RGBD for CS and CV by 0.6% and 1.4%, respectively, while Kinetics-skeleton Top-1 and Top-5 are 1.4% improved, UCF-101 has improved 2.34% accuracy and HMDB-51 dataset has significantly improved 1.8% accuracy. According to the results, our model has performed better than the other models. Our model consistently outperformed other models, and the results were statistically significant that demonstrating the superiority of our model for the task of HAR and its ability to provide the most reliable and accurate results.

Segment-Based Unsupervised Deep Learning for Human Activity Recognition using Accelerometer Data and SBOA based Channel Attention Networks

The deep learning community has increasingly focused on the critical challenges of human activity segmentation and detection based on sensors, which have numerous real-world applications. In most prior efforts, activity segmentation and recognition have been treated as separate processes, relying on pre-segmented sensor streams. This research proposes an unsupervised deep learning approach for Human Activity Recognition (HAR) that is segment-based, with an emphasis on activity continuity. The approach integrates segment-based SimCLR with Segment Feature Decorrelation (SDFD) and a new framework that leverages pairs of segment data for contrastive learning of visual representations. Furthermore, the Secretary Bird Optimization Algorithm (SBOA) and Channel Attention with Spatial Attention Network (CASANet) are utilized to enhance the performance of sensor-based human activity detection. CASANet effectively extracts key features and spatial dependencies in sensor data, while SBOA optimizes the model for greater accuracy and generalization. Evaluations on two publicly available datasets—Mhealth and PAMAP2—demonstrated an average F1 score of 98%, highlighting the approach’s efficacy in improving activity recognition performance.

The Adversarial Attacks towards Human Action Recognition Models: A Comparison between unimodal and multimodal Models

Human-action recognition models are neural networks that analyse visual inputs and provide classification or text outputs. This technology has and will significantly impact society in security, education, healthcare, etc. However, Human-Action Recognition models, like other neural networks, are still susceptible to malicious adversarial attacks. Therefore, this paper proposes an experimental adversarial attack towards ResNet-18 using FGSM. First, ResNet-18 is finetuned using the UCF-101 dataset, and keyframes are selected from sample videos. The keyframes will be given to ResNet-18 for classification while FGSM will be implemented, and ResNet will do another classification of the attached sample. The classification results (Original and Attacked) are given to the language model (GPT-4o) through a prompt that provides the language model with a specific role (e.g. a smart home assistant), and this section is regarded as unimodal. The original and attacked frames will be sent directly instead of the labels in the multimodal section. Lastly, this paper proposes to observe the effects on textual responses generated based on a given prompt and the classification result and evaluate the impact of the attack through Cosine Similarities and Human Evaluation.

Benchmarking In-Sensor Machine Learning Computing: An Extension to the MLCommons-Tiny Suite

This paper proposes a new benchmark specifically designed for in-sensor digital machine learning computing to meet an ultra-low embedded memory requirement. With the exponential growth of edge devices, efficient local processing is essential to mitigate economic costs, latency, and privacy concerns associated with the centralized cloud processing. Emerging intelligent sensors equipped with computing assets to run neural network inferences and embedded in the same package, which hosts the sensing elements, present new challenges due to their limited memory resources and computational skills. This benchmark evaluates models trained with Quantization Aware Training (QAT) and compares their performance with Post-Training Quantization (PTQ) across three use cases: Human Activity Recognition (HAR) by means of the SHL dataset, Physical Activity Monitoring (PAM) by means of the PAMAP2 dataset, and superficial electromyography (sEMG) regression with the NINAPRO DB8 dataset. The results demonstrate the effectiveness of QAT over PTQ in most scenarios, highlighting the potential for deploying advanced AI models on highly resource-constrained sensors. The INT8 versions of the models always outperformed their FP32, regarding memory and latency reductions, except for the activations for CNN. The CNN model exhibited reduced memory usage and latency with respect to its Dense counterpart, allowing it to meet the stringent 8KiB data RAM and 32 KiB program RAM limits of the ISPU. The TCN model proved to be too large to fit within the memory constraints of the ISPU, primarily due to its greater capacity in terms of number of parameters, designed for processing more complex signals like EMG. This benchmark aims to guide the development of efficient AI solutions for In-Sensor Machine Learning Computing, fostering innovation in the field of Edge AI benchmarking, such as the one conducted by the MLCommons-Tiny working group.

Retraction Note: Rule-based multi-view human activity recognition system in real time using skeleton data from RGB-D sensor

Retraction Note: Rule-based multi-view human activity recognition system in real time using skeleton data from RGB-D sensor

Retraction Note: An improved Wi-Fi sensing-based human activity recognition using multi-stage deep learning model

Retraction Note: An improved Wi-Fi sensing-based human activity recognition using multi-stage deep learning model

Belief-Rule-Based System with Self-organizing and Multi-temporal Modeling for Sensor-based Human Activity Recognition.

Smart environment is an efficient and cost- effective way to afford intelligent supports for the elderly people. Human activity recognition (HAR) is a crucial aspect of the research field of smart environments, and it has attracted widespread attention lately. The goal of this study is to develop an effective sensor-based HAR model based on the belief-rule-based system (BRBS), which is one of representative rule-based expert systems. Specially, a new belief rule base (BRB) modeling approach is proposed by taking into account the self- organizing rule generation method and the multi-temporal rule representation scheme, in order to address the problem of combination explosion that existed in the traditional BRB modelling procedure and the time correlation found in continuous sensor data in chronological order. The new BRB modeling approach is so called self-organizing and multi-temporal BRB (SOMT-BRB) modeling procedure. A case study is further deducted to validate the effectiveness of the SOMT-BRB modeling procedure. By comparing with some conventional BRBSs and classical activity recognition models, the results show a significant improvement of the BRBS in terms of the number of belief rules, modelling efficiency, and activity recognition accuracy.

Ensem-DeepHAR: Identification of human activity in smart environments using ensemble of deep learning methods and motion sensor data

Recognizing human activity plays a crucial role in many applications such as medical care services in smart healthcare environments. Inertial or motion sensors can measure physiognomies such as acceleration and angular velocity of body movement while performing the activities and we can use them to learn the models capable of activity recognition. Over the past decades, many state-of-the-art activity recognition systems have been developed but there is still room to improve. In this paper, we have proposed a novel approach to identify human activity from motion sensor data by employing an enormous analysis of sensor data. Based on data analysis, we yielded quality data by preprocessing using a preprocessing chain for human activity recognition (PC-HAR) which also includes the Synthetic Minority Over-sampling Technique to balance the data of the dataset. As a recognition model, we proposed an ensemble of three different deep learning algorithms, namely, modified DeepConvLSTM, modified InceptionTime, and modified ResNet which is named ‘Ensem-DeepHAR’. The outcome of the proposed model is carried out by stacking predictions from each of the mentioned models and then a Random Forest as a meta-model uses those predictions to recognize the final activity. We evaluated our method on both person-dependent and person-independent cases and achieved 99.31 %, 99.08 %, and 97.52 % accuracies for the former case and 97.95 %, 98.11 %, and 99.51 % accuracies for the latter case using three common benchmark datasets: WISDM_ar_v1.1, PAMAP2, and UCI-HAR respectively. The various performance metrics and measures of experimental results establish the supremacy of the proposed model over the state-of-the-arts.