Human Action Recognition Method Research Articles

Position-Aware Indoor Human Activity Recognition Using Multisensors Embedded in Smartphones

Composite indoor human activity recognition is very important in elderly health monitoring and is more difficult than identifying individual human movements. This article proposes a sensor-based human indoor activity recognition method that integrates indoor positioning. Convolutional neural networks are used to extract spatial information contained in geomagnetic sensors and ambient light sensors, while transform encoders are used to extract temporal motion features collected by gyroscopes and accelerometers. We established an indoor activity recognition model with a multimodal feature fusion structure. In order to explore the possibility of using only smartphones to complete the above tasks, we collected and established a multisensor indoor activity dataset. Extensive experiments verified the effectiveness of the proposed method. Compared with algorithms that do not consider the location information, our method has a 13.65% improvement in recognition accuracy.

Lightweight human activity recognition method based on the MobileHARC model

In recent years, Human activity recognition (HAR) based on wearable devices has been widely applied in health applications and other fields. Currently, most HAR models are based on the Convolutional Neural Network (CNN), Long Short-Term Memory (LSTM), or their combination. Recently, there have been proposals based on Transformer and its variant models. However, due to the fact that these models have sequential network structures and are unable to simultaneously focus on local and global features, thus, resulting in a reduction in recognition performance. In addition, along with the substantial computational resources required by Transformers, they are not suitable for resource-constrained devices. In this paper, the primary distinction of our proposed model from other hybrid models that combine CNN and Transformer is that our model adopts a completely new parallel network architecture and primarily focuses on lightweight design. Particularly, We proposed the Mobile Human Activity Recognition Conformer (MobileHARC), which adopts the parallel structure with a lightweight Transformer and CNN as the backbone networks. Furthermore, we proposed the Inverted Residual Lightweight Convolution Block and Multiscale Lightweight Multi-Head Self-Attention Mechanism. We systematically evaluate the proposed models on four public datasets. Experimental results show that MobileHARC achieves superior recognition performance, and uses fewer Floating-Point Operations per Second (FLOPs) and parameters compared to current models.

Research on Human Action Recognition Method Based on Machine Learning

Human action recognition (HAR) is an important research direction in the field of computer vision, and human action recognition technology provides an important foundation for computers to understand and simulate human behavior. It has been widely used in many practical applications, including human-computer interaction, gesture recognition, motion analysis, motion capture, virtual reality and augmented reality. Early methods were mainly based on the characteristics of manual design and traditional machine learning methods, such as support vector machine (SVM) and Random Forest. These methods usually rely on manually extracted features, such as edge, texture and color information, but they do not perform well in complex scenes and changing environments. With the rise of deep learning, especially the successful application of Convolutional Neural Network (CNN), great breakthroughs have been made in human action recognition. Using CNN, we can learn the feature representation directly from the original image data, which avoids the trouble of manually designing features. This paper expounds the research progress and future development trend of human action recognition methods based on machine learning.

Deep learning method for human activity recognition using heaped LSTM and image pattern of activity

Deep learning method for human activity recognition using heaped LSTM and image pattern of activity

A profile similarity-based personalized federated learning method for wearable sensor-based human activity recognition

Wearable sensor-based human activity recognition (HAR) utilizes artificial intelligence models to analyze real-time data like accelerometer data to recognize daily human activities. While it greatly benefits the life of senior citizens and postoperative patients, it conventionally requires the collected data to be uploaded to a central server to train AI models, raising critical security and privacy concerns. Though Federated learning (FL) emerges as a viable way to cope with these problems, it is confronted by the data heterogeneity problem, where the varying activity patterns of different individuals result in non-identically distributed local data. Some FL models have been proposed to solve the data heterogeneity problem by leveraging the similarity between individuals to create a personalized global model for each individual. However, they are still limited by increased computation or unreliable relationships in the similarity computation. This study proposes a novel profile similarity-based personalized federated learning for wearable sensor-based HAR where the similarity between individuals can be reflected in their profile, such as age, gender, height, and weight. When personalizing a model for an individual, we compute the weighted sum of all clients’ local models, where the weight is determined by the similarity value computed from the profile. In this way, the local models from individuals who have higher similarity values will contribute more towards personalizing a model for a targeted individual than those who are less similar. Experiment results demonstrate that the proposed model outperformed the baseline FL and centralized learning on both RealWorld and SisFall datasets. We also discuss the tradeoff between privacy and personalization and FL's advantages over centralized learning.

CSI-GLSTN: A Location-Independent CSI Human Activity Recognition Method Based on Spatio-Temporal and Channel Feature Fusion

CSI-GLSTN: A Location-Independent CSI Human Activity Recognition Method Based on Spatio-Temporal and Channel Feature Fusion

Deep Wavelet Convolutional Neural Networks for Multimodal Human Activity Recognition Using Wearable Inertial Sensors.

Recent advances in wearable systems have made inertial sensors, such as accelerometers and gyroscopes, compact, lightweight, multimodal, low-cost, and highly accurate. Wearable inertial sensor-based multimodal human activity recognition (HAR) methods utilize the rich sensing data from embedded multimodal sensors to infer human activities. However, existing HAR approaches either rely on domain knowledge or fail to address the time-frequency dependencies of multimodal sensor signals. In this paper, we propose a novel method called deep wavelet convolutional neural networks (DWCNN) designed to learn features from the time-frequency domain and improve accuracy for multimodal HAR. DWCNN introduces a framework that combines continuous wavelet transforms (CWT) with enhanced deep convolutional neural networks (DCNN) to capture the dependencies of sensing signals in the time-frequency domain, thereby enhancing the feature representation ability for multiple wearable inertial sensor-based HAR tasks. Within the CWT, we further propose an algorithm to estimate the wavelet scale parameter. This helps enhance the performance of CWT when computing the time-frequency representation of the input signals. The output of the CWT then serves as input for the proposed DCNN, which consists of residual blocks for extracting features from different modalities and attention blocks for fusing these features of multimodal signals. We conducted extensive experiments on five benchmark HAR datasets: WISDM, UCI-HAR, Heterogeneous, PAMAP2, and UniMiB SHAR. The experimental results demonstrate the superior performance of the proposed model over existing competitors.

A Novel Lightweight Human Activity Recognition Method Via L-CTCN.

Wi-Fi-based human activity recognition has attracted significant attention. Deep learning methods are widely used to achieve feature representation and activity sensing. While more learnable parameters in the neural networks model lead to richer feature extraction, it results in significant resource consumption, rendering the model unsuitable for lightweight Internet of Things (IoT) devices. Furthermore, the sensing performance heavily relies on the quality and quantity of data, which is a time-consuming and labor-intensive task. Therefore, there is a need to explore methods that reduce the dependence on the quality and quantity of the dataset while ensuring recognition performance and decreasing model complexity to adapt to ubiquitous lightweight IoT devices. In this paper, we propose a novel Lightweight-Complex Temporal Convolution Network (L-CTCN) for human activity recognition. Specifically, this approach effectively combines complex convolution with a Temporal Convolution Network (TCN). Complex convolution can extract richer information from limited raw complex data, reducing the reliance on the quality and quantity of training samples. Based on the designed TCN framework with 1D convolution and residual blocks, the proposed model can achieve lightweight human activity recognition. Extensive experiments verify the effectiveness of the proposed method. We can achieve an average recognition accuracy of 96.6% with only 0.17 M parameter size. This method performs well under conditions of low sampling rates and a low number of subcarriers and samples.

Review of Literature on Human Activity Detection and Recognition

Purpose: The objective of this research article is to methodically combine the existing literature on Human Activity Recognition (HAR) and provide an understanding of the present state of the HAR literature. Additionally, the article aims to suggest an appropriate HAR system that can be used for detecting real-time activities such as suspicious behavior, surveillance, and healthcare. Objective: This review study intends to delve into the current state of human activity detection and recognition methods, while also pointing towards promising avenues for further research and development in the field, particularly with regards to complex and multi-task human activity recognition across different domains. Design/Methodology/Approach: A systematic literature review methodology was adopted by collecting and analyzing the required literature available from international and national journals, conferences, databases and other resources searched through the Google Scholar and other search engines. Findings/Result: The systematic review of literature uncovered the various approaches of Human activity detection and recognition. Even though the prevailing literature reports the investigations of several aspects of Human activity detection and recognition, there is still room for exploring the role of this technology in various domains to enhance its robustness in detecting and recognizing of multiple human actions from preloaded CCTV cameras, which can aid in detecting abnormal and suspicious activities and ultimately reduce aberrant human actions in society. Originality/Value: This paper follows a systematic approach to examine the factors that impact the detection and recognition of Human activity and suggests a concept map. The study undertaken supplements the expanding literature on knowledge sharing highlighting its significance. Paper Type: Review Paper.

Towards efficient video-based action recognition: context-aware memory attention network

Abstract Given the prevalence of surveillance cameras in our daily lives, human action recognition from videos holds significant practical applications. A persistent challenge in this field is to develop more efficient models capable of real-time recognition with high accuracy for widespread implementation. In this research paper, we introduce a novel human action recognition model named Context-Aware Memory Attention Network (CAMA-Net), which eliminates the need for optical flow extraction and 3D convolution which are computationally intensive. By removing these components, CAMA-Net achieves superior efficiency compared to many existing approaches in terms of computation efficiency. A pivotal component of CAMA-Net is the Context-Aware Memory Attention Module, an attention module that computes the relevance score between key-value pairs obtained from the 2D ResNet backbone. This process establishes correspondences between video frames. To validate our method, we conduct experiments on four well-known action recognition datasets: ActivityNet, Diving48, HMDB51 and UCF101. The experimental results convincingly demonstrate the effectiveness of our proposed model, surpassing the performance of existing 2D-CNN based baseline models. Article Highlights Recent human action recognition models are not yet ready for practical applications due to high computation needs. We propose a 2D CNN-based human action recognition method to reduce the computation load. The proposed method achieves competitive performance compared to most SOTA 2D CNN-based methods on public datasets.

MAWKDN: A Multimodal Fusion Wavelet Knowledge Distillation Approach Based on Cross-View Attention for Action Recognition

The recognition prerformance of existing vision-based human action recognition (HAR) methods is greatly reduced in the case of low camera resolution or occlusion. Wearable sensors can provide complementary information to alleviate this problem. It is challenging to construct a robust HAR model using multimodal wearable-sensor data. In this paper, we propose a cross-Attention-based Multimodal fusion Wavelet Knowledge Distillation Network (MAWKDN) method to guide recognition from video data by acquiring complementary information from wearable sensors and reduce the noise effects through wavelet knowledge distillation, which improves the robustness of the model. A multi-attention dilated convolution kernel residual network including dilated convolution and an attention mechanism is constructed to extract features from various sensor modalities and fuse the various modal data through the cross-view attention method to acquire additional information from different modalities. To reduce the modal differences between different modalities of the teacher and student networks and acquire similar semantic knowledge, we learn the information between different modalities by constructing a graph structure of convolutional layer features, and computing the semantic preservation loss between the teacher and student networks. To reduce the influence of noise in the input data, we construct the loss of wavelet knowledge distillation, which transforms the image through the discrete wavelet transform and only retains the low frequency features to extract the useful information. The top-1 accuracy achieved on the UTD-MHAD (99.31%), Berkeley-MHAD (99.40%) and the F1-score on the MMAct (85.26% based on cross-session) dataset prove the superior performance of MAWKDN compared with the state-of-the-art HAR methods. Moreover, we demonstrate the robustness of the MAWKDN approach on the noise-added UTD-MHAD dataset.

Occlusion-Aware Graph Neural Networks for Skeleton Action Recognition

Due to its important applications in industry, the research of action recognition has attracted many attention. In particular, skeleton-based human action recognition methods have evolved to become more and more competitive. Nevertheless, occlusion is still a very challenging task in human action recognition as yet. Off-the-shelf works are usually based on complete skeleton data, few people consider action recognition in occlusion. For the sake of improving the recognition accuracy on occluded skeleton data, we put forward an occlusion-aware multi-stream fusion graph convolutional network (dubbed MSFGCN). Multiple streams are comprised in MSFGCN, and different occlusion cases can be disposed by different streams. Besides, in order to construct more discriminative features, multi-modal features were extracted simultaneously, such as joint coordinates, relative coordinates, small-scale temporal differences and large-scale temporal differences. In particular, it is the first time to take advantage of motion features at large and small scales at the same time, which helps to distinguish actions with different motion amplitudes. What's more, considering the different importance of different parts of the human body for different action recognition, the content adaptation operation is used to further optimize the recognition performance. Effective training strategies are also presented to further improve the performance of the model. Experimental results show that the proposed MSFGCN has a great advantage over other methods on occluded skeleton datasets. To showcase the effectiveness of different modules, extensive ablation experiments are performed on various skeleton action recognition datasets.

MMTSA

Multimodal sensors provide complementary information to develop accurate machine-learning methods for human activity recognition (HAR), but introduce significantly higher computational load, which reduces efficiency. This paper proposes an efficient multimodal neural architecture for HAR using an RGB camera and inertial measurement units (IMUs) called Multimodal Temporal Segment Attention Network (MMTSA). MMTSA first transforms IMU sensor data into a temporal and structure-preserving gray-scale image using the Gramian Angular Field (GAF), representing the inherent properties of human activities. MMTSA then applies a multimodal sparse sampling method to reduce data redundancy. Lastly, MMTSA adopts an inter-segment attention module for efficient multimodal fusion. Using three well-established public datasets, we evaluated MMTSA's effectiveness and efficiency in HAR. Results show that our method achieves superior performance improvements (11.13% of cross-subject F1-score on the MMAct dataset) than the previous state-of-the-art (SOTA) methods. The ablation study and analysis suggest that MMTSA's effectiveness in fusing multimodal data for accurate HAR. The efficiency evaluation on an edge device showed that MMTSA achieved significantly better accuracy, lower computational load, and lower inference latency than SOTA methods.

Human Activity Recognition Based on Continuous-Wave Radar and Bidirectional Gate Recurrent Unit

The technology for human activity recognition has diverse applications within the Internet of Things spectrum, including medical sensing, security measures, smart home systems, and more. Predominantly, human activity recognition methods have relied on contact sensors, and some research uses inertial sensors embedded in smartphones or other devices, which present several limitations. Additionally, most research has concentrated on recognizing discrete activities, even though activities in real-life scenarios tend to be continuous. In this paper, we introduce a method to classify continuous human activities, such as walking, running, squatting, standing, and jumping. Our approach hinges on the micro-Doppler (MD) features derived from continuous-wave radar signals. We first process the radar echo signals generated from human activities to produce MD spectrograms. Subsequently, a bidirectional gate recurrent unit (Bi-GRU) network is employed to train and test these extracted features. Preliminary results highlight the efficacy of our approach, with an average recognition accuracy exceeding 90%.

A multi-graph convolutional network based wearable human activity recognition method using multi-sensors

A multi-graph convolutional network based wearable human activity recognition method using multi-sensors

Human activity recognition method using joint deep learning and acceleration signal

Many studies have been conducted on human activity recognition (HAR) in the last decade. Accordingly, deep learning (DL) algorithms have been given more attention in terms of classification of human daily activities. Deep neural networks (DNNs) compute and extract complex features on voluminous data through some hidden layers that require large memory and powerful graphics processing units (GPUs). So, this study proposes a new joint learning (JL) approach to classify human activities using inertial sensors. To this end, a large complex donor model based on a convolutional neural network (CNN) is used to transfer knowledge to a smaller model based on CNN referred to as the acceptor model. The acceptor model can be deployed on mobile devices and low-power hardware due to decreased computing costs and memory consumption. The wireless sensor data mining (WISDM) dataset is used to test the proposed model. According to the experimental results, the HAR system based on the JL algorithm outperforms than other methods.

SKELTER: unsupervised skeleton action denoising and recognition using transformers

Unsupervised Human Action Recognition (U-HAR) methods currently leverage large-scale datasets of human poses to solve this challenging problem. As most of the approaches are dedicated to reaching the best recognition accuracies, no attention has been put into analyzing the resilience of such methods given perturbed data, a likely occurrence in real in-the-wild testing scenarios. Our first contribution is to systematically validate the decrease in performance of current U-HAR state-of-the-art using perturbed or altered data (e.g., obtained by removing some skeletal joints, rotating the entire pose, and injecting geometrical aberrations). Then, we propose a novel framework based on a transformer encoder–decoder with remarkable de-noising capabilities to counter such perturbations effectively. Moreover, we also present additional losses to have robust representations against rotation variances and provide temporal motion consistency. Our model, SKELTER, shows limited drops in performance when skeleton noise is present compared with previous approaches, favoring its use in challenging in-the-wild settings.

Wearable sensor-based human activity recognition with ensemble learning: a comparison study

<p>The spectacular growth of wearable sensors has provided a key contribution to the field of human activity recognition. Due to its effective and versatile usage and application in various fields such as smart homes and medical areas, human activity recognition has always been an appealing research topic in artificial intelligence. From this perspective, there are a lot of existing works that make use of accelerometer and gyroscope sensor data for recognizing human activities. This paper presents a comparative study of ensemble learning methods for human activity recognition. The methods include random forest, adaptive boosting, gradient boosting, extreme gradient boosting, and light gradient boosting machine (LightGBM). Among the ensemble learning methods in comparison, light gradient boosting machine and random forest demonstrate the best performance. The experimental results revealed that light gradient boosting machine yields the highest accuracy of 94.50% on UCI-HAR dataset and 100% on single accelerometer dataset while random forest records the highest accuracy of 93.41% on motion sense dataset.</p>

A Human Activity Recognition Method Based on Lightweight Feature Extraction Combined With Pruned and Quantized CNN for Wearable Device

Human Activity Recognition (HAR) is becoming an essential part of human life care. Existing HAR methods are usually developed using a two-level approach, wherein a first-level Machine Learning (ML) classifier is employed to distinguish the static and dynamic activities, followed by a second-level classifier to identify the specific activity. These approaches are not suitable for wearable devices, due to the high computational and memory consumption. Our rigorous analysis of various HAR datasets opens up a new possibility that static or dynamic activities can be discriminated against through a simple statistical technique. Therefore, we propose to utilize a statistical feature extraction technique to replace the first-level ML classifier, thus achieving more lightweight computation. Next, we employ Random Forest (RF) and Convolutional Neural Networks (CNN) to classify the specific activities, achieving higher accuracy compared to the state-of-the-art results. We further reduce the computation and memory consumption of the above combined approach by applying pruning and quantizing techniques to CNN (PQ-CNN). Experimental results show the proposed lightweight HAR method achieved an F1 score of 0.9417 and 0.9438 for unbalanced and balanced datasets, respectively. On top of lightweight and accuracy, the proposed HAR method is practical for wearable devices by using a single accelerometer.

Human Action Representation Learning Using an Attention-Driven Residual 3DCNN Network

The recognition of human activities using vision-based techniques has become a crucial research field in video analytics. Over the last decade, there have been numerous advancements in deep learning algorithms aimed at accurately detecting complex human actions in video streams. While these algorithms have demonstrated impressive performance in activity recognition, they often exhibit a bias towards either model performance or computational efficiency. This biased trade-off between robustness and efficiency poses challenges when addressing complex human activity recognition problems. To address this issue, this paper presents a computationally efficient yet robust approach, exploiting saliency-aware spatial and temporal features for human action recognition in videos. To achieve effective representation of human actions, we propose an efficient approach called the dual-attentional Residual 3D Convolutional Neural Network (DA-R3DCNN). Our proposed method utilizes a unified channel-spatial attention mechanism, allowing it to efficiently extract significant human-centric features from video frames. By combining dual channel-spatial attention layers with residual 3D convolution layers, the network becomes more discerning in capturing spatial receptive fields containing objects within the feature maps. To assess the effectiveness and robustness of our proposed method, we have conducted extensive experiments on four well-established benchmark datasets for human action recognition. The quantitative results obtained validate the efficiency of our method, showcasing significant improvements in accuracy of up to 11% as compared to state-of-the-art human action recognition methods. Additionally, our evaluation of inference time reveals that the proposed method achieves up to a 74× improvement in frames per second (FPS) compared to existing approaches, thus showing the suitability and effectiveness of the proposed DA-R3DCNN for real-time human activity recognition.