Networks For Human Activity Recognition Research Articles

Attention-Based Hybrid Deep Learning Network for Human Activity Recognition Using WiFi Channel State Information

The recognition of human movements is a crucial aspect of AI-related research fields. Although methods using vision and sensors provide more valuable data, they come at the expense of inconvenience to users and social limitations including privacy issues. WiFi-based sensing methods are increasingly being used to collect data on human activity due to their ubiquity, versatility, and high performance. Channel state information (CSI), a characteristic of WiFi signals, can be employed to identify various human activities. Traditional machine learning approaches depend on manually designed features, so recent studies propose leveraging deep learning capabilities to automatically extract features from raw CSI data. This research introduces a versatile framework for recognizing human activities by utilizing CSI data and evaluates its effectiveness on different deep learning networks. A hybrid deep learning network called CNN-GRU-AttNet is proposed to automatically extract informative spatial-temporal features from raw CSI data and efficiently classify activities. The effectiveness of a hybrid model is assessed by comparing it with five conventional deep learning models (CNN, LSTM, BiLSTM, GRU, and BiGRU) on two widely recognized benchmark datasets (CSI-HAR and StanWiFi). The experimental results demonstrate that the CNN-GRU-AttNet model surpasses previous state-of-the-art techniques, leading to an average accuracy improvement of up to 4.62%. Therefore, the proposed hybrid model is suitable for identifying human actions using CSI data.

Temporal-channel convolution with self-attention network for human activity recognition using wearable sensors

Human activity recognition (HAR) is an essential task in many applications such as health monitoring, rehabilitation, and sports training. Sensor-based HAR has received increasing attention due to the widespread availability of sensors in daily life. In this paper, we propose two novel architectures, the convolution with self-attention network (CSNet) and the temporal-channel convolution with self-attention network (TCCSNet), for classifying sequences of human activity data from different sensors. CSNet leverages both convolution and self-attention to capture both local and global dependencies in the input data, while TCCSNet exploits both temporal and inter-channel dependencies through two branches of convolutions and self-attentions for extracting time-wise and channel-wise information. The proposed methods are evaluated on seven different sensor-based HAR datasets, namely: MHEALTH, PAMAP2, UTD1, UTD2, WHARF, USC-HAD, and WISDM, using the leave-one-subject-out cross-validation protocol. Our experiments show that the proposed models outperform other modern approaches, such as Transformers and long short-term memory (LSTM) based models.

A Prior Knowledge-Guided Graph Convolutional Neural Network for Human Action Recognition in Solar Panel Installation Process

Human action recognition algorithms have garnered significant research interest due to their vast potential for applications. Existing human behavior recognition algorithms primarily focus on recognizing general behaviors using a large number of datasets. However, in industrial applications, there are typically constraints such as limited sample sizes and high accuracy requirements, necessitating algorithmic improvements. This article proposes a graph convolution neural network model that combines prior knowledge supervision and attention mechanisms, designed to fulfill the specific action recognition requirements for workers installing solar panels. The model extracts prior knowledge from training data, improving the training effectiveness of action recognition models and enhancing the recognition reliability of special actions. The experimental results demonstrate that the method proposed in this paper surpasses traditional models in terms of recognizing solar panel installation actions accurately. The proposed method satisfies the need for highly accurate recognition of designated person behavior in industrial applications, showing promising application prospects.

Fine-Tuned Temporal Dense Sampling with 1D Convolutional Neural Network for Human Action Recognition.

Human action recognition is a constantly evolving field that is driven by numerous applications. In recent years, significant progress has been made in this area due to the development of advanced representation learning techniques. Despite this progress, human action recognition still poses significant challenges, particularly due to the unpredictable variations in the visual appearance of an image sequence. To address these challenges, we propose the fine-tuned temporal dense sampling with 1D convolutional neural network (FTDS-1DConvNet). Our method involves the use of temporal segmentation and temporal dense sampling, which help to capture the most important features of a human action video. First, the human action video is partitioned into segments through temporal segmentation. Each segment is then processed through a fine-tuned Inception-ResNet-V2 model, where max pooling is performed along the temporal axis to encode the most significant features as a fixed-length representation. This representation is then fed into a 1DConvNet for further representation learning and classification. The experiments on UCF101 and HMDB51 demonstrate that the proposed FTDS-1DConvNet outperforms the state-of-the-art methods, with a classification accuracy of 88.43% on the UCF101 dataset and 56.23% on the HMDB51 dataset.

Design of cuckoo search optimization with deep belief network for human activity recognition and classification

Design of cuckoo search optimization with deep belief network for human activity recognition and classification

Micro-network-based deep convolutional neural network for human activity recognition from realistic and multi-view visual data

Micro-network-based deep convolutional neural network for human activity recognition from realistic and multi-view visual data

Multi-level channel attention excitation network for human action recognition in videos

Channel attention mechanism has continuously attracted strong interests and shown great potential in enhancing the performance of deep CNNs. However, when applied to video-based human action recognition task, most existing methods generally learn channel attention at frame level, which ignores the temporal dependencies and may limit the recognition performance. In this paper, we propose a novel multi-level channel attention excitation (MCAE) module to model the temporal-related channel attention at both frame and video levels. Specifically, based on video convolutional feature maps, frame-level channel attention (FCA) is generated by exploring time-channel correlations, and video-level channel attention (VCA) is generated by aggregating global motion variations. MCAE firstly recalibrates video feature responses with frame-wise FCA, and then activates the motion-sensitive channel features with motion-aware VCA. MCAE module learns the channel discriminability from multiple levels and can act as a guidance to facilitate efficient spatiotemporal feature modeling in activated motion-sensitive channels. It can be flexibly embedded into 2D networks with very limited extra computation cost to construct MCAE-Net, which effectively enhances the spatiotemporal representation of 2D models for video action recognition task Extensive experiments on five human action datasets show that our method achieves superior or very competitive performance compared with the state-of-the-arts, which demonstrates the effectiveness of the proposed method for improving the performance of human action recognition.

Two-Level Attention Module Based on Spurious-3D Residual Networks for Human Action Recognition.

In recent years, deep learning techniques have excelled in video action recognition. However, currently commonly used video action recognition models minimize the importance of different video frames and spatial regions within some specific frames when performing action recognition, which makes it difficult for the models to adequately extract spatiotemporal features from the video data. In this paper, an action recognition method based on improved residual convolutional neural networks (CNNs) for video frames and spatial attention modules is proposed to address this problem. The network can guide what and where to emphasize or suppress with essentially little computational cost using the video frame attention module and the spatial attention module. It also employs a two-level attention module to emphasize feature information along the temporal and spatial dimensions, respectively, highlighting the more important frames in the overall video sequence and the more important spatial regions in some specific frames. Specifically, we create the video frame and spatial attention map by successively adding the video frame attention module and the spatial attention module to aggregate the spatial and temporal dimensions of the intermediate feature maps of the CNNs to obtain different feature descriptors, thus directing the network to focus more on important video frames and more contributing spatial regions. The experimental results further show that the network performs well on the UCF-101 and HMDB-51 datasets.

WMNN: Wearables-Based Multi-Column Neural Network for Human Activity Recognition.

In recent years, human activity recognition (HAR) technologies in e-health have triggered broad interest. In literature, mainstream works focus on the body's spatial information (i.e. postures) which lacks the interpretation of key bioinformatics associated with movements, limiting the use in applications requiring comprehensively evaluating motion tasks' correctness. To address the issue, in this article, a Wearables-based Multi-column Neural Network (WMNN) for HAR based on multi-sensor fusion and deep learning is presented. Here, the Tai Chi Eight Methods were utilized as an example as in which both postures and muscle activity strengths are significant. The research work was validated by recruiting 14 subjects in total, and we experimentally show 96.9% and 92.5% accuracy for training and testing, for a total of 144 postures and corresponding muscle activities. The method is then provided with a human-machine interface (HMI), which returns users with motion suggestions (i.e. postures and muscle strength). The report demonstrates that the proposed HAR technique can enhance users' self-training efficiency, potentially promoting the development of the HAR area.

STGL-GCN: Spatial–Temporal Mixing of Global and Local Self-Attention Graph Convolutional Networks for Human Action Recognition

Human action recognition methods based on skeleton data have been widely studied owing to their strong robustness to illumination and complex backgrounds. Existing methods have achieved good recognition results; however, they have certain challenges, such as the fixed topological structure of the graph, the omission of nonphysical joint correlation, and the inability to extract local spatial–temporal features. Herein, we propose spatial–temporal mixing of global and local self-attention graph convolutional networks (STGL-GCN) using skeleton data. The global self-attention matrix captures the potential dependencies of nonphysical correlations between joints, and the local self-attention matrix determines the connection strength of the physical edges of joints. The matrices are updated together with the convolution parameters in each network layer as the model is trained for optimal graph structure to achieve accurate action expressions Experiments on the NTU-RGBD dataset demonstrate that our model accurately recognizes actions.

Attention-Based Residual BiLSTM Networks for Human Activity Recognition

Attention-Based Residual BiLSTM Networks for Human Activity Recognition

A PSL-based Approach to Human Activity Recognition in Smart Home Environments

Human activity recognition is widely used in smart cities, public safety and other fields, especially in smart home systems where it has a pivotal role. The study addresses the shortcomings of Markov logic networks for human activity recognition and proposes a human activity recognition method in smart home scenarios - an activity recognition framework based on Probabilistic Soft Logic (PSL). The framework is able to deal with logical uncertainty problems and provides expression and inference mechanisms for data uncertainty problems on this basis. The framework utilizes Deng entropy evidence theory to provide an evaluation method for sensor event uncertainty, and combines event calculus for activity modeling. Comparing the PSL method with three other common recognition methods, Ontology, Hidden Markov Model (HMM), and Markov logic network, on a public dataset, it was found that the PSL method has a much better ability to handle data uncertainty than the other three algorithms. The average recognition rates on the ADL and ADL-E sub datasets were 82.87% and 80.33%, respectively. In experiments to verify the ability of PSL to handle temporal complexity, PSL showed the least significant decrease in the average recognition rate and maintained an average recognition rate of 81.02% in the presence of concurrent and alternating activities. The human activity recognition method based on PSL has a better performance in handling both data uncertainty and temporal complexity.

ViT-ReT: Vision and Recurrent Transformer Neural Networks for Human Activity Recognition in Videos

ViT-ReT: Vision and Recurrent Transformer Neural Networks for Human Activity Recognition in Videos

A fuzzy convolutional attention-based GRU network for human activity recognition

Human activity recognition has become a pillar of today intelligent Human–Computer Interfaces as it typically provides more comfortable and ubiquitous interaction. This paper proposes a novel fuzzy-based deep learning-based algorithm to predict future sequences of activities from a given sequence of daily living activities of a subject wearing a lower limb exoskeleton. The engineering application concerns the challenging task of recognizing locomotion activities of the wearer in real-time, which is needed to ensure appropriate control of the robot during daily living activities. Indeed, real-time locomotion activity recognition is very challenging for controlling lower-limb exoskeletons. The model proposes a new adaptive kernel, based on the data features derived from the fuzzy rules on the input sequences to enrich the features of the activity sequences. Then, a CNN is applied to extract local subsequences from the whole sequences to identify local patterns in the convolution window. Finally, an attention-based GRU is incorporated into the model to extract meaningful parts of the time-series sequences. The results show high accuracy in the estimation of the transition between gait modes which is critical to ensure smooth control of the exoskeleton. The performance of the model is evaluated using the dynamic activity data gathered from different subjects. The proposed model outperforms the traditional models used in the literature.

Speeding up deep neural architecture search for wearable activity recognition with early prediction of converged performance

Neural architecture search (NAS) has the potential to uncover more performant networks for human activity recognition from wearable sensor data. However, a naive evaluation of the search space is computationally expensive. We introduce neural regression methods for predicting the converged performance of a deep neural network (DNN) using validation performance in early epochs and topological and computational statistics. Our approach shows a significant improvement in predicting converged testing performance over a naive approach taking the ranking of the DNNs at an early epoch as an indication of their ranking on convergence. We apply this to the optimization of the convolutional feature extractor of an LSTM recurrent network using NAS with deep Q-learning, optimizing the kernel size, number of kernels, number of layers, and the connections between layers, allowing for arbitrary skip connections and dimensionality reduction with pooling layers. We find architectures which achieve up to 4% better F1 score on the recognition of gestures in the Opportunity dataset than our implementation of DeepConvLSTM and 0.8% better F1 score than our implementation of state-of-the-art model Attend and Discriminate, while reducing the search time by more than 90% over a random search. This opens the way to rapidly search for well-performing dataset-specific architectures. We describe the computational implementation of the system (software frameworks, computing resources) to enable replication of this work. Finally, we lay out several future research directions for NAS which the community may pursue to address ongoing challenges in human activity recognition, such as optimizing architectures to minimize power, minimize sensor usage, or minimize training data needs.

Artificial neural networks for human activity recognition using sensor based dataset

Artificial neural networks for human activity recognition using sensor based dataset

Activity Graph Based Convolutional Neural Network for Human Activity Recognition Using Acceleration and Gyroscope Data

Human activity recognition (HAR) using smartphone sensors have been recently studied in various applications including healthcare, fitness, and smart home. Their recognition accuracy often depends on high-quality feature design and effectiveness of classification algorithms, where existing work mostly replies on laborious hand-crafted design and shallow feature learning architecture. Recent deep learning techniques demonstrate outstanding effectiveness in performing automatic feature learning and outperform traditional models in terms of accuracy. But their performance is limited by the quality and volumes of available labelled data. It is challenging to achieve accurate multisubject HAR with only smartphone sensing data. This article proposes a novel optimal activity graph generation model incorporating a deep learning framework for automatic and accurate HAR with multiple subjects using only acceleration and gyroscope data. The activity graph generation model presents a multisensory integration mechanism with three-steps sorting algorithms for producing optimal activity graphs containing alignments of neighbored signals in their width and height. Then, we propose a deep convolutional neural network to automatically learn distinguishable features from the graphs for HAR. By leveraging superior presentation of correlations between human activities and neighbored signals alignments via optimal activity graphs, the learned features are endowed with more discriminative power. The experimental evaluation was carried out on several benchmark datasets (i.e., UCI, USCHAD, and UTD-MHAD). The results showed that our approach improved the average recognition accuracy by about 5% when compared with other state-of-the-art HAR methods. Particularly towards multisubject HAR cases (UTD-MHAD dataset with 21 subjects), it achieved up to 10% accuracy gain over other methods. These improvements show the advantage and potential of our method dealing with complex HAR problems with multiple subjects using limited sensing data.

VirtualActionNet: A strong two-stream point cloud sequence network for human action recognition

In this paper, we propose a strong two-stream point cloud sequence network VirtualActionNet for 3D human action recognition. In the data preprocessing stage, we transform the depth sequence into a point cloud sequence as the input of our VirtualActionNet. In order to encode intra-frame appearance structures, static point cloud technologies are first employed as a virtual action generation sequence module to abstract the point cloud sequence into a virtual action sequence. Then, a two-stream network framework is presented to model the virtual action sequence. Specifically, we design an appearance stream module for aggregating all the appearance information preserved in each virtual action frame. Moreover, a motion stream module is introduced to capture dynamic changes along the time dimension. Finally, a joint loss strategy is adopted during data training to improve the action prediction accuracy of the two-stream network. Extensive experiments on three publicly available datasets demonstrate the effectiveness of the proposed VirtualActionNet.

Gesture recognition from RGB images using convolutional neural network‐attention based system

SummaryHand gestures can be categorized based on their applications as conversational, manipulative, controlling, and communicative gestures. Hand gesture recognition is an aspect of human action recognition, which plays a notable role in communication among the deaf and dumb community. Accurate recognition and classification of hand gestures with similar postures is still a complex problem. The main objective of this work is to employ deep learning based convolutional neural networks for human action recognition. This article introduces a convolutional neural network based on VGG16 architecture with an attention approach to mitigate the issue. Incorporating an attention‐based module with the VGG16 architecture is the prime reason for potentially learning distinguishing image features by the network. Besides, the proposed robust system collectively recognizes and classifies all the available 36 hand gesture classes of the Massey database, where most previous work selected only a few distinguishable gestures. Experimental results show that the average recognition accuracy for characters with similar hand postures like “m” and “n” is better than the state‐of‐the‐art networks. Additionally, the proposed method attains about 3% higher recognition accuracy than the state‐of‐the‐art networks for all gesture classes. The efficacy of the proposed architecture has also been validated via precision, recall, and F‐score.

Quantum behaved Intelligent Variant of Gravitational Search Algorithm with Deep Neural Networks for Human Activity Recognition

Human activity recognition (HAR) encompasses the detection of daily routine activities to advance usability in detecting crime and preventing dangerous activities. The recognition of activities from videos and image sequences with higher exactitude is a major challenge due to system complexities. The efficient feature optimization approach can reduce system complexities by removing ineffective features, which also improves the activity recognition performance. This research work presents a novel quantum behaved intelligent gravitational search algorithm to optimize the features for human activity recognition. The proposed intelligent variant is termed as INQGSA, which optimizes the features by using the advantageous attributes of quantum computing (QC) and intelligent gravitational search algorithm (INGSA). In INQGSA, the intelligent factor avoids the trapping of mass agents in later iterations by using the information of the best and worst agents to update the position of agents. The addition of quantum computing based attributes (such as quantum bits, their superposition, and quantum gates, etc.) ensures a better diversity of discrete optimized features. To analyze the superiority of INQGSA, the feature optimization is also conducted with the gravitational search algorithm (GSA) and the quantum-inspired binary gravitational search algorithm (QBGSA). Finally, the optimized selected features are utilized by the deep neural networks (DNN) of ResNet-50V2 and ResNet-101V2 for the classification of activities. The activity recognition experiments are conducted on the UCF101 and HMDB51 datasets. The performance comparison of the proposed HAR system with state-of-the-art techniques signifies that the proposed system is superior and effective in detecting the different activities.