Problem Of Human Action Recognition Research Articles

STO-CVAE: state transition-oriented conditional variational autoencoder for data augmentation in disability classification

AbstractThe class imbalance problem occurs when there is an unequal distribution of classes in a dataset and is a significant issue in various artificial intelligence applications. This study focuses on the severe multiclass imbalance problem of human activity recognition in rehabilitation exercises for people with disabilities. To overcome this problem, we present a novel human action-centric augmentation method for human skeleton-based pose estimation. This study proposes the state transition-oriented conditional variational autoencoder (STO-CVAE) to capture action patterns in repeated exercises. The proposed approach generates action samples by capturing temporal information of human skeletons to improve the identification of minority disability classes. We conducted experimental studies with a real-world dataset gathered from rehabilitation exercises and confirmed the superiority and effectiveness of the proposed method. Specifically, all investigated classifiers (i.e., random forest, support vector machine, extreme gradient boosting, light gradient boosting machine, and TabNet) trained with the proposed augmentation method outperformed the models trained without augmentation in terms of the F1-score and accuracy, with F1-score showing the most improvement. Overall, the prediction accuracy of most classes was improved; in particular, the prediction accuracy of the minority classes was greatly improved. Hence, the proposed STO-CVAE can be used to improve the accuracy of disability classification in the field of physical medicine and rehabilitation and to provide suitable personal training and rehabilitation exercise programs.

Graph-Structure-Based Multigranular Belief Fusion for Human Activity Recognition.

The belief functions (BFs) introduced by Shafer in the mid of 1970s are widely applied in information fusion to model epistemic uncertainty and to reason about uncertainty. Their success in applications is however limited because of their high-computational complexity in the fusion process, especially when the number of focal elements is large. To reduce the complexity of reasoning with BFs, we can envisage as a first method to reduce the number of focal elements involved in the fusion process to convert the original basic belief assignments (BBAs) into simpler ones, or as a second method to use a simple rule of combination with potentially a loss of the specificity and pertinence of the fusion result, or to apply both methods jointly. In this article, we focus on the first method and propose a new BBA granulation method inspired by the community clustering of nodes in graph networks. This article studies a novel efficient multigranular belief fusion (MGBF) method. Specifically, focal elements are regarded as nodes in the graph structure, and the distance between nodes will be used to discover the local community relationship of focal elements. Afterward, the nodes belonging to the decision-making community are specially selected, and then the derived multigranular sources of evidence can be efficiently combined. To evaluate the effectiveness of the proposed graph-based MGBF, we further apply this new approach to combine the outputs of convolutional neural networks + attention (CNN + Attention) in the human activity recognition (HAR) problem. The experimental results obtained with real datasets prove the potential interest and feasibility of our proposed strategy with respect to classical BF fusion methods.

Higher Order Polynomial Transformer for Fine-Grained Freezing of Gait Detection.

Freezing of Gait (FoG) is a common symptom of Parkinson's disease (PD), manifesting as a brief, episodic absence, or marked reduction in walking, despite a patient's intention to move. Clinical assessment of FoG events from manual observations by experts is both time-consuming and highly subjective. Therefore, machine learning-based FoG identification methods would be desirable. In this article, we address this task as a fine-grained human action recognition problem based on vision inputs. A novel deep learning architecture, namely, higher order polynomial transformer (HP-Transformer), is proposed to incorporate pose and appearance feature sequences to formulate fine-grained FoG patterns. In particular, a higher order self-attention mechanism is proposed based on higher order polynomials. To this end, linear, bilinear, and trilinear transformers are formulated in pursuit of discriminative fine-grained representations. These representations are treated as multiple streams and further fused by a cross-order fusion strategy for FoG detection. Comprehensive experiments on a large in-house dataset collected during clinical assessments demonstrate the effectiveness of the proposed method, and an area under the receiver operating characteristic (ROC) curve (AUC) of 0.92 is achieved for detecting FoG.

Human Behavior Recognition Based on CNN-LSTM Hybrid and Multi-Sensing Feature Information Fusion

To address the human activity recognition problem and its application in practical situations, a CNN-LSTM hybrid neural network model capable of automatically extracting sensor data features and memorizing temporal activity data is designed and improved by integrating CNN and gated recurrent units as a variant of RNN. A multi-channel spatiotemporal fusion network-based two-person interaction behavior recognition method is proposed for two-person skeletal sequential behavior recognition. Firstly, a viewpoint invariant feature extraction method is used to extract two-player skeleton features, then a two-layer cascaded spatiotemporal fusion network model is designed, and finally, a multi-channel spatiotemporal fusion network is used to learn multiple sets of two-player skeleton features separately to obtain multi-channel fusion features, and the fusion features are used to recognize the interaction behavior, and the weights are shared among the channels. Applying the algorithm in the paper to the UCF101 dataset for experiments, the accuracy of the two-person cross-object experiment can reach 96.42% and the accuracy of the cross-view experiment can reach 97.46%. The method in the paper shows better performance in two-player interaction behavior recognition compared to typical methods in this field.

Human pose estimation and action recognition for fitness movements

Human pose estimation and action recognition have important applications in the areas of security, medicine and sports. In this work, we propose an improved YOLOv7-Pose algorithm to solve the problem of human posture and action recognition for fitness movements in various scenarios. For the algorithm based on YOLOv7-Pose, we add the function of classification to the original network. And we introduce Coordinate Attention to the network to improve the model’s ability to identify important features of human skeletal joints and action classifications. The improved ConvNeXt network structure is introduced to replace the CBS convolution kernel of the original model to improve the accuracy of human key point detection and action classification of the model. We optimise the spatial pyramid pool structure, which can reduce the loss function and accelerate the convergence rate of the model. We adopt EIOU as the regression function of the target detection frame to improve the accuracy of the coordinate regression. Experiments show that the improved YOLOv7-Pose has an mAP of 95.9% on a homemade test set of fitness actions, which is 5.4% higher than HRNet, with a 4.2% improvement over the original YOLOv7 algorithm, suggesting that the accuracy of action recognition and key point estimation has significantly improved.

Hybrid Learning Models for IMU-Based HAR with Feature Analysis and Data Correction.

This paper proposes a novel approach to tackle the human activity recognition (HAR) problem. Four classes of body movement datasets, namely stand-up, sit-down, run, and walk, are applied to perform HAR. Instead of using vision-based solutions, we address the HAR challenge by implementing a real-time HAR system architecture with a wearable inertial measurement unit (IMU) sensor, which aims to achieve networked sensing and data sampling of human activity, data pre-processing and feature analysis, data generation and correction, and activity classification using hybrid learning models. Referring to the experimental results, the proposed system selects the pre-trained eXtreme Gradient Boosting (XGBoost) model and the Convolutional Variational Autoencoder (CVAE) model as the classifier and generator, respectively, with 96.03% classification accuracy.

Human Action Representation Learning Using an Attention-Driven Residual 3DCNN Network

The recognition of human activities using vision-based techniques has become a crucial research field in video analytics. Over the last decade, there have been numerous advancements in deep learning algorithms aimed at accurately detecting complex human actions in video streams. While these algorithms have demonstrated impressive performance in activity recognition, they often exhibit a bias towards either model performance or computational efficiency. This biased trade-off between robustness and efficiency poses challenges when addressing complex human activity recognition problems. To address this issue, this paper presents a computationally efficient yet robust approach, exploiting saliency-aware spatial and temporal features for human action recognition in videos. To achieve effective representation of human actions, we propose an efficient approach called the dual-attentional Residual 3D Convolutional Neural Network (DA-R3DCNN). Our proposed method utilizes a unified channel-spatial attention mechanism, allowing it to efficiently extract significant human-centric features from video frames. By combining dual channel-spatial attention layers with residual 3D convolution layers, the network becomes more discerning in capturing spatial receptive fields containing objects within the feature maps. To assess the effectiveness and robustness of our proposed method, we have conducted extensive experiments on four well-established benchmark datasets for human action recognition. The quantitative results obtained validate the efficiency of our method, showcasing significant improvements in accuracy of up to 11% as compared to state-of-the-art human action recognition methods. Additionally, our evaluation of inference time reveals that the proposed method achieves up to a 74× improvement in frames per second (FPS) compared to existing approaches, thus showing the suitability and effectiveness of the proposed DA-R3DCNN for real-time human activity recognition.

The basic assembly of skeletal models in the fall detection problem

The paper considers the appliance of the featureless approach to the human activity recognition problem, which exclude the direct anthropomorphic and visual characteristics of human figure from further analysis and thus increase the privacy of the monitoring system. A generalized pairwise comparison function of two human skeletal models, invariant to the sensor type, is used to project the object of interest to the secondary feature space, formed by the basic assembly of skeletons. A sequence of such projections in time forms an activity map, which allows an application of deep learning methods based on convolution neural networks for activity recognition. The proper ordering of skeletal models in a basic assembly plays an important role in secondary space design. The study of ordering of the basic assembly by the shortest unclosed path algorithm and correspondent activity maps for video streams from the TST Fall Detection v2 database are presented.

Wrapper-based deep feature optimization for activity recognition in the wearable sensor networks of healthcare systems

The Human Activity Recognition (HAR) problem leverages pattern recognition to classify physical human activities as they are captured by several sensor modalities. Remote monitoring of an individual’s activities has gained importance due to the reduction in travel and physical activities during the pandemic. Research on HAR enables one person to either remotely monitor or recognize another person’s activity via the ubiquitous mobile device or by using sensor-based Internet of Things (IoT). Our proposed work focuses on the accurate classification of daily human activities from both accelerometer and gyroscope sensor data after converting into spectrogram images. The feature extraction process follows by leveraging the pre-trained weights of two popular and efficient transfer learning convolutional neural network models. Finally, a wrapper-based feature selection method has been employed for selecting the optimal feature subset that both reduces the training time and improves the final classification performance. The proposed HAR model has been tested on the three benchmark datasets namely, HARTH, KU-HAR and HuGaDB and has achieved 88.89%, 97.97% and 93.82% respectively on these datasets. It is to be noted that the proposed HAR model achieves an improvement of about 21%, 20% and 6% in the overall classification accuracies while utilizing only 52%, 45% and 60% of the original feature set for HuGaDB, KU-HAR and HARTH datasets respectively. This proves the effectiveness of our proposed wrapper-based feature selection HAR methodology.

A Multimodal Fusion Approach for Human Activity Recognition

The problem of human activity recognition (HAR) has been increasingly attracting the efforts of the research community, having several applications. It consists of recognizing human motion and/or behavior within a given image or a video sequence, using as input raw sensor measurements. In this paper, a multimodal approach addressing the task of video-based HAR is proposed. It is based on 3D visual data that are collected using an RGB + depth camera, resulting to both raw video and 3D skeletal sequences. These data are transformed into six different 2D image representations; four of them are in the spectral domain, another is a pseudo-colored image. The aforementioned representations are based on skeletal data. The last representation is a "dynamic" image which is actually an artificially created image that summarizes RGB data of the whole video sequence, in a visually comprehensible way. In order to classify a given activity video, first, all the aforementioned 2D images are extracted and then six trained convolutional neural networks are used so as to extract visual features. The latter are fused so as to form a single feature vector and are fed into a support vector machine for classification into human activities. For evaluation purposes, a challenging motion activity recognition dataset is used, while single-view, cross-view and cross-subject experiments are performed. Moreover, the proposed approach is compared to three other state-of-the-art methods, demonstrating superior performance in most experiments.

Matched Filter Interpretation of CNN Classifiers with Application to HAR.

Time series classification is an active research topic due to its wide range of applications and the proliferation of sensory data. Convolutional neural networks (CNNs) are ubiquitous in modern machine learning (ML) models. In this work, we present a matched filter (MF) interpretation of CNN classifiers accompanied by an experimental proof of concept using a carefully developed synthetic dataset. We exploit this interpretation to develop an MF CNN model for time series classification comprising a stack of a Conv1D layer followed by a GlobalMaxPooling layer acting as a typical MF for automated feature extraction and a fully connected layer with softmax activation for computing class probabilities. The presented interpretation enables developing superlight highly accurate classifier models that meet the tight requirements of edge inference. Edge inference is emerging research that addresses the latency, availability, privacy, and connectivity concerns of the commonly deployed cloud inference. The MF-based CNN model has been applied to the sensor-based human activity recognition (HAR) problem due to its significant importance in a broad range of applications. The UCI-HAR, WISDM-AR, and MotionSense datasets are used for model training and testing. The proposed classifier is tested and benchmarked on an android smartphone with average accuracy and scores of 98% and 97%, respectively, which outperforms state-of-the-art HAR methods in terms of classification accuracy and run-time performance. The proposed model size is less than 150 KB, and the average inference time is less than 1 ms. The presented interpretation helps develop a better understanding of CNN operation and decision mechanisms. The proposed model is distinguished from related work by jointly featuring interpretability, high accuracy, and low computational cost, enabling its ready deployment on a wide set of mobile devices for a broad range of applications.

Improving the Performance of Closet-Set Classification in Human Activity Recognition by Applying a Residual Neural Network Architecture

The Time Series Classification Residual Network (TSCResNet) deep learning model is introduced in this study to improve the classification performance in the human activity recognition (HAR) problem. Specifically in the context of closed-set classification where all labels or classes are present during the model training phase. This contrasts with open-set classification where new, unseen activities are introduced to the HAR system after the training phase. The proposed TSCResNet model is evaluated with the benchmark PAMAP2 Physical Activity Monitoring Dataset. By using the same quantitative methods and data preprocessing protocols as previous research in the field of closed-set HAR, results show that the TSCResNet model architecture is able to achieve improved classification results across accuracy, the weighted F1-score, and mean F1-score.

A Semi-Supervised Transfer Learning with Dynamic Associate Domain Adaptation for Human Activity Recognition Using WiFi Signals.

Human activity recognition without equipment plays a vital role in smart home applications, freeing humans from the shackles of wearable devices. In this paper, by using the channel state information (CSI) of the WiFi signal, semi-supervised transfer learning with dynamic associate domain adaptation is proposed for human activity recognition. In order to improve the CSI quality and denoising of CSI, we carried out missing packet filling, burst noise removal, background estimation, feature extraction, feature enhancement, and data augmentation in the data pre-processing stage. This paper considers the problem of environment-independent human activity recognition, also known as domain adaptation. The pre-trained model is trained from the source domain by collecting a complete labeled dataset of all of the CSI of human activity patterns. Then, the pre-trained model is transferred to the target environment through the semi-supervised transfer learning stage. Therefore, when humans move to different target domains, a partial labeled dataset of the target domain is required for fine-tuning. In this paper, we propose a dynamic associate domain adaptation called DADA. By modifying the existing associate domain adaptation algorithm, the target domain can provide a dynamic ratio of labeled dataset/unlabeled dataset, while the existing associate domain adaptation algorithm only allows target domains with the unlabeled dataset. The advantage of DADA is that it provides a dynamic strategy to eliminate different effects on different environments. In addition, we further designed an attention-based DenseNet model, or AD, as our training network, which is modified by an existing DenseNet by adding the attention function. The solution we proposed was simplified to DADA-AD throughout the paper. The experimental results show that for domain adaptation in different domains, the accuracy of human activity recognition of the DADA-AD scheme is 97.4%. It also shows that DADA-AD has advantages over existing semi-supervised learning schemes.

A Hierarchical Approach to Activity Recognition and Fall Detection Using Wavelets and Adaptive Pooling.

Human activity recognition has been a key study topic in the development of cyber physical systems and assisted living applications. In particular, inertial sensor based systems have become increasingly popular because they do not restrict users’ movement and are also relatively simple to implement compared to other approaches. In this paper, we present a hierarchical classification framework based on wavelets and adaptive pooling for activity recognition and fall detection predicting fall direction and severity. To accomplish this, windowed segments were extracted from each recording of inertial measurements from the SisFall dataset. A combination of wavelet based feature extraction and adaptive pooling was used before a classification framework was applied to determine the output class. Furthermore, tests were performed to determine the best observation window size and the sensor modality to use. Based on the experiments the best window size was found to be 3 s and the best sensor modality was found to be a combination of accelerometer and gyroscope measurements. These were used to perform activity recognition and fall detection with a resulting weighted F1 score of 94.67%. This framework is novel in terms of the approach to the human activity recognition and fall detection problem as it provides a scheme that is computationally less intensive while providing promising results and therefore can contribute to edge deployment of such systems.

Double constrained bag of words for human action recognition

Various improved methods based on the strategy of bag of words (BoW) are widely used to solve the problem of human action recognition. However, the spatial relationship between features is measured and utilized by these methods in a relatively single way. It limits the recognition performance of these methods. To solve this problem, double constrained bag of words (DC-BoW) is proposed to utilize the spatial distribution information between features belonging to three levels, which include descriptor-level, presentation-level and hidden layer features. Aiming at the problem that most coding methods only rely on Euclidean distance to constrain the relationship between descriptor-level features, the constraints of the difference in length and cosine of angle between visual word and local feature are designed to construct the loss function to obtain the length and angle constrained linear coding (LACLC) method. In order to improve the recognizability of the representation-level features, the spatial distribution between the encoded features around each cluster center is considered. Hierarchical weighting and LACLC are jointly applied to the distribution to construct aggregated word group feature (AWGF). At the same time, the constraint form of correntropy is changed according to the principle of constructing constraints in LACLC. The hidden layer features are combined with new constraint forms to construct double constrained extreme learning machine (DC-ELM), which improves the classification performance of the network while avoiding iterative training of correntropy weight. In order to verify the feasibility of DC-BoW, experiments are conducted on KTH, Olympic Sports, UCF11, Hollywood2 and UCF101 datasets. Experimental results show that the proposed DC-BoW can further utilize the spatial distribution information between features to obtain excellent recognition accuracy compared with other improved methods based on BoW.

RECOGNITION OF HUMAN ACTIONS IN VIDEO SEQUENCES USING THE LSTM NETWORK

The development of computer vision and the wide applicability of its applied components determine the relevance of research in this field of science. One of the most interesting tasks of computer vision is to monitor the behavior of people, which includes the analysis of their actions and carried out for various purposes. Examples of use are systems for monitoring compliance with safety regulations and the wearing of personal protective equipment by workers in factories, systems such as “smart home”, which track actions, systems for monitoring the condition of people in medical or social institutions, home systems for monitoring the condition of the elderly, which are able to notify relatives in cases of emergency situations. There is no comprehensive program that can solve the described problem and its variations, regardless of the scope of application. Therefore, the development of its prototype, which is a module that solves the human action recognition problem in the video, is an important problem. The article describes the creation of a software module that solves the human action recognition problem in a video. An overview of existing data sets suitable for training a neural network is provided, and data collection and processing for a custom dataset for actions of four different classes is described. The key features of the stages of creating, training and testing a neural network with the LSTM (Long Short-Term Memory) architecture, as well as options for its practical application, are described below. The developed module is quite flexible, there is a possibility to increase the number of classes of recognized actions depending on the scope of its application, as well as the possibility of integration with other modules for monitoring the behavior of people who have a similar device.

A deep survey on supervised learning based human detection and activity classification methods

Human detection and activity recognition is very important research area in the healthcare, video surveillance, pedestrian detection, intelligent vehicle system and home care center. Among the various human activity detection frameworks, the statistical based approach were most intensively studied and used in practice in which pattern recognition was traditionally formulated. More recently, supervised learning based techniques and methods imported from statistical learning theory have deserved increasing attention. Many new supervised learning methods such as transfer learning, multi-instance learning, and the new trends in deep learning techniques have used for the formulation of solutions to the human activity detection. This paper reviews the automatic human detection and their activity recognition in the video sequences and static images. We explain several problems of human detection and activity recognition in different steps such as processing, segmentation of human features extraction and classification. Moreover, discuss the problems in each step and provide the recent state- of-the-art methods, gaps between recent methods, technical difficulties, applications and their challenges. Several features extraction techniques and corresponding problems for human classification have been discussed in details. Special emphasis have been given on convolution neural network that solves the problem of human segmentation, efficient classification and activity recognition. The objective of this review paper is to summarize and review related of the established and recent methods used in various stages of a human detection and activity classification system and identify research topics and applications that are at the forefront of this exciting and challenging field. Further, the evaluation protocols (i.e. datasets and simulation tools) and possible solution of current limitation have been discussed briefly in this survey.

Exploiting spatio-temporal representation for 3D human action recognition from depth map sequences

Human action recognition based on 3D data is attracting increasing attention because it could provide more abundant spatial and temporal information compared with RGB videos. The challenge of the depth map based method is to capture the cues between spatial appearances and temporal motions. In this paper, we propose a straightforward and efficient framework for modeling the human action based on depth map sequences, considering the short-term and long-term dependencies. A frame-level feature, termed depth-oriented gradient vector (DOGV), is developed to capture the appearance and motion in a short-term duration. For a long-term dependence, we construct convolutional neural networks (CNNs) based backbone to aggregate frame-level features in the space and time sequence. The proposed method is comprehensively evaluated on four public benchmark datasets, including NTU RGB+D, NTU RGB+D 120, PKU-MMD and UOW LSC. The experimental results demonstrate that the proposed approach can solve the problem of 3D human action recognition in an efficient way and achieve the state-of-the-art performance.

Unsupervised Human Activity Representation Learning with Multi-task Deep Clustering

Human activity recognition (HAR) based on sensing data from wearable and mobile devices has become an active research area in ubiquitous computing, and it envisions a wide range of application scenarios in mobile social networking, environmental context sensing, health and well-being monitoring, etc. However, activity recognition based on manually annotated sensing data is manpower-expensive, time-consuming, and privacy-sensitive, which prevents HAR systems from being really deployed in scale. In this paper, we address the problem of unsupervised human activity recognition, which infers activities from unlabeled datasets without the need of domain knowledge. We propose an end-to-end multi-task deep clustering framework to solve the problem. Taking the unlabeled multi-dimensional sensing signals as input, we firstly apply a CNN-BiLSTM autoencoder to form a compressed latent feature representation. Then we apply a K-means clustering algorithm based on the extracted features to partition the dataset into different groups, which produces pseudo labels for the instances. We further train a deep neural network (DNN) with the latent features and pseudo labels for activity recognition. The tasks of feature representation, clustering, and classification are integrated into a uniform multi-task learning framework and optimized jointly to achieve unsupervised activity classification. We conduct extensive experiments based on three public datasets. It is shown that the proposed approach outperforms shallow unsupervised learning approaches, and it performs close to the state-of-the-art supervised approaches by fine-tuning with a small number of labeled data. The proposed approach significantly reduces the cost of human-based data annotation and narrows down the gap between unsupervised and supervised human activity recognition.

Attend and Discriminate

Wearables are fundamental to improving our understanding of human activities, especially for an increasing number of healthcare applications from rehabilitation to fine-grained gait analysis. Although our collective know-how to solve Human Activity Recognition (HAR) problems with wearables has progressed immensely with end-to-end deep learning paradigms, several fundamental opportunities remain overlooked. We rigorously explore these new opportunities to learn enriched and highly discriminating activity representations. We propose: i) learning to exploit the latent relationships between multi-channel sensor modalities and specific activities; ii) investigating the effectiveness of data-agnostic augmentation for multi-modal sensor data streams to regularize deep HAR models; and iii) incorporating a classification loss criterion to encourage minimal intra-class representation differences whilst maximising inter-class differences to achieve more discriminative features. Our contributions achieves new state-of-the-art performance on four diverse activity recognition problem benchmarks with large margins---with up to 6% relative margin improvement. We extensively validate the contributions from our design concepts through extensive experiments, including activity misalignment measures, ablation studies and insights shared through both quantitative and qualitative studies. The code base and trained network parameters are open-sourced on GitHub https://github.com/AdelaideAuto-IDLab/Attend-And-Discriminate to support further research.