Human Action Recognition Method Research Articles

RGB+2D skeleton: local hand-crafted and 3D convolution feature coding for action recognition

Most 3D skeleton feature-based human action recognition methods are sensitive to changes in viewpoints, motion scales, and human scales. In addition, acquiring depth information from a real scene in outdoor environments results in poor precision or high computational costs. To address these drawbacks, in this study, we propose a new RGB video and 2D skeleton-based action recognition method including local joint trajectory volume representation and feature coding. First, a video is transferred to a set of volumes, which are called the local joint trajectory volumes. Then, hand-crafted and convolutional networks are used to calculate the features of each volume-based RGB image sequence. Different from most works that use convolutional networks to learn global video features, in this paper, the problem of using a convolutional network to represent local video regions is discussed. Finally, the feature set of each joint is encoded into the Fisher vector as an action feature. The classifier is trained by a linear SVM. The experimental results show that skeleton joint-based features result in a more compact and effective action representation approach than other approaches.

Moving the Lab into the Mountains: A Pilot Study of Human Activity Recognition in Unstructured Environments.

Goal: To develop and validate a field-based data collection and assessment method for human activity recognition in the mountains with variations in terrain and fatigue using a single accelerometer and a deep learning model. Methods: The protocol generated an unsupervised labelled dataset of various long-term field-based activities including run, walk, stand, lay and obstacle climb. Activity was voluntary so transitions could not be determined a priori. Terrain variations included slope, crossing rivers, obstacles and surfaces including road, gravel, clay, mud, long grass and rough track. Fatigue levels were modulated between rested to physical exhaustion. The dataset was used to train a deep learning convolutional neural network (CNN) capable of being deployed on battery powered devices. The human activity recognition results were compared to a lab-based dataset with 1,098,204 samples and six features, uniform smooth surfaces, non-fatigued supervised participants and activity labelling defined by the protocol. Results: The trail run dataset had 3,829,759 samples with five features. The repetitive activities and single instance activities required hyper parameter tuning to reach an overall accuracy 0.978 with a minimum class precision for the one-off activity (climbing gate) of 0.802. Conclusion: The experimental results showed that the CNN deep learning model performed well with terrain and fatigue variations compared to the lab equivalents (accuracy 97.8% vs. 97.7% for trail vs. lab). Significance: To the authors knowledge this study demonstrated the first successful human activity recognition (HAR) in a mountain environment. A robust and repeatable protocol was developed to generate a validated trail running dataset when there were no observers present and activity types changed on a voluntary basis across variations in terrain surface and both cognitive and physical fatigue levels.

Disparity-augmented trajectories for human activity recognition

Numerous methods for human activity recognition (HAR) have been proposed in the past two decades. Many of these methods are based on sparse representations, which describe the whole video content by a set of local features. Trajectories, as mid-level sparse features, are capable of describing the movements of interest points in two-dimensional (2D) space. However, 2D trajectories might be affected by viewpoint changes, potentially decreasing their accuracies. In this paper, we first propose and compare different 2D trajectory-based algorithms for human activity recognition. Then, we propose a new way of augmenting 2D trajectories with disparity information, without the calculation of the 3D reconstruction. Our obtained HAR results have shown a 2.76% improvement when using disparity-augmented trajectories, compared to using classical 2D trajectory information only. Furthermore, we have also tested our method on the challenging Hollywood 3D dataset, on which we have obtained competitive results, at a much faster speed.

Context-aware incremental learning-based method for personalized human activity recognition

This study introduces an ensemble-based personalized human activity recognition method relying on incremental learning, which is a method for continuous learning, that can not only learn from streaming data but also adapt to different contexts and changes in context. This adaptation is based on a novel weighting approach which gives bigger weight to those base models of the ensemble which are the most suitable to the current context. In this article, contexts are different body positions for inertial sensors. The experiments are performed in two scenarios: (S1) adapting model to a known context, and (S2) adapting model to a previously unknown context. In both scenarios, the models had to also adapt to the data of previously unknown person, as the initial user-independent dataset did not include any data from the studied user. In the experiments, the proposed ensemble-based approach is compared to non-weighted personalization method relying on ensemble-based classifier and to static user-independent model. Both ensemble models are experimented using three different base classifiers (linear discriminant analysis, quadratic discriminant analysis, and classification and regression tree). The results show that the proposed ensemble method performs much better than non-weighted ensemble model for personalization in both scenarios no matter which base classifier is used. Moreover, the proposed method outperforms user-independent models. In scenario 1, the error rate of balanced accuracy using user-independent model was 13.3%, using non-weighted personalization method 13.8%, and using the proposed method 6.4%. The difference is even bigger in scenario 2, where the error rate using user-independent model is 36.6%, using non-weighted personalization method 36.9%, and using the proposed method 14.1%. In addition, F1 scores also show that the proposed method performs much better in both scenarios that the rival methods. Moreover, as a side result, it was noted that the presented method can also be used to recognize body position of the sensor.

Human Activity Recognition Based on Acceleration Data From Smartphones Using HMMs

Smartphones are among the most popular wearable devices to monitor human activities. Several existing methods for Human Activity Recognition (HAR) using data from smartphones are based on conventional pattern recognition techniques, but they generate handcrafted feature vectors. This drawback is overcome by deep learning techniques which unfortunately require lots of computing resources, while generating less interpretable feature vectors. The current paper addresses these limitations through the proposal of a Hidden Markov Model (HMM)-based technique for HAR. More formally, the sequential variations of spatial locations within the raw data vectors are initially captured in Markov chains, which are later used for the initialization and the training of HMMs. Meta-data extracted from these models are then saved as the components of the feature vectors. The meta-data are related to the overall time spent by the model observing every symbol for a long time span, irrespective of the state from which this symbol is observed. Classification experiments involving four classification tasks have been carried out on the recently constructed UniMiB SHAR database which contains 17 classes, including 9 types of activities of daily living and 8 types of falls. As a result, the proposed approach has shown best accuracies between 92% and 98.85% for all the classification tasks. This performance is more than 10% better than prior work for 2 out of 4 classification tasks.

Enhanced view-independent representation method for skeleton-based human action recognition

Enhanced view-independent representation method for skeleton-based human action recognition

Enhanced view-independent representation method for skeleton-based human action recognition

Human action recognition is an important branch of computer vision science. It is a challenging task based on skeletal data because joints have complex spatiotemporal information. In this work, we propose a method for action recognition, which consists of three parts: view-independent representation, combination with cumulative Euclidean distance, and combined model. First, the action sequence becomes view-independent representations independent of the view. Second, these representations are further combined with cumulative Euclidean distances, so the joints more closely associated with the action are emphasised. Then, a combined model is adopted to extract these representation features and classify actions. It consists of two parts, a regular three-layer BLSTM network, and a temporal attention module. Experimental results on two multi-view benchmark datasets, Northwestern-UCLA and NTU RGB + D, demonstrate the effectiveness of our complete method. Despite its simple architecture and the use of only one type of action feature, it can still significantly improve recognition performance and has strong robustness.

A Novel Sensor-Based Human Activity Recognition Method Based on Hybrid Feature Selection and Combinational Optimization

In recent years, sensor-based human activity recognition (HAR) has become a hot topic due to the advancement of sensing technologies, wireless communication technologies and nano-technologies. Since the sensor signals are usually non-stationary and quite noisy, both selecting the discriminant feature representations and finding out the optimal parameters for recognition algorithm play an important role for the enhanced performance and robustness of an HAR system. However, most of the previous research focused on one of them ignoring their interactions. Very few studies focused on these two aspects simultaneously. Considering the two factors separately may lead to inferior HAR performance. This paper presents a novel HAR framework which can optimize the feature set and the parameters of recognition algorithm synchronously for robust and optimal system performance. A new hybrid feature selection methodology using game-theory based feature selection (GTFS) and binary firefly algorithm (BFA), called GTFS-BFA, is proposed. GTFS-BFA is a hybrid methodology combining evidence from both filter and wrapper feature selection methods. It consists of two phases, namely pre-selection phase and re-selection phase. Pre-selection phase relies on game-theory-based filter method, while the re-selection phase uses binary firefly algorithm (BFA) as a wrapper method. The popular and efficient algorithm kernel extreme learning machine (KELM) is utilized as a classifier. The experimental results indicate that the proposed method can obtain better comprehensive performance in terms of four performance measures through a comparison to other existing methods on daily activity dataset from five body positions.

Human Activity Recognition and Embedded Application Based on Convolutional Neural Network

With the improvement of people's living standards, the demand for health monitoring and exercise detection is increasing. It is of great significance to study human activity recognition methods that are different from traditional feature extraction methods. This article uses convolutional neural network algorithms in deep learning to automatically extract features of activities related to human life. It uses a stochastic gradient descent algorithm to optimize the parameters of the convolutional neural network. The trained network model is compressed on STM32CubeMX-AI. Finally, this article introduces the use of neural networks on embedded devices to recognize six human activities of daily life, such as sitting, standing, walking, jogging, upstairs and downstairs. The acceleration sensor related to human activity information is used to obtain the relevant characteristics of the activity, thereby solving the human activity recognition (HAR) problem. The network structure of the constructed CNN model is shown in Figure 1, including an input layer, two convolutional layers and two pooling layers. After comparing the average accuracy of each set of experiments and the test set of the best model obtained from it, the best model is then selected.

A GAN‐based data augmentation method for human activity recognition via the caching ability

Sensor‐based human activity recognition (HAR) provides vital information for an ocean of health care and entertainment applications. However, data acquisition of such studies usually requires huge time and manpower, which delays the progress of the whole project. Data augmentation is a promising way to solve this issue. In this paper, we expand the origin dataset via a GAN‐based augmentation neural network. The experimental results suggest that the generated data has a certain substitution and complementary effect in terms of the real data based on the caching deployment ability. In addition, the quasi‐real data also gains good performance on identifying certain activities.

Richly Activated Graph Convolutional Network for Robust Skeleton-Based Action Recognition

Current methods for skeleton-based human action recognition usually work with complete skeletons. However, in real scenarios, it is inevitable to capture incomplete or noisy skeletons, which could significantly deteriorate the performance of current methods when some informative joints are occluded or disturbed. To improve the robustness of action recognition models, a multi-stream graph convolutional network (GCN) is proposed to explore sufficient discriminative features spreading over all skeleton joints, so that the distributed redundant representation reduces the sensitivity of the action models to non-standard skeletons. Concretely, the backbone GCN is extended by a series of ordered streams which is responsible for learning discriminative features from the joints less activated by preceding streams. Here, the activation degrees of skeleton joints of each GCN stream are measured by the class activation maps (CAM), and only the information from the unactivated joints will be passed to the next stream, by which rich features over all active joints are obtained. Thus, the proposed method is termed richly activated GCN (RA-GCN). Compared to the state-of-the-art (SOTA) methods, the RA-GCN achieves comparable performance on the standard NTU RGB+D 60 and 120 datasets. More crucially, on the synthetic occlusion and jittering datasets, the performance deterioration due to the occluded and disturbed joints can be significantly alleviated by utilizing the proposed RA-GCN.

A fast human action recognition network based on spatio-temporal features

Artificial intelligence models are widely used in the field of human activity recognition, and human action recognition is an important aspect of human activity recognition. The core of human action recognition is to understand the temporal relationship between video frames. Almost all state-of-the-art methods of human action recognition in videos use optical flow. However, traditional local optical flow estimation methods areexpensive and not trained end-to-end. In this paper, we propose a fast network for human action recognition. Our purpose is to improve the efficiency of optical flow feature extraction and explore the fusion method of spatio-temporal features. For spatio-temporal features, our method combines spatial features and temporal features into fusion features. In addition, we propose CNN with OFF instead of the VGG16 network, which is used to process optical flow features to obtain abundant features. Our model only needs RGB inputs to get the state-of-the-art accuracy of 91.5% on UCF-101, 67.9% on HMDB51, 83.3% on MSR Daily Activity3D, and 91.25% on Florence 3D action, respectively. Compared with most state-of-the-art video action recognition models, our proposed model can effectively improve the accuracy of human action recognition.

Homogeneous Data Normalization and Deep Learning: A Case Study in Human Activity Classification

One class of applications for human activity recognition methods is found in mobile devices for monitoring older adults and people with special needs. Recently, many studies were performed to create intelligent methods for the recognition of human activities. However, the different mobile devices in the market acquire the data from sensors at different frequencies. This paper focuses on implementing four data normalization techniques, i.e., MaxAbsScaler, MinMaxScaler, RobustScaler, and Z-Score. Subsequently, we evaluate the impact of the normalization algorithms with deep neural networks (DNN) for the classification of the human activities. The impact of the data normalization was counterintuitive, resulting in a degradation of performance. Namely, when using the accelerometer data, the accuracy dropped from about 79% to only 53% for the best normalization approach. Similarly, for the gyroscope data, the accuracy without normalization was about 81.5%, whereas with the best normalization, it was only 60%. It can be concluded that data normalization techniques are not helpful in classification problems with homogeneous data.

Human action recognition using distance transform and entropy based features

Human action recognition based on silhouette images has wide applications in computer vision, human computer interaction and intelligent surveillance. It is a challenging task due to the complex actions in nature. In this paper, a human action recognition method is proposed which is based on the distance transform and entropy features of human silhouettes. In the first stage, background subtraction is performed by applying correlation coefficient based frame difference technique to extract silhouette images. In the second stage, distance transform based features and entropy features are extracted from the silhouette images. The distance transform based features and entropy features provide the shape and local variation information. These features are given as input to neural networks to recognize various human actions. The proposed method is tested on three different datasets viz., Weizmann, KTH and UCF50. The proposed method obtains an accuracy of 92.5%, 91.4% and 80% for Weizmann, KTH and UCF50 datasets respectively. The experimental results show that the proposed method for human action recognition is comparable to other state-of-the-art human action recognition methods.

Skeleton edge motion networks for human action recognition

Human skeleton is receiving increasing attention from the community of human action recognition due to its robustness to complex image backgrounds. Previous methods usually utilize body joint-based representation, i.e., joint locations, while leaving edge-based movement poorly investigated. In this paper, we propose a new human action recognition method, skeleton edge motion networks (SEMN), to further explore the motion information of human body parts. Specifically, we address the movement of skeleton edge by using the angle changes of skeleton edge and the movement of the corresponding body joints. We then devise the proposed skeleton edge motion networks by stacking multiple spatial-temporal blocks to learn a robust deep representation from skeleton sequences. Furthermore, we propose a new progressive ranking loss to help the proposed skeleton edge motion networks maintain temporal order information in a self-supervised manner. Experimental results on five popular human action recognition datasets, PennAction, UTD-MHAD, NTU RGB+D, NTU RGB+D 120, and CSL, demonstrate the effectiveness of the proposed method.

Human action recognition method based on conformal geometric algebra and recurrent neural network

Deep Learning (DL) plays an important role in machine learning and artificial intelligence. DL is widely applied in many fields with high dimensional data, including natural language processing, image recognition. High dimensional data can lead to problems in machine learning such as overfitting, degradation of accuracy. To address these issues, some methods, such as Principal Components Analysis (PCA), principal component regression (PCR), Multi-class Linear Discriminant Analysis (MLDA), were proposed to reduce dimensions of the data and computational complexity simultaneously. The drawback of these methods is that they only work well on data distributed on the plane. In the case of the data distributed on the hyper-sphere, such as objects moving in space, the processing results are not so good as expected. In this paper, we propose the use of Conformal Geometric Algebra (CGA) to extract features and simultaneously reduce the dimensionality of a dataset for human activity recognition using Recurrent Neural Network (RNN). First, human activity data in a 3-dimensional coordinate system is pre-processed and normalized by calculating deviations from the mean coordinate. Next, the data is transformed to vectors in CGA space and its dimensions are reduced to return the feature vectors. Finally, we use the RNN model to train feature vectors. Empirical results performed on the CMU eight actions dataset show that the CGA combined with RNN gave the best test results of 92.5%.

NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding.

Research on depth-based human activity analysis achieved outstanding performance and demonstrated the effectiveness of 3D representation for action recognition. The existing depth-based and RGB+D-based action recognition benchmarks have a number of limitations, including the lack of large-scale training samples, realistic number of distinct class categories, diversity in camera views, varied environmental conditions, and variety of human subjects. In this work, we introduce a large-scale dataset for RGB+D human action recognition, which is collected from 106 distinct subjects and contains more than 114 thousand video samples and 8 million frames. This dataset contains 120 different action classes including daily, mutual, and health-related activities. We evaluate the performance of a series of existing 3D activity analysis methods on this dataset, and show the advantage of applying deep learning methods for 3D-based human action recognition. Furthermore, we investigate a novel one-shot 3D activity recognition problem on our dataset, and a simple yet effective Action-Part Semantic Relevance-aware (APSR) framework is proposed for this task, which yields promising results for recognition of the novel action classes. We believe the introduction of this large-scale dataset will enable the community to apply, adapt, and develop various data-hungry learning techniques for depth-based and RGB+D-based human activity understanding.

View transform graph attention recurrent networks for skeleton-based action recognition

Human action recognition based on skeleton recently has attracted attention of researchers due to the accessibility and popularity of the 3D skeleton data. However, it is complicated to effectively represent spatial–temporal skeleton sequences given the large variations of action representations when they are captured from different viewpoints. In order to get a better representation of the spatial–temporal skeletal features, this paper introduces a view transform graph attention recurrent networks (VT+GARN) method for view-invariant human action recognition. We design a view-invariant transform strategy based on the sequence to reduce the influence of different views on the spatial–temporal position of skeleton joint. Then, the graph attention recurrent network automatically calculates the coefficient of attention and learns the representation of spatiotemporal skeletal features after the transformation and outputs the classification result. Ablation studies and extensive experiments on three challenging datasets, Northwestern-UCLA, NTU RGB+D and UWA3DII, demonstrate the effectiveness and superiority of our method

A data augmentation method for human action recognition using dense joint motion images

With the development of deep learning and neural network techniques, human action recognition has made great progress in recent years. However, it remains challenging to analyse temporal information and identify human actions with few training samples. In this paper, an effective motion image called a dense joint motion image (DJMI) was proposed to transform an action to an image. Our method was compared with state-of-the-art methods, and its contributions are mainly reflected in three characteristics. First, in contrast to the current classic joint trajectory map (JTM), every pixel of the DJMI is useful and contains essential spatio-temporal information. Thus, the input parameters of the deep neural network (DNN) are reduced by an order of magnitude, and the efficiency of action recognition is improved. Second, each frame of an action video is encoded as an independent slice of the DJMI, which avoids the information loss caused by action trajectory overlap. Third, by using DJMIs, proven algorithms for graphics and images can be used to generate training samples. Compared with the original image, the generated DJMIs contain new and different spatio-temporal information, which enables DNNs to be trained well on very few samples. Our method was evaluated on three benchmark datasets, namely, Florence-3D, UTKinect-Action3D and MSR Action3D. The results showed that our method achieved a recognition speed of 37 fps with competitive accuracy on these datasets. The time efficiency and few-shot learning capability of our method enable it to be used in real-time surveillance.

Human Action Recognition Method Based on Dynamic Bayesian Network

Human Action Recognition Method Based on Dynamic Bayesian Network