Human Action Recognition Method Research Articles

Human Action Recognition Using Multilevel Depth Motion Maps

The advent of depth sensors opens up new opportunities for human action recognition by providing depth information. The main purpose of this paper is to present an effective method for human action recognition from depth images. A multilevel frame select sampling (MFSS) method are proposed to generate three levels of temporal samples from the input depth sequences first. Then, the proposed motion and static mapping (MSM) method is used to obtain the representation of MFSS sequences. After that, this paper exploits the block-based LBP feature extraction approach to extract features information from the MSM. Finally, the fisher kernel representation is applied to aggregate the block features, which is then combined with the kernel-based extreme learning machine classifier. The developed framework is evaluated on three public datasets captured by depth cameras. The experimental results demonstrate the great performance compared with the existing approaches.

Action Recognition From Thermal Videos

Human action recognition using a camera-based surveillance system remains a challenging task. In particular, action recognition is difficult when a human is not visible in an image captured in a dark environment. The existing studies have utilized near-infrared (NIR) and thermal cameras to solve this problem. Compared to NIR cameras, thermal cameras enable long- and short-distance objects to be visible without an additional illuminator. However, thermal cameras have two major disadvantages: a halo effect and a temperature similarity. A halo effect occurs around an object with a high temperature. In a human object, such a halo effect is similar to a shadow under the body area. It is more difficult to segment a human area from an image with a halo effect. Moreover, if the background and human object have similar temperatures, it becomes more difficult to segment the human area. These disadvantages influence not only the accuracy of the segmentation of the human area but also the performance of human action recognition. Unfortunately, no studies have considered these issues. To address these problems, this study proposes the cycle-consistent generative adversarial network (CycleGAN)-based methods for removing halo effects from thermal images and restoring the areas of the human bodies. In addition, this study also considered a method for creating a skeleton image from a thermal image to analyze body movements. To extract more spatial and temporal features from skeleton image sequences thus created, a method for human action recognition that combines a convolutional neural network (CNN) and long short-term memory (LSTM) was proposed. In an experiment using an open database (Dongguk activities & actions database (DA&A-DB2)), the proposed method demonstrated a better performance than the existing methods.

Broad learning system for human activity recognition using sensor data

In a multi-sensor environment, it is efficient to record and reflect people's information of activities, using the large amount of data. However, the data cannot directly display the form of activity itself so that it is necessary to do the further job of exploration and processing. Deep Learning (DL) methods have attracted more attention and have shown some superior performance, while they have the problem of structural complexity. Therefore, this paper creatively uses Broad Learning System (BLS) method for human activity recognition. We use sliding window to get the data segmented. The weights involved are fine-tuned by pseudo-inverse and ridge regression algorithms, and we achieve an accurate classification of activities. The method is verified by using OPPORTUNITY data set. The results show that this method can greatly shorten the learning time and improve the accuracy, as well as the performance in comparison with traditional methods.

Broad learning system for human activity recognition using sensor data

In a multi-sensor environment, it is efficient to record and reflect people's information of activities, using the large amount of data. However, the data cannot directly display the form of activity itself so that it is necessary to do the further job of exploration and processing. Deep Learning (DL) methods have attracted more attention and have shown some superior performance, while they have the problem of structural complexity. Therefore, this paper creatively uses Broad Learning System (BLS) method for human activity recognition. We use sliding window to get the data segmented. The weights involved are fine-tuned by pseudo-inverse and ridge regression algorithms, and we achieve an accurate classification of activities. The method is verified by using OPPORTUNITY data set. The results show that this method can greatly shorten the learning time and improve the accuracy, as well as the performance in comparison with traditional methods.

单传感器数据驱动的人体日常短时行为识别方法

单传感器数据驱动的人体日常短时行为识别方法

Human behaviour recognition algorithm based on improved DMM and Fisher coding

Human behaviour recognition has become a key technology in intelligent sensing, location and tracking tasks. In view of the different speed of action execution and DMM loss of time dimension information, this paper proposes a human action recognition method based on improved DMM and Fisher coding. First, in consideration of the different action speeds of long and short video, this paper adopts two different video segmentation strategies. Second, in order to make video-based DMM contain more time dimension information, this paper proposes an improved DMM; then, in order to better express the texture information of the image, this paper improves the extraction of LBP features by DMM. Finally, due to the different feature lengths and high dimensions obtained by the long and short video, this paper adopts the Fisher vector for feature encoding and combines SVM to complete the action recognition. In the public action recognition database MSRAction3D and gesture recognition database MSRGesture3D, the accuracy rate of the algorithm is 96.25% and 96.00%, respectively, and it has higher recognition rate than many existing algorithms.

Segmentation and Recognition of Basic and Transitional Activities for Continuous Physical Human Activity

Human activity recognition (HAR) is a hot research topic which aims to understand human behavior and can be applied in various applications. However, transitions between activities are usually disregarded due to their low incidence and short duration when compared against other activities, while in fact, transitions can affect the performance of the recognition system if not dealt with properly. In this paper, we propose and implement a systematic human activity recognition method to recognize basic activities (BA) and transitional activities (TA) in a continuous sensor data stream. First, raw sensor data are segmented into fragments with sliding window and the features are constructed based on window segmentation. Then, cluster analysis with K-Means is used to aggregate activity fragments into periods. Next, generally, realize the classification of BA and TA according to the shortest duration of the BA, and then deal with the hidden phenomenon of BA. Third, the fragments between adjacent BA are evaluated to decide whether they are TA or disturbance process. Finally, random forest classifier is used to accurately recognize BA and TA. The proposed method is evaluated on the public dataset SBHAR. The results demonstrate that our method effectively recognizes different activities and can deliver high accuracy with all activities considered.

Design of human activity recognition algorithms based on a single wearable IMU sensor

In recent years, with the rapid development of inertial measurement unit (IMU) technology, wireless body area network and pattern recognition theory, human motion recognition based on wearable technology has gradually gained the attention of researchers. In this paper, the human activity recognition method based on wearable sensor motion information fusion is studied. On the existing wearable system platform, the time domain analysis and frequency domain analysis of human motion information are used to distinguish the daily activity of the human body, and based on the human motion data acquisition experiment, time domain features, frequency domain features and attitude angles of the human motion data are used as identification features. On that basis multi-classification activity recognition algorithm based on support vector machine is proposed and human motion pattern recognition is carried out. The experimental results show that the system can accurately identify the daily activity of the human body.

Design of human activity recognition algorithms based on a single wearable IMU sensor

In recent years, with the rapid development of inertial measurement unit (IMU) technology, wireless body area network and pattern recognition theory, human motion recognition based on wearable technology has gradually gained the attention of researchers. In this paper, the human activity recognition method based on wearable sensor motion information fusion is studied. On the existing wearable system platform, the time domain analysis and frequency domain analysis of human motion information are used to distinguish the daily activity of the human body, and based on the human motion data acquisition experiment, time domain features, frequency domain features and attitude angles of the human motion data are used as identification features. On that basis multi-classification activity recognition algorithm based on support vector machine is proposed and human motion pattern recognition is carried out. The experimental results show that the system can accurately identify the daily activity of the human body.

Group Sparse Regression-Based Learning Model for Real-Time Depth-Based Human Action Prediction

This paper addresses the problem of predicting human actions in depth videos. Due to the complex spatiotemporal structure of human actions, it is difficult to infer ongoing human actions before they are fully executed. To handle this challenging issue, we first propose two new depth-based features called pairwise relative joint orientations (PRJOs) and depth patch motion maps (DPMMs) to represent the relative movements between each pair of joints and human-object interactions, respectively. The two proposed depth-based features are suitable for recognizing and predicting human actions in real-time fashion. Then, we propose a regression-based learning approach with a group sparsity inducing regularizer to learn action predictor based on the combination of PRJOs and DPMMs for a sparse set of joints. Experimental results on benchmark datasets have demonstrated that our proposed approach significantly outperforms existing methods for real-time human action recognition and prediction from depth data.

A novel gradient based method for RGB-D human action recognition

Most action recognition approaches proposed over the years that were designed for RGB sequences cannot utilize the rich 3D-structural information to reduce large intra-class variations. This paper addresses the issue of handling sole depth structural information for accurate human action recognition. It presents and evaluates the saliency based P-DmHOG features for human action representation. Saliency based P-DmHOG features are depth features inspired by the well-known HOG descriptor. Good results namely 96.78%, 97.78% and 93.13% are achieved on three public benchmark datasets: MSR Actions 3D, MSR Action Pairs 3D, and MSR Daily Activity 3D, which show the efficiency of proposed method to spatio-temporal and shape information.

Iss2Image: A Novel Signal-Encoding Technique for CNN-Based Human Activity Recognition.

The most significant barrier to success in human activity recognition is extracting and selecting the right features. In traditional methods, the features are chosen by humans, which requires the user to have expert knowledge or to do a large amount of empirical study. Newly developed deep learning technology can automatically extract and select features. Among the various deep learning methods, convolutional neural networks (CNNs) have the advantages of local dependency and scale invariance and are suitable for temporal data such as accelerometer (ACC) signals. In this paper, we propose an efficient human activity recognition method, namely Iss2Image (Inertial sensor signal to Image), a novel encoding technique for transforming an inertial sensor signal into an image with minimum distortion and a CNN model for image-based activity classification. Iss2Image converts real number values from the X, Y, and Z axes into three color channels to precisely infer correlations among successive sensor signal values in three different dimensions. We experimentally evaluated our method using several well-known datasets and our own dataset collected from a smartphone and smartwatch. The proposed method shows higher accuracy than other state-of-the-art approaches on the tested datasets.

Skeleton-Based Human Action Recognition by Pose Specificity and Weighted Voting

This paper introduces a human action recognition method based on skeletal data captured by Kinect or other depth sensors. After a series of pre-processing, action features such as position, velocity, and acceleration have been extracted from each frame to capture both dynamic and static information of human motion, which can make full use of the human skeletal data. The most challenging problem in skeleton-based human action recognition is the large variability within and across subjects. To handle this problem, we propose to divide human poses into two major categories: the discriminating pose and the common pose. A pose specificity metric has been proposed to quantify the discriminative level of different poses. Finally, the action recognition is actualized by a weighted voting method. This method uses the k nearest neighbors found from the training dataset for voting and uses the pose specificity as the weight of a ballot. Experiments on two benchmark datasets have been carried out, the results have illustrated that the proposed method outperforms the state-of-the-art methods.

A probabilistic data-driven method for human activity recognition

This paper proposes a probabilistic, time efficient, data-driven method for human low and medium level activity recognition and indoor tracking. The obtained results can be applied to a probabilistic reasoner for high level activity recognition. The proposed method is tested on Opportunity, a dataset consisting of daily morning activities in a highly sensor-rich environment. The main objective of this research is to suggest and apply methods suitable for batch processing of big data. In this case, performance in terms of CPU time and efficiency in storage usage are the top priorities. We applied fast signal processing methods to compute proper features from different collections of sensor signals. The relevant collections of features are selected and fed into a classifier to obtain results in the form of probability for each instance belonging to available classes. Additionally, the most probable locations of each subject in the room are calculated by processing noisy data from location tags on the subjects' body. Afterwards, the proposed probabilistic data smoothing method is applied to further increase accuracy. To evaluate the methods, the most probable recognitions are benchmarked against the results of the Opportunity Challenge competitions as well as provided results by the Opportunity group. We also implemented a couple of well-known methods on the current dataset and compared them with ours. Moreover, the performance of different sensors assemblies is investigated. Our proposed method could obtain very close results in terms of accuracy while it is more optimal in terms of number of features and required time.

RapidHARe: A computationally inexpensive method for real-time human activity recognition from wearable sensors

Recent human activity recognition (HAR) methods, based on on-body inertial sensors, have achieved increasing performance; however, this is at the expense of longer CPU calculations and greater energy consumption. Therefore, these complex models might not be suitable for real-time prediction in mobile systems, e.g., in elder-care support and long-term health-monitoring systems. Here, we present a new method called RapidHARe for real-time human activity recognition based on modeling the distribution of a raw data in a half-second context window using dynamic Bayesian networks. Our method does not employ any dynamic-programming-based algorithms, which are notoriously slow for inference, nor does it employ feature extraction or selection methods. In our comparative tests, we show that RapidHARe is an extremely fast predictor, one and a half times faster than artificial neural networks (ANNs) methods, and more than eight times faster than recurrent neural networks (RNNs) and hidden Markov models (HMMs). Moreover, in performance, RapidHare achieves an F1 score of 94.27\% and accuracy of 98.94\%, and when compared to ANN, RNN, HMM, it reduces the F1-score error rate by 45\%, 65\%, and 63\% and the accuracy error rate by 41\%, 55\%, and 62\%, respectively. Therefore, RapidHARe is suitable for real-time recognition in mobile devices.

PSO based combined kernel learning framework for recognition of first-person activity in a video

This paper presents human activity recognition problem from first-person view-point (ego-centric video). The task is to understand the activities of a person by an observer (wearable camera or robot) from real-time video data. An efficient human activity recognition system demands the choice of useful traits and the suitable kernels for those traits. In this work, we have proposed a combined kernel learning (CKL) framework using PSO as optimization algorithm for first-person activity recognition in a video. This framework does appropriate feature selection and combines those features from their respective kernels from the video data in a productive way. The proposed algorithm learns an optimal composite kernel from the combination of the basis kernel constructed from different motion-related features of the first-person video. To determine both basis kernel and their combination, this method can optimize a data-dependent kernel evaluation measure. The performance of the proposed CKL is evaluated by combining different types of motion features from the first-person video (JPL-interaction dataset). The result shows a comparatively better rate of accuracy than that of other state-of-the-art human activity recognition methods.

Human action recognition based on the Grassmann multi-graph embedding

In this paper, a human action recognition method based on the kernelized Grassmann manifold learning is introduced. The goal is to find a map which transfers the high-dimensional data to a discriminative low-dimensional space by considering the geometry of the manifold. To this end, a multi-graph embedding method using three graphs named as center-class, within-class and between-class similarity graphs is proposed. These graphs capture the local and semi-global information of data which is the benefit of the proposed method. Graphs play an important role in subspace learning methods. Most of the graph-based methods ignore the geometry of the manifold-valued data because of using Euclidean distance in graph construction. So, these methods are sensitive to noise and outliers. To handle these problems, the geodesic distance is used to build the neighborhood graphs. We analyze the performance of the proposed method on both noisy (complex) and less noisy (simple) datasets. So, two geodesic distance metrics are used to calculate the geodesic distance of these datasets. Experimental results show the performance of the proposed method.

DeepGesture: Deep learning-based gesture recognition scheme using motion sensors

Recent advancement in smart phones and sensor technology has promoted research in gesture recognition. This has made designing of efficient gesture interface easy. However, human activity recognition (HAR) through gestures is not trivial since each person may pose the same gesture differently. In this paper, we propose deepGesture algorithm, a new arm gesture recognition method based on gyroscope and accelerometer sensors using deep convolution and recurrent neural networks. This method uses four deep convolution layers to automate feature learning in raw sensor data. The features of the convolution layers are used as input of the gated recurrent unit (GRU) which is based on the state-of-the-art recurrent neural network (RNN) structure to capture long-term dependency and model sequential data. The input data of the proposed algorithm is obtained through motion sequence data extracted using a wrist-type smart band device equipped with gyroscope and accelerometer sensors. The data is initially segmented in fixed length segments. The segmented data is labeled and we construct the database. Then the labeled data is used in our learning algorithm. To verify the applicability of the algorithm, several experiments have been performed to measure the accuracy of gesture classification. Compared to the human activity recognition method, our experimental results show that the proposed deepGesture algorithm can increase the average F1-score for recognition of nine defined arm gestures by 6%.

Simultaneous joint and object trajectory templates for human activity recognition from 3-D data

Availability of low-cost range sensors and the development of relatively robust algorithms for the extraction of skeleton joint locations have inspired many researchers to develop human activity recognition methods using 3-D data. In this paper, an effective method for the recognition of human activities from the normalized joint trajectories is proposed. We represent the actions as multidimensional signals and introduce a novel method for generating action templates by averaging the samples in a “dynamic time” sense. Then, in order to deal with the variations in speed and style of performing actions, we warp the samples with action templates by an efficient algorithm and employ wavelet filters to extract meaningful spatiotemporal features. The proposed method is also capable of modeling the human-object interactions, by performing the template generation and temporal warping procedure via the joint and object trajectories simultaneously. Experimental evaluations on several challenging datasets demonstrates the effectiveness of our method compared to the state-of-the-arts as well as its robustness against different sources of noise.

Human activity recognition using dynamic representation and matching of skeleton feature sequences from RGB-D images

Segmenting a skeleton feature sequence into pose and motion feature segments can improve the effectiveness of key pose based human activity recognition. Nevertheless, the fixed number of atomic motions in feature segments results in the lost of temporal information, it makes the segmentation technique hardly suitable for all scenarios of feature sequences. To address this issue, this paper proposes a human activity recognition method using dynamic representation and matching of skeleton feature sequences based on the segmentation technique. In our method, a skeleton feature sequence is first segmented into key pose and atomic motion segments according to the potential differences. Afterwards, a learning strategy is proposed to select some frame sets with high confidence value, and we then extract series of poses (i.e., atomic motion series) with different numbers of poses from the learnt frame sets to represent motion feature segments, while a fixed number of centroids obtained by K-Means are used to represent pose feature segments. Finally, the shape dynamic time warping (shapeDTW) algorithm is utilized to measure the distance between the corresponding motion feature segments of two feature sequences. Comprehensive experiments are conducted on three public human activity datasets, and the results show that our method achieves superior performances in comparison with some state-of-the-art methods.