Human Activity Recognition Dataset Research Articles

Benchmarking a Multimodal and Multiview and Interactive Dataset for Human Action Recognition.

Human action recognition is an active research area in both computer vision and machine learning communities. In the past decades, the machine learning problem has evolved from conventional single-view learning problem, to cross-view learning, cross-domain learning and multitask learning, where a large number of algorithms have been proposed in the literature. Despite having large number of action recognition datasets, most of them are designed for a subset of the four learning problems, where the comparisons between algorithms can further limited by variances within datasets, experimental configurations, and other factors. To the best of our knowledge, there exists no dataset that allows concurrent analysis on the four learning problems. In this paper, we introduce a novel multimodal and multiview and interactive (M2I) dataset, which is designed for the evaluation of human action recognition methods under all four scenarios. This dataset consists of 1760 action samples from 22 action categories, including nine person-person interactive actions and 13 person-object interactive actions. We systematically benchmark state-of-the-art approaches on M2I dataset on all four learning problems. Overall, we evaluated 13 approaches with nine popular feature and descriptor combinations. Our comprehensive analysis demonstrates that M2I dataset is challenging due to significant intraclass and view variations, and multiple similar action categories, as well as provides solid foundation for the evaluation of existing state-of-the-art algorithms.

PERFORMANCE ANALYSIS OF MLP AND SVM BASED CLASSIFIERS FOR HUMAN ACTIVITY RECOGNITION USING SMARTPHONE SENSORS DATA

Days have gone when Mobile Phone used to be matter luxury, it has become a significant need for rapidly evolving fast track world. Intelligent aspects of the computing device enhance the importance of the interface development since; efficient collaboration always relies on the good communication between man and machine. The information about mobile device and user activity, environment, other devices, location and time can be utilized in different situations to enhance the interaction between the user and the device. Here, we presented our work in which certain types of human physical activities using accelerometer and gyroscope data generated by a mobile device. The benchmark Human Activity Recognition dataset is considered for this work is acquired from UCI Machine Learning Repository which is available in public domain. MLP and SVM Classifiers were tested using various time domain and frequency domain features. We found the using Multi Layer Perceptron with Processing Element 6, Learning rate 0.05, Momentum 0.1 reach an overall accuracy of 98.11%. Keywords— Context; Framework; Adaptive User Interfaces; Classifier, Multilayer Perceptron, Support Vector

Learning semantic context feature-tree for action recognition via nearest neighbor fusion

The spatio-temporal context learnt by the traditional methods for action recognition lacks the semantic meanings and temporal relationships. In order to deal with the drawbacks, we propose a novel semantic context feature-tree model to model the video clip for efficient human action recognition. The proposed method enforces spatio-temporal interest points (STIPs) within an irregular spatio-temporal volume to construct a semantic trees-structured relationship by nearest neighbor fusion. Specifically, we firstly extract STIPs, and moreover, utilize super-pixels to segment the motion image obtained by STIP detection. The points, which fall into super-pixel, are viewed as spatial semantic co-occurring features to represent one body part. Secondly, by patch matching, the point sets which are temporal nearest neighbors are merged into a new node of the next layer to describe the context of one moving part. After matching and associating, the sets of the frame indexes are renewed to group for the next fusion process until the conditions of the recursive process do not satisfy. Using KTH, UCF-YouTube and HOHA action datasets for human action recognition, our representation based on the learnt tree-structured features enhances the discriminative power of action descriptor, and obtains promising results.

Human activity recognition with smartphone sensors using deep learning neural networks

Human activities are inherently translation invariant and hierarchical. Human activity recognition (HAR), a field that has garnered a lot of attention in recent years due to its high demand in various application domains, makes use of time-series sensor data to infer activities. In this paper, a deep convolutional neural network (convnet) is proposed to perform efficient and effective HAR using smartphone sensors by exploiting the inherent characteristics of activities and 1D time-series signals, at the same time providing a way to automatically and data-adaptively extract robust features from raw data. Experiments show that convnets indeed derive relevant and more complex features with every additional layer, although difference of feature complexity level decreases with every additional layer. A wider time span of temporal local correlation can be exploited (1×9–1×14) and a low pooling size (1×2–1×3) is shown to be beneficial. Convnets also achieved an almost perfect classification on moving activities, especially very similar ones which were previously perceived to be very difficult to classify. Lastly, convnets outperform other state-of-the-art data mining techniques in HAR for the benchmark dataset collected from 30 volunteer subjects, achieving an overall performance of 94.79% on the test set with raw sensor data, and 95.75% with additional information of temporal fast Fourier transform of the HAR data set.

Convolutional neural random fields for action recognition

A deep discriminative structured model, convolutional neural random fields (CNRFs), is proposed for action recognition problem. In the proposed model, a spatio-temporal convolutional neural network (CNN) is developed for invariant feature learning from raw input frames, and the CNN is combined with conditional random fields (CRFs) for capturing the interdependencies between outputs. The parameters from both CRF and CNN are learned in a joint fashion which enables structured prediction and feature learning as well. We also explore different combinations of observation and transition feature functions based on the learned high level features from convolution part. The approach enjoys the advantages of both CNN and CRF, it has the invariant feature learning ability possessed by the former and structured prediction ability of the latter. The experimental results on both segmented and unsegmented human action recognition datasets show that CNRF boosts the performance over the comparison methods by a large margin.

Co-Occurrence Feature Learning for Skeleton Based Action Recognition Using Regularized Deep LSTM Networks

Skeleton based action recognition distinguishes human actions using the trajectories of skeleton joints, which provide a very good representation for describing actions. Considering that recurrent neural networks (RNNs) with Long Short-Term Memory (LSTM) can learn feature representations and model long-term temporal dependencies automatically, we propose an end-to-end fully connected deep LSTM network for skeleton based action recognition. Inspired by the observation that the co-occurrences of the joints intrinsically characterize human actions, we take the skeleton as the input at each time slot and introduce a novel regularization scheme to learn the co-occurrence features of skeleton joints. To train the deep LSTM network effectively, we propose a new dropout algorithm which simultaneously operates on the gates, cells, and output responses of the LSTM neurons. Experimental results on three human action recognition datasets consistently demonstrate the effectiveness of the proposed model.

Histogram of Oriented Principal Components for Cross-View Action Recognition.

Existing techniques for 3D action recognition are sensitive to viewpoint variations because they extract features from depth images which are viewpoint dependent. In contrast, we directly process pointclouds for cross-view action recognition from unknown and unseen views. We propose the histogram of oriented principal components (HOPC) descriptor that is robust to noise, viewpoint, scale and action speed variations. At a 3D point, HOPC is computed by projecting the three scaled eigenvectors of the pointcloud within its local spatio-temporal support volume onto the vertices of a regular dodecahedron. HOPC is also used for the detection of spatio-temporal keypoints (STK) in 3D pointcloud sequences so that view-invariant STK descriptors (or Local HOPC descriptors) at these key locations only are used for action recognition. We also propose a global descriptor computed from the normalized spatio-temporal distribution of STKs in 4-D, which we refer to as STK-D. We have evaluated the performance of our proposed descriptors against nine existing techniques on two cross-view and three single-view human action recognition datasets. The experimental results show that our techniques provide significant improvement over state-of-the-art methods.

P-Laplacian Regularized Sparse Coding for Human Activity Recognition

Human activity analysis in videos has increasingly attracted attention in computer vision research with the massive number of videos now accessible online. Although many recognition algorithms have been reported recently, activity representation is challenging. Recently, manifold regularized sparse coding has obtained promising performance in action recognition, because it simultaneously learns the sparse representation and preserves the manifold structure. In this paper, we propose a generalized version of Laplacian regularized sparse coding for human activity recognition called $p$ -Laplacian regularized sparse coding (pLSC). The proposed method exploits $p$ -Laplacian regularization to preserve the local geometry. The $p$ -Laplacian is a nonlinear generalization of standard graph Laplacian and has tighter isoperimetric inequality. As a result, pLSC provides superior theoretical evidence than standard Laplacian regularized sparse coding with a proper $p$ . We also provide a fast iterative shrinkage-thresholding algorithm for the optimization of pLSC. Finally, we input the sparse codes learned by the pLSC algorithm into support vector machines and conduct extensive experiments on the unstructured social activity attribute dataset and human motion database (HMDB51) for human activity recognition. The experimental results demonstrate that the proposed pLSC algorithm outperforms the manifold regularized sparse coding algorithms including the standard Laplacian regularized sparse coding algorithm with a proper $p$ .

A Review of Human Activity Recognition Methods

Recognizing human activities from video sequences or still images is a challenging task due to problems such as background clutter, partial occlusion, changes in scale, viewpoint, lighting, and appearance. Many applications, including video surveillance systems, human-computer interaction, and robotics for human behavior characterization, require a multiple activity recognition system. In this work, we provide a detailed review of recent and state-of-the-art research advances in the field of human activity classification. We propose a categorization of human activity methodologies and discuss their advantages and limitations. In particular, we divide human activity classification methods into two large categories according to whether they use data from different modalities or not. Then, each of these categories is further analyzed into sub-categories, which reflect how they model human activities and what type of activities they are interested in. Moreover, we provide a comprehensive analysis of the existing, publicly available human activity classification datasets and examine the requirements for an ideal human activity recognition dataset. Finally, we report the characteristics of future research directions and present some open issues on human activity recognition.

Multiview human activity recognition system based on spatiotemporal template for video surveillance system

An efficient view invariant framework for the recognition of human activities from an input video sequence is presented. The proposed framework is composed of three consecutive modules: (i) detect and locate people by background subtraction, (ii) view invariant spatiotemporal template creation for different activities, (iii) and finally, template matching is performed for view invariant activity recognition. The foreground objects present in a scene are extracted using change detection and background modeling. The view invariant templates are constructed using the motion history images and object shape information for different human activities in a video sequence. For matching the spatiotemporal templates for various activities, the moment invariants and Mahalanobis distance are used. The proposed approach is tested successfully on our own viewpoint dataset, KTH action recognition dataset, i3DPost multiview dataset, MSR viewpoint action dataset, VideoWeb multiview dataset, and WVU multiview human action recognition dataset. From the experimental results and analysis over the chosen datasets, it is observed that the proposed framework is robust, flexible, and efficient with respect to multiple views activity recognition, scale, and phase variations.

A general framework for co-training and its applications

Co-training is one of the major semi-supervised learning paradigms in which two classifiers are alternately trained on two distinct views and they teach each other by adding the predictions of unlabeled data to the training set of the other view. Co-training can achieve promising performance, especially when there is only a small number of labeled data. Hence, co-training has received considerable attention, and many variant co-training algorithms have been developed. It is essential and informative to provide a systematic framework for a better understanding of the common properties and differences in these algorithms. In this paper, we propose a general framework for co-training according to the diverse learners constructed in co-training. Specifically, we provide three types of co-training implementations, including co-training on multiple views, co-training on multiple classifiers, and co-training on multiple manifolds. Finally, comprehensive experiments of different methods are conducted on the UCF-iPhone dataset for human action recognition and the USAA dataset for social activity recognition. The experimental results demonstrate the effectiveness of the proposed solutions.

Multipe/single-view human action recognition via part-induced multitask structural learning.

This paper proposes a unified framework for multiple/single-view human action recognition. First, we propose the hierarchical partwise bag-of-words representation which encodes both local and global visual saliency based on the body structure cue. Then, we formulate the multiple/single-view human action recognition as a part-regularized multitask structural learning (MTSL) problem which has two advantages on both model learning and feature selection: 1) preserving the consistence between the body-based action classification and the part-based action classification with the complementary information among different action categories and multiple views and 2) discovering both action-specific and action-shared feature subspaces to strengthen the generalization ability of model learning. Moreover, we contribute two novel human action recognition datasets, TJU (a single-view multimodal dataset) and MV-TJU (a multiview multimodal dataset). The proposed method is validated on three kinds of challenging datasets, including two single-view RGB datasets (KTH and TJU), two well-known depth dataset (MSR action 3-D and MSR daily activity 3-D), and one novel multiview multimodal dataset (MV-TJU). The extensive experimental results show that this method can outperform the popular 2-D/3-D part model-based methods and several other competing methods for multiple/single-view human action recognition in both RGB and depth modalities. To our knowledge, this paper is the first to demonstrate the applicability of MTSL with part-based regularization on multiple/single-view human action recognition in both RGB and depth modalities.

Human Action Recognition Via Multi-modality Information

In this paper, we propose pyramid appearance and global structure action descriptors on both RGB and depth motion history images and a model-free method for human action recognition. In proposed algorithm, we firstly construct motion history image for both RGB and depth channels, at the same time, depth information is employed to filter RGB information, after that, different action descriptors are extracted from depth and RGB MHIs to represent these actions, and then multimodality information collaborative representation and recognition model, in which multi-modality information are put into object function naturally, and information fusion and action recognition also be done together, is proposed to classify human actions. To demonstrate the superiority of the proposed method, we evaluate it on MSR Action3D and DHA datasets, the well-known dataset for human action recognition. Large scale experiment shows our descriptors are robust, stable and efficient, when comparing with the-state-of-the-art algorithms, the performances of our descriptors are better than that of them, further, the performance of combined descriptors is much better than just using sole descriptor. What is more, our proposed model outperforms the state-of-the-art methods on both MSR Action3D and DHA datasets.

Human Action Recognition Using Pyramid Histograms of Oriented Gradients and Collaborative Multi-task Learning

In this paper, human action recognition using pyramid histograms of oriented gradients and collaborative multi-task learning is proposed. First, we accumulate global activities and construct motion history image (MHI) for both RGB and depth channels respectively to encode the dynamics of one action in different modalities, and then different action descriptors are extracted from depth and RGB MHI to represent global textual and structural characteristics of these actions. Specially, average value in hierarchical block, GIST and pyramid histograms of oriented gradients descriptors are employed to represent human motion. To demonstrate the superiority of the proposed method, we evaluate them by KNN, SVM with linear and RBF kernels, SRC and CRC models on DHA dataset, the well-known dataset for human action recognition. Large scale experimental results show our descriptors are robust, stable and efficient, and outperform the state-of-the-art methods. In addition, we investigate the performance of our descriptors further by combining these descriptors on DHA dataset, and observe that the performances of combined descriptors are much better than just using only sole descriptor. With multimodal features, we also propose a collaborative multi-task learning method for model learning and inference based on transfer learning theory. The main contributions lie in four aspects: 1) the proposed encoding the scheme can filter the stationary part of human body and reduce noise interference; 2) different kind of features and models are assessed, and the neighbor gradients information and pyramid layers are very helpful for representing these actions; 3) The proposed model can fuse the features from different modalities regardless of the sensor types, the ranges of the value, and the dimensions of different features; 4) The latent common knowledge among different modalities can be discovered by transfer learning to boost the performance.

Skin detection by dual maximization of detectors agreement for video monitoring

This paper presents an approach for skin detection which is able to adapt its parameters to image data captured from video monitoring tasks with a medium field of view. It is composed of two detectors designed to get high and low probable skin pixels (respectively, regions and isolated pixels). Each one is based on thresholding two color channels, which are dynamically selected. Adaptation is based on the agreement maximization framework, whose aim is to find the configuration with the highest similarity between the channel results. Moreover, we improve such framework by learning how detector parameters are related and proposing an agreement function to consider expected skin properties. Finally, both detectors are combined by morphological reconstruction filtering to keep the skin regions whilst removing wrongly detected regions. The proposed approach is evaluated on heterogeneous human activity recognition datasets outperforming the most relevant state-of-the-art approaches.

3D flow estimation for human action recognition from colored point clouds

Abstract Motion perception and classification are key elements exploited by humans for recognizing actions. The same principles can serve as a basis for building cognitive architectures which can recognize human actions, thus enhancing challenging applications such as human robot interaction, visual surveillance, content-based video analysis and motion capture. In this paper, we propose an autonomous system for real-time human action recognition based on 3D motion flow estimation. We exploit colored point cloud data acquired with a Microsoft Kinect and we summarize the motion information by means of a 3D grid-based descriptor. Finally, temporal sequences of descriptors are classified with the Nearest Neighbor technique. We also present a newly created public dataset for RGB-D human action recognition which contains 15 actions performed by 12 different people. Our overall system is tested on this dataset and on the dataset used in Ballin, Munaro, and Menegatti (2012) , showing the effectiveness of the proposed approach in recognizing about 90% of the actions.

A survey of video datasets for human action and activity recognition

Vision-based human action and activity recognition has an increasing importance among the computer vision community with applications to visual surveillance, video retrieval and human–computer interaction. In recent years, more and more datasets dedicated to human action and activity recognition have been created. The use of these datasets allows us to compare different recognition systems with the same input data. The survey introduced in this paper tries to cover the lack of a complete description of the most important public datasets for video-based human activity and action recognition and to guide researchers in the election of the most suitable dataset for benchmarking their algorithms.

A real-time system for motion retrieval and interpretation

This paper proposes a new examplar-based method for real-time human motion recognition using Motion Capture (MoCap) data. We have formalized streamed recognizable actions, coming from an online MoCap engine, into a motion graph that is similar to an animation motion graph. This graph is used as an automaton to recognize known actions as well as to add new ones. We have defined and used a spatio-temporal metric for similarity measurements to achieve more accurate feedbacks on classification. The proposed method has the advantage of being linear and incremental, making the recognition process very fast and the addition of a new action straightforward. Furthermore, actions can be recognized with a score even before they are fully completed. Thanks to the use of a skeleton-centric coordinate system, our recognition method has become view-invariant. We have successfully tested our action recognition method on both synthetic and real data. We have also compared our results with four state-of-the-art methods using three well known datasets for human action recognition. In particular, the comparisons have clearly shown the advantage of our method through better recognition rates.

Spatiotemporal Sparsity Induced Similarity Measure for Human Action Recognition

In this paper, we propose a model-free method for human action recognition via sparse spatiotemporal representation. Similar to the template matching method, the philosophy of the proposed method is to decompose each video sample containing one kind of human actions as a 1 sparse linear combination of several video samples containing multiple kinds of human actions. To realize this goal, we mainly focus on three problems, feature point detection and description for human action, spatiotemporal feature construction for action video representation, and action video decomposition via sparse representation. The contributions of this method lies in three-folds: 1) the proposed method does not depend on complicated model selection and learning; 2) it can handle the action recognition under difference scenarios by multiple persons; 3) the generalization ability of the method can be easily extended by simply adding bases, the new labeled action video. To demonstrate the superiority of the proposed method, we evaluate it on KTH dataset, the well known dataset for human action recognition. Large scale experiment shows the accuracy and robustness of the method. Moreover, the proposed method outperforms most of the state-of-art methods for human behavior recognition.