Problem Of Human Action Recognition Research Articles

P.1.11 Inflammatory markers in anorexia nervosa: An exploratory study

In this paper, we propose a semi-supervised classifier termed as Fast Laplacian Twin Support Vector Machine (FLap−TWSVM) with an objective to reduce the requirement of labeled data and simultaneously lessen the training time complexity of a traditional Laplacian Twin Support Vector Machine semi-supervised classifier. FLap−TWSVM is faster than existing Laplacian twin support vector machine as it solves a smaller size Quadratic Programming Problem (QPP) along with an Unconstrained Minimization Problem (UMP) to obtain decision hyperplanes which can also handle heteroscedastic noise present in the training data. Traditional semi-supervised classifiers generally have no explicit control over the choice of labeled data available for training, hence to overcome this limitation, we propose a pool-based active learning framework which identifies most informative examples to train the learning model. Moreover, the aforementioned framework has been extended to deal with multi-category classification scenarios. Several experiments have been performed on machine learning benchmark datasets which proves the utility of the proposed classifier over traditional Laplacian Twin Support Vector Machine (Lap−TWSVM) and active learning based Support Vector Machine (SVMAL). The efficacy of the proposed framework has also been tested on human activity recognition problem and content based image retrieval system.

Efficient Human Activity Recognition Solving the Confusing Activities Via Deep Ensemble Learning

The ubiquity of smartphones and their rich set of on-board sensors has created many exciting new opportunities, where smartphones are used as powerful computing platforms to sense and analyze pervasive data. One important application of mobile sensing is activity recognition based on smartphone inertial sensors, which is a fundamental building block for a variety of scenarios, such as indoor pedestrian tracking, mobile health care, and smart cities. Although many approaches have been proposed to address the human activity recognition problem, several challenges are still present: 1) people’s motion modes are very different for different individuals; 2) there is only a very limited amount of training data; 3) human activities can be arbitrary and complex, thus handcrafted feature engineering often fails to work; and 4) the recognition accuracy tends to be limited due to confusing activities. To tackle those challenges, in this paper, we propose a human activity recognition framework based on convolutional neural networks (CNNs) with two convolutional layers using the smartphone-based accelerometer, gyroscope, and magnetometer. To solve the confusion between highly similar activities like going upstairs and walking, this paper presents a novel ensemble model of CNN to further improve the identification accuracy. The extensive experiments have been conducted using 235 977 sensory samples from a total of 100 subjects. The results have shown that the classification accuracy of the proposed model can be up to 96.11%, which proves the effectiveness of the proposed model.

InnoHAR: A Deep Neural Network for Complex Human Activity Recognition

Human activity recognition (HAR) based on sensor networks is an important research direction in the fields of pervasive computing and body area network. Existing researches often use statistical machine learning methods to manually extract and construct features of different motions. However, in the face of extremely fast-growing waveform data with no obvious laws, the traditional feature engineering methods are becoming more and more incapable. With the development of deep learning technology, we do not need to manually extract features and can improve the performance in complex human activity recognition problems. By migrating deep neural network experience in image recognition, we propose a deep learning model (InnoHAR) based on the combination of inception neural network and recurrent neural network. The model inputs the waveform data of multi-channel sensors end-to-end. Multi-dimensional features are extracted by inception-like modules by using various kernel-based convolution layers. Combined with GRU, modeling for time series features is realized, making full use of data characteristics to complete classification tasks. Through experimental verification on three most widely used public HAR datasets, our proposed method shows consistent superior performance and has good generalization performance, when compared with the state-of-the-art.

Efficiency investigation from shallow to deep neural network techniques in human activity recognition

In the last years, several researchers measured different recognition rates with different artificial neural network (ANN) techniques on public data sets in the human activity recognition (HAR) problem. However an overall investigation does not exist in the literature and the efficiency of complex and deeper ANNs over shallow networks is not clear. The purpose of this paper is to investigate the recognition rate and time requirement of different kinds of ANN approaches in HAR. This work examines the performance of shallow ANN architectures with different hyper-parameters, ANN ensembles, binary ANN classifier groups, and convolutional neural networks on two public databases. Although the popularity of binary classifiers, classifier ensembles and deep learning have been significantly increasing, this study shows that shallow ANNs with appropriate hyper-parameters in combination with extracted features can reach similar or higher recognition rate in less time than other artificial neural network methods in HAR. With a well-tuned ANN we outperformed all previous results on two public databases. Consequently, instead of the more complex ANN techniques, the usage of simple ANN with two or three layers can be an appropriate choice for activity recognition.

Fast Laplacian twin support vector machine with active learning for pattern classification

In this paper, we propose a semi-supervised classifier termed as Fast Laplacian Twin Support Vector Machine (FLap−TWSVM) with an objective to reduce the requirement of labeled data and simultaneously lessen the training time complexity of a traditional Laplacian Twin Support Vector Machine semi-supervised classifier. FLap−TWSVM is faster than existing Laplacian twin support vector machine as it solves a smaller size Quadratic Programming Problem (QPP) along with an Unconstrained Minimization Problem (UMP) to obtain decision hyperplanes which can also handle heteroscedastic noise present in the training data. Traditional semi-supervised classifiers generally have no explicit control over the choice of labeled data available for training, hence to overcome this limitation, we propose a pool-based active learning framework which identifies most informative examples to train the learning model. Moreover, the aforementioned framework has been extended to deal with multi-category classification scenarios. Several experiments have been performed on machine learning benchmark datasets which proves the utility of the proposed classifier over traditional Laplacian Twin Support Vector Machine (Lap−TWSVM) and active learning based Support Vector Machine (SVMAL). The efficacy of the proposed framework has also been tested on human activity recognition problem and content based image retrieval system.

Learning Compact Features for Human Activity Recognition Via Probabilistic First-Take-All

With the popularity of mobile sensor technology, smart wearable devices open a unprecedented opportunity to solve the challenging human activity recognition (HAR) problem by learning expressive representations from the multi-dimensional daily sensor signals. This inspires us to develop a new algorithm applicable to both camera-based and wearable sensor-based HAR systems. Although competitive classification accuracy has been reported, existing methods often face the challenge of distinguishing visually similar activities composed of activity patterns in different temporal orders. In this paper, we propose a novel probabilistic algorithm to compactly encode temporal orders of activity patterns for HAR. Specifically, the algorithm learns an optimal set of latent patterns such that their temporal structures really matter in recognizing different human activities. Then, a novel probabilistic First-Take-All (pFTA) approach is introduced to generate compact features from the orders of these latent patterns to encode the entire sequence, and the temporal structural similarity between different sequences can be efficiently measured by the Hamming distance between compact features. Experiments on three public HAR datasets show the proposed pFTA approach can achieve competitive performance in terms of accuracy as well as efficiency.

Sequential Human Activity Recognition Based on Deep Convolutional Network and Extreme Learning Machine Using Wearable Sensors

Human activity recognition (HAR) problems have traditionally been solved by using engineered features obtained by heuristic methods. These methods ignore the time information of the streaming sensor data and cannot achieve sequential human activity recognition. With the use of traditional statistical learning methods, results could easily plunge into the local minimum other than the global optimal and also face the problem of low efficiency. Therefore, we propose a hybrid deep framework based on convolution operations, LSTM recurrent units, and ELM classifier; the advantages are as follows: (1) does not require expert knowledge in extracting features; (2) models temporal dynamics of features; and (3) is more suitable to classify the extracted features and shortens the runtime. All of these unique advantages make it superior to other HAR algorithms. We evaluate our framework on OPPORTUNITY dataset which has been used in OPPORTUNITY challenge. Results show that our proposed method outperforms deep nonrecurrent networks by 6%, outperforming the previous reported best result by 8%. When compared with neural network using BP algorithm, testing time reduced by 38%.

PSO based combined kernel learning framework for recognition of first-person activity in a video

This paper presents human activity recognition problem from first-person view-point (ego-centric video). The task is to understand the activities of a person by an observer (wearable camera or robot) from real-time video data. An efficient human activity recognition system demands the choice of useful traits and the suitable kernels for those traits. In this work, we have proposed a combined kernel learning (CKL) framework using PSO as optimization algorithm for first-person activity recognition in a video. This framework does appropriate feature selection and combines those features from their respective kernels from the video data in a productive way. The proposed algorithm learns an optimal composite kernel from the combination of the basis kernel constructed from different motion-related features of the first-person video. To determine both basis kernel and their combination, this method can optimize a data-dependent kernel evaluation measure. The performance of the proposed CKL is evaluated by combining different types of motion features from the first-person video (JPL-interaction dataset). The result shows a comparatively better rate of accuracy than that of other state-of-the-art human activity recognition methods.

Human Action Recognition Using 3D Reconstruction Data

In this paper, the problem of human action recognition using 3D reconstruction data is deeply investigated. 3D reconstruction techniques are employed for addressing two of the most challenging issues related to human action recognition in the general case, namely view variance (i.e., when the same action is observed from different viewpoints) and the presence of (self-) occlusions (i.e., when for a given point of view a body part of an individual conceals another body part of the same or another subject). The main contributions of this paper are summarized as follows. The first one is a detailed examination of the use of 3D reconstruction data for performing human action recognition. The latter includes the introduction of appropriate local/global flow/shape descriptors, extensive experiments in challenging publicly available datasets and exhaustive comparisons with state-of-art approaches. The second one is a new local-level 3D flow descriptor, which incorporates spatial and surface information in the flow representation and efficiently handles the problem of defining 3D orientation at every local neighborhood. The third one is a new global-level 3D flow descriptor that efficiently encodes the global motion characteristics in a compact way. The fourth one is a novel global temporal-shape descriptor that extends the notion of 3D shape descriptions for action recognition, by incorporating the temporal dimension. The proposed descriptor efficiently addresses the inherent problems of temporal alignment and compact representation, while also being robust in the presence of noise (compared with similar tracking-based methods of the literature). Overall, this paper significantly improves the state-of-art performance and introduces new research directions in the field of 3D action recognition, following the recent development and wide-spread use of portable, affordable, high-quality and accurate motion capturing devices (e.g., Microsoft Kinect).

Information Fusion for Human Action Recognition via Biset/Multiset Globality Locality Preserving Canonical Correlation Analysis.

In this paper, we study the problem of human action recognition, in which each action is captured by multiple sensors and represented by multisets. We propose two novel information fusion techniques for fusing the information from multisets. The first technique is biset globality locality preserving canonical correlation analysis (BGLPCCA), which aims to learn the common feature subspace between two sets. The second technique is multiset globality locality preserving canonical correlation analysis (MGLPCCA), which aims to deal with three or more sets. The proposed BGLPCCA and MGLPCCA are able to learn a low-dimensional common subspace that preserves the local and global structures of data samples. Moreover, two novel descriptors are presented for both depth and skeleton. We then propose a new human action recognition framework employing the proposed BGLPCCA or MGLPCCA to learn the shared subspace from multiple sets of features including skeleton, depth, and optical flow. Extensive experiments on five publicly available datasets (MSR Action3D, UTD multimodal human action dataset, multimodal action database, Kinect activity recognition dataset, and SBU Kinect interaction dataset) demonstrate the effectiveness of the proposed framework.

Evolutionary Design of Convolutional Neural Networks for Human Activity Recognition in Sensor-Rich Environments.

Human activity recognition is a challenging problem for context-aware systems and applications. It is gaining interest due to the ubiquity of different sensor sources, wearable smart objects, ambient sensors, etc. This task is usually approached as a supervised machine learning problem, where a label is to be predicted given some input data, such as the signals retrieved from different sensors. For tackling the human activity recognition problem in sensor network environments, in this paper we propose the use of deep learning (convolutional neural networks) to perform activity recognition using the publicly available OPPORTUNITY dataset. Instead of manually choosing a suitable topology, we will let an evolutionary algorithm design the optimal topology in order to maximize the classification F1 score. After that, we will also explore the performance of committees of the models resulting from the evolutionary process. Results analysis indicates that the proposed model was able to perform activity recognition within a heterogeneous sensor network environment, achieving very high accuracies when tested with new sensor data. Based on all conducted experiments, the proposed neuroevolutionary system has proved to be able to systematically find a classification model which is capable of outperforming previous results reported in the state-of-the-art, showing that this approach is useful and improves upon previously manually-designed architectures.

Human Activity Recognition from Body Sensor Data using Deep Learning.

In recent years, human activity recognition from body sensor data or wearable sensor data has become a considerable research attention from academia and health industry. This research can be useful for various e-health applications such as monitoring elderly and physical impaired people at Smart home to improve their rehabilitation processes. However, it is not easy to accurately and automatically recognize physical human activity through wearable sensors due to the complexity and variety of body activities. In this paper, we address the human activity recognition problem as a classification problem using wearable body sensor data. In particular, we propose to utilize a Deep Belief Network (DBN) model for successful human activity recognition. First, we extract the important initial features from the raw body sensor data. Then, a kernel principal component analysis (KPCA) and linear discriminant analysis (LDA) are performed to further process the features and make them more robust to be useful for fast activity recognition. Finally, the DBN is trained by these features. Various experiments were performed on a real-world wearable sensor dataset to verify the effectiveness of the deep learning algorithm. The results show that the proposed DBN outperformed other algorithms and achieves satisfactory activity recognition performance.

Bayesian optimization on graph-structured search spaces: Optimizing deep multimodal fusion architectures

Abstract A popular testbed for deep learning has been multimodal recognition of human activity or gesture involving diverse inputs like video, audio, skeletal pose and depth images. Deep learning architectures have excelled on such problems due to their ability to combine modality representations at different levels of nonlinear feature extraction. However, designing an optimal architecture in which to fuse such learned representations has largely been a non-trivial human engineering effort. We treat fusion structure optimization as a hyperparameter search and cast it as a discrete optimization problem under the Bayesian optimization framework. We propose two methods to compute structural similarities in the search space of tree-structured multimodal architectures, and demonstrate their effectiveness on two challenging multimodal human activity recognition problems.

Human Action Recognition Algorithm Based on Adaptive Initialization of Deep Learning Model Parameters and Support Vector Machine

Action recognition is a trending topic and key research direction in computer vision, machine learning, artificial intelligence, and other fields. This research seeks to identify human action in image and video data. Its research results have been widely used in the fields of safety monitoring, disability monitoring, understanding multimedia content, human–computer interaction, virtual reality, and so on. However, the existing traditional human action recognition technology has many limitations in practical application, such as low accuracy and weak adaptive ability. Although the action recognition based on deep learning can self-learn and improve the action recognition accuracy, there are many difficulties in training the deep neural network model, such as gradient disappearance, gradient explosion, and overfitting. Therefore, this paper will reduce the abovementioned difficulties in deep neural network model training from the perspective of deep neural network model parameter initialization and then propose a model parameter initialization method based on the multilayer maxout network activation function to solve the difficulties in deep neural network model training. Then, on this basis, a method of learning the temporal and spatial characteristics of human action based on the deep neural network model is proposed. First, the method detects and tracks the human action and uses the restricted Boltzmann machine (RBM) to encode the temporal and spatial features of various parts of the human body. Second, the temporal and spatial feature codes of various parts of the human body are integrated into a global temporal and spatial feature representation method of the action video through a RBM neural network. Finally, the trained SVM classifiers are used to recognize human action. Experiments show that the human action recognition method proposed in this paper not only has high recognition accuracy but also has great adaptability. Thus, this method extracts temporal and spatial features from the shape feature sequences of various parts of the human body, thus opening up a new way to extract human action features and solving the problem of human action recognition in complex scenes. Its proposal provides an exploratory technical method and approach for self-adaptive recognition of human action. It also gives directional enlightenment to the development and improvement of self-adaptive human action methods.

Deep Dilation on Multimodality Time Series for Human Activity Recognition

Providing accurate information on people’s activities and behaviors plays an important role in innumerable applications, such as medical, security, and entertainment. In recent years, deep learning has been applied in human activity recognition, and achieved a better performance. However, if the spatial dependency of inter-sensors is considered, it is possible to enhance the discriminative ability. In this paper, we present a novel deep learning framework for human activity recognition problems. First, on the basis of our previous work, we utilize dilated convolutional neural network to extract features of inter-sensors and intra-sensors. Since the extracted features are local and short-temporal, it is necessary to utilize RNN to model the long-temporal dependencies. However, duo to that LSTM and GRU often rely on the completely previous computations, it will result in slow inference and hard convergence. Hence, inspired by the idea of dilation operation, we present a novel recurrent model to learn the temporal dependencies at different time scales. Then, at the topmost layer, a fully-connected layer with softmax function is utilized to generate a class probability distribution, and the predicted activity is obtained. Eventually, we evaluate the proposed framework in two open human activity datasets, OPPORTUNITY and PAMAP2. Results demonstrate that the proposed framework achieves a higher classification performance than the state-of-the-art methods. Moreover, it takes the least time to recognize an activity. Besides, it also performs faster and easier to converge in the training stage.

Convolutional Neural Networks and Long Short-Term Memory for skeleton-based human activity and hand gesture recognition

In this work, we address human activity and hand gesture recognition problems using 3D data sequences obtained from full-body and hand skeletons, respectively. To this aim, we propose a deep learning-based approach for temporal 3D pose recognition problems based on a combination of a Convolutional Neural Network (CNN) and a Long Short-Term Memory (LSTM) recurrent network. We also present a two-stage training strategy which firstly focuses on CNN training and, secondly, adjusts the full method (CNN+LSTM). Experimental testing demonstrated that our training method obtains better results than a single-stage training strategy. Additionally, we propose a data augmentation method that has also been validated experimentally. Finally, we perform an extensive experimental study on publicly available data benchmarks. The results obtained show how the proposed approach reaches state-of-the-art performance when compared to the methods identified in the literature. The best results were obtained for small datasets, where the proposed data augmentation strategy has greater impact.

Naïve Bayes switching linear dynamical system: A model for dynamic system modelling, classification, and information fusion

The Naïve Bayes Switching Linear Dynamical System (NB-SLDS) is proposed as a novel variant of the switching linear dynamical system (SLDS). The variant models multi-variable systems that undergo regime changes in their dynamics. The model may be applied to identify regime changes or classify systems according to their dynamics. The NB-SLDS provides the means to fuse multiple sequential data sources into a single model. A key feature of the model is that it is able to handle missing and unsynchronised data. Filtering and smoothing algorithms for inference and an expectation maximisation algorithm for parameter learning in the NB-SLDS are presented. The model is demonstrated and compared to the SLDS and hidden Markov model (HMM) in a human action recognition problem.

A Comprehensive Analysis on Wearable Acceleration Sensors in Human Activity Recognition.

Sensor-based motion recognition integrates the emerging area of wearable sensors with novel machine learning techniques to make sense of low-level sensor data and provide rich contextual information in a real-life application. Although Human Activity Recognition (HAR) problem has been drawing the attention of researchers, it is still a subject of much debate due to the diverse nature of human activities and their tracking methods. Finding the best predictive model in this problem while considering different sources of heterogeneities can be very difficult to analyze theoretically, which stresses the need of an experimental study. Therefore, in this paper, we first create the most complete dataset, focusing on accelerometer sensors, with various sources of heterogeneities. We then conduct an extensive analysis on feature representations and classification techniques (the most comprehensive comparison yet with 293 classifiers) for activity recognition. Principal component analysis is applied to reduce the feature vector dimension while keeping essential information. The average classification accuracy of eight sensor positions is reported to be 96.44% ± 1.62% with 10-fold evaluation, whereas accuracy of 79.92% ± 9.68% is reached in the subject-independent evaluation. This study presents significant evidence that we can build predictive models for HAR problem under more realistic conditions, and still achieve highly accurate results.

Internal Transfer Learning for Improving Performance in Human Action Recognition for Small Datasets

Human action recognition nowadays plays a key role in varieties of computer vision applications. Many computer vision methods focus on algorithms designing classifiers with handcrafted features which are complex and inflexible. In this paper, we focus on the human action recognition problem and utilize 3D convolutional neural networks to automatically extract both spatial and temporal features for classification. Specifically, in order to address the training problems with small data sets, we propose an internal transfer learning strategy adapted to this framework, by incorporating the sub-data classification method into transfer learning. We evaluate our method on several data sets and obtain promising results. With the proposed strategy, the performance of human action recognition is improved obviously.

Multi-view human activity recognition based on silhouette and uniform rotation invariant local binary patterns

This paper addresses the problem of silhouette-based human activity recognition. Most of the previous work on silhouette based human activity recognition focus on recognition from a single view and...