Human Activity Recognition Model Research Articles

Collaborative multimodal feature learning for RGB-D action recognition

The emergence of cost-effective depth sensors opens up a new dimension for RGB-D based human action recognition. In this paper, we propose a collaborative multimodal feature learning (CMFL) model for human action recognition from RGB-D sequences. Specifically, we propose a robust spatio-temporal pyramid feature (RSTPF) to capture dynamic local patterns around each human joint. The proposed CMFL model fuses multimodal data (skeleton, depth and RGB), and learns action classifiers using the fused features. The original low-level feature matrices are factorized to learn shared features and modality-specific features under a supervised fashion. The shared features describe the common structures among the three modalities while the modality-specific features capture intrinsic information of each modality. We formulate shared-specific features mining and action classifiers learning in a unified max-margin framework, and solve the formulation using an iterative optimization algorithm. Experimental results on four action datasets demonstrate the efficacy of the proposed method.

Unsupervised learning of finite full covariance multivariate generalized Gaussian mixture models for human activity recognition

We propose in this paper to recognize human activities through an unsupervised learning of finite multivariate generalized Gaussian mixture model. We address an important cue in finite mixture model which is the estimation of the mixture model’s parameters for a full covariance matrix. We have developed a novel learning algorithm based on Fixed-point covariance matrix estimator combined with the Expectation-Maximization algorithm. Furthermore, we have proposed an appropriate minimum message length (MML) criterion to deal with model selection problem. We evaluated our proposed method on synthetic datasets and a challenging application namely : Human activity recognition from images and videos. The obtained resutls show clearly the merits of our proposed framework which has better capabilities with full covariance matrix when modeling correlated data.

Robust Human Activity Recognition Using Multimodal Feature-Level Fusion

Automated recognition of human activities or actions has great significance as it incorporates wide-ranging applications, including surveillance, robotics, and personal health monitoring. Over the past few years, many computer vision-based methods have been developed for recognizing human actions from RGB and depth camera videos. These methods include space-time trajectory, motion encoding, key poses extraction, space-time occupancy patterns, depth motion maps, and skeleton joints. However, these camera-based approaches are affected by background clutter and illumination changes and applicable to a limited field of view only. Wearable inertial sensors provide a viable solution to these challenges but are subject to several limitations such as location and orientation sensitivity. Due to the complementary trait of the data obtained from the camera and inertial sensors, the utilization of multiple sensing modalities for accurate recognition of human actions is gradually increasing. This paper presents a viable multimodal feature-level fusion approach for robust human action recognition, which utilizes data from multiple sensors, including RGB camera, depth sensor, and wearable inertial sensors. We extracted the computationally efficient features from the data obtained from RGB-D video camera and inertial body sensors. These features include densely extracted histogram of oriented gradient (HOG) features from RGB/depth videos and statistical signal attributes from wearable sensors data. The proposed human action recognition (HAR) framework is tested on a publicly available multimodal human action dataset UTD-MHAD consisting of 27 different human actions. K-nearest neighbor and support vector machine classifiers are used for training and testing the proposed fusion model for HAR. The experimental results indicate that the proposed scheme achieves better recognition results as compared to the state of the art. The feature-level fusion of RGB and inertial sensors provides the overall best performance for the proposed system, with an accuracy rate of 97.6%.

Hierarchical Human Action Recognition with Self-Selection Classifiers via Skeleton Data**Supported by the National Nature Science Foundation of China under Grant Nos. 11475003, 61603003, and 11471093; the Key Project of Cultivation of Leading Talents in Universities of Anhui Province under Grant No. gxfxZD2016174; Funds of Integration of Cloud Computing and Big Data; Innovation of Science and Technology of Ministry of Education

Human action recognition has become one of the most active research topics in human-computer interaction and artificial intelligence, and has attracted much attention. Here, we employ a low-cost optical sensor Kinect to capture the action information of the human skeleton. We then propose a two-level hierarchical human action recognition model with self-selection classifiers via skeleton data. Especially different optimal classifiers are selected by probability voting mechanism and 10 times 10-fold cross validation at different coarse grained levels. Extensive simulations on a well-known open dataset and results demonstrate that our proposed method is efficient in human action recognition, achieving 94.19% the average recognition rate and 95.61% the best rate.

An efficient method to resolve intraclass variability using highly refined HOG description model for human action recognition

SummaryRecognizing the human action is an interesting field and promising area of research due to its importance in various applications. HAR system seeks to interpret the actions being accomplished by the human within the video sequence instinctively and tagging their actions for the primary need of an intelligent video system. Recently, the research attention becomes more focused on recognizing the human actions in unrestrained videos, as the variations in scale, illumination, etc, are often will be the case for performance degradation. Hence, we introduced a novel refined gradient model to normalize the HOG descriptor for the scale‐invariant and appearance modeling. In addition, as different subjects can perform the same action in a different manner. Their different appearance or style variation leads to errors. To resolve this Intraclass variability of actions a discriminative approach has been proposed to explore action attributes. Thus, this system unites these two to reduce the discrimination error. This new approach gives near‐perfect ways for reducing overall dense trajectory displacements with significant accuracy enhancements. The complete action recognition system was evaluated on a number of test videos from real‐world data sets and compared against state‐of‐the‐art methods.

Associative memory and recall model with KID model for human activity recognition

Learning, memorizing and recalling knowledge are the basic functions of cognitive models. These models must prioritize which stimulants to respond to as well as package acquired knowledge in an easy to retrieve manner. The human brain is a cognitive model that derives information from sensor data such as vision, associates different patterns to create knowledge, and uses chunking mechanisms to package the acquired knowledge in manageable entities. The use of chunking mechanisms by the brain aids it to overcome its short-term memory (STM) capacity limitation. Through chunking, each entity held in the STM is a chunk containing more associations (knowledge) in it. By mimicking the human brain, this study proposes an associative memory and recall (AMR) model that stores associative knowledge from sensor data. Using chunking mechanisms, AMR can organize human activity knowledge in the manner that is efficient and effective to store and recall. The knowledge–information–data (KID) model is used for learning associative knowledge while the AMR continuously looks for associations among knowledge units and merges related units using merging mechanisms. The chunking mechanisms used in this study are inspired by the chunking mechanisms of the brain i.e. goal oriented chunking and automatic chunking.

Multi‐feature consultation model for human action recognition in depth video sequence

In the field of computer vision research, the research on human action recognition of depth video sequence is an important research direction. Herein, considering the characteristics of temporal and spatial depth video sequence, the authors propose a framework of the consultation model of several action sequence features to solve the classification problem in-depth video sequence. According to the characteristics of the 3D human action space, a variety of action sequence feature data is obtained, and then these data is projected to three coordinate planes, the acquired fusion features are used to train the consultation model, and finally the model is validated through the data. The authors have achieved good results by comparing the two publicly available datasets with the other methods. Experimental results demonstrate that the model performs well in existing identification methods.

A Survey on Bayesian Learning Model for Human Action Recognition

Human action recognition is to judge human action by analyzing action characteristics in the fields of computer vision and video surveillance. As the development of machine learning technique, the application of Bayesian Learning Model is increasing in related fields. In order to analyze the characteristics of human action and then recognize human action, this paper introduce a survey on Bayesian Learning Model for Human Action Recognition. The paper focuses on Bayesian handcrafted and deep learning models, and evaluate the state-of-the-art benchmark datasets, e.g., Weizmann, KTH, MSR-3D, HOHA, and UCF101. In this paper, all papers are published ranging from 2007 to 2016, which provides an overview of the progress in this area.

Human Activity Recognition for Mobile Robot

Due to the increasing number of mobile robots including domestic robots for cleaning and maintenance in developed countries, human activity recognition is inevitable for congruent human-robot interaction. Needless to say that this is indeed a challenging task for robots, it is expedient to learn human activities for autonomous mobile robots (AMR) for navigating in an uncontrolled environment without any guidance. Building a correct classifier for complex human action is non-trivial since simple actions can be combined to recognize a complex human activity. In this paper, we trained a model for human activity recognition using convolutional neural network. We trained and validated our model using the Vicon physical action dataset and also tested the model on our generated dataset (VMCUHK). Our experiment shows that our method performs with high accuracy, human activity recognition task both on the Vicon physical action dataset and VMCUHK dataset.

Personalized Human Activity Recognition using Wearables: A Manifold Learning-based Knowledge Transfer.

Human activity recognition (HAR) is an important component in health-care systems. For example, it can enable context-aware applications such as elderly care and patient monitoring. Relying on a set of training data, supervised machine learning algorithms form the core intelligence of most existing HAR systems. Meanwhile, the accuracy of an HAR model highly depends on the similarity between the training and the operating context. Therefore, there is a need for developing machine learning algorithms that can easily adapt to the operating context at hand. In this paper, we propose a cross-subject transfer learning algorithm that links source and target subjects by constructing manifolds from feature-level representation of the source subject(s). Our algorithm assigns labels to the unlabeled data in the current context using the manifold learned from the source subject(s). The newly labeled data is used to develop a personalized HAR model for the current context (i.e., target subject). We demonstrate the efficacy of the algorithm using a publicly available dataset on HAR. We show that the proposed framework improves the accuracy of activity recognition by up to 24%.

A unified model for human activity recognition using spatial distribution of gradients and difference of Gaussian kernel

Understanding of human action and activity from video data is growing field and received rapid importance due to surveillance, security, entertainment and personal logging. In this work, a new hybrid technique is proposed for the description of human action and activity in video sequences. The unified framework endows a robust feature vector wrapping both global and local information strengthening discriminative depiction of action recognition. Initially, entropy-based texture segmentation is used for human silhouette extraction followed by construction of average energy silhouette images (AEIs). AEIs are the 2D binary projection of human silhouette frames of the video sequences, which reduces the feature vector generation time complexity. Spatial Distribution Gradients are computed at different levels of resolution of sub-images of AEI consisting overall shape variations of human silhouette during the activity. Due to scale, rotation and translation invariant properties of STIPs, the vocabulary of DoG-based STIPs are created using vector quantization which is unique for each class of the activity. Extensive experiments are conducted to validate the performance of the proposed approach on four standard benchmarks, i.e., Weizmann, KTH, Ballet Movements, Multi-view IXMAS. Promising results are obtained when compared with the similar state of the arts, demonstrating the robustness of the proposed hybrid feature vector for different types of challenges—illumination, view variations posed by the datasets.

Learned Model for Human Activity Recognition Based on Dimensionality Reduction

Pattern recognition is an interesting research area, especially for human activity recognition (HAR). The purpose of HAR study is to automatically detect human activities from the information acquired from different sensors using statistical techniques. Despite the great efforts for HAR is still a challenging task because of high dimensionality problems. Several approaches have recently been developed to solve these problems. The mean objective of this work is to deal with high dimensionality in HAR modeling to minimize the number of features and maximize the classification accuracy.

Learning structures of interval-based Bayesian networks in probabilistic generative model for human complex activity recognition

Complex activity recognition is challenging due to the inherent uncertainty and diversity of performing a complex activity. Normally, each instance of a complex activity has its own configuration of atomic actions and their temporal dependencies. In our previous work, we proposed an atomic action-based Bayesian model that constructs Allen’s interval relation networks to characterize complex activities in a probabilistic generative way: By introducing latent variables from the Chinese restaurant process, our approach is able to capture all possible styles of a particular complex activity as a unique set of distributions over atomic actions and relations. However, a major limitation of our previous models is their fixed network structures, which may lead to an overtrained or undertrained model owing to unnecessary or missing links in a network. In this work, we present an improved model that network structures can be automatically learned from empirical data, allowing itself to characterize complex activities with structural varieties. In addition, a new dataset of complex hand activities has been constructed and made publicly available, which is much larger in size than any existing datasets. Empirical evaluations on benchmark datasets as well as our in-house dataset demonstrate the competitiveness of our approach.

Human Activity Recognition from Body Sensor Data using Deep Learning.

In recent years, human activity recognition from body sensor data or wearable sensor data has become a considerable research attention from academia and health industry. This research can be useful for various e-health applications such as monitoring elderly and physical impaired people at Smart home to improve their rehabilitation processes. However, it is not easy to accurately and automatically recognize physical human activity through wearable sensors due to the complexity and variety of body activities. In this paper, we address the human activity recognition problem as a classification problem using wearable body sensor data. In particular, we propose to utilize a Deep Belief Network (DBN) model for successful human activity recognition. First, we extract the important initial features from the raw body sensor data. Then, a kernel principal component analysis (KPCA) and linear discriminant analysis (LDA) are performed to further process the features and make them more robust to be useful for fast activity recognition. Finally, the DBN is trained by these features. Various experiments were performed on a real-world wearable sensor dataset to verify the effectiveness of the deep learning algorithm. The results show that the proposed DBN outperformed other algorithms and achieves satisfactory activity recognition performance.

Log-Viterbi algorithm applied on second-order hidden Markov model for human activity recognition

Recognition of human activities is getting into the limelight among researchers in the field of pervasive computing, ambient intelligence, robotic, and monitoring such as assistive living, elderly care, and health care. Many platforms, models, and algorithms have been developed and implemented to recognize the human activities. However, existing approaches suffer from low-activity accuracy and high time complexity. Therefore, we proposed probabilistic log-Viterbi algorithm on second-order hidden Markov model that facilitates our algorithm by reducing the time complexity with increased accuracy. Second-order hidden Markov model is efficient relevance between previous two activities, current activity, and current observation that incorporate more information into recognition procedure. The log-Viterbi algorithm converts the products of a large number of probabilities into additions and finds the most likely activity from observation sequence under given model. Therefore, this approach maximizes the probability of activity recognition with improved accuracy and reduced time complexity. We compared our proposed algorithm among other famous probabilistic models such as Naïve Bayes, condition random field, hidden Markov model, and hidden semi-Markov model using three datasets in the smart home environment. The recognition possibility of our proposed method is significantly better in accuracy and time complexity than early proposed method. Moreover, this improved algorithm for activity recognition is much effective for almost all the dynamic environments such as assistive living, elderly care, healthcare applications, and home automation.

DFTerNet: Towards 2-bit Dynamic Fusion Networks for Accurate Human Activity Recognition

Deep Convolutional Neural Networks (DCNNs) are currently popular in human activity recognition applications. However, in the face of modern artificial intelligence sensor-based games, many research achievements cannot be practically applied on portable devices. DCNNs are typically resource-intensive and too large to be deployed on portable devices, thus this limits the practical application of complex activity detection. In addition, since portable devices do not possess high-performance Graphic Processing Units (GPUs), there is hardly any improvement in Action Game (ACT) experience. Besides, in order to deal with multi-sensor collaboration, all previous human activity recognition models typically treated the representations from different sensor signal sources equally. However, distinct types of activities should adopt different fusion strategies. In this paper, a novel scheme is proposed. This scheme is used to train 2-bit Convolutional Neural Networks with weights and activations constrained to {-0.5,0,0.5}. It takes into account the correlation between different sensor signal sources and the activity types. This model, which we refer to as DFTerNet, aims at producing a more reliable inference and better trade-offs for practical applications. Our basic idea is to exploit quantization of weights and activations directly in pre-trained filter banks and adopt dynamic fusion strategies for different activity types. Experiments demonstrate that by using dynamic fusion strategy can exceed the baseline model performance by up to ~5% on activity recognition like OPPORTUNITY and PAMAP2 datasets. Using the quantization method proposed, we were able to achieve performances closer to that of full-precision counterpart. These results were also verified using the UniMiB-SHAR dataset. In addition, the proposed method can achieve ~9x acceleration on CPUs and ~11x memory saving.

Class-balanced siamese neural networks

This paper focuses on metric learning with Siamese Neural Networks (SNN). Without any prior, SNNs learn to compute a non-linear metric using only similarity and dissimilarity relationships between input data. Our SNN model proposes three contributions: a tuple-based architecture, an objective function with a norm regularisation and a polar sine-based angular reformulation for cosine dissimilarity learning. Applying our SNN model for Human Action Recognition (HAR) gives very competitive results using only one accelerometer or one motion capture point on the Multimodal Human Action Dataset (MHAD). Performances and properties of our proposals in terms of accuracy, convergence and complexity are assessed, with very favourable results. Additional experiments on the ”Challenge for Multimodal Mid-Air Gesture Recognition for Close Human Computer Interaction” Dataset (ChAirGest) confirm the competitive comparison of our proposals with state-of-the-arts models.

Latent feature learning for activity recognition using simple sensors in smart homes

Activity recognition is an important step towards monitoring and evaluating the functional health of an individual, and it potentially promotes human-centric ubiquitous applications in smart homes particularly for senior healthcare. The nature of human activity characterized by a high degree of complexity and uncertainty, however, poses a great challenge to the design of good feature representations and the optimization of classifiers towards building a robust model for human activity recognition. In this study, we propose to exploit deep learning techniques to automatically learn high-level features from the binary sensor data under the assumption that there exist discriminative latent patterns inherent in the simple low-level features. Specifically, we extract high-level features with a stacked autoencoder that has a deep and hierarchy architecture, and combine feature learning and classifier construction into a unified framework to obtain a jointly optimized activity recognizer. Besides, we investigate two different original feature representations of the sensor data for latent feature learning. To evaluate the performance of the proposed method, we conduct extensive experiments on three publicly available smart home datasets, and compare it with a range of shallow models in terms of time-slice accuracy and class accuracy. Experimental results show that our proposed model achieves better recognition rates and generalizes better across different original feature representations, indicating its applicability to the real-world activity recognition.

A Wireless Signal Denoising Model for Human Activity Recognition

Some pioneer WIFI signal based human activity recognition systems have been proposed. The common characteristic is to use the information of CSI(Channel State Information). Experimental results show that the extracted features of the PCA method are more obvious than that of the traditional denoising method. Even in a static environment, CSI values in Wifi signals fluctuate because WiFi devices are susceptible to surrounding electromagnetic noises. General purpose denoising methods, such as low-pass filters or mean filters, do not perform well in removing these impulse and burst noises. In this paper, we propose a method which use the low pass filter and principal component analysis simultaneously. Experimental results show that the extracted features of the PCA method are more obvious than that of the traditional methods.

3D human action recognition model based on image set and regularized multi-task leaning

With in-depth research of action recognition, the conceptions and algorithms of multi-view had been proposed and demonstrated its superiority by researchers, but these algorithms usually neglected the relatedness of multi-views. For the sake of overcome the shortage, this paper proposes a 3D human action recognition model based on image set and regularized multi-task learning. Specifically, we first extract dense trajectory feature for each camera, and then propose to construct the shared codebook by k-means for all cameras, after that, Bag-of-Word (BOW) weight scheme is employed to code dense trajectory feature by the shared codebook for each camera, respectively. Furthermore, we formulate the 3D human action recognition into regularized multi-task learning problem penalized by image set and relevant and irrelevant decomposition to discover the underlying relationship among different tasks and different views, and consequently boost the performances. Large scale experimental results on three public multi-view action3D datasets – IXMAX, UCLA and CVS-MV-RGBD-SINGLE, show that multi-task learning approach is very helpful for discovering the latent relationships among different tasks, which can significantly improve performance over the single-task learning method. Moreover, the image set regularized term, which is utilized to combine different views, can further improve the performance. In a word, the performance of our proposed method is comparable to the state-of-the-art methods.