Human Activity Prediction Research Articles

Human activity prediction by mapping grouplets to recurrent Self-Organizing Map

Human activity prediction is defined as inferring the high-level activity category with the observation of only a few action units. It is very meaningful for time-critical applications such as emergency surveillance. For efficient prediction, we represent the ongoing human activity by using body part movements and taking full advantage of inherent sequentiality, then find the best matching activity template by a proper aligning measurement.In streaming videos, dense spatio-temporal interest points (STIPs) are first extracted as low-level descriptors for their high detection efficiency. Then, sparse grouplets, i.e., clustered point groups, are located to represent body part movements, for which we propose a scale-adaptive mean shift method that can determine grouplet number and scale for each frame adaptively. To learn the sequentiality, located grouplets are successively mapped to Recurrent Self-Organizing Map (RSOM), which has been pre-trained to preserve the temporal topology of grouplet sequences. During this mapping, a growing RSOM trajectory, which represents the ongoing activity, is obtained. For the special structure of RSOM trajectory, a combination of dynamic time warping (DTW) distance and edit distance, called DTW-E distance, is designed for similarity measurement. Four activity datasets with different characteristics such as complex scenes and inter-class ambiguities serve for performance evaluation. Experimental results confirm that our method is very efficient for predicting human activity and yields better performance than state-of-the-art works.

人間行動の統計的モデリングのための遅れを伴う状況-行動間の因果関係パターン抽出手法

SUMMARY A modeling approach for human action is the focus of this paper. We design a human action model based on the stored data obtained during long-term monitoring of a person. This approach consists of the following two processes. First, several kinds of partial time series data that occur frequently are extracted from the stored data and taken to be human action patterns. Next, the extracted time series data are modeled based on a statistical modeling method such as the hidden Markov model. In this research, we focus on the extraction method for the time series data that occur frequently in the stored data. A person changes his actions according to changes in the situation around him. Moreover, it takes some time for him to perform his action after he recognizes the situation around himself. This time is called the delay time in this paper. A human action model that takes this delay time into consideration leads to greater accuracy in recognition and prediction of human actions based on that action. It is necessary to extract time series data for a situation and an action with the delay time as training data in order to generate the above human action model. In conventional methods, multidimensional time series data are used as the stored data without a distinction between the situation and action data. Also, some partial time series data that occur frequently are extracted from the stored data. Therefore, the delay time is not taken into account. In this paper, we propose a new extraction method for time series data that occur frequently with the time delay taken into consideration. In this method, sets of partial time series data that occur frequently with the delay time are extracted by evaluating the repeatability and similarity between the partial time series data sets with different occurrence frequencies. In the experiment, we extract the interaction motions that occur frequently between two subjects. The utility of the proposed method is examined based on the experimental results.

Unsupervised feature learning improves prediction of human brain activity in response to natural images.

Encoding and decoding in functional magnetic resonance imaging has recently emerged as an area of research to noninvasively characterize the relationship between stimulus features and human brain activity. To overcome the challenge of formalizing what stimulus features should modulate single voxel responses, we introduce a general approach for making directly testable predictions of single voxel responses to statistically adapted representations of ecologically valid stimuli. These representations are learned from unlabeled data without supervision. Our approach is validated using a parsimonious computational model of (i) how early visual cortical representations are adapted to statistical regularities in natural images and (ii) how populations of these representations are pooled by single voxels. This computational model is used to predict single voxel responses to natural images and identify natural images from stimulus-evoked multiple voxel responses. We show that statistically adapted low-level sparse and invariant representations of natural images better span the space of early visual cortical representations and can be more effectively exploited in stimulus identification than hand-designed Gabor wavelets. Our results demonstrate the potential of our approach to better probe unknown cortical representations.

Prediction of Human Activity by Discovering Temporal Sequence Patterns.

Early prediction of ongoing human activity has become more valuable in a large variety of time-critical applications. To build an effective representation for prediction, human activities can be characterized by a complex temporal composition of constituent simple actions and interacting objects. Different from early detection on short-duration simple actions, we propose a novel framework for long -duration complex activity prediction by discovering three key aspects of activity: Causality, Context-cue, and Predictability. The major contributions of our work include: (1) a general framework is proposed to systematically address the problem of complex activity prediction by mining temporal sequence patterns; (2) probabilistic suffix tree (PST) is introduced to model causal relationships between constituent actions, where both large and small order Markov dependencies between action units are captured; (3) the context-cue, especially interactive objects information, is modeled through sequential pattern mining (SPM), where a series of action and object co-occurrence are encoded as a complex symbolic sequence; (4) we also present a predictive accumulative function (PAF) to depict the predictability of each kind of activity. The effectiveness of our approach is evaluated on two experimental scenarios with two data sets for each: action-only prediction and context-aware prediction. Our method achieves superior performance for predicting global activity classes and local action units.

Reinforced learning systems based on merged and cumulative knowledge to predict human actions

This paper defines learning systems that contribute to the incremental building of knowledge to optimize the prediction of human actions. Two modes of knowledge building are proposed: merged approach and cumulative approach. In a merged learning system, a limited amount of knowledge is reinforced with new input data. In a cumulative learning system, knowledge is continuously acquired and reinforced by considering new input data as potential new knowledge. Both cumulative and merged approaches reinforce prior knowledge to predict subsequent output data, such as human actions. The structure of the proposed systems contains two main controllers: the Feedforward Action Controller (FAC), which pertains to the prediction process, and the Feedback Knowledge Controller (FKC), which manages the system knowledge. Two modes of use are proposed for these systems: a single mode that compares the prediction results of the merged and cumulative learning systems, and a joint mode that combines the prediction results of the merged and cumulative systems. The merged and cumulative FAC–FKC systems are validated using railway and car driving simulations to predict human actions and learn from them. Our results were satisfactory and were compared with those obtained by a learning system based on Self-Organizing Maps. The joint mode provides improved prediction by taking advantage of both merged and cumulative FAC–FKC systems.

Prediction of human actions: Expertise and task‐related effects on neural activation of the action observation network

The action observation network (AON) is supposed to play a crucial role when athletes anticipate the effect of others' actions in sports such as tennis. We used functional magnetic resonance imaging to explore whether motor expertise leads to a differential activation pattern within the AON during effect anticipation and whether spatial and motor anticipation tasks are associated with a differential activation pattern within the AON depending on participant expertise level. Expert (N=16) and novice (N=16) tennis players observed video clips depicting forehand strokes with the instruction to either indicate the predicted direction of ball flight (spatial anticipation) or to decide on an appropriate response to the observed action (motor anticipation). The experts performed better than novices on both tennis anticipation tasks, with the experts showing stronger neural activation in areas of the AON, namely, the superior parietal lobe, the intraparietal sulcus, the inferior frontal gyrus, and the cerebellum. When novices were contrasted with experts, motor anticipation resulted in stronger activation of the ventral premotor cortex, the supplementary motor area, and the superior parietal lobe than spatial anticipation task did. In experts, the comparison of motor and spatial anticipation revealed no increased activation. We suggest that the stronger activation of areas in the AON during the anticipation of action effects in experts reflects their use of the more fine-tuned motor representations they have acquired and improved during years of training. Furthermore, results suggest that the neural processing of different anticipation tasks depends on the expertise level.

A Desirable Engineer Providing Manifold Prospects Enhanced by Synergy Effect of Science and Liberal Arts

Quick globalization makes prediction of future human activity on production and economy more difficult. This is mainly because of increase in factors affecting relationship among different people with different way of life, culture, tradition and so on. To survive in such complicated world each engineer is desired to provide the knowledge of liberal arts as well as highly specialized knowledge. A synergy power generated by collaborative work of science-and liberal arts-minded way of thinking is a promising tool to unveil difficult world. This paper first describes degradation of liberal arts education during past two decades. Then, several movements for stopping that degradation are introduced in conjunction with the author′s overseas experiences. Finally, a necessity of bringing up well competitive desirable engineer through collaborative work by university and company is proposed.

A Nonsupervised Learning Framework of Human Behavior Patterns Based on Sequential Actions

In designing autonomous service systems such as assistive robots for the aged and the disabled, discovery and prediction of human actions are important and often crucial. Patterns of human behavior, however, involve ambiguity, uncertainty, complexity, and inconsistency caused by physical, logical, and emotional factors, and thus their modeling and recognition are known to be difficult. In this paper, a nonsupervised learning framework of human behavior patterns is suggested in consideration of human behavioral characteristics. Our approach consists of two steps. In the first step, a meaningful structure of data is discovered by using Agglomerative Iterative Bayesian Fuzzy Clustering (AIBFC) with a newly proposed cluster validity index. In the second step, the sequence of actions is learned on the basis of the structure discovered in the first step and by utilizing the proposed Fuzzy-state Q--learning (FSQL) process. These two learning steps are incorporated in an amalgamated framework, AIBFC-FSQL, which is capable of learning human behavior patterns in a nonsupervised manner and predicting subsequent human actions. Through a number of simulations with typical benchmark data sets, we show that the proposed learning method outperforms several well-known methods. We further conduct experiments with two challenging real-world databases to demonstrate its usefulness from a practical perspective.

Predictive modeling of human operators using parametric and neuro-fuzzy models by means of computer-based identification experiment

Driving a car and piloting an airplane are the most common examples for manual control of complicated processes. Human operators are known to be nonlinear, adaptive, time varying and intelligent controllers. In some cases, the human operator may or may not be well trained or an expert, showing different dynamics from operator to operator as in driving example. Therefore, it is very difficult to obtain mathematical models of human operators in a human-in-the-loop-manual control tasks. The goal of this research is to find a simple dynamic model for the prediction of the human operator actions in a manual control system. A computer-based experiment has been designed using the system identification theory to collect data from human operators. The autoregressive with exogenous inputs (ARX), as a parametric model and the adaptive-network-based fuzzy inference system (ANFIS), as an intelligent modeling approach that has the advantages of both neural networks and fuzzy logic, have been investigated and compared for simple and fast implementation to predict the response of human operators. ANFIS, having only 32 rules, provided much better prediction results than ARX model.

A Model of Human Actions by a Petri Net and Prediction of Human Acts. For Generation of Home Robots Movement based on Prediction of Human Actions.

A home robot (a robot used in the home) must recognize the intention of a person in the home, in order to support that person. It is important to construct intelligence with which a robot can predict the person's next act and help him or her if required. In this paper, a robust system which predicts vague human actions is proposed. Human actions can be modeled by a stochastic process in digital time and discrete states: this stochastic process is visualized using a Petri net. A series of human actions is a chain of states and acts. A human state is expressed as a place, and a human act as a transition. A stochastic act is described as a predicting transition, and transition probability to plural states is given. If the system makes a wrong prediction, the transition probability is renewed. As time progresses, the level of this intelligence increases, and the prediction will be accurate with high probalility.