UT-Interaction Dataset Research Articles

On an algorithm for human action recognition

Human action recognition which needs video processing in real time, requires large memory size and execution time. This work proposes a local maxima of difference image (LMDI) based interest point detection technique, random projection tree with overlapping split and modified voting score for human action recognition. In LMDI based interest point detection method, difference images are obtained using consecutive frame differencing technique and next, 3D peak detection is applied on the bunch of calculated difference images. Histogram of oriented gradients and histogram of optical flow as local features are extracted by defining a block of size 16 × 16 around each of the interest point. These local features are then indexed by random projection trees. Overlapping split is used during tree structuring to reduce failure probability. Hough voting technique is applied on testing video to compute highest similarity matching score with individual training classes. In addition to Hough voting score, the number of matched interest points of a single query video with each training class, is considered for recognition. The proposed method is evaluated on segmented UT-interaction dataset, J-HMDB dataset and UCF101 dataset. The experimental results indicate that the proposed technique provides better performance compared to earlier reported techniques.

Local Temporal Block Difference Pattern for Action Recognition in Surveillance Videos using Tree Based Classifiers

Intelligent video classification and prediction is a fundamental step towards effective retrieval system. Huge volume of video is available for navigation today and managing such video and prediction of the activity before its completion gains importance in video surveillance,human computer recognition, gesture recognition etc., An eminent Local Temporal Block Difference Pattern (LTBDP) is introducedwhich enable efficient feature extraction that could be given to Tree Classifiers like Random Forest and REPTree for further prediction.The proposed pattern has been evaluated on UT-interaction dataset which enable researchers to predict ongoing human actions in an efficient manner. Experimental results using LTBDP in Random Forest and REPTree classifier gives 85.6% and 66.45% accuracy respectively.

Leveraging Structural Context Models and Ranking Score Fusion for Human Interaction Prediction

Predicting an interaction before it is fully executed is very important in applications such as human-robot interaction and video surveillance. In a two-human interaction scenario, there often contextual dependency structure between the global interaction context of the two humans and the local context of the different body parts of each human. In this paper, we propose to learn the structure of the interaction contexts, and combine it with the spatial and temporal information of a video sequence for a better prediction of the interaction class. The structural models, including the spatial and the temporal models, are learned with Long Short Term Memory (LSTM) networks to capture the dependency of the global and local contexts of each RGB frame and each optical flow image, respectively. LSTM networks are also capable of detecting the key information from the global and local interaction contexts. Moreover, to effectively combine the structural models with the spatial and temporal models for interaction prediction, a ranking score fusion method is also introduced to automatically compute the optimal weight of each model for score fusion. Experimental results on the BIT Interaction and the UT-Interaction datasets clearly demonstrate the benefits of the proposed method.

Human interaction recognition based on sparse representation of feature covariance matrices

A new method for interaction recognition based on sparse representation of feature covariance matrices was presented. Firstly, the dense trajectories (DT) extracted from the video were clustered into different groups to eliminate the irrelevant trajectories, which could greatly reduce the noise influence on feature extraction. Then, the trajectory tunnels were characterized by means of feature covariance matrices. In this way, the discriminative descriptors could be extracted, which was also an effective solution to the problem that the description of the feature second-order statistics is insufficient. After that, an over-complete dictionary was learned with the descriptors and all the descriptors were encoded using sparse coding (SC). Classification was achieved using multiple instance learning (MIL), which was more suitable for complex environments. The proposed method was tested and evaluated on the WEB Interaction dataset and the UT interaction dataset. The experimental results demonstrated the superior efficiency.

Efficient extraction of spatial relations for extended objects vis-à-vis human activity recognition in video

Human activity recognition (HAR) deals with recognition of activities or interactions that include humans within a video. Entities occurring in a video frame can be abstracted in variety of ways, ranging from the detailed silhouette of the entity to the very basic axis-aligned minimum bounding rectangles (MBR). On one end of the spectrum, using a detailed silhouette is not only demanding in terms of storage and computational resources but is also easily susceptible to noise. On the other end of the spectrum, MBRs require less storage, computation and abstracts out noise and video specific details. However, for abstraction of human bodies in a video an MBR does not offer an adequate solution because in addition to abstracting away noise, it also abstracts out important details such as the posture of the human body. For a more precise description, which offers a reasonable tradeoff between efficiency and noise elimination, a human body can be abstracted using a set of MBRs corresponding to different body parts. However, for a representation of activities as relations between interacting objects, a simplistic approximation assuming each MBR to be an independent entity leads to computation of redundant relations. In this paper, we explore a representation schema for interaction between entities that are considered as sets of rectangles, also referred to as extended objects. We further show that, given the representation schema, a simple recursive algorithm can be used to opportunistically extract topological, directional and distance information in O(n l o g n) time. We evaluate our representation schema for HAR on the Mind’s Eye dataset ( http://www.visint.org ), the UT-Interaction (Ryoo and Aggarwal 2010) dataset and the SBU Kinect Interaction dataset (Yun et al. 2012).

A New Framework of Human Interaction Recognition Based on Multiple Stage Probability Fusion

Visual-based human interactive behavior recognition is a challenging research topic in computer vision. There exist some important problems in the current interaction recognition algorithms, such as very complex feature representation and inaccurate feature extraction induced by wrong human body segmentation. In order to solve these problems, a novel human interaction recognition method based on multiple stage probability fusion is proposed in this paper. According to the human body’s contact in interaction as a cut-off point, the process of the interaction can be divided into three stages: start stage, execution stage and end stage. Two persons’ motions are respectively extracted and recognizes in the start stage and the finish stage when there is no contact between those persons. The two persons’ motion is extracted as a whole and recognized in the execution stage. In the recognition process, the final recognition results are obtained by the weighted fusing these probabilities in different stages. The proposed method not only simplifies the extraction and representation of features, but also avoids the wrong feature extraction caused by occlusion. Experiment results on the UT-interaction dataset demonstrated that the proposed method results in a better performance than other recent interaction recognition methods.

A Study on Recognition of Two-Person Interactions Based on Concatenated History Images

Computer vision’s demand in applications like visual surveillance, video retrieval and human-computer interaction are increasing day by day. Human action recognition is one such vital research area under computer vision. Many datasets dedicated to human action recognition have been created and are accompanied with equally large number of techniques of recognition in recent years. In this paper, we have proposed a technique to recognize two-person interactions based on a benchmark dataset from University of Texas, Austin called UT interaction Dataset. The task is challenging due to variations in motion, dynamic background, recording settings and inter-personal differences. Our technique includes concatenated Motion History Images (cMHI). The produced cMHI templates are incorporated with Local Binary Pattern (LBP) and Histogram of Oriented Gradient (HOG) to develop smart feature vector for each class. Finally, Support Vector Machine (SVM) is exploited to classify various actions.

Anticipating Human Activities from Surveillance Videos

Human action recognition from the unconstrained surveillance videos and anticipating human action from onset of video is a challenging task. This work focuses on the study of action recognition, action classification followed by action anticipation. We have used the UT Interaction data set containing interactive videos with six types of activities, the system developed classifies three interactive actions punching, pushing and kicking with accuracy. Anticipating action is a probabilistic process and the all possible outcomes are predicted during anticipation. The framework developed can take any interactive video as input from the web camera or any other camera connected to the laptop and will classify the three interactive actions as said above; similarly when the onset of video is given it produces the probable predictions. Action is modeled as a sequence of changes in Spatio Temporal features and histogram of the gradients helps in identifying the changes. The motion boundary descriptors, histogram of oriented gradients and optical flow features are extracted and SVM algorithm is used for classification. Bag of words approach is used for anticipation. Prediction methodologies are highly needed in surveillance environments for efficient monitoring and management.

Interactive activity recognition using pose-based spatio–temporal relation features and four-level Pachinko Allocation Model

In this paper, we go beyond the problem of recognizing video-based human interactive activities. We propose a novel approach that permits to deeply understand complex person-person activities based on the knowledge coming from human pose analysis. The joint coordinates of interactive objects are first located by an efficient human pose estimation algorithm. The relation features consisting of the intra and inter-person features of joint distance and angle, are suggested to use for describing the relationships between body components of the individual persons and the interacting two participants in the spatio-temporal dimension. These features are then provided to the codebook construction process, in which two types of codeword are generated corresponding to distance and angle features. In order to explain the relationships between poses, a flexible hierarchical topic model constructed by four layers is proposed using the Pachinko Allocation Model. The model is able to represent the full correlation between the relation features of body components as codewords, the interactive poselets as subtopics, and the interactive activities as super topics. Discrimination of complex activities presenting similar postures is further obtained by the proposed model. We validate our interaction recognition method on two practical data sets, the BIT-Interaction data set and the UT-Interaction data set. The experimental results demonstrate that the proposed approach outperforms recent interaction recognition approaches in terms of recognition accuracy.

Human action classification using adaptive key frame interval for feature extraction

Human action classification based on the adaptive key frame interval (AKFI) feature extraction is presented. Since human movement periods are different, the action intervals that contain the intensive and compact motion information are considered in this work. We specify AKFI by analyzing an amount of motion through time. The key frame is defined to be the local minimum interframe motion, which is computed by using frame differencing between consecutive frames. Once key frames are detected, the features within a segmented period are encoded by adaptive motion history image and key pose history image. The action representation consists of the local orientation histogram of the features during AKFI. The experimental results on Weizmann dataset, KTH dataset, and UT Interaction dataset demonstrate that the features can effectively classify action and can classify irregular cases of walking compared to other well-known algorithms.

Human Interaction Recognition Using Hierarchical Invariant Features

Human interaction recognition has been widely studied because it has great scientific importance and many potential practical applications. However, recognizing human interactions is also very challenging especially in realistic environments where the background is dynamic or has varying lighting conditions. Most existing methods rely on either spatio-temporal local features (i.e. SIFT) or human poses, or human joints to model human interactions. As a result, they are not fully unsupervised processes because they require either hand-designed features or human detection results. Motivated by the recent success of deep learning networks, we investigate a three-layer convolutional network which uses the Independent Subspace Analysis (ISA) algorithm to learn hierarchical invariant features from videos. Using the invariant features learned by the ISA, we build a bag-of-features (BOF) model to recognize human interactions. We also evaluate the performance of our approach and the effectiveness of hierarchical invariant features on video sequences of the UT-Interaction dataset which contain both interacting persons and irrelevant pedestrians in the scenes. The dataset imposes several challenging factors including moving backgrounds, clutter scenes, scales and camera jitters. Experimental results show that our three-layer convolutional ISA network is able to learn features which are effective to represent complex activities such as human interactions in realistic environments.

From constrained to unconstrained datasets: an evaluation of local action descriptors and fusion strategies for interaction recognition

As an important task in computer vision, the interaction recognition has attracted extensive attention due to its widely potential applications. The existing methods mainly focus on the interaction recognition problem on constrained datasets with few variations of scenes, viewpoints, background clutter for the experimental purpose. The performance of the recently proposed methods on the available constrained dataset almost approaches to saturation, which is not adaptive to further evaluate the robustness of new methods. In this paper, we introduce a new unconstrained dataset, called WEB-interaction, collected from the Internet. Our WEB-interaction more represents realistic scenes and has much more challenges than existing datasets. Besides, we evaluate the state-of-the-art pipeline of interaction recognition on both WEB-interaction and UT-interaction datasets. The evaluation results reveal that MBHx and MBHy of Motion Boundary Histogram (MBH) are important feature descriptors for interaction recognition and MBHx has relatively dominative information. For fusion strategy, the late fusion benefits more to performance than early fusion. Filming condition effects are also evaluated on WEB-interaction dataset. In addition, the best average precision(AP) result of different features on our WEB-interaction dataset is 44.2 % and the mean is around 38 %. Compare to the UT-interaction dataset, our dataset has bigger improvement space, which is more significant to promote new methods.

Recognizing human interactions by genetic algorithm-based random forest spatio-temporal correlation

Recognizing human interactions is a more challenging task than recognizing single person activities and has attracted much attention of the computer vision community. This paper proposes an innovative and effective way to recognize human interactions, which incorporates the advantages of both global motion context (MC) feature and spatio-temporal (S-T) correlation of local spatio-temporal interest point feature. The MC feature is used to train a random forest where genetic algorithm (GA) is applied to the training phase to achieve a good compromise between reliability and efficiency. Besides, we propose S-T correlation-based match, where MC's structure and Needleman---Wunsch algorithm are used to calculate the spatial and temporal correlation score of two videos, respectively. Experiments on the UT-Interaction dataset show that our approaches outperform other prevalent machine learning methods, and that the combination of GA search-based random forest and S-T correlation achieves the state-of-the-art performance.

Interactive Phrases: Semantic Descriptions for Human Interaction Recognition.

This paper addresses the problem of recognizing human interactions from videos. We propose a novel approach that recognizes human interactions by the learned high-level descriptions, interactive phrases. Interactive phrases describe motion relationships between interacting people. These phrases naturally exploit human knowledge and allow us to construct a more descriptive model for recognizing human interactions. We propose a discriminative model to encode interactive phrases based on the latent SVM formulation. Interactive phrases are treated as latent variables and are used as mid-level features. To complement manually specified interactive phrases, we also discover data-driven phrases from data in order to find potentially useful and discriminative phrases for differentiating human interactions. An information-theoretic approach is employed to learn the data-driven phrases. The interdependencies between interactive phrases are explicitly captured in the model to deal with motion ambiguity and partial occlusion in the interactions. We evaluate our method on the BIT-Interaction data set, UT-Interaction data set, and Collective Activity data set. Experimental results show that our approach achieves superior performance over previous approaches.

Realistic human action recognition by Fast HOG3D and self-organization feature map

Nowadays, local features are very popular in vision-based human action recognition, especially in "wild" or unconstrained videos. This paper proposes a novel framework that combines Fast HOG3D and self-organization feature map (SOM) network for action recognition from unconstrained videos, bypassing the demanding preprocessing such as human detection, tracking or contour extraction. The contributions of our work not only lie in creating a more compact and computational effective local feature descriptor than original HOG3D, but also lie in first successfully applying SOM to realistic action recognition task and studying its training parameters' influence. We mainly test our approach on the UCF-YouTube dataset with 11 realistic sport actions, achieving promising results that outperform local feature-based support vector machine and are comparable with bag-of-words. Experiments are also carried out on KTH and UT-Interaction datasets for comparison. Results on all the three datasets confirm that our work has comparable, if not better, performance comparing with state-of-the-art.

A motion-based approach to detect persons in low-resolution video

The paper proposes a motion-based technique to detect persons in a low-resolution video, where the persons look like tiny blobs. The tiny blob-like appearance of the persons are due to camera position which is at a distance from the person(s). The proposed technique uses integral matrix based different spatial and temporal features. Gradient weighted optical flow (GWOF) is calculated for each frame of the video clip to minimize background noise. Spatial filters are used to extract motion features from the GWOF based integral matrices. The combination of image gradient and GWOF features extracts static and moving persons present in the video. The AdaBoost learning technique is used for training. The training is performed using features derived from the positive samples of bounding boxes in a video frame containing a person and negative samples with bounding boxes without a person. The proposed technique is applied on benchmark Tower dataset, UT-Interaction dataset and PETS 2007 dataset. We have obtained approximately 2 to 10 % improvement in the performance compared to the states-of-the-art.

SOFT-ASSIGNMENT RANDOM-FOREST WITH AN APPLICATION TO DISCRIMINATIVE REPRESENTATION OF HUMAN ACTIONS IN VIDEOS

The bag-of-features model is a distinctive and robust approach to detect human actions in videos. The discriminative power of this model relies heavily on the quantization of the video features into visual words. The quantization determines how well the visual words describe the human action. Random forests have proven to efficiently transform the features into distinctive visual words. A major disadvantage of the random forest is that it makes binary decisions on the feature values, and thus not taking into account uncertainties of the values. We propose a soft-assignment random forest, which is a generalization of the random forest, by substitution of the binary decisions inside the tree nodes by a sigmoid function. The slope of the sigmoid models the degree of uncertainty about a feature's value. The results demonstrate that the soft-assignment random forest improves significantly the action detection accuracy compared to the original random forest. The human actions that are hard to detect — because they involve interactions with or manipulations of some (typically small) item — are structurally improved. Most prominent improvements are reported for a person handing, throwing, dropping, hauling, taking, closing or opening some item. Improvements are achieved for the state-of-the-art on the IXMAS and UT-Interaction datasets by using the soft-assignment random forest.

Selection of negative samples and two-stage combination of multiple features for action detection in thousands of videos

In this paper, a system is presented that can detect 48 human actions in realistic videos, ranging from simple actions such as `walk' to complex actions such as `exchange'. We propose a method that gives a major contribution in performance. The reason for this major improvement is related to a different approach on three themes: sample selection, two-stage classification, and the combination of multiple features. First, we show that the sampling can be improved by smart selection of the negatives. Second, we show that exploiting all 48 actions' posteriors by two-stage classification greatly improves its detection. Third, we show how low-level motion and high-level object features should be combined. These three yield a performance improvement of a factor 2.37 for human action detection in the visint.org test set of 1,294 realistic videos. In addition, we demonstrate that selective sampling and the two-stage setup improve on standard bag-of-feature methods on the UT-interaction dataset, and our method outperforms state-of-the-art for the IXMAS dataset.

Spatio-temporal layout of human actions for improved bag-of-words action detection

We investigate how human action recognition can be improved by considering spatio-temporal layout of actions. From literature, we adopt a pipeline consisting of STIP features, a random forest to quantize the features into histograms, and an SVM classifier. Our goal is to detect 48 human actions, ranging from simple actions such as walk to complex actions such as exchange. Our contribution to improve the performance of this pipeline by exploiting a novel spatio-temporal layout of the 48 actions. Here each STIP feature does not in the video contributes to the histogram bins by a unity value, but rather by a weight given by its spatio-temporal probability. We propose 6 configurations of spatio-temporal layout, where the varied parameters are the coordinate system and the modeling of the action and its context. Our model of layout does not change any other parameter of the pipeline, it requires no re-learning of the random forest, yields a limited increase of the size of its resulting representation by only a factor two, and at a minimal additional computational cost of only a handful of operations per feature. Extensive experiments show that the layout is demonstrated to be distinctive of actions that involve trajectories, (dis)appearance, kinematics, and interactions. The visualization of each action’s layout illustrates that our approach is indeed able to model spatio-temporal patterns of each action. Each layout is experimentally shown to be optimal for a specific set of actions. Generally, the context has more effect than the choice of coordinate system. The most impressive improvements are achieved for complex actions involving items. For 43 out of 48 human actions, the performance is better or equal when spatio-temporal layout is included. In addition, we show our method outperforms state-of-the-art for the IXMAS and UT-Interaction datasets.