Spatio-temporal Interest Points Research Articles

Human action tracking design of neural network algorithm based on GA-PSO in physical training

In order to solve human action recognition algorithm in traditional physical training, the problem of some kind of action recognition without universality is usually solved emphatically, and a recognition system aimed at human action tracking in physical training is proposed. Firstly, local feature description of human action in physical training based on self-similarity matrix is constructed by using generalized self-similarity concept changed along with time and local feature extraction methods of optical flow field of spatio-temporal interest point; secondly, a new neural network training algorithm of particle swarm optimization (PSO) with self-adaptive genetic operator is proposed. Through probability control, at the same time of using PSO algorithm to optimize neural network, selection, intersection, variation and other genetic operation are implemented for optional particles adaptively to realize promotion of algorithm performance; simulation experiment indicates that proposed scheme can obviously improve algorithm efficiency of human action recognition and recognition accuracy in physical training.

GA-STIP: Action Recognition in Multi-Channel Videos With Geometric Algebra Based Spatio-Temporal Interest Points

Spatiotemporal interest points (STIPs) are often extracted for action recognition in videos. But most of the current interest point detection methods are not suitable for multi-channel videos. In this paper, a new geometric algebra (GA)-based STIP detection method is proposed for action recognition in multi-channel videos. Specifically, we take the advantage of GA to represent multi-channel videos and develop a new method to obtain multi-scale space, and then GA-based difference of Gaussian is computed to detect STIPs. Experimental results show that GA-STIP method can achieve better performance for multi-channel video action recognition.

A Cuboid Bi-Level Log Operator for Action Classification

We propose a 3-D cuboid bi-level Laplacian-of-Gaussian (CBLoG) operator with high speed and invariant space–time scale for detecting abrupt changes of signals in videos. This 3-D CBLoG operator is applied in the detection of spatio-temporal interest points in videos. Then, we extract rich feature descriptors, including HOG, HOF, and MBH (motion boundary histogram) around the detected points, and use Fisher vector to encoding these descriptors and their combinations as action representation. In our experiments, we evaluated our recognition method on two standard data sets: KTH and UCF Sport action data sets. The best accuracy on the KTH data set achieves at 99.3%, and on the UCF Sports data set, it is 90.0%. Compared with the existing approaches, our method outperforms the state of the art tested on the KTH data set and achieves impressive performance on the UCF sports data set.

A Novel SURF Based on a Unified Model of Appearance and Motion-Variation

The speeded-up robust features (SURFs) algorithm is the best and most efficient local invariant feature algorithm for application to 2-D images and is widely applied in the fields of 2-D image processing and computer vision. Compared to 2-D images, a video has motion information in addition to its appearance information. Here, to make full use of the video appearance and motion information, we use geometric algebra as the mathematical calculation and analysis framework to obtain the embedded appearance and motion information on a local area of a video. In our proposed model of appearance and motion variation (UMAMV), we developed SURF feature detection and description algorithms operating on the spatio-temporal domain with video appearance and motion information. First of all, a model of appearance and motion variation, which contains video appearance and motion information in the framework of geometric algebra, is proposed. Then, based on this model, we propose a novel detection algorithm, theUMAMV-SURF detector, which mainly contains Hessian matrix construction, Hessian matrix determinant approximation calculation, and non-maximal suppression determination feature points as its key steps. Then, we introduce the UMAMV-SURF description algorithm, which mainly includes determining the dominant orientation of UMAMV-SURF feature points and generating the UMAMV-SURF feature descriptors. Finally, by experimenting with the Weizman and UCF101 datasets, the experimental results show that the proposed UMAMV-SURF algorithm can detect those SURF feature points which can have unique appearance information in the spatial domain and reflect motion change in the temporal domain. Moreover, it offers a higher accuracy than other spatio-temporal interest point algorithms in human behavior recognition of video footage.

Spatio-Temporal Scale Selection in Video Data

This work presents a theory and methodology for simultaneous detection of local spatial and temporal scales in video data. The underlying idea is that if we process video data by spatio-temporal receptive fields at multiple spatial and temporal scales, we would like to generate hypotheses about the spatial extent and the temporal duration of the underlying spatio-temporal image structures that gave rise to the feature responses. For two types of spatio-temporal scale-space representations, (i) a non-causal Gaussian spatio-temporal scale space for offline analysis of pre-recorded video sequences and (ii) a time-causal and time-recursive spatio-temporal scale space for online analysis of real-time video streams, we express sufficient conditions for spatio-temporal feature detectors in terms of spatio-temporal receptive fields to deliver scale-covariant and scale-invariant feature responses. We present an in-depth theoretical analysis of the scale selection properties of eight types of spatio-temporal interest point detectors in terms of either: (i)–(ii) the spatial Laplacian applied to the first- and second-order temporal derivatives, (iii)–(iv) the determinant of the spatial Hessian applied to the first- and second-order temporal derivatives, (v) the determinant of the spatio-temporal Hessian matrix, (vi) the spatio-temporal Laplacian and (vii)–(viii) the first- and second-order temporal derivatives of the determinant of the spatial Hessian matrix. It is shown that seven of these spatio-temporal feature detectors allow for provable scale covariance and scale invariance. Then, we describe a time-causal and time-recursive algorithm for detecting sparse spatio-temporal interest points from video streams and show that it leads to intuitively reasonable results. An experimental quantification of the accuracy of the spatio-temporal scale estimates and the amount of temporal delay obtained from these spatio-temporal interest point detectors is given, showing that: (i) the spatial and temporal scale selection properties predicted by the continuous theory are well preserved in the discrete implementation and (ii) the spatial Laplacian or the determinant of the spatial Hessian applied to the first- and second-order temporal derivatives leads to much shorter temporal delays in a time-causal implementation compared to the determinant of the spatio-temporal Hessian or the first- and second-order temporal derivatives of the determinant of the spatial Hessian matrix.

A unified model of appearance and motion of video and its application in STIP detection

Spatio-temporal interest points (STIPs) are local invariant features of a video and are both distinctive and descriptive and therefore can be applied in action recognition; however, most existing STIP detectors extend spatial descriptions by adding a temporal component for the appearance description, which separate spatio-temporal domain correlations in the spatio-temporal domain and only implicitly capture motion information. Therefore, by regarding the video as a 3-d structure, this research aims to develop a novel STIP detector which synthetically exploits appearance and motion information using Clifford algebra. This study firstly establishes a general Clifford algebra model for video and then builds the unified model of appearance and motion (UMAM) based thereon to synthetically analyse appearance and motion information. Subsequently, in the spirit of the well-known Harris 3-d detector, a UMAM-based spatio-temporal Harris corner detector (UMAM-Harris) for videos is developed. The experimental results indicate that the UMAM-Harris detector proposed in this study extracts the UMAM-Harris corners that contain distinctive features in the spatial domain and reflect substantial motion in the time domain, and it offers a better performance than the traditional STIP detection algorithms used in video action recognition.

Improving Bag-of-Visual-Words model using visual n-grams for human action classification

The Bag-of-Visual-Words model has emerged as an effective approach to represent local video features for human actions classification. However, one of the major challenges in this model is the generation of the visual vocabulary. In the case of human action recognition, losing spatial-temporal relationships is one of the important reasons that provokes the low descriptive power of classic visual words. In this work we propose a three-level approach to construct visual n-grams for human action classification. First, in order to reduce the number of non-descriptive words generated by K-means clustering of the spatio-temporal interest points, we propose to apply a variant of the classical Leader-Follower clustering algorithm to create an optimal vocabulary from a pre-established number of visual words. Second, with the aim of incorporating spatial and temporal constraints to the Bag-of-Visual-Words model, we exploit the spatio-temporal relationships between interest points to build a graph-based representation of the video. Frequent subgraphs are extracted for each action class and a visual vocabulary of n-grams is constructed from the labels (descriptors) of selected subgraphs. Finally, we build a histogram by using the frequency of each n-gram in the graph representing a video of human action. The proposed approach combines the representational power of graphs with the efficiency of the Bag-of-Visual-Words model. Extensive validation on five challenging human actions datasets demonstrates the effectiveness of the proposed model compared to state-of-the-art methods.

Moving object detection based on on-line block-robust principal component analysis decomposition

Robust principal component analysis (RPCA) decomposition is widely applied in moving object detection due to its ability in suppressing environmental noises while separating sparse foreground from low rank background. However, it may suffer from constant punishing parameters (resulting in confusion between foreground and background) and holistic processing of all input frames (leading to bad real-time performance). Improvements to these issues are studied in this paper. A block-RPCA decomposition approach was proposed to handle the confusion while separating foreground from background. Input frame was initially separated into blocks using three-frame difference. Then, punishing parameter of each block was computed by its motion saliency acquired based on selective spatio-temporal interesting points. Aiming to improve the real-time performance of the proposed method, an on-line solution to block-RPCA decomposition was utilized. Both qualitative and quantitative tests were implemented and the results indicate the superiority of our method to some state-of-the-art approaches in detection accuracy or real-time performance, or both of them.

Spatio-temporal feature using optical flow based distribution for violence detection

Violence detection in video surveillance is a challenging task in computer vision. In this paper, we propose a new feature that describes violence in video based on interest points, detected in both space and time domains, and optical flow information. The proposed feature estimates the bivariate distribution of the optical flow magnitude and orientation computed around spatio-temporal interest points (STIP). The estimation of the distribution is performed using the bivariate kernel density estimation method. Our proposed descriptor, named Distribution of Magnitude and Orientation of Local Interest Frame (DiMOLIF), is used to learn a SVM based binary classifier. Experiments are conducted on two well-known benchmark datasets for violence detection in both crowded and uncrowded scenes. Obtained results show that our proposed feature is efficient and outperforms the state of the art descriptors.

Fusion of appearance and motion-based sparse representations for multi-shot person re-identification

We present in this paper a multi-shot human re-identification system from video sequences based on interest points (IPs) matching. Our contribution is to take advantage of the complementary of person’s appearance and style of its movement that leads to a more robust description with respect to various complexity factors. The proposed contributions include person’s description and features matching. For person’s description, we propose to exploit a fusion strategy of two complementary features provided by appearance and motion description. We describe motion using spatiotemporal IPs, and use spatial IPs for describing the appearance. For feature matching, we use Sparse Representation (SR) as a local matching method between IPs. The fusion strategy is based on the weighted sum of matched IPs votes and then applying the rule of majority vote. This approach is evaluated on a large public dataset, PRID-2011. The experimental results show that our approach clearly outperforms current state-of-the-art.

Graph formulation of video activities for abnormal activity recognition

Abnormal activity recognition is a challenging task in surveillance videos. In this paper, we propose an approach for abnormal activity recognition based on graph formulation of video activities and graph kernel support vector machine. The interaction of the entities in a video is formulated as a graph of geometric relations among space–time interest points. The vertices of the graph are spatio-temporal interest points and an edge represents the relation between appearance and dynamics around the interest points. Once the activity is represented using a graph, then for classification of the activities into normal or abnormal classes, we use binary support vector machine with graph kernel. These graph kernels provide robustness to slight topological deformations in comparing two graphs, which may occur due to the presence of noise in data. We demonstrate the efficacy of the proposed method on the publicly available standard datasets viz. UCSDped1, UCSDped2 and UMN. Our experiments demonstrate high rate of recognition and outperform the state-of-the-art algorithms.

Survey of Spatio-Temporal Interest Point Detection Algorithms in Video

Recently, increasing attention has been paid to the detection of spatio-temporal interest points (STIPs), which has become a key technique and research focus in the field of computer vision. Its applications include human action recognition, video surveillance, video summarization, and content-based video retrieval. Amount of work has been done by many researchers in STIP detection. This paper presents a comprehensive review on STIP detection algorithms. We first propose the detailed introductions and analysis of the existing STIP detection algorithms. STIP detection algorithms are robust in detecting interest points for video in the spatio-temporal domain. Next, we summarize the existing challenges in the STIP detection for video, such as low time efficiency, poor robustness with respect to camera movement, illumination change, perspective occlusion, and background clutter. This paper also presents the application situations of STIP and discusses the potential development trends of STIP detection.

A novel local feature descriptor based on energy information for human activity recognition

In this paper we propose a novel local feature descriptor based on energy information for human activity recognition. Instead of detecting spatio-temporal interest points, we combine the kinetic energy, gesture potential energy of 3D skeleton joints and others as a feature matrix. The semantic features are obtained by the Bag of Word (BOW) based on k-means clustering. These features conform to not only kinematics and biology of human action, but also the natural visual saliency for action recognition. During the activity recognition, we first present a temporal segmentation method based on kinetic features of human skeleton to cut the long videos into the sub-action segments. Then the sub-action units are iteratively incorporated in the meaningful groups by considering similarity of feature information. Finally, SVM based on kernel function is used to carry out human activity recognition. The experimental results show that our approach outperforms several state-of-the-art algorithms based on our proposed low dimensional features of energy information.

Facial expression recognition based on spatio-temporal interest points for depth sequences

Of the numerous facial expression recognition methods previously proposed, most are based on texture frames or sequences. Recently, the development of depth sensors has raised new possibilities of dealing with 3D data. The proposed method extracts histograms of oriented gradient (HOG) and optical flow (HOF) for STIPs directly from depth sequences rather than involving registration/deformation techniques to find correspondence in 3D scan data. Mutual information score (MIS) and weighted matching score (WMS) are, respectively, calculated on the basis of naïve-Bayes mutual information maximization and constrained matching pairs. Finally, the MIS and WMS results are concatenated into feature vectors which are then fed into a support vector machine for facial expression classification. The proposed method is applied to the public BU-4DFE (Binghamton University 4D Facial Expression) database for six different facial expressions: anger, disgust, fear, happy, sad and surprise. Experimental results confirm that the proposed method is simple but effective.

Frontal gait recognition based on spatio-temporal interest points

This paper proposes a spatio-temporal interest point (STIP)-based human gait recognition method for frontal gait videos. The proposed method extracts histograms of oriented gradient (HOG) for STIPs directly from gait videos without involving a gait cycle segment, which is required by silhouette analysis as used by most gait recognition methods. Moreover, silhouette extraction may be affected by noise and carried objects leading to consequent recognition error. Matches of STIPs between two gait videos are then found in terms of HOG to measure the similarity of two videos so as to achieve the goal of gait recognition. The experimental results offer evidence that our method outperforms the existing methods on the Carnegie-Mellon University Motion of Body database and the Chinese Academy of Sciences Institute of Automation data-set B.

Learning semantic context feature-tree for action recognition via nearest neighbor fusion

The spatio-temporal context learnt by the traditional methods for action recognition lacks the semantic meanings and temporal relationships. In order to deal with the drawbacks, we propose a novel semantic context feature-tree model to model the video clip for efficient human action recognition. The proposed method enforces spatio-temporal interest points (STIPs) within an irregular spatio-temporal volume to construct a semantic trees-structured relationship by nearest neighbor fusion. Specifically, we firstly extract STIPs, and moreover, utilize super-pixels to segment the motion image obtained by STIP detection. The points, which fall into super-pixel, are viewed as spatial semantic co-occurring features to represent one body part. Secondly, by patch matching, the point sets which are temporal nearest neighbors are merged into a new node of the next layer to describe the context of one moving part. After matching and associating, the sets of the frame indexes are renewed to group for the next fusion process until the conditions of the recursive process do not satisfy. Using KTH, UCF-YouTube and HOHA action datasets for human action recognition, our representation based on the learnt tree-structured features enhances the discriminative power of action descriptor, and obtains promising results.

Two‐layer discriminative model for human activity recognition

Most of recent methods for action/activity recognition, usually based on static classifiers, have achieved improvements by integrating context of local interest point (IP) features such as spatiotemporal IPs by characterising their neighbourhood under different scales. In this study, the authors propose a new approach that explicitly models the sequential aspect of activities. First, a sliding window segmentation technique splits the video stream into overlapping short segments. Each window is characterised by a local bag of words of IPs encoded by motion information. A first-layer support vector machine provides for each window a vector of conditional class probabilities that summarises all discriminant information that is relevant for sequence recognition. The sequence of these stochastic vectors is then fed to a hidden conditional random field for inference at the sequence level. They also show how their approach can be naturally extended to the problem of conjoint segmentation and recognition of a sequence of action classes within a continuous video stream. They have tested their model on various human action and activity datasets and the obtained results compare favourably with current state of the art.

Supervised framework for automatic recognition and retrieval of interaction: a framework for classification and retrieving videos with similar human interactions

This study presents supervised framework for automatic recognition and retrieval of interactions (SAFARRIs), a supervised learning framework to recognise interactions such as pushing, punching, and hugging, between a pair of human performers in a video shot. The primary contribution of the study is to extend the vectors of locally aggregated descriptors (VLADs) as a compact and discriminative video encoding representation, to solve the complex class partitioning problem of recognising human interaction. An initial codebook is generated from the training set of video shots, by extracting feature descriptors around the spatiotemporal interest points computed across frames. A bag of action words is generated by encoding the first‐order statistics of the visual words using VLAD. Support vector machine classifiers (1 against all) are trained using these codebooks. The authors have verified SAFARRI's accuracy for classification and retrieval (query by example). SAFARRI is free from tracking or recognition of body parts and capable of identifying the region of interaction in video shots. It gives superior retrieval and classification performances over recently proposed methods, on two publicly available human interaction datasets.

Action Recognition Using Hierarchical STIP Saliency and Mixed Neighborhood Features

In video action recognition, the Dollar detector has been widely used to extract Spatio-Temporal Interest Points (STIPs) from action video sequence. It generates two kinds of information: STIP position and the respond value. However, in many cases, the detector respond, which measures the strength of local motion changes, is ignored. By utilizing such information, we propose to build a Hierarchical STIP Saliency (HSS) framework to provide different types of motion information. A novel local feature named Mixed Neighborhood Feature (MNF), which integrates the similarity and position relationship between local features, is put forward, and encoded by localityconstrained linear coding. Then, by partitioning video sequence along temporal direction, a group of sub-STVs are produced, and their corresponding descriptors are obtained with a max-pooling-on-absolute-value technique. In classification stage, Locality-constrained Group Sparse Representation (LGSR) is adopted as classifier to utilize the intrinsic group information of these sub-STV features. The experiments on the KTH and UCF Sports datasets show that in contrast to the classical recognition systems published recently, our recognition system based on the HSS and MNF achieves good performance.

Action recognition via spatio-temporal local features: A comprehensive study

Local methods based on spatio-temporal interest points (STIPs) have shown their effectiveness for human action recognition. The bag-of-words (BoW) model has been widely used and dominated in this field. Recently, a large number of techniques based on local features including improved variants of the BoW model, sparse coding (SC), Fisher kernels (FK), vector of locally aggregated descriptors (VLAD) as well as the naive Bayes nearest neighbor (NBNN) classifier have been proposed and developed for visual recognition. However, some of them are proposed in the image domain and have not yet been applied to the video domain and it is still unclear how effectively these techniques would perform on action recognition. In this paper, we provide a comprehensive study on these local methods for human action recognition. We implement these techniques and conduct comparison under unified experimental settings on three widely used benchmarks, i.e., the KTH, UCF-YouTube and HMDB51 datasets. We discuss insightfully the findings from the experimental results and draw useful conclusions, which are expected to guide practical applications and future work for the action recognition community.