Spatio-temporal Interest Point Detector Research Articles

Spatiotemporal interest point detector exploiting appearance and motion-variation information

As a local invariant feature of videos, the spatiotemporal interest point (STIP) has been widely used in computer vision and pattern recognition. However, existing STIP detectors are generally extended from detection algorithms constructed for local invariant features of two-dimensional images, which does not explicitly exploit the motion information inherent in the temporal domain of videos, thus weakening the performance of existing STIP detectors in a video context. To remedy this, we aim to develop an STIP detector that uniformly captures appearance and motion information for video, thus yielding substantial performance improvement. Specifically, under the framework of geometric algebra, we first develop a spatiotemporal unified model of appearance and motion-variation information (UMAMV), and then a UMAMV-based scale space of the spatiotemporal domain is proposed to synthetically analyze appearance information and motion information in a video. Based on this model, we propose an STIP feature of UMAMV-SIFT that embraces both appearance and motion variation information of the videos. Three datasets with different sizes are utilized to evaluate the proposed model and the STIP detector. We present experimental results to show that the UMAMV-SIFT achieves state-of-the-art performance and is particularly effective when dataset is small.

GA-STIP: Action Recognition in Multi-Channel Videos With Geometric Algebra Based Spatio-Temporal Interest Points

Spatiotemporal interest points (STIPs) are often extracted for action recognition in videos. But most of the current interest point detection methods are not suitable for multi-channel videos. In this paper, a new geometric algebra (GA)-based STIP detection method is proposed for action recognition in multi-channel videos. Specifically, we take the advantage of GA to represent multi-channel videos and develop a new method to obtain multi-scale space, and then GA-based difference of Gaussian is computed to detect STIPs. Experimental results show that GA-STIP method can achieve better performance for multi-channel video action recognition.

Spatio-Temporal Scale Selection in Video Data

This work presents a theory and methodology for simultaneous detection of local spatial and temporal scales in video data. The underlying idea is that if we process video data by spatio-temporal receptive fields at multiple spatial and temporal scales, we would like to generate hypotheses about the spatial extent and the temporal duration of the underlying spatio-temporal image structures that gave rise to the feature responses. For two types of spatio-temporal scale-space representations, (i) a non-causal Gaussian spatio-temporal scale space for offline analysis of pre-recorded video sequences and (ii) a time-causal and time-recursive spatio-temporal scale space for online analysis of real-time video streams, we express sufficient conditions for spatio-temporal feature detectors in terms of spatio-temporal receptive fields to deliver scale-covariant and scale-invariant feature responses. We present an in-depth theoretical analysis of the scale selection properties of eight types of spatio-temporal interest point detectors in terms of either: (i)–(ii) the spatial Laplacian applied to the first- and second-order temporal derivatives, (iii)–(iv) the determinant of the spatial Hessian applied to the first- and second-order temporal derivatives, (v) the determinant of the spatio-temporal Hessian matrix, (vi) the spatio-temporal Laplacian and (vii)–(viii) the first- and second-order temporal derivatives of the determinant of the spatial Hessian matrix. It is shown that seven of these spatio-temporal feature detectors allow for provable scale covariance and scale invariance. Then, we describe a time-causal and time-recursive algorithm for detecting sparse spatio-temporal interest points from video streams and show that it leads to intuitively reasonable results. An experimental quantification of the accuracy of the spatio-temporal scale estimates and the amount of temporal delay obtained from these spatio-temporal interest point detectors is given, showing that: (i) the spatial and temporal scale selection properties predicted by the continuous theory are well preserved in the discrete implementation and (ii) the spatial Laplacian or the determinant of the spatial Hessian applied to the first- and second-order temporal derivatives leads to much shorter temporal delays in a time-causal implementation compared to the determinant of the spatio-temporal Hessian or the first- and second-order temporal derivatives of the determinant of the spatial Hessian matrix.

A unified model of appearance and motion of video and its application in STIP detection

Spatio-temporal interest points (STIPs) are local invariant features of a video and are both distinctive and descriptive and therefore can be applied in action recognition; however, most existing STIP detectors extend spatial descriptions by adding a temporal component for the appearance description, which separate spatio-temporal domain correlations in the spatio-temporal domain and only implicitly capture motion information. Therefore, by regarding the video as a 3-d structure, this research aims to develop a novel STIP detector which synthetically exploits appearance and motion information using Clifford algebra. This study firstly establishes a general Clifford algebra model for video and then builds the unified model of appearance and motion (UMAM) based thereon to synthetically analyse appearance and motion information. Subsequently, in the spirit of the well-known Harris 3-d detector, a UMAM-based spatio-temporal Harris corner detector (UMAM-Harris) for videos is developed. The experimental results indicate that the UMAM-Harris detector proposed in this study extracts the UMAM-Harris corners that contain distinctive features in the spatial domain and reflect substantial motion in the time domain, and it offers a better performance than the traditional STIP detection algorithms used in video action recognition.

Survey of Spatio-Temporal Interest Point Detection Algorithms in Video

Recently, increasing attention has been paid to the detection of spatio-temporal interest points (STIPs), which has become a key technique and research focus in the field of computer vision. Its applications include human action recognition, video surveillance, video summarization, and content-based video retrieval. Amount of work has been done by many researchers in STIP detection. This paper presents a comprehensive review on STIP detection algorithms. We first propose the detailed introductions and analysis of the existing STIP detection algorithms. STIP detection algorithms are robust in detecting interest points for video in the spatio-temporal domain. Next, we summarize the existing challenges in the STIP detection for video, such as low time efficiency, poor robustness with respect to camera movement, illumination change, perspective occlusion, and background clutter. This paper also presents the application situations of STIP and discusses the potential development trends of STIP detection.

Learning semantic context feature-tree for action recognition via nearest neighbor fusion

The spatio-temporal context learnt by the traditional methods for action recognition lacks the semantic meanings and temporal relationships. In order to deal with the drawbacks, we propose a novel semantic context feature-tree model to model the video clip for efficient human action recognition. The proposed method enforces spatio-temporal interest points (STIPs) within an irregular spatio-temporal volume to construct a semantic trees-structured relationship by nearest neighbor fusion. Specifically, we firstly extract STIPs, and moreover, utilize super-pixels to segment the motion image obtained by STIP detection. The points, which fall into super-pixel, are viewed as spatial semantic co-occurring features to represent one body part. Secondly, by patch matching, the point sets which are temporal nearest neighbors are merged into a new node of the next layer to describe the context of one moving part. After matching and associating, the sets of the frame indexes are renewed to group for the next fusion process until the conditions of the recursive process do not satisfy. Using KTH, UCF-YouTube and HOHA action datasets for human action recognition, our representation based on the learnt tree-structured features enhances the discriminative power of action descriptor, and obtains promising results.

Action Recognition in Realistic Scenes via Local Spatio-temporal Representation

In this paper, we consider the problem of human action recognition in realistic scenes. Mainstream methods adopt local spatio-temporal patterns matching strategy that detect STIPs (Spatio-Temporal Interest Points) and compute features from raw video frames, and then classify the features into some predefined action categories. However, these researches have made certain assumptions seldom hold in the real-world environment such as small scale, without viewpoint changes and background motion. To obtain more effective detection and description for human action videos of complex scenes, we improve the performance of STIPs detection in realistic scenes by filtering out noises resulted from low resolution or cluttered background. In addition, we propose a new feature description method that uses 3D local self-similarities to compensate for motion estimation error due to camera motion. BoW (Bag-of-Words) model is applied to learn a codebook in training stage, and a nonlinear SVM classifier is used for action recognition finally. Experimental results show that our approach achieve superior performance on more challenge dataset (Youtube) in comparison to baseline method.

Automatic Definition of Regions of Interest on Renal Scintigraphic Images

Renography is a radionuclide technique introduced in medicine since the early sixties in the daily practice. It allows measuring the function of each kidney separately in case of unilateral or bilateral uro-nephrological disease. A new system for analysis of Renography is presented in this paper. Our system aims to show a fully automated method for drawing the renal regions of interest based on a multi-agent system that incorporates spatio-temporal interest points detection on scintigraphic images by using the HOG3D descriptor to initialize agents. We used both types of agents, namely supervisor agents and explorator agents, they communicate among themselves and they inspire in their behavior from the Fast Marching method. Our system was tested on many real patients’ cases and we have obtained encouraging results. Clinical validation showed a constant agreement between the approach presented and manual segmentation by professionals in the nuclear medicine.

Large scale continuous visual event recognition using max-margin Hough transformation framework

In this paper we propose a novel method for continuous visual event recognition (CVER) on a large scale video dataset using max-margin Hough transformation framework. Due to high scalability, diverse real environmental state and wide scene variability direct application of action recognition/detection methods such as spatio-temporal interest point (STIP)-local feature based technique, on the whole dataset is practically infeasible. To address this problem, we apply a motion region extraction technique which is based on motion segmentation and region clustering to identify possible candidate “event of interest” as a preprocessing step. On these candidate regions a STIP detector is applied and local motion features are computed. For activity representation we use generalized Hough transform framework where each feature point casts a weighted vote for possible activity class centre. A max-margin frame work is applied to learn the feature codebook weight. For activity detection, peaks in the Hough voting space are taken into account and initial event hypothesis is generated using the spatio-temporal information of the participating STIPs. For event recognition a verification Support Vector Machine is used. An extensive evaluation on benchmark large scale video surveillance dataset (VIRAT) and as well on a small scale benchmark dataset (MSR) shows that the proposed method is applicable on a wide range of continuous visual event recognition applications having extremely challenging conditions.

Selective spatio-temporal interest points

Recent progress in the field of human action recognition points towards the use of Spatio-Temporal Interest Points (STIPs) for local descriptor-based recognition strategies. In this paper, we present a novel approach for robust and selective STIP detection, by applying surround suppression combined with local and temporal constraints. This new method is significantly different from existing STIP detection techniques and improves the performance by detecting more repeatable, stable and distinctive STIPs for human actors, while suppressing unwanted background STIPs. For action representation we use a bag-of-video words (BoV) model of local N-jet features to build a vocabulary of visual-words. To this end, we introduce a novel vocabulary building strategy by combining spatial pyramid and vocabulary compression techniques, resulting in improved performance and efficiency. Action class specific Support Vector Machine (SVM) classifiers are trained for categorization of human actions. A comprehensive set of experiments on popular benchmark datasets (KTH and Weizmann), more challenging datasets of complex scenes with background clutter and camera motion (CVC and CMU), movie and YouTube video clips (Hollywood 2 and YouTube), and complex scenes with multiple actors (MSR I and Multi-KTH), validates our approach and show state-of-the-art performance. Due to the unavailability of ground truth action annotation data for the Multi-KTH dataset, we introduce an actor specific spatio-temporal clustering of STIPs to address the problem of automatic action annotation of multiple simultaneous actors. Additionally, we perform cross-data action recognition by training on source datasets (KTH and Weizmann) and testing on completely different and more challenging target datasets (CVC, CMU, MSR I and Multi-KTH). This documents the robustness of our proposed approach in the realistic scenario, using separate training and test datasets, which in general has been a shortcoming in the performance evaluation of human action recognition techniques.