Space-time Interest Points Research Articles

Multi-dimensional CNN Based Feature Extraction with Feature Fusion and SVM for Human Activity Recognition in Surveillance Videos

Background/Objectives: The accurate recognition of human activities from video sequences is very challenging due to low resolution, cluttered background, partial occlusion, and different viewpoints. Machine learning (ML) based automated HAR from surveillance videos is required with the fusion of various feature extraction techniques. Methods: In this paper, SVM with feature fusion is utilized for automatic recognition from surveillance videos. A Histogram of Oriented Gradient (HOG) is used to segment the frame to differentiate humans from other objects or background noise in the input video frames. The multi-feature extraction can be accomplished in terms of Gabor Wavelet Transform (GWT), Autocorrelogram, Gray-Level Co-Occurrence Matrix (GLCM), HSV histogram, and Multi-dimensional CNN. The proposed approach is implemented in MATLAB software and compared with existing approaches like Space-Time Interest Point (STIP) and Histogram of Optical Flow (HOF). Findings: The proposed approach outperforms the existing approaches in terms of reduced time consumption and high accuracy, 99.886% when using the UCF101 dataset and 99.538% when using the UTKinect dataset. Novelty: The most discriminative feature information is obtained with the feature-level fusion technique. From the feature information, various human actions are recognized with the classification algorithm. Keywords: Human activity recognition, Machine Learning, Surveillance Videos, Human detection algorithm, Feature extraction, SVM classifier

Graph based anomaly detection in human action video sequence

Abstract In our paper, we have proposed to use graphs to detect anomaly in human action video. Although the detection of anomaly is a widely researched topic, but very few researchers have detected anomaly in action video using graphs. in our proposed method we have represented the smaller section (sub-section) of our input video as a graph where vertices of the graph are the space time interest points in the sub-section video and the association between the space time interest points exists. Thus, graphs for each sub section are created to look for a repeated substructure. We believe most of the actions inherently are repeated in nature. Thus, we have tried to capture the repetitive sub-structure of the action represented as a graph and used this repetitive sub-structure to compress the graph. If the compressed graph has few elements that have not been compressed, we suspect them as anomaly. But the threshold value takes care not to make the proposed method very much sensitive towards the few uncompressed elements. Our proposed method has been implemented on locally created “extended KTH” and “extended Weizmann” datasets with good accuracy score. The proposed method can also be extended for few more applications such as training athletes and taking elderly care.

Track color space-time interest points in video

Color Space-Time Interest Points (CSTIP) are among all the interesting low-level features which can be extracted from videos; they provide an efficient characterization of moving objects. The CSTIP are simple and can be used for video stabilization, camera motion estimation, and object tracking. In this paper, we show how the resulting features often reflect interesting events that can be used for a compact representation of video data as well as for tracking. To increase the robustness of CSTIP features extraction, we suggest a pre-processing step which is based on a Color Video Decomposition and can decompose the input images into a dynamic color texture and structure components. We compute the new Color Space Time Interest Points (CSTIP) associated to the dynamic color texture components by using the proposed algorithm of the detection of Color Space- Time Interest Points. The point tracker object tracks a set of Color Space-Time Interest Points using the robust Zero-Mean Normalized Cross-Correlation (ZNCC), feature-tracking algorithm. Experimental results are obtained from very different types of videos, namely sport videos and animation movies.

Stochastic Gradient Descent–Whale Optimization Algorithm-Based Deep Convolutional Neural Network To Crowd Emotion Understanding

AbstractCrowd emotion understanding is an interesting research area that assists the security personnel to read the emotion/activity of the crowd in the locality. Most of the traditional methods utilize the low-level visual features to understand the crowd emotions that extend the gap between the low- and the high-level features. With the aim to develop an automatic method for emotion recognition, this paper utilizes the deep convolutional neural network (deep CNN). For the effective emotion recognition, it is essential to select the key frames of the video using the wavelet-based Bhattacharya distance. The key frames are fed to the space-time interest points descriptor that extracts the features and forms the input vector to the classifier. Deep CNN is trained using the proposed Stochastic Gradient Descent–Whale Optimization Algorithm, which is the unification of the standard stochastic gradient descent algorithm with whale optimization algorithm. The proposed classifier recognizes the emotions of the crowd, such as angry, escape, fight, happy, normal, running/walking and violence. The analysis of the method proved that the proposed approaches acquired a maximal accuracy, specificity and sensitivity of 0.9693, 0.9936 and 0.9675, respectively.

Hybrid features-enabled dragon deep belief neural network for activity recognition

ABSTRACTActivity recognition is a challenging task in computer vision that finds widespread applications in various fields, such as motion capture, video retrieval, security, and video surveillance. The objective of this work is to present a technique for recognizing human activities in videos using Dragon Deep Belief Network (DDBN) and hybrid features, which comprises of features like shape, coverage factor, and Space-Time Interest (STI) points. Initially, the keyframes from the input video sequence are extracted using Structural Similarity (SSIM) measure. Then, the features, such as shape, coverage factor, and STI points, are extracted from the keyframes. Based on the feature vector extracted, the proposed DDBN classifier, which is designed by the effective combination of DBN and Dragonfly Algorithm (DA), a classification on human activities, such as walk, bend, etc. in videos. In DDBN, the weights in the network are selected optimally using DA. The weight update using the DA for each incoming feature improves the performance of the DDBN classifier. Further it improves the accuracy in classification of actions. The proposed DDBN classifier is experimented using KTH and Weizmann datasets based on three evaluation parameters, such as accuracy, sensitivity, and specificity. From the performance evaluation, the proposed DDBN classifier could attain better performance with the probability of 98.5% accuracy, 0.96 sensitivity, and 0.959 specificity, respectively.

Spatial-temporal human gesture recognition under degraded conditions using three-dimensional integral imaging.

We present spatial-temporal human gesture recognition in degraded conditions including low light levels and occlusions using passive sensing three-dimensional (3D) integral imaging (InIm) system and 3D correlation filters. The 4D (lateral, longitudinal, and temporal) reconstructed data is processed using a variety of algorithms including linear and non-linear distortion-invariant filters; and compared with previously reported space-time interest points (STIP) feature detector, 3D histogram of oriented gradients (3D HOG) feature descriptor, with a standard bag-of-features support vector machine (SVM) framework, etc. The gesture recognition results with different classification algorithms are compared using a variety of performance metrics such as receiver operating characteristic (ROC) curves, area under the curve (AUC), SNR, the probability of classification errors, and confusion matrix. Integral imaging video sequences of human gestures are captured under degraded conditions such as low light illumination and in the presence of partial occlusions. A four-dimensional (4D) reconstructed video sequence is computed that provides lateral and depth information of a scene over time i.e. (x, y, z, t). The total-variation denoising algorithm is applied to the signal to further reduce noise and preserve data in the video frames. We show that the 4D signal consists of decreased scene noise, partial occlusion removal, and improved SNR due to the computational InIm and/or denoising algorithms. Finally, gesture recognition is processed with classification algorithms, such as distortion-invariant correlation filters; and STIP, 3D HOG with SVM, which are applied to the reconstructed 4D gesture signal to classify the human gesture. Experiments are conducted using a synthetic aperture InIm system in ambient light. Our experiments indicate that the proposed approach is promising in detection of human gestures in degraded conditions such as low illumination conditions with partial occlusion. To the best of our knowledge, this is the first report on spatial-temporal human gesture recognition in degraded conditions using passive sensing 4D integral imaging with nonlinear correlation filters.

Evaluating a bag-of-visual features approach using spatio-temporal features for action recognition

The detection of the spatial-temporal interest points has a key role in human action recognition algorithms. This research work aims to exploit the existing strength of bag-of-visual features and presents a method for automatic action recognition in realistic and complex scenarios. This paper provides a better feature representation by combining the benefit of both a well-known feature detector and descriptor i.e. the 3D Harris space-time interest point detector and the 3D Scale-Invariant Feature Transform descriptor. Finally, action videos are represented using a histogram of visual features by following the traditional bag-of-visual feature approach. Apart from video representation, a support vector machine (SVM) classifier is used for training and testing. A large number of experiments show the effectiveness of our method on existing benchmark datasets and shows state-of-the-art performance. This article reports 68.1% mean Average Precision (mAP), 94% and 91.8% average accuracy for Hollywood-2, UCF Sports and KTH datasets respectively.

Graph formulation of video activities for abnormal activity recognition

Abnormal activity recognition is a challenging task in surveillance videos. In this paper, we propose an approach for abnormal activity recognition based on graph formulation of video activities and graph kernel support vector machine. The interaction of the entities in a video is formulated as a graph of geometric relations among space–time interest points. The vertices of the graph are spatio-temporal interest points and an edge represents the relation between appearance and dynamics around the interest points. Once the activity is represented using a graph, then for classification of the activities into normal or abnormal classes, we use binary support vector machine with graph kernel. These graph kernels provide robustness to slight topological deformations in comparing two graphs, which may occur due to the presence of noise in data. We demonstrate the efficacy of the proposed method on the publicly available standard datasets viz. UCSDped1, UCSDped2 and UMN. Our experiments demonstrate high rate of recognition and outperform the state-of-the-art algorithms.

Obstacle regions extraction method for unmanned aerial vehicles based on space–time tensor descriptor

Obstacle avoidance is an important and challenging task for the autonomous flight of unmanned aerial vehicles. Obstacle regions extraction from image sequences is a critical prerequisite in obstacle avoidance. We propose an obstacle regions extraction method based on space–time tensor descriptor. In our method, first, the space–time tensor descriptor is defined and a criterion function based on the descriptor of extracting space–time interest points (STIPs) is designed. Then a self-adaptive clustering of STIPs approach is presented to locate the possible obstacle regions. Finally, an improved level set algorithm is applied with the result of clustering to extract the obstacle regions. We demonstrate the experiments of obstacle regions extraction by our method on image sequences. Sequences are captured in indoor simulative obstacle avoidance environments and outdoor real flight obstacle avoidance environments. Experimental results validate that our method can effectively complete extraction and segmentation of obstacle region with captured images. Compared with the state-of-the-art methods, our method performs well to extract the contours of obstacle regions on the whole and significantly improves segmentation speed.

Attribute-based learning for gait recognition using spatio-temporal interest points

This paper proposes a new method to extract a gait feature from a raw gait video directly. The Space–Time Interest Points (STIPs) are detected where there are significant movements of human body along both spatial and temporal directions in local spatio-temporal volumes of a raw gait video. Then, a histogram of STIP descriptors (HSD) is constructed as a gait feature. In the classification stage, the support vector machine (SVM) is applied to recognize gaits based on HSDs. In this study, the standard multi-class (i.e. multiple subjects) classification can often be computationally infeasible at test phase, when gait recognition is performed by using every possible classifiers (i.e. SVM models) trained for all individual subjects. In this paper, the attribute-based classification is applied to reduce the number of SVM models needed for recognizing each probe gait. This process will significantly reduce the test-time computational complexity and also retain or even improve the recognition accuracy. When compared with other existing methods in the literature, the proposed method is shown to have the promising performance for the case of normal walking, and the outstanding performance for the cases of walking with variations such as walking with carrying a bag and walking with varying a type of clothes.

일반화된 4차원 특징을 이용한 행동 방향 인식

본 논문에서는 4차원 시공간 (4D-ST, [x,y,z,t]) 특징을 이용하여 행동 방향을 인식하는 방법을 제안한다. 이를 위해 4차원 시공간 특징점 (4D-STIPs, [x,y,z,t])을 제안하였고, 이는 여러 다른 뷰에서 촬영한 이미지들로부터 복원된 3차원 공간 (3D-S, [x,y,z]) 볼륨으로부터 계산된다. 3차원 공간정보를 갖고 있는 3D-S 볼륨과 4D-STIPs는 2차원 공간 (2D-S, [x,y]) 뷰로 사영을 하여 임의의 2D-S 뷰에서의 특징을 생성해 낼 수 있다. 이 때, 사영 방향을 결정 할 수 있으므로, 학습 시 방향에 대한 정보를 포함하여 행동 방향을 인식 할 수 있다. 행동 방향을 인식하는 과정은 두 단계로 나눌 수 있는데, 우선 어떤 행동인지를 인식하고 그 후, 방향 정보를 이용하여 최종적으로 행동 방향을 인식한다. 행동 인식과 방향 인식을 위해, 사영된 3D-S 볼륨과 4D-STIPs은 각각 움직이는 부분과 움직이지 않는 부분에 대한 정보를 담고 있는 motion history images (MHIs)와 non-motion history images (NMHIs)로 구성된다. 이러한 특징들은 행동 인식을 위해, 방향 정보에 상관없이 같은 행동이면 같은 클래스로 분류되어 support vector data description (SVDD) 분류기로 학습되고, support vector domain density description (SVDDD)을 이용하여 인식된다. 인식된 행동에서 최종적으로 방향을 인식하기 위해 각 행동을 방향 클래스로 분류하여 SVDD 분류기로 학습하고 SVDDD로 인식한다. 제안된 방법의 성능을 보이기 위해서 INRIA Xmas Motion Acquisition Sequences (IXMAS) 데이터셋에서 제공하는 3D-S 볼륨을 사용하여 학습을 하고, 행동 방향 인식 실험이 가능한 SNU 데이터셋을 구축하여 인식 실험을 하였다. In this paper, we propose a method to recognize the action direction of human by developing 4D space-time (4D-ST, [x,y,z,t]) features. For this, we propose 4D space-time interest points (4D-STIPs, [x,y,z,t]) which are extracted using 3D space (3D-S, [x,y,z]) volumes reconstructed from images of a finite number of different views. Since the proposed features are constructed using volumetric information, the features for arbitrary 2D space (2D-S, [x,y]) viewpoint can be generated by projecting the 3D-S volumes and 4D-STIPs on corresponding image planes in training step. We can recognize the directions of actors in the test video since our training sets, which are projections of 3D-S volumes and 4D-STIPs to various image planes, contain the direction information. The process for recognizing action direction is divided into two steps, firstly we recognize the class of actions and then recognize the action direction using direction information. For the action and direction of action recognition, with the projected 3D-S volumes and 4D-STIPs we construct motion history images (MHIs) and non-motion history images (NMHIs) which encode the moving and non-moving parts of an action respectively. For the action recognition, features are trained by support vector data description (SVDD) according to the action class and recognized by support vector domain density description (SVDDD). For the action direction recognition after recognizing actions, each actions are trained using SVDD according to the direction class and then recognized by SVDDD. In experiments, we train the models using 3D-S volumes from INRIA Xmas Motion Acquisition Sequences (IXMAS) dataset and recognize action direction by constructing a new SNU dataset made for evaluating the action direction recognition.

Recognizing Gaits on Spatio-Temporal Feature Domain

Gait has been known as an effective biometric feature to identify a person at a distance, e.g., in video surveillance applications. Many methods have been proposed for gait recognitions from various different perspectives. It is found that these methods rely on appearance (e.g., shape contour, silhouette)-based analyses, which require preprocessing of foreground–background segmentation (FG/BG). This process not only causes additional time complexity, but also adversely influences performances of gait analyses due to imperfections of existing FG/BG methods. Besides, appearance-based gait recognitions are sensitive to several variations and partial occlusions, e.g., caused by carrying a bag and varying a cloth type. To avoid these limitations, this paper proposes a new framework to construct a new gait feature directly from a raw video. The proposed gait feature extraction process is performed in the spatio-temporal domain. The space-time interest points (STIPs) are detected by considering large variations along both spatial and temporal directions in local spatio-temporal volumes of a raw gait video sequence. Thus, STIPs are allocated, where there are significant movements of human body in both space and time. A histogram of oriented gradients and a histogram of optical flow are computed on a 3D video patch in a neighborhood of each detected STIP, as a STIP descriptor. Then, the bag-of-words model is applied on each set of STIP descriptors to construct a gait feature for representing and recognizing an individual gait. When compared with other existing methods in the literature, it has been shown that the performance of the proposed method is promising for the case of normal walking, and is outstanding for the case of partial occlusion caused by walking with carrying a bag and walking with varying a cloth type.

Fights Behavior Detection Based on Space-Time Interest Points

Human action recognition belongs to the senior visual analysis of computer vision, which involves image processing, artificial intelligence, pattern recognition and so on, is becoming one of the most hot research topic in recent years. In this paper, on the basis of comparative analysis and study towards current methods related to human action recognition, we propose a novel fights behavior detection method which is based on spatial-temporal interest point. Since most information of human action in video are indicated by the space-time interest points of video, we combine spatial-temporal features with motion energy image to describe information of video, and local spatial-temporal features are applied to extract fights behavior model by bags of words. Experimental results show that this method can achieve high accuracy and certain practical value.

View invariant action recognition using generalized 4D features

In this paper, we propose a method to recognize human actions independently of viewpoints by developing 4D space–time features which can generalize the information from a finite number of views in training phase so as to show a satisfactory performance in arbitrary testing views. This 4D space–time interest points (4D-STIPs, [x,y,z,t]) are extracted using 3D space volumes reconstructed from images of a finite number of different views. Since the proposed features are constructed using volumetric information, the features for arbitrary 2D space viewpoint in testing can be generated by projecting the 3D space volumes and 4D-STIPs on corresponding test image planes. This enables action recognition in any camera viewpoint even after training with images from only a finite number of views. We also propose the variant of 3D space–time interest points, which take into account the simultaneous gradient variation in all 3 dimensions to focus on the motion of important spatial corner points. 3D space volumes and 4D-STIPs can be projected to arbitrary viewpoints for training each action to get generalization capability of the classifier. With these projected features, we construct motion history images and non-motion history images which encode the moving and non-moving parts of an action respectively. After reducing the feature dimension, the final features are learned by support vector data description method. In experiments, we train the models using IXMAS dataset constructed from five views and test them with a new SNU dataset made for evaluating the generalization performance for arbitrary view videos.

Human Behavior Recognition Based on Motion Decomposition

Human behavior recognition is an active research field in computer vision and image processing. A novel method is proposed for human behavior recognition in video image sequences. First of all, a video sequence is represented by extracting space-time interest points. Then Human behavior is represented by activities through Motion Decomposition. The activity comprises labeled bags that are composed of unlabeled instances comprising to action. Final labeled activities are used to train a strong classifier which is used to predict the labels of unseen behavior bags. Experimental results show the effectiveness of the proposed method in comparison with other related works in the literature and can also tolerate noise and interference conditions.

Space-time Facet Model for Human Activity Classification

This paper presents a novel space-time feature-based human activity analysis system. We detect Space Time Interest Points (STIP) and generate their description based on the facet model. The proposed approach detects interest points in video data using the three-dimensional facet model efficiently. Then we describe each interest point by three-dimensional Haar wavelet transform and time derivatives of different order obtained from said facet model. Here we represent each video clip following the bag-of-words approach by learning feature specific dictionary. Finally, classification is done using non-linear SVM with χ 2 -kernel. We evaluate the performance of our system on standard datasets like Weizmann, KTH, UCF sports, ICD, UCF YouTube, and UCF50 and get better, or at least comparable results compared to other state-of-the-art systems.

Graph-based approach for human action recognition using spatio-temporal features

Due to the exponential growth of the video data stored and uploaded in the Internet websites especially YouTube, an effective analysis of video actions has become very necessary. In this paper, we tackle the challenging problem of human action recognition in realistic video sequences. The proposed system combines the efficiency of the Bag-of-visual-Words strategy and the power of graphs for structural representation of features. It is built upon the commonly used Space–Time Interest Points (STIP) local features followed by a graph-based video representation which models the spatio-temporal relations among these features. The experiments are realized on two challenging datasets: Hollywood2 and UCF YouTube Action. The experimental results show the effectiveness of the proposed method.

Vision-based action recognition of earthmoving equipment using spatio-temporal features and support vector machine classifiers

Video recordings of earthmoving construction operations provide understandable data that can be used for benchmarking and analyzing their performance. These recordings further support project managers to take corrective actions on performance deviations and in turn improve operational efficiency. Despite these benefits, manual stopwatch studies of previously recorded videos can be labor-intensive, may suffer from biases of the observers, and are impractical after substantial period of observations. This paper presents a new computer vision based algorithm for recognizing single actions of earthmoving construction equipment. This is particularly a challenging task as equipment can be partially occluded in site video streams and usually come in wide variety of sizes and appearances. The scale and pose of the equipment actions can also significantly vary based on the camera configurations. In the proposed method, a video is initially represented as a collection of spatio-temporal visual features by extracting space–time interest points and describing each feature with a Histogram of Oriented Gradients (HOG). The algorithm automatically learns the distributions of the spatio-temporal features and action categories using a multi-class Support Vector Machine (SVM) classifier. This strategy handles noisy feature points arisen from typical dynamic backgrounds. Given a video sequence captured from a fixed camera, the multi-class SVM classifier recognizes and localizes equipment actions. For the purpose of evaluation, a new video dataset is introduced which contains 859 sequences from excavator and truck actions. This dataset contains large variations of equipment pose and scale, and has varied backgrounds and levels of occlusion. The experimental results with average accuracies of 86.33% and 98.33% show that our supervised method outperforms previous algorithms for excavator and truck action recognition. The results hold the promise for applicability of the proposed method for construction activity analysis.

Discriminative two-level feature selection for realistic human action recognition

Constructing the bag-of-features model from Space–time interest points (STIPs) has been successfully utilized for human action recognition. However, how to eliminate a large number of irrelevant STIPs for representing a specific action in realistic scenarios as well as how to select discriminative codewords for effective bag-of-features model still need to be further investigated. In this paper, we propose to select more representative codewords based on our pruned interest points algorithm so as to reduce computational cost as well as improve recognition performance. By taking human perception into account, attention based saliency map is employed to choose salient interest points which fall into salient regions, since visual saliency can provide strong evidence for the location of acting subjects. After salient interest points are identified, each human action is represented with the bag-of-features model. In order to obtain more discriminative codewords, an unsupervised codeword selection algorithm is utilized. Finally, the Support Vector Machine (SVM) method is employed to perform human action recognition. Comprehensive experimental results on the widely used and challenging Hollywood-2 Human Action (HOHA-2) dataset and YouTube dataset demonstrate that our proposed method is computationally efficient while achieving improved performance in recognizing realistic human actions.

Realistic Human Action Recognition With Multimodal Feature Selection and Fusion

Although promising results have been achieved for human action recognition under well-controlled conditions, it is very challenging to recognize human actions in realistic scenarios due to increased difficulties such as dynamic backgrounds. In this paper, we propose to take multimodal (i.e., audiovisual) characteristics of realistic human action videos into account in human action recognition for the first time, since, in realistic scenarios, audio signals accompanying an action generally provide a cue to the nature of the action, such as phone ringing to answering the phone . In order to cope with diverse audio cues of an action in realistic scenarios, we propose to identify effective features from a large number of audio features with the generalized multiple kernel learning algorithm. The widely used space-time interest point descriptors are utilized as visual features, and a support vector machine is employed for both audio- and video-based classifications. At the final stage, fuzzy integral is utilized to fuse recognition results of both audio and visual modalities. Experimental results on the challenging Hollywood-2 Human Action data set demonstrate that the proposed approach is able to achieve better recognition performance improvement than that of integrating scene context. It is also discovered how audio context influences realistic action recognition from our comprehensive experiments.