Spatio-temporal Interest Points Research Articles

A Review Work: Human Action Recognition in Video Surveillance Using Deep Learning Techniques

Despite being extensively used in numerous uses, precise and effective human activity identification continues to be an interesting research issue in the area of vision for computers. Currently, a lot of investigation is being done on themes like pedestrian activity recognition and ways to recognize people's movements employing depth data, 3D skeletal data, still picture data, or strategies that utilize spatiotemporal interest points. This study aims to investigate and evaluate DL approaches for detecting human activity in video. The focus has been on multiple structures for detecting human activities that use DL as their primary strategy. Based on the application, including identifying faces, emotion identification, action identification, and anomaly identification, the human occurrence forecasts are divided into four different subcategories. The literature has been carried several research based on these recognitions for predicting human behavior and activity for video surveillance applications. The state of the art of four different applications' DL techniques is contrasted. This paper also presents the application areas, scientific issues, and potential goals in the field of DL-based human behavior and activity recognition/detection.

Machine Vision-Based Human Action Recognition Using Spatio-Temporal Motion Features (STMF) with Difference Intensity Distance Group Pattern (DIDGP)

In recent years, human action recognition is modeled as a spatial-temporal video volume. Such aspects have recently expanded greatly due to their explosively evolving real-world uses, such as visual surveillance, autonomous driving, and entertainment. Specifically, the spatio-temporal interest points (STIPs) approach has been widely and efficiently used in action representation for recognition. In this work, a novel approach based on the STIPs is proposed for action descriptors i.e., Two Dimensional-Difference Intensity Distance Group Pattern (2D-DIDGP) and Three Dimensional-Difference Intensity Distance Group Pattern (3D-DIDGP) for representing and recognizing the human actions in video sequences. Initially, this approach captures the local motion in a video that is invariant to size and shape changes. This approach extends further to build unique and discriminative feature description methods to enhance the action recognition rate. The transformation methods, such as DCT (Discrete cosine transform), DWT (Discrete wavelet transforms), and hybrid DWT+DCT, are utilized. The proposed approach is validated on the UT-Interaction dataset that has been extensively studied by past researchers. Then, the classification methods, such as Support Vector Machines (SVM) and Random Forest (RF) classifiers, are exploited. From the observed results, it is perceived that the proposed descriptors especially the DIDGP based descriptor yield promising results on action recognition. Notably, the 3D-DIDGP outperforms the state-of-the-art algorithm predominantly.

Motion Feature Retrieval in Basketball Match Video Based on Multisource Motion Feature Fusion

Both the human body and its motion are three-dimensional information, while the traditional feature description method of two-person interaction based on RGB video has a low degree of discrimination due to the lack of depth information. According to the respective advantages and complementary characteristics of RGB video and depth video, a retrieval algorithm based on multisource motion feature fusion is proposed. Firstly, the algorithm uses the combination of spatiotemporal interest points and word bag model to represent the features of RGB video. Then, the directional gradient histogram is used to represent the feature of the depth video frame. The statistical features of key frames are introduced to represent the histogram features of depth video. Finally, the multifeature image fusion algorithm is used to fuse the two video features. The experimental results show that multisource feature fusion can greatly improve the retrieval accuracy of motion features.

Performance evaluation of entropy-based cuboid selection technique for human interaction recognition

Local feature-based approaches are proven to be most successful for the application of human action recognition. This work aims to study the effect of the number of cuboids chosen on the performance of human action recognition. A 3D region, called cuboid, is extracted around every spatio-temporal interest point and an entropy-based cuboid selection method is implemented for choosing the cuboids having maximum information. The proposed method is evaluated using UT interaction and SBU kinect interaction datasets. Results show that maximum classification accuracy and F1 Score are obtained when the top 40 cuboids are selected for the feature extraction. The classification accuracy and F1 Score increase when the number of selected cuboids is increased from 10 to 40 and remain constant when the number is increased beyond 40. The selected 40 cuboids are then used for feature extraction. To carry out this experiment, wavelet coefficients extracted from selected cuboids are used as features.

Edge computing enabled video segmentation for real-time traffic monitoring in internet of vehicles

In the Internet of Things enabled intelligent transportation systems, a huge amount of vehicle video data has been generated and real-time and accurate video analysis are very important and challenging work, especially in situations with complex street scenes. Therefore, we propose edge computing based video pre-processing to eliminate the redundant frames, so that we migrate the partial or all the video processing task to the edge, thereby diminishing the computing, storage and network bandwidth requirements of the cloud center, and enhancing the effectiveness of video analyzes. To eliminate the redundancy of the traffic video, the magnitude of motion detection based on spatio-temporal interest points (STIP) and the multi-modal linear features combination are presented which splits a video into super frame segments of interests. After that, we select the key frames from these interesting segments of the long videos with the design and detection of the prominent region. Finally, the extensive numerical experimental verification results show our methods are superior to the previous algorithms for different stages of the redundancy elimination, video segmentation, key frame selection and vehicle detection.

ADMT: Advanced Driver’s Movement Tracking System Using Spatio-Temporal Interest Points and Maneuver Anticipation Using Deep Neural Networks

Assistive driving is a complex engineering problem and is influenced by several factors such as the sporadic nature of the quality of the environment, the response of the driver, and the standard of the roads on which the vehicle is being driven. The authors track the driver's anticipation based on his head movements using Spatio-Temporal Interest Point (STIP) extraction and enhance the anticipation of action accuracy well before using the RNN-LSTM framework. This research tackles a fundamental problem of lane change assistance by developing a novel model called Advanced Driver's Movement Tracking (ADMT). ADMT uses customized convolution-based deep learning networks by using Recurrent Convolutional Neural Network (RCNN). STIP with eye gaze extraction and RCNN performed in ADMT on brain4cars dataset for driver movement tracking. Its performance is compared with the traditional machine learning and deep learning models, namely Support Vector Machines (SVM), Hidden Markov Model (HMM), Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), Long Short-Term Memory (LSTM), and provided an increment of almost 12% in the prediction accuracy and 44% in the anticipation time. Furthermore, ADMT systems outperformed all of the models in terms of both the accuracy of the system and the previously mentioned time of anticipation that is discussed at length in the paper. Thus it assists the driver with additional anticipation time to access the typical reaction time for better preparedness to respond to undesired future behavior. The driver is then assured of a safe and assisted driving experience with the proposed system.

Human gesture recognition under degraded environments using 3D-integral imaging and deep learning.

In this paper, we propose a spatio-temporal human gesture recognition algorithm under degraded conditions using three-dimensional integral imaging and deep learning. The proposed algorithm leverages the advantages of integral imaging with deep learning to provide an efficient human gesture recognition system under degraded environments such as occlusion and low illumination conditions. The 3D data captured using integral imaging serves as the input to a convolutional neural network (CNN). The spatial features extracted by the convolutional and pooling layers of the neural network are fed into a bi-directional long short-term memory (BiLSTM) network. The BiLSTM network is designed to capture the temporal variation in the input data. We have compared the proposed approach with conventional 2D imaging and with the previously reported approaches using spatio-temporal interest points with support vector machines (STIP-SVMs) and distortion invariant non-linear correlation-based filters. Our experimental results suggest that the proposed approach is promising, especially in degraded environments. Using the proposed approach, we find a substantial improvement over previously published methods and find 3D integral imaging to provide superior performance over the conventional 2D imaging system. To the best of our knowledge, this is the first report that examines deep learning algorithms based on 3D integral imaging for human activity recognition in degraded environments.

Active Player Detection in Handball Scenes Based on Activity Measures.

In team sports training scenes, it is common to have many players on the court, each with his own ball performing different actions. Our goal is to detect all players in the handball court and determine the most active player who performs the given handball technique. This is a very challenging task, for which, apart from an accurate object detector, which is able to deal with complex cluttered scenes, additional information is needed to determine the active player. We propose an active player detection method that combines the Yolo object detector, activity measures, and tracking methods to detect and track active players in time. Different ways of computing player activity were considered and three activity measures are proposed based on optical flow, spatiotemporal interest points, and convolutional neural networks. For tracking, we consider the use of the Hungarian assignment algorithm and the more complex Deep SORT tracker that uses additional visual appearance features to assist the assignment process. We have proposed the evaluation measure to evaluate the performance of the proposed active player detection method. The method is successfully tested on a custom handball video dataset that was acquired in the wild and on basketball video sequences. The results are commented on and some of the typical cases and issues are shown.

Efficient Sparse Representation based Action Recognition in video

Human Action Recognition (HAR) is an interesting and helpful topic in various real-life applications such as surveillance based security system, computer vision and robotics. The selected features and feature representation methods, classification algorithms decides the accuracy of the HAR systems. A new feature called, Skeletonized STIP (Spatio Temporal Interest Points) is identified and used in this work. The skeletonization on the action video’s foreground frames are performed and the new feature is generated as STIP values of the skeleton frame sequence. Then the feature set is used for initial dictionary construction in sparse coding. The data for action recognition is huge, since the feature set is represented using the sparse representation. To refine the sparse representation the max pooling method is used and the action recognition is performed using SVM classifier. The proposed approach outperforms on the benchmark datasets.

Fuzzy Based Deep Belief Network for Activity Recognition

In the automated machinery world, human-computer interaction plays a major role, in which human actions or activities are major key components for a better performance. Hence, activity recognition becomes an active research area, where a lot of algorithms and methods were evolved quickly. Still there is lot of challenges like view occlusions, clothing and speed of action sequence. This paper proposes a novel classifier with skeleton features, to recognize human activities. This work integrates fuzzy with Dragon Deep Belief Network in order to improve the accuracy in complex activities. From the given input videos, initially key frames are selected to extract sufficient features for classification using SSIM (Structure Similarity Measure) then Scale Invariant Feature Transform (SIFT) has been applied to selected Spatial Invariant features and Spatio-Temporal interest points has also extracted to retain temporal features. Finally, combined spatial and temporal features are used for training and testing the classifier. For implementation, input videos are chosen from two common datasets, namely KTH, Weizienam. Detailed performance analyses were done with various actions like walking, running and boxing. The proposed work improves the performance in accuracy better with the maximum range up to 1.

Human action recognition based on HOIRM feature fusion and AP clustering BOW

In this paper, we propose a human action recognition method using HOIRM (histogram of oriented interest region motion) feature fusion and a BOW (bag of words) model based on AP (affinity propagation) clustering. First, a HOIRM feature extraction method based on spatiotemporal interest points ROI is proposed. HOIRM can be regarded as a middle-level feature between local and global features. Then, HOIRM is fused with 3D HOG and 3D HOF local features using a cumulative histogram. The method further improves the robustness of local features to camera view angle and distance variations in complex scenes, which in turn improves the correct rate of action recognition. Finally, a BOW model based on AP clustering is proposed and applied to action classification. It obtains the appropriate visual dictionary capacity and achieves better clustering effect for the joint description of a variety of features. The experimental results demonstrate that by using the fused features with the proposed BOW model, the average recognition rate is 95.75% in the KTH database, and 88.25% in the UCF database, which are both higher than those by using only 3D HOG+3D HOF or HOIRM features. Moreover, the average recognition rate achieved by the proposed method in the two databases is higher than that obtained by other methods.

Detecting anomalous crowd scenes by oriented Tracklets’ approach in active contour region

Video imagery based crowd analysis has become a topic of great interest for public safety at the venues of mass gathering events. This paper presents a novel approach to detect anomalous scene in high density crowded places. We propose an oriented tracklets approach in active contour region and measure the entropy and temporal occupancy deviation of oriented tracklets over the frames. To this end, an oriented trajectory algorithm is designed to extract tracklets of moving crowd. For trajectory extraction, spatio-temporal interest points are detected by adopting Harris corner features. The detected interest points are tracked over the frames within the optimized region. An active contour segmentation approach is applied to optimize the tracking region as a moving crowd is not distributed in entire frame region. The flow direction of each oriented tracklet is distributed into histogram bins at a specified interval, which defines the flow of collective motion pattern. A real-time scene updating procedure is also followed to adapt the changes of crowd scenes. Further, an entropy of histogram of oriented tracklets is computed based on the probability of occurrence of the tracklets. It has been shown that entropy of flow direction changes markedly in the unusual state of affairs. A simulation on a large number of the anomalous scene has been exercised to see the characteristics of an entropy. Also, temporal occupancy deviation is computed which measures the area occupied by the extracted tracklets of the crowd during a certain interval of time. If entropy and temporal occupancy deviation increase beyond a certain threshold, an alert is issued to detect anomaly to prevent potentially dangerous crowd-related disasters. Experiments conducted on three publicly available benchmark crowd datasets such as UMN, UCF Web, and Violent Flows, obtained interesting and promising results. We also evaluated some manually collected challenging real-world crowd video sequences. We compared the proposed approach with various state-of-the-art methods, and achieve remarkable accuracy while maintaining the lower computational complexity.

Segmentation of crowd flow by trajectory clustering in active contours

Crowd analysis has become an important topic of research for visual surveillance community. This paper proposes an active contour-based trajectory clustering approach for crowd flow segmentation. To this end, the active contour method is applied to segment the foreground crowd region with an aim to optimize further tracking. From the segmented foreground region, spatiotemporal interest points are detected and tracked to extract crowd trajectories. The trajectories are then parameterized by their shape, location information, flow direction, and neighborhood density. A clustering algorithm is designed to cluster these trajectories, and further flow patterns are segmented by merging trajectory clusters on the basis of their spatial overlapping and distinction in location and in flow direction. Once the flow patterns are segmented, trajectory density of each segment is estimated to analyze crowd flow. Experiments are conducted on three publicly available UCF Web, Collective Motion, and Violent Flows crowd datasets. The proposed work is compared with various state-of-the-art methods and achieves remarkable accuracy while maintaining the lower computational complexity.

Spatiotemporal interest point detector exploiting appearance and motion-variation information

As a local invariant feature of videos, the spatiotemporal interest point (STIP) has been widely used in computer vision and pattern recognition. However, existing STIP detectors are generally extended from detection algorithms constructed for local invariant features of two-dimensional images, which does not explicitly exploit the motion information inherent in the temporal domain of videos, thus weakening the performance of existing STIP detectors in a video context. To remedy this, we aim to develop an STIP detector that uniformly captures appearance and motion information for video, thus yielding substantial performance improvement. Specifically, under the framework of geometric algebra, we first develop a spatiotemporal unified model of appearance and motion-variation information (UMAMV), and then a UMAMV-based scale space of the spatiotemporal domain is proposed to synthetically analyze appearance information and motion information in a video. Based on this model, we propose an STIP feature of UMAMV-SIFT that embraces both appearance and motion variation information of the videos. Three datasets with different sizes are utilized to evaluate the proposed model and the STIP detector. We present experimental results to show that the UMAMV-SIFT achieves state-of-the-art performance and is particularly effective when dataset is small.

Early anticipation of driver’s maneuver in semiautonomous vehicles using deep learning

Making machines to anticipate human action is a complex research problem. Some of the recent research studies on computer vision and assistive driving have reported that the anticipation of driver’s action few seconds in advance is a challenging problem. These studies are based on the driver’s head movement tracking, eye gaze tracking, and spatiotemporal interest points. The study is aimed to address an important question of how to anticipate a driver’s action while driving and improve the anticipation time. The goal of this study is to review the existing deep learning framework for assistive driving. This paper differs from the existing solutions in two ways. First, it proposes a simplified framework using the driver’s inside video data and develops a driver’s movement tracking (DMT) algorithm. Majority of the existing state of the art is based on inside and outside features of the vehicles. Second, the proposed work tends to improve the image pattern recognition by introducing a fusion of spatiotemporal data points (STIPs) for movement tracking along with eye cuboids and then action anticipation by using deep learning. The proposed DMT algorithm tracks the driver’s movement using STIPs from the input video. Also, a fast eye gaze algorithm tracks eye movements. The features extracted from STIP and eye gaze are fused and analyzed by a deep recurrent neural network to improve the prediction time, thereby giving a few extra seconds to anticipate the driver’s correct action. The performance of the DMT algorithm is compared with the previous algorithms and found that DMT offers 30% improvement with regards to anticipating driver’s action over two recently proposed deep learning algorithms.

A long video caption generation algorithm for big video data retrieval

Videos captured by people are often tied to certain important moments of their lives. But with the era of big data coming, the time required to retrieval and watch can be daunting. In this paper, novel techniques are proposed for the application of long video segmentation, which can effectively shorten the retrieval time. The motion extent of long video is detected by the improved of the spatio-temporal interest points (STIPs) detection algorithm. After that, the superframe segmentation of the filtered long video is performed to gain the interesting clip of long video. In the selection of keyframes, the region of interest is constructed by the use of the STIP already obtained on the video clips, and the saliency detection of these regions of interest is utilized to screen out video keyframes. Finally, we generate the video captions by adding attention vectors to the traditional LSTM. Our method is benchmarked on the VideoSet dataset, and evaluated by the BLEU, Meteor and Rouge.

Human action recognition based on spatio-temporal three-dimensional scattering transform descriptor and an improved VLAD feature encoding algorithm

The local spatio-temporal descriptor and feature encoding algorithm are two crucial key steps for human action recognition based on spatio-temporal interest points (STIP). Since the local descriptors for STIP are essentially a type of motion information based on the texture, the key point of local feature description is to extract invariable, robust and distinguishable local texture features and motion information in reference spatio-temporal volume. Scattering transform is an image transform method based on directional wavelet transform and scale convolution, which has local translation invariance, rotation invariance and elastic deformation stability for local texture features. A novel local descriptor for STIP based on spatio-temporal three-dimensional scattering transform is proposed in this paper, which extends the original scattering transform to spatio-temporal three-dimensional space. Compared to the traditional descriptors, such as HOG, HOF and so on, the proposed scattering transform coefficients based histogram of oriented gradients (STC-HOG) descriptor can capture more robust and distinguishable motion information of local texture for STIP. In order to incorporate the local descriptors into action video representation, the feature encoding algorithm is indispensable. For the problem that vector of locally aggregated descriptors (VLAD) loses feature distribution location information during feature encoding, a histogram of distribution vector of locally aggregated descriptors (HOD-VALD) based on Gaussian kernel is proposed. We validated the proposed algorithm for human action recognition on multiple public available datasets, such as KTH, UCF Sports, HMDB51 and so on. The evaluation experiment results indicate that the proposed descriptor and encoding method can improve the efficiency of human action recognition and the recognition accuracy.

Violence detection in surveillance video using low-level features.

It is very important to automatically detect violent behaviors in video surveillance scenarios, for instance, railway stations, gymnasiums and psychiatric centers. However, the previous detection methods usually extract descriptors around the spatiotemporal interesting points or extract statistic features in the motion regions, leading to limited abilities to effectively detect video-based violence activities. To address this issue, we propose a novel method to detect violence sequences. Firstly, the motion regions are segmented according to the distribution of optical flow fields. Secondly, in the motion regions, we propose to extract two kinds of low-level features to represent the appearance and dynamics for violent behaviors. The proposed low-level features are the Local Histogram of Oriented Gradient (LHOG) descriptor extracted from RGB images and the Local Histogram of Optical Flow (LHOF) descriptor extracted from optical flow images. Thirdly, the extracted features are coded using Bag of Words (BoW) model to eliminate redundant information and a specific-length vector is obtained for each video clip. At last, the video-level vectors are classified by Support Vector Machine (SVM). Experimental results on three challenging benchmark datasets demonstrate that the proposed detection approach is superior to the previous methods.

Action recognition based on histogram of oriented gradients and spatio-temporal interest points

In modern years large extent of the work has been carried out to recognize human actions perhaps because of its wide range of applications in the field of surveillance, human-machine interaction and video analysis. Several methods were proposed by researchers to resolve action recognition challenges such as variations in viewpoints, occlusion, cluttered backgrounds and camera motion. To address these challenges, we propose a novel method comprise of features extraction using histogram of oriented gradients (HOG), and their classification using k-nearest neighbor (k-NN) and support vector machine (SVM). Six different experimentations were carried out on the basis of hybrid combinations of feature extractors and classifiers. Two gold standard datasets; KTH and Weizmann were used for training and testing purpose. The quantitative parameters such as recognition accuracy, training time and prediction speed were used for evaluation. To validate the applicability of proposed algorithm, its performance has been compared with spatio-temporal interest points (STIP) technique which was proposed as state of art method in the domain.

Video‐based road traffic monitoring and prediction using dynamic Bayesian networks

The varied road conditions, chaotic and unstructured traffic, lack of lane discipline and wide variety of vehicles in countries like India, Pakistan and so on pose a need for a novel traffic monitoring system. In this study, the authors propose a novel camera-based traffic monitoring and prediction scheme without identifying or tracking vehicles. Spatial interest points (SIPs) and spatio-temporal interest points (STIPs) are extracted from the video stream of road traffic. SIP represents the number of vehicles and STIP represents the number of moving vehicles. The distributions of these features are then classified using Gaussian mixture model. In the proposed method, they learn the road state pattern using dynamic Bayesian network and predict the future road traffic state within a specific time delay. The predicted road state information can be used for traffic planning. The proposed method is computationally light, yet very powerful and efficient. The algorithm is tested for different weather conditions as well. They have validated their algorithm using Synchro Studio simulator and got 95.7% as average accuracy and on real-time video we got an accuracy of 84%.