Spatiotemporal Feature Descriptor Research Articles

A software package for the investigation of spatio-temporal feature descriptors

At present time researchers investigate approaches to the automatic facial micro-expression analysis to implement the possibility of the recognition of true human emotions from the image sequence of his face. The modern technologies are based on the use of complex mathematical algorithms with many adjusted parameters; in particular, such algorithms are the algorithms of spatio-temporal feature descriptors. The scientists who are not technical specialists, e.g., psychologists or psychotherapists, when investigating these algorithms may encounter difficulties in writing the program code. Therefore, it became necessary to develop a special software package for the investigation of spatio-temporal feature descriptors. Aim of the work is development of a special software package for studying various spatial and temporal descriptors of facial micromovement features in video images A software package for the investigation of spatio-temporal feature descriptors with a graphical interface is developed. The software package makes it possible to investigate the influence of the selection of the descriptor (LBP-TOP, FHOOF or FHOFO) and its parameters on the micro-facial movement detection accuracy for different input datasets of human face image sequences. The developed software package for the investigation of spatio-temporal feature descriptors can be applied as a software tool for carrying out investigations in the facial micro-expression analysis field.

Harvesting spatiotemporal correlation from sky image sequence to improve ultra-short-term solar irradiance forecasting

With photovoltaic power being increasingly integrated into power grid, accurately forecasting solar irradiance is of critical for ensuring stable and economical operation of power systems. The forecast accuracy in ultra-short-term horizons can be greatly improved by employing ground-based sky images. Although wide range of computer vision methods have been used for modelling, effectively extracting spatiotemporal features from sky image sequence is still a tough task. In this study, a sparse spatiotemporal feature descriptor is introduced to enhance the process of dynamic spatiotemporal information extraction from continuous grayscale sky images, while spatial pyramid pooling is used for feature refinement. Parallelly, dense convolutional network is used to extract static features from the nearest single-frame RGB sky images. Both dynamic and static spatiotemporal features were adequately extracted and subsequently fused for the multi-step prediction of global horizontal irradiance. In addition, various competitive models in object detection are adopted as benchmarks for comparison. The experimental results revealed that the proposed method outperformed baseline models, with up to 5.51% reduction on normalized root mean square error (NRMSE) and 9.38% improvement on ramp event forecast. The proposed method can be widely applied to photovoltaic stations equipped with all-sky-imagers.

A Novel Multi-Modal Network-Based Dynamic Scene Understanding

In recent years, dynamic scene understanding has gained attention from researchers because of its widespread applications. The main important factor in successfully understanding the dynamic scenes lies in jointly representing the appearance and motion features to obtain an informative description. Numerous methods have been introduced to solve dynamic scene recognition problem, nevertheless, a few concerns still need to be investigated. In this article, we introduce a novel multi-modal network for dynamic scene understanding from video data, which captures both spatial appearance and temporal dynamics effectively. Furthermore, two-level joint tuning layers are proposed to integrate the global and local spatial features as well as spatial and temporal stream deep features. In order to extract the temporal information, we present a novel dynamic descriptor, namely, Volume Symmetric Gradient Local Graph Structure ( VSGLGS ), which generates temporal feature maps similar to optical flow maps. However, this approach overcomes the issues of optical flow maps. Additionally, Volume Local Directional Transition Pattern ( VLDTP ) based handcrafted spatiotemporal feature descriptor is also introduced, which extracts the directional information through exploiting edge responses. Lastly, a stacked Bidirectional Long Short-Term Memory ( Bi-LSTM ) network along with a temporal mixed pooling scheme is designed to achieve the dynamic information without noise interference. The extensive experimental investigation proves that the proposed multi-modal network outperforms most of the state-of-the-art approaches for dynamic scene understanding.

Gaze dynamics with spatiotemporal guided feature descriptor for prediction of driver’s maneuver behavior

At different levels of driving automation, driver’s gaze maintains great indispensable importance on semantic perception of the surround. In this work, we model gaze dynamics and clarify its relationship with driver’s maneuver behaviors from personalized driving style. Firstly, this paper proposes an Occlusion-immune Face Detector (OFD) for facial landmark detection, which can adaptively solve the facial occlusion introduced by the body and glasses frame in the real-world driving scenarios. Meanwhile, an Eye-head Coordination Model is brought up to bridge the error gap in gaze direction through determining eye pose and head pose fused pattern. Then, a vectorized spatiotemporal guidance feature (STGF) descriptor combining gaze accumulation and gaze transition frequency is proposed to construct gaze dynamics within a time window. Finally, we predict driver’s maneuver behavior through STGF descriptor considering different driving styles to clarify the relationship between gaze dynamics and driving maneuver task. Natural driving data are sampled in a dedicated instrumented vehicle testbed, on which 15 drivers with three kind of driving styles participated. Experimental results show that the prediction model achieves the best performance, estimating driver’s behavior an average of 1 s ahead of actual behavior with 83.6% accuracy considering driving style, compared with other approaches.

Video Anomaly Detection Using the Optimization-Enabled Deep Convolutional Neural Network

Abstract In video surveillance, automatic detection of the anomalies is the active research area in computer technology. Even though various video anomaly detection methods are introduced, detecting anomalous events, such as illegal actions and crimes, is a major challenging issue in video surveillance. Thus, an effective automatic video anomaly detection strategy based on the deep convolutional neural network (deep CNN) is developed in this research. Initially, the input video surveillance is passed into the spatiotemporal feature descriptor, named Histograms of Optical Flow Orientation and Magnitude. The features obtained from the descriptor provide the optical flow details with the aspect of normal patterns from the scene. These patterns are further subjected to the deep CNN, which is trained using the proposed dragonfly-rider optimization algorithm (DragROA) to assure the classification either as an anomalous activity or normal. The proposed DragROA is the combination of the standard dragonfly optimization algorithm and the standard rider optimization algorithm. The implementation of the proposed DragROA-based deep CNN is carried out using two datasets, namely anomaly detection dataset and UMN dataset; the performance is analyzed using the metrics, namely accuracy, sensitivity and specificity. From the analysis, it is depicted that the proposed method obtains the maximum accuracy, sensitivity and specificity of 0.9922, 0.9809 and 1, respectively, for the UCSD dataset.

Fusing pyramid histogram of gradients and optical flow for hand gesture recognition

Human computer interaction systems based on hand gestures catch the eye of the research community for implementing natural communication between man and machines. However, different persons perform the same gestures differently in terms of velocity and motion scale. This poses a challenging issue in minimising the variations between different persons and maximises the coherence of the same gestures. In this paper, the original pyramid histogram of gradients in three orthogonal planes combining with the dense optical flow to create dynamic descriptor is explored in to discriminate features for recognition of hand gestures. The shape and motion features of images in a video sequence are captured to obtain the geometric and illumination invariant spatio-temporal feature descriptor for classification. A multiclass support vector machine classifier is used to recognise the hand gestures. The proposed method gives an excellent recognition rate and excels the existing approaches.

Dual-channel CNN for efficient abnormal behavior identification through crowd feature engineering

This research has been investigating an automatic and online crowd anomaly detection model by exploring a novel compound image descriptor generated from live video streams. A dual-channel convolutional neural network (DCCNN) has been set up for efficiently processing scene-related and motion-related crowd information inherited from raw frames and the compound descriptor instances. The novelty of the work stemmed from the creation of the spatio-temporal cuboids in online (or near real-time) manner through dynamically extracting local feature tracklets within the temporal space while handling the foreground region-of-interests (i.e., moving targets) through the exploration of Gaussian Mixture Model in the spatial space. Hence, the extracted foreground blocks can effectively eliminate irrelevant backgrounds and noises from the live streams for reducing the computational costs in the subsequent detecting phases. The devised compound feature descriptor, named as spatio-temporal feature descriptor (STFD), is capable of characterizing the crowd attributes through the measures such as collectiveness, stability, conflict and density in each online generated spatio-temporal cuboid. A STFD instance registers not only the dynamic variation of the targeted crowd over time based on local feature tracklets, but also the interaction information of neighborhoods within a crowd, e.g., the interaction force through the K-nearest neighbor (K-NN) analysis. The DCCNN developed in this research enables online identification of suspicious crowd behaviors based on analyzing the live-feed images and their STFD instances. The proposed model has been developed and evaluated against benchmarking techniques and databases. Experimental results have shown substantial improvements in terms of detection accuracy and efficiency for online crowd abnormal behavior identification.

Towards Intelligent Crowd Behavior Understanding Through the STFD Descriptor Exploration

Realizing the automated and online detection of crowd anomalies from surveillance CCTVs is a research-intensive and application-demanding task. This research proposes a novel technique for detecting crowd abnormalities through analyzing the spatial and temporal features of input video signals. This integrated solution defines an image descriptor (named spatio-temporal feature descriptor—STFD) that reflects the global motion pattern of crowds over time. A designed convolutional neural network (CNN) has then been adopted to classify dominant or large-scale crowd abnormal behaviors. The work reported has focused on: (1) detecting moving objects in online (or near real-time) manner through spatio-temporal segmentations of crowds identified by the similarity of group trajectory structures in the temporal space and the foreground blocks based on the Gaussian mixture model in the spatial space; (2) dividing multiple clustered groups based on the spectral clustering methods through treating image pixels from segmented regions as dynamic particles; (3) creating STFD descriptor instances by calculating corresponding attributes such as collectiveness, stability, conflict and crowd density for individuals (particles) in the corresponding groups; (4) inputting generated STFD descriptor instances into the devised CNN to detect suspicious crowd behaviors. For the test and evaluation of the devised models and techniques, the PETS database has been selected as the primary experimental data sets. Results against benchmarking models and systems have shown promising advancements of this novel approach in terms of accuracy and efficiency for crowd anomaly detection.

Histograms of Optical Flow Orientation and Magnitude and Entropy to Detect Anomalous Events in Videos

This paper presents an approach for detecting anomalous events in videos with crowds. The main goal is to recognize patterns that might lead to an anomalous event. An anomalous event might be characterized by the deviation from the normal or usual, but not necessarily in an undesirable manner, e.g., an anomalous event might just be different from normal but not a suspicious event from the surveillance point of view. One of the main challenges of detecting such events is the difficulty to create models due to their unpredictability and their dependency on the context of the scene. Based on these challenges, we present a model that uses general concepts, such as orientation, velocity, and entropy to capture anomalies. Using such a type of information, we can define models for different cases and environments. Assuming images captured from a single static camera, we propose a novel spatiotemporal feature descriptor, called histograms of optical flow orientation and magnitude and entropy , based on optical flow information. To determine the normality or abnormality of an event, the proposed model is composed of training and test steps. In the training, we learn the normal patterns. Then, during test, events are described and if they differ significantly from the normal patterns learned, they are considered as anomalous. The experimental results demonstrate that our model can handle different situations and is able to recognize anomalous events with success. We use the well-known UCSD and Subway data sets and introduce a new data set, namely, Badminton.

Use of trajectory and spatiotemporal features for retrieval of videos with a prominent moving foreground object

This paper presents generalized spatiotemporal analysis and lookup tool (GESTALT), an unsupervised framework for content-based video retrieval. GESTALT takes a query video and retrieves “similar” videos from the database. Motion and dynamics of appearance (shape) patterns of a prominent moving foreground object are considered as the key components of the video content and captured using corresponding feature descriptors. GESTALT automatically segments the moving foreground object from the given query video shot and estimates the motion trajectory. A graph-based framework is used to explicitly capture the structural and kinematics property of the motion trajectory, while an improved version of an existing spatiotemporal feature descriptor is proposed to model the change in object shape and movement over time. A combined match cost is computed as a convex combination of the two match scores, using these two feature descriptors, which is used to rank-order the retrieved video shots. Effectiveness of GESTALT is shown using extensive experimentation, and comparative study with recent techniques exhibits its superiority.