Extract Spatiotemporal Features Research Articles

ST-LSTM: A Deep Learning Approach Combined Spatio-Temporal Features for Short-Term Forecast in Rail Transit

The short-term forecast of rail transit is one of the most essential issues in urban intelligent transportation system (ITS). Accurate forecast result can provide support for the forewarning of flow outburst and enables passengers to make an appropriate travel plan. Therefore, it is significant to develop a more accurate forecast model. Long short-term memory (LSTM) network has been proved to be effective on data with temporal features. However, it cannot process the correlation between time and space in rail transit. As a result, a novel forecast model combining spatio-temporal features based on LSTM network (ST-LSTM) is proposed. Different from other forecast methods, ST-LSTM network uses a new method to extract spatio-temporal features from the data and combines them together as the input. Compared with other conventional models, ST-LSTM network can achieve a better performance in experiments.

Deep learning and process understanding for data-driven Earth system science.

Machine learning approaches are increasingly used to extract patterns and insights from the ever-increasing stream of geospatial data, but current approaches may not be optimal when system behaviour is dominated by spatial or temporal context. Here, rather than amending classical machine learning, we argue that these contextual cues should be used as part of deep learning (an approach that is able to extract spatio-temporal features automatically) to gain further process understanding of Earth system science problems, improving the predictive ability of seasonal forecasting and modelling of long-range spatial connections across multiple timescales, for example. The next step will be a hybrid modelling approach, coupling physical process models with the versatility of data-driven machine learning.

Attentive models in vision: Computing saliency maps in the deep learning era

Estimating the focus of attention of a person looking at an image or a video is a crucial step which can enhance many vision-based inference mechanisms: image segmentation and annotation, video captioning, autonomous driving are some examples. The early stages of the attentive behavior are typically bottom-up; reproducing the same mechanism means to find the saliency embodied in the images, i.e. which parts of an image pop out of a visual scene. This process has been studied for decades in neuroscience and in terms of computational models for reproducing the human cortical process. In the last few years, early models have been replaced by deep learning architectures, that outperform any early approach compared against public datasets. In this paper, we propose a discussion on why convolutional neural networks (CNNs) are so accurate in saliency prediction. We present our DL architectures which combine both bottom-up cues and higher-level semantics, and incorporate the concept of time in the attentional process through LSTM recurrent architectures. Eventually, we present a video-specific architecture based on the C3D network, which can extracts spatio-temporal features by means of 3D convolutions to model task-driven attentive behaviors. The merit of this work is to show how these deep networks are not mere brute-force methods tuned on massive amount of data, but represent well-defined architectures which recall very closely the early saliency models, although improved with the semantics learned by human ground-truth.

Spatiotemporal Feature Descriptor for Micro-Expression Recognition Using Local Cube Binary Pattern

Micro-expression recognition has been an active research area in recent years, it plays an important role in psychology and public security. Due to the aspects of short duration and subtle movement, it is challenging to extract spatiotemporal features of micro-expressions. The existing methods only extract features in the three-dimensional orthogonal plane and fail to make full use of that information. To solve this problem, we propose a new Local Cubes Binary Patterns (LCBP) method for micro-expression recognition. LCBP is cascaded by the motion information LCBP direction , the amplitude information LCBP amplitudes , and the spatial information LCBP3D to obtain the spatiotemporal features. The advantage of LCBP is its ability to preserve the spatiotemporal information and the low feature dimension. Furthermore, to increase the discrimination of features in micro-expression sequences, we apply a differential calculation energy map to find regions of interest (ROI) for getting a weighted energy map. The final micro-expression feature acquired by fusing the LCBP features and the weighted energy map are classified through the Support Vector Machine (SVM). We evaluate the proposed method on four published micro-expression databases including SMIC, CASME, CASME2, SAMM. Experimental results demonstrate that our proposed method achieves promising performance for micro-expression recognition.

Three-dimensional convolutional restricted Boltzmann machine for human behavior recognition from RGB-D video

This paper provides a novel approach for recognizing human behavior from RGB-D video data. The three-dimensional convolutional restricted Boltzmann machine (3DCRBM) is proposed which can extract features from the raw RGB-D data. In a physical model, the 3DCRBM differs from the restricted Boltzmann machine (RBM) as its weights are shared among all locations in the input and preserving spatial locality. Adjacent frames of the RGB image and the corresponding adjacent frames of the depth image are set as the input of 3DCRBM. Then, multiple 3D convolutional kernels can be applied to these four frames to extract spatio-temporal features. In the experiment of human behavior recognition, the deep belief network (DBN) is established by a layer of 3DCRBM network, convolutional neural network (CNN), and back propagation (BP) network. 3DCRBM is adapted for unsupervised training and getting a feature, while CNN and BP are used for supervised training and classifying the human behavior. The experiment results demonstrate that the correct differentiation rate of 95.7% is achieved, so the effectiveness of our approach could be validated.

A spatiotemporal model of video quality assessment via 3D gradient differencing

Some of the video quality assessment (VQA) metrics assess the video quality using spatial and temporal features separately, however, the video distortions are highly related to spatial and temporal content simultaneously. The accuracy of VQA may be reduced by separated spatial and temporal aspects. The distortion of video is under the twin impacts of spatial and temporal distortions. 3D gradient is a spatiotemporal structural capturer to extract both spatial and temporal features at the same time. Hence, this paper proposed a novel VQA metric via spatiotemporal 3D gradient differencing to describe the degradation of video quality effectively. Firstly, 3D gradient is constructed to extract spatiotemporal feature of video sequences. And then the differencing of 3D gradients between the reference and the distorted video is used to predict perceptual local quality of frames. Sequence pooling strategy is applied to assemble the frame quality indices into the score of video quality. Experimental results show that the proposed metric has a good consistency with the subjective perception and performs better than traditional metrics.

Large-Scale Gesture Recognition With a Fusion of RGB-D Data Based on Saliency Theory and C3D Model

Gesture recognition has raised wide attention in computer vision owing to its many applications. However, the task of video-based large-scale gesture recognition yet faces many challenges, since many gesture-irrelevant factors like the background may disturb the recognition accuracy. To better recognize gestures with large-scale videos, we propose a method based on RGB-D data in this paper, where the “RGB-D” means RGB and depth data captured simultaneously by specific devices like Kinect. To learn gesture details better, we first use an adaptive frame unification strategy to unify the frame number of inputs, and then the RGB and depth data are sent to the C3D model to extract spatiotemporal features, respectively. In order to alleviate the interference of gesture-irrelevant factors, the saliency theory is also employed to generate auxiliary data. Next the features of these data are combined to boost the performance, which can also avoid unreasonable synthetic data, since the dimension of C3D features is uniform. Finally the performances of several classifiers are tested and the best one of SVM classifier is selected to output the ultimate accuracy. Our approach achieves 52.04% and 59.43% accuracy on the validation and testing subset of the Chalearn LAP IsoGD, respectively, both of which outperform our results in the chalearn LAP Large-scale Gesture Recognition Challenge as reported in ICPR 2016.

A Time-Distributed Spatiotemporal Feature Learning Method for Machine Health Monitoring with Multi-Sensor Time Series.

Data-driven methods with multi-sensor time series data are the most promising approaches for monitoring machine health. Extracting fault-sensitive features from multi-sensor time series is a daunting task for both traditional data-driven methods and current deep learning models. A novel hybrid end-to-end deep learning framework named Time-distributed ConvLSTM model (TDConvLSTM) is proposed in the paper for machine health monitoring, which works directly on raw multi-sensor time series. In TDConvLSTM, the normalized multi-sensor data is first segmented into a collection of subsequences by a sliding window along the temporal dimension. Time-distributed local feature extractors are simultaneously applied to each subsequence to extract local spatiotemporal features. Then a holistic ConvLSTM layer is designed to extract holistic spatiotemporal features between subsequences. At last, a fully-connected layer and a supervised learning layer are stacked on the top of the model to obtain the target. TDConvLSTM can extract spatiotemporal features on different time scales without any handcrafted feature engineering. The proposed model can achieve better performance in both time series classification tasks and regression prediction tasks than some state-of-the-art models, which has been verified in the gearbox fault diagnosis experiment and the tool wear prediction experiment.

Spatio-temporal convolutional features with nested LSTM for facial expression recognition

Abstract In this paper, we propose a novel end-to-end architecture termed Spatio-Temporal Convolutional features with Nested LSTM (STC-NLSTM), which learns the muti-level appearance features and temporal dynamics of facial expressions in a joint fashion. More precisely, 3DCNN is used to extract spatio-temporal convolutional features from the image sequences that represent facial expressions, and the dynamics of expressions are modeled by Nested LSTM, which is actually coupled by two sub-LSTMs, saying T-LSTM and C-LSTM. Namely, T-LSTM is used to model the temporal dynamics of the spatio-temporal features in each convolutional layer, and C-LSTM is adopted to integrate the outputs of all T-LSTMs together so as to encode the multi-level features encoded in the intermediate layers of the network. We conduct experiments on four benchmark databases, CK+, Oulu-CASIA, MMI and BP4D, and the results show that the proposed method achieves a performance superior to the state-of-the-art methods.

Deep Motion Analysis for Epileptic Seizure Classification.

Visual motion clues such as facial expression and pose are natural semiology features which an epileptologist observes to identify epileptic seizures. However, these cues have not been effectively exploited for automatic detection due to the diverse variations in seizure appearance within and between patients. Here we present a multi-modal analysis approach to quantitatively classify patients with mesial temporal lobe (MTLE) and extra-temporal lobe (ETLE) epilepsy, relying on the fusion of facial expressions and pose dynamics. We propose a new deep learning approach that leverages recent advances in Convolutional Neural Networks (CNN) and Long Short-Term Memory (LSTM) networks to automatically extract spatiotemporal features from facial and pose semiology using recorded videos. A video dataset from 12 patients with MTLE and 6 patients with ETLEin an Australian hospital has been collected for experiments. Our experiments show that facial semiology and body movements can be effectively recognized and tracked, and that they provide useful evidence to identify the type of epilepsy. A multi-fold cross-validation of the fusion model exhibited an average test accuracy of 92.10%, while a leave-one-subject-out cross-validation scheme, which is the first in the literature, achieves an accuracy of 58.49%. The proposed approach is capable of modelling semiology features which effectively discriminate between seizures arising from temporal and extra-temporal brain areas. Our approach can be used as a virtual assistant, which will save time, improve patient safety and provide objective clinical analysis to assist with clinical decision making.

Anomaly detection in spatiotemporal data via regularized non-negative tensor analysis

Anomaly detection in multidimensional data is a challenging task. Detecting anomalous mobility patterns in a city needs to take spatial, temporal, and traffic information into consideration. Although existing techniques are able to extract spatiotemporal features for anomaly analysis, few systematic analysis about how different factors contribute to or affect the anomalous patterns has been proposed. In this paper, we propose a novel technique to localize spatiotemporal anomalous events based on tensor decomposition. The proposed method employs a spatial-feature-temporal tensor model and analyzes latent mobility patterns through unsupervised learning. We first train the model based on historical data and then use the model to capture the anomalies, i.e., the mobility patterns that are significantly different from the normal patterns. The proposed technique is evaluated based on the yellow-cab dataset collected from New York City. The results show several interesting latent mobility patterns and traffic anomalies that can be deemed as anomalous events in the city, suggesting the effectiveness of the proposed anomaly detection method.

3D Panoramic Virtual Reality Video Quality Assessment Based on 3D Convolutional Neural Networks

Virtual reality (VR), a new type of simulation and interaction technology, has aroused widespread attention and research interest. It is necessary to evaluate the VR quality and provide a standard for the rapidly developing technology. To the best of our knowledge, a few researchers have built benchmark databases and designed related algorithms, which has hindered the further development of the VR technology. In this paper, a free available data set (VRQ-TJU) for VR quality assessment is proposed with subjective scores for each sample data. The validity for the designed database has been proved based on the traditional multimedia quality assessment metrics. In addition, an end-to-end 3-D convolutional neural network is introduced to predict the VR video quality without a referenced VR video. This method can extract spatiotemporal features and does not require using hand-crafted features. At the same time, a new score fusion strategy is designed based on the characteristics of the VR video projection process. Taking the pre-processed VR video patches as input, the network captures local spatiotemporal features and gets the score of every patch. Then, the new quality score fusion strategy is applied to get the final score. Such approach shows advanced performance on this database.

Probe data-driven travel time forecasting for urban expressways by matching similar spatiotemporal traffic patterns

Travel time is an effective measure of roadway traffic conditions. The provision of accurate travel time information enables travelers to make smart decisions about departure time, route choice and congestion avoidance. Based on a vast amount of probe vehicle data, this study proposes a simple but efficient pattern-matching method for travel time forecasting. Unlike previous approaches that directly employ travel time as the input variable, the proposed approach resorts to matching large-scale spatiotemporal traffic patterns for multi-step travel time forecasting. Specifically, the Gray-Level Co-occurrence Matrix (GLCM) is first employed to extract spatiotemporal traffic features. The Normalized Squared Differences (NSD) between the GLCMs of current and historical datasets serve as a basis for distance measurements of similar traffic patterns. Then, a screening process with a time constraint window is implemented for the selection of the best-matched candidates. Finally, future travel times are forecasted as a negative exponential weighted combination of each candidate’s experienced travel time for a given departure. The proposed approach is tested on Ring 2, which is a 32km urban expressway in Beijing, China. The intermediate procedures of the methodology are visualized by providing an in-depth quantitative analysis on the speed pattern matching and examples of matched speed contour plots. The prediction results confirm the desirable performance of the proposed approach and its robustness and effectiveness in various traffic conditions.

HOTS: A Hierarchy of Event-Based Time-Surfaces for Pattern Recognition.

This paper describes novel event-based spatio-temporal features called time-surfaces and how they can be used to create a hierarchical event-based pattern recognition architecture. Unlike existing hierarchical architectures for pattern recognition, the presented model relies on a time oriented approach to extract spatio-temporal features from the asynchronously acquired dynamics of a visual scene. These dynamics are acquired using biologically inspired frameless asynchronous event-driven vision sensors. Similarly to cortical structures, subsequent layers in our hierarchy extract increasingly abstract features using increasingly large spatio-temporal windows. The central concept is to use the rich temporal information provided by events to create contexts in the form of time-surfaces which represent the recent temporal activity within a local spatial neighborhood. We demonstrate that this concept can robustly be used at all stages of an event-based hierarchical model. First layer feature units operate on groups of pixels, while subsequent layer feature units operate on the output of lower level feature units. We report results on a previously published 36 class character recognition task and a four class canonical dynamic card pip task, achieving near 100 percent accuracy on each. We introduce a new seven class moving face recognition task, achieving 79 percent accuracy.This paper describes novel event-based spatio-temporal features called time-surfaces and how they can be used to create a hierarchical event-based pattern recognition architecture. Unlike existing hierarchical architectures for pattern recognition, the presented model relies on a time oriented approach to extract spatio-temporal features from the asynchronously acquired dynamics of a visual scene. These dynamics are acquired using biologically inspired frameless asynchronous event-driven vision sensors. Similarly to cortical structures, subsequent layers in our hierarchy extract increasingly abstract features using increasingly large spatio-temporal windows. The central concept is to use the rich temporal information provided by events to create contexts in the form of time-surfaces which represent the recent temporal activity within a local spatial neighborhood. We demonstrate that this concept can robustly be used at all stages of an event-based hierarchical model. First layer feature units operate on groups of pixels, while subsequent layer feature units operate on the output of lower level feature units. We report results on a previously published 36 class character recognition task and a four class canonical dynamic card pip task, achieving near 100 percent accuracy on each. We introduce a new seven class moving face recognition task, achieving 79 percent accuracy.

Dimensionality Reduction in Transient Simulations: A Diffusion Maps Approach

A mathematically rigorous framework combining diffusion map analysis techniques with simulation data from electromagnetic transient programs is proposed to characterize the collective dynamics of transient processes. The framework can be used to compress and analyze large and complex data sets obtained from transient simulations. Through this formulation, the structures that dominate the system response, and the global measurements that contribute most to this response can be directly calculated. The resulting coordinates can be correlated with the time scales of specific motion or be effectively used to extract spatiotemporal features of high-dimensional data. The application of the method is illustrated on both, electromagnetic transient simulations and transient stability data. Simulation studies reveal that diffusion-map data-analysis techniques generate accurate estimates of modal parameters even in the presence of measurement noise.

A Robust Approach for Action Recognition Based on Spatio-Temporal Features in RGB-D Sequences

Recognizing human action is attractive research topic in computer vision since it plays an important role on the applications such as human-computer interaction, intelligent surveillance, human actions retrieval system, health care, smart home, robotics and so on. The availability the low-cost Microsoft Kinect sensor, which can capture real-time high-resolution RGB and visual depth information, has opened an opportunity to significantly increase the capabilities of many automated vision based recognition tasks. In this paper, we propose new framework for action recognition in RGB-D video. We extract spatiotemporal features from RGB-D data that capture both visual, shape and motion information. Moreover, the segmentation technique is applied to present the temporal structure of action. Firstly, we use STIP to detect interest points both of RGB and depth channels. Secondly, we apply HOG3D descriptor for RGB channel and 3DS-HONV descriptor for depth channel. In addition, we also extract HOF2.5D from fusing RGB and Depth to capture human’s motion. Thirdly, we divide the video into segments and apply GMM to create feature vectors for each segment. So, we have three feature vectors (HOG3D, 3DS-HONV, and HOF2.5D) that represent for each segment. Next, the max pooling technique is applied to create a final vector for each descriptor. Then, we concatenate the feature vectors from the previous step into the final vector for action representation. Lastly, we use SVM method for classification step. We evaluated our proposed method on three benchmark datasets to demonstrate generalizability. And, the experimental results shown to be more accurate for action recognition compared to the previous works. We obtain overall accuracies of 93.5%, 99.16% and 89.38% with our proposed method on the UTKinect-Action, 3D Action Pairs and MSR-Daily Activity 3D dataset, respectively. These results show that our method is feasible and superior performance over the-state-of-the-art methods on these datasets.

Combining unsupervised learning and discrimination for 3D action recognition

Previous work on 3D action recognition has focused on using hand-designed features, either from depth videos or 2D videos. In this work, we present an effective way to combine unsupervised feature learning with discriminative feature mining. Unsupervised feature learning allows us to extract spatio-temporal features from unlabeled video data. With this, we can avoid the cumbersome process of designing feature extraction by hand. We propose an ensemble approach using a discriminative learning algorithm, where each base learner is a discriminative multi-kernel-learning classifier, trained to learn an optimal combination of joint-based features. Our evaluation includes a comparison to state-of-the-art methods on the MSRAction 3D dataset, where our method, abbreviated EnMkl, outperforms earlier methods. Furthermore, we analyze the efficiency of our approach in a 3D action recognition system.

Sequential Markov random fields for human body parts tracking

This paper presents a novel human body part tracker called sequential Markov random fields (SMRFs), which can be used to extract spatiotemporal features in human action recognition. Given a video sequence of human action in the monocular settings, SMRF can effectively detect the key spatiotemporal feature points on human body parts. We also develop efficient learning algorithms for the SMRF tracker using relaxation labeling (RL). Our results show that the SMRF tracker performs better than some state-of-the-art trackers for human action recognition.

Video-based Indoor Human Gait Recognition Using Depth Imaging and Hidden Markov Model: A Smart System for Smart Home

Smart homes that are capable of home healthcare and e-Health services are receiving much attention due to their potential for better care of the elderly and disabled in an indoor environment. Recently the Center for Sustainable Healthy Buildings at Kyung Hee University has developed a novel indoor human activity recognition methodology based on depth imaging of a user’s activities. This system utilizes Independent Component Analysis to extract spatiotemporal features from a series of depth silhouettes of various activities. To recognise the activities from the spatiotemporal features, trained Hidden Markov Models of the activities would be used. In this study, this technique has been extended to recognise human gaits (including normal and abnormal). Since this system could be of great significance for the caring of the elderly, to promote and preserve their health and independence, the gait recognition system would be considered a primary function of the smart system for smart homes. The indoor gait recognition system is trained to detect abnormal gait patterns and generate warnings. The system works in real-time and is aimed to be installed at smart homes. This paper provides the information for further development of the system for their application in the future.

A diffusion framework for detection of moving vehicles

We introduce a novel real-time algorithm for automatic acoustic-based vehicle detection. Commonly, surveillance systems for this task use a microphone that is placed in a target area. The recorded sounds are processed in order to detect vehicles as they pass by. The proposed algorithm uses the wavelet-packet transform in order to extract spatio-temporal characteristic features from the recordings. These features constitute a unique acoustic signature for each of the recordings. A more compact signature is derived by the application of the Diffusion Maps (DM) dimensionality reduction algorithm. A new recording is classified according to its compact acoustic signature in the DM reduced-dimension space. The signature is efficiently obtained via the Geometric Harmonics (GH) algorithm. The introduced algorithm is generic and can be applied to various signal types for solving different detection and classification problems.