Spatiotemporal Attention Model Research Articles

DPT‐tracker: Dual pooling transformer for efficient visual tracking

AbstractTransformer tracking always takes paired template and search images as encoder input and conduct feature extraction and target‐search feature correlation by self and/or cross attention operations, thus the model complexity will grow quadratically with the number of input images. To alleviate the burden of this tracking paradigm and facilitate practical deployment of Transformer‐based trackers, we propose a dual pooling transformer tracking framework, dubbed as DPT, which consists of three components: a simple yet efficient spatiotemporal attention model (SAM), a mutual correlation pooling Transformer (MCPT) and a multiscale aggregation pooling Transformer (MAPT). SAM is designed to gracefully aggregates temporal dynamics and spatial appearance information of multi‐frame templates along space‐time dimensions. MCPT aims to capture multi‐scale pooled and correlated contextual features, which is followed by MAPT that aggregates multi‐scale features into a unified feature representation for tracking prediction. DPT tracker achieves AUC score of 69.5 on LaSOT and precision score of 82.8 on TrackingNet while maintaining a shorter sequence length of attention tokens, fewer parameters and FLOPs compared to existing state‐of‐the‐art (SOTA) Transformer tracking methods. Extensive experiments demonstrate that DPT tracker yields a strong real‐time tracking baseline with a good trade‐off between tracking performance and inference efficiency.

Multistep traffic speed prediction: A sequence-to-sequence spatio-temporal attention model

Multistep traffic speed prediction plays a crucial role in alleviating road congestion and improving transport efficiency. In actual traffic networks, the spatio-temporal dependence among roads dynamically changes over time due to factors such as road conditions and unforeseen incidents, which brings great challenges to multistep traffic speed prediction. Additionally, multistep traffic speed prediction commonly faces the problem of error accumulation, resulting in a loss of prediction accuracy. To address these issues, we propose a Sequence-to-Sequence Spatio-Temporal Attention model (STSSTA) for multistep traffic speed prediction. Specifically, we build an adaptive tuning module to select road preferences and automatically obtain global information about the roads. Then we construct a Sequence-to-Sequence architecture for spatio-temporal feature learning and multistep traffic speed prediction. In particular, in the encoder, we design a Diffusion Graph Convolutional Network (DGCN) and combine it with a Gated Recurrent Unit (GRU) to effectively capture the complex spatio-temporal features within the traffic networks. In the decoder, we introduce a Recalling attention mechanism to alleviate the problem of local information loss caused by encoder compression, thereby reducing error accumulation in multistep prediction. Experiments on METR-LA and PeMS-BAY datasets demonstrate that STSSTA outperforms the baseline models in long-term prediction.

Imbalanced learning for wind turbine blade icing detection via spatio-temporal attention model with a self-adaptive weight loss function

Accurate blade icing detection of wind turbines is of great significance to avoid secondary damages or accidents. Recently, data-driven methods have attracted increasing attention due to the availability of supervisory control and data acquisition (SCADA) data with their low-cost and high volume. However, SCADA data usually present complex characteristics of high dimensionality and strong spatio-temporal correlations among different sensor variables and severe class imbalance. It is challenging to extract effective features for accurate and reliable icing detection from imbalanced SCADA data. To address these challenges, we propose an imbalanced learning method for blade icing detection. First, we propose a spatio-temporal attention model to extract and assign weights to temporal-spatial features based on their contributions to blade icing detection. Then, to address the class imbalance issue, we design a self-adaptive weight (SAW) loss function to divide SCADA data into multiple batches and then classify them during model training. Unlike conventional loss functions, our SAW loss can adaptively assign weights to each data category according to their numbers in the divided batches. The effectiveness of the proposed method is verified on real SCADA datasets. The results demonstrate our designed spatio-temporal attention and SAW loss achieved better classification results and superior self-adaptability.

An optimized task assignment framework based on crowdsourcing knowledge graph and prediction

In spatial crowdsourcing, both workers and tasks are added to the system on the fly, making it a real-time platform with complex circumstances. Most research has focused on optimizing the total number of assigned tasks, with little attention paid to factors such as worker preferences and task completion rate in a practical environment. To capture the complex spatiotemporal correlation and the latent supply–demand relationship, this work proposes a convolutional spatiotemporal attention model (Conv-STAN) and a cluster-based time-weighted voting method (CT-Voting) for predicting the future distribution of crowdsourcing entities. We develop a crowdsourcing knowledge graph (CKG) that reflects the semantic linkages between entities and provide an innovative and effective state representation approach (CKG2vec) for quantifying the preference to demonstrate the impact of a dynamically changing environment on worker preferences. In addition, a local subgraph incremental updating approach (LSG-IncrUpdate) is constructed to simulate the interaction between various entities in a crowdsourcing setup. Finally, we introduce a reinforcement learning framework (CKG-RLTA) that integrates prediction and the crowdsourcing knowledge graph to conduct task assignment. The experimental findings show that our approach is sensitive to workers’ preferences and improves task completion rate, even when those preferences are subject to change.

A Spatio-Temporal Attention Mechanism Based Approach for Remaining Useful Life Prediction of Turbofan Engine.

The time-series data generated by turbofan engines has a great degree of complexity and dynamics. At present, recurrent neural networks are commonly used to model and forecast the remaining useful life (RUL). The relationship of the sample data is not taken into account, and there are issues such as gradient explosion. In view of this, a spatio-temporal attention model is proposed, which comprehensively relates to the temporal association of data features and the hidden state of data features in space. At the same time, position coding is performed on the temporal relationship, avoiding the use of recurrent neural networks. Experimental results show that by combining the two dimensions, the predictive performance of the model is significantly improved. Compared with different methods on the four data sets of the commercial modular aerospace propulsion system simulation (C-MAPSS), the stability and prediction accuracy of the spatio-temporal attention model are better than that of alternative methods.

DESA: a novel hybrid decomposing-ensemble and spatiotemporal attention model for PM2.5 forecasting.

Exposure to fine particulate matter can easily lead to health issues. PM2.5 concentrations are associated with various spatiotemporal factors, which makes the prediction of PM2.5 concentrations still a challenging task. One of the reasons that makes the accurate prediction by statistical learning method difficult is severe fluctuations in input data. In addition, the abstraction method of space will also affect the prediction results. To address these important issues, a novel hybrid decomposing-ensemble and spatiotemporal attention (DESA) model is proposed to improve the prediction accuracy by decomposing the mode-mixed time series into single-mode series and automatically assign weights to the spatiotemporal factors. In our proposed framework, raw PM2.5 series are firstly decomposed into simple sub-series via the complete ensemble empirical mode decomposition (CEEMD) method. Then, to keep the results independent of the spatial abstraction method, a data-driven approach called multiscale spatiotemporal attention network is employed to extract spatiotemporal features from the sub-series. Finally, the predictions of each sub-series are processed separately and combined to obtain the final prediction results. The experimental results indicate that the proposed model achieved the better performance with RMSE of 11.15, 17.49, 24.84, and 26.93 for 6-, 12-, 24-, and 36-h forecasting, respectively. The proposed method is expected to be applied in fine prediction of air pollution and controlling programs and therefore provide decision support or useful guidance.

A Spatiotemporal Attention Model for Severe Precipitation Estimation

Quantitative precipitation estimation (QPE) is an essential task in meteorology and hydrology and is of great significance for disaster prevention and control. The starting point of QPE is to establish a point-by-point mapping relationship between atmospheric observations and rain gauges. Traditional methods called Z-R relationships fit the parameters in a given paradigm to perform QPE under meteorology prior guidance. Methods based on machine learning (ML) construct the QPE models from statistical views, which could benefit from large historical data. However, in operational applications, these methods are challenging to estimate severe precipitation accurately. The reason is that severe precipitation is usually caused by convective systems. The point-by-point QPEs only focus on fixed isolated points and are difficult to characterize convective systems that cause precipitation effectively. In this letter, a spatiotemporal attention model is proposed for one-hour QPE. For each pixel, the spatiotemporal attention guides the model to find and focus on the most worthy attention region at each moment instead of a definite isolated point, making the model view more flexible and insightful. In experiments, the radar data from 2015 to 2016 in North China are used to train and evaluate the model. Compared with other methods, the results show that the spatiotemporal attention model could effectively improve the accuracy of QPE, especially for intense precipitation. The case study also shows that the operation of our model is more consistent with meteorological perspectives.

Multi-Level Two-Stream Fusion-Based Spatio-Temporal Attention Model for Violence Detection and Localization

Detection of violent human behavior is necessary for public safety and monitoring. However, it demands constant human observation and attention in human-based surveillance systems, which is a challenging task. Autonomous detection of violent human behavior is therefore essential for continuous uninterrupted video surveillance. In this paper, we propose a novel method for violence detection and localization in videos using the fusion of spatio-temporal features and attention model. The model consists of Fusion Convolutional Neural Network (Fusion-CNN), spatio-temporal attention modules and Bi-directional Convolutional LSTMs (BiConvLSTM). The Fusion-CNN learns both spatial and temporal features by combining multi-level inter-layer features from both RGB and Optical flow input frames. The spatial attention module is used to generate an importance mask to focus on the most important areas of the image frame. The temporal attention part, which is based on BiConvLSTM, identifies the most significant video frames which are related to violent activity. The proposed model can also localize and discriminate prominent regions in both spatial and temporal domains, given the weakly supervised training with only video-level classification labels. Experimental results evaluated on different publicly available benchmarking datasets show the superior performance of the proposed model in comparison with the existing methods. Our model achieves the improved accuracies (ACC) of 89.1%, 99.1% and 98.15% for RWF-2000, HockeyFight and Crowd-Violence datasets, respectively. For CCTV-FIGHTS dataset, we choose the mean average precision (mAp) performance metric and our model obtained 80.7% mAp.

Spatial–Temporal Attention Two-Stream Convolution Neural Network for Smoke Region Detection

Smoke detection is of great significance for fire location and fire behavior analysis in a fire video surveillance system. Smoke image classification methods based on a deep convolution network have achieved high accuracy. However, the combustion of different types of fuel can produce smoke with different colors, such as black smoke, grey smoke, and white smoke. Additionally, the diffusion characteristic of smoke can lead to transparent smoke regions accompanied by colors and textures of background objects. Therefore, compared with smoke image classification, smoke region detection is a challenging task. This paper proposes a two-stream convolutional neural network based on spatio-temporal attention mechanism for smoke region segmentation (STCNNsmoke). The spatial stream extracts spatial features of foreground objects using the semi-supervised ranking model. The temporal stream uses optical flow characteristics to represent the dynamic characteristics of smoke such as diffusion and flutter features. Specifically, the spatio-temporal attention mechanism is presented to fuse the spatial and temporal characteristics of smoke and pay more attention to the moving regions with smoke colors and textures by predicting attention weights of channels. Furthermore, the spatio-temporal attention model improves the channel response of smoke-moving regions for the segmentation of complete smoke regions. The proposed method is evaluated and analyzed from multiple perspectives such as region detection accuracy and anti-interference. The experimental results showed that the proposed method significantly improved the ability of segmenting thin smoke and small smoke.

Cross-scene foreground segmentation with supervised and unsupervised model communication

In this paper11Dong Liang, Bin Kang and Xinyu Liu contributed equally to this work., we investigate cross-scene video foreground segmentation via supervised and unsupervised model communication. Traditional unsupervised background subtraction methods often face the challenging problem of updating the statistical background model online. In contrast, supervised foreground segmentation methods, such as those that are based on deep learning, rely on large amounts of training data, thereby limiting their cross-scene performance. Our method leverages segmented masks from a cross-scene trained deep model (spatio-temporal attention model (STAM), pyramid scene parsing network (PSPNet), or DeepLabV3+) to seed online updates for the statistical background model (CPB), thereby refining the foreground segmentation. More flexible than methods that require scene-specific training and more data-efficient than unsupervised models, our method outperforms state-of-the-art approaches on CDNet2014, WallFlower, and LIMU according to our experimental results. The proposed framework can be integrated into a video surveillance system in a plug-and-play form to realize cross-scene foreground segmentation.

Pain-attentive network: a deep spatio-temporal attention model for pain estimation

In the video surveillance of medical institutions, pain intensity is a significant clue to the state of patients. Of late, some approaches leverage various spatio-temporal methods to capture the dynamic pain information of videos for accomplishing pain estimation automatically. However, there is still a challenge in the spatio-temporal saliency, which means pain is always reflected in some important regions of informative image frames in a video sequence. To this end, we propose a deep spatio-temporal attention model called as Pain-Attentive Network (PAN), which pays more attention on the saliency in the extraction of dynamic features. PAN consists of two subnetworks: spatial and temporal subnetwork. Especially, in spatial subnetwork, a proposed spatial attention module is embedded to make the spatial feature extraction more targeted. Also, a devised temporal attention module is inserted in temporal subnetwork, so that the temporal features focus on informative image frames. Extensive experiment results on the UNBC-McMaster Shoulder Pain database show that our proposed PAN achieves compelling performances. In addition, to evaluate the generalization, we report competitive results of our proposed method in the Remote Collaborative and Affective database.

An Improved Attention-Based Spatiotemporal-Stream Model for Action Recognition in Videos

Action recognition is an important yet challenging task in computer vision. Attention mechanism not only tells where to focus but when to focus. It plays a key role in extracting discriminative spatial and temporal features for solving the task. In this paper, we propose an improved spatiotemporal attention model based on the two-stream structure to recognize the different actions in videos. Specifically, we first extract the intra-frame spatial features and inter-frame optical flow features for each video data. Then we implement an effective attention module, which sequentially infers attention maps along three separate dimensions: channel, spatial and temporal. After adaptive feature refinement based on the attention maps, we perform a temporal pooling process to squeeze the temporal dimension. Then, these achieved spatial and temporal features are fed into the spatial LSTM and temporal LSTM, respectively. Finally, we fuse the spatial feature, temporal feature and two-stream fusion feature to classify the actions in videos. Additionally, we also collect and construct a new Ping-Pong action dataset for subsequent human-robot interaction task from YouTube. It contains 2400 labeled videos for 4 categories. We compare with other action recognition algorithms and validate the feasibility and effectiveness of the proposed method on Ping-Pong action dataset and HMDB51 dataset.

Spatio-Temporal Attention Model for Foreground Detection in Cross-Scene Surveillance Videos.

Foreground detection is an important theme in video surveillance. Conventional background modeling approaches build sophisticated temporal statistical model to detect foreground based on low-level features, while modern semantic/instance segmentation approaches generate high-level foreground annotation, but ignore the temporal relevance among consecutive frames. In this paper, we propose a Spatio-Temporal Attention Model (STAM) for cross-scene foreground detection. To fill the semantic gap between low and high level features, appearance and optical flow features are synthesized by attention modules via the feature learning procedure. Experimental results on CDnet 2014 benchmarks validate it and outperformed many state-of-the-art methods in seven evaluation metrics. With the attention modules and optical flow, its F-measure increased and respectively. The model without any tuning showed its cross-scene generalization on Wallflower and PETS datasets. The processing speed was 10.8 fps with the frame size 256 by 256.

StagNet: An Attentive Semantic RNN for Group Activity and Individual Action Recognition

In real life, group activity recognition plays a significant and fundamental role in a variety of applications, e.g. sports video analysis, abnormal behavior detection, and intelligent surveillance. In a complex dynamic scene, a crucial yet challenging issue is how to better model the spatio-temporal contextual information and inter-person relationship. In this paper, we present a novel attentive semantic recurrent neural network (RNN), namely, stagNet, for understanding group activities and individual actions in videos, by combining the spatio-temporal attention mechanism and semantic graph modeling. Specifically, a structured semantic graph is explicitly modeled to express the spatial contextual content of the whole scene, which is further incorporated with the temporal factor through structural-RNN. By virtue of the “factor sharing” and “message passing” mechanisms, our stagNet is capable of extracting discriminative and informative spatio-temporal representations and capturing inter-person relationships. Moreover, we adopt a spatio-temporal attention model to focus on key persons/frames for improved recognition performance. Besides, a body-region attention and a global-part feature pooling strategy are devised for individual action recognition. In experiments, four widely-used public datasets are adopted for performance evaluation, and the extensive results demonstrate the superiority and effectiveness of our method.

An End-to-End Spatio-Temporal Attention Model for Human Action Recognition from Skeleton Data

Human action recognition is an important task in computer vision. Extracting discriminative spatial and temporal features to model the spatial and temporal evolutions of different actions plays a key role in accomplishing this task. In this work, we propose an end-to-end spatial and temporal attention model for human action recognition from skeleton data. We build our model on top of the Recurrent Neural Networks (RNNs) with Long Short-Term Memory (LSTM), which learns to selectively focus on discriminative joints of skeleton within each frame of the inputs and pays different levels of attention to the outputs of different frames. Furthermore, to ensure effective training of the network, we propose a regularized cross-entropy loss to drive the model learning process and develop a joint training strategy accordingly. Experimental results demonstrate the effectiveness of the proposed model, both on the small human action recognition dataset of SBU and the currently largest NTU dataset.

Multimodal Saliency and Fusion for Movie Summarization Based on Aural, Visual, and Textual Attention

Multimodal streams of sensory information are naturally parsed and integrated by humans using signal-level feature extraction and higher level cognitive processes. Detection of attention-invoking audiovisual segments is formulated in this work on the basis of saliency models for the audio, visual, and textual information conveyed in a video stream. Aural or auditory saliency is assessed by cues that quantify multifrequency waveform modulations, extracted through nonlinear operators and energy tracking. Visual saliency is measured through a spatiotemporal attention model driven by intensity, color, and orientation. Textual or linguistic saliency is extracted from part-of-speech tagging on the subtitles information available with most movie distributions. The individual saliency streams, obtained from modality-depended cues, are integrated in a multimodal saliency curve, modeling the time-varying perceptual importance of the composite video stream and signifying prevailing sensory events. The multimodal saliency representation forms the basis of a generic, bottom-up video summarization algorithm. Different fusion schemes are evaluated on a movie database of multimodal saliency annotations with comparative results provided across modalities. The produced summaries, based on low-level features and content-independent fusion and selection, are of subjectively high aesthetic and informative quality.