Spatial-Temporal Attention Network Research Articles

Adaptive Graph Spatial-Temporal Attention Networks for long lead ENSO prediction

El Niño-Southern Oscillation (ENSO) is a crucial factor in global climate change, which can lead to disasters such as floods, droughts, and heavy rainfall worldwide. Although accurate long-lead prediction of ENSO is essential, the intricate time-varying spatial correlations make it challenging. In this paper, we propose a novel deep learning architecture, called Adaptive Graph Spatial-Temporal Attention Network (AGSTAN), to model the extensive spatial–temporal interactions in ENSO. We first develop a dynamic spatial graph module with self-attention mechanism, wherein the structure is adapted to the characteristics of the evolving states. Furthermore, we introduce the long-term temporal sequence module to capture long-range dependency between time-series outputs and inputs, and generate accurate long lead predictions with only one forward step. To validate its efficacy and robustness, we conducted experiments on the simulated and historical climate datasets. Results showcase that AGSTAN possesses superior performance in ENSO prediction, outperforming contemporary deep learning models and boasting a 23-month forecasting horizon with heightened correlation and minimized bias.

DSTAN: A Deformable Spatial-temporal Attention Network with Bidirectional Sequence Feature Refinement for Speckle Noise Removal in Thyroid Ultrasound Video.

Thyroid ultrasound video provides significant value for thyroid diseases diagnosis, but the ultrasound imaging process is often affected by the speckle noise, resulting in poor quality of the ultrasound video. Numerous video denoising methods have been proposed to remove noise while preserving texture details. However, existing methods still suffer from the following problems: (1) relevant temporal features in the low-contrast ultrasound video cannot be accurately aligned and effectively aggregated by simple optical flow or motion estimation, resulting in the artifacts and motion blur in the video; (2) fixed receptive field in spatial features integration lacks the flexibility of aggregating features in the global region of interest and is susceptible to interference from irrelevant noisy regions. In this work, we propose a deformable spatial-temporal attention denoising network to remove speckle noise in thyroid ultrasound video. The entire network follows the bidirectional feature propagation mechanism to efficiently exploit the spatial-temporal information of the whole video sequence. In this process, two modules are proposed to address the above problems: (1) a deformable temporal attention module (DTAM) is designed after optical flow pre-alignment to further capture and aggregate relevant temporal features according to the learned offsets between frames, so that inter-frame information can be better exploited even with the imprecise flow estimation under the low contrast of ultrasound video; (2) a deformable spatial attention module (DSAM) is proposed to flexibly integrate spatial features in the global region of interest through the learned intra-frame offsets, so that irrelevant noisy information can be ignored and essential information can be precisely exploited. Finally, all these refined features are rectified and merged through residual convolution blocks to recover the clean video frames. Experimental results on our thyroid ultrasound video (US-V) dataset and the DDTI dataset demonstrate that our proposed method exceeds 1.2 1.3dB on PSNR and has clearer texture detail compared to other state-of-the-art methods. In the meantime, the proposed model can also assist thyroid nodule segmentation methods to achieve more accurate segmentation effect, which provides an important basis for thyroid diagnosis. In the future, the proposed model can be improved and extended to other medical image sequence datasets, including CT and MRI slice denoising. The code and datasets are provided at https://github.com/Meta-MJ/DSTAN .

Physics Guided Deep Learning-Based Model for Short-Term Origin–Destination Demand Prediction in Urban Rail Transit Systems Under Pandemic

Accurate origin–destination (OD) demand prediction is crucial for the efficient operation and management of urban rail transit (URT) systems, particularly during a pandemic. However, this task faces several limitations, including real-time availability, sparsity, and high-dimensionality issues, and the impact of the pandemic. Consequently, this study proposes a unified framework called the physics-guided adaptive graph spatial–temporal attention network (PAG-STAN) for metro OD demand prediction under pandemic conditions. Specifically, PAG-STAN introduces a real-time OD estimation module to estimate real-time complete OD demand matrices. Subsequently, a novel dynamic OD demand matrix compression module is proposed to generate dense real-time OD demand matrices. Thereafter, PAG-STAN leverages various heterogeneous data to learn the evolutionary trend of future OD ridership during the pandemic. Finally, a masked physics-guided loss function (MPG-loss function) incorporates the physical quantity information between the OD demand and inbound flow into the loss function to enhance model interpretability. PAG-STAN demonstrated favorable performance on two real-world metro OD demand datasets under the pandemic and conventional scenarios, highlighting its robustness and sensitivity for metro OD demand prediction. A series of ablation studies were conducted to verify the indispensability of each module in PAG-STAN.

Multi-object Tracking with Spatial-Temporal Tracklet Association

Recently, the tracking-by-detection methods have achieved excellent performance in Multi-Object Tracking (MOT), which focuses on obtaining a robust feature for each object and generating tracklets based on feature similarity. However, they are confronted with two issues: (1) unstable features in short-term occlusion and (2) insufficient matching in long-term occlusion. Specifically, the unstable feature is caused by the appearance variation under occlusion, and the association with the current unstable feature will lead to insufficient matching in long-term occlusion. To address the above issues, we propose a two-stage tracklet-level association method, Spatial-Temporal Tracklet Association (STTA), to effectively combine spatial-temporal context between feature extraction and data association. In the first stage, we propose the Tracklet-guided Spatial-Temporal Attention network (TSTA) to generate robust and stable features. Specifically, TSTA captures spatial-temporal context to obtain the most salient regions between the current and previous clips. In the second stage, we design the Bi-Tracklet Spatial-Temporal association (BTST) module to fully exploit the spatial-temporal context in data association. Specifically, we leverage BTST to merge different tracklets into long-term trajectories by jointly learning visual feature and spatial-temporal context and designing a bidirectional interpolation to recover the missed objects between matched tracklets. Extensive experiments of public and private detections on four benchmarks demonstrate the robustness of STTA. Furthermore, the proposed method is a model-agnostic method, which can be plugged and played with existing methods to boost their performance, e.g., obtain 11.0%, 10.1%, 2.9%, 3.2%, and 7.8% improvement on IDF1 in the MOT16 validation dataset for Tracktor, CenterTrack, Deepsort, JDE, and CTracker, respectively.

Hyperspectral Image Change Detection Based on Gated Spectral–Spatial–Temporal Attention Network With Spectral Similarity Filtering

Hyperspectral Image Change Detection Based on Gated Spectral–Spatial–Temporal Attention Network With Spectral Similarity Filtering

A multi-graph spatial-temporal attention network for air-quality prediction

Air pollution poses a grave threat to human health and everyday life. Accurate air-quality prediction plays a crucial role in effectively preventing and controlling air pollution. A multi-graph spatial-temporal attention network is proposed to predict the air quality in a given area by analyzing the interconnections between stations and the individual characteristics of each station. Firstly, this paper constructs multi-scale spatial-temporal graphs from spatial and temporal perspectives to effectively capture the interconnections between stations, thereby comprehensively understanding the spatial-temporal relationships among them. Secondly, incorporating the nonlinear temporal correlation of station data, this paper proposes a temporal multi-graph attention-fusion module to integrate information from both neighboring stations and the station itself. Finally, predictions are made using a spatial-temporal graph network. The experiments in this paper were based on air-quality data from Beijing and Tianjin, and the following experimental results were obtained. (1) The proposed model outperforms the baselines significantly in both single-step and multi-step prediction tasks. (2) The ablation study confirms that the graph constructed in this paper contributes to improving the performance of the model. (3) Comparing the performance of attention components, the proposed attention mechanism exhibits better performance than other attention mechanisms.

State Transition Graph-Based Spatial–Temporal Attention Network for Cell-Level Mobile Traffic Prediction

Mobile traffic prediction enables the efficient utilization of network resources and enhances user experience. In this paper, we propose a state transition graph-based spatial–temporal attention network (STG-STAN) for cell-level mobile traffic prediction, which is designed to exploit the underlying spatial–temporal dynamic information hidden in the historical mobile traffic data. Specifically, we first identify the semantic context information over different segments of the historical data by constructing the state transition graphs, which may reveal different patterns of random fluctuation. Then, based on the state transition graphs, a spatial attention extraction module using graph convolutional networks (GCNs) is designed to aggregate the spatial information of different nodes in the state transition graph. Moreover, a temporal extraction module is employed to capture the dynamic evolution and temporal correlation of the state transition graphs over time. Such a spatial–temporal attention network can be further integrated with a parallel long short-term memory (LSTM) module to improve the accuracy of mobile traffic prediction. Extensive experiments demonstrate that the STG-STAN can better exploit the spatial–temporal information hidden in the state transition graphs, achieving superior performance compared with several baselines.

Spatial–Temporal Attention Network for Depression Recognition from facial videos

Recent studies focus on the utilization of deep learning approaches to recognize depression from facial videos. However, these approaches have been hindered by their limited performance, which can be attributed to the inadequate consideration of global spatial–temporal relationships in significant local regions within faces. In this paper, we propose Spatial–Temporal Attention Depression Recognition Network (STA-DRN) for depression recognition to enhance feature extraction and increase the relevance of depression recognition by capturing the global and local spatial–temporal information. Our proposed approach includes a novel Spatial–Temporal Attention (STA) mechanism, which generates spatial and temporal attention vectors to capture the global and local spatial–temporal relationships of features. To the best of our knowledge, this is the first attempt to incorporate pixel-wise STA mechanisms for depression recognition based on 3D video analysis. Additionally, we propose an attention vector-wise fusion strategy in the STA module, which combines information from both spatial and temporal domains. We then design the STA-DRN by stacking STA modules ResNet-style. The experimental results on AVEC 2013 and AVEC 2014 show that our method achieves competitive performance, with mean absolute error/root mean square error (MAE/RMSE) scores of 6.15/7.98 and 6.00/7.75, respectively. Moreover, visualization analysis demonstrates that the STA-DRN responds significantly in specific locations related to depression. The code is available at: https://github.com/divertingPan/STA-DRN.

Predicting future production system bottlenecks with a graph neural network approach

Predicting throughput bottlenecks has received an increasing research attention in the recent years. Several statistical and machine learning models have been developed, with the objective of identifying future bottleneck locations for more efficient and cost-saving productions. However, bottleneck prediction remains a challenging task due to the complexity of manufacturing systems and the interactive effects from many impacting factors. This paper proposed the BSTAN (bottleneck spatial-temporal attention network) – an interpretable modeling framework to improve system bottlenecks prediction capability for complex manufacturing systems by: (1) characterizing line topological features using graph representations; (2) modeling spatial-temporal relationships and interactions between stations using graph attention network with gated recurrent units. In this proposed framework, the production line is first transformed into the representation of a weighted undirected graph to depict network structural information of a production line. An attention-based graph neural network with gated recurrent units is applied to capture both the temporal trends and station interactions to predict each station’s blockage and starvation. Future bottleneck stations are identified by analyzing the forecasted blockage and starvation distribution. To evaluate the model performance, an industrial case study is performed with data collected from a one-year production at an automotive powertrain assembly line. With the proper model construction, BSTAN has outperformed both statistical and machine learning benchmark models, achieving up to 15% lower root mean square error compared to the best benchmark method in predicting overall system blockage and starvation time.

Bi-STAN: bilinear spatial-temporal attention network for wearable human activity recognition

Bi-STAN: bilinear spatial-temporal attention network for wearable human activity recognition

Unified Spatial-Temporal Neighbor Attention Network for Dynamic Traffic Prediction

Traffic prediction plays an essential role in many real-world applications ranging from route planning to vehicular communications. The goal of making accurate prediction is challenging due to the dynamic and various spatial-temporal correlations in traffic network. Numerous methods have been proposed to capture the spatial-temporal correlations, however, there are two major challenges: <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i</i> ) The task of synchronously capturing dynamic and various spatial-temporal correlations has not been fully explored yet. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ii</i> ) Lack of an adaptive mechanism to model different influences caused by different external factors. To overcome these challenges, we propose a unified spatial-temporal attention network entitled USTAN. First, it constructs the spatial neighbor graph and the temporal neighbor array to describe the spatial and temporal correlations respectively. By expanding the spatial neighbors in temporal perspective according to the temporal neighbor array, all the spatial-temporal neighbors can be gathered together. Second, a single unified attention component is performed on all the spatial-temporal neighbors for capturing the spatial-temporal correlations all at once. In addition, a gated fusion module is designed to fuse the external factors adaptively to reduce predicting error. Extensive experiments on real-world traffic prediction tasks demonstrate the superiority of our proposed framework.

RPPG-Based Heart Rate Estimation Using Spatial-Temporal Attention Network

Remote photoplethysmography (rPPG) based on the computer vision technology is widely used to calculate the heart rate (HR) from facial videos. Existing rPPG techniques have been subject to some limitations [e.g., highly redundant spatial information, head movement noise, and region of interest (ROI) selection]. To address these limitations, this article introduces an effective spatial-temporal attention network. A temporal fusion module is first proposed to fully exploit the time-domain information, aiming to reduce the redundant video information and strengthen the association relationships of long-range videos. Furthermore, a spatial attention mechanism is designed in the backbone net to precisely target the skin ROIs. Finally, to assist the network in learning the weights between channels, we project the RGB images using the plane orthogonal to skin (POS) algorithm and add motion representation to complement physiological signals’ extraction. The extensive experiments on the public PURE, MMSE-HR, and UBFC-rPPG datasets demonstrate that our model achieves competitive results compared with other methods.

Weakly supervised spatial–temporal attention network driven by tracking and consistency loss for action detection

This study proposes a novel network model for video action tube detection. This model is based on a location-interactive weakly supervised spatial–temporal attention mechanism driven by multiple loss functions. It is especially costly and time consuming to annotate every target location in video frames. Thus, we first propose a cross-domain weakly supervised learning method with a spatial–temporal attention mechanism for action tube detection. In source domain, we trained a newly designed multi-loss spatial–temporal attention–convolution network on the source data set, which has both object location and classification annotations. In target domain, we introduced internal tracking loss and neighbor-consistency loss; we trained the network with the pre-trained model on the target data set, which only has inaccurate action temporal positions. Although this is a location-unsupervised method, its performance outperforms typical weakly supervised methods, and even shows comparable results with some recent fully supervised methods. We also visualize the activation maps, which reveal the intrinsic reason behind the higher performance of the proposed method.

A Dynamic Spatial-Temporal Attention Network for Early Anticipation of Traffic Accidents

The rapid advancement of sensor technologies and artificial intelligence are creating new opportunities for traffic safety enhancement. Dashboard cameras (dashcams) have been widely deployed on both human driving vehicles and automated driving vehicles. A computational intelligence model that can accurately and promptly predict accidents from the dashcam video will enhance the preparedness for accident prevention. The spatial-temporal interaction of traffic agents is complex. Visual cues for predicting a future accident are embedded deeply in dashcam video data. Therefore, the early anticipation of traffic accidents remains a challenge. Inspired by the attention behavior of humans in visually perceiving accident risks, this paper proposes a Dynamic Spatial-Temporal Attention (DSTA) network for the early accident anticipation from dashcam videos. The DSTA-network learns to select discriminative temporal segments of a video sequence with a Dynamic Temporal Attention (DTA) module. It also learns to focus on the informative spatial regions of frames with a Dynamic Spatial Attention (DSA) module. A Gated Recurrent Unit (GRU) is trained jointly with the attention modules to predict the probability of a future accident. The evaluation of the DSTA-network on two benchmark datasets confirms that it has exceeded the state-of-the-art performance. A thorough ablation study that assesses the DSTA-network at the component level reveals how the network achieves such performance. Furthermore, this paper proposes a method to fuse the prediction scores from two complementary models and verifies its effectiveness in further boosting the performance of early accident anticipation.

Joint Variation Learning of Fusion and Difference Features for Change Detection in Remote Sensing Images

Remote sensing (RS) image change detection (CD) is an earth observation technique for detecting surface changes in the same area during a period. With the rapid development of deep learning, various deep neural networks especially Siamese ones have been widely used in the field of CD. However, they have the deficiency of insufficient contextual information aggregation, resulting in false and missed detections, and it is difficult to refine the detection of change edges. To alleviate these problems and obtain more accurate results, we propose an efficient self-weighted spatial-temporal attention network (SSANet). In contrast to the Siamese structure, our network is a novel joint learning framework composed of fusion sub-network, difference sub-network, and decoder. Fusion sub-network is used to extract multiscale object features where we propose a multi-core channel-aligning attention (MCA) module to capture the long-range semantic information for multi-scale context aggregation. Difference sub-network is used to extract the difference variation features, where we propose a feature differential reconfiguration (FDR) module to learn the temporal change information. FDR can effectively filter change information and reconstruct features to improve the perception of changed regions. To better balance the MCA and FDR modules, an asymmetric weighting (AW) module is proposed in the decoder to self-weight the multi-scale features and generate the change map. Experiments demonstrate the efficiency of proposed sub-networks and modules, and the state-of-the-art performance of SSANet.

Spatial-temporal attention network for multistep-ahead forecasting of chlorophyll

The multistep-ahead prediction of chlorophyll provides an effective means for early warning of red tide. However, since multistep-ahead forecasting presents challenges, such as vague interactive relationships among ocean factors, long-term dependence modeling, and accumulative errors, existing methods mostly concentrate on the current time or one-step-ahead forecasting. In this paper, a hierarchical multistep-ahead forecasting model spatial-temporal attention network(STAN), which integrates the spatial context extractor network(SCE-net), long short-term memory network(LSTM), and the temporal attention mechanism, is proposed for the prediction of chlorophyll. In STAN, the input layer utilizes SCE-net to excavate relationships among ocean factors and generate high-level semantic via embedding factors into a continuous low-dimensional space. The middle layer applies LSTM to build the long-term dependencies of corresponding semantic representations. The output layer uses another LSTM with temporal attention to reduce accumulative errors and maintain temporal continuity. The attention can assign different weights to the middle layer’s hidden state and generate a context vector. Then the context vector and the final predicted value are considered as the current input for better forecasting. The buoy observation data of the Xiamen coastal area monitored in 2009–2011 is used to verify the efficiency of STAN. Experimental results prove that STAN outperforms the state-of-the-art methods of multistep-ahead prediction. When using 7 observation steps to forecast 15 steps, the MAPE of STAN is 0.3209, and the MAE is 0.1 lower than the values of the baselines approaches.

STA-Net: spatial-temporal attention network for video salient object detection

This paper conducts a systematic study on the role of spatial and temporal attention mechanism in the video salient object detection (VSOD) task. We present a two-stage spatial-temporal attention network, named STA-Net, which makes two major contributions. In the first stage, we devise a Multi-Scale-Spatial-Attention (MSSA) module to reduce calculation cost on non-salient regions while exploiting multi-scale saliency information. Such a sliced attention method offers an individual way to efficiently exploit the high-level features of the network with an enlarged receptive field. The second stage is to propose a Pyramid-Saliency-Shift-Aware (PSSA) module, which puts emphasis on the importance of dynamic object information since it offers a valid shift cue to confirm salient object and capture temporal information. Such a temporal detection module is able to encourage precise salient region detection. Exhaustive experiments show that the proposed STA-Net is effective for video salient object detection task, and achieves compelling performance in comparison with state-of-the-art.

Weather Forecasting Using Ensemble of Spatial-Temporal Attention Network and Multi-Layer Perceptron

Weather forecasting is a challenging task, which is especially suited for artificial intelligence due to the large amount of data involved. This paper proposed an end-to-end hybrid regression model, called Ensemble of Spatial-Temporal Attention Network and Multi-Layer Perceptron (E-STAN-MLP), to forecast surface temperature, humidity, wind speed, and wind direction at 24 automatic weather stations in Beijing. Combining the data from historical observations with the data from the numerical weather prediction (NWP) system, our proposed model give better results than the NWP system or previously reported algorithms. Our E-STAN-MLP model consists of two parts. One is to use the spatial-temporal attention based recurrent neural network to model the time series of meteorological elements. The other is a simple but efficient multi-layer perceptron architecture forecasts the regression value while ignoring time dependence. Results at each time stamp are integrated together using a step-wise fusion strategy. Moreover, we use a joint loss step integrating both the regression loss function and the classification loss function to simultaneously forecast the wind speed and direction. Experiments demonstrate that our proposed E-STAN-MLP model achieves state-of-the-art results in weather forecasting.

Video Question Answering via Knowledge-based Progressive Spatial-Temporal Attention Network

Visual Question Answering (VQA) is a challenging task that has gained increasing attention from both the computer vision and the natural language processing communities in recent years. Given a question in natural language, a VQA system is designed to automatically generate the answer according to the referenced visual content. Though there recently has been much intereset in this topic, the existing work of visual question answering mainly focuses on a single static image, which is only a small part of the dynamic and sequential visual data in the real world. As a natural extension, video question answering (VideoQA) is less explored. Because of the inherent temporal structure in the video, the approaches of ImageQA may be ineffectively applied to video question answering. In this article, we not only take the spatial and temporal dimension of video content into account but also employ an external knowledge base to improve the answering ability of the network. More specifically, we propose a knowledge-based progressive spatial-temporal attention network to tackle this problem. We obtain both objects and region features of the video frames from a region proposal network. The knowledge representation is generated by a word-level attention mechanism using the comment information of each object that is extracted from DBpedia. Then, we develop a question-knowledge-guided progressive spatial-temporal attention network to learn the joint video representation for video question answering task. We construct a large-scale video question answering dataset. The extensive experiments based on two different datasets validate the effectiveness of our method.

R-STAN: Residual Spatial-Temporal Attention Network for Action Recognition

Two-stream network architecture has the ability to capture temporal and spatial features from videos simultaneously and has achieved excellent performance on video action recognition tasks. However, there is a fair amount of redundant information in both temporal and spatial dimensions in videos, which increases the complexity of network learning. To solve this problem, we propose residual spatial-temporal attention network (R-STAN), a feed-forward convolutional neural network using residual learning and spatial-temporal attention mechanism for video action recognition, which makes the network focus more on discriminative temporal and spatial features. In our R-STAN, each stream is constructed by stacking residual spatial-temporal attention blocks (R-STAB), the spatial-temporal attention modules integrated in the residual blocks have the ability to generate attention-aware features along temporal and spatial dimensions, which largely reduce the redundant information. Together with the specific characteristic of residual learning, we are able to construct a very deep network for learning spatial-temporal information in videos. With the layers going deeper, the attention-aware features from the different R-STABs can change adaptively. We validate our R-STAN through a large number of experiments on UCF101 and HMDB51 datasets. Our experiments show that our proposed network combined with residual learning and spatial-temporal attention mechanism contributes substantially to the performance of video action recognition.