Spatiotemporal Attention Research Articles

A Hybrid End-to-End Spatiotemporal Attention Neural Network With Graph-Smooth Signals for EEG Emotion Recognition

A Hybrid End-to-End Spatiotemporal Attention Neural Network With Graph-Smooth Signals for EEG Emotion Recognition

Enhanced HAR using Dynamic STGAT

Action recognition has seen significant advancements with the integration of spatio-temporal representations, particularly leveraging skeleton-based models and cross-modal data fusion techniques. However, existing approaches face challenges in capturing long- range dependencies within the human body skeleton and effectively balancing features from diverse modalities. To address these limitations, a novel framework, the Dynamic Spatio-Temporal Graph Attention Transformer (D-STGAT), is proposed, which seamlessly integrates the strengths of dynamic graph attention mechanisms and transformer architectures for enhanced action recognition. The framework builds upon recent innovations in graph attention networks (GAT) and transformer models. First, the Spatial-Temporal Dynamic Graph Attention Network (ST-DGAT) is introduced, extending traditional GAT by incorporating a dynamic attention mechanism to capture spatial- temporal patterns within skeleton sequences. By reordering the weighted vector operations in GAT, the approach achieves a global approximate attention function, significantly enhancing its expressivity and capturing long-distance dependencies more effectively than static attention mechanisms. Furthermore, to address the challenges of cross-modal feature representation and fusion, the spatio-temporal Cross Attention Transformer (ST-CAT) is introduced. This model efficiently integrates spatio-temporal information from both video frames and skeleton sequences by employing a combination of full spatio-temporal attention (FAttn), zigzag spatio-temporal attention (ZAttn), and binary spatio-temporal attention (BAttn) modules. Through the proper arrangement of these modules within the transformer encoder and decoder, ST-CAT learns a multi-feature representation that effectively captures the intricate spatiotemporal dynamics inherent in action recognition tasks. Experimental results on the Penn- Action, NTU-RGB+D 60, and 120 datasets showcase the efficacy of the approach, yielding promising performance improvements over previous state-of-the-art methods. In summary, the proposed D-STGAT and ST-CAT frameworks offer novel solutions for action recognition tasks by leveraging dynamic graph attention mechanisms and transformer architectures to effectively capture and fuse spatiotemporal features from diverse modalities, leading to superior performance compared to existing approaches.

Multi-scale spatiotemporal attention network for neuron based motor imagery EEG classification

BackgroundIn recent times, the expeditious expansion of Brain-Computer Interface (BCI) technology in neuroscience, which relies on electroencephalogram (EEG) signals associated with motor imagery, has yielded outcomes that rival conventional approaches, notably due to the triumph of deep learning. Nevertheless, the task of developing and training a comprehensive network to extract the underlying characteristics of motor imagining EEG data continues to pose challenges. New methodThis paper presents a multi-scale spatiotemporal self-attention (SA) network model that relies on an attention mechanism. This model aims to classify motor imagination EEG signals into four classes (left hand, right hand, foot, tongue/rest) by considering the temporal and spatial properties of EEG. It is employed to autonomously allocate greater weights to channels linked to motor activity and lesser weights to channels not related to movement, thus choosing the most suitable channels. Neuron utilises parallel multi-scale Temporal Convolutional Network (TCN) layers to extract feature information in the temporal domain at various scales, effectively eliminating temporal domain noise. ResultsThe suggested model achieves accuracies of 79.26%, 85.90%, and 96.96% on the BCI competition datasets IV-2a, IV-2b, and HGD, respectively. Comparison with existing methodsIn terms of single-subject classification accuracy, this strategy demonstrates superior performance compared to existing methods. ConclusionThe results indicate that the proposed strategy exhibits favourable performance, resilience, and transfer learning capabilities.

MPSTAN: Metapopulation-Based Spatio-Temporal Attention Network for Epidemic Forecasting.

Accurate epidemic forecasting plays a vital role for governments to develop effective prevention measures for suppressing epidemics. Most of the present spatio-temporal models cannot provide a general framework for stable and accurate forecasting of epidemics with diverse evolutionary trends. Incorporating epidemiological domain knowledge ranging from single-patch to multi-patch into neural networks is expected to improve forecasting accuracy. However, relying solely on single-patch knowledge neglects inter-patch interactions, while constructing multi-patch knowledge is challenging without population mobility data. To address the aforementioned problems, we propose a novel hybrid model called metapopulation-based spatio-temporal attention network (MPSTAN). This model aims to improve the accuracy of epidemic forecasting by incorporating multi-patch epidemiological knowledge into a spatio-temporal model and adaptively defining inter-patch interactions. Moreover, we incorporate inter-patch epidemiological knowledge into both model construction and the loss function to help the model learn epidemic transmission dynamics. Extensive experiments conducted on two representative datasets with different epidemiological evolution trends demonstrate that our proposed model outperforms the baselines and provides more accurate and stable short- and long-term forecasting. We confirm the effectiveness of domain knowledge in the learning model and investigate the impact of different ways of integrating domain knowledge on forecasting. We observe that using domain knowledge in both model construction and the loss function leads to more efficient forecasting, and selecting appropriate domain knowledge can improve accuracy further.

Motion Deblurring via Spatial-Temporal Collaboration of Frames and Events

Motion deblurring can be advanced by exploiting informative features from supplementary sensors such as event cameras, which can capture rich motion information asynchronously with high temporal resolution. Existing event-based motion deblurring methods neither consider the modality redundancy in spatial fusion nor temporal cooperation between events and frames. To tackle these limitations, a novel spatial-temporal collaboration network (STCNet) is proposed for event-based motion deblurring. Firstly, we propose a differential-modality based cross-modal calibration strategy to suppress redundancy for complementarity enhancement, and then bimodal spatial fusion is achieved with an elaborate cross-modal co-attention mechanism to weight the contributions of them for importance balance. Besides, we present a frame-event mutual spatio-temporal attention scheme to alleviate the errors of relying only on frames to compute cross-temporal similarities when the motion blur is significant, and then the spatio-temporal features from both frames and events are aggregated with the custom cross-temporal coordinate attention. Extensive experiments on both synthetic and real-world datasets demonstrate that our method achieves state-of-the-art performance. Project website: https://github.com/wyang-vis/STCNet.

Short-term forecasting of surface solar incident radiation on edge intelligence based on AttUNet

Solar energy has emerged as a key industry in the field of renewable energy due to its universality, harmlessness, and sustainability. Accurate prediction of solar radiation is crucial for optimizing the economic benefits of photovoltaic power plants. In this paper, we propose a novel spatiotemporal attention mechanism model based on an encoder-translator-decoder architecture. Our model is built upon a temporal AttUNet network and incorporates an auxiliary attention branch to enhance the extraction of spatiotemporal correlation information from input images. And utilize the powerful ability of edge intelligence to process meteorological data and solar radiation parameters in real-time, adjust the prediction model in real-time, thereby improving the real-time performance of prediction. The dataset utilized in this study is sourced from the total surface solar incident radiation (SSI) product provided by the geostationary meteorological satellite FY4A. After experiments, the SSIM has been improved to 0.86. Compared with other existing models, our model has obvious advantages and has great prospects for short-term prediction of surface solar incident radiation.

LCDFormer: Long-term correlations dual-graph transformer for traffic forecasting

Traffic forecasting has always been a critical component of intelligent transportation systems. Due to the complexity of traffic prediction models, most research just only consider short-term historical data in the temporal dimension. However, learning temporal patterns necessitates the involvement of long-term historical data. Additionally, many models are limited in capturing spatial features by only considering short-distance spatial information connected to the target node. To solve these problems, we propose a dual-graph transformer, namely Long-term Correlations Dual-graph transFormer (LCDFormer), designed to capture long-term correlations and long-distance spatial correlations. It is entirely based on attention mechanisms, and as far as we know, there is limited research adopting this approach. Our work addresses this gap in the literature. In particular, we have devised a time aggregation method capable of consolidating long-term historical time series, concurrently addressing the impact of long-term temporal correlations while minimizing the influence of redundant data. Subsequently, we have introduced a novel spatio-temporal attention module that compresses spatial information to generate short-term input sequences while modeling dynamic long-range spatial correlations. We conducted extensive experiments with LCDFormer on five real-world traffic datasets. The results indicate that LCDFormer, considering long-term spatio-temporal correlations, is better able to learn the spatio-temporal patterns of traffic data. Compared to the current state-of-the-art baseline, our model has demonstrated outstanding predictive performance with a maximum improvement of 5.02% in mean absolute error, 4.33% in root mean square error and 7.32% in mean absolute percentage error. The source codes are available at: https://github.com/NanakiC/LCDFormer.

DPT‐tracker: Dual pooling transformer for efficient visual tracking

AbstractTransformer tracking always takes paired template and search images as encoder input and conduct feature extraction and target‐search feature correlation by self and/or cross attention operations, thus the model complexity will grow quadratically with the number of input images. To alleviate the burden of this tracking paradigm and facilitate practical deployment of Transformer‐based trackers, we propose a dual pooling transformer tracking framework, dubbed as DPT, which consists of three components: a simple yet efficient spatiotemporal attention model (SAM), a mutual correlation pooling Transformer (MCPT) and a multiscale aggregation pooling Transformer (MAPT). SAM is designed to gracefully aggregates temporal dynamics and spatial appearance information of multi‐frame templates along space‐time dimensions. MCPT aims to capture multi‐scale pooled and correlated contextual features, which is followed by MAPT that aggregates multi‐scale features into a unified feature representation for tracking prediction. DPT tracker achieves AUC score of 69.5 on LaSOT and precision score of 82.8 on TrackingNet while maintaining a shorter sequence length of attention tokens, fewer parameters and FLOPs compared to existing state‐of‐the‐art (SOTA) Transformer tracking methods. Extensive experiments demonstrate that DPT tracker yields a strong real‐time tracking baseline with a good trade‐off between tracking performance and inference efficiency.

Learning Sequence Descriptor Based on Spatio-Temporal Attention for Visual Place Recognition

Learning Sequence Descriptor Based on Spatio-Temporal Attention for Visual Place Recognition

Weather Interaction-Aware Spatio-Temporal Attention Networks for Urban Traffic Flow Prediction

As the cornerstone of intelligent transportation systems, accurate traffic prediction can reduce the pressure of urban traffic, reduce the cost of residents’ travel time, and provide a reference basis for urban construction planning. Existing traffic prediction methods focus on spatio-temporal dependence modeling, ignoring the influence of weather factors on spatio-temporal characteristics, and the prediction task has complexity and an uneven distribution in different spatio-temporal scenarios and weather changes. In view of this, we propose a weather interaction-aware spatio-temporal attention network (WST-ANet), in which we integrate feature models and dynamic graph modules in the encoder and decoder, and use a spatio-temporal weather interaction perception module for prediction. Firstly, the contextual semantics of the traffic flows are fused using a feature embedding module to improve the adaptability to weather drivers; then, an encoder–decoder is constructed by combining the Dynamic Graph Module and the WSTA Block, to extract spatio-temporal aggregated correlations in the roadway network; finally, the feature information of the encoder was weighted and aggregated using the cross-focusing mechanism, and attention was paid to the hidden state of the encoding. Traffic flow was predicted using the PeMS04 and PeMS08 datasets and compared with multiple typical baseline models. It was learned through extensive experiments that the accuracy evaluation result is the smallest in WST-ANet, which demonstrated the superiority of the proposed model. This can more accurately predict future changes in traffic in different weather conditions, providing decision makers with a basis for optimizing scenarios.

Multistep traffic speed prediction: A sequence-to-sequence spatio-temporal attention model

Multistep traffic speed prediction plays a crucial role in alleviating road congestion and improving transport efficiency. In actual traffic networks, the spatio-temporal dependence among roads dynamically changes over time due to factors such as road conditions and unforeseen incidents, which brings great challenges to multistep traffic speed prediction. Additionally, multistep traffic speed prediction commonly faces the problem of error accumulation, resulting in a loss of prediction accuracy. To address these issues, we propose a Sequence-to-Sequence Spatio-Temporal Attention model (STSSTA) for multistep traffic speed prediction. Specifically, we build an adaptive tuning module to select road preferences and automatically obtain global information about the roads. Then we construct a Sequence-to-Sequence architecture for spatio-temporal feature learning and multistep traffic speed prediction. In particular, in the encoder, we design a Diffusion Graph Convolutional Network (DGCN) and combine it with a Gated Recurrent Unit (GRU) to effectively capture the complex spatio-temporal features within the traffic networks. In the decoder, we introduce a Recalling attention mechanism to alleviate the problem of local information loss caused by encoder compression, thereby reducing error accumulation in multistep prediction. Experiments on METR-LA and PeMS-BAY datasets demonstrate that STSSTA outperforms the baseline models in long-term prediction.

AST3DRNet: Attention-Based Spatio-Temporal 3D Residual Neural Networks for Traffic Congestion Prediction.

Traffic congestion prediction has become an indispensable component of an intelligent transport system. However, one limitation of the existing methods is that they treat the effects of spatio-temporal correlations on traffic prediction as invariable during modeling spatio-temporal features, which results in inadequate modeling. In this paper, we propose an attention-based spatio-temporal 3D residual neural network, named AST3DRNet, to directly forecast the congestion levels of road networks in a city. AST3DRNet combines a 3D residual network and a self-attention mechanism together to efficiently model the spatial and temporal information of traffic congestion data. Specifically, by stacking 3D residual units and 3D convolution, we proposed a 3D convolution module that can simultaneously capture various spatio-temporal correlations. Furthermore, a novel spatio-temporal attention module is proposed to explicitly model the different contributions of spatio-temporal correlations in both spatial and temporal dimensions through the self-attention mechanism. Extensive experiments are conducted on a real-world traffic congestion dataset in Kunming, and the results demonstrate that AST3DRNet outperforms the baselines in short-term (5/10/15 min) traffic congestion predictions with an average accuracy improvement of 59.05%, 64.69%, and 48.22%, respectively.

Trajectory-User Linking via Hierarchical Spatio-Temporal Attention Networks

Trajectory-User Linking (TUL) is crucial for human mobility modeling by linking different trajectories to users with the exploration of complex mobility patterns. Existing works mainly rely on the recurrent neural framework to encode the temporal dependencies in trajectories, have fall short in capturing spatial-temporal global context for TUL prediction. To fill this gap, this work presents a new hierarchical spatio-temporal attention neural network, called AttnTUL , to jointly encode the local trajectory transitional patterns and global spatial dependencies for TUL. Specifically, our first model component is built over the graph neural architecture to preserve the local and global context and enhance the representation paradigm of geographical regions and user trajectories. Additionally, a hierarchically structured attention network is designed to simultaneously encode the intra-trajectory and inter-trajectory dependencies, with the integration of the temporal attention mechanism and global elastic attentional encoder. Extensive experiments demonstrate the superiority of our AttnTUL method as compared to state-of-the-art baselines on various trajectory datasets. The source code of our model is available at https://github.com/Onedean/AttnTUL .

An Effective Video Transformer With Synchronized Spatiotemporal and Spatial Self-Attention for Action Recognition.

Convolutional neural networks (CNNs) have come to dominate vision-based deep neural network structures in both image and video models over the past decade. However, convolution-free vision Transformers (ViTs) have recently outperformed CNN-based models in image recognition. Despite this progress, building and designing video Transformers have not yet obtained the same attention in research as image-based Transformers. While there have been attempts to build video Transformers by adapting image-based Transformers for video understanding, these Transformers still lack efficiency due to the large gap between CNN-based models and Transformers regarding the number of parameters and the training settings. In this work, we propose three techniques to improve video understanding with video Transformers. First, to derive better spatiotemporal feature representation, we propose a new spatiotemporal attention scheme, termed synchronized spatiotemporal and spatial attention (SSTSA), which derives the spatiotemporal features with temporal and spatial multiheaded self-attention (MSA) modules. It also preserves the best spatial attention by another spatial self-attention module in parallel, thereby resulting in an effective Transformer encoder. Second, a motion spotlighting module is proposed to embed the short-term motion of the consecutive input frames to the regular RGB input, which is then processed with a single-stream video Transformer. Third, a simple intraclass frame interlacing method of the input clips is proposed that serves as an effective video augmentation method. Finally, our proposed techniques have been evaluated and validated with a set of extensive experiments in this study. Our video Transformer outperforms its previous counterparts on two well-known datasets, Kinetics400 and Something-Something-v2.

Unified Spatio-Temporal Attention Models for Advanced Human Action Recognition & Detection

Abstract: This research paper introduces innovative approaches to enhance human action analytics in computer vision through the development of spatial and temporal attention models. The first model is based on Recurrent Neural Networks (RNNs) with Long Short-Term Memory (LSTM) units, aiming to extract discriminative spatial and temporal features from skeleton data for improved human action recognition and detection. The model selectively focuses on discriminative joints within each frame, utilizing a regularized cross-entropy loss for effective training. Additionally, a joint training strategy is proposed to optimize the learning process. The second model, a spatio-temporal attention (STA) network, is designed to address the limitations of existing 3D Convolutional Neural Networks (3D CNNs) in treating all video frames equally. The STA network characterizes beneficial information at both the frame and channel levels, leveraging differences in spatial and temporal dimensions to enhance the learning capability of 3D convolutions. The proposed STA method can be seamlessly integrated into state-of-the-art 3D CNN architectures for video action detection and recognition. Evaluation of diverse datasets, including SBU Kinect Interaction, NTU RGB+D, PKU-MMD, UCF-101, HMDB-51, and THUMOS 2014, demonstrates the effectiveness of both proposed models in achieving state-of-the-art performance for action recognition and detection tasks. The spatial and temporal attention models contribute significantly to capturing discriminative features, showcasing their potential in advancing the field of human action analytics in computer vision.

Remaining useful life estimation of bearing using spatio-temporal convolutional transformer

The maintenance costs, productivity, health, and safety of mechanical equipment all heavily rely on the remaining usable life (RUL) of the bearings. Recently, the transformer has been widely used in the RUL field due to its ability to capture some of the degradation information of the bearing. However, the transformer is weak in acquiring local information and fails to extract temporal features from the degradation process. To solve the above problems, this paper proposes a spatio-temporal convolutional transformer (STCT) model, which mainly consists of the dual convolutional spatio-temporal network (DCSTN) and multi-scale transformer (MST). It not only captures the degradation features of the bearings from the temporal and spatial perspectives but also enhances the ability of the transformer to acquire local information. We propose DCSTN as a feature extraction module, and the proposed spatio-temporal attention as the core of the DCSTN can capture the relevant degradation state features at different moments. In addition, MST uses a new module of multi-scale dilated causal convolution combined with multi-head attention to realize the combination of global degradation information and local contextual information capturing ability. We demonstrate the effectiveness and sophistication of the STCT model by conducting comparative experiments with ablation experiments on publicly available datasets.

CL-SPO2Net: Contrastive Learning Spatiotemporal Attention Network for Non-Contact Video-Based SpO2 Estimation.

Video-based peripheral oxygen saturation (SpO2) estimation, utilizing solely RGB cameras, offers a non-contact approach to measuring blood oxygen levels. Previous studies set a stable and unchanging environment as the premise for non-contact blood oxygen estimation. Additionally, they utilized a small amount of labeled data for system training and learning. However, it is challenging to train optimal model parameters with a small dataset. The accuracy of blood oxygen detection is easily affected by ambient light and subject movement. To address these issues, this paper proposes a contrastive learning spatiotemporal attention network (CL-SPO2Net), an innovative semi-supervised network for video-based SpO2 estimation. Spatiotemporal similarities in remote photoplethysmography (rPPG) signals were found in video segments containing facial or hand regions. Subsequently, integrating deep neural networks with machine learning expertise enabled the estimation of SpO2. The method had good feasibility in the case of small-scale labeled datasets, with the mean absolute error between the camera and the reference pulse oximeter of 0.85% in the stable environment, 1.13% with lighting fluctuations, and 1.20% in the facial rotation situation.

Image Super-Resolution Reconstruction Model Based on Multi-Feature Fusion

Due to the limitations of imaging equipment and image transmission conditions on daily image acquisition, the images acquired are usually low-resolution images, and it will cost a lot of time and economic costs to increase image resolution by upgrading hardware equipment. In this paper, we propose an image super-resolution reconstruction algorithm based on spatio-temporal-dependent residual network MSRN, which fuses multiple features. The algorithm uses the surface feature extraction module to extract the input features of the image, and then uses the deep residual aggregation module to adaptively learn the deep features, and then fuses multiple features and learns the global residual. Finally, the high-resolution image is obtained through the up-sampling module and the reconstruction module. In the model structure, different convolution kernels and jump connections are used to extract more high-frequency information, and spatio-temporal attention mechanism is introduced to focus on more image details. The experimental results show that compared with SRGAN, VDSR and Laplacian Pyramid SRN, the proposed algorithm finally achieves better reconstruction effect, and the image texture details are clearer under different scaling factors. In objective evaluation, the peak signal-to-noise ratio (PSNR) and structure similarity (SSIM) of the proposed algorithm are improved compared with SRGAN.

Construction of Mining Robot Equipment Fault Prediction Model Based on Deep Learning

In the field of mining robot maintenance, in order to enhance the research on predictive modeling, we introduce the LODS model (long short-term memory network (LSTM) optimized deep fusion neural network (DFNN) with spatiotemporal attention network (STAN)). Traditional models have shortcomings in handling the long-term dependencies of time series data and mining the complexity of spatiotemporal information in the field of mine maintenance. The LODS model integrates the advantages of LSTM, DFNN and STAN, providing a comprehensive method for effective feature extraction and prediction. Through experimental evaluation on multiple data sets, the experimental results show that the LODS model achieves more accurate predictions, compared with traditional models and optimization strategies, and achieves significant reductions in MAE, MAPE, RMSE and MSE of 15.76, 5.59, 2.02 and 11.96, respectively, as well as significant reductions in the number of parameters and computational complexity. It also achieves higher efficiency in terms of the inference time and training time. The LODS model performs well in all the evaluation indexes and has significant advantages; thus, it can provide reliable support for the equipment failure prediction of the mine maintenance robot.

A Radar Echo Extrapolation Model Based on a Dual-Branch Encoder–Decoder and Spatiotemporal GRU

Precipitation forecasting is an immensely significant aspect of meteorological prediction. Accurate weather predictions facilitate services in sectors such as transportation, agriculture, and tourism. In recent years, deep learning-based radar echo extrapolation techniques have found effective applications in precipitation forecasting. However, the ability of existing methods to extract and characterize complex spatiotemporal features from radar echo images remains insufficient, resulting in suboptimal forecasting accuracy. This paper proposes a novel extrapolation algorithm based on a dual-branch encoder–decoder and spatiotemporal Gated Recurrent Unit. In this model, the dual-branch encoder–decoder structure independently encodes radar echo images in the temporal and spatial domains, thereby avoiding interference between spatiotemporal information. Additionally, we introduce a Multi-Scale Channel Attention Module (MSCAM) to learn global and local feature information from each encoder layer, thereby enhancing focus on radar image details. Furthermore, we propose a Spatiotemporal Attention Gated Recurrent Unit (STAGRU) that integrates attention mechanisms to handle temporal evolution and spatial relationships within radar data, enabling the extraction of spatiotemporal information from a broader receptive field. Experimental results demonstrate the model’s ability to accurately predict morphological changes and motion trajectories of radar images on real radar datasets, exhibiting superior performance compared to existing models in terms of various evaluation metrics. This study effectively improves the accuracy of precipitation forecasting in radar echo images, provides technical support for the short-range forecasting of precipitation, and has good application prospects.