Spatiotemporal Attention Research Articles

Image Super-Resolution Reconstruction Model Based on Multi-Feature Fusion

Due to the limitations of imaging equipment and image transmission conditions on daily image acquisition, the images acquired are usually low-resolution images, and it will cost a lot of time and economic costs to increase image resolution by upgrading hardware equipment. In this paper, we propose an image super-resolution reconstruction algorithm based on spatio-temporal-dependent residual network MSRN, which fuses multiple features. The algorithm uses the surface feature extraction module to extract the input features of the image, and then uses the deep residual aggregation module to adaptively learn the deep features, and then fuses multiple features and learns the global residual. Finally, the high-resolution image is obtained through the up-sampling module and the reconstruction module. In the model structure, different convolution kernels and jump connections are used to extract more high-frequency information, and spatio-temporal attention mechanism is introduced to focus on more image details. The experimental results show that compared with SRGAN, VDSR and Laplacian Pyramid SRN, the proposed algorithm finally achieves better reconstruction effect, and the image texture details are clearer under different scaling factors. In objective evaluation, the peak signal-to-noise ratio (PSNR) and structure similarity (SSIM) of the proposed algorithm are improved compared with SRGAN.

Construction of Mining Robot Equipment Fault Prediction Model Based on Deep Learning

In the field of mining robot maintenance, in order to enhance the research on predictive modeling, we introduce the LODS model (long short-term memory network (LSTM) optimized deep fusion neural network (DFNN) with spatiotemporal attention network (STAN)). Traditional models have shortcomings in handling the long-term dependencies of time series data and mining the complexity of spatiotemporal information in the field of mine maintenance. The LODS model integrates the advantages of LSTM, DFNN and STAN, providing a comprehensive method for effective feature extraction and prediction. Through experimental evaluation on multiple data sets, the experimental results show that the LODS model achieves more accurate predictions, compared with traditional models and optimization strategies, and achieves significant reductions in MAE, MAPE, RMSE and MSE of 15.76, 5.59, 2.02 and 11.96, respectively, as well as significant reductions in the number of parameters and computational complexity. It also achieves higher efficiency in terms of the inference time and training time. The LODS model performs well in all the evaluation indexes and has significant advantages; thus, it can provide reliable support for the equipment failure prediction of the mine maintenance robot.

A Radar Echo Extrapolation Model Based on a Dual-Branch Encoder–Decoder and Spatiotemporal GRU

Precipitation forecasting is an immensely significant aspect of meteorological prediction. Accurate weather predictions facilitate services in sectors such as transportation, agriculture, and tourism. In recent years, deep learning-based radar echo extrapolation techniques have found effective applications in precipitation forecasting. However, the ability of existing methods to extract and characterize complex spatiotemporal features from radar echo images remains insufficient, resulting in suboptimal forecasting accuracy. This paper proposes a novel extrapolation algorithm based on a dual-branch encoder–decoder and spatiotemporal Gated Recurrent Unit. In this model, the dual-branch encoder–decoder structure independently encodes radar echo images in the temporal and spatial domains, thereby avoiding interference between spatiotemporal information. Additionally, we introduce a Multi-Scale Channel Attention Module (MSCAM) to learn global and local feature information from each encoder layer, thereby enhancing focus on radar image details. Furthermore, we propose a Spatiotemporal Attention Gated Recurrent Unit (STAGRU) that integrates attention mechanisms to handle temporal evolution and spatial relationships within radar data, enabling the extraction of spatiotemporal information from a broader receptive field. Experimental results demonstrate the model’s ability to accurately predict morphological changes and motion trajectories of radar images on real radar datasets, exhibiting superior performance compared to existing models in terms of various evaluation metrics. This study effectively improves the accuracy of precipitation forecasting in radar echo images, provides technical support for the short-range forecasting of precipitation, and has good application prospects.

Multistage spatio-temporal attention network based on NODE for short-term PV power forecasting

Photovoltaic (PV) power has attracted widespread attention from many countries around the world due to its clean and renewable characteristics. To ensure the stable operation of the power system, accurate PV power forecasting has become a mandatory and challenging task. Currently, deep learning methods have become a vital approach in the field of PV power forecasting. In this work, a multistage attention neural network based on neural ordinary differential equation (MANODE) is proposed to address the main limitations of previous deep learning methods applied to PV power forecasting. Based on the neural ordinary differential equation (NODE), MANODE optimizes the long short-term memory network (LSTM) and temporal convolutional neural network (TCN), and combines the attention mechanism to achieve fine-grained spatio-temporal information extraction of PV series. In addition, the proposed MANODE model is applied to three different PV series collected from the Alice Springs meteorological station. Compared to previous state-of-the-art methods, the proposed method reduces the PV power forecasting error by at least 12.05%, 13.15%, and 9.71% on three different PV datasets, in terms of mean absolute error metric. The average errors of the MANODE method in four-hour-ahead PV power forecasting on the three datasets are 0.321, 0.350, and 0.567.

Spatiotemporal wave forecast with transformer-based network: A case study for the northwestern Pacific Ocean

The forecast of ocean waves relies mostly on complex dynamic-based models, which are expensive in computation and demanding in professional skills to run. Diverse deep learning methods have been proposed to tackle this problem, yet the architecture of Transformer (i.e., the self-attention) was seldom tested for such a learning problem. To bridge this gap, we apply a state-of-the-art spatiotemporal attention network, the EarthFormer, and apply it for wave forecasting in the northwestern Pacific. To train and validate the EarthFormer, the fifth-generation atmospheric reanalysis dataset (ERA5) from the European Centre for Medium-Range Weather Forecast product with hourly resolution was adopted, with the sequence of wind as the input and the three key variables (i.e., significance wave height, mean wave period, and mean wave direction) of ocean waves as the output. The prediction can generate accurate wave forecasts up to 12 h ahead, with considerably low root-mean-squared error (RMSE). Overall, the predicted waves resulted in RMSE for the significance wave height of 0.22 m, for the mean wave period of 0.68 s, and for the mean wave direction of 0.28 rad. These wave height and wave period values are ∼13 % and ∼10 % of corresponding spatiotemporal mean values. The EarthFormer outperformed a popular spatiotemporal forecast network, the ConvLSTM, in our problem. For site-wise forecasts against buoy observations near Taiwan, the EarthFormer forecast also presents considerably high accuracy comparable to the ERA5 forecasts. This method can therefore provide an accurate and prompt way to forecast the large-scale distribution of waves to better model the marine dynamic and its climate effects, which could have a high potential for disaster prevention and climate modeling.

Dynamic-guided Spatiotemporal Attention for Echocardiography Video Segmentation.

Left ventricle (LV) endocardium segmentation in echocardiography video has received much attention as an important step in quantifying LV ejection fraction. Most existing methods are dedicated to exploiting temporal information on top of 2D convolutional networks. In addition to single appearance semantic learning, some research attempted to introduce motion cues through the optical flow estimation (OFE) task to enhance temporal consistency modeling. However, OFE in these methods is tightly coupled to LV endocardium segmentation, resulting in noisy inter-frame flow prediction, and post-optimization based on these flows accumulates errors. To address these drawbacks, we propose dynamic-guided spatiotemporal attention (DSA) for semi-supervised echocardiography video segmentation. We first fine-tune the off-the-shelf OFE network RAFT on echocardiography data to provide dynamic information. Taking inter-frame flows as additional input, we use a dual-encoder structure to extract motion and appearance features separately. Based on the connection between dynamic continuity and semantic consistency, we propose a bilateral feature calibration module to enhance both features. For temporal consistency modeling, the DSA is proposed to aggregate neighboring frame context using deformable attention that is realized by offsets grid attention. Dynamic information is introduced into DSA through a bilateral offset estimation module to effectively combine with appearance semantics and predict attention offsets, thereby guiding semantic-based spatiotemporal attention. We evaluated our method on two popular echocardiography datasets, CAMUS and EchoNet-Dynamic, and achieved state-of-the-art.

Locating X-Ray Coronary Angiogram Keyframes via Long Short-Term Spatiotemporal Attention With Image-to-Patch Contrastive Learning.

Locating the start, apex and end keyframes of moving contrast agents for keyframe counting in X-ray coronary angiography (XCA) is very important for the diagnosis and treatment of cardiovascular diseases. To locate these keyframes from the class-imbalanced and boundary-agnostic foreground vessel actions that overlap complex backgrounds, we propose long short-term spatiotemporal attention by integrating a convolutional long short-term memory (CLSTM) network into a multiscale Transformer to learn the segment- and sequence-level dependencies in the consecutive-frame-based deep features. Image-to-patch contrastive learning is further embedded between the CLSTM-based long-term spatiotemporal attention and Transformer-based short-term attention modules. The imagewise contrastive module reuses the long-term attention to contrast image-level foreground/background of XCA sequence, while patchwise contrastive projection selects the random patches of backgrounds as convolution kernels to project foreground/background frames into different latent spaces. A new XCA video dataset is collected to evaluate the proposed method. The experimental results show that the proposed method achieves a mAP (mean average precision) of 72.45% and a F-score of 0.8296, considerably outperforming the state-of-the-art methods. The source code is available at https://github.com/Binjie-Qin/STA-IPCon.

Hand Gesture Authentication by Discovering Fine-Grained Spatiotemporal Identity Characteristics

Dynamic hand gesture is an emerging and promising biometric trait containing both physiological and behavioral characteristics. Possessing the two kinds of characteristics makes dynamic hand gesture have more identity information enabling more accurate and secure authentication theoretically, but also poses a challenge of efficient fine-grained spatiotemporal feature extraction. This challenge involves a seemingly paradoxical problem that high-frame-rate videos are required for behavioral characteristic analysis, but they can also introduce high computational costs. To mitigate this issue, we propose a Frequency Spatiotemporal Attention Network (FSTA-Net) with a focus on satisfying the high-performance and low-computation requirements of authentication systems. The FSTA-Net is established with a two-stage identity characteristic analysis paradigm for short- and long-term modeling. Specifically, considering that models prefer to analyze physiological characteristics which are relatively straightforward to understand, we first design a Behavior Enhanced (BE) module to emphasize hand motions and reduce redundant information to facilitate local identity feature distillation in the first stage. We then present a Frequency Spatiotemporal Attention (FSTA) module to summarize global identity features with decent FLOPs and GPU memory occupation in the second stage. Incorporating the BE and FSTA modules enables them to complement each other’s strengths, resulting in a clear-cut improvement in equal error rate and running speed. Extensive experiments on the SCUT-DHGA dataset demonstrate the superiority of the FSTA-Net. The code is available at https://github.com/SCUT-BIP-Lab/FSTA-Net.

M2DC: A Meta-Learning Framework for Generalizable Diagnostic Classification of Major Depressive Disorder.

Psychiatric diseases are bringing heavy burdens for both individual health and social stability. The accurate and timely diagnosis of the diseases is essential for effective treatment and intervention. Thanks to the rapid development of brain imaging technology and machine learning algorithms, diagnostic classification of psychiatric diseases can be achieved based on brain images. However, due to divergences in scanning machines or parameters, the generalization capability of diagnostic classification models has always been an issue. We propose Meta-learning with Meta batch normalization and Distance Constraint (M2DC) for training diagnostic classification models. The framework can simulate the train-test domain shift situation and promote intra-class cohesion, as well as inter-class separation, which can lead to clearer classification margins and more generalizable models. To better encode dynamic brain graphs, we propose a concatenated spatiotemporal attention graph isomorphism network (CSTAGIN) as the backbone. The network is trained for the diagnostic classification of major depressive disorder (MDD) based on multi-site brain graphs. Extensive experiments on brain images from over 3261 subjects show that models trained by M2DC achieve the best performance on cross-site diagnostic classification tasks compared to various contemporary domain generalization methods and SOTA studies. The proposed M2DC is by far the first framework for multi-source closed-set domain generalizable training of diagnostic classification models for MDD and the trained models can be applied to reliable auxiliary diagnosis on novel data.

RPM: RF-Based Pose Machines

Radio-frequency (RF) based human sensing technologies, due to their great practical value in various applications and privacy-preserving nature, have gained tremendous attention in recent years. However, without fully exploiting the characteristics of radio signals, the performance of existing methods are still limited. First, RF features of the moving human body have different representations in dimensions such as channel and scale, which is challenging when performing feature fusion. Besides, the human body is specularly reflective with respect to the radar, which means the human body cannot be fully captured by a single RF snapshot. Therefore, the radar signal reflected by the human body is sparse and incomplete, which is difficult to extract high-quality features for 3D human pose estimation. In this paper, we present the RF-based Pose Machines (RPM), a novel framework which can generate 3D skeletons from RF signals. Considering the characteristics of RF signals, RPM includes several modules to overcome the challenges. Firstly, a Feature Fusion Network (FFN) is designed to effectively fuse radio signals from horizontal and vertical planes based on the channels' correlation and maintain high-quality feature via a multi-scale fusion block. A Spatio-Temporal Attention network is then designed to reconstruct 3D skeletons from the sparse and incomplete RF signals. Specifically, a spatial attention module is designed to model non-local relationships among joints and reconstruct body parts that a single RF snapshot cannot capture. Afterwards, a temporal attention module is proposed to refine 3D pose based on temporal coherency learned from frame queries. To evaluate the performance of our RPM framework, we construct a large-scale dataset of synchronized 3d skeletons and RF signals, RFSkeleton3D. Our experimental results show that RPM locates 3D key points of the human body with an average error of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$5.71 cm$</tex-math></inline-formula> and maintains its performance in new environments with occlusion or bad illumination. The dataset and codes will be made in public.

Grasp Stability Assessment Through Spatio-Temporal Attention Mechanism and Adaptive Gate Fusion

Grasp Stability Assessment Through Spatio-Temporal Attention Mechanism and Adaptive Gate Fusion

基于下采样交互时空注意力网络的交通流量预测

基于下采样交互时空注意力网络的交通流量预测

Adaptive Spatial–Temporal Aware Graph Learning for EEG-Based Emotion Recognition

An intelligent emotion recognition system based on electroencephalography (EEG) signals shows considerable potential in various domains such as healthcare, entertainment, and education, thanks to its portability, high temporal resolution, and real-time capabilities. However, the existing research in this field faces limitations stemming from the nonstationary nature and individual variability of EEG signals. In this study, we present a novel EEG emotion recognition model, named GraphEmotionNet, designed to enhance the accuracy of EEG-based emotion recognition through the incorporation of a spatiotemporal attention mechanism and transfer learning. The proposed GraphEmotionNet model can effectively learn the intrinsic connections between EEG channels and construct an adaptive graph. This graph’s adaptive nature is crucial in optimizing spatial–temporal graph convolutions, which in turn enhances spatial–temporal feature characterization and contributes to the process of emotion classification. Moreover, an integration of domain adaptation aligns the extracted features across different domains, further alleviating the impact of individual EEG variability. We evaluate the model performance on two benchmark databases, employing two types of cross-validation protocols: within-subject cross-validation and cross-subject cross-validation. The experimental results affirm the model’s efficacy in extracting EEG features linked to emotional semantics and demonstrate its promising performance in emotion recognition.

RPM 2.0: RF-Based Pose Machines for Multi-Person 3D Pose Estimation

Advanced human sensing technologies based on radio frequency (RF) signals have gained widespread attention in recent years. However, due to the sparsity and incompleteness of RF signals, fine-grained RF-based multi-person 3D pose estimation has progressed more slowly. In this paper, we present RF-based Pose Machine (RPM 2.0) for multi-person 3D pose estimation using RF signals. Specifically, we first develop a lightweight anchor-free detector module to locate and crop regions of interest from horizontal and vertical RF signals. Afterward, we treat the horizontal and vertical millimeter-wave radars as “RF cameras” with different viewing angles and propose a Multi-view Fusion Network to unproject the RF signals into a unified latent feature space, and then calculate the correlation for weighted fusion. Finally, a Spatio-Temporal Attention Network is designed to reconstruct the multi-person 3D skeleton sequences, in which the spatial attention module is proposed to recover invisible body parts using non-local correlations among joints and the temporal attention module refines the 3D pose sequences using temporal coherency learned from frame queries. We evaluate the performance of the proposed RPM 2.0 and state-of-the-art methods on a large-scale dataset with multi-person 3D pose labels and corresponding radar signals. The experimental results show that RPM 2.0 outperforms all of the baseline methods, which locates multi-person 3D key points with an average error of 73 <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">mm</i> and generalizes well in new data such as occlusion, low illumination.

LiDAR video object segmentation with dynamic kernel refinement

In this paper, we formalize memory- and tracking-based methods to perform the LiDAR-based Video Object Segmentation (VOS) task, which segments points of the specific 3D target (given in the first frame) in a LiDAR sequence. LiDAR-based VOS can directly provide target-aware geometric information for practical application scenarios like behavior analysis and anticipating danger. We first construct a LiDAR-based VOS dataset named KITTI-VOS based on SemanticKITTI, which acts as a testbed and facilitates comprehensive evaluations of algorithm performance. Next, we provide two types of baselines, i.e., memory-based and tracking-based baselines, to explore this task. Specifically, the first memory-based pipeline is built on a space–time memory network equipped with the non-local spatiotemporal attention-based memory bank. We further design a more potent variant to introduce the locality into the spatiotemporal attention module by local self-attention and cross-attention modules. For the second tracking-based baseline, we modify two representative 3D object tracking methods to adapt to LiDAR-based VOS tasks. Finally, we propose a refine module that takes mask priors and generates object-aware kernels, which could boost all the baselines’ performance. We evaluate the proposed methods on the dataset and demonstrate their effectiveness.

Spatio-Temporal-Attention-Based Vehicle Trajectory Prediction Considering Multi-Vehicle Interaction in Mixed Traffic Flow

As a link connecting the environmental perception system and the decision-making system, accurate obstacle trajectory prediction provides a reliable guarantee of correct decision-making by autonomous vehicles. Oriented toward a mixed human-driven and machine-driven traffic environment, a vehicle trajectory prediction algorithm based on an encoding–decoding framework composed of a multiple-attention mechanism is proposed. Firstly, a directed graph is used to describe vehicle–vehicle motion dependencies. Then, by calculating the repulsive force between vehicles using a priori edge information based on the artificial potential field theory, vehicle–vehicle interaction coefficients are extracted via a graph attention mechanism (GAT). Subsequently, after concatenating the vehicle–vehicle interaction feature with the encoded vehicle trajectory vectors, a spatio-temporal attention mechanism is applied to determine the coupling relationship of hidden vectors. Finally, the predicted trajectory is generated by a gated recurrent unit (GRU) decoder. The training and evaluation of the proposed model were conducted on the NGSIM public dataset. The test results demonstrated that compared with existing baseline models, our approach has fewer prediction errors and better robustness. In addition, introducing artificial potential fields into the attention mechanism causes the model to have better interpretability.

Local motion feature extraction and spatiotemporal attention mechanism for action recognition

Local motion feature extraction and spatiotemporal attention mechanism for action recognition

Action Recognition Based on Multi-Level Topological Channel Attention of Human Skeleton.

In action recognition, obtaining skeleton data from human poses is valuable. This process can help eliminate negative effects of environmental noise, including changes in background and lighting conditions. Although GCN can learn unique action features, it fails to fully utilize the prior knowledge of human body structure and the coordination relations between limbs. To address these issues, this paper proposes a Multi-level Topological Channel Attention Network algorithm: Firstly, the Multi-level Topology and Channel Attention Module incorporates prior knowledge of human body structure using a coarse-to-fine approach, effectively extracting action features. Secondly, the Coordination Module utilizes contralateral and ipsilateral coordinated movements in human kinematics. Lastly, the Multi-scale Global Spatio-temporal Attention Module captures spatiotemporal features of different granularities and incorporates a causal convolution block and masked temporal attention to prevent non-causal relationships. This method achieved accuracy rates of 91.9% (Xsub), 96.3% (Xview), 88.5% (Xsub), and 90.3% (Xset) on NTU-RGB+D 60 and NTU-RGB+D 120, respectively.

Dynamic alpha power modulations and slow negative potentials track natural shifts of spatio-temporal attention.

Alpha power modulations and slow negative potentials have previously been associated with anticipatory processes in spatial and temporal top-down attention. In typical experimental designs, however, neural responses triggered by transient stimulus onsets can interfere with attention-driven activity patterns and our interpretation of such. Here, we investigated these signatures of spatio-temporal attention in a dynamic paradigm free from potentially confounding stimulus-driven activity using electroencephalography. Participants attended the cued side of a bilateral stimulus rotation and mentally counted how often one of two remembered sample orientations (i.e., the target) was displayed while ignoring the uncued side and non-target orientation. Afterwards, participants performed a delayed match-to-sample task, in which they indicated if the orientation of a probe stimulus matched the corresponding sample orientation (previously target or non-target). We observed dynamic alpha power reductions and slow negative waves around task-relevant points in space and time (i.e., onset of the target orientation in the cued hemifield) over posterior electrodes contralateral to the locus of attention. In contrast to static alpha power lateralization, these dynamic signatures correlated with subsequent memory performance (primarily detriments for matching probes of the non-target orientation), suggesting a preferential allocation of attention to task-relevant locations and time points at the expense of reduced resources and impaired performance for information outside the current focus of attention. Our findings suggest that humans can naturally and dynamically focus their attention at relevant points in space and time and that such spatio-temporal attention shifts can be reflected by dynamic alpha power modulations and slow negative potentials.

Flood Forecasting Using Hybrid LSTM and GRU Models with Lag Time Preprocessing

Climate change and urbanization have increased the frequency of floods worldwide, resulting in substantial casualties and property loss. Accurate flood forecasting can offer governments early warnings about impending flood disasters, giving them a chance to evacuate and save lives. Deep learning is used in flood forecasting to improve the timeliness and accuracy of flood water level predictions. While various deep learning models similar to Long Short-Term Memory (LSTM) have achieved notable results, they have complex structures with low computational efficiency, and often lack generalizability and stability. This study applies a spatiotemporal Attention Gated Recurrent Unit (STA-GRU) model for flood prediction to increase the models’ computing efficiency. Another salient feature of our methodology is the incorporation of lag time during data preprocessing before the training of the model. Notably, for 12-h forecasting, the STA-GRU model’s R-squared (R2) value increased from 0.8125 to 0.9215. Concurrently, the model manifested reduced root mean squared error (RMSE) and mean absolute error (MAE) metrics. For a more extended 24-h forecasting, the R2 value of the STA-GRU model improved from 0.6181 to 0.7283, accompanied by diminishing RMSE and MAE values. Seven typical deep learning models—the LSTM, the Convolutional Neural Networks LSTM (CNNLSTM), the Convolutional LSTM (ConvLSTM), the spatiotemporal Attention Long Short-Term Memory (STA-LSTM), the GRU, the Convolutional Neural Networks GRU (CNNGRU), and the STA-GRU—are compared for water level prediction. Comparative analysis delineated that the use of the STA-GRU model and the application of the lag time pre-processing method significantly improved the reliability and accuracy of flood forecasting.