Spatiotemporal Attention Research Articles

A New Accurate Aircraft Trajectory Prediction in Terminal Airspace Based on Spatio-Temporal Attention Mechanism

Trajectory prediction serves as a prerequisite for future trajectory-based operation, significantly reducing the uncertainty of aircraft movement information within airspace by scientifically forecasting the three-dimensional positions of aircraft over a certain period. As convergence points in the aviation network, airport terminal airspace exhibits the most complex traffic conditions in the entire air route network. It has stronger mutual influences and interactions among aircraft compared to the en-route phase. Current research typically uses the trajectory time series information of a single aircraft as input for subsequent predictions. However, it often lacks consideration of the close-range spatial interactions between multiple aircraft in the terminal airspace. This results in a gap in the study of aircraft trajectory prediction that couples spatiotemporal features. This paper aims to predict the four-dimensional trajectories of aircraft in terminal airspace, constructing a Spatio-Temporal Transformer (ST-Transformer) prediction model based on temporal and spatial attention mechanisms. Using radar aircraft trajectory data from the Guangzhou Baiyun Airport terminal airspace, the results indicate that the proposed ST-Transformer model has a smaller prediction error compared to mainstream deep learning prediction models. This demonstrates that the model can better integrate the temporal sequence correlation of trajectory features and the potential spatial interaction information among trajectories for accurate prediction.

Traffic Speed Prediction Based on Dynamic Spatial-Temporal Attention Graph Convolutional Network

Abstract The prediction of traffic speed represents a crucial branch within the broader field of traffic forecasting. Its objective is to forecast the average speed or speed distribution on road segments or within a network for a designated future time period . The prediction results can provide scientific decision support for traffic management departments to effectively control and guide traffic flow. However, given the limitations of urban road network topology and its dynamic temporal shifts, traffic prediction has consistently been deemed a perpetual scientific challenge. To simultaneously encapsulate both spatial and temporal dependencies, we introduce an innovative Dynamic Spatio-Temporal Attention Network (DSTANet). This model integrates spatio-temporal attention with graph convolutional networks and temporal convolution to enhance its analytical capabilities. Specifically, the graph convolutional network is leveraged to learn the intricate topology for capturing spatial relations, while temporal convolution is employed to ascertain the dynamic fluctuations of traffic data, thereby acquiring temporal dependencies. Subsequently, we implement our model to forecast traffic patterns within the framework of road networks. Experimental results demonstrate that our model effectively captures the spatio-temporal dependencies among roadway junctions derived from traffic data, outperforming state-of-the-art benchmarks on real-world traffic datasets in terms of predictive accuracy.

STHKT: Spatiotemporal Knowledge Tracing with Topological Hawkes Process

Knowledge Tracing (KT) is a method that seeks to forecast students’ future performance based on their historical interactions with intelligent tutoring systems. Various KT techniques have been developed from distinct perspectives, such as probability, logic, and deep learning, to scrutinize interaction sequences. However, most current methods predominantly concentrate on temporal or structural information, considering simple question sequences or interaction sequences, but fail to describe the topological dependencies between sequences, which can lead to changes in causal structures,as referred to as dynamic structural information in the paper. The dynamic information includes the clustering effect among concepts and questions, as well as the group effect among students and questions: Questions with the same knowledge concepts tend to present increasingly similar response patterns in the sequence, and concurrently, students who answer similar questions often exhibit closer knowledge states. We introduce the STHKT method, which employs a tripartite heterogeneous graph to capture relationships among concepts, questions, and students. By incorporating temporal information through a topological graph, we employ two Graph Convolution Networks (GCNs) to model the two effects separately. Following this, we apply the Hawkes Process to introduce a spatiotemporal attention mechanism that combines both effects. Through comparative experiments, ablation studies and case studies, we validate the efficacy of key components in the model. Experiment results indicate that the STHKT model can effectively model the two major effects, integrate dynamic structural information, and to some extent improve the accuracy of predicting students’ response probabilities. It is the first time that these two effects have been considered in the process of knowledge tracing, and it is adaptable to large-scale online education platforms.

Multi-Scale Spatio-Temporal Attention Networks for Network-Scale Traffic Learning and Forecasting.

Accurate and timely forecasting of traffic on local road networks is crucial for deploying effective dynamic traffic control, advanced route planning, and navigation services. This task is particularly challenging due to complex spatio-temporal dependencies arising from non-Euclidean spatial relations in road networks and non-linear temporal dynamics influenced by changing road conditions. This paper introduces the spatio-temporal network embedding (STNE) model, a novel deep learning framework tailored for learning and forecasting graph-structured traffic data over extended input sequences. Unlike traditional convolutional neural networks (CNNs), the model employs graph convolutional networks (GCNs) to capture the spatial characteristics of local road network topologies. Moreover, the segmentation of very long input traffic data into multiple sub-sequences, based on significant temporal properties such as closeness, periodicity, and trend, is performed. Multi-dimensional long short-term memory neural networks (MDLSTM) are utilized to flexibly access multi-dimensional context. Experimental results demonstrate that the STNE model surpasses state-of-the-art traffic forecasting benchmarks on two large-scale real-world traffic datasets.

Managing remaining useful life of cyber-aeroengine systems using a graph spatio-temporal attention recurrent network with phase-lag index

The decision-making process concerning the maintenance of equipment significantly impacts the safety and reliability of the aeroengine system. A cyber-aeroengine system (CAS) integrates intricate aeroengine components with a cyber decision-making space based on the communication network, thereby facilitating computational support for forecasting the health status of the aeroengine system. Within the cyber realm, we introduce a novel approach termed the graph spatio-temporal attention recurrent network with phase-lag index (PLI-GSTARN) to assess the degradation states of aeroengine systems. This method constructs a spatio-temporal graph based on sensor signal correlations and information reachability across distributed sensors within the physical equipment. Subsequently, the proposed framework employs a graph attention network to capture spatial features and a long short-term memory (LSTM) layer equipped with an attention mechanism to capture temporal features inherent in the constructed graph. Ultimately, this methodology is applied to cyber–physical systems within aircraft engines to manage remaining health life and diagnose faults, yielding commendable outcomes.

Computer aided diagnostic system with reduced electrode set for depression detection using spatio-temporal attention mechanism

Computer-aided diagnostic systems (CAD) have been extensively used in medical applications to identify neurological abnormalities. It enhances diagnostic accuracy, decreases misdiagnosis rates, and provides reliable decision support. The proposed manuscript has introduced an automated CAD system that uses EEG signals to detect depression. The proposed automated CAD system is designed to work with a reduced set of electrodes. This manuscript introduces a method for electrode ranking based on the Fisher score. Electrodes are selected based on the threshold criteria. EEG is spatio-temporal data. The spatial and temporal self-attention mechanism is used to extract spatial and temporal features in the spatial and temporal domain. The proposed automated CAD system reduces the requirements of electrodes by an order of magnitude without compromising the performance of the model. The proposed CAD system has a higher degree of automation. It reduces the cost of the scan, experimental time, and complexity of the system.

Measuring the velocity of spatio-temporal attention waves

A sequence of visual or auditory events may be perceived as a single continuing sequence or as two or more separate sequences occurring in parallel. The latter percept occurs when the perceived distance between events is large, and the timing is fast, and is referred to as “streaming.” Several researchers have previously argued that streaming indicates a velocity constraint on the movement of attention. To test this hypothesis the present experiment measured tradeoffs between distance and timing for the onset or loss of streaming in a rectangular pattern of displayed lights. Two linear tradeoffs were found, one corresponding to the loss of streaming when the light pattern was slowed down, and one corresponding to the onset of streaming when the light pattern was sped up. The slopes of these linear relations are interpreted as integer multiples of the velocity of spatio-temporal attention waves. A process model postulates that participants adjust the wavelength of their spatio-temporal attentional traveling wave to match the height of the displayed rectangle. Streaming is assumed to occur when peaks in the attentional traveling wave coincide with the onsets of lights at the top and bottom of the displayed rectangle. Additional supporting evidence for temporal and spatial attention waves is reviewed. This model may be useful for understanding some forms of attentional deficits as well as expert attentional skills arising in musical performance, sports, meditation, and other tasks.

Enhancing Major Depressive Disorder Diagnosis With Dynamic-Static Fusion Graph Neural Networks.

Major Depressive Disorder (MDD) is a debilitating, complex mental condition with unclear mechanisms hindering diagnostic progress. Research links MDD to abnormal brain connectivity using functional magnetic resonance imaging (fMRI). Yet, existing fMRI-based MDD models suffer from limitations, including neglecting dynamic network traits, lacking interpretability, and struggling with small datasets. We present DSFGNN, a novel graph neural network framework addressing these issues for improved MDD diagnosis. DSFGNN employs a graph isomorphism encoder to model static and dynamic brain networks, achieving effective fusion of temporal and spatial information through a spatiotemporal attention mechanism, thereby enhancing interpretability. Furthermore, we incorporate a causal disentangling module and orthogonal regularization module to augment the model's expressiveness. We evaluate DSFGNN on the Rest-meta-MDD dataset, yielding superior results compared to the best baseline. Besides, extensive ablation studies and interpretability analysis confirm DSFGNN's effectiveness and potential for biomarker discovery.

A Dual Pipeline With Spatio-Temporal Attention Fusion Approach for Human Activity Recognition

A Dual Pipeline With Spatio-Temporal Attention Fusion Approach for Human Activity Recognition

Parking Lot Traffic Prediction Based on Fusion of Multifaceted Spatio-Temporal Features.

With the rapid growth of population and vehicles, issues such as traffic congestion are becoming increasingly apparent. Parking guidance and information (PGI) systems are becoming more critical, with one of the most important tasks being the prediction of traffic flow in parking lots. Predicting parking traffic can effectively improve parking efficiency and alleviate traffic congestion, traffic accidents, and other problems. However, due to the complex characteristics of parking spatio-temporal data, high levels of noise, and the intricate influence of external factors, there are three challenges to predicting parking traffic in a city effectively: (1) how to better model the nonlinear, asymmetric, and complex spatial relationships among parking lots; (2) how to model the temporal autocorrelation of parking flow more accurately for each parking lot, whether periodic or aperiodic; and (3) how to model the correlation between external influences, such as holiday weekends, POIs (points of interest), and weather factors. In this context, this paper proposes a parking lot traffic prediction model based on the fusion of multifaceted spatio-temporal features (MFF-STGCN). The model consists of a feature embedding module, a spatio-temporal attention mechanism module, and a spatio-temporal convolution module. The feature embedding module embeds external features such as weekend holidays, geographic POIs, and weather features into the time series, the spatio-temporal attention mechanism module captures the dynamic spatio-temporal correlation of parking traffic, and the spatio-temporal convolution module captures the spatio-temporal features by using graph convolution and gated recursion units. Finally, the outputs of adjacent time series, daily series, and weekly series are weighted and fused to obtain the final prediction results, thus predicting the parking lot traffic flow more accurately and effectively. Results on real datasets demonstrate that the proposed model enhances prediction performance.

MGSFformer: A Multi-Granularity Spatiotemporal Fusion Transformer for air quality prediction

Air quality spatiotemporal prediction can provide technical support for environmental governance and sustainable city development. As a classic multi-source spatiotemporal data, effective multi-source information fusion is key to achieving accurate air quality predictions. However, due to not fully fusing two pieces of information, classical deep learning models struggle to achieve satisfactory prediction results: (1) Multi-granularity: each air monitoring station collects air quality data at different sampling intervals, which show distinct time series patterns. (2) Spatiotemporal correlation: due to human activities and atmospheric diffusion, there exist correlations between air quality data from different air monitoring stations, necessitating the consideration of other air monitoring stations' influences when modeling each air quality time series. In this study, to achieve satisfactory prediction results, we propose the Multi-Granularity Spatiotemporal Fusion Transformer, comprised of the residual de-redundant block, spatiotemporal attention block, and dynamic fusion block. Specifically, the residual de-redundant block eliminates information redundancy between data with different granularities and prevents the model from being misled by redundant information. The spatiotemporal attention block captures the spatiotemporal correlation of air quality data and facilitates prediction modeling. The dynamic fusion block evaluates the importance of data with different granularities and integrates the prediction results. Experimental results demonstrate that the proposed model surpasses 11 baselines by 5% in performance on three real-world datasets.

TE-Spikformer:Temporal-enhanced spiking neural network with transformer

The integration of Spiking Neural Networks (Maass, 1997) and Transformers (Vaswani et al., 2017) has significantly enhanced performance in the field, achieving substantial improvements while ensuring energy efficiency. This amalgamation bears significant exploratory significance. This paper aims to enhance the network’s capacity for extracting temporal information from neuromorphic datasets. By leveraging the Spikformer (Zhou et al., 2023b) architecture, we introduce a novel network named TE-Spikformer. Through thorough analysis, we identified an imbalance in the Average Firing Rates(AFR) of neurons in the layer preceding the classification head across temporal steps. We thoroughly investigate the root cause of this issue, attributing it to the limitations of the network’s use of Batch Normalization (BN) (Ioffe and Szegedy, 2015) layer. To address these challenges, we propose a Batch Group Normalization (BGN) layer. While maintaining the stability of temporal characteristics in the data, we also introduce the Spike Spatio-Temporal Attention(SSTA) module to enhance the network’s ability to capture temporal information. To validate the effectiveness of our approach, we conducted multiple experiments using neuromorphic datasets, including DVS-CIFAR10, DVS128 Gesture, and N-Caltech101. The experimental results demonstrate that our algorithm consistently outperforms baseline methods, achieving accuracy rates of 99.30%, 89.60%, and 87.42%, respectively, thereby attaining state-of-the-art performance in the field.

SRU-Net: a novel spatiotemporal attention network for sclera segmentation and recognition

SRU-Net: a novel spatiotemporal attention network for sclera segmentation and recognition

Novel STAttention GraphWaveNet model for residential household appliance prediction and energy structure optimization

The multifaceted challenge of carbon dioxide (CO2) emissions during building operations, construction, and material production perpetuates energy shortages and climate adversities. Urgent imperatives underscore the need for effective energy reduction and carbon mitigation, driving the building sector toward a paradigm shift in clean, low-carbon evolution. However, the complex multidimensional and nonlinear characteristics of influencing factors of building energy consumption make it difficult in precisely understanding the spatial and temporal relationships between the features in the predictive modeling of buildings. Therefore, a novel Graph Wavenet (GWN) based on the spatio-temporal attention fusion mechanism (STA-GWN) is proposed for spatio-temporal graph prediction modeling of structures, which can capture hidden spatial relationships using an adaptive dependency matrix. To overcome the drawbacks of standard predefined adjacency matrices and their inability to capture faraway dependencies, a spatio-temporal attention fusion mechanism is introduced to extract latent spatio-temporal features dynamically. By combining extracted features, building energy consumption can be predicted more precisely. Moreover, the proposed model becomes more interpretable, making it easier to reveal how spatio-temporal features relate to building energy consumption. Lastly, the proposed method is applied to energy conservation and emission reduction in the building industry. A predictive model for electrical energy consumption during two public datasets with an operational phase of buildings is established to analyze and optimize energy utilization in residential structures that rely on electricity. The experimental results demonstrate that the STA-GWN model surpasses other deep learning methods in terms of prediction accuracy and robustness. Moreover, the proposed model can offer optimal solutions for building energy processes, reducing approximately 201.44 kg (Kg) and 64.08 Kg of CO2 emissions in the appliances energy prediction dataset and the individual-household-electric-power-consumption (IHEPC) dataset, respectively.

An ensemble model for accurate prediction of key water quality parameters in river based on deep learning methods

Deep learning models provide a more powerful method for accurate and stable prediction of water quality in rivers, which is crucial for the intelligent management and control of the water environment. To increase the accuracy of predicting the water quality parameters and learn more about the impact of complex spatial information based on deep learning models, this study proposes two ensemble models TNX (with temporal attention) and STNX (with spatio-temporal attention) based on seasonal and trend decomposition (STL) method to predict water quality using geo-sensory time series data. Dissolved oxygen, total phosphorus, and ammonia nitrogen were predicted in short-step (1 h, and 2 h) and long-step (12 h, and 24 h) with seven water quality monitoring sites in a river. The ensemble model TNX improved the performance by 2.1%–6.1% and 4.3%–22.0% relative to the best baseline deep learning model for the short-step and long-step water quality prediction, and it can capture the variation pattern of water quality parameters by only predicting the trend component of raw data after STL decomposition. The STNX model, with spatio-temporal attention, obtained 0.5%–2.4% and 2.3%–5.7% higher performance compared to the TNX model for the short-step and long-step water quality prediction, and such improvement was more effective in mitigating the prediction shift patterns of long-step prediction. Moreover, the model interpretation results consistently demonstrated positive relationship patterns across all monitoring sites. However, the significance of seven specific monitoring sites diminished as the distance between the predicted and input monitoring sites increased. This study provides an ensemble modeling approach based on STL decomposition for improving short-step and long-step prediction of river water quality parameter, and understands the impact of complex spatial information on deep learning model.

DPP: A Novel Disease Progression Prediction Method for Ginkgo Leaf Disease Based on Image Sequences

Ginkgo leaf disease poses a grave threat to Ginkgo biloba. The current management of Ginkgo leaf disease lacks precision guidance and intelligent technologies. To provide precision guidance for disease management and to evaluate the effectiveness of the implemented measures, the present study proposes a novel disease progression prediction (DPP) method for Ginkgo leaf blight with a multi-level feature translation architecture and enhanced spatiotemporal attention module (eSTA). The proposed DPP method is capable of capturing key spatiotemporal dependencies of disease symptoms at various feature levels. Experiments demonstrated that the DPP method achieves state-of-the-art prediction performance in disease progression prediction. Compared to the top-performing spatiotemporal predictive learning method (SimVP + TAU), our method significantly reduced the mean absolute error (MAE) by 19.95% and the mean square error (MSE) by 25.35%. Moreover, it achieved a higher structure similarity index measure (SSIM) of 0.970 and superior peak signal-to-noise ratio (PSNR) of 37.746 dB. The proposed method can accurately forecast the progression of Ginkgo leaf blight to a large extent, which is expected to provide valuable insights for precision and intelligent disease management. Additionally, this study presents a novel perspective for the extensive research on plant disease prediction.

Interpretable multi-graph convolution network integrating spatio-temporal attention and dynamic combination for wind power forecasting

Graph neural networks (GNN) can effectively improve wind power forecasting accuracy due to their ability to capture spatial correlations between wind farms. However, conventional GNN-based forecasting models constructing the graphs from only one specific perspective fail to characterize the spatial correlation patterns comprehensively and accurately. Furthermore, their black-box structure leads to deficient interpretability of the forecasting results, which hampers the quantification of forecasting uncertainty and reduces the credibility of applying these forecasting models in practice. To address the above problems, an interpretable ultra-short-term wind power forecasting method is proposed based on multi-graph convolution network integrating spatio-temporal attention and dynamic graph combination. Four undirected weight graphs are constructed to form a multi-graph scheme by considering various geographical and statistical aspects, and then graph convolution network (GCN) and temporal convolution network (TCN) are used to enhance the forecasting performance by leveraging information in both space and time dimensions. Meanwhile, it consists of temporal attention that deeply mines the temporal dependencies and spatial attention that interprets the influence of wind farms on the forecasts. Additionally, to boost the adaptability and generalization capability of the proposed method, an improved reinforcement learning is elaborately designed to combine the results of different graphs by dynamically optimizing their weights to obtain the final forecasts. Experiment results suggest that the proposed method exhibits superior forecasting accuracy and reasonable interpretability compared with other benchmark methods.

Multi‐graph representation spatio‐temporal attention networks for traffic forecasting in the cinematic metaverse

AbstractThe cinematic metaverse aims to create a virtual space with the context of a film. Users can enter this space in the form of avatars, experiencing the cinematic plot firsthand in an immersive manner. This requires us to design a rational computation resource allocation and synchronization algorithm to meet the demands of multi‐objective joint optimization, such as low latency and high throughput, which ensures that users can seamlessly switch between virtual and real worlds and acquire immersive experiences. Unfortunately, the explosive growth in the number of users makes it difficult to jointly optimize multiple objectives. Predicting traffic generated by the users' avatars in the cinematic metaverse is significant for the optimization process. Although graph neural networks‐based traffic prediction models achieve superior prediction accuracy, these methods rely only on physical distances‐based topological graph information, while failing to comprehensively reflect the real relationships between avatars in the cinematic metaverse. To address this issue, we present a novel Multi‐Graph Representation Spatio‐Temporal Attention Networks (MGRSTANet) for traffic prediction in the cinematic metaverse. Specifically, based on multiple topological graph information (e.g., physical distances, centerity, and similarity), we first design Multi‐Graph Embedding (MGE) module to generate multiple graph representations, thus reflecting on the real relationships between avatars more comprehensively. The Spatio‐Temporal Attention (STAtt) module is then proposed to extract spatio‐temporal correlations in each graph representations, thus improving prediction accuracy. We conduct simulation experiments to evaluate the effectiveness of MGRSTANet. The experimental results demonstrate that our proposed model outperforms the state‐of‐the‐art baselines in terms of prediction accuracy, making it appropriate for traffic forecasting in the cinematic metaverse.

Shear-Wave Velocity Prediction Based on the CNN-BiGRU Integrated Network with Spatiotemporal Attention Mechanism

Shear wave velocity is one of the important parameters reflecting the lithological and physical properties of reservoirs, and it is widely used in the fields of lithology and fluid property identification, reservoir evaluation, seismic data processing, and interpretation. However, due to the high cost and challenge of obtaining shear wave velocity, only a few key wells are measured. Considering the intricate nonlinear mapping relationship between shear wave velocity and conventional logging data, an integrated network incorporating an attention mechanism, a convolutional neural network, and a bidirectional gated recurrent unit (STACBiN) is proposed for predicting shear wave velocity. The impact of conventional logging data on shear wave velocity is analyzed, thus employing the attention mechanism to focus on data correlated with shear wave velocity, which can enable the prediction results of the method proposed superior to those of conventional methods. Additionally, the prediction results of this method are compared with the prediction results of the two-dimensional convolutional neural network (2DCNN) and bidirectional gated recurrent unit (BiGRU). It is verified that the network proposed can effectively predict the shear wave velocity, with minimal error between predicted and true values.

Low and high beta rhythms have different motor cortical sources and distinct roles in movement control and spatiotemporal attention.

Low and high beta frequency rhythms were observed in the motor cortex, but their respective sources and behavioral correlates remain unknown. We studied local field potentials (LFPs) during pre-cued reaching behavior in macaques. They contained a low beta band (<20 Hz) dominant in primary motor cortex and a high beta band (>20 Hz) dominant in dorsal premotor cortex (PMd). Low beta correlated positively with reaction time (RT) from visual cue onset and negatively with uninstructed hand postural micro-movements throughout the trial. High beta reflected temporal task prediction, with selective modulations before and during cues, which were enhanced in moments of increased focal attention when the gaze was on the work area. This double-dissociation in sources and behavioral correlates of motor cortical low and high beta, with respect to both task-instructed and spontaneous behavior, reconciles the largely disparate roles proposed for the beta rhythm, by suggesting band-specific roles in both movement control and spatiotemporal attention.