Spatial-temporal Transformation Research Articles

Multi-View Graph Learning for Path-Level Aging-Aware Timing Prediction

As CMOS technology continues to scale down, the aging effect—known as negative bias temperature instability (NBTI)—has become increasingly prominent, gradually emerging as a key factor affecting device reliability. Accurate aging-aware static timing analysis (STA) at the early design phase is critical for establishing appropriate timing margins to ensure circuit reliability throughout the chip lifecycle. However, traditional aging-aware timing analysis methods, typically based on Simulation Program with Integrated Circuit Emphasis (SPICE) simulations or aging-aware timing libraries, struggle to balance prediction accuracy with computational cost. In this paper, we propose a multi-view graph learning framework for path-level aging-aware timing prediction, which combines the strengths of the spatial–temporal Transformer network (STTN) and graph attention network (GAT) models to extract the aging timing features of paths from both timing-sensitive and workload-sensitive perspectives. Experimental results demonstrate that our proposed framework achieves an average MAPE score of 3.96% and reduces the average MAPE by 5.8 times compared to FFNN and 2.2 times compared to PNA, while maintaining acceptable increases in processing time.

Spatial-Temporal Transformer Networks for Traffic Flow Forecasting Using a Pre-Trained Language Model.

Most current methods use spatial-temporal graph neural networks (STGNNs) to analyze complex spatial-temporal information from traffic data collected from hundreds of sensors. STGNNs combine graph neural networks (GNNs) and sequence models to create hybrid structures that allow for the two networks to collaborate. However, this collaboration has made the model increasingly complex. This study proposes a framework that relies solely on original Transformer architecture and carefully designs embeddings to efficiently extract spatial-temporal dependencies in traffic flow. Additionally, we used pre-trained language models to enhance forecasting performance. We compared our new framework with current state-of-the-art STGNNs and Transformer-based models using four real-world traffic datasets: PEMS04, PEMS08, METR-LA, and PEMS-BAY. The experimental results demonstrate that our framework outperforms the other models in most metrics.

Exploring the conflict risk characteristics of air weaving sections in Metroplex terminal areas with flight trajectory data and adaptive graph spatial-temporal transformer

Metroplex terminal areas is involved with numerous air weaving sections, where various types of arrival and departure traffic flows intersect or converge, leading to high-risk aircraft conflicts and inefficient Metroplex operation. The primary objective of this study is to explore the conflict risk characteristics of operational interactions between airports in Metroplex terminal areas with large-scale trajectory data, including risk evaluation, risk propagation and risk prediction. One complete month of ADS-B trajectory data is collected from a representative multi-airport system in Central and South China to illustrate the procedure. Specifically, flight trajectories are firstly clustered within Metroplex terminal areas, and a trajectory-tube model is then built to characterize the airport flows within the same cluster. Then, a clustering-based collision probability calculation method is proposed to evaluate the risk of air weaving sections. The spatial patterns of risks in air weaving sections reveal that operational interactions among Metroplex airports are quite different under various runway configurations. Furthermore, an adaptive graph spatial-temporal transformer (ASTT) network is developed to predict the future risk of each air weaving section in Metroplex terminal areas. The results indicate that the developed ASTT network can collaboratively encode the spatiotemporal characteristics in the air weaving section risk data, which achieves more accurate and robust prediction performance than other compared models during multiple look-ahead time steps. Moreover, the adaptive spatial graph technique designed in the ASTT can also reveal the hidden risk propagation effects among air weaving sections. The results of this study could provide insightful suggestions to air traffic authorities for developing effective coordination and conflict mitigation strategies, such as redesigning flight procedures, reallocating routes, and optimizing aircraft sequences to enhance the efficiency and safety of Metroplex operations.

Emotion recognition using hierarchical spatial–temporal learning transformer from regional to global brain

Emotion recognition is an essential but challenging task in human–computer interaction systems due to the distinctive spatial structures and dynamic temporal dependencies associated with each emotion. However, current approaches fail to accurately capture the intricate effects of electroencephalogram (EEG) signals across different brain regions on emotion recognition. Therefore, this paper designs a transformer-based method, denoted by R2G-STLT, which relies on a spatial–temporal transformer encoder with regional to global hierarchical learning that learns the representative spatiotemporal features from the electrode level to the brain-region level. The regional spatial–temporal transformer (RST-Trans) encoder is designed to obtain spatial information and context dependence at the electrode level aiming to learn the regional spatiotemporal features. Then, the global spatial–temporal transformer (GST-Trans) encoder is utilized to extract reliable global spatiotemporal features, reflecting the impact of various brain regions on emotion recognition tasks. Moreover, the multi-head attention mechanism is placed into the GST-Trans encoder to empower it to capture the long-range spatial–temporal information among the brain regions. Finally, subject-independent experiments are conducted on each frequency band of the DEAP, SEED, and SEED-IV datasets to assess the performance of the proposed model. Results indicate that the R2G-STLT model surpasses several state-of-the-art approaches.

An Intelligent Deep Learning Framework for Traffic Flow Imputation and Short-term Prediction Based on Dynamic Features

The accurate prediction of traffic flow has emerged as a focal point in the cutting-edge sphere of intelligent transportation. Extant methodologies rely on deep learning for short-term forecasts but are hampered by two critical limitations: the omission of vital traffic flow data and an overemphasis on static rather than dynamic attributes. This paper presents a solution to these challenges through the creation of a dynamic attention generative adversarial network (DATGAN) that is specifically engineered to impute missing traffic data accurately. DATGAN introduces two groundbreaking features: first, an innovative attention module that is adept at identifying local correlations, and second, novel spatial-correction graph convolutional layers that are designed to capture the ever-changing spatial dynamic dependencies. Building on this foundation, the parallel spatial–temporal Transformer (PSTTransformer) is proposed, which integrates both spatial and temporal attention blocks to manage dynamic spatial-temporal dependencies for future short-term traffic predictions. This dual-block approach not only embeds periodic information via a fusion self-attention mechanism in the temporal attention blocks, but also synergizes graph convolutional networks with self-attention mechanisms in the spatial attention blocks through a gate mechanism. A comprehensive comparative analysis conducted on traffic datasets revealed that our methodology outperforms current leading models by reducing the mean squared error by 5% in data imputation tasks and shows robust performance in traffic prediction scenarios.

PRECYSE: Predicting Cybersickness using Transformer for Multimodal Time-Series Sensor Data

Cybersickness, a factor that hinders user immersion in VR, has been the subject of ongoing attempts to predict it using AI. Previous studies have used CNN and LSTM for prediction models and used attention mechanisms and XAI for data analysis, yet none explored a transformer that can better reflect the spatial and temporal characteristics of the data, beneficial for enhancing prediction and feature importance analysis. In this paper, we propose cybersickness prediction models using multimodal time-series sensor data (i.e., eye movement, head movement, and physiological signals) based on a transformer algorithm, considering sensor data pre-processing and multimodal data fusion methods. We constructed the MSCVR dataset consisting of normalized sensor data, spectrogram formatted sensor data, and cybersickness levels collected from 45 participants through a user study. We proposed two methods for embedding multimodal time-series sensor data into the transformer: modality-specific spatial and temporal transformer encoders for normalized sensor data (MS-STTN) and modality-specific spatial-temporal transformer encoder for spectrogram (MS-STTS). MS-STTN yielded the highest performance in the ablation study and the comparison of the existing models. Furthermore, by analyzing the importance of data features, we determined their relevance to cybersickness over time, especially the salience of eye movement features. Our results and insights derived from multimodal time-series sensor data and the transformer model provide a comprehensive understanding of cybersickness and its association with sensor data. Our MSCVR dataset and code are publicly available: https://github.com/dayoung-jeong/PRECYSE.git.

A multi-channel spatial-temporal transformer model for traffic flow forecasting

Traffic flow forecasting is a crucial task in transportation management and planning. The main challenges for traffic flow forecasting are that (1) as the length of prediction time increases, the accuracy of prediction will decrease; (2) the predicted results greatly rely on the extraction of temporal and spatial dependencies from the road networks. To overcome the challenges mentioned above, we propose a multi-channel spatial-temporal transformer model for traffic flow forecasting, which improves the accuracy of the prediction by fusing results from different channels of traffic data. Our approach leverages graph convolutional network to extract spatial features from each channel while using a transformer-based architecture to capture temporal dependencies across channels. We introduce an adaptive adjacency matrix to overcome limitations in feature extraction from fixed topological structures. Experimental results on six real-world datasets demonstrate that introducing a multi-channel mechanism into the temporal model enhances performance and our proposed model outperforms state-of-the-art models in terms of accuracy.

An efficient spatial-temporal transformer with temporal aggregation and spatial memory for traffic forecasting

Traffic forecasting technology has widespread applications in various domains, such as urban traffic planning and intelligent transportation systems. Traffic forecasting encounters challenges in effectively capturing the intricate spatial–temporal correlations in traffic data. While the latest methods have achieved satisfactory performance, they still suffer from two limitations: (i) Most methods overlook the memory of valuable traffic patterns at each traffic node, thus making it struggle to reveal dynamic spatial–temporal correlations using their inherent periodicity and trend characteristics from a broader perspective. (ii) As the research progresses, recently proposed models become increasingly complex and massive. To address these issues, we propose a Spatial-Temporal Aggregation Memory Transformer (STAMT) for traffic forecasting. Specifically, we propose a memory bank to enhance vanilla spatial attention and cache the traffic patterns of historical input. By querying these traffic patterns, which contain rich spatial–temporal semantic information, the model can optimize prediction performance by extracting trends and regularities across various periods. To reduce the computational costs, we introduce a temporal module to capture temporal correlations while reducing temporal dimension information. In addition, we leverage the random feature map and matrix multiplication associativity property, which reduce the quadratic complexity of spatial attention to linearity with regard to the number of nodes. Ultimately, our theoretical analysis concludes that a single-layer spatial attention network is sufficient to capture spatial–temporal correlations deeply without stacking. Extensive experiments on nine real-world datasets demonstrate that STAMT outperforms state-of-the-art baselines in regular, long-range, and large-scale traffic forecasting tasks while significantly reducing the computational costs. Codes are available at https://github.com/LiuAoyu1998/STAMT.

HybridGait: A Benchmark for Spatial-Temporal Cloth-Changing Gait Recognition with Hybrid Explorations

Existing gait recognition benchmarks mostly include minor clothing variations in the laboratory environments, but lack persistent changes in appearance over time and space. In this paper, we propose the first in-the-wild benchmark CCGait for cloth-changing gait recognition, which incorporates diverse clothing changes, indoor and outdoor scenes, and multi-modal statistics over 92 days. To further address the coupling effect of clothing and viewpoint variations, we propose a hybrid approach HybridGait that exploits both temporal dynamics and the projected 2D information of 3D human meshes. Specifically, we introduce a Canonical Alignment Spatial-Temporal Transformer (CA-STT) module to encode human joint position-aware features, and fully exploit 3D dense priors via a Silhouette-guided Deformation with 3D-2D Appearance Projection (SilD) strategy. Our contributions are twofold: we provide a challenging benchmark CCGait that captures realistic appearance changes over expanded time and space, and we propose a hybrid framework HybridGait that outperforms prior works on CCGait and Gait3D benchmarks. Our project page is available at https://github.com/HCVLab/HybridGait.

STViT: Improving Self-Supervised Multi-Camera Depth Estimation with Spatial-Temporal Context and Adversarial Geometry Regularization (Student Abstract)

Multi-camera depth estimation has recently garnered significant attention due to its substantial practical implications in the realm of autonomous driving. In this paper, we delve into the task of self-supervised multi-camera depth estimation and propose an innovative framework, STViT, featuring several noteworthy enhancements: 1) we propose a Spatial-Temporal Transformer to comprehensively exploit both local connectivity and the global context of image features, meanwhile learning enriched spatial-temporal cross-view correlations to recover 3D geometry. 2) to alleviate the severe effect of adverse conditions, e.g., rainy weather and nighttime driving, we introduce a GAN-based Adversarial Geometry Regularization Module (AGR) to further constrain the depth estimation with unpaired normal-condition depth maps and prevent the model from being incorrectly trained. Experiments on challenging autonomous driving datasets Nuscenes and DDAD show that our method achieves state-of-the-art performance.

A Cross-Modal Generative Adversarial Network for Scenarios Generation of Renewable Energy

Scenarios data of renewable energy resources plays an essential role in the study of mitigating the risk in the power system due to their intermittent nature. Existing researches have mainly focused on how to generate high-quality time series based scenarios with corrupted or missing observed data. However, the multimodality of renewable energy resources is largely ignored from the perspective of the spatial-temporal correlation which can significantly strengthen the renewable energy prediction and optimization. To make full use of these multi-modal data and further improve the data quality of scenarios, a cross-modal Generative Adversarial Network (cGAN) model is proposed with the setting of two spatial-temporal transformer models. In cGAN, the task of scenarios generation is formulated as a probability approximation problem. Then, a cross-modal scenarios generation framework is constructed to process data from multi-modal observations. Theoretically, it can generate infinite number of uncertain scenarios data via the repeated random noise sampling technique. Extensive experiments are carried out at a NREL photovoltaic power dataset and a real-world wind power dataset, respectively. The results validate the state-of-the-art performance of cGAN, even though the validation dataset is missing at random. Moreover, the results also show that the setting of spatial-temporal transformer clearly explains the contributions of different modal data, and stabilize the training process of cGAN. Finally, a stochastic day-head economic dispatch is also studied to show the practical value of cGAN.

MVSTT: A Multiview Spatial-Temporal Transformer Network for Traffic-Flow Forecasting.

Accurate traffic-flow prediction remains a critical challenge due to complicated spatial dependencies, temporal factors, and unpredictable events. Most existing approaches focus on single-or dual-view learning and thus face limitations in systematically learning complex spatial-temporal features. In this work, we propose a novel multiview spatial-temporal transformer (MVSTT) network that can effectively learn complex spatial-temporal domain correlations and potential patterns from multiple views. First, we examine a temporal view and design a short-range gated convolution component from a short-term subview, and a long-range gated convolution component from a long-term subview. These two components effectively aggregate knowledge of the temporal domain at multiple granularities and mine patterns of node evolution across time steps. Meanwhile, in the spatial view, we design a dual-graph spatial learning module that captures fixed and dynamic spatial dependencies of nodes, as well as the evolution patterns of edges, from the static and dynamic graph subviews, respectively. In addition, we further design a spatial-temporal transformer to mine different levels of spatial-temporal features through multiview knowledge fusion. Extensive experiments on four real-world traffic datasets show that our method consistently outperforms the state-of-the-art baseline. The code of MVSTT is available at https://github.com/JianSoL/MVSTT.

Self-supervised spatial–temporal transformer fusion based federated framework for 4D cardiovascular image segmentation

Availability of high-quality large annotated data is indeed a significant challenge in healthcare. In addition, privacy concerns and data-sharing restrictions often hinder access to large and diverse medical image datasets. To reduce the requirement for annotated training data, self-supervised pre-training strategies on nonannotated data have been extensively used, whereas collaborative algorithm training without the need to exchange the underlying data. In this paper, we introduce a novel federated learning-based self-supervised spatial–temporal transformer’s fusion (SSFL) for cardiovascular image segmentation. The integration of spatial–temporal swin transformer is used to extract the features from 3D SAX multiple phases (full cycle of cardiac heart). An efficient self-supervised contrastive framework consisting of a spatial–temporal transformer network with 25 encoders is used to model the temporal features. The spatial and temporal features are fused and forwarded to the decoder for cardiac heart segmentation using cine MRI images. To further improve segmentation, we use an attention-based unpaired GAN model to map or transfer the style from ACDC to M&Ms and use synthetically generated volumes in the proposed self-supervised approach. Experiments with three different cardiovascular image segmentation tasks, such as segmentation of the right ventricle, left ventricle, and myocardium, showed significant improvement compared to the state-of-the-art segmentation framework.

Graph Spatial-Temporal Transformer Network for Traffic Prediction

Traffic information can reflect the operating status of a city, and accurate traffic forecasting is critical in intelligent transportation systems (ITS) and urban planning. However, traffic information has complex nonlinearity and dynamic spatial-temporal dependencies due to human mobility, bringing new traffic forecasting challenges. This paper proposed a graph spatial-temporal transformer network for traffic prediction (GSTTN) to cope with the above problems. Specifically, the proposed framework explores spatial characteristics of the across-road network of traffic information hidden in human behavior patterns via a multi-view graph convolutional network (GCN). Furthermore, the transformer network with a multi-head attention mechanism is adopted to capture the random disturbance in the time series characteristics of traffic information. As a result, these two components can be used to model spatial relations and temporal trends. Finally, we examine real-world datasets, and the experiments show that the proposed framework outperforms the current state-of-the-art baselines.

An Accurate Prediction Method of Human Assembly Motion for Human–Robot Collaboration

In the process of human–robot collaborative assembly, robots need to recognize and predict human behaviors accurately, and then perform autonomous control and work route planning in real-time. To support the judgment of human intervention behaviors and meet the need of real-time human–robot collaboration, the Fast Spatial–Temporal Transformer Network (FST-Trans), an accurate prediction method of human assembly actions, is proposed. We tried to maximize the symmetry between the prediction results and the actual action while meeting the real-time requirement. With concise and efficient structural design, FST-Trans can learn about the spatial–temporal interactions of human joints during assembly in the same latent space and capture more complex motion dynamics. Considering the inconsistent assembly rates of different individuals, the network is forced to learn more motion variations by introducing velocity–acceleration loss, realizing accurate prediction of assembly actions. An assembly dataset was collected and constructed for detailed comparative experiments and ablation studies, and the experimental results demonstrate the effectiveness of the proposed method.

Landscape pattern changes in coal resource-based cities: A case study of Huainan City in China

Currently, coal resource-based cities (CRBCs) are facing challenges such as ecological destruction, resource exhaustion, and disordered urban development. By analyzing the landscape pattern, the understanding of urban land use can be clarified, and optimization strategies can be proposed for urban transformation and sustainable development. In this study, based on the interpretation of remote sensing data for three dates, the landscape pattern changes in the urban area of Huainan City, a typical coal resource-based city in Anhui Province, China were empirically investigated. The results indicate that: (1) There is a significant spatial-temporal transformation of land use, with construction land gradually replacing arable land as the dominant land use type in the region. (2) Landscape indices are helpful to reveal the characteristics of land transfer and distribution of human activities during a process. At the landscape type level, construction land, grassland, and water bodies are increasingly affected by human activities. At the landscape composition level, the number of landscape types increases, and the distribution of different types of patches becomes more balanced. In addition, to address the problems caused by the coal mining subsidence areas in Huainan city, three landscape pattern optimization strategies are proposed at both macro and micro levels. The research findings contribute to a better understanding of land use changes and their driving forces, and offer valuable alternatives for ecological environment optimization.

Multi-Task Spatial-Temporal Transformer for Multi-Variable Meteorological Forecasting

Multi-Task Spatial-Temporal Transformer for Multi-Variable Meteorological Forecasting

Event Graph Guided Compositional Spatial-Temporal Reasoning for Video Question Answering.

Video question answering (VideoQA) is challenging since it requires the model to extract and combine multi-level visual concepts from local objects to global actions from complex events for compositional reasoning. Existing works represent the video with fixed-duration clip features that make the model struggle in capturing the crucial concepts in multiple granularities. To overcome this shortcoming, we propose to represent the video with an Event Graph in a hierarchical structure whose nodes correspond to visual concepts of different levels (object, relation, scene and action) and edges indicate their spatial-temporal relationships. We further propose a H ierarchical S patial- T emporal T ransformer (HSTT) which takes nodes from the graph as visual input to realize compositional reasoning guided by the event graph. To fully exploit the spatial-temporal context delivered from the graph structure, on the one hand, we encode the nodes in the order of their semantic hierarchy (depth) and occurrence time (breadth) with our improved graph search algorithm; On the other hand, we introduce edge-guided attention to combine the spatial-temporal context among nodes according to their edge connections. HSTT then performs QA by cross-modal interactions guaranteed by the hierarchical correspondence between the multi-level event graph and the cross-level question. Experiments on the recent challenging AGQA and STAR datasets show that the proposed method clearly outperforms the existing VideoQA models by a large margin, including those pre-trained with large-scale external data. Our code is available at https://github.com/ByZ0e/HSTT.

A Spatial-Temporal Transformer Network for City-Level Cellular Traffic Analysis and Prediction

With the accelerated popularization of 5G applications, accurate cellular traffic prediction is becoming increasingly important for efficient network management. Currently, the latest algorithms for cellular traffic prediction generally neglect extraction of the shallow features of cellular traffic and the prediction accuracy is hence limited. Therefore, we propose a global-local spatial-temporal transformer network (GLSTTN) that can fully excavate diverse spatial-temporal characteristics of cellular traffic for accurate cellular traffic prediction. Specifically, GLSTTN achieves this goal by constructing two modules: the global spatial-temporal module and the local spatial-temporal module. In the global spatial-temporal module, GLSTTN captures global correlations using stacked spatial-temporal blocks, where each block is composed of one spatial transformer and one temporal transformer. A skip connection is then used in each block to strengthen feature propagation. In the local spatial-temporal module, GLSTTN fully extracts the local spatial-temporal dependencies hidden in globally encoded features using densely connected convolutional neural networks. Extensive experiments demonstrate that GLSTTN achieves more accurate cellular traffic prediction than existing approaches on a real-world cellular traffic dataset.

Tourism demand forecasting: a deep learning model based on spatial-temporal transformer

Purpose Incorporating dynamic spatial effects exhibits considerable potential in improving the accuracy of forecasting tourism demands. This study aims to propose an innovative deep learning model for capturing dynamic spatial effects. Design/methodology/approach A novel deep learning model founded on the transformer architecture, called the spatiotemporal transformer network, is presented. This model has three components: the temporal transformer, spatial transformer and spatiotemporal fusion modules. The dynamic temporal dependencies of each attraction are extracted efficiently by the temporal transformer module. The dynamic spatial correlations between attractions are extracted efficiently by the spatial transformer module. The extracted dynamic temporal and spatial features are fused in a learnable manner in the spatiotemporal fusion module. Convolutional operations are implemented to generate the final forecasts. Findings The results indicate that the proposed model performs better in forecasting accuracy than some popular benchmark models, demonstrating its significant forecasting performance. Incorporating dynamic spatiotemporal features is an effective strategy for improving forecasting. It can provide an important reference to related studies. Practical implications The proposed model leverages high-frequency data to achieve accurate predictions at the micro level by incorporating dynamic spatial effects. Destination managers should fully consider the dynamic spatial effects of attractions when planning and marketing to promote tourism resources. Originality/value This study incorporates dynamic spatial effects into tourism demand forecasting models by using a transformer neural network. It advances the development of methodologies in related fields.