Spatial-temporal Transformation Research Articles

Spatial–Temporal Variations in Water Use Efficiency and Its Influencing Factors in the Li River Basin, China

As a vital indicator for measuring the coupled carbon–water cycle of an ecosystem, water use efficiency (WUE) can also reflect the adaptive capacity of plants in different ecosystems. Located in Southwest China, the Li River Basin has a representative karst landform, and the uneven rainfall in the region leads to severe water shortage. In this study, we analyzed the spatial–temporal transformation characteristics of the WUE of the basin and its relationship with different influencing factors from 2001 to 2020 based on a correlation analysis and trend analysis. The main conclusions are as follows: (1) The average value of WUE in the Li River Basin was 1.8251 gC· mm−1·m−2, and it kept decreasing at a rate of 0.0072 gC· mm−1·m−2·a−1 in the past 20 years. With respect to the spatial distribution of the multi-year average of WUE, it exhibits a gradual increasing trend from west to east. (2) Between gross primary productivity (GPP) and evapotranspiration (ET), it was found that ET was the primary influencing factor of WUE. Precipitation was positively correlated with WUE in the Li River Basin, accounting for 67.22% of the total area of the basin. The air temperature was negatively correlated with WUE, and the area was negatively correlated with WUE, accounting for 92.67% of the basin area. (3) The normalized difference vegetation index (NDVI) and leaf area index (LAI) were negatively correlated with WUE, and the proportions of negatively correlated areas to the total area of the basin were similar; both were between 60 and 70%. The growth of vegetation inhibited the increase in WUE in the basin to a certain extent. Regarding Vapor Pressure Deficit (VPD), the proportions of positive and negative correlation areas with WUE were similar, accounting for 49.58% and 50.42%, respectively. (4) The occurrence of drought events and the enhancement in its degree led to a continuous increase in WUE in the basin; for different land cover types, the correlation of the standardized precipitation evapotranspiration index (SPEI) was in the following order from strongest to weakest: grassland > cropland > forest > shrubland.

Enhancing Turbidity Predictions in Coastal Environments by Removing Obstructions from Unmanned Aerial Vehicle Multispectral Imagery Using Inpainting Techniques

High-resolution remote sensing of turbidity in the coastal environment with unmanned aerial vehicles (UAVs) can be adversely affected by the presence of obstructions of vessels and marine objects in images, which can introduce significant errors in turbidity modeling and predictions. This study evaluates the use of two deep-learning-based inpainting methods, namely, Decoupled Spatial–Temporal Transformer (DSTT) and Deep Image Prior (DIP), to recover the obstructed information. Aerial images of turbidity plumes in the coastal environment were first acquired using a UAV system with a multispectral sensor that included obstructions on the water surface at various obstruction percentages. The performance of the two inpainting models was then assessed through both qualitative and quantitative analyses of the inpainted data, focusing on the accuracy of turbidity retrieval. The results show that the DIP model performs well across a wide range of obstruction percentages from 10 to 70%. In comparison, the DSTT model produces good accuracy only with low percentages of less than 20% and performs poorly when the obstruction percentage increases.

Multi-View Graph Learning for Path-Level Aging-Aware Timing Prediction

As CMOS technology continues to scale down, the aging effect—known as negative bias temperature instability (NBTI)—has become increasingly prominent, gradually emerging as a key factor affecting device reliability. Accurate aging-aware static timing analysis (STA) at the early design phase is critical for establishing appropriate timing margins to ensure circuit reliability throughout the chip lifecycle. However, traditional aging-aware timing analysis methods, typically based on Simulation Program with Integrated Circuit Emphasis (SPICE) simulations or aging-aware timing libraries, struggle to balance prediction accuracy with computational cost. In this paper, we propose a multi-view graph learning framework for path-level aging-aware timing prediction, which combines the strengths of the spatial–temporal Transformer network (STTN) and graph attention network (GAT) models to extract the aging timing features of paths from both timing-sensitive and workload-sensitive perspectives. Experimental results demonstrate that our proposed framework achieves an average MAPE score of 3.96% and reduces the average MAPE by 5.8 times compared to FFNN and 2.2 times compared to PNA, while maintaining acceptable increases in processing time.

Spatial-Temporal Transformer Networks for Traffic Flow Forecasting Using a Pre-Trained Language Model.

Most current methods use spatial-temporal graph neural networks (STGNNs) to analyze complex spatial-temporal information from traffic data collected from hundreds of sensors. STGNNs combine graph neural networks (GNNs) and sequence models to create hybrid structures that allow for the two networks to collaborate. However, this collaboration has made the model increasingly complex. This study proposes a framework that relies solely on original Transformer architecture and carefully designs embeddings to efficiently extract spatial-temporal dependencies in traffic flow. Additionally, we used pre-trained language models to enhance forecasting performance. We compared our new framework with current state-of-the-art STGNNs and Transformer-based models using four real-world traffic datasets: PEMS04, PEMS08, METR-LA, and PEMS-BAY. The experimental results demonstrate that our framework outperforms the other models in most metrics.

Exploring the conflict risk characteristics of air weaving sections in Metroplex terminal areas with flight trajectory data and adaptive graph spatial-temporal transformer

Metroplex terminal areas is involved with numerous air weaving sections, where various types of arrival and departure traffic flows intersect or converge, leading to high-risk aircraft conflicts and inefficient Metroplex operation. The primary objective of this study is to explore the conflict risk characteristics of operational interactions between airports in Metroplex terminal areas with large-scale trajectory data, including risk evaluation, risk propagation and risk prediction. One complete month of ADS-B trajectory data is collected from a representative multi-airport system in Central and South China to illustrate the procedure. Specifically, flight trajectories are firstly clustered within Metroplex terminal areas, and a trajectory-tube model is then built to characterize the airport flows within the same cluster. Then, a clustering-based collision probability calculation method is proposed to evaluate the risk of air weaving sections. The spatial patterns of risks in air weaving sections reveal that operational interactions among Metroplex airports are quite different under various runway configurations. Furthermore, an adaptive graph spatial-temporal transformer (ASTT) network is developed to predict the future risk of each air weaving section in Metroplex terminal areas. The results indicate that the developed ASTT network can collaboratively encode the spatiotemporal characteristics in the air weaving section risk data, which achieves more accurate and robust prediction performance than other compared models during multiple look-ahead time steps. Moreover, the adaptive spatial graph technique designed in the ASTT can also reveal the hidden risk propagation effects among air weaving sections. The results of this study could provide insightful suggestions to air traffic authorities for developing effective coordination and conflict mitigation strategies, such as redesigning flight procedures, reallocating routes, and optimizing aircraft sequences to enhance the efficiency and safety of Metroplex operations.

Emotion recognition using hierarchical spatial–temporal learning transformer from regional to global brain

Emotion recognition is an essential but challenging task in human–computer interaction systems due to the distinctive spatial structures and dynamic temporal dependencies associated with each emotion. However, current approaches fail to accurately capture the intricate effects of electroencephalogram (EEG) signals across different brain regions on emotion recognition. Therefore, this paper designs a transformer-based method, denoted by R2G-STLT, which relies on a spatial–temporal transformer encoder with regional to global hierarchical learning that learns the representative spatiotemporal features from the electrode level to the brain-region level. The regional spatial–temporal transformer (RST-Trans) encoder is designed to obtain spatial information and context dependence at the electrode level aiming to learn the regional spatiotemporal features. Then, the global spatial–temporal transformer (GST-Trans) encoder is utilized to extract reliable global spatiotemporal features, reflecting the impact of various brain regions on emotion recognition tasks. Moreover, the multi-head attention mechanism is placed into the GST-Trans encoder to empower it to capture the long-range spatial–temporal information among the brain regions. Finally, subject-independent experiments are conducted on each frequency band of the DEAP, SEED, and SEED-IV datasets to assess the performance of the proposed model. Results indicate that the R2G-STLT model surpasses several state-of-the-art approaches.

An Intelligent Deep Learning Framework for Traffic Flow Imputation and Short-term Prediction Based on Dynamic Features

The accurate prediction of traffic flow has emerged as a focal point in the cutting-edge sphere of intelligent transportation. Extant methodologies rely on deep learning for short-term forecasts but are hampered by two critical limitations: the omission of vital traffic flow data and an overemphasis on static rather than dynamic attributes. This paper presents a solution to these challenges through the creation of a dynamic attention generative adversarial network (DATGAN) that is specifically engineered to impute missing traffic data accurately. DATGAN introduces two groundbreaking features: first, an innovative attention module that is adept at identifying local correlations, and second, novel spatial-correction graph convolutional layers that are designed to capture the ever-changing spatial dynamic dependencies. Building on this foundation, the parallel spatial–temporal Transformer (PSTTransformer) is proposed, which integrates both spatial and temporal attention blocks to manage dynamic spatial-temporal dependencies for future short-term traffic predictions. This dual-block approach not only embeds periodic information via a fusion self-attention mechanism in the temporal attention blocks, but also synergizes graph convolutional networks with self-attention mechanisms in the spatial attention blocks through a gate mechanism. A comprehensive comparative analysis conducted on traffic datasets revealed that our methodology outperforms current leading models by reducing the mean squared error by 5% in data imputation tasks and shows robust performance in traffic prediction scenarios.

STTD: spatial-temporal transformer with double recurrent graph convolutional cooperative network for traffic flow prediction

STTD: spatial-temporal transformer with double recurrent graph convolutional cooperative network for traffic flow prediction

PRECYSE: Predicting Cybersickness using Transformer for Multimodal Time-Series Sensor Data

Cybersickness, a factor that hinders user immersion in VR, has been the subject of ongoing attempts to predict it using AI. Previous studies have used CNN and LSTM for prediction models and used attention mechanisms and XAI for data analysis, yet none explored a transformer that can better reflect the spatial and temporal characteristics of the data, beneficial for enhancing prediction and feature importance analysis. In this paper, we propose cybersickness prediction models using multimodal time-series sensor data (i.e., eye movement, head movement, and physiological signals) based on a transformer algorithm, considering sensor data pre-processing and multimodal data fusion methods. We constructed the MSCVR dataset consisting of normalized sensor data, spectrogram formatted sensor data, and cybersickness levels collected from 45 participants through a user study. We proposed two methods for embedding multimodal time-series sensor data into the transformer: modality-specific spatial and temporal transformer encoders for normalized sensor data (MS-STTN) and modality-specific spatial-temporal transformer encoder for spectrogram (MS-STTS). MS-STTN yielded the highest performance in the ablation study and the comparison of the existing models. Furthermore, by analyzing the importance of data features, we determined their relevance to cybersickness over time, especially the salience of eye movement features. Our results and insights derived from multimodal time-series sensor data and the transformer model provide a comprehensive understanding of cybersickness and its association with sensor data. Our MSCVR dataset and code are publicly available: https://github.com/dayoung-jeong/PRECYSE.git.

A multi-channel spatial-temporal transformer model for traffic flow forecasting

Traffic flow forecasting is a crucial task in transportation management and planning. The main challenges for traffic flow forecasting are that (1) as the length of prediction time increases, the accuracy of prediction will decrease; (2) the predicted results greatly rely on the extraction of temporal and spatial dependencies from the road networks. To overcome the challenges mentioned above, we propose a multi-channel spatial-temporal transformer model for traffic flow forecasting, which improves the accuracy of the prediction by fusing results from different channels of traffic data. Our approach leverages graph convolutional network to extract spatial features from each channel while using a transformer-based architecture to capture temporal dependencies across channels. We introduce an adaptive adjacency matrix to overcome limitations in feature extraction from fixed topological structures. Experimental results on six real-world datasets demonstrate that introducing a multi-channel mechanism into the temporal model enhances performance and our proposed model outperforms state-of-the-art models in terms of accuracy.

An efficient spatial-temporal transformer with temporal aggregation and spatial memory for traffic forecasting

Traffic forecasting technology has widespread applications in various domains, such as urban traffic planning and intelligent transportation systems. Traffic forecasting encounters challenges in effectively capturing the intricate spatial–temporal correlations in traffic data. While the latest methods have achieved satisfactory performance, they still suffer from two limitations: (i) Most methods overlook the memory of valuable traffic patterns at each traffic node, thus making it struggle to reveal dynamic spatial–temporal correlations using their inherent periodicity and trend characteristics from a broader perspective. (ii) As the research progresses, recently proposed models become increasingly complex and massive. To address these issues, we propose a Spatial-Temporal Aggregation Memory Transformer (STAMT) for traffic forecasting. Specifically, we propose a memory bank to enhance vanilla spatial attention and cache the traffic patterns of historical input. By querying these traffic patterns, which contain rich spatial–temporal semantic information, the model can optimize prediction performance by extracting trends and regularities across various periods. To reduce the computational costs, we introduce a temporal module to capture temporal correlations while reducing temporal dimension information. In addition, we leverage the random feature map and matrix multiplication associativity property, which reduce the quadratic complexity of spatial attention to linearity with regard to the number of nodes. Ultimately, our theoretical analysis concludes that a single-layer spatial attention network is sufficient to capture spatial–temporal correlations deeply without stacking. Extensive experiments on nine real-world datasets demonstrate that STAMT outperforms state-of-the-art baselines in regular, long-range, and large-scale traffic forecasting tasks while significantly reducing the computational costs. Codes are available at https://github.com/LiuAoyu1998/STAMT.

HybridGait: A Benchmark for Spatial-Temporal Cloth-Changing Gait Recognition with Hybrid Explorations

Existing gait recognition benchmarks mostly include minor clothing variations in the laboratory environments, but lack persistent changes in appearance over time and space. In this paper, we propose the first in-the-wild benchmark CCGait for cloth-changing gait recognition, which incorporates diverse clothing changes, indoor and outdoor scenes, and multi-modal statistics over 92 days. To further address the coupling effect of clothing and viewpoint variations, we propose a hybrid approach HybridGait that exploits both temporal dynamics and the projected 2D information of 3D human meshes. Specifically, we introduce a Canonical Alignment Spatial-Temporal Transformer (CA-STT) module to encode human joint position-aware features, and fully exploit 3D dense priors via a Silhouette-guided Deformation with 3D-2D Appearance Projection (SilD) strategy. Our contributions are twofold: we provide a challenging benchmark CCGait that captures realistic appearance changes over expanded time and space, and we propose a hybrid framework HybridGait that outperforms prior works on CCGait and Gait3D benchmarks. Our project page is available at https://github.com/HCVLab/HybridGait.

STViT: Improving Self-Supervised Multi-Camera Depth Estimation with Spatial-Temporal Context and Adversarial Geometry Regularization (Student Abstract)

Multi-camera depth estimation has recently garnered significant attention due to its substantial practical implications in the realm of autonomous driving. In this paper, we delve into the task of self-supervised multi-camera depth estimation and propose an innovative framework, STViT, featuring several noteworthy enhancements: 1) we propose a Spatial-Temporal Transformer to comprehensively exploit both local connectivity and the global context of image features, meanwhile learning enriched spatial-temporal cross-view correlations to recover 3D geometry. 2) to alleviate the severe effect of adverse conditions, e.g., rainy weather and nighttime driving, we introduce a GAN-based Adversarial Geometry Regularization Module (AGR) to further constrain the depth estimation with unpaired normal-condition depth maps and prevent the model from being incorrectly trained. Experiments on challenging autonomous driving datasets Nuscenes and DDAD show that our method achieves state-of-the-art performance.

A Cross-Modal Generative Adversarial Network for Scenarios Generation of Renewable Energy

Scenarios data of renewable energy resources plays an essential role in the study of mitigating the risk in the power system due to their intermittent nature. Existing researches have mainly focused on how to generate high-quality time series based scenarios with corrupted or missing observed data. However, the multimodality of renewable energy resources is largely ignored from the perspective of the spatial-temporal correlation which can significantly strengthen the renewable energy prediction and optimization. To make full use of these multi-modal data and further improve the data quality of scenarios, a cross-modal Generative Adversarial Network (cGAN) model is proposed with the setting of two spatial-temporal transformer models. In cGAN, the task of scenarios generation is formulated as a probability approximation problem. Then, a cross-modal scenarios generation framework is constructed to process data from multi-modal observations. Theoretically, it can generate infinite number of uncertain scenarios data via the repeated random noise sampling technique. Extensive experiments are carried out at a NREL photovoltaic power dataset and a real-world wind power dataset, respectively. The results validate the state-of-the-art performance of cGAN, even though the validation dataset is missing at random. Moreover, the results also show that the setting of spatial-temporal transformer clearly explains the contributions of different modal data, and stabilize the training process of cGAN. Finally, a stochastic day-head economic dispatch is also studied to show the practical value of cGAN.

MVSTT: A Multiview Spatial-Temporal Transformer Network for Traffic-Flow Forecasting.

Accurate traffic-flow prediction remains a critical challenge due to complicated spatial dependencies, temporal factors, and unpredictable events. Most existing approaches focus on single-or dual-view learning and thus face limitations in systematically learning complex spatial-temporal features. In this work, we propose a novel multiview spatial-temporal transformer (MVSTT) network that can effectively learn complex spatial-temporal domain correlations and potential patterns from multiple views. First, we examine a temporal view and design a short-range gated convolution component from a short-term subview, and a long-range gated convolution component from a long-term subview. These two components effectively aggregate knowledge of the temporal domain at multiple granularities and mine patterns of node evolution across time steps. Meanwhile, in the spatial view, we design a dual-graph spatial learning module that captures fixed and dynamic spatial dependencies of nodes, as well as the evolution patterns of edges, from the static and dynamic graph subviews, respectively. In addition, we further design a spatial-temporal transformer to mine different levels of spatial-temporal features through multiview knowledge fusion. Extensive experiments on four real-world traffic datasets show that our method consistently outperforms the state-of-the-art baseline. The code of MVSTT is available at https://github.com/JianSoL/MVSTT.

Self-supervised spatial–temporal transformer fusion based federated framework for 4D cardiovascular image segmentation

Availability of high-quality large annotated data is indeed a significant challenge in healthcare. In addition, privacy concerns and data-sharing restrictions often hinder access to large and diverse medical image datasets. To reduce the requirement for annotated training data, self-supervised pre-training strategies on nonannotated data have been extensively used, whereas collaborative algorithm training without the need to exchange the underlying data. In this paper, we introduce a novel federated learning-based self-supervised spatial–temporal transformer’s fusion (SSFL) for cardiovascular image segmentation. The integration of spatial–temporal swin transformer is used to extract the features from 3D SAX multiple phases (full cycle of cardiac heart). An efficient self-supervised contrastive framework consisting of a spatial–temporal transformer network with 25 encoders is used to model the temporal features. The spatial and temporal features are fused and forwarded to the decoder for cardiac heart segmentation using cine MRI images. To further improve segmentation, we use an attention-based unpaired GAN model to map or transfer the style from ACDC to M&Ms and use synthetically generated volumes in the proposed self-supervised approach. Experiments with three different cardiovascular image segmentation tasks, such as segmentation of the right ventricle, left ventricle, and myocardium, showed significant improvement compared to the state-of-the-art segmentation framework.

Graph Spatial-Temporal Transformer Network for Traffic Prediction

Traffic information can reflect the operating status of a city, and accurate traffic forecasting is critical in intelligent transportation systems (ITS) and urban planning. However, traffic information has complex nonlinearity and dynamic spatial-temporal dependencies due to human mobility, bringing new traffic forecasting challenges. This paper proposed a graph spatial-temporal transformer network for traffic prediction (GSTTN) to cope with the above problems. Specifically, the proposed framework explores spatial characteristics of the across-road network of traffic information hidden in human behavior patterns via a multi-view graph convolutional network (GCN). Furthermore, the transformer network with a multi-head attention mechanism is adopted to capture the random disturbance in the time series characteristics of traffic information. As a result, these two components can be used to model spatial relations and temporal trends. Finally, we examine real-world datasets, and the experiments show that the proposed framework outperforms the current state-of-the-art baselines.

An Accurate Prediction Method of Human Assembly Motion for Human–Robot Collaboration

In the process of human–robot collaborative assembly, robots need to recognize and predict human behaviors accurately, and then perform autonomous control and work route planning in real-time. To support the judgment of human intervention behaviors and meet the need of real-time human–robot collaboration, the Fast Spatial–Temporal Transformer Network (FST-Trans), an accurate prediction method of human assembly actions, is proposed. We tried to maximize the symmetry between the prediction results and the actual action while meeting the real-time requirement. With concise and efficient structural design, FST-Trans can learn about the spatial–temporal interactions of human joints during assembly in the same latent space and capture more complex motion dynamics. Considering the inconsistent assembly rates of different individuals, the network is forced to learn more motion variations by introducing velocity–acceleration loss, realizing accurate prediction of assembly actions. An assembly dataset was collected and constructed for detailed comparative experiments and ablation studies, and the experimental results demonstrate the effectiveness of the proposed method.

Landscape pattern changes in coal resource-based cities: A case study of Huainan City in China

Currently, coal resource-based cities (CRBCs) are facing challenges such as ecological destruction, resource exhaustion, and disordered urban development. By analyzing the landscape pattern, the understanding of urban land use can be clarified, and optimization strategies can be proposed for urban transformation and sustainable development. In this study, based on the interpretation of remote sensing data for three dates, the landscape pattern changes in the urban area of Huainan City, a typical coal resource-based city in Anhui Province, China were empirically investigated. The results indicate that: (1) There is a significant spatial-temporal transformation of land use, with construction land gradually replacing arable land as the dominant land use type in the region. (2) Landscape indices are helpful to reveal the characteristics of land transfer and distribution of human activities during a process. At the landscape type level, construction land, grassland, and water bodies are increasingly affected by human activities. At the landscape composition level, the number of landscape types increases, and the distribution of different types of patches becomes more balanced. In addition, to address the problems caused by the coal mining subsidence areas in Huainan city, three landscape pattern optimization strategies are proposed at both macro and micro levels. The research findings contribute to a better understanding of land use changes and their driving forces, and offer valuable alternatives for ecological environment optimization.

Tagged-to-Cine MRI Sequence Synthesis via Light Spatial-Temporal Transformer.

Tagged magnetic resonance imaging (MRI) has been successfully used to track the motion of internal tissue points within moving organs. Typically, to analyze motion using tagged MRI, cine MRI data in the same coordinate system are acquired, incurring additional time and costs. Consequently, tagged-to-cine MR synthesis holds the potential to reduce the extra acquisition time and costs associated with cine MRI, without disrupting downstream motion analysis tasks. Previous approaches have processed each frame independently, thereby overlooking the fact that complementary information from occluded regions of the tag patterns could be present in neighboring frames exhibiting motion. Furthermore, the inconsistent visual appearance, e.g., tag fading, across frames can reduce synthesis performance. To address this, we propose an efficient framework for tagged-to-cine MR sequence synthesis, leveraging both spatial and temporal information with relatively limited data. Specifically, we follow a split-and-integral protocol to balance spatialtemporal modeling efficiency and consistency. The light spatial-temporal transformer (LiST2) is designed to exploit the local and global attention in motion sequence with relatively lightweight training parameters. The directional product relative position-time bias is adapted to make the model aware of the spatial-temporal correlation, while the shifted window is used for motion alignment. Then, a recurrent sliding fine-tuning (ReST) scheme is applied to further enhance the temporal consistency. Our framework is evaluated on paired tagged and cine MRI sequences, demonstrating superior performance over comparison methods.