Spatial-temporal Transformation Research Articles

PRECYSE: Predicting Cybersickness using Transformer for Multimodal Time-Series Sensor Data

Cybersickness, a factor that hinders user immersion in VR, has been the subject of ongoing attempts to predict it using AI. Previous studies have used CNN and LSTM for prediction models and used attention mechanisms and XAI for data analysis, yet none explored a transformer that can better reflect the spatial and temporal characteristics of the data, beneficial for enhancing prediction and feature importance analysis. In this paper, we propose cybersickness prediction models using multimodal time-series sensor data (i.e., eye movement, head movement, and physiological signals) based on a transformer algorithm, considering sensor data pre-processing and multimodal data fusion methods. We constructed the MSCVR dataset consisting of normalized sensor data, spectrogram formatted sensor data, and cybersickness levels collected from 45 participants through a user study. We proposed two methods for embedding multimodal time-series sensor data into the transformer: modality-specific spatial and temporal transformer encoders for normalized sensor data (MS-STTN) and modality-specific spatial-temporal transformer encoder for spectrogram (MS-STTS). MS-STTN yielded the highest performance in the ablation study and the comparison of the existing models. Furthermore, by analyzing the importance of data features, we determined their relevance to cybersickness over time, especially the salience of eye movement features. Our results and insights derived from multimodal time-series sensor data and the transformer model provide a comprehensive understanding of cybersickness and its association with sensor data. Our MSCVR dataset and code are publicly available: https://github.com/dayoung-jeong/PRECYSE.git.

A multi-channel spatial-temporal transformer model for traffic flow forecasting

Traffic flow forecasting is a crucial task in transportation management and planning. The main challenges for traffic flow forecasting are that (1) as the length of prediction time increases, the accuracy of prediction will decrease; (2) the predicted results greatly rely on the extraction of temporal and spatial dependencies from the road networks. To overcome the challenges mentioned above, we propose a multi-channel spatial-temporal transformer model for traffic flow forecasting, which improves the accuracy of the prediction by fusing results from different channels of traffic data. Our approach leverages graph convolutional network to extract spatial features from each channel while using a transformer-based architecture to capture temporal dependencies across channels. We introduce an adaptive adjacency matrix to overcome limitations in feature extraction from fixed topological structures. Experimental results on six real-world datasets demonstrate that introducing a multi-channel mechanism into the temporal model enhances performance and our proposed model outperforms state-of-the-art models in terms of accuracy.

HybridGait: A Benchmark for Spatial-Temporal Cloth-Changing Gait Recognition with Hybrid Explorations

Existing gait recognition benchmarks mostly include minor clothing variations in the laboratory environments, but lack persistent changes in appearance over time and space. In this paper, we propose the first in-the-wild benchmark CCGait for cloth-changing gait recognition, which incorporates diverse clothing changes, indoor and outdoor scenes, and multi-modal statistics over 92 days. To further address the coupling effect of clothing and viewpoint variations, we propose a hybrid approach HybridGait that exploits both temporal dynamics and the projected 2D information of 3D human meshes. Specifically, we introduce a Canonical Alignment Spatial-Temporal Transformer (CA-STT) module to encode human joint position-aware features, and fully exploit 3D dense priors via a Silhouette-guided Deformation with 3D-2D Appearance Projection (SilD) strategy. Our contributions are twofold: we provide a challenging benchmark CCGait that captures realistic appearance changes over expanded time and space, and we propose a hybrid framework HybridGait that outperforms prior works on CCGait and Gait3D benchmarks. Our project page is available at https://github.com/HCVLab/HybridGait.

STViT: Improving Self-Supervised Multi-Camera Depth Estimation with Spatial-Temporal Context and Adversarial Geometry Regularization (Student Abstract)

Multi-camera depth estimation has recently garnered significant attention due to its substantial practical implications in the realm of autonomous driving. In this paper, we delve into the task of self-supervised multi-camera depth estimation and propose an innovative framework, STViT, featuring several noteworthy enhancements: 1) we propose a Spatial-Temporal Transformer to comprehensively exploit both local connectivity and the global context of image features, meanwhile learning enriched spatial-temporal cross-view correlations to recover 3D geometry. 2) to alleviate the severe effect of adverse conditions, e.g., rainy weather and nighttime driving, we introduce a GAN-based Adversarial Geometry Regularization Module (AGR) to further constrain the depth estimation with unpaired normal-condition depth maps and prevent the model from being incorrectly trained. Experiments on challenging autonomous driving datasets Nuscenes and DDAD show that our method achieves state-of-the-art performance.

A Cross-Modal Generative Adversarial Network for Scenarios Generation of Renewable Energy

Scenarios data of renewable energy resources plays an essential role in the study of mitigating the risk in the power system due to their intermittent nature. Existing researches have mainly focused on how to generate high-quality time series based scenarios with corrupted or missing observed data. However, the multimodality of renewable energy resources is largely ignored from the perspective of the spatial-temporal correlation which can significantly strengthen the renewable energy prediction and optimization. To make full use of these multi-modal data and further improve the data quality of scenarios, a cross-modal Generative Adversarial Network (cGAN) model is proposed with the setting of two spatial-temporal transformer models. In cGAN, the task of scenarios generation is formulated as a probability approximation problem. Then, a cross-modal scenarios generation framework is constructed to process data from multi-modal observations. Theoretically, it can generate infinite number of uncertain scenarios data via the repeated random noise sampling technique. Extensive experiments are carried out at a NREL photovoltaic power dataset and a real-world wind power dataset, respectively. The results validate the state-of-the-art performance of cGAN, even though the validation dataset is missing at random. Moreover, the results also show that the setting of spatial-temporal transformer clearly explains the contributions of different modal data, and stabilize the training process of cGAN. Finally, a stochastic day-head economic dispatch is also studied to show the practical value of cGAN.

MVSTT: A Multiview Spatial-Temporal Transformer Network for Traffic-Flow Forecasting.

Accurate traffic-flow prediction remains a critical challenge due to complicated spatial dependencies, temporal factors, and unpredictable events. Most existing approaches focus on single-or dual-view learning and thus face limitations in systematically learning complex spatial-temporal features. In this work, we propose a novel multiview spatial-temporal transformer (MVSTT) network that can effectively learn complex spatial-temporal domain correlations and potential patterns from multiple views. First, we examine a temporal view and design a short-range gated convolution component from a short-term subview, and a long-range gated convolution component from a long-term subview. These two components effectively aggregate knowledge of the temporal domain at multiple granularities and mine patterns of node evolution across time steps. Meanwhile, in the spatial view, we design a dual-graph spatial learning module that captures fixed and dynamic spatial dependencies of nodes, as well as the evolution patterns of edges, from the static and dynamic graph subviews, respectively. In addition, we further design a spatial-temporal transformer to mine different levels of spatial-temporal features through multiview knowledge fusion. Extensive experiments on four real-world traffic datasets show that our method consistently outperforms the state-of-the-art baseline. The code of MVSTT is available at https://github.com/JianSoL/MVSTT.

Self-supervised spatial–temporal transformer fusion based federated framework for 4D cardiovascular image segmentation

Availability of high-quality large annotated data is indeed a significant challenge in healthcare. In addition, privacy concerns and data-sharing restrictions often hinder access to large and diverse medical image datasets. To reduce the requirement for annotated training data, self-supervised pre-training strategies on nonannotated data have been extensively used, whereas collaborative algorithm training without the need to exchange the underlying data. In this paper, we introduce a novel federated learning-based self-supervised spatial–temporal transformer’s fusion (SSFL) for cardiovascular image segmentation. The integration of spatial–temporal swin transformer is used to extract the features from 3D SAX multiple phases (full cycle of cardiac heart). An efficient self-supervised contrastive framework consisting of a spatial–temporal transformer network with 25 encoders is used to model the temporal features. The spatial and temporal features are fused and forwarded to the decoder for cardiac heart segmentation using cine MRI images. To further improve segmentation, we use an attention-based unpaired GAN model to map or transfer the style from ACDC to M&Ms and use synthetically generated volumes in the proposed self-supervised approach. Experiments with three different cardiovascular image segmentation tasks, such as segmentation of the right ventricle, left ventricle, and myocardium, showed significant improvement compared to the state-of-the-art segmentation framework.

Graph Spatial-Temporal Transformer Network for Traffic Prediction

Traffic information can reflect the operating status of a city, and accurate traffic forecasting is critical in intelligent transportation systems (ITS) and urban planning. However, traffic information has complex nonlinearity and dynamic spatial-temporal dependencies due to human mobility, bringing new traffic forecasting challenges. This paper proposed a graph spatial-temporal transformer network for traffic prediction (GSTTN) to cope with the above problems. Specifically, the proposed framework explores spatial characteristics of the across-road network of traffic information hidden in human behavior patterns via a multi-view graph convolutional network (GCN). Furthermore, the transformer network with a multi-head attention mechanism is adopted to capture the random disturbance in the time series characteristics of traffic information. As a result, these two components can be used to model spatial relations and temporal trends. Finally, we examine real-world datasets, and the experiments show that the proposed framework outperforms the current state-of-the-art baselines.

An Accurate Prediction Method of Human Assembly Motion for Human–Robot Collaboration

In the process of human–robot collaborative assembly, robots need to recognize and predict human behaviors accurately, and then perform autonomous control and work route planning in real-time. To support the judgment of human intervention behaviors and meet the need of real-time human–robot collaboration, the Fast Spatial–Temporal Transformer Network (FST-Trans), an accurate prediction method of human assembly actions, is proposed. We tried to maximize the symmetry between the prediction results and the actual action while meeting the real-time requirement. With concise and efficient structural design, FST-Trans can learn about the spatial–temporal interactions of human joints during assembly in the same latent space and capture more complex motion dynamics. Considering the inconsistent assembly rates of different individuals, the network is forced to learn more motion variations by introducing velocity–acceleration loss, realizing accurate prediction of assembly actions. An assembly dataset was collected and constructed for detailed comparative experiments and ablation studies, and the experimental results demonstrate the effectiveness of the proposed method.

Landscape pattern changes in coal resource-based cities: A case study of Huainan City in China

Currently, coal resource-based cities (CRBCs) are facing challenges such as ecological destruction, resource exhaustion, and disordered urban development. By analyzing the landscape pattern, the understanding of urban land use can be clarified, and optimization strategies can be proposed for urban transformation and sustainable development. In this study, based on the interpretation of remote sensing data for three dates, the landscape pattern changes in the urban area of Huainan City, a typical coal resource-based city in Anhui Province, China were empirically investigated. The results indicate that: (1) There is a significant spatial-temporal transformation of land use, with construction land gradually replacing arable land as the dominant land use type in the region. (2) Landscape indices are helpful to reveal the characteristics of land transfer and distribution of human activities during a process. At the landscape type level, construction land, grassland, and water bodies are increasingly affected by human activities. At the landscape composition level, the number of landscape types increases, and the distribution of different types of patches becomes more balanced. In addition, to address the problems caused by the coal mining subsidence areas in Huainan city, three landscape pattern optimization strategies are proposed at both macro and micro levels. The research findings contribute to a better understanding of land use changes and their driving forces, and offer valuable alternatives for ecological environment optimization.

Event Graph Guided Compositional Spatial-Temporal Reasoning for Video Question Answering.

Video question answering (VideoQA) is challenging since it requires the model to extract and combine multi-level visual concepts from local objects to global actions from complex events for compositional reasoning. Existing works represent the video with fixed-duration clip features that make the model struggle in capturing the crucial concepts in multiple granularities. To overcome this shortcoming, we propose to represent the video with an Event Graph in a hierarchical structure whose nodes correspond to visual concepts of different levels (object, relation, scene and action) and edges indicate their spatial-temporal relationships. We further propose a H ierarchical S patial- T emporal T ransformer (HSTT) which takes nodes from the graph as visual input to realize compositional reasoning guided by the event graph. To fully exploit the spatial-temporal context delivered from the graph structure, on the one hand, we encode the nodes in the order of their semantic hierarchy (depth) and occurrence time (breadth) with our improved graph search algorithm; On the other hand, we introduce edge-guided attention to combine the spatial-temporal context among nodes according to their edge connections. HSTT then performs QA by cross-modal interactions guaranteed by the hierarchical correspondence between the multi-level event graph and the cross-level question. Experiments on the recent challenging AGQA and STAR datasets show that the proposed method clearly outperforms the existing VideoQA models by a large margin, including those pre-trained with large-scale external data. Our code is available at https://github.com/ByZ0e/HSTT.

A Spatial-Temporal Transformer Network for City-Level Cellular Traffic Analysis and Prediction

With the accelerated popularization of 5G applications, accurate cellular traffic prediction is becoming increasingly important for efficient network management. Currently, the latest algorithms for cellular traffic prediction generally neglect extraction of the shallow features of cellular traffic and the prediction accuracy is hence limited. Therefore, we propose a global-local spatial-temporal transformer network (GLSTTN) that can fully excavate diverse spatial-temporal characteristics of cellular traffic for accurate cellular traffic prediction. Specifically, GLSTTN achieves this goal by constructing two modules: the global spatial-temporal module and the local spatial-temporal module. In the global spatial-temporal module, GLSTTN captures global correlations using stacked spatial-temporal blocks, where each block is composed of one spatial transformer and one temporal transformer. A skip connection is then used in each block to strengthen feature propagation. In the local spatial-temporal module, GLSTTN fully extracts the local spatial-temporal dependencies hidden in globally encoded features using densely connected convolutional neural networks. Extensive experiments demonstrate that GLSTTN achieves more accurate cellular traffic prediction than existing approaches on a real-world cellular traffic dataset.

Tourism demand forecasting: a deep learning model based on spatial-temporal transformer

Purpose Incorporating dynamic spatial effects exhibits considerable potential in improving the accuracy of forecasting tourism demands. This study aims to propose an innovative deep learning model for capturing dynamic spatial effects. Design/methodology/approach A novel deep learning model founded on the transformer architecture, called the spatiotemporal transformer network, is presented. This model has three components: the temporal transformer, spatial transformer and spatiotemporal fusion modules. The dynamic temporal dependencies of each attraction are extracted efficiently by the temporal transformer module. The dynamic spatial correlations between attractions are extracted efficiently by the spatial transformer module. The extracted dynamic temporal and spatial features are fused in a learnable manner in the spatiotemporal fusion module. Convolutional operations are implemented to generate the final forecasts. Findings The results indicate that the proposed model performs better in forecasting accuracy than some popular benchmark models, demonstrating its significant forecasting performance. Incorporating dynamic spatiotemporal features is an effective strategy for improving forecasting. It can provide an important reference to related studies. Practical implications The proposed model leverages high-frequency data to achieve accurate predictions at the micro level by incorporating dynamic spatial effects. Destination managers should fully consider the dynamic spatial effects of attractions when planning and marketing to promote tourism resources. Originality/value This study incorporates dynamic spatial effects into tourism demand forecasting models by using a transformer neural network. It advances the development of methodologies in related fields.

UAV-based Anomaly Detection via a novel Spatial-Temporal Transformer for Precision Agriculture

UAV-based Anomaly Detection via a novel Spatial-Temporal Transformer for Precision Agriculture

Demand forecasting and predictability identification of ride-sourcing via bidirectional spatial-temporal transformer neural processes

Understanding the spatial–temporal stochasticity in shared mobility is crucial for ride-sourcing demand forecasting, supply–demand management, and vehicle dispatch optimization. In contrast to conventional deep learning methods that typically provide point predictions or deterministic predictions, this paper introduces the bidirectional spatial–temporal Transformer neural processes (Bi-STTNP) prediction model, which stands out from conventional deep learning methods by providing probabilistic predictions and uncertainty estimations for ride-sourcing demand. Bi-STTNP captures the multivariate spatial–temporal Gaussian distribution of demand, offering not only demand expectations but also comprehensive uncertainty representations. We propose a predictability identification process based on predictive distributions to assess varying predictability across time slots and regions, improving interpretability. Our model, consisting of the bidirectional supply–demand attention module and spatial–temporal Transformer module, maintains interpretability while ensuring accurate demand expectation predictions. Extensive experiments on a real-world dataset of 15 million ride-sourcing orders in Hangzhou, China, demonstrate that Bi-STTNP outperforms baseline models in predicting demand expectation and quantifying demand uncertainty. Furthermore, we compute loose spatial–temporal predictability lower bounds and categorize regions by predictability, providing insights for optimizing passenger pricing strategies, driver incentives, and vehicle dispatching in ride-sourcing platforms.

Joint spatio-temporal modeling for visual tracking

Similarity-based approaches have made significant progress in visual object tracking (VOT). Although these methods work well in simple scenes, they ignore the continuous spatio-temporal connection of the object in the video sequence. For this reason, tracking by spatial matching solely can lead to tracking failures because of distractors and occlusion. In this paper, we propose a spatio-temporal joint-modeling tracker named STTrack which implicitly builds continuous connections between the temporal and spatial aspects of the sequence. Specifically, we first design a time-sequence iteration strategy (TSIS) to concentrate on the temporal connection of the object in the video sequence. Then, we propose a novel spatial temporal interaction Transformer network (STIN) to capture the spatio-temporal correlation of the object between frames. The proposed STIN module is robust in object occlusion because it explores the dynamic state change dependencies of the object. Finally, we introduce a spatio-temporal query to suppress distractors by iteratively propagating the target prior. Extensive experiments on six tracking benchmark datasets demonstrate that the proposed STTrack achieves excellent performance while operating in real-time. The code is publicly available at https://github.com/nubsym/STTrack.

Research on Traffic Flow Forecasting Based on Dynamic Spatial-Temporal Transformer

Accurate traffic flow forecasting is crucial for urban traffic control and route planning. Aiming at the difficulty in capturing dynamic spatio-temporal complexity of traffic flow, a dynamic spatio-temporal transformer (DST-Trans) model capable of modeling dynamic correlation of traffic flow is proposed, which consists of gated temporal convolutional network (GTCN), graph convolutional network (GCN), and spatio-temporal transformer (ST-TF). GTCN and GCN are utilized to capture the temporal and spatial characteristics of traffic flow, respectively. ST-TF includes a temporal transformer using temporal gated convolution and temporal multi-head self-attention to capture short-long term temporal features, and spatial transformer using spatial gated graph convolution and spatial multi-head self-attention to capture local-global dynamic spatial features. In addition, to take full advantage of the dynamic and static associations of road networks, multi-graph models of road relationship graph, similarity graph, and adaptive dynamic graph with SGGC are constructed. Experimental results show that the DST-Trans model in this paper shows good prediction performance in short-term (15 min), medium-term (30 min), and long-term (60 min) prediction, outperforming existing state-of-the-art models by up to approximately 7%.

A Lightweight and Accurate Spatial-Temporal Transformer for Traffic Forecasting

A Lightweight and Accurate Spatial-Temporal Transformer for Traffic Forecasting

FSTT: Flow-Guided Spatial Temporal Transformer for Deep Video Inpainting

Video inpainting aims to complete the missing regions with content that is consistent both spatially and temporally. How to effectively utilize the spatio-temporal information in videos is critical for video inpainting. Recent advances in video inpainting methods combine both optical flow and transformers to capture spatio-temporal information. However, these methods fail to fully explore the potential of optical flow within the transformer. Furthermore, the designed transformer block cannot effectively integrate spatio-temporal information across frames. To address the above problems, we propose a novel video inpainting model, named Flow-Guided Spatial Temporal Transformer (FSTT), which effectively establishes correspondences between missing regions and valid regions in both spatial and temporal dimensions under the guidance of completed optical flow. Specifically, a Flow-Guided Fusion Feed-Forward module is developed to enhance features with the assistance of optical flow, mitigating the inaccuracies caused by hole pixels when performing MHSA. Additionally, a decomposed spatio-temporal MHSA module is proposed to effectively capture spatio-temporal dependencies in videos. To improve the efficiency of the model, a Global–Local Temporal MHSA module is further designed based on the window partition strategy. Extensive quantitative and qualitative experiments on the DAVIS and YouTube-VOS datasets demonstrate the superiority of our proposed method.

A decentralized federated learning-based spatial–temporal model for freight traffic speed forecasting

Accurately understanding the spatial–temporal information of future freight traffic speed in the metropolitan area is of vital importance to formulate freight-related traffic management strategies. In this study, we develop a novel decentralized federated learning-based spatial–temporal model for freight traffic speed forecasting while implementing collaborative training among multiple participants instead of requiring an exclusive cloud center server for centralized data processing. First, a tailored spatial–temporal transformer network is proposed to substitute the existing graph convolutional network for local personalized learning of each participant. Second, a decentralized federated learning model is designed to fuse local personalization models for freight traffic speed forecasting. The associated convergence properties are theoretically illustrated. rgb]0,0,0Finally, the experiments based on a real-world freight dataset of member cities in the Nanjing Metropolitan Area demonstrate that the proposed approach can accurately forecast freight traffic speed and outperform existing traffic speed forecasting methods, with the average improvement of 8.3% for MAE, 8.2% for RMSE, and 8.6% for MAPE respectively. The visualization results reveal that the proposed approach is able to effectively capture the internal spatial–temporal dependencies among urban regions from various neighboring cities, providing the insights for collaboratively developing proactive freight-related traffic management strategies.