Spatio-temporal Network Research Articles

A dynamic electric fence planning framework for dockless bike-sharing systems based on inventory prediction

While dockless bike-sharing systems are growing in popularity due to their convenience, indiscriminate parking creates disorder in cities. To address this problem, this paper proposes a novel prediction-based approach for planning dynamic electric fences. These fences can adapt to dynamic usage patterns and efficiently guide parking behavior. First, graded clustering and spatial point integration algorithms are used to determine the locations of electric fence candidates. Second, initial parking and scheduling simulations are developed to calculate the inventory of each candidate. Finally, a spatiotemporal graph neural network is utilized to predict inventory and generate real-time deployment plans for electric fences. Case studies are conducted in two regions in Shanghai, China. Compared to a non-electric fence scenario, extensive experiments show that our framework can satisfy 98.6% of the parking demand and reduce spatial entropy by 15% within a reasonable walking distance. The results contribute to improving urban orderliness and promoting the sustainability of bike-sharing systems.

A coupled multi-factor synergistic graph network model for traffic prediction

Abstract Since only the spatio-temporal structure itself is considered in the current traffic prediction process, and there is a lack of multi-factor information about the scene, which leads to problems such as lack of universality and authenticity in the spatio-temporal scene, we proposed a coupled multi-factor synergistic graph network (CM-SGN). For this network, this paper firstly adopts a fusion mechanism to fuse traffic features with coupled scene features, constructs a complex feature correlation matrix, and enhances the multi-feature fusion effect of spatio-temporal scenes; secondly, it introduces a spatio-temporal feature network constrained by an attention mechanism to ensure the spatio-temporal stability of the fused features; and lastly, it reduces the sensitivity of the scene-traffic interdependent targets by using the regression layer combinations of the loss function to improve the prediction generalization capability. Experiments were conducted on two real datasets. Experiments were conducted on two real datasets, and the results show that the model proposed in this paper meets the complex spatio-temporal scenarios in traffic prediction, effectively captures the intrinsic spatio-temporal dependencies, and has good stability and generalization ability.

Dynamic mode decomposition and short-time prediction of PM2.5 using the graph Neural Koopman network

Analyzing and accurately predicting the spatiotemporal dynamics of PM2.5 remain challenging. The existing spatiotemporal prediction approaches are associated with high model complexity and limited interpretability. Conventional methods combining Koopman theory and deep learning often neglect spatial correlations in spatiotemporal data. This study used the hourly PM2.5 dataset of the Beijing-Tianjin-Hebei region to reveal its spatiotemporal hierarchy using Koopman mode decomposition to identify the key dynamic modes. Furthermore, a Spatial Physics Constrained Learning (SPCL) model utilizing a graph representation learning method was proposed to combine the graph topological information of the PM2.5 spatial features with the Koopman feature function. The results showed that PM2.5 has growth, decay, and oscillation modes as well as daily, weekly, monthly, and yearly periods. SPCL achieved mean absolute error, root mean square error (RMSE), correlation r, and index of agreement values of 9.678, 13.922, 0.864, and 0.921, respectively. The average RMSE at 12 h improved by 16.1%, 12.7%, 0.9%, and 3.5% compared with using Long short-term Memory, Graph Convolutional Networks and Long Short-Term Memory Networks, Spatio-Temporal Graph Convolutional Networks, and Dynamic Spatiotemporal Graph Convolution Network, respectively. By discretizing the neural network hidden layers, the explanatory key of PM2.5 modes was elucidated, which demonstrated enhanced stability.

Spatio-Temporal Graph Neural Networks for Predictive Learning in Urban Computing: A Survey

Spatio-Temporal Graph Neural Networks for Predictive Learning in Urban Computing: A Survey

Multimodal Spatio-Temporal Framework for Real-World Affect Recognition

Deep learning models show great potential in applications involving video-based affect recognition, including human-computer interaction, robotic interfaces, stress and depression assessment, and Alzheimer's disease detection. The low complex Multimodal Diverse Spatio-Temporal Network (MDSTN) has been analysed to effectively capture spatio-temporal information from audio-visual modalities for affect recognition using the Acted Facial Expressions in the Wild (AFEW) dataset. The scarcity of data is handled by data augmented parallel feature extraction for visual network. Visual features extracted by carefully reviewing and customizing Convolutional 3D architecture over different ranges are combined to train a neural network for classification. Multi-resolution Cochleagram (MRCG) features from speech, along with spectral and prosodic audio features, are processed by a supervised classifier. The late fusion technique is explored to integrate audio and video modalities, considering their processing over different temporal spans. The MDSTN approach significantly boosts the accuracy of basic emotion recognition to 71.54% on the AFEW dataset. It demonstrates exceptional proficiency in identifying emotions such as disgust and surprise, thus exceeding current benchmarks in real-world affect recognition.

Disentangled traffic forecasting via efficient graph neural network

Abstract Traffic prediction is important to intelligent transportation systems, but it poses significant challenges due to the complex spatio-temporal correlation of traffic data. To address these challenges, we decompose complex traffic time series into events and trends using discrete wavelet transform. However, the high-frequency information decomposed by wavelet transform contains noise, so we designed a module to denoise the high-frequency events. We model the spatio-temporal correlations of trends and events separately using two mutually independent spatio-temporal networks. We use causal dilation convolution with local receptive fields and attention with global receptive fields to capture the correlation of fluctuating time series and the correlation of stable time series, respectively. We capture spatial correlation using a GNN optimized by an attention mechanism. Finally, we merge useful information from volatile events into predictable trends using adaptive event fusion. Our model significantly outperforms nine existing traffic prediction methods across four real datasets.

Adaptive and Interactive Multi-Level Spatio-Temporal Network for Traffic Forecasting

Adaptive and Interactive Multi-Level Spatio-Temporal Network for Traffic Forecasting

Transfer Learning for Cross-City Traffic Prediction to Solve Data Scarcity

Deep learning models have demonstrated significant achievements in traffic prediction. However, their predictive performance substantially declines when faced with the scarcity of urban traffic data. Addressing the challenges of data scarcity and heterogeneity between cities, cross-city transfer learning has emerged as a promising solution. This paper proposes a domain adaptation cross-city model, which integrates traffic data with auxiliary urban data for domain adaptation in cross-city transfer learning. Specifically, we designed a domain fusion module to measure the differences between cities. Firstly, the knowledge extractor within the domain fusion module learns the knowledge from urban auxiliary data, such as road networks and points of interest, and calculates transferable knowledge. Then, dynamic time warping is used to measure the similarity of traffic time series. By combining these two aspects, we derive the domain differences between cities. Finally, the spatiotemporal network undergoes pre-learning using abundant data from the source city. According to the differences between cities, the model is fine-tuned to improve the adaptability of the model to the target domain. Experimental results on real-world data validate the effectiveness of the proposed model.

Deep learning algorithms for predicting routes of snowplows in European cities

Most cities are faced with annual snowfall but are finding it difficult to deal with their snow plowing activities. Even though winter road maintenance has been studied for many decades, most of the papers do not present models that can be scaled up to allow for incorporation of side constraints that are often met in practical applications. Subject of the paper: Comparison of deep learning algorithms for forecasting snowplow routes in European cities in terms of spatial analysis, routing and traffic flow. The study focuses on extending neural networks including Recurrent Neural Network (RNN), Convolutional Neural Network (CNN) and Graph Neural Network (GNN) to overcome the problematics of snowplow in urban setting. Temporal and spatial data are incorporated using sophisticated models such as Spatio-Temporal Graph Convolutional Networks (STGCNs) using examples that capture the capacity to control the changes in the snowplow routes in reaction to the real-time traffic flow and weather information. The report also looks at how transformer models among other emerging technologies could be harnessed to improve predictive accuracy as well as efficiency. Comparisons made to these other methods evaluating deep learning methods’ benefits and drawbacks and their application to urban infrastructure and traffic flow. The results, therefore, point out that integrating these complicated formulas can go a long way in enhancing the efficiency and safety of snowplowing in the European cities to create better lit and safer transport networks

Bridging spatiotemporal feature gap for video salient object detection

The mutual transfer of spatiotemporal features is the main challenge for the two-stream video salient object detection. Current methods adopt the spatiotemporal feature interaction to achieve it. However, these methods still have two issues: modal feature gap and layer feature gap. To address these, we propose a Bridging Spatiotemporal feature Gap Network (BSGNet) with a global correspondence interaction and gate filtering (GCGF) module, a global-local distribution consistency (GLDC) module, and a modality-layer feature fusion framework (MLFF). Compared with previous works, BSGNet not only explores more effective interaction by GCGF, but also bridges modality and layer feature gaps by GLDC and MLFF. Firstly, GCGF achieves the spatiotemporal feature interaction by modeling intra-modal and inter-modal global correspondences. Besides, GCGF employs a gate mechanism to control the proportion of message transfer between appearance and motion information, which characterizes the contribution provided by spatiotemporal features. Secondly, at both global and local levels, GLDC pushes the spatiotemporal feature distribution between same scenes, and pulls the spatiotemporal feature distribution between different scenes. This can enhance the distribution consistency to align spatiotemporal features and bridge modal feature gap. Finally, MLFF designs an inter-modal and inter-layer feature fusion framework to bridge the layer feature gap brought by the different modalities and different receptive fields. Extensive experiments on five benchmarks reveal that our BSGNet outperforms state-of-the-arts.

Spatio-temporal fusion with motion masks for the moving small target detection from remote-sensing videos

Moving small target detection is an eye-catching research topic, full of great challenges in object detection field. Its primary difficulties exist in (i) the very small size proportion of the target region in background images, (ii) the quite weak contrast to background and (iii) the perception of slight target motion. Currently-existing detection methods mainly utilize the image features from spatial domain. This kind of features is quite weak to small targets. More new feature kinds need to be concerned. In view of these, besides traditional image clues, motion features are currently attracting more and more attention in small target detection. In order to promote detection performance, this paper proposes a Spatio-Temporal Fusion Network (STFNet) with two parallel feature extraction branches for detecting moving small targets. In this network, one branch is designed to capture the traditional semantic features from spatial domain, and the other is devised to perceive the slight target motion features hidden in temporal domain. Meanwhile, to optimize the extraction of spatio-temporal features, a group of motion masks is specially designed to guide feature extractors to pay more precise attention to the positions where small targets appear. Moreover, we design a new bridging module to enhance the cross-domain and cross-scale fusion of spatio-temporal features. Through extensive comparison and ablation experiments on seven sub-datasets, it is proved that our new STFNet is effective and it is obviously superior to the compared ones in detecting moving small targets from satellite sequence images. It achieves a Precision of 0.93, an mAP50 of 71.10% and an F1 of 0.84, evidently higher than existing state-of-the-art methods. Our codes are available at https://github.com/UESTC-nnLab/STF.

Multi-area short-term load forecasting based on spatiotemporal graph neural network

Short term power load forecasting can accurately evaluate the overall power load changes and provide accurate reference for power system operation decision-making. To address the limitations of traditional load forecasting methods, which are unable to capture spatial correlations and simultaneously predict load changes in multiple areas, this paper proposes a load forecasting model based on the spatiotemporal attention convolutional mechanism. The proposed model is composed of two key components: a spatiotemporal attention module and a spatiotemporal convolution module. First, the dynamic spatiotemporal correlations between different electricity load areas are captured and analyzed by using the spatiotemporal attention mechanism. Secondly, the spatial pattern and temporal features of the load data sequence are effectively obtained by spatiotemporal convolutional layers. Finally, this paper verifies the accuracy and effectiveness of the proposed method through a real electricity consumption load dataset. The experimental results demonstrate that, in comparison to various benchmark prediction methods, the proposed method can fully explore and utilize the spatiotemporal correlation between different electricity load areas, thereby improving the accuracy of load prediction.

Depression detection based on the temporal-spatial-frequency feature fusion of EEG

Depression is a prevalent affective psychiatric disorder projected to be the leading contributor to the world’s disease burden by 2030. Due to its high prevalence and low recognition rate, an objective and effective detection method is urgently needed. Deep learning methods based on electroencephalography (EEG) have shown significant potential in depression detection. However, excessive channels can increase redundancy and computational complexity in EEG, while irrelevant channels may reduce accuracy. Additionally, existing models often overlook the complementarity between the temporal-, spatial-, and frequency-domain features of EEG, limiting their detection capabilities. To address these issues, we propose a method that fuses the temporal, spatial, and frequency domain features of EEG to enhance the detection accuracy while eliminating redundant channels. We introduce an EEG channel selection method based on frequency domain weighting that automatically adjusts the channel weights to select the EEG channels that best capture spatial information across the delta, theta, alpha, beta, and gamma bands, thereby optimizing the extraction of spatial-frequency features. In addition, we designed a multiscale spatiotemporal convolutional attention network to extract the spatiotemporal features of EEG. In this network, the multiscale convolutional attention module enhanced the model’s ability to capture spatial features, whereas the temporal trend-aware self-attention module extracted long-term temporal features by analyzing global correlations across different time points. Experimental results on the MODMA dataset show that our method achieved a 97.24% detection accuracy, surpassing current state-of-the-art models. This study offers a novel approach for constructing depression detection models, providing a foundation for future research and application.

A complex-valued convolutional fusion-type multi-stream spatiotemporal network for automatic modulation classification

Automatic Modulation Classification (AMC) is crucial in non-cooperative communication systems as it facilitates the identification of interference signals with minimal prior knowledge. Although there have been significant advancements in Deep Learning (DL) within the field of AMC, leveraging the inherent relationships between In-phase (I) and Quadrature-phase (Q) components, and enhance recognition accuracy under low signal-to-noise ratio (SNR) conditions remains a challenge. This study introduces a complex-valued convolutional fusion-type multi-stream spatiotemporal network (CC-MSNet) for AMC, which combines spatial and temporal feature extraction modules for modulation recognition. Experimental results demonstrate that the CC-MSNet performs well on three benchmark datasets, RML2016.10a, RML2016.10b, and RML2016.04c, with average recognition accuracy of 62.86%, 65.08%, and 71.12%. It also performs excellently in low SNR environments below 0dB, significantly outperforming other networks.

TEDR: A spatiotemporal attention radar extrapolation network constrained by optical flow and distribution correction

In recent years, deep learning has been widely applied to meteorological radar extrapolation due to the shortcomings of traditional optical flow methods in predicting the genesis and dissipation of radar echoes. However, it still faces challenges in addressing issues of clarity and overall intensity attenuation caused by uncertainty. This study implemented a dual-path spatiotemporal attention network that integrates optical flow techniques by employing intra-frame static attention and inter-frame dynamic attention, which could simulate motion fields and the overall intensity distribution of radar echoes separately. Our approach effectively resolve the issues of systematic intensity attenuation and clarity degradation introduced by deep learning methods. Through the comparisons of key metrics such as MSE, SSIM, CSI20, CSI30, and CSI40, the results demonstrated significant improvements over traditional approaches, particularly in CSI30 and CSI40, where the metrics improved by more than 35 %.

A deep spatiotemporal interaction network for multimodal sentimental analysis and emotion recognition

One of the challenges of sentiment analysis and emotion recognition is how to effectively fuse the multimodal inputs. The transformer-based models have achieved great success in applications of multimodal sentiment analysis and emotion recognition recently. However, the transformer-based model often neglects the coherence of human emotion due to its parallel structure. Additionally, a low-rank bottleneck created by multi- attention-head causes an inadequate fitting ability of models. To tackle these issues, a Deep Spatiotemporal Interaction Network (DSIN) is proposed in this study. It consists of two main components, i.e., a cross-modal transformer with a cross-talking attention module and a hierarchically temporal fusion module, where the cross-modal transformer is used to model the spatial interactions between different modalities and the hierarchically temporal fusion network is utilized to model the temporal coherence of emotion. Therefore, the DSIN can model the spatiotemporal interactions of multimodal inputs by incorporating the time-dependency into the parallel structure of transformer and decrease the redundancy of embedded features by implanting their spatiotemporal interactions into a hybrid memory network in a hierarchical manner. The experimental results on two benchmark datasets indicate that DSIN achieves superior performance compared with the state-of-the-art models, and some useful insights are derived from the results.

Novel spatio-temporal attention causal convolutional neural network for multi-site PM2.5 prediction

Multi-site PM2.5 prediction has emerged as a crucial approach, given that the accuracy of prediction models based solely on data from a single monitoring station may be constrained. However, existing multi-site PM2.5 prediction methods predominantly rely on recurrent networks for extracting temporal dependencies and overlook the domain knowledge related to air quality pollutant dispersion. This study aims to explore whether a superior prediction architecture exists that not only approximates the prediction performance of recurrent networks through feedforward networks but also integrates domain knowledge of PM2.5. Consequently, we propose a novel spatio-temporal attention causal convolutional neural network (Causal-STAN) architecture for predicting PM2.5 concentrations at multiple sites in the Yangtze River Delta region of China. Causal-STAN comprises two components: a multi-site spatio-temporal feature integration module, which identifies temporal local correlation trends and spatial correlations in the spatio-temporal data, and extracts inter-site PM2.5 concentrations from the directional residual block to delineate directional features of PM2.5 concentration dispersion between sites; and a temporal causal attention convolutional network that captures the internal correlation information and long-term dependencies in the time series. Causal-STAN was evaluated using one-year data from 247 sites in mainland China. Compared to six state-of-the-art baseline models, Causal-STAN achieves optimal performance in 6-hour future predictions, surpassing the recurrent network model and reducing the prediction error by 8%–10%.

A novel spatio-temporal attention-based bidirectional LSTM model for moisture content prediction in drying process

Accurate prediction of moisture content is significantly crucial for ensuring process stability and product quality in the cylinder drying process. However, the drying process exhibits complex spatio-temporal characteristics and strong interference, which make accurate prediction challenging for the deep learning approach. To address this issue, this article proposes a new spatio-temporal attention-based bidirectional long-short temporal memory network (STA-BiLSTM) model for accurate moisture content prediction. First, Maximum Relevance Minimum Redundancy (mRMR) is adopted to identify optimal features highly related to moisture content. Secondly, bidirectional long-short temporal memory (Bi-LSTM) network is utilized to extract temporal dependencies from the sequential data. Subsequently, spatio-temporal attention mechanisms are designed to adaptively focus on the most relevant features and timesteps, enhancing the model’s generalization ability. Finally, due to the harsh industrial environment, eXtreme Gradient Boosting (XGBoost) is adapted to improve generalizability and robustness. Extensive experiments on a real industrial dataset of the drying process demonstrate that the proposed STA-BiLSTM approach significantly outperforms alternative approaches for predicting moisture content, validating its effectiveness and superiority.

A multi-task spatio-temporal fully convolutional model incorporating interaction patterns for traffic flow prediction

Previous traffic flow prediction studies have utilized spatio-temporal neural networks combined with the multi-task learning framework to seek complementary information for enhancing prediction performance. However, the existing methods still face two challenges: they fail to capture global interaction patterns between regions and lack consideration for inter-correlations within interaction patterns. To solve these issues, we propose a novel multi-task spatio-temporal fully convolutional model named MSTFCM. First, the model includes the interaction tensor and raster tensor as task inputs, where the interaction tensor extends the raster tensor by incorporating global interaction patterns between regions. Second, a multi-task framework combined spatio-temporal convolutional block was used to learn generalized features and interaction features. A channel spatio-temporal attention is added to adaptively adjust feature weights and capture inter-correlations. To train the MSTFCM, the uncertainty loss was designed as the learnable loss functions, which capture various flow fluctuations, to facilitate multi-task optimization. The proposed model was validated on two real-world traffic datasets collected in Xiamen, China. Experimental results showed that MSTFCM outperformed nine baselines in one-step and multi-step prediction, with slower performance degradation as predicted time intervals and steps increased. We further validated the model’s effectiveness through designed variants and visualization results.

A short-term wind power prediction method via self-adaptive adjacency matrix and spatiotemporal graph neural networks

Graph neural networks (GNN) recently have been successfully applied in wind power prediction by employing geo-information to construct the graphs that serve as the GNN-based prediction model. However, these graphs cannot maintain the latent correlation and attribute of the wind turbines, leading to inferior performance. This research proposes an optimizing wind power prediction model through attention mechanism and spatiotemporal graph neural networks. Initially, the spectral clustering and a self-adjacency matrix to construct the graph nodes and edges. Subsequently, the proposed ASTGNN combined from graph convolutional networks and a gated recurrent unit with an attention mechanism is designed to act as the prediction model. A comprehensive set of experiments has been carried out and the proposed method not only improved prediction accuracy but also enhanced computational efficiency. According to the results obtained, the developed model demonstrated an improvement that is up to 13 % in terms of RMSE and 6.7 % in terms of computation time compared to the latest models.