Spatio-temporal Graph Research Articles

Learning to Generate Temporal Origin-destination Flow Based-on Urban Regional Features and Traffic Information

Origin-destination (OD) flow contains population mobility information between every two regions in the city, which is of great value in urban planning and transportation management. Nevertheless, the collection of OD flow data is extremely difficult due to the hindrance of privacy issues and collection costs. Significant efforts have been made to generate OD flow based on urban regional features, e.g., demographics, land use, and so on, since spatial heterogeneity of urban function is the primary cause that drives people to move from one place to another. On the other hand, people travel through various routes between OD, which will have effects on urban traffic, e.g., road travel speed and time. These effects of OD flows reveal the fine-grained spatiotemporal patterns of population mobility. Few works have explored the effectiveness of incorporating urban traffic information into OD generation. To bridge this gap, we propose to generate real-world daily temporal OD flows enhanced by urban traffic information in this paper. Our model consists of two modules: Urban2OD and OD2Traffic . In the Urban2OD module, we devise a spatiotemporal graph neural network to model the complex dependencies between daily temporal OD flows and regional features. In the OD2Traffic module, we introduce an attention-based neural network to predict urban traffic based on OD flow from the Urban2OD module. Then, by utilizing gradient backpropagation, these two modules are able to enhance each other to generate high-quality OD flow data. Extensive experiments conducted on real-world datasets demonstrate the superiority of our proposed model over the state of the art.

A large-scale graph clustering method for cell conditions spatio-temporal localization in aluminum electrolysis

Sensor nodes (industrial nodes) clustering is an efficacious method for gaining local working conditions real-timely in the production process. The existing deep graph neural networks-based industrial nodes clustering studies construct static graphs to describe the potential spatial correlation among industrial nodes, but the temporal correlation among industrial nodes is ignored, resulting in inferior real-time performance. Meanwhile, these deep learning models fail to obtain good clustering results due to the large-scale and noise-containing characteristics of industrial nodes. In this paper, a large-scale graph clustering method (LsGCM) is proposed for cell conditions spatio-temporal localization in aluminum electrolysis to address the above issues. Specifically, a spatio-temporal graph construction method based on industrial process mechanism knowledge and local working condition boundary types statistical feature fusion is proposed for capturing effective spatio-temporal correlations between industrial nodes. Then, hysteresis knowledge of industrial production system is used to guide the nodes merging, thereby avoiding the incorrect merging of time-asynchronous nodes when performing the spatio-temporal region detection of local working conditions. Finally, combined with the actual spatio-temporal location information of industrial nodes, a novel spatio-temporal information expansion method (StIEM) of the sub-spatio-temporal graph describing local working condition is proposed for obtaining spatio-temporal feature information of the neighborhood, which is beneficial to improve the performance of industrial node clustering. Validation studies on a numerical simulation example and multiple real aluminum electrolysis industrial datasets certificate the effectiveness and superiority of the proposed method. In particular, compared with existing graph clustering methods, our method improves the clustering accuracy by 7.79%.

CSGAT-Net: a conditional pedestrian trajectory prediction network based on scene semantic maps and spatiotemporal graph attention

CSGAT-Net: a conditional pedestrian trajectory prediction network based on scene semantic maps and spatiotemporal graph attention

A heterogeneous traffic spatio-temporal graph convolution model for traffic prediction

Smart cities require advanced traffic management systems. Traffic forecasting is an essential task of the advanced transportation system. Traffic spatio-temporal data are often heterogeneous. Most existing traffic prediction models predominantly use separate components to extract the temporal and spatial features of traffic data. However, this overlooks the intrinsic connections between the spatio-temporal features of traffic data. To directly mine the spatio-temporal heterogeneity, this study constructs a global heterogeneous traffic spatio-temporal graph and proposes the Heterogeneous Traffic Spatio-Temporal Graph Convolution (HTSTGC). To reduce the complexity of the model, Simple Graph Convolution (SGC) is used to extract semi-structured meta-graph information. The receptive fields that capture temporal and spatial features can be flexibly adjusted separately through clever design, which can balance the performance and efficiency of the model. Finally, the feature fusion module applies Gated Graph Neural Network (GGNN) to fuse temporal and spatial features. The results on the PEMS datasets reveal that jointly modeling different types of relationships can improve the traffic prediction performance of the model. The proposed HTSTGC has better performance than the baseline methods in most cases. The research results can support urban traffic control, traffic pollution reduction, and sustainable urban development.

Intelligent Analysis of E-commerce Reviews based on Complex-Valued Spatio-temporal Graph Convolutional Neural Network Optimized Tyrannosaurus Optimization Algorithm

Intelligent analysis of E-commerce reviews based on multimodal social media big data using applies the benefit of innovative artificial intelligence method, especially deep learning algorithms, to study and realize web-based product consider and social network data connected to e-commerce. In this manuscript, Intelligent Analysis of E-commerce reviews based on Complex-Valued Spatio-temporal Graph Convolutional Neural Network Optimized Tyrannosaurus Optimization Algorithm (IAER-CVSGCNN-TOA)is proposed for Intelligent Analysis of E-commerce Reviews. The input data are collected from E-commerce big Dataset. Then data are pre-processing utilizing federated neural collaborative filtering (FNCF) to remove the noise and clean the data. The pre-processed data is provided to the CVSGCNN is used for intelligent analysis of e-commerce reviews. In general, CVSGCNN does no express adapting optimization approaches to determine optimal parameters to ensure accurate prediction. Hence, proposed to utilize the Tyrannosaurus Optimization Algorithm enhancement CVSGCNN for intelligent analysis of e-commerce reviews. The proposed IAER-CVSGCNN-TOA method is implemented on python. Then performance of proposed technique is analyzed with other existing techniques. The proposed technique attains 30.26%, 28.70% and 46.25% higher accuracy, 20.53%, 32.60%, and 53.43% higher precision, 25.23%, 23.15%, and 22.33%higher recall, 25.75%, 44.30% and 53.70%higher specificity comparing with the existing methods such as a Retracted: Online Troll Reviewer Detection Utilizing Deep Learning Methods (IAER-CNN), Developing an Intelligent System and DL Algorithms for Sentiment Analysis of E-Commerce Product Reviews(IAER-LSTM), Intelligent Perception System of Big Data Decision in Cross-Border e-Commerce Depend on Data Fusion(IAER-DFN) respectively.

Artistic Creation and Animation Scene Designing using Hierarchical Spatio-Temporal Graph Convolutional Neural Network Optimized with Gazelle Optimization Algorithm

Artistic Creation and Animation Scene Designing using Hierarchical Spatio-Temporal Graph Convolutional Neural Network Optimized with Gazelle Optimization Algorithm

Enhanced HAR using Dynamic STGAT

Action recognition has seen significant advancements with the integration of spatio-temporal representations, particularly leveraging skeleton-based models and cross-modal data fusion techniques. However, existing approaches face challenges in capturing long- range dependencies within the human body skeleton and effectively balancing features from diverse modalities. To address these limitations, a novel framework, the Dynamic Spatio-Temporal Graph Attention Transformer (D-STGAT), is proposed, which seamlessly integrates the strengths of dynamic graph attention mechanisms and transformer architectures for enhanced action recognition. The framework builds upon recent innovations in graph attention networks (GAT) and transformer models. First, the Spatial-Temporal Dynamic Graph Attention Network (ST-DGAT) is introduced, extending traditional GAT by incorporating a dynamic attention mechanism to capture spatial- temporal patterns within skeleton sequences. By reordering the weighted vector operations in GAT, the approach achieves a global approximate attention function, significantly enhancing its expressivity and capturing long-distance dependencies more effectively than static attention mechanisms. Furthermore, to address the challenges of cross-modal feature representation and fusion, the spatio-temporal Cross Attention Transformer (ST-CAT) is introduced. This model efficiently integrates spatio-temporal information from both video frames and skeleton sequences by employing a combination of full spatio-temporal attention (FAttn), zigzag spatio-temporal attention (ZAttn), and binary spatio-temporal attention (BAttn) modules. Through the proper arrangement of these modules within the transformer encoder and decoder, ST-CAT learns a multi-feature representation that effectively captures the intricate spatiotemporal dynamics inherent in action recognition tasks. Experimental results on the Penn- Action, NTU-RGB+D 60, and 120 datasets showcase the efficacy of the approach, yielding promising performance improvements over previous state-of-the-art methods. In summary, the proposed D-STGAT and ST-CAT frameworks offer novel solutions for action recognition tasks by leveraging dynamic graph attention mechanisms and transformer architectures to effectively capture and fuse spatiotemporal features from diverse modalities, leading to superior performance compared to existing approaches.

Passenger spatiotemporal distribution prediction in airport terminals based on physics-guided spatio-temporal graph convolutional network and its effect on indoor environment prediction

The airport as an important transportation hub plays a leading role in promoting sustainable cities and new-type urbanization. To boost safe, environmental-friendly and technologically advanced airports, the passenger travel behavior as a core that decides the resource allocation, system tuning and capacity dispatching, must be grasped. Previous research in passenger distribution prediction focused on physics-based methods or only mining temporal dynamics. In this work, a refined passenger distribution prediction was modeled based on a learning-based method embedding physical prior knowledge, and then its effects on indoor environment prediction were analyzed. Among them, based on insect intelligent building architecture, a virtual spatial graph was defined in Guangzhou Baiyun International Airport Terminal 2, then a Wi-Fi positioning system was constructed; Next, a physics-guided spatio-temporal graph convolutional network, considering both the spatial dependencies and the passenger arrival pattern extracted from cost-free flight schedules, was developed for domestic and international passenger distribution predictions with R2 over 0.87 and 0.76 respectively; Lastly, the contributions of predicted occupant densities to the indoor environment prediction were evaluated with results showing that the average R2 for indoor temperature, relative humidity and CO2 concentration prediction was enhanced by 0.4 %∼91.5 %, 0.2 %∼29.7 % and 0.4 %∼45.4 % respectively as the prediction horizon broadening.

A spatio-temporal graph neural network for fall prediction with inertial sensors

Falls are the leading cause of unintentional human injury, having become a public health event of strong social concern. The fall prediction technology based on wearable inertial sensors is a relatively reliable solution in human activity monitoring, a user scenario with mobility and high information privacy sensitivity, and has the advantages of low cost, small size, and high precision. However, a key challenge in sensor-based fall prediction is the fact that a fall event can often occur in various configurations of fall poses together with their own spatio-temporal dependencies. This leads us to define a spatio-temporal model to explicitly characterize these internal configurations of poses. In particular, we introduce a graph neural network with spatio-temporal topological structure to encode such latent relations among poses by capturing representative patterns in fall events. Moreover, a human body orientation estimator is devised to represent human low limbs information and as a result, separate pose dependencies are globally consistent. Empirical evaluations on two benchmark datasets and one in-house dataset suggest our approach significantly outperforms the state-of-the-art methods.

Graph-Based Prediction and Planning Policy Network (GP3Net) for Scalable Self-Driving in Dynamic Environments Using Deep Reinforcement Learning

Recent advancements in motion planning for Autonomous Vehicles (AVs) show great promise in using expert driver behaviors in non-stationary driving environments. However, learning only through expert drivers needs more generalizability to recover from domain shifts and near-failure scenarios due to the dynamic behavior of traffic participants and weather conditions. A deep Graph-based Prediction and Planning Policy Network (GP3Net) framework is proposed for non-stationary environments that encodes the interactions between traffic participants with contextual information and provides a decision for safe maneuver for AV. A spatio-temporal graph models the interactions between traffic participants for predicting the future trajectories of those participants. The predicted trajectories are utilized to generate a future occupancy map around the AV with uncertainties embedded to anticipate the evolving non-stationary driving environments. Then the contextual information and future occupancy maps are input to the policy network of the GP3Net framework and trained using Proximal Policy Optimization (PPO) algorithm. The proposed GP3Net performance is evaluated on standard CARLA benchmarking scenarios with domain shifts of traffic patterns (urban, highway, and mixed). The results show that the GP3Net outperforms previous state-of-the-art imitation learning-based planning models for different towns. Further, in unseen new weather conditions, GP3Net completes the desired route with fewer traffic infractions. Finally, the results emphasize the advantage of including the prediction module to enhance safety measures in non-stationary environments.

Structure-Aware Multimodal Sequential Learning for Visual Dialog

With the ability to collect vast amounts of image and natural language data from the web, there has been a remarkable advancement in Large-scale Language Models (LLMs). This progress has led to the emergence of chatbots and dialogue systems capable of fluent conversations with humans. As the variety of devices enabling interactions between humans and agents expands, and the performance of text-based dialogue systems improves, there has been recently proposed research on visual dialog. However, visual dialog requires understanding sequences of pairs consisting of images and sentences, making it challenging to gather sufficient data for training large-scale models from the web. In this paper, we propose a new multimodal learning method leveraging existing large-scale models designed for each modality, to enable model training for visual dialog with small visual dialog datasets. The key ideas of our approach are: 1) storing the history or context during the progression of visual dialog in the form of spatiotemporal graphs, and 2) introducing small modulation blocks between modality-specific models and the graphs to align the semantic spaces. For implementation, we introduce a novel structure-aware cross-attention method, which retrieves relevant image and text knowledge for utterance generation from the pretrained models. For experiments, we achieved a new state-of-the-art performance on three visual dialog datasets, including the most challenging one COMET.

Advancing storm surge forecasting from scarce observation data: A causal-inference based Spatio-Temporal Graph Neural Network approach

Rapid and precise forecasting of storm surge in coastal regions is crucial for ensuring safety of coastal communities’ life and property. Yet, learning a data-driven forecasting model from observation data such as gauges and post-event reconnaissance remains challenging, due to the observation data scarcity and the real-world complexity. Recently, deep learning has received increasing attention, but existing deep learning approaches solely focus on individual site scenarios, ignoring the value of information contained in neighboring sites’ observations. In this study, we propose to integrate graph neural networks (GNN) and gated recurrent unit (GRU) to capture the spatial and temporal storm surge dependencies across multiple observation stations. GNN provides the unique capability to model non-Euclidean complex spatial relationship across observation stations, while GRU enhances the data efficiency of temporal dependency modeling. To account for the effect of complex coastline topography, the Liang–Kleeman information flow theory is employed to establish a causal-inference based graph edge scheme connecting multiple observation stations. The Causal-inference based Spatio-Temporal Graph Neural Network (CSTGNN) were trained and evaluated on 13-year observation data from 4 observation stations along Florida coastline. Experiments affirm the competence of CSTGNN, which outperformed six commonly used competitive baselines across different metrics and observation stations, under lead times up to six hours. Furthermore, benefits of capturing the spatial dependency and leveraging causal inference are also comprehensively examined. To conclude, we believe that this novel spatio-temporal forecasting framework can result in enhanced coastal resilience by its improved storm surge forecasting capability.

Misbehavior detection with spatio-temporal graph neural networks

Graph Neural Networks (GNNs) gained the attention of researchers following advancements in Representational Learning. Unlike classical machine learning (ML) methods, its ability to represent more concepts on its data structure gives GNN a clear head start. On the other hand, Misbehavior Detection (MBD) has become a security solution for authorized vehicles to protect against malicious activities on Vehicles to Everything (V2X) Networks. Although authorities standardize MBD, there are not enough MBD datasets for ML, and existing ones are unsuitable for GNNs. In this study, we address this issue by providing an algorithm to convert classical MBD datasets for GNN. Then, a novel GNN model is proposed to detect misbehaving activity on V2X networks. Obtained results show that the proposed GNN model outperforms existing methods with 99.92% accuracy, 99.9196% recall, 31% better runtime efficiency, and 97.35% Matthews correlation coefficient.

Simultaneous forecasting of wind speed for multiple stations based on attribute-augmented spatiotemporal graph convolutional network and tree-structured parzen estimator

The global increase in energy demand and environmental concerns have made the development of wind energy increasingly important. Accurate wind speed prediction is crucial for maximizing the benefits of wind energy. In order to further improve the accuracy of multi-site wind speed prediction, this study adopts an evolutionary algorithm-based deep learning model that fully considers the spatiotemporal relationships among multiple station's wind speed data. Firstly, the mutual information (MI) method is used to select variables with stronger correlations to wind speed as auxiliary input factors. Then, an improved version of Attribute-Augmented Spatiotemporal Graph Convolutional Network (IASTGCN) is employed to process data from multiple stations, taking into account both temporal and spatial factors. Additionally, an MI-based wind speed data relationship matrix between multiple stations is calculated to replace the original distance relationship matrix in the model, enabling the model to better capture and utilize the relationships between stations. Next, the Tree-structured Parzen Estimator (TPE) is used to optimize the hyperparameters of the model. This ultimately achieves multi-site multi-step wind speed prediction. Experimental results demonstrate that the proposed model outperforms baseline models and models that only consider either temporal or spatial factors in various scenarios, exhibiting better predictive performance.

LSTTN: A Long-Short Term Transformer-based spatiotemporal neural network for traffic flow forecasting

Accurate traffic forecasting is a fundamental problem in intelligent transportation systems and learning long-range traffic representations with key information through spatiotemporal graph neural networks (STGNNs) is a basic assumption of current traffic flow prediction models. However, due to structural limitations, existing STGNNs can only utilize short-range traffic flow data; therefore, the models cannot adequately learn the complex trends and periodic features in traffic flow. Besides, it is challenging to extract the key temporal information from the long historical traffic series and obtain a compact representation. To solve the above problems, we propose a novel LSTTN (Long-Short Term Transformer-based Network) framework comprehensively considering the long- and short-term features in historical traffic flow. First, we employ a masked subseries Transformer to infer the content of masked subseries from a small portion of unmasked subseries and their temporal context in a pretraining manner, forcing the model to efficiently learn compressed and contextual subseries temporal representations from long historical series. Then, based on the learned representations, long-term trend is extracted by using stacked 1D dilated convolution layers, and periodic features are extracted by dynamic graph convolution layers. For the difficulties in making time-step level prediction, LSTTN adopts a short-term trend extractor to learn fine-grained short-term temporal features. Finally, LSTTN fuses the long-term trend, periodic features and short-term features to obtain the prediction results. Experiments on four real-world datasets show that in 60-minute-ahead long-term forecasting, the LSTTN model achieves a minimum improvement of 5.63% and a maximum improvement of 16.78% over baseline models. The source code is availble at https://github.com/GeoX-Lab/LSTTN.

A novel spatiotemporal graph convolutional network framework for functional connectivity biomarkers identification of Alzheimer’s disease

BackgroundFunctional connectivity (FC) biomarkers play a crucial role in the early diagnosis and mechanistic study of Alzheimer’s disease (AD). However, the identification of effective FC biomarkers remains challenging. In this study, we introduce a novel approach, the spatiotemporal graph convolutional network (ST-GCN) combined with the gradient-based class activation mapping (Grad-CAM) model (STGC-GCAM), to effectively identify FC biomarkers for AD.MethodsThis multi-center cross-racial retrospective study involved 2,272 participants, including 1,105 cognitively normal (CN) subjects, 790 mild cognitive impairment (MCI) individuals, and 377 AD patients. All participants underwent functional magnetic resonance imaging (fMRI) and T1-weighted MRI scans. In this study, firstly, we optimized the STGC-GCAM model to enhance classification accuracy. Secondly, we identified novel AD-associated biomarkers using the optimized model. Thirdly, we validated the imaging biomarkers using Kaplan–Meier analysis. Lastly, we performed correlation analysis and causal mediation analysis to confirm the physiological significance of the identified biomarkers.ResultsThe STGC-GCAM model demonstrated great classification performance (The average area under the curve (AUC) values for different categories were: CN vs MCI = 0.98, CN vs AD = 0.95, MCI vs AD = 0.96, stable MCI vs progressive MCI = 0.79). Notably, the model identified specific brain regions, including the sensorimotor network (SMN), visual network (VN), and default mode network (DMN), as key differentiators between patients and CN individuals. These brain regions exhibited significant associations with the severity of cognitive impairment (p < 0.05). Moreover, the topological features of important brain regions demonstrated excellent predictive capability for the conversion from MCI to AD (Hazard ratio = 3.885, p < 0.001). Additionally, our findings revealed that the topological features of these brain regions mediated the impact of amyloid beta (Aβ) deposition (bootstrapped average causal mediation effect: β = -0.01 [-0.025, 0.00], p < 0.001) and brain glucose metabolism (bootstrapped average causal mediation effect: β = -0.02 [-0.04, -0.001], p < 0.001) on cognitive status.ConclusionsThis study presents the STGC-GCAM framework, which identifies FC biomarkers using a large multi-site fMRI dataset.

A fish appetite assessment method based on improved ByteTrack and spatiotemporal graph convolutional network

The appetite of fish significantly influences aquaculture efficiency and fish welfare. However, accurately assessing fish appetite has posed a challenging problem. Currently, the study of fish feeding behaviour relies primarily on the overall information obtained from images of fish schools, often overlooking the distinctive behavioural traits of individual fish. Analysing the behaviour of individual fish is hindered by challenges such as intraclass variation and cross-occlusion within real aquaculture environments. To address these challenges, this paper introduces a novel method for assessing appetite based on individual fish behaviour. The ByteTrack model was improved to enable stable tracking of each fish within a school under complex conditions. Additionally, this paper employs the spatiotemporal graph convolutional neural network (ST-GCN) to extract the movement characteristics of individual fish, facilitating accurate appetite assessment. The experimental results demonstrate that the proposed method achieves 98.47% accuracy in appetite assessment, surpassing the performance of other state-of-the-art methods. This paper provides a new opportunity and effective means for analysing fish behaviour and appetite in intricate environments.

Human-Robot handover task intention recognition framework by fusing human digital twin and deep domain adaptation

Industry 5.0 is conceived to achieve the expected high level of smart manufacturing and the more proactive and adaptive human-robot collaboration (HRC). In particular, human-robot handover tasks (HRHTs) are a notably crucial aspect of HRC, and resolving how to motivate robots to comprehend human handover intentions constitutes an exigent issue that demands resolution. To address these problems, A framework that integrates hierarchical human digital twin (HHDT) and deep domain adaptation is proposed, where HHDT model is built to digitise the human physical and physiological state and enhance the robot's perception of humans, a feature extractor-based on bidirectional long-short-term memory and spatio-temporal graph convolution network (BiLSTM-ST-GCN) is devised to extract latent spatio-temporal features of HRHTs intentions. Additionally, the framework incorporates a deep domain adaptation layer (DDAL) that facilitates knowledge transfer of HRHTs intentions by accounting for the distribution differences between the source and target operator objects. Extensive experiments demonstrate the superior performance of the proposed framework for HRHTs intention recognition and extraction of domain invariant features.

STFormer: A dual-stage transformer model utilizing spatio-temporal graph embedding for multivariate time series forecasting

Multivariate Time Series (MTS) forecasting has gained significant importance in diverse domains. Although Recurrent Neural Network (RNN)-based approaches have made notable advancements in MTS forecasting, they do not effectively tackle the challenges posed by noise and unordered data. Drawing inspiration from advancing the Transformer model, we introduce a transformer-based method called STFormer to address this predicament. The STFormer utilizes a two-stage Transformer to capture spatio-temporal relationships and tackle the issue of noise. Furthermore, the MTS incorporates adaptive spatio-temporal graph structures to tackle the issue of unordered data specifically. The Transformer incorporates graph embedding to combine spatial position information with long-term temporal connections. Experimental results based on typical finance and environment datasets demonstrate that STFormer surpasses alternative baseline forecasting models and achieves state-of-the-art results for single-step horizon and multistep horizon forecasting.

A novel dynamic spatio-temporal graph convolutional network for wind speed interval prediction

It is crucial to predict the wind speed for the utilization of renewable wind energy and the operation of transmission lines with increased capacity. The intermittency and stochastic fluctuations of wind speed pose a significant challenge for the high-quality wind speed prediction, and a novel wind speed interval prediction (WSIP) model is constructed in this study by employing the residual estimation (RE)-oriented dynamic spatio-temporal graph convolutional network (DSTGCN) approach. Firstly, a dynamic adjacency matrix is designed to obtain time-varying global spatial weight allocations among each wind speed node. Then, the spatio-temporal features are extracted by using gated recurrent units (GRUs) and GCNs to construct the wind speed graph networks. Moreover, the RE-oriented strategy incorporating the pinball loss is designed to provide a guidance the parameter training of the constructed model, thus eliminating the quantile crossings problem. As a result, the deterministic point prediction of the wind speed is expanded to the quantile-based probabilistic interval prediction. Finally, the experimental results are presented to demonstrate the validity and superiority of proposed scheme in both qualitative and quantitative performance.