Spatial Temporal Graph Convolutional Network Research Articles

Multi-relation spatiotemporal graph residual network model with multi-level feature attention: A novel approach for landslide displacement prediction

Accurate prediction of landslide displacement is crucial for effective early warning of landslide disasters. While most existing prediction methods focus on time-series forecasting for individual monitoring points, there is limited research on the spatiotemporal characteristics of landslide deformation. This paper proposes a novel Multi-Relation Spatiotemporal Graph Residual Network with Multi-Level Feature Attention (MFA-MRSTGRN) that effectively improves the prediction performance of landslide displacement through spatiotemporal fusion. This model integrates internal seepage factors as data feature enhancements with external triggering factors, allowing for accurate capture of the complex spatiotemporal characteristics of landslide displacement and the construction of a multi-source heterogeneous dataset. The MFA-MRSTGRN model incorporates dynamic graph theory and four key modules: multi-level feature attention, temporal-residual decomposition, spatial multi-relational graph convolution, and spatiotemporal fusion prediction. This comprehensive approach enables the efficient analyses of multi-source heterogeneous datasets, facilitating adaptive exploration of the evolving multi-relational, multi-dimensional spatiotemporal complexities in landslides. When applying this model to predict the displacement of the Liangshuijing landslide, we demonstrate that the MFA-MRSTGRN model surpasses traditional models, such as random forest (RF), long short-term memory (LSTM), and spatial temporal graph convolutional networks (ST-GCN) models in terms of various evaluation metrics including mean absolute error (MAE = 1.27 mm), root mean square error (RMSE = 1.49 mm), mean absolute percentage error (MAPE = 0.026), and R-squared (R2 = 0.88). Furthermore, feature ablation experiments indicate that incorporating internal seepage factors improves the predictive performance of landslide displacement models. This research provides an advanced and reliable method for landslide displacement prediction.

The Analysis of Regional Ice and Snow Tourist Destinations under Back Propagation Neural Network

This study aims to analyze the evolutionary characteristics and development levels of regional ice and snow tourist destinations by integrating the Back Propagation Neural Network (BPNN) within an Internet of Things (IoT) framework. Data from multiple sources are gathered through web scraping technology from various online platforms and are then subjected to cleaning, standardization, and normalization. A feature recognition model for ice and snow tourism is constructed based on a BPNN combined with a Spatial-Temporal Graph Convolutional Network (ST-GCN) algorithm. Experimental results demonstrate that this model excels in convergence speed and prediction accuracy, achieving a final convergence value of 0.059 and a prediction accuracy of 95.74%, which is at least 4% higher than that of the baseline BPNN algorithm. Additionally, the model yields Recall and F1 scores of 91.57% and 89.31%, respectively. After 98 iterations, the Root Mean Square Error (RMSE) is 6.26, significantly outperforming other model algorithms. These results indicate that the proposed model offers substantial advantages in enhancing the management and service quality of ice and snow tourist destinations, thereby providing valuable technical support and guidance for future intelligent tourism management.

Dynamic multi-scale spatial-temporal graph convolutional network for traffic flow prediction

This paper proposes a dynamic multi-scale spatial-temporal graph convolutional network (DS-STGCN) for traffic flow prediction. The network aims to comprehensively extract global and local dependencies in dynamic spatial-temporal data by inputting traffic network flow data to construct node feature graphs, topology graphs, and time slot feature graphs, capturing the complexity and dynamics of traffic flow. DS-STGCN interprets feature information of the traffic network from both spatial and temporal dimensions through dynamic multi-scale graph convolutional blocks. In the spatial dimension, these blocks use constraints at different levels to balance fine-grained local features and extensive global features, revealing the intrinsic structure of traffic flow data. In the temporal dimension, these blocks jointly learn with temporal convolutional blocks to capture multi-frequency time patterns and handle long sequence data, effectively extracting potential dependencies of time series. Furthermore, DS-STGCN effectively models the changing spatial-temporal relationships in road network flow by constructing dynamically adaptive updated adjacency tensors, generating dynamic graph structures to address the challenge of changing spatial-temporal relationships in the transportation system. Experimental results show that our method significantly outperforms other competing methods on five real traffic datasets (PEMS03, PEMS04, PEMS07, PEMS08 and METR-LA).

SWL-LSE: A Dataset of Health-Related Signs in Spanish Sign Language with an ISLR Baseline Method

Progress in automatic sign language recognition and translation has been hindered by the scarcity of datasets available for the training of machine learning algorithms, a challenge that is even more acute for languages with smaller signing communities, such as Spanish. In this paper, we introduce a dataset of 300 isolated signs in Spanish Sign Language, collected online via a web application with contributions from 124 participants, resulting in a total of 8000 instances. This dataset, which is openly available, includes keypoints extracted using MediaPipe Holistic. The goal of this paper is to describe the construction and characteristics of the dataset and to provide a baseline classification method using a spatial–temporal graph convolutional network (ST-GCN) model, encouraging the scientific community to improve upon it. The experimental section offers a comparative analysis of the method’s performance on the new dataset, as well as on two other well-known datasets. The dataset, code, and web app used for data collection are freely available, and the web app can also be used to test classifier performance on-line in real-time.

Interpretable Spatial–Temporal Graph Convolutional Network for System Log Anomaly Detection

To ensure seamless information flow and operational integrity, computer systems need effectively to manage their system logs, but the expansion in their scale and complexity makes it hard to detect anomalies. Current methodologies exhibit deficiencies, including inefficiencies in handling abnormal sequences, lack of interpretability, and limited consideration of both temporal and spatial information. To improve, this paper develops a semi-supervised graph neural network model termed the Interpretable Spatial–Temporal Graph Convolutional Network (IST-GCN). By integrating temporal and event similarity perspectives, the IST-GCN harnesses directed and undirected graphs to capture the temporal and spatial aspects of system log events. Hence, the IST-GCN offers temporal and spatial interpretability. Further, a lightweight feature regularization technique is developed to enhance interpretability in both time and space domains, and thus facilitates anomaly detection efficiently. Comprehensive testing verifies that the IST-GCN approach surpasses nearly all state-of-the-art methods across five public log anomaly datasets. On average, IST-GCN improves Average Precision (AP) by approximately 3% and ROC AUC (RC) by about 4% compared to the best-performing baseline methods, underscoring its effectiveness and robustness.

Multi-granularity spatial temporal graph convolution network with consecutive attention for human motion prediction

Human motion prediction is attracting increasing attention for its numerous potential applications in fields including autonomous driving, video surveillance and virtual reality. However, accurate motion prediction is challenging due to the complex spatial dependencies, dynamic temporal correlations and high dimension of human pose sequences. Existing graph-based methods rarely consider positional and channel information of the feature map, resulting in lower prediction accuracy. Therefore, we propose a novel multi-granularity spatial temporal graph convolution network with consecutive attention (MSTCA) for human motion prediction. Firstly, a multi-granularity spatial convolution network is introduced to capture spatial joint features through multiple kernel sizes. Then, consecutive attention module is proposed to capture both positional and channel information of the feature map. Next, MSTCA uses multi-granularity temporal convolutional network to extract temporal correlations with multiple receptive fields and predict future poses. Finally, a decoder composed of a 2D convolution layer and several PRelu layers integrates the output of the whole model. Experimental results on the GTA-IM and PROX datasets demonstrate that our method significantly improves the accuracy of human motion prediction in comparison to the existing approaches.

Forecasting road network functionality states during extreme rainfall events to facilitate real-time emergency response planning

Rapid prediction of Road Network Functionality (RNF) during extreme rainfall-induced flooding is crucial for supporting proactive and real-time emergency planning, such as rescue, evacuation planning, and emergency supply distribution. Unlike normal operational conditions, extreme rainfall events introduce complex non-stationary, non-Euclidean characteristics to RNF due to intricate meteorological and hydrological processes, as well as the role of a community's road network in emergency response planning. Conventional physics-based flood simulations and flow-based road network analyses typically lack the computational efficiency required for real-time RNF predictions, hindering timely risk mitigation decisions. This study leverages the accuracy of physics-based simulations and the efficacy of deep-learning technologies to develop a deep learning-based surrogate model for Rain-to-RNF (R2R) predictions. This model couples Long Short-Term Memory (LSTM) networks with Spatial-Temporal Graph Convolutional Networks (ST-GCNs) to uniquely capture the spatiotemporal dynamics of RNF under extreme rainfall events. The predictive accuracy, stability, and versatility of the R2R surrogate model are demonstrated in four flood-prone communities in Zhejiang Province. Its implementation during Typhoon Fitow (2013) over a 30-hour intense rainfall showcases its promising predictive capacity and unparalleled computational efficiency. This research advances disaster management, enhancing the resilience and responsiveness of community infrastructure during extreme weather events.

AdpSTGCN: Adaptive spatial–temporal graph convolutional network for traffic forecasting

Traffic flow forecasting plays a crucial role in applications such as intelligent transportation systems. Despite significant research in this field, the current methods have limitations that hinder the realization of highly accurate predictions. Existing GCN-based approaches typically rely on a definite graph structure derived from a physical topology or learned from node features, which is insufficient for building intricate spatial relationships among nodes. To address this challenge, we propose an adaptive spatial–temporal graph convolutional network for traffic forecasting. Our approach exploits a multi-head attention mechanism to construct multi-view feature graphs. We then introduce an adaptive graph convolution method to dynamically aggregate and propagate information from both the topology graph and multi-view feature graphs, which are capable of capturing complex spatial correlations across diverse proximity ranges. Furthermore, we designed a cascaded structural framework that combines temporal information with node features using gated dilated causal convolution to ensure the integrated modeling of spatial–temporal dynamics in traffic flow. Experiments on real-world datasets demonstrate that our proposed method outperforms the current mainstream methods, achieving better performance in traffic flow forecasting. The code is available at https://github.com/dhxdla/AdpSTGCN.git.

Spatial and temporal attention embedded spatial temporal graph convolutional networks for skeleton based gait recognition with multiple IMUs

Gait recognition is one of the key technologies for exoskeleton robot control, while the current IMU-based gait recognition methods only use inertial data and do not fully consider the interconnections of human spatial structure and human joints. In this regard, a skeleton-based gait recognition approach with inertial measurement units using spatial temporal graph convolutional networks with spatial and temporal attention is proposed. A human forward kinematics solver module was used for constructing different human skeleton models and a temporal attention module was added for capturing the more important time frames in the gait cycle. Moreover, the two-stream structure was used to construct spatial temporal graph convolutional networks with spatial and temporal attention for gait recognition, and an average accuracy of about 99% was obtained in user experiments, which is the best performance compared to other algorithms, provides certain reference for gait recognition and real-time control of exoskeleton robots.

One Size Fits All: A Unified Traffic Predictor for Capturing the Essential Spatial-Temporal Dependency.

Traffic prediction is a keystone for building smart cities in the new era and has found wide applications in traffic scheduling and management, environment policy making, public safety, and so on. Instead of creating a traffic predictor for each city, this article focuses on designing a unified network model that could be directly applied for traffic prediction in any city, by learning the essential spatial-temporal dependencies, i.e., the mutual relationship between traffic and the corresponding fine-grained road network. To achieve this goal, this article proposes a joint knowledge- and data-driven mechanism that novelly divides dependencies into three kinds of correlations, i.e., road segment, intra-intersection, and inter-intersection correlation, which capture the microcosmic, middle, and macroscopic dependencies between traffic and the road network, respectively. Specifically, we first construct traffic datasets that could cover all road segments from real-world trajectory datasets, which makes it possible to model the whole road network as a graph, with the help of fine-grained road topology. Then, we propose meta road segment learner, connection-aware spatial-temporal graph convolutional network (GCN), and multiscale residual networks for capturing the microcosmic, middle, and macroscopic dependencies, respectively. Our experiments on three real-world datasets demonstrate that our proposed method could: 1) achieve better prediction accuracy compared with several approaches and 2) capture the mutual relationship between traffic and the fine-grained road network since our model trained only using data from the source city achieves good performance when it is directly applied for traffic prediction in the target city, without any fine-tuning. The codes will be made publicly available.

Attention Mechanism Based Spatial-Temporal Graph Convolution Network for Traffic Prediction

Attention Mechanism Based Spatial-Temporal Graph Convolution Network for Traffic Prediction

Partitioning Leakage Detection in Water Distribution Systems: A Specialized Deep Learning Framework Enhanced by Spatial–Temporal Graph Convolutional Networks

Partitioning Leakage Detection in Water Distribution Systems: A Specialized Deep Learning Framework Enhanced by Spatial–Temporal Graph Convolutional Networks

Rehabilitation exercise quality assessment through supervised contrastive learning with hard and soft negatives.

Exercise-based rehabilitation programs have proven to be effective in enhancing the quality of life and reducing mortality and rehospitalization rates. AI-driven virtual rehabilitation, which allows patients to independently complete exercises at home, utilizes AI algorithms to analyze exercise data, providing feedback to patients and updating clinicians on their progress. These programs commonly prescribe a variety of exercise types, leading to a distinct challenge in rehabilitation exercise assessment datasets: while abundant in overall training samples, these datasets often have a limited number of samples for each individual exercise type. This disparity hampers the ability of existing approaches to train generalizable models with such a small sample size per exercise type. Addressing this issue, this paper introduces a novel supervised contrastive learning framework with hard and soft negative samples that effectively utilizes the entire dataset to train a single model applicable to all exercise types. This model, with a Spatial-Temporal Graph Convolutional Network (ST-GCN) architecture, demonstrated enhanced generalizability across exercises and a decrease in overall complexity. Through extensive experiments on three publicly available rehabilitation exercise assessment datasets, UI-PRMD, IRDS, and KIMORE, our method has proven to surpass existing methods, setting a new benchmark in rehabilitation exercise quality assessment.

Appearance-posture fusion network for distracted driving behavior recognition

In recent years, detection techniques using computer vision and deep learning have shown promise in assessing driver distraction. This paper proposes a fusion network that combines a Spatial-Temporal Graph Convolutional Network (ST-GCN) and a hybrid convolutional network to integrate multimodal input data for recognizing distracted driver behavior. Specifically, to address the limitations of the ST-GCN method in modeling long-distance joint interaction features and inadequate temporal feature extraction, we design the Spatially Symmetric Configuration Partitioning Graph Convolutional Network (SSCP-GCN) to model relative motion information of symmetric relationships between limbs. Specifically, we utilize densely connected blocks for processing multi-scale temporal information between consecutive frames, thereby enhancing the reuse of bottom features. Furthermore, the expression of important temporal information is augmented by the introduction of the channel attention mechanism. To tackle the problem that the Mixed Convolution (MC) combining 3D convolution with 2D convolution cannot extract higher-order timing information and has limitations in modeling global dependency relationships, we compensate for its inability to capture higher-order temporal semantic information using the Time Shift Module (TSM) without consuming additional computational resources. Additionally, the 3D Multi-Head Self-Attention mechanism (3D MHSA) is employed to integrate global spatial–temporal information of high-level features, avoiding the issue of model complexity proliferation caused by the deep stacking design of Convolutional Neural Networks (CNN). Lastly, we introduce a multistream network framework that integrates driver posture and appearance features to harness complementary advantages, enabling us to combine multimodal input features to achieve better model performance. Experimental results indicate that the accuracy of the network designed in this paper reaches 95.6% and 94.3% on ASU dataset and NTU-RGB+D dataset, respectively. The small size of the model offers the possibility for practical application of the algorithm.

Multiscale information enhanced spatial-temporal graph convolutional network for multivariate traffic flow forecasting via magnifying perceptual scope

Graph Convolutional Networks (GCNs), which can model data in non-Euclidean space, have received extensive attention in multivariate traffic flow forecasting in recent years. However, most of the existing GCN forecasting methods rely on adjacency matrices defined by road network knowledge, which cannot accurately capture the inherent spatial-temporal dependencies of complex multivariate traffic flows. In this paper, a multiscale information enhanced spatial-temporal GCN is proposed, which utilizes an adaptive graph learning layer to automatically extract the dependencies between variables and constructs adaptive adjacency matrices for different datasets. In addition, an information enhancement module is introduced to magnify perceptual scope of the model and strengthen the connection between information, so that the spatial-temporal graph convolution module can better capture the spatial-temporal dependencies in the traffic flow. We conducted experiments on multivariate traffic flow data from three different real-world scenarios to evaluate the effectiveness of the proposed method. Comparative experimental results consistently show that the proposed method outperforms existing GCNs, achieving an average improvement of 21.16% on Mean Absolute Error (MAE), 18.87% on Root Mean Square Error (RMSE), and 8.27% on R2 score (R2) in the three datasets, respectively, and the results demonstrate its excellent forecasting performance in the field of multivariate traffic flow forecasting.

Shale Gas Production Forecasting with Well Interference Based on Spatial-Temporal Graph Convolutional Network

Summary One of the core assumptions of most deep-learning-based data-driven models is that samples are independent. However, this assumption poses a key challenge in production forecasting—performance is influenced by well interference and reservoir connectivity. Most shale gas wells are hydraulically fractured and exist in complex fracture systems, and the neighboring well characteristics should also be considered when constructing data-driven forecast models. Researchers have explored using the graph convolutional network (GCN) to address this issue by incorporating neighboring well characteristics into production forecasting models. However, applying GCN to field-scale studies is problematic, as it requires training on a full batch, leading to gigantic cache allocation. In addition, the transductive nature of GCN poses challenges for direct generalization to unseen nodes. To overcome these limitations, we adopt the graph sampling and aggregation (GraphSAGE) network architecture, which allows training large graphs with batches and generalizing predictions for previously unseen nodes. By utilizing the gated recurrent unit (GRU) network, the proposed spatial-temporal (ST)-GraphSAGE model can capture cross-time relationships between the target and the neighboring wells and generate promising prediction time series for the target wells, even if they are newly drilled wells. The proposed approach is validated and tested using the field data from 2,240 Montney shale gas wells, including formation properties, hydraulic fracture parameters, production history, and operational data. The algorithm aggregates the first-hop information to the target node for each timestep. The encoder-decoder (ED) architecture is used to generate forecasts for the subsequent 3-year production rate by using the 1-year production history of the wells. The trained model enables the evaluation of production predictions for newly developed wells at any location. We evaluate the model’s performance using P10, P50, and P90 of the test data set’s root mean square error (RMSE). Our method preserves the topological characteristics of wells and generalizes the prediction to unseen nodes while significantly reducing training complexity, making it applicable to larger data sets. By incorporating information from adjacent wells and integrating ST data, our ST-GraphSAGE model outperforms the traditional GRU-ED model and shows enhanced interpretability.

Fall Detection Method for Infrared Videos Based on Spatial-Temporal Graph Convolutional Network.

The timely detection of falls and alerting medical aid is critical for health monitoring in elderly individuals living alone. This paper mainly focuses on issues such as poor adaptability, privacy infringement, and low recognition accuracy associated with traditional visual sensor-based fall detection. We propose an infrared video-based fall detection method utilizing spatial-temporal graph convolutional networks (ST-GCNs) to address these challenges. Our method used fine-tuned AlphaPose to extract 2D human skeleton sequences from infrared videos. Subsequently, the skeleton data was represented in Cartesian and polar coordinates and processed through a two-stream ST-GCN to recognize fall behaviors promptly. To enhance the network's recognition capability for fall actions, we improved the adjacency matrix of graph convolutional units and introduced multi-scale temporal graph convolution units. To facilitate practical deployment, we optimized time window and network depth of the ST-GCN, striking a balance between model accuracy and speed. The experimental results on a proprietary infrared human action recognition dataset demonstrated that our proposed algorithm accurately identifies fall behaviors with the highest accuracy of 96%. Moreover, our algorithm performed robustly, identifying falls in both near-infrared and thermal-infrared videos.

Cross- and Context-Aware Attention Based Spatial-Temporal Graph Convolutional Networks for Human Mobility Prediction

The COVID-19 pandemic has dramatically transformed human mobility patterns. Therefore, human mobility prediction for the “new normal” is crucial to infrastructure redesign, emergency management, and urban planning post the pandemic. This paper aims to predict people’s number of visits to various locations in New York City using COVID and mobility data in the past two years. To quantitatively model the impact of COVID cases on human mobility patterns and predict mobility patterns across the pandemic period, this paper develops a model CCAAT-GCN ( C ross- and C ontext- A ttention based Spatial-Temporal G raph C onvolutional N etworks). The proposed model is validated using SafeGraph data in New York City from August 2020 to April 2022. A rich set of baselines are performed to demonstrate the performance of our proposed model. Results demonstrate the superior performance of our proposed method. Also, the attention matrix learned by our model exhibits a strong alignment with the COVID-19 situation and the points of interest within the geographic region. This alignment suggests that the model effectively captures the intricate relationships between COVID-19 case rates and human mobility patterns. The developed model and findings can offer insights into the mobility pattern prediction for future disruptive events and pandemics, so as to assist with emergency preparedness for planners, decision-makers and policymakers.

InfoSTGCAN: An Information-Maximizing Spatial-Temporal Graph Convolutional Attention Network for Heterogeneous Human Trajectory Prediction

Predicting the future trajectories of multiple interacting pedestrians within a scene has increasingly gained importance in various fields, e.g., autonomous driving, human–robot interaction, and so on. The complexity of this problem is heightened due to the social dynamics among different pedestrians and their heterogeneous implicit preferences. In this paper, we present Information Maximizing Spatial-Temporal Graph Convolutional Attention Network (InfoSTGCAN), which takes into account both pedestrian interactions and heterogeneous behavior choice modeling. To effectively capture the complex interactions among pedestrians, we integrate spatial-temporal graph convolution and spatial-temporal graph attention. For grasping the heterogeneity in pedestrians’ behavior choices, our model goes a step further by learning to predict an individual-level latent code for each pedestrian. Each latent code represents a distinct pattern of movement choice. Finally, based on the observed historical trajectory and the learned latent code, the proposed method is trained to cover the ground-truth future trajectory of this pedestrian with a bi-variate Gaussian distribution. We evaluate the proposed method through a comprehensive list of experiments and demonstrate that our method outperforms all baseline methods on the commonly used metrics, Average Displacement Error and Final Displacement Error. Notably, visualizations of the generated trajectories reveal our method’s capacity to handle different scenarios.

Improved fault detection and diagnosis using graph auto encoder and attention-based graph convolution networks

A powerful fault detection and diagnosis (FDD) system plays a pivotal role in achieving operational excellence by maximizing system performance, optimizing maintenance strategies, and ensuring the longevity and resilience of process plants. In the context of FDD for multivariate sensor data, this study presents an improved FDD approach using graph-based neural networks. This graph neural network uses an adjacency matrix developed by extracting the expert domain knowledge and topological information of the multi-sensor system. This additional graph representation of the system is incorporated along with multivariate sensor data to capture the spatial and temporal information in neural networks efficiently. In this regard, we propose and evaluate: 1) A Graph Auto Encoder (GAE) based fault detection strategy and 2) An Attention-based Spatial Temporal Graph Convolution Network (ASTGCN) based fault diagnosis methodology. By leveraging the additional knowledge in the form of graphs, the GAE captures the complex relationships and dependencies among sensors, enabling effective anomaly detection, which identifies abnormal patterns and deviations from normal behavior, thus indicating potential faults in the system. The ASTGCN incorporates attention mechanisms to selectively focus on relevant sensor nodes and capture their spatial and temporal dependencies for fault diagnosis. The effectiveness of the proposed FDD approach is demonstrated using the benchmark Tennessee Eastman Process (TEP) problem. The results show that the proposed approaches outperform traditional methods and highlight the importance of leveraging graph-based knowledge for FDD in complex systems.