Spatial Temporal Graph Convolutional Network Research Articles

Dual-module spatial temporal information enhancement graph convolutional network for recognizing traffic police command gestures

Dual-module spatial temporal information enhancement graph convolutional network for recognizing traffic police command gestures

Stgcn-pad: a spatial-temporal graph convolutional network for detecting abnormal pedestrian motion patterns at grade crossings

This paper presents a Spatial-Temporal Graph Convolutional Network-based Pedestrians’ behaviors Anomaly Detection system (STGCN-PAD) for grade crossings. The behaviors of pedestrians are represented in a structured manner by skeleton trajectories that are generated using a pose estimation model. The ST-GCN components are sequentially applied to capture the spatial dependencies between skeleton key points within a single video frame and the temporal relationships for each of them. Based on these features, the system reconstructs input trajectories with a constant sliding window size, and the reconstruction error is used to distinguish abnormal behaviors from those normal. To accelerate the processing of extracted multi-dimensional feature maps, an MLP-Mixer model-based reconstruction network is developed as an alternative to the traditional convolution neural network. Only trajectories of normal walking behavior are included for model training. Anomalies, such as lingering and squatting activities, can be identified as outliers by observing the magnitude of reconstruction errors. The case studies demonstrate the salient feasibility and efficiency of the proposed system, which achieves at least comparable performance (approximately 88% in the AUC evaluation metric) with several state-of-the-art approaches while using the MLP-Mixer model accelerates model inference by 10× relative to our previous effort (Song et al. in Appl Intell 53:21676–21691, 2023).

Identifying and Animating Movement of Zeibekiko Sequences by Spatial Temporal Graph Convolutional Network with Multi Attention Modules

Identifying and Animating Movement of Zeibekiko Sequences by Spatial Temporal Graph Convolutional Network with Multi Attention Modules

Multi-relation spatiotemporal graph residual network model with multi-level feature attention: A novel approach for landslide displacement prediction

Accurate prediction of landslide displacement is crucial for effective early warning of landslide disasters. While most existing prediction methods focus on time-series forecasting for individual monitoring points, there is limited research on the spatiotemporal characteristics of landslide deformation. This paper proposes a novel Multi-Relation Spatiotemporal Graph Residual Network with Multi-Level Feature Attention (MFA-MRSTGRN) that effectively improves the prediction performance of landslide displacement through spatiotemporal fusion. This model integrates internal seepage factors as data feature enhancements with external triggering factors, allowing for accurate capture of the complex spatiotemporal characteristics of landslide displacement and the construction of a multi-source heterogeneous dataset. The MFA-MRSTGRN model incorporates dynamic graph theory and four key modules: multi-level feature attention, temporal-residual decomposition, spatial multi-relational graph convolution, and spatiotemporal fusion prediction. This comprehensive approach enables the efficient analyses of multi-source heterogeneous datasets, facilitating adaptive exploration of the evolving multi-relational, multi-dimensional spatiotemporal complexities in landslides. When applying this model to predict the displacement of the Liangshuijing landslide, we demonstrate that the MFA-MRSTGRN model surpasses traditional models, such as random forest (RF), long short-term memory (LSTM), and spatial temporal graph convolutional networks (ST-GCN) models in terms of various evaluation metrics including mean absolute error (MAE = 1.27 mm), root mean square error (RMSE = 1.49 mm), mean absolute percentage error (MAPE = 0.026), and R-squared (R2 = 0.88). Furthermore, feature ablation experiments indicate that incorporating internal seepage factors improves the predictive performance of landslide displacement models. This research provides an advanced and reliable method for landslide displacement prediction.

The analysis of regional ice and snow tourist destinations under back propagation neural network

This study aims to analyze the evolutionary characteristics and development levels of regional ice and snow tourist destinations by integrating the Back Propagation Neural Network (BPNN) within an Internet of Things (IoT) framework. Data from multiple sources are gathered through web scraping technology from various online platforms and are then subjected to cleaning, standardization, and normalization. A feature recognition model for ice and snow tourism is constructed based on a BPNN combined with a Spatial-Temporal Graph Convolutional Network (ST-GCN) algorithm. Experimental results demonstrate that this model excels in convergence speed and prediction accuracy, achieving a final convergence value of 0.059 and a prediction accuracy of 95.74 %, which is at least 4 % higher than that of the baseline BPNN algorithm. Additionally, the model yields Recall and F1 scores of 91.57 % and 89.31 %, respectively. After 98 iterations, the Root Mean Square Error (RMSE) is 6.26, significantly outperforming other model algorithms. These results indicate that the proposed model offers substantial advantages in enhancing the management and service quality of ice and snow tourist destinations, thereby providing valuable technical support and guidance for future intelligent tourism management.

Beyond homophily in spatial–temporal traffic flow forecasting

Traffic flow forecasting is a crucial yet complex task due to the intricate spatial–temporal correlations arising from road interactions. Recent methods model these interactions using message-passing Graph Convolution Networks (GCNs), which work for homophily graphs where connected nodes primarily exhibit close observations. However, relying solely on homophily graphs presents inherent limitations in traffic modeling, as road interactions can yield not only close but also distant observations over time, revealing diverse and dynamic node-wise correlations. We designate this phenomenon as homophily-heterophily dynamics, which has been largely overlooked in previous works. To address this gap, we propose a Homophily-Heterophily Spatial-Temporal Graph Convolution Network (H2STGCN) that exploits both homophily and heterophily components in the spatial–temporal domain. Specifically, we first adopt time-related node attributes to disentangle the diverse and dynamic node-wise relations across time, thereby obtaining homophily and heterophily Spatial-Temporal Graphs (STGs), which provide comprehensive insights into road interactions. Subsequently, we construct dual information propagation branches, each outfitted with a specific type of STG, to exploit multiple ranges of spatial–temporal correlations from distinct perspectives through dilated causal spatial–temporal graph convolution operations on STGs. Additionally, we introduce a Graph Collaborative Learning Module (GCLM) to capture the complementary information of these two branches via mutual information transfer. Experimental evaluation on four real-world traffic datasets reveals that our model outperforms state-of-the-art methods.

Dynamic multi-scale spatial-temporal graph convolutional network for traffic flow prediction

This paper proposes a dynamic multi-scale spatial-temporal graph convolutional network (DS-STGCN) for traffic flow prediction. The network aims to comprehensively extract global and local dependencies in dynamic spatial-temporal data by inputting traffic network flow data to construct node feature graphs, topology graphs, and time slot feature graphs, capturing the complexity and dynamics of traffic flow. DS-STGCN interprets feature information of the traffic network from both spatial and temporal dimensions through dynamic multi-scale graph convolutional blocks. In the spatial dimension, these blocks use constraints at different levels to balance fine-grained local features and extensive global features, revealing the intrinsic structure of traffic flow data. In the temporal dimension, these blocks jointly learn with temporal convolutional blocks to capture multi-frequency time patterns and handle long sequence data, effectively extracting potential dependencies of time series. Furthermore, DS-STGCN effectively models the changing spatial-temporal relationships in road network flow by constructing dynamically adaptive updated adjacency tensors, generating dynamic graph structures to address the challenge of changing spatial-temporal relationships in the transportation system. Experimental results show that our method significantly outperforms other competing methods on five real traffic datasets (PEMS03, PEMS04, PEMS07, PEMS08 and METR-LA).

SWL-LSE: A Dataset of Health-Related Signs in Spanish Sign Language with an ISLR Baseline Method

Progress in automatic sign language recognition and translation has been hindered by the scarcity of datasets available for the training of machine learning algorithms, a challenge that is even more acute for languages with smaller signing communities, such as Spanish. In this paper, we introduce a dataset of 300 isolated signs in Spanish Sign Language, collected online via a web application with contributions from 124 participants, resulting in a total of 8000 instances. This dataset, which is openly available, includes keypoints extracted using MediaPipe Holistic. The goal of this paper is to describe the construction and characteristics of the dataset and to provide a baseline classification method using a spatial–temporal graph convolutional network (ST-GCN) model, encouraging the scientific community to improve upon it. The experimental section offers a comparative analysis of the method’s performance on the new dataset, as well as on two other well-known datasets. The dataset, code, and web app used for data collection are freely available, and the web app can also be used to test classifier performance on-line in real-time.

Interpretable Spatial–Temporal Graph Convolutional Network for System Log Anomaly Detection

To ensure seamless information flow and operational integrity, computer systems need effectively to manage their system logs, but the expansion in their scale and complexity makes it hard to detect anomalies. Current methodologies exhibit deficiencies, including inefficiencies in handling abnormal sequences, lack of interpretability, and limited consideration of both temporal and spatial information. To improve, this paper develops a semi-supervised graph neural network model termed the Interpretable Spatial–Temporal Graph Convolutional Network (IST-GCN). By integrating temporal and event similarity perspectives, the IST-GCN harnesses directed and undirected graphs to capture the temporal and spatial aspects of system log events. Hence, the IST-GCN offers temporal and spatial interpretability. Further, a lightweight feature regularization technique is developed to enhance interpretability in both time and space domains, and thus facilitates anomaly detection efficiently. Comprehensive testing verifies that the IST-GCN approach surpasses nearly all state-of-the-art methods across five public log anomaly datasets. On average, IST-GCN improves Average Precision (AP) by approximately 3% and ROC AUC (RC) by about 4% compared to the best-performing baseline methods, underscoring its effectiveness and robustness.

Multi-granularity spatial temporal graph convolution network with consecutive attention for human motion prediction

Human motion prediction is attracting increasing attention for its numerous potential applications in fields including autonomous driving, video surveillance and virtual reality. However, accurate motion prediction is challenging due to the complex spatial dependencies, dynamic temporal correlations and high dimension of human pose sequences. Existing graph-based methods rarely consider positional and channel information of the feature map, resulting in lower prediction accuracy. Therefore, we propose a novel multi-granularity spatial temporal graph convolution network with consecutive attention (MSTCA) for human motion prediction. Firstly, a multi-granularity spatial convolution network is introduced to capture spatial joint features through multiple kernel sizes. Then, consecutive attention module is proposed to capture both positional and channel information of the feature map. Next, MSTCA uses multi-granularity temporal convolutional network to extract temporal correlations with multiple receptive fields and predict future poses. Finally, a decoder composed of a 2D convolution layer and several PRelu layers integrates the output of the whole model. Experimental results on the GTA-IM and PROX datasets demonstrate that our method significantly improves the accuracy of human motion prediction in comparison to the existing approaches.

Forecasting road network functionality states during extreme rainfall events to facilitate real-time emergency response planning

Rapid prediction of Road Network Functionality (RNF) during extreme rainfall-induced flooding is crucial for supporting proactive and real-time emergency planning, such as rescue, evacuation planning, and emergency supply distribution. Unlike normal operational conditions, extreme rainfall events introduce complex non-stationary, non-Euclidean characteristics to RNF due to intricate meteorological and hydrological processes, as well as the role of a community's road network in emergency response planning. Conventional physics-based flood simulations and flow-based road network analyses typically lack the computational efficiency required for real-time RNF predictions, hindering timely risk mitigation decisions. This study leverages the accuracy of physics-based simulations and the efficacy of deep-learning technologies to develop a deep learning-based surrogate model for Rain-to-RNF (R2R) predictions. This model couples Long Short-Term Memory (LSTM) networks with Spatial-Temporal Graph Convolutional Networks (ST-GCNs) to uniquely capture the spatiotemporal dynamics of RNF under extreme rainfall events. The predictive accuracy, stability, and versatility of the R2R surrogate model are demonstrated in four flood-prone communities in Zhejiang Province. Its implementation during Typhoon Fitow (2013) over a 30-hour intense rainfall showcases its promising predictive capacity and unparalleled computational efficiency. This research advances disaster management, enhancing the resilience and responsiveness of community infrastructure during extreme weather events.

AdpSTGCN: Adaptive spatial–temporal graph convolutional network for traffic forecasting

Traffic flow forecasting plays a crucial role in applications such as intelligent transportation systems. Despite significant research in this field, the current methods have limitations that hinder the realization of highly accurate predictions. Existing GCN-based approaches typically rely on a definite graph structure derived from a physical topology or learned from node features, which is insufficient for building intricate spatial relationships among nodes. To address this challenge, we propose an adaptive spatial–temporal graph convolutional network for traffic forecasting. Our approach exploits a multi-head attention mechanism to construct multi-view feature graphs. We then introduce an adaptive graph convolution method to dynamically aggregate and propagate information from both the topology graph and multi-view feature graphs, which are capable of capturing complex spatial correlations across diverse proximity ranges. Furthermore, we designed a cascaded structural framework that combines temporal information with node features using gated dilated causal convolution to ensure the integrated modeling of spatial–temporal dynamics in traffic flow. Experiments on real-world datasets demonstrate that our proposed method outperforms the current mainstream methods, achieving better performance in traffic flow forecasting. The code is available at https://github.com/dhxdla/AdpSTGCN.git.

Spatial and temporal attention embedded spatial temporal graph convolutional networks for skeleton based gait recognition with multiple IMUs

Gait recognition is one of the key technologies for exoskeleton robot control, while the current IMU-based gait recognition methods only use inertial data and do not fully consider the interconnections of human spatial structure and human joints. In this regard, a skeleton-based gait recognition approach with inertial measurement units using spatial temporal graph convolutional networks with spatial and temporal attention is proposed. A human forward kinematics solver module was used for constructing different human skeleton models and a temporal attention module was added for capturing the more important time frames in the gait cycle. Moreover, the two-stream structure was used to construct spatial temporal graph convolutional networks with spatial and temporal attention for gait recognition, and an average accuracy of about 99% was obtained in user experiments, which is the best performance compared to other algorithms, provides certain reference for gait recognition and real-time control of exoskeleton robots.

One Size Fits All: A Unified Traffic Predictor for Capturing the Essential Spatial-Temporal Dependency.

Traffic prediction is a keystone for building smart cities in the new era and has found wide applications in traffic scheduling and management, environment policy making, public safety, and so on. Instead of creating a traffic predictor for each city, this article focuses on designing a unified network model that could be directly applied for traffic prediction in any city, by learning the essential spatial-temporal dependencies, i.e., the mutual relationship between traffic and the corresponding fine-grained road network. To achieve this goal, this article proposes a joint knowledge- and data-driven mechanism that novelly divides dependencies into three kinds of correlations, i.e., road segment, intra-intersection, and inter-intersection correlation, which capture the microcosmic, middle, and macroscopic dependencies between traffic and the road network, respectively. Specifically, we first construct traffic datasets that could cover all road segments from real-world trajectory datasets, which makes it possible to model the whole road network as a graph, with the help of fine-grained road topology. Then, we propose meta road segment learner, connection-aware spatial-temporal graph convolutional network (GCN), and multiscale residual networks for capturing the microcosmic, middle, and macroscopic dependencies, respectively. Our experiments on three real-world datasets demonstrate that our proposed method could: 1) achieve better prediction accuracy compared with several approaches and 2) capture the mutual relationship between traffic and the fine-grained road network since our model trained only using data from the source city achieves good performance when it is directly applied for traffic prediction in the target city, without any fine-tuning. The codes will be made publicly available.

Attention Mechanism Based Spatial-Temporal Graph Convolution Network for Traffic Prediction

Attention Mechanism Based Spatial-Temporal Graph Convolution Network for Traffic Prediction

Partitioning Leakage Detection in Water Distribution Systems: A Specialized Deep Learning Framework Enhanced by Spatial–Temporal Graph Convolutional Networks

Partitioning Leakage Detection in Water Distribution Systems: A Specialized Deep Learning Framework Enhanced by Spatial–Temporal Graph Convolutional Networks

Rehabilitation exercise quality assessment through supervised contrastive learning with hard and soft negatives.

Exercise-based rehabilitation programs have proven to be effective in enhancing the quality of life and reducing mortality and rehospitalization rates. AI-driven virtual rehabilitation, which allows patients to independently complete exercises at home, utilizes AI algorithms to analyze exercise data, providing feedback to patients and updating clinicians on their progress. These programs commonly prescribe a variety of exercise types, leading to a distinct challenge in rehabilitation exercise assessment datasets: while abundant in overall training samples, these datasets often have a limited number of samples for each individual exercise type. This disparity hampers the ability of existing approaches to train generalizable models with such a small sample size per exercise type. Addressing this issue, this paper introduces a novel supervised contrastive learning framework with hard and soft negative samples that effectively utilizes the entire dataset to train a single model applicable to all exercise types. This model, with a Spatial-Temporal Graph Convolutional Network (ST-GCN) architecture, demonstrated enhanced generalizability across exercises and a decrease in overall complexity. Through extensive experiments on three publicly available rehabilitation exercise assessment datasets, UI-PRMD, IRDS, and KIMORE, our method has proven to surpass existing methods, setting a new benchmark in rehabilitation exercise quality assessment.

Appearance-posture fusion network for distracted driving behavior recognition

In recent years, detection techniques using computer vision and deep learning have shown promise in assessing driver distraction. This paper proposes a fusion network that combines a Spatial-Temporal Graph Convolutional Network (ST-GCN) and a hybrid convolutional network to integrate multimodal input data for recognizing distracted driver behavior. Specifically, to address the limitations of the ST-GCN method in modeling long-distance joint interaction features and inadequate temporal feature extraction, we design the Spatially Symmetric Configuration Partitioning Graph Convolutional Network (SSCP-GCN) to model relative motion information of symmetric relationships between limbs. Specifically, we utilize densely connected blocks for processing multi-scale temporal information between consecutive frames, thereby enhancing the reuse of bottom features. Furthermore, the expression of important temporal information is augmented by the introduction of the channel attention mechanism. To tackle the problem that the Mixed Convolution (MC) combining 3D convolution with 2D convolution cannot extract higher-order timing information and has limitations in modeling global dependency relationships, we compensate for its inability to capture higher-order temporal semantic information using the Time Shift Module (TSM) without consuming additional computational resources. Additionally, the 3D Multi-Head Self-Attention mechanism (3D MHSA) is employed to integrate global spatial–temporal information of high-level features, avoiding the issue of model complexity proliferation caused by the deep stacking design of Convolutional Neural Networks (CNN). Lastly, we introduce a multistream network framework that integrates driver posture and appearance features to harness complementary advantages, enabling us to combine multimodal input features to achieve better model performance. Experimental results indicate that the accuracy of the network designed in this paper reaches 95.6% and 94.3% on ASU dataset and NTU-RGB+D dataset, respectively. The small size of the model offers the possibility for practical application of the algorithm.

Multiscale information enhanced spatial-temporal graph convolutional network for multivariate traffic flow forecasting via magnifying perceptual scope

Graph Convolutional Networks (GCNs), which can model data in non-Euclidean space, have received extensive attention in multivariate traffic flow forecasting in recent years. However, most of the existing GCN forecasting methods rely on adjacency matrices defined by road network knowledge, which cannot accurately capture the inherent spatial-temporal dependencies of complex multivariate traffic flows. In this paper, a multiscale information enhanced spatial-temporal GCN is proposed, which utilizes an adaptive graph learning layer to automatically extract the dependencies between variables and constructs adaptive adjacency matrices for different datasets. In addition, an information enhancement module is introduced to magnify perceptual scope of the model and strengthen the connection between information, so that the spatial-temporal graph convolution module can better capture the spatial-temporal dependencies in the traffic flow. We conducted experiments on multivariate traffic flow data from three different real-world scenarios to evaluate the effectiveness of the proposed method. Comparative experimental results consistently show that the proposed method outperforms existing GCNs, achieving an average improvement of 21.16% on Mean Absolute Error (MAE), 18.87% on Root Mean Square Error (RMSE), and 8.27% on R2 score (R2) in the three datasets, respectively, and the results demonstrate its excellent forecasting performance in the field of multivariate traffic flow forecasting.

Shale Gas Production Forecasting with Well Interference Based on Spatial-Temporal Graph Convolutional Network

Summary One of the core assumptions of most deep-learning-based data-driven models is that samples are independent. However, this assumption poses a key challenge in production forecasting—performance is influenced by well interference and reservoir connectivity. Most shale gas wells are hydraulically fractured and exist in complex fracture systems, and the neighboring well characteristics should also be considered when constructing data-driven forecast models. Researchers have explored using the graph convolutional network (GCN) to address this issue by incorporating neighboring well characteristics into production forecasting models. However, applying GCN to field-scale studies is problematic, as it requires training on a full batch, leading to gigantic cache allocation. In addition, the transductive nature of GCN poses challenges for direct generalization to unseen nodes. To overcome these limitations, we adopt the graph sampling and aggregation (GraphSAGE) network architecture, which allows training large graphs with batches and generalizing predictions for previously unseen nodes. By utilizing the gated recurrent unit (GRU) network, the proposed spatial-temporal (ST)-GraphSAGE model can capture cross-time relationships between the target and the neighboring wells and generate promising prediction time series for the target wells, even if they are newly drilled wells. The proposed approach is validated and tested using the field data from 2,240 Montney shale gas wells, including formation properties, hydraulic fracture parameters, production history, and operational data. The algorithm aggregates the first-hop information to the target node for each timestep. The encoder-decoder (ED) architecture is used to generate forecasts for the subsequent 3-year production rate by using the 1-year production history of the wells. The trained model enables the evaluation of production predictions for newly developed wells at any location. We evaluate the model’s performance using P10, P50, and P90 of the test data set’s root mean square error (RMSE). Our method preserves the topological characteristics of wells and generalizes the prediction to unseen nodes while significantly reducing training complexity, making it applicable to larger data sets. By incorporating information from adjacent wells and integrating ST data, our ST-GraphSAGE model outperforms the traditional GRU-ED model and shows enhanced interpretability.