Spatio-temporal Graph Convolutional Network Research Articles

Dual-branch spatio-temporal graph neural networks for pedestrian trajectory prediction

Pedestrian trajectory prediction is an important area in computer vision, with wide applications in autonomous driving, robot path planning, and surveillance systems. The core underlying technique of these applications is pattern recognition. A key challenge in this area is modeling social interactions between pedestrians, such as pedestrian view area and group behaviors. However, although many methods have been proposed to model social interactions, pedestrian view area and group behaviors have not been explored together to account for complex situations. Additionally, most existing studies require additional detectors and manual annotations to handle view area and group interactions, respectively. In this paper, we propose a dual-branch spatio-temporal graph neural network to automatically model view area and grouping together. Specifically, a spatio-temporal graph attention network (STGAT) branch is designed to handle pedestrian view area, and a spatio-temporal graph convolutional network (STGCN) branch is designed to model group interactions. The features of these branches are then fused to provide better feature representations, on which a temporal convolution operation (TCN) is performed for trajectory prediction. Experiments on public standard datasets demonstrate that the proposed method achieves very competitive performance and predicts socially acceptable trajectories in different challenging scenarios.

ICE-GCN: An interactional channel excitation-enhanced graph convolutional network for skeleton-based action recognition

Thanks to the development of depth sensors and pose estimation algorithms, skeleton-based action recognition has become prevalent in the computer vision community. Most of the existing works are based on spatio-temporal graph convolutional network frameworks, which learn and treat all spatial or temporal features equally, ignoring the interaction with channel dimension to explore different contributions of different spatio-temporal patterns along the channel direction and thus losing the ability to distinguish confusing actions with subtle differences. In this paper, an interactional channel excitation (ICE) module is proposed to explore discriminative spatio-temporal features of actions by adaptively recalibrating channel-wise pattern maps. More specifically, a channel-wise spatial excitation (CSE) is incorporated to capture the crucial body global structure patterns to excite the spatial-sensitive channels. A channel-wise temporal excitation (CTE) is designed to learn temporal inter-frame dynamics information to excite the temporal-sensitive channels. ICE enhances different backbones as a plug-and-play module. Furthermore, we systematically investigate the strategies of graph topology and argue that complementary information is necessary for sophisticated action description. Finally, together equipped with ICE, an interactional channel excited graph convolutional network with complementary topology (ICE-GCN) is proposed and evaluated on three large-scale datasets, NTU RGB+D 60, NTU RGB+D 120, and Kinetics-Skeleton. Extensive experimental results and ablation studies demonstrate that our method outperforms other SOTAs and proves the effectiveness of individual sub-modules. The code will be published at https://github.com/shuxiwang/ICE-GCN.

Hierarchical Spatio–Temporal Graph Convolutional Networks and Transformer Network for Traffic Flow Forecasting

Graph convolutional networks (GCN) have been applied in the traffic flow forecasting tasks with the graph capability in describing the irregular topology structures of road networks. However, GCN based traffic flow forecasting methods often fail to simultaneously capture the short-term and long-term temporal relations carried by the traffic flow data, and also suffer the over-smoothing problem. To overcome the problems, we propose a hierarchical traffic flow forecasting network by merging newly designed the long-term temporal Transformer network (LTT) and the spatio-temporal graph convolutional networks (STGC). Specifically, LTT aims to learn the long-term temporal relations among the traffic flow data, while the STGC module aims to capture the short-term temporal relations and spatial relations among the traffic flow data, respectively, via cascading between the one-dimensional convolution and the graph convolution. In addition, an attention fusion mechanism is proposed to combine the long-term with the short-term temporal relations as the input of the graph convolution layer in STGC, in order to mitigate the over-smoothing problem of GCN. Experimental results on three public traffic flow datasets prove the effectiveness and robustness of the proposed method.

Spatio-Temporal Graph Convolutional Networks via View Fusion for Trajectory Data Analytics

Trajectory data contains rich spatial and temporal information. Turning trajectories into graphs and then analyzing them efficiently in an AI-empowered way is a representative branch of trajectory analysis in IoV and ITS environments, which is of great significance. This research attempts to project trajectories onto road networks to predict traffic conditions. Extracting accurate spatio-temporal dependencies is the key to improving the analysis. However, two problems exist in the current study. The first one is the focus on the network structure while ignoring node features, and the second one is that the structure cannot be fully utilized. In addition, the static spatial structure may not accurately reflect the dynamic real spatial dependency. In response to these problems, a novel Spatio-Temporal Graph Convolutional Networks via View Fusion for Trajectory Data Analytics (STFGCN) model is designed. It contains two independent views: the structural view and feature view. The view fusion layer is further designed. It includes an extended graph convolutional module and a causal dilated module. The extended graph convolutional module fully extracts dynamic spatial dependencies, while the causal dilated module captures time tendencies. Stacked view fusion layers and a view fusion module perform fusion operations based on the advantages of the two views, efficiently integrating information from both. Several experiments are performed on two real-world trajectory datasets. The results show that a better prediction performance is obtained, especially on the long-range time prediction task.

Aircraft Trajectory Prediction for Terminal Airspace Employing Social Spatiotemporal Graph Convolutional Network

Four-dimensional (4-D) trajectory prediction is the core of air traffic management technologies such as flow management, conflict detection and resolution, arrival and departure sequencing, and aircraft abnormal behavior monitoring. The airport terminal airspace has a complex airspace structure, and the flight status of aircraft is changeable, which poses a challenge to trajectory prediction. To this end, aiming at the problem of track prediction in airport terminal area, we propose a 4-D trajectory prediction model of social spatiotemporal graph convolutional neural network (S-STGCNN) based on pattern matching. For each type of flight pattern, an S-STGCNN is trained to improve the robustness. Taking each aircraft as a node of a graph, the spatiotemporal graph convolution is used to extract features of the graph so as to simultaneously characterize the time dependence of trajectories and the interaction between aircraft. The time extrapolation convolutional neural network is used to generate the predicted trajectory. Experimental results show that the trajectory prediction method proposed in this paper has a higher prediction accuracy and a generalization ability than other models.

Determination of workers' compliance to safety regulations using a spatio-temporal graph convolution network

The safety of workers in construction remains a critical issue despite the automation of several tasks with fewer workers on site. As fatal accidents of workers account for a significant number of construction accidents, considerable effort has been made to monitor workers’ safety behaviors with additional personnel for supervising workers. With the advancement of data analytics, recent research has reported various human activity recognition methods based on image data to perform automated worker monitoring without additional labor. Nevertheless, unlike existing approaches based on a single image, a method that can capture a series of actions from sequential images is required to monitor workers’ compliance with safety behavior. To this end, an approach based on OpenPose and a spatio-temporal graph convolutional network is proposed in this study to evaluate workers’ compliance with safety regulations using sequential videos. The two primary functions of the developed method include 1) classifying each safety behavior among five representative behaviors stipulated in construction, and 2) determining the compliance of workers with each safety regulation. The results indicate that the developed approach can capture momentary safety behaviors and workers’ compliance with feasible accuracy of an average F1 score greater than 0.8. Furthermore, the proposed method can be extended to safety intervention policies with behavior-based feedback to inform workers of their non-compliance with safety behaviors. Therefore, this study contributes to proactive safety management by focusing on workers’ behavioral levels rather than on accident rate-based management.

Continual spatio-temporal graph convolutional networks

Graph-based reasoning over skeleton data has emerged as a promising approach for human action recognition. However, the application of prior graph-based methods, which predominantly employ whole temporal sequences as their input, to the setting of online inference entails considerable computational redundancy. In this paper, we tackle this issue by reformulating the Spatio-Temporal Graph Convolutional Neural Network as a Continual Inference Network, which can perform step-by-step predictions in time without repeat frame processing. To evaluate our method, we create a continual version of ST-GCN, CoST-GCN, alongside two derived methods with different self-attention mechanisms, CoAGCN and CoS-TR. We investigate weight transfer strategies and architectural modifications for inference acceleration, and perform experiments on the NTU RGB+D 60, NTU RGB+D 120, and Kinetics Skeleton 400 datasets. Retaining similar predictive accuracy, we observe up to 109× reduction in time complexity, on-hardware accelerations of 26×, and reductions in maximum allocated memory of 52% during online inference.

Operating Status Prediction Model at EV Charging Stations With Fusing Spatiotemporal Graph Convolutional Network

This article proposes the operating status prediction model at electric vehicle (EV) charging stations based on the spatiotemporal graph convolutional network (SGCN). The SGCN combines graph convolutional network (GCN) and the gated recurrent unit (GRU), alleviating the queuing time at charging stations due to the lack of information for EV users. First, an urban charging station-traffic flow model is established to portray the interrelationship between charging stations and traffic. Second, a multistep prediction model based on SGCN for operating status at charging stations is proposed to forecast the occupancy of charging stations over the next tens of minutes. The comparison case study with the forecast and actual data reveals that the mean forecast error is around 19.21% when estimating 18 min ahead. Incrementing errors are subtle even after adding random noise to the original data. Finally, the model is applied to charging guidance decisions. Our model can reduce the number of EV queuing by 60% during high charging demand. It also shortens the average charging waiting time by 4 min.

Learning from explainable data-driven tunneling graphs: A spatio-temporal graph convolutional network for clogging detection

Clogging challenges caused by mud cake in shield tunneling can hinder the efficiency, making it of great significance for clogging detection during tunneling. However, low explainability of previous intelligent computing methods block their applications. An explainable spatiotemporal graph convolutional network for clogging detection is therefore proposed to judge the risk level of clogging for each ring. The graphs are transformed from original multi-dimensional time-series monitoring data, which include spatio-temporal information, whose risk are leveled by experts as labels. A practical tunneling case is applied to discuss the results of the model and the data from another tunneling line is applied to validate the robustness. The hierarchical pooling process and attention mechanism highlight the important nodes in each graph as the spatio-temporal explainability. Typical clogging characteristics can be observed in those important nodes. With complex network indices, more detailed explanations are obtained, showing potential value in learning from the explainable AI.

A Hierarchical Spatio-Temporal Graph Convolutional Neural Network for Anomaly Detection in Videos

Deep learning models have been widely used for anomaly detection in surveillance videos. Typical models are equipped with the capability to reconstruct normal videos and evaluate the reconstruction errors on anomalous videos to indicate the extent of abnormalities. However, existing approaches suffer from two disadvantages. Firstly, they can only encode the movements of each identity independently, without considering the interactions among identities which may also indicate anomalies. Secondly, they leverage inflexible models whose structures are fixed under different scenes, this configuration disables the understanding of scenes. In this paper, we propose a Hierarchical Spatio-Temporal Graph Convolutional Neural Network (HSTGCNN) to address these problems, the HSTGCNN is composed of multiple branches that correspond to different levels of graph representations. High-level graph representations encode the trajectories of people and the interactions among multiple identities while low-level graph representations encode the local body postures of each person. Furthermore, we propose to weightedly combine multiple branches that are better at different scenes. An improvement over single-level graph representations is achieved in this way. An understanding of scenes is achieved and serves anomaly detection. High-level graph representations are assigned higher weights to encode moving speed and directions of people in low-resolution videos while low-level graph representations are assigned higher weights to encode human skeletons in high-resolution videos. Experimental results show that the proposed HSTGCNN significantly outperforms current state-of-the-art models on four benchmark datasets (UCSD Pedestrian, ShanghaiTech, CUHK Avenue and IITB-Corridor) by using much less learnable parameters.

Unrolled Spatiotemporal Graph Convolutional Network for Distribution System State Estimation and Forecasting

Timely perception of distribution system states is critical for the control and operation of power grids. Recently, it has been seriously challenged by the dramatic voltage fluctuations induced by high renewables. To address this issue, an Unrolled Spatiotemporal Graph Convolutional Network (USGCN) is proposed for distribution system state estimation (DSSE) and forecasting with augmented consideration of the underlying complex spatiotemporal correlations of renewable energy sources (RES). Specifically, the interconnection among individual spatial graphs of adjacent time steps will lead to an unrolled spatiotemporal graph and benefit the synchronous capture of spatial and temporal correlations to achieve enhanced accuracy. On top of this, the node-embedding technique is employed in the unrolled spatiotemporal convolutional layer to reveal the hidden nonlinear spatiotemporal correlations of RES outputs without relying on full prior knowledge. Moreover, the proposed USGCN stacks the unrolled spatiotemporal convolutional layers, leading to the perception of longtime correlations to obtain effective ahead-of-time state forecasting results robustly. The simulation results have been provided to verify the accuracy and efficiency of the proposed model in 118-node and 1746-node distribution systems.

Facial Action Unit Detection via Adaptive Attention and Relation

Facial action unit (AU) detection is challenging due to the difficulty in capturing correlated information from subtle and dynamic AUs. Existing methods often resort to the localization of correlated regions of AUs, in which predefining local AU attentions by correlated facial landmarks often discards essential parts, or learning global attention maps often contains irrelevant areas. Furthermore, existing relational reasoning methods often employ common patterns for all AUs while ignoring the specific way of each AU. To tackle these limitations, we propose a novel adaptive attention and relation (AAR) framework for facial AU detection. Specifically, we propose an adaptive attention regression network to regress the global attention map of each AU under the constraint of attention predefinition and the guidance of AU detection, which is beneficial for capturing both specified dependencies by landmarks in strongly correlated regions and facial globally distributed dependencies in weakly correlated regions. Moreover, considering the diversity and dynamics of AUs, we propose an adaptive spatio-temporal graph convolutional network to simultaneously reason the independent pattern of each AU, the inter-dependencies among AUs, as well as the temporal dependencies. Extensive experiments show that our approach (i) achieves competitive performance on challenging benchmarks including BP4D, DISFA, and GFT in constrained scenarios and Aff-Wild2 in unconstrained scenarios, and (ii) can precisely learn the regional correlation distribution of each AU.

Production Forecasting Based on Attribute-Augmented Spatiotemporal Graph Convolutional Network for a Typical Carbonate Reservoir in the Middle East

Production forecasting plays an important role in development plans during the entire period of petroleum exploration and development. Artificial intelligence has been extensively investigated in recent years because of its capacity to extensively analyze and interpret complex data. With the emergence of spatiotemporal models that can integrate graph convolutional networks (GCN) and recurrent neural networks (RNN), it is now possible to achieve multi-well production prediction by considering the impact of interactions between producers and historical production data simultaneously. Moreover, an accurate prediction not only depends on historical production data but also on the influence of neighboring injectors’ historical gas injection rate (GIR). Therefore, based on the assumption that introducing GIR can enhance prediction accuracy, this paper proposes a deep learning-based hybrid production forecasting model that is aimed at considering both the spatiotemporal characteristics of producers and the GIR of neighboring injectors. Specifically, we integrated spatiotemporal characteristics and GIR into an attribute-augmented spatiotemporal graph convolutional network (AST-GCN) and gated recurrent units (GRU) neural network to extract intricate temporal correlations from historical data. The method proposed in this paper has been successfully applied in a well pattern (including five producers and seven gas injectors) in a low-permeability carbonate reservoir in the Middle East. In single well production forecasting, the error of AST-GCN is 63.2%, 37.3%, and 16.1% lower in MedAE, MAE, and RMSE compared with GRU and 62.9%, 44.6%, and 28.9% lower compared with RNS. Similarly, the accuracy of AST-GCN is 15.9% and 35.8% higher than GRU and RNS in single well prediction. In well-pattern production forecasting, the error of AST-GCN is 41.2%, 64.2%, and 75.2% lower in RMSE, MAE, and MedAE compared with RNS, while the accuracy of AST-GCN is 29.3% higher. After different degrees of Gaussian noise are added to the actual data, the average change in AST-GCN is 3.3%, 0.4%, and 1.2% in MedAE, MAE, and RMSE, which indicates the robustness of the proposed model. The results show that the proposed model can consider the production data, gas injection data, and spatial correlation at the same time, which performs well in oil production forecasts.

Electrical-STGCN: An Electrical Spatio-Temporal Graph Convolutional Network for Intelligent Predictive Maintenance

With the rapid improvement of Industrial Internet of Things and artificial intelligence, predictive maintenance (PdM) has attracted great attention from both academia and industrial practitioners. When equipment is running, the electrical attributes have intrinsic relations. Meanwhile, they are changing over time. However, existing PdM models are often limited as they lack considering both attribute interactions and temporal dependence of the dynamic working system. To address the problem, in this article, we propose an electrical spatio-temporal graph convolutional network (Electrical-STGCN) for PdM. First, it takes a sequence of electrical records as input. Next, both attribute interactions and temporal dependence are established to extract features. Then, the extracted features are fed into a prediction component. Finally, the output of the Electrical-STGCN (i.e., remaining useful life) can help the workers decide whether to carry out equipment maintenance. The effectiveness of the proposed method is verified in real-world cases. Our method achieves 85.2% Accuracy and 0.9 F1-Score, which are better than the other approaches.

Retracted: Recognition of Hitting Action in Cyclic Anaerobic Volleyball by Acute Cooling Based on Improved Spatiotemporal Graph Convolutional Network

Retracted: Recognition of Hitting Action in Cyclic Anaerobic Volleyball by Acute Cooling Based on Improved Spatiotemporal Graph Convolutional Network

Dynamic traffic correlations based spatio-temporal graph convolutional network for urban traffic prediction

Accurate urban traffic prediction is a critical issue in Intelligent Transportation Systems (ITS). It is challenging since urban traffic usually indicates high dynamic spatio-temporal correlations, leading to uncertainty and complexity of traffic status. Since the transportation network is a graph structure practically, existing works have applied Graph Convolutional Network (GCN) on urban traffic prediction with a pre-defined adjacency matrix based on node distance or connectivity. However, in many urban traffic scenarios, spatio-temporal dependencies among traffic data usually change over time, so using a fixed adjacency matrix cannot describe the dynamic dependencies. To track the dynamic spatio-temporal dependencies among traffic data, we propose a novel deep learning framework, Dynamic Traffic Correlation-based Spatio-Temporal Graph Convolutional network (DTC-STGCN), to forecast traffic flow and speed accurately. DTC-STGCN extracts a dynamic adjacency matrix from different traffic characters to describe dynamic spatio-temporal correlations. Moreover, an attention and dynamic adjacency matrix-based GCNs framework is proposed to capture urban traffic dynamic spatial features, while a long-short-term memory network (LSTM) is used to capture urban traffic temporal features, respectively. Finally, we feed the spatio-temporal features generated by GCN and LSTM, with real road segments into a hybrid graph convolution framework to simultaneously model the dynamic spatial and temporal dependencies for traffic predictions. The experiments on two real-world datasets demonstrate that the proposed DTC-STGCN model consistently outperforms the state-of-the-art traffic prediction baselines on MAE and RMSE over 10%, and achieve a stable performance for two specific tasks (long-term traffic prediction and peak time prediction). And ablation study validates the effectiveness of dynamic adjacency matrix, attention mechanism, respectively.

A Spatio-Temporal Graph Convolutional Network Model for Internet of Medical Things (IoMT).

In order to provide intelligent and efficient healthcare services in the Internet of Medical Things (IoMT), human action recognition (HAR) can play a crucial role. As a result of their stringent requirements, such as high computational complexity and memory efficiency, classical HAR techniques are not applicable to modern and intelligent healthcare services, e.g., IoMT. To address these issues, we present in this paper a novel HAR technique for healthcare services in IoMT. This model, referred to as the spatio-temporal graph convolutional network (STGCN), primarily aims at skeleton-based human-machine interfaces. By independently extracting spatial and temporal features, STGCN significantly reduces information loss. Spatio-temporal information is extracted independently of the exact spatial and temporal point, ensuring the extraction of useful features for HAR. Using only joint data and fewer parameters, we demonstrate that our proposed STGCN achieved 92.2% accuracy on the skeleton dataset. Unlike multi-channel methods, which use a combination of joint and bone data and have a large number of parameters, multi-channel methods use both joint and bone data. As a result, STGCN offers a good balance between accuracy, memory consumption, and processing time, making it suitable for detecting medical conditions.

Spatiotemporal Graph Convolutional Networks for Earthquake Source Characterization.

Accurate earthquake location and magnitude estimation play critical roles in seismology. Recent deep learning frameworks have produced encouraging results on various seismological tasks (e.g., earthquake detection, phase picking, seismic classification, and earthquake early warning). Many existing machine learning earthquake location methods utilize waveform information from a single station. However, multiple stations contain more complete information for earthquake source characterization. Inspired by recent successes in applying graph neural networks (GNNs) in graph-structured data, we develop a Spatiotemporal Graph Neural Network (STGNN) for estimating earthquake locations and magnitudes. Our graph neural network leverages geographical and waveform information from multiple stations to construct graphs automatically and dynamically by adaptive message passing based on graphs' edges. Using a recent graph neural network and a fully convolutional neural network as baselines, we apply STGNN to earthquakes recorded by the Southern California Seismic Network from 2000 to 2019 and earthquakes collected in Oklahoma from 2014 to 2015. STGNN yields more accurate earthquake locations than those obtained by the baseline models and performs comparably in terms of depth and magnitude prediction, though the ability to predict depth and magnitude remains weak for all tested models. Our work demonstrates the potential of using GNNs and multiple stations for better automatic estimation of earthquake epicenters.

IS-STGCNN: An Improved Social spatial-temporal graph convolutional neural network for ship trajectory prediction

Trajectory prediction is a critical technology for ensuring ship navigation safety, improving the efficiency of marine traffic control, and efficiently searching for maritime targets. This research proposes an Improved Social-STGCNN (IS-STGCNN) to increase the accuracy of ship trajectory prediction. This technique successfully combines the benefits of the spatio-temporal graph convolutional neural network (STGCNN) for extracting ship trajectory features and the benefits of the time-extrapolated convolutional neural network (TXP-CNN) for generating future ship trajectory using the obtained features. This method proposes a social-sampling strategy based on prior knowledge, i.e., the bumper model of ship safety assessment, to carry out negative sampling and enhances the effectiveness of the IS-STGCNN method in learning the concept of negative examples (such as collisions). The interaction force between ships is estimated in a novel way in the spatio-temporal graph convolutional neural network to improve high-precision feature extraction from ship trajectory. Model predictive control (MPC) is used to correct the output of trajectory prediction in order to fulfill kinematic constraints and assure prediction kinematic feasibility. The experiments are carried out using AIS data, and the findings reveal that IS-STGCNN outperforms the state-of-the-arts.

Dynamic Prediction of Air Traffic Situation in Large-Scale Airspace

Air traffic situation prediction is critical for traffic flow management and the optimal allocation of airspace resources. In this study, the multi-sector airspace scenario is abstracted into an undirected graph. A spatiotemporal graph convolutional network (STGCN) model is developed to portray the spatiotemporal correlation between the sector operational situation changes. The model can predict multi-sector operational situations using time series data such as sector operational situation data and traffic volume within the sector. Experimenting on the air traffic situation dataset of 30 area sectors in the Shanghai control area revealed that the STGCN model has a prediction accuracy of above 90%, and it outperforms the benchmark method of traditional traffic prediction. This proves the effectiveness of the proposed situation prediction model.