Spatial Temporal Graph Convolutional Network Research Articles

Dynamic Origin-Destination Flow Prediction Using Spatial-Temporal Graph Convolution Network With Mobile Phone Data

Massive mobile phone data provide continuous and large-scale dynamic origin–destination (OD) flow information for multiple modes of transportation. In this study, we represent the dynamic OD flows obtained from mobile phone data as time-dependent graphs and propose two novel spatial-temporal graph convolutional network (STGCN)-based models to predict dynamic OD flows. Both models directly operate on the graph-structured OD flows, capture correlations among OD flows far apart in the Euclidean space, and fully explore the complex spatial-temporal features. We first formulate OD flows as explicit edges that specify the travels between two locations and propose an edge-focused STGCN. The edge-focused STGCN applies a novel three-step strategy to effectively update edge features in large-scale graphs. Second, we formulate OD flows as vertices in graph and propose a vertex-focused STGCN. The vertex-focused STGCN infers the relations among OD flows by establishing an adjacency matrix based on the temporal similarity between OD flows. The proposed models were validated using real-world mobile phone data collected in Kunshan, China. OD flows in the next hour were predicted, and the mean absolute percent errors of the edge-focused STGCN and the vertex-focused STGCN were 1.755% and 1.672%, respectively; both were significantly lower than the current baseline models.

Hierarchical Traffic Flow Prediction Based on Spatial-Temporal Graph Convolutional Network

In recent years, traffic flow prediction has attracted more and more interest from both academia and industry since such information can provide effective guidance for traffic management or driving planning and enhance traffic safety and efficiency. But due to the complicated spatial-temporal dependence in actual roads and the limitation of intersection monitoring equipment, there are still many challenges in spatial-temporal traffic flow prediction. In this paper, we propose a novel hierarchical traffic flow prediction protocol based on spatial-temporal graph convolutional network (ST-GCN), which incorporates both spatial and temporal dependence of intersection traffic to achieve a more accurate traffic flow prediction. Different from existing works, our proposed protocol with the Adjacent-Similar algorithm can also effectively predict the traffic flow of the intersections without historical data. Experiments based on practical traffic data of the city of Qingdao, China demonstrate that our proposed ST-GCN-based traffic flow prediction protocol outperforms the state-of-the-art baseline models. Moreover, as for the intersections without historical data, we can also obtain a good prediction accuracy.

An Energy Harvesting Roadside Unit communication load prediction and energy scheduling based on graph convolutional neural networks for spatial‐temporal vehicle data

The Energy Harvesting Roadside Unit (EH-RSU) with self-powered module will not only effectively reduce the communication load of regional Vehicular Ad Hoc Networks, but also enjoys a low deployment cost. Given the imbalance in communication demands invoked by transportation systems, the EH-RSU should allocate energy appropriately in accordance with its energy harvesting rate to ensure the communication safety of vehicles within its coverage. Firstly, we propose a novel attention-based spatial-temporal graph convolutional network (ASTGCN) to predict the communication load around the EH-RSU in the road network through the surrounding vehicle information. Secondly, we use the predicted communication load as part of the input parameters to neural network and leverage a double deep Q network to ameliorate the operating states switching strategy of EH-RSUs by reinforcement learning so that they achieve a more satisfying effective time with limited resources. Finally, we built a dataset by simulation to validate the effectiveness of our model. The results show that our prediction model has a better accuracy and the improved strategy has higher efficiency compared with other methods.

Deep Reinforcement Learning Based on Social Spatial–Temporal Graph Convolution Network for Crowd Navigation

In the crowd navigation, reinforcement learning based on graph neural network is a promising method, which effectively solves the poor navigation effect based on social interaction model and the freezing behavior of robot in extreme cases. However, since the information correlation of human trajectory has not been involved in the method, its performance still needs improvement. Therefore, we proposed a deep reinforcement learning model based on Social Spatial–Temporal Graph Convolution Network (SSTGCN) to handle the crowd navigation problem, in which the spatial–temporal information of human trajectory has been taken advantage to predict human behavior intentions and help robot plan path more efficiently. The model consists of graph learning module and robot forward planning module. In the graph learning module, the latent features of agents are taken advantage to reason about the relations among the agents, and SSTGCN is used to update feature matrix. In addition, value estimation module calculates state representation and state prediction module predicts the next state. The robot forward planning module makes use of k-step planning to estimate the quality of state and searches the best k steps planning. We tested our model in the Crowd-Nav platform, and the results show that our model has high navigation success rate and short navigation time. In addition, it has good robustness to crowd changes.

Enhancing the accuracy of a human emotion recognition method using spatial temporal graph convolutional networks

Enhancing the accuracy of a human emotion recognition method using spatial temporal graph convolutional networks

A spatial temporal graph neural network model for predicting flashover in arbitrary building floorplans

Rapid fire progression, such as flashover, has been one of the leading causes for firefighter deaths and injuries in residential building environments. Due to long computational time of and the required prior knowledge about the fire scene, existing models cannot be used to predict the potential occurrence of flashover in practical firefighting applications. In this paper, a scene-agnostic model (FlashNet) is proposed to predict flashover based on limited heat detector temperature information up to 150 °C. FlashNet utilizes spatial temporal graph convolutional neural networks to effectively learn features from the limited temperature information and to tackle building structure variations. The proposed model is benchmarked against five different state-of-the-art flashover prediction models. Results show that FlashNet outperforms the existing flashover prediction models and it can reliably predict flashover 30 s preceding its occurrence with an overall accuracy of about 92.1%. Ablation study is carried out to examine the effectiveness of different key model components and geometric average adjacency matrix. The research outcomes from this study are expected to enhance firefighters’ situational awareness in the fire scene, protecting them from hazardous fire environments and to pave the way for the development of data-driven prediction systems.

A Novel Early Warning Model for Hand, Foot and Mouth Disease Prediction Based on a Graph Convolutional Network

ObjectivesHand, foot and mouth disease (HFMD) is a widespread infectious disease that causes a significant disease burden on society. To achieve early intervention and to prevent outbreaks of disease, we propose a novel warning model that can accurately predict the incidence of HFMD. MethodsWe propose a spatial-temporal graph convolutional network (STGCN) that combines spatial factors for surrounding cities with historical incidence over a certain time period to predict the future occurrence of HFMD in Guangdong and Shandong between 2011 and 2019. The 2011–2018 data served as the training and verification set, while data from 2019 served as the prediction set. Six important parameters were selected and verified in this model and the deviation was displayed by the root mean square error and the mean absolute error. ResultsAs the first application using a STGCN for disease forecasting, we succeeded in accurately predicting the incidence of HFMD over a 12-week period at the prefecture level, especially for cities of significant concern. ConclusionsThis model provides a novel approach for infectious disease prediction and may help health administrative departments implement effective control measures up to 3 months in advance, which may significantly reduce the morbidity associated with HFMD in the future.

AFSTGCN: Prediction for multivariate time series using an adaptive fused spatial-temporal graph convolutional network

The prediction for Multivariate Time Series (MTS) explores the interrelationships among variables at historical moments, extracts their relevant characteristics, and is widely used in finance, weather, complex industries and other fields. Furthermore, it is important to construct a digital twin system. However, existing methods do not take full advantage of the potential properties of variables, which results in poor predicted accuracy. In this paper, we propose the Adaptive Fused Spatial-Temporal Graph Convolutional Network (AFSTGCN). First, to address the problem of the unknown spatial-temporal structure, we construct the Adaptive Fused Spatial-Temporal Graph (AFSTG) layer. Specifically, we fuse the spatial-temporal graph based on the interrelationship of spatial graphs. Simultaneously, we construct the adaptive adjacency matrix of the spatial-temporal graph using node embedding methods. Subsequently, to overcome the insufficient extraction of disordered correlation features, we construct the Adaptive Fused Spatial-Temporal Graph Convolutional (AFSTGC) module. The module forces the reordering of disordered temporal, spatial and spatial-temporal dependencies into rule-like data. AFSTGCN dynamically and synchronously acquires potential temporal, spatial and spatial-temporal correlations, thereby fully extracting rich hierarchical feature information to enhance the predicted accuracy. Experiments on different types of MTS datasets demonstrate that the model achieves state-of-the-art single-step and multi-step performance compared with eight other deep learning models.

DSTGCN: Dynamic Spatial-Temporal Graph Convolutional Network for Traffic Prediction

Traffic prediction is an important part of building a smart city. Reasonable traffic prediction can help the relevant departments to make important decisions and help people to plan their travel routes. However, due to its complex spatial-temporal correlation has been a challenging task, and even though current research has progressed to some extent, it still generally focuses on modelling relationships between node pairs and between node history information, neglecting the analysis of node properties, leading to performance bottlenecks. To overcome these problems, we propose a dynamic spatial-temporal graph convolutional network (DSTGCN). Specifically, we design a dynamic graph generation module that collects information on geographical proximity and spatial heterogeneity between node pairs in advance and adaptively fuses the two types of information at each time step to generate a new dynamic graph. The dynamic graph module gives DSTGCN the ability to capture dynamic traffic information. In addition, we construct a graph convolution cycle module that captures local temporal dependencies on the basis of merging spatial relationships. It complements the dynamic graph module to jointly capture the spatial-temporal dependence of the traffic data. We validate the effectiveness of our model on two types of traffic prediction tasks, with DSTGCN outperforming the majority of baseline models.

Spatial-temporal graph convolutional network for Alzheimer classification based on brain functional connectivity imaging of electroencephalogram.

Functional connectivity of the human brain, representing statistical dependence of information flow between cortical regions, significantly contributes to the study of the intrinsic brain network and its functional mechanism. To fully explore its potential in the early diagnosis of Alzheimer's disease (AD) using electroencephalogram (EEG) recordings, this article introduces a novel dynamical spatial-temporal graph convolutional neural network (ST-GCN) for better classification performance. Different from existing studies that are based on either topological brain function characteristics or temporal features of EEG, the proposed ST-GCN considers both the adjacency matrix of functional connectivity from multiple EEG channels and corresponding dynamics of signal EEG channel simultaneously. Different from the traditional graph convolutional neural networks, the proposed ST-GCN makes full use of the constrained spatial topology of functional connectivity and the discriminative dynamic temporal information represented by the 1D convolution. We conducted extensive experiments on the clinical EEG data set of AD patients and Healthy Controls. The results demonstrate that the proposed method achieves better classification performance (92.3%) than the state-of-the-art methods. This approach can not only help diagnose AD but also better understand the effect of normal ageing on brain network characteristics before we can accurately diagnose the condition based on resting-state EEG.

Multi-actor activity detection by modeling object relationships in extended videos based on deep learning

We present a Multi-actor Activity Detection Framework (MADF) to model the interactive relationship among multiple actors for activity detection in extended videos. MADF can detect 3 groups of multi-actor activities with different kinds of actors, which involves three stages: detection, classification and post-processing. In the detection stage, both interaction proposals and actor proposals are generated in each video clip, in order to eliminate irrelevant background in the scene. In the classification stage, 3 different classification networks are proposed to classify the 3 groups of activities. And further, for person–object interaction, an attention mechanism is adopted to help the person–object classification network to pay more attention to the small-scale objects; for person–person interaction, a suppression module is used to improve the accuracy of the person–person activity detection; for person–vehicle interaction, a spatial–temporal graph convolution network (GCN) module is embedded to model the fine-grained relationship between the person and vehicle in the person–vehicle classification network, with a proposed Mutually Exclusive Category Loss (MECLoss) helping this network distinguish mutually exclusive activities. At last, we use the off-the-shelf post-processing methods to re-score the proposals for more stable results. The proposed system achieves a great progress on our baseline and achieves the state-of-the-art results in TRECVID 2021 ActEV challenge.

An improved spatial temporal graph convolutional network for robust skeleton-based action recognition

An improved spatial temporal graph convolutional network for robust skeleton-based action recognition

SAST-GCN: Segmentation Adaptive Spatial Temporal-Graph Convolutional Network for P3-Based Video Target Detection.

Detecting video-induced P3 is crucial to building the video target detection system based on the brain-computer interface. However, studies have shown that the brain response patterns corresponding to video-induced P3 are dynamic and determined by the interaction of multiple brain regions. This paper proposes a segmentation adaptive spatial-temporal graph convolutional network (SAST-GCN) for P3-based video target detection. To make full use of the dynamic characteristics of the P3 signal data, the data is segmented according to the processing stages of the video-induced P3, and the brain network connections are constructed correspondingly. Then, the spatial-temporal feature of EEG data is extracted by adaptive spatial-temporal graph convolution to discriminate the target and non-target in the video. Especially, a style-based recalibration module is added to select feature maps with higher contributions and increase the feature extraction ability of the network. The experimental results demonstrate the superiority of our proposed model over the baseline methods. Also, the ablation experiments indicate that the segmentation of data to construct the brain connection can effectively improve the recognition performance by reflecting the dynamic connection relationship between EEG channels more accurately.

Multi‐branch angle aware spatial temporal graph convolutional neural network for model‐based gait recognition

AbstractModel‐based gait recognition with skeleton data input has attracted more attention in recent years. The model‐based gait recognition methods take skeletons constructed by body joints as input, which are invariant to changing carrying and clothing conditions. However, previous methods limitedly model the skeleton information in either spatial or temporal domains and ignore the pose variety under different view angles, which results in poor performance for gait recognition. To solve the above problems, we propose the Multi‐Branch Angle Aware Spatial Temporal Graph Convolutional Neural Network to better depict the spatial‐temporal relationship while minimising the interference from the view angles. The model adopts the legacy Spatial Temporal Graph Neural Network (ST‐GCN) as its backbone and relocates it to create independent ST‐GCN branches. The novel Angle Estimator module is designed to predict the skeletons' view angles, which enables the network robust to the changing views. To balance the weights of different body parts and sequence frames, we build a Part‐Frame‐Importance module to redistribute them. Our experiments on the challenging CASIA‐B dataset have proved the efficacy of the proposed method, which achieves state‐of‐the‐art performance under different carrying and clothing conditions.

Prediction of Spread Trend of Epidemic Based on Spatial-Temporal Sequence

Coronavirus Disease 2019 (COVID-19) continues to spread throughout the world, and it is necessary for us to implement effective methods to prevent and control the spread of the epidemic. In this paper, we propose a new model called Spatial–Temporal Attention Graph Convolutional Networks (STAGCN) that can analyze the long-term trend of the COVID-19 epidemic with high accuracy. The STAGCN employs a spatial graph attention network layer and a temporal gated attention convolutional network layer to capture the spatial and temporal features of infectious disease data, respectively. While the new model inherits the symmetric “space-time space” structure of Spatial–Temporal Graph Convolutional Networks (STGCN), it enhances its ability to identify infectious diseases using spatial–temporal correlation features by replacing the graph convolutional network layer with a graph attention network layer that can pay more attention to important features based on adaptively adjusted feature weights at different time points. The experimental results show that our model has the lowest error rate compared to other models. The paper also analyzes the prediction results of the model using interpretable analysis methods to provide a more reliable guide for the decision-making process during epidemic prevention and control.

Automated freezing of gait assessment with marker-based motion capture and multi-stage spatial-temporal graph convolutional neural networks

BackgroundFreezing of gait (FOG) is a common and debilitating gait impairment in Parkinson’s disease. Further insight into this phenomenon is hampered by the difficulty to objectively assess FOG. To meet this clinical need, this paper proposes an automated motion-capture-based FOG assessment method driven by a novel deep neural network.MethodsAutomated FOG assessment can be formulated as an action segmentation problem, where temporal models are tasked to recognize and temporally localize the FOG segments in untrimmed motion capture trials. This paper takes a closer look at the performance of state-of-the-art action segmentation models when tasked to automatically assess FOG. Furthermore, a novel deep neural network architecture is proposed that aims to better capture the spatial and temporal dependencies than the state-of-the-art baselines. The proposed network, termed multi-stage spatial-temporal graph convolutional network (MS-GCN), combines the spatial-temporal graph convolutional network (ST-GCN) and the multi-stage temporal convolutional network (MS-TCN). The ST-GCN captures the hierarchical spatial-temporal motion among the joints inherent to motion capture, while the multi-stage component reduces over-segmentation errors by refining the predictions over multiple stages. The proposed model was validated on a dataset of fourteen freezers, fourteen non-freezers, and fourteen healthy control subjects.ResultsThe experiments indicate that the proposed model outperforms four state-of-the-art baselines. Moreover, FOG outcomes derived from MS-GCN predictions had an excellent (r = 0.93 [0.87, 0.97]) and moderately strong (r = 0.75 [0.55, 0.87]) linear relationship with FOG outcomes derived from manual annotations.ConclusionsThe proposed MS-GCN may provide an automated and objective alternative to labor-intensive clinician-based FOG assessment. Future work is now possible that aims to assess the generalization of MS-GCN to a larger and more varied verification cohort.

Expressway Speed Prediction Based on Electronic Toll Collection Data

Expressway section speed can visually reflect the section operation condition, and accurate short time section speed prediction has a wide range of applications in path planning and traffic guidance. However, existing expressway speed prediction data have defects, such as sparse density and incomplete object challenges. Thus, this paper proposes a framework for a combined expressway traffic speed prediction model based on wavelet transform and spatial-temporal graph convolutional network (WSTGCN) of the Electronic Toll Collection (ETC) gantry transaction data. First, the framework pre-processes the ETC gantry transaction data to construct the section speeds. Then wavelet decomposition and single-branch reconstruction are performed on the section speed sequences, and the spatial features are captured by graph convolutional network (GCN) for each reconstructed single-branch sequence, and the temporal features are extracted by connecting the gated recurrent unit (GRU). The experiments use the ETC gantry transaction data of the expressway from Quanzhou to Xiamen. The results indicate that the WSTGCN model makes notable improvements compared to the model of the baseline for different prediction ranges.

Attention-based dynamic spatial-temporal graph convolutional networks for traffic speed forecasting

In recent years, spatial–temporal graph modeling based on graph convolutional neural networks (GCN) has become an effective method for mining spatial–temporal dependencies in traffic forecasting research. However, existing studies lack the capability of dynamic spatial–temporal modeling of traffic speeds. Furthermore, long-term forecasting is difficult because of the diversity of traffic conditions. In addition, traditional studies capture only the features of fixed graph structures, which do not reflect real spatial dependence. To address these challenges, this study proposes a novel attention-based dynamic spatial–temporal graph convolutional network (ADSTGCN) model. ADSTGCN mainly consists of multiple dynamic spatial–temporal blocks, each of which contains three modules: 1) a dynamic adjustment module to model the dynamic spatial–temporal correlations of traffic speed, 2) a gated dilated convolution module to mine long-term dependencies, and 3) a spatial convolution module to capture hidden spatial dependencies. Experiments on three public traffic datasets demonstrated the good performance of the model.

Improved GCN Framework for Human Motion Recognition

Human recognition models based on spatial-temporal graph convolutional neural networks have been gradually developed, and we present an improved spatial-temporal graph convolutional neural network to solve the problems of the high number of parameters and low accuracy of this type of model. The method mainly draws on the inception structure. First, the tensor rotation is added to the graph convolution layer to realize the conversion between graph node dimension and channel dimension and enhance the model’s ability to capture global information for small-scale tasks. Then the inception temporal convolution layer is added to build a multiscale temporal convolution filter to perceive temporal information under different time domains hierarchically from 4-time dimensions. It overcomes the shortcomings of temporal graph convolutional networks in the field of joint relevance of hidden layers and compensates for the information omission of small-scale graph tasks. It also limits the volume of parameters, decreases the arithmetic power, and speeds up the computation. In our experiments, we verify our model on the public dataset NTU RGB + D. Our method reduces the number of the model parameters by 50% and achieves an accuracy of 90% in the CS evaluation system and 94% in the CV evaluation system. The results show that our method not only has high recognition accuracy and good robustness in human behavior recognition applications but also has a small number of model parameters, which can effectively reduce the computational cost.

Estimating Parkinsonism Severity in Natural Gait Videos of Older Adults With Dementia.

Drug-induced parkinsonism affects many older adults with dementia, often causing gait disturbances. New advances in vision-based human pose- estimation have opened possibilities for frequent and unobtrusive analysis of gait in long-term care settings. This work leverages spatial-temporal graph convolutional network (ST-GCN) architectures and training procedures to predict clinical scores of parkinsonism in gait from video of individuals with dementia. We propose a two-stage training approach consisting of a self-supervised pretraining stage that encourages the ST-GCN model to learn about gait patterns before predicting clinical scores in the finetuning stage. The proposed ST-GCN models are evaluated on joint trajectories extracted from video and are compared against traditional (ordinal, linear, random forest) regression models and temporal convolutional network baselines. Three 2D human pose-estimation libraries (OpenPose, Detectron, AlphaPose) and the Microsoft Kinect (2D and 3D) are used to extract joint trajectories of 4787 natural walking bouts from 53 older adults with dementia. A subset of 399 walks from 14 participants is annotated with scores of parkinsonism severity on the gait criteria of the Unified Parkinson's Disease Rating Scale (UPDRS) and the Simpson-Angus Scale (SAS). Our results demonstrate that ST-GCN models operating on 3D joint trajectories extracted from the Kinect consistently outperform all other models and feature sets. Prediction of parkinsonism scores in natural walking bouts of unseen participants remains a challenging task, with the best models achieving macro-averaged F1-scores of 0.53 ± 0.03 and 0.40 ± 0.02 for UPDRS-gait and SAS-gait, respectively. Pre-trained model and demo code for this work is available.1.