Spatial Temporal Graph Research Articles

An Energy Harvesting Roadside Unit communication load prediction and energy scheduling based on graph convolutional neural networks for spatial‐temporal vehicle data

The Energy Harvesting Roadside Unit (EH-RSU) with self-powered module will not only effectively reduce the communication load of regional Vehicular Ad Hoc Networks, but also enjoys a low deployment cost. Given the imbalance in communication demands invoked by transportation systems, the EH-RSU should allocate energy appropriately in accordance with its energy harvesting rate to ensure the communication safety of vehicles within its coverage. Firstly, we propose a novel attention-based spatial-temporal graph convolutional network (ASTGCN) to predict the communication load around the EH-RSU in the road network through the surrounding vehicle information. Secondly, we use the predicted communication load as part of the input parameters to neural network and leverage a double deep Q network to ameliorate the operating states switching strategy of EH-RSUs by reinforcement learning so that they achieve a more satisfying effective time with limited resources. Finally, we built a dataset by simulation to validate the effectiveness of our model. The results show that our prediction model has a better accuracy and the improved strategy has higher efficiency compared with other methods.

Deep Reinforcement Learning Based on Social Spatial–Temporal Graph Convolution Network for Crowd Navigation

In the crowd navigation, reinforcement learning based on graph neural network is a promising method, which effectively solves the poor navigation effect based on social interaction model and the freezing behavior of robot in extreme cases. However, since the information correlation of human trajectory has not been involved in the method, its performance still needs improvement. Therefore, we proposed a deep reinforcement learning model based on Social Spatial–Temporal Graph Convolution Network (SSTGCN) to handle the crowd navigation problem, in which the spatial–temporal information of human trajectory has been taken advantage to predict human behavior intentions and help robot plan path more efficiently. The model consists of graph learning module and robot forward planning module. In the graph learning module, the latent features of agents are taken advantage to reason about the relations among the agents, and SSTGCN is used to update feature matrix. In addition, value estimation module calculates state representation and state prediction module predicts the next state. The robot forward planning module makes use of k-step planning to estimate the quality of state and searches the best k steps planning. We tested our model in the Crowd-Nav platform, and the results show that our model has high navigation success rate and short navigation time. In addition, it has good robustness to crowd changes.

MST-GAT: A multimodal spatial–temporal graph attention network for time series anomaly detection

Multimodal time series (MTS) anomaly detection is crucial for maintaining the safety and stability of working devices (e.g., water treatment system and spacecraft), whose data are characterized by multivariate time series with diverse modalities. Although recent deep learning methods show great potential in anomaly detection, they do not explicitly capture spatial–temporal relationships between univariate time series of different modalities, resulting in more false negatives and false positives. In this paper, we propose a multimodal spatial–temporal graph attention network (MST-GAT) to tackle this problem. MST-GAT first employs a multimodal graph attention network (M-GAT) and a temporal convolution network to capture the spatial–temporal correlation in multimodal time series. Specifically, M-GAT uses a multi-head attention module and two relational attention modules (i.e., intra- and inter-modal attention) to model modal correlations explicitly. Furthermore, MST-GAT optimizes the reconstruction and prediction modules simultaneously. Experimental results on four multimodal benchmarks demonstrate that MST-GAT outperforms the state-of-the-art baselines. Further analysis indicates that MST-GAT strengthens the interpretability of detected anomalies by locating the most anomalous univariate time series.

Enhancing the accuracy of a human emotion recognition method using spatial temporal graph convolutional networks

Enhancing the accuracy of a human emotion recognition method using spatial temporal graph convolutional networks

Global Correlation Enhanced Hand Action Recognition Based on NST-GCN

Hand action recognition is an important part of intelligent monitoring, human–computer interaction, robotics and other fields. Compared with other methods, the hand action recognition method using skeleton information can ignore the error effects caused by complex background and movement speed changes, and the computational cost is relatively small. The spatial-temporal graph convolution networks (ST-GCN) model has excellent performance in the field of skeleton-based action recognition. In order to solve the problem of the root joint and the further joint not being closely connected, resulting in a poor hand-action-recognition effect, this paper firstly uses the dilated convolution to replace the standard convolution in the temporal dimension. This is in order to process the time series features of the hand action video, which increases the receptive field in the temporal dimension and enhances the connection between features. Then, by adding non-physical connections, the connection between the joints of the fingertip and the root of the finger is established, and a new partition strategy is adopted to strengthen the hand correlation of each joint point information. This helps to improve the network’s ability to extract the spatial-temporal features of the hand. The improved model is tested on public datasets and real scenarios. The experimental results show that compared with the original model, the 14-category top-1 and 28-category top-1 evaluation indicators of the dataset have been improved by 4.82% and 6.96%. In the real scene, the recognition effect of the categories with large changes in hand movements is better, and the recognition results of the categories with similar trends of hand movements are poor, so there is still room for improvement.

A spatial temporal graph neural network model for predicting flashover in arbitrary building floorplans

Rapid fire progression, such as flashover, has been one of the leading causes for firefighter deaths and injuries in residential building environments. Due to long computational time of and the required prior knowledge about the fire scene, existing models cannot be used to predict the potential occurrence of flashover in practical firefighting applications. In this paper, a scene-agnostic model (FlashNet) is proposed to predict flashover based on limited heat detector temperature information up to 150 °C. FlashNet utilizes spatial temporal graph convolutional neural networks to effectively learn features from the limited temperature information and to tackle building structure variations. The proposed model is benchmarked against five different state-of-the-art flashover prediction models. Results show that FlashNet outperforms the existing flashover prediction models and it can reliably predict flashover 30 s preceding its occurrence with an overall accuracy of about 92.1%. Ablation study is carried out to examine the effectiveness of different key model components and geometric average adjacency matrix. The research outcomes from this study are expected to enhance firefighters’ situational awareness in the fire scene, protecting them from hazardous fire environments and to pave the way for the development of data-driven prediction systems.

Multivariate time-series classification with hierarchical variational graph pooling

In recent years, multivariate time-series classification (MTSC) has attracted considerable attention owing to the advancement of sensing technology. Existing deep-learning-based MTSC techniques, which mostly rely on convolutional or recurrent neural networks, focus primarily on the temporal dependency of a single time series. Based on this, complex pairwise dependencies among multivariate variables can be better described using advanced graph methods, where each variable is regarded as a node in the graph, and their dependencies are regarded as edges. Furthermore, current spatial–temporal modeling (e.g., graph classification) methodologies based on graph neural networks (GNNs) are inherently flat and cannot hierarchically aggregate node information. To address these limitations, we propose a novel graph-pooling-based framework, MTPool, to obtain an expressive global representation of MTS. We first convert MTS slices into graphs using the interactions of variables via a graph structure learning module and obtain the spatial–temporal graph node features via a temporal convolutional module. To obtain global graph-level representation, we design an “encoder-decoder”-based variational graph pooling module to create adaptive centroids for cluster assignments. Then, we combine GNNs and our proposed variational graph pooling layers for joint graph representation learning and graph coarsening, after which the graph is progressively coarsened to one node. Finally, a differentiable classifier uses this coarsened representation to obtain the final predicted class. Experiments on ten benchmark datasets showed that MTPool outperforms state-of-the-art strategies in the MTSC task.

A protein complex recognition method based on spatial-temporal graph convolution neural network

To propose a new method for mining complexes in dynamic protein network using spatiotemporal convolution neural network. The edge strength, node strength and edge existence probability are defined for modeling of the dynamic protein network. Based on the time series information and structure information on the graph, two convolution operators were designed using Hilbert-Huang transform, attention mechanism and residual connection technology to represent and learn the characteristics of the proteins in the network, and the dynamic protein network characteristic map was constructed. Finally, spectral clustering was used to identify the protein complexes. The simulation results on several public biological datasets showed that the F value of the proposed algorithm exceeded 90% on DIP dataset and MIPS dataset. Compared with 4 other recognition algorithms (DPCMNE, GE-CFI, VGAE and NOCD), the proposed algorithm improved the recognition efficiency by 34.5%, 28.7%, 25.4% and 17.6%, respectively. The application of deep learning technology can improve the efficiency in analysis of dynamic protein networks.

Subgraph-aware graph structure revision for spatial–temporal graph modeling

Spatial–temporal graph modeling has been widely studied in many fields, such as traffic forecasting and energy analysis, where data has time and space properties. Existing methods focus on capturing stable and dynamic spatial correlations by constructing physical and virtual graphs along with graph convolution and temporal modeling. However, existing methods tending to smooth node features may obscure the spatial–temporal patterns among nodes. Worse, the graph structure is not always available in some fields, while the manually constructed stable or dynamic graphs cannot necessarily reflect the true spatial correlations either. This paper proposes a Subgraph-Aware Graph Structure Revision network (SAGSR) to overcome these limitations. Architecturally, a subgraph-aware structure revision graph convolution module (SASR-GCM) is designed, which revises the learned stable graph to obtain a dynamic one to automatically infer the dynamics of spatial correlations. Each of these two graphs is separated into one homophilic subgraph and one heterophilic subgraph by a subgraph-aware graph convolution mechanism, which aggregates similar nodes in the homophilic subgraph with positive weights, while keeping nodes with dissimilar features in the heterophilic subgraph mutually away with negative aggregation weights to avoid pattern obfuscation. By combining a gated multi-scale temporal convolution module (GMS-TCM) for temporal modeling, SAGSR can efficiently capture the spatial–temporal correlations and extract complex spatial–temporal graph features. Extensive experiments, conducted on two specific tasks: traffic flow forecasting and energy consumption forecasting, indicate the effectiveness and superiority of our proposed approach over several competitive baselines.

Multi-View Spatial–Temporal Graph Neural Network for Traffic Prediction

Abstract Spatial–temporal graph neural network has drawn more and more attention in recent years and is widely used to various real-world applications. However, learning the spatial–temporal graph neural network structure presents unique challenges including: (i) the dynamic spatial correlation; (ii) the dynamic temporal correlation. Even the existing methods take into account the spatial correlation, they still learn the static road network structure information, which cannot reflect the dynamic of road relations. Some of the works has focused on modeling the long-term time series, but the improvements have been limited tightly. To overcome these challenges, we proposed a novel approach called Multi-View Spatial–Temporal Graph Neural Network. Differ from the existing research, we designed a multi-view temporal transformer module to extract dynamic temporal correlation and enhance the expression of medium and long-term temporal features. We propose a multi-view spatial structure and a corresponding multi-view graph convolutional module, which are capable of simultaneously combining the features of static road network structure and dynamic changes. Compared with 11 baselines, our proposed model has achieved significant improvement in the accuracy of prediction.

A Novel Early Warning Model for Hand, Foot and Mouth Disease Prediction Based on a Graph Convolutional Network

ObjectivesHand, foot and mouth disease (HFMD) is a widespread infectious disease that causes a significant disease burden on society. To achieve early intervention and to prevent outbreaks of disease, we propose a novel warning model that can accurately predict the incidence of HFMD. MethodsWe propose a spatial-temporal graph convolutional network (STGCN) that combines spatial factors for surrounding cities with historical incidence over a certain time period to predict the future occurrence of HFMD in Guangdong and Shandong between 2011 and 2019. The 2011–2018 data served as the training and verification set, while data from 2019 served as the prediction set. Six important parameters were selected and verified in this model and the deviation was displayed by the root mean square error and the mean absolute error. ResultsAs the first application using a STGCN for disease forecasting, we succeeded in accurately predicting the incidence of HFMD over a 12-week period at the prefecture level, especially for cities of significant concern. ConclusionsThis model provides a novel approach for infectious disease prediction and may help health administrative departments implement effective control measures up to 3 months in advance, which may significantly reduce the morbidity associated with HFMD in the future.

AFSTGCN: Prediction for multivariate time series using an adaptive fused spatial-temporal graph convolutional network

The prediction for Multivariate Time Series (MTS) explores the interrelationships among variables at historical moments, extracts their relevant characteristics, and is widely used in finance, weather, complex industries and other fields. Furthermore, it is important to construct a digital twin system. However, existing methods do not take full advantage of the potential properties of variables, which results in poor predicted accuracy. In this paper, we propose the Adaptive Fused Spatial-Temporal Graph Convolutional Network (AFSTGCN). First, to address the problem of the unknown spatial-temporal structure, we construct the Adaptive Fused Spatial-Temporal Graph (AFSTG) layer. Specifically, we fuse the spatial-temporal graph based on the interrelationship of spatial graphs. Simultaneously, we construct the adaptive adjacency matrix of the spatial-temporal graph using node embedding methods. Subsequently, to overcome the insufficient extraction of disordered correlation features, we construct the Adaptive Fused Spatial-Temporal Graph Convolutional (AFSTGC) module. The module forces the reordering of disordered temporal, spatial and spatial-temporal dependencies into rule-like data. AFSTGCN dynamically and synchronously acquires potential temporal, spatial and spatial-temporal correlations, thereby fully extracting rich hierarchical feature information to enhance the predicted accuracy. Experiments on different types of MTS datasets demonstrate that the model achieves state-of-the-art single-step and multi-step performance compared with eight other deep learning models.

Decoupled dynamic spatial-temporal graph neural network for traffic forecasting

We all depend on mobility, and vehicular transportation affects the daily lives of most of us. Thus, the ability to forecast the state of traffic in a road network is an important functionality and a challenging task. Traffic data is often obtained from sensors deployed in a road network. Recent proposals on spatial-temporal graph neural networks have achieved great progress at modeling complex spatial-temporal correlations in traffic data, by modeling traffic data as a diffusion process. However, intuitively, traffic data encompasses two different kinds of hidden time series signals, namely the diffusion signals and inherent signals. Unfortunately, nearly all previous works coarsely consider traffic signals entirely as the outcome of the diffusion, while neglecting the inherent signals, which impacts model performance negatively. To improve modeling performance, we propose a novel Decoupled Spatial-Temporal Framework (DSTF) that separates the diffusion and inherent traffic information in a data-driven manner, which encompasses a unique estimation gate and a residual decomposition mechanism. The separated signals can be handled subsequently by the diffusion and inherent modules separately. Further, we propose an instantiation of DSTF, Decoupled Dynamic Spatial-Temporal Graph Neural Network (D 2 STGNN), that captures spatial-temporal correlations and also features a dynamic graph learning module that targets the learning of the dynamic characteristics of traffic networks. Extensive experiments with four real-world traffic datasets demonstrate that the framework is capable of advancing the state-of-the-art.

DSTGCN: Dynamic Spatial-Temporal Graph Convolutional Network for Traffic Prediction

Traffic prediction is an important part of building a smart city. Reasonable traffic prediction can help the relevant departments to make important decisions and help people to plan their travel routes. However, due to its complex spatial-temporal correlation has been a challenging task, and even though current research has progressed to some extent, it still generally focuses on modelling relationships between node pairs and between node history information, neglecting the analysis of node properties, leading to performance bottlenecks. To overcome these problems, we propose a dynamic spatial-temporal graph convolutional network (DSTGCN). Specifically, we design a dynamic graph generation module that collects information on geographical proximity and spatial heterogeneity between node pairs in advance and adaptively fuses the two types of information at each time step to generate a new dynamic graph. The dynamic graph module gives DSTGCN the ability to capture dynamic traffic information. In addition, we construct a graph convolution cycle module that captures local temporal dependencies on the basis of merging spatial relationships. It complements the dynamic graph module to jointly capture the spatial-temporal dependence of the traffic data. We validate the effectiveness of our model on two types of traffic prediction tasks, with DSTGCN outperforming the majority of baseline models.

A gated graph convolutional network with multi-sensor signals for remaining useful life prediction

With the advent of industry 4.0, multi-sensors are utilized to monitor the degradation process of machinery. When machinery operating, multi-sensor signals have potential relation with each other. However, existing deep-learning-based prognosis models are often limited by lacks of (1) considering temporal and spatial dependencies in multi-sensor signals collected from the dynamic and complex machinery system; (2) uncertainty quantification of remaining useful life (RUL) which plays an essential role in maintenance schedules and spare parts management. Therefore, how to fuse multi-sensor signals and manage uncertainty are two major concerns in deep-learning-based prognosis approaches with multi-sensor signals. To tackle these challenges, a gated graph convolutional network (GGCN) is developed for multi-sensor signal fusion and RUL prediction. Firstly, spatial–temporal graphs are constructed from multi-sensor signals as input of the prognosis model. Next, gated graph convolutional layers are built to accurately extract degradation features by simultaneously modeling the temporal and spatial dependencies in multi-sensor signals. Finally, the extracted features are fed into a quantile regression layer to estimate the RUL and its confidence interval. Experimental results on a simulated graph dataset, a bearing dataset from real wind farm, a turbofan engine dataset and a tool wear dataset validate the effectiveness of the proposed GGCN-based prognosis framework.

Keypoint Message Passing for Video-Based Person Re-identification

Video-based person re-identification~(re-ID) is an important technique in visual surveillance systems which aims to match video snippets of people captured by different cameras. Existing methods are mostly based on convolutional neural networks~(CNNs), whose building blocks either process local neighbor pixels at a time, or, when 3D convolutions are used to model temporal information, suffer from the misalignment problem caused by person movement. In this paper, we propose to overcome the limitations of normal convolutions with a human-oriented graph method. Specifically, features located at person joint keypoints are extracted and connected as a spatial-temporal graph. These keypoint features are then updated by message passing from their connected nodes with a graph convolutional network~(GCN). During training, the GCN can be attached to any CNN-based person re-ID model to assist representation learning on feature maps, whilst it can be dropped after training for better inference speed. Our method brings significant improvements over the CNN-based baseline model on the MARS dataset with generated person keypoints and a newly annotated dataset: PoseTrackReID. It also defines a new state-of-the-art method in terms of top-1 accuracy and mean average precision in comparison to prior works.

Spatial-temporal graph convolutional network for Alzheimer classification based on brain functional connectivity imaging of electroencephalogram.

Functional connectivity of the human brain, representing statistical dependence of information flow between cortical regions, significantly contributes to the study of the intrinsic brain network and its functional mechanism. To fully explore its potential in the early diagnosis of Alzheimer's disease (AD) using electroencephalogram (EEG) recordings, this article introduces a novel dynamical spatial-temporal graph convolutional neural network (ST-GCN) for better classification performance. Different from existing studies that are based on either topological brain function characteristics or temporal features of EEG, the proposed ST-GCN considers both the adjacency matrix of functional connectivity from multiple EEG channels and corresponding dynamics of signal EEG channel simultaneously. Different from the traditional graph convolutional neural networks, the proposed ST-GCN makes full use of the constrained spatial topology of functional connectivity and the discriminative dynamic temporal information represented by the 1D convolution. We conducted extensive experiments on the clinical EEG data set of AD patients and Healthy Controls. The results demonstrate that the proposed method achieves better classification performance (92.3%) than the state-of-the-art methods. This approach can not only help diagnose AD but also better understand the effect of normal ageing on brain network characteristics before we can accurately diagnose the condition based on resting-state EEG.

Multi-actor activity detection by modeling object relationships in extended videos based on deep learning

We present a Multi-actor Activity Detection Framework (MADF) to model the interactive relationship among multiple actors for activity detection in extended videos. MADF can detect 3 groups of multi-actor activities with different kinds of actors, which involves three stages: detection, classification and post-processing. In the detection stage, both interaction proposals and actor proposals are generated in each video clip, in order to eliminate irrelevant background in the scene. In the classification stage, 3 different classification networks are proposed to classify the 3 groups of activities. And further, for person–object interaction, an attention mechanism is adopted to help the person–object classification network to pay more attention to the small-scale objects; for person–person interaction, a suppression module is used to improve the accuracy of the person–person activity detection; for person–vehicle interaction, a spatial–temporal graph convolution network (GCN) module is embedded to model the fine-grained relationship between the person and vehicle in the person–vehicle classification network, with a proposed Mutually Exclusive Category Loss (MECLoss) helping this network distinguish mutually exclusive activities. At last, we use the off-the-shelf post-processing methods to re-score the proposals for more stable results. The proposed system achieves a great progress on our baseline and achieves the state-of-the-art results in TRECVID 2021 ActEV challenge.

Automatic Traffic Anomaly Detection on the Road Network with Spatial-Temporal Graph Neural Network Representation Learning

Traffic anomaly detection is an essential part of an intelligent transportation system. Automatic traffic anomaly detection can provide sufficient decision-support information for road network operators, travelers, and other stakeholders. This research proposes a novel automatic traffic anomaly detection method based on spatial-temporal graph neural network representation learning. We divide traffic anomaly detection into two steps: first is learning the implicit graph feature representation of multivariate time series of traffic flows based on a graph attention model to predict the traffic states. Second, traffic anomalies are detected using graph deviation score calculation to compare the deviation of predicted traffic states with the observed traffic states. Experiments on real network datasets show that with an end-to-end workflow and spatial-temporal representation of traffic states, this method can detect traffic anomalies accurately and automatically and achieves better performance over baselines.

Spatial–Temporal Graph Data Mining for IoT-Enabled Air Mobility Prediction

Big data analytics and mining have the potential to enable real-time decision making and control in a range of Internet of Things (IoT) application domains, such as the Internet of Vehicles, the Internet of Wings, and the Airport of Things. The prediction toward air mobility, which is essential to the studies of air traffic management, has been a challenging task due to the complex spatial and temporal dependencies in air traffic data with highly nonlinear and variational patterns. Existing works for air traffic prediction only focus on either modeling static traffic patterns of individual flight or temporal correlation, with no or limited addressing of the spatial impact, namely, the propagation of traffic perturbation among airports. In this article, we propose to leverage the concept of graph and model the airports as nodes with time-series features and conduct data mining on graph-structured data. To be specific, first, airline on-time performance (AOTP) data is preprocessed to generate a temporal graph data set, which includes three features: 1) the number; 2) average delay; and 3) average taxiing time of departure and arrival flights. Then, a spatial–temporal graph neural networks model is implemented to forecast the mobility level at each airport over time, where a combination of graph convolution and time-dimensional convolution is used to capture the spatial and temporal correlation simultaneously. Experiments on the data set demonstrate the advantage of the model on spatial–temporal air mobility prediction, together with the impact of different priors on adjacency matrices and the effectiveness of the temporal attention mechanism. Finally, we analyze the prediction performance and discuss the capability of our model. The prediction framework proposed in this work has the potential to be generalized to other spatial–temporal tasks in IoT.