Spatial Temporal Graph Research Articles

Fault Diagnosis of Wheeled Robot Based on Prior Knowledge and Spatial-Temporal Difference Graph Convolutional Network

The critical issue of wheeled robot fault diagnosis is to comprehensively evaluate its health condition using multi-sensor data, but traditional deep learning-based methods are hard to model the relationships among sensor measurements. Unlike these methods, the graph convolutional network (GCN), which uses the graph-structured data along with the association graph as input, is more efficient for relationship modeling. However, existing GCN-based fault diagnosis methods suffer from the following weaknesses: 1) the association graphs are obtained according to the similarity of data samples or their features, which cannot guarantee accuracy; 2) these models are focused on spatial correlations and neglect temporal correlations. To address these problems, we propose to construct the association graph based on prior knowledge, i.e., a simplified mathematical model of the wheeled robot. Moreover, we develop a spatial-temporal difference graph convolutional network (STDGCN) for wheeled robot fault diagnosis. This network contains a difference layer that utilizes localized difference properties for feature enhancement, and the spatial-temporal graph convolutional modules are introduced to jointly capture the spatial-temporal correlations. To verify the effectiveness of STDGCN for fault diagnosis, experiments are carried out, and the results show that STDGCN achieves superior performance.

MVSTGN: A Multi-View Spatial-Temporal Graph Network for Cellular Traffic Prediction

Timely and accurate cellular traffic prediction is difficult to achieve due to the complex spatial-temporal characteristics of cellular traffic. The latest approaches mainly aim to model local spatial-temporal dependencies of cellular traffic based on deep learning techniques but lack the consideration of diverse global spatial-temporal correlations hidden in cellular traffic. To tackle this issue, we propose a novel multi-view spatial-temporal graph network (MVSTGN), which combines attention and convolution mechanisms into traffic pattern analysis, enabling the comprehensive excavation of spatial-temporal characteristics. Specifically, the MVSTGN realizes the above statement from three spatial-temporal views: 1) From a global spatial view, two spatial attention modules are proposed to capture the global spatial correlations between different regions at node and trend levels; 2) From a global temporal view, a temporal attention module is employed to capture and encode global temporal correlations between traffic at different times; 3) From a local spatial-temporal view, a dense convolution module is developed to further excavate the local spatial-temporal dependencies in cellular traffic. Consequently, a successful cellular traffic prediction strategy is constructed to fully explore the spatial-temporal characteristics from multiple views. The experimental results on a popular real-world cellular traffic dataset demonstrate that the MVSTGN achieves obvious improvements over baselines.

A spatial–temporal graph deep learning model for urban flood nowcasting leveraging heterogeneous community features

Flood nowcasting refers to near-future prediction of flood status as an extreme weather event unfolds to enhance situational awareness. The objective of this study was to adopt and test a novel structured deep-learning model for urban flood nowcasting by integrating physics-based and human-sensed features. We present a new computational modeling framework including an attention-based spatial–temporal graph convolution network (ASTGCN) model and different streams of data that are collected in real-time, preprocessed, and fed into the model to consider spatial and temporal information and dependencies that improve flood nowcasting. The novelty of the computational modeling framework is threefold: first, the model is capable of considering spatial and temporal dependencies in inundation propagation thanks to the spatial and temporal graph convolutional modules; second, it enables capturing the influence of heterogeneous temporal data streams that can signal flooding status, including physics-based features (e.g., rainfall intensity and water elevation) and human-sensed data (e.g., residents’ flood reports and fluctuations of human activity) on flood nowcasting. Third, its attention mechanism enables the model to direct its focus to the most influential features that vary dynamically and influence the flood nowcasting. We show the application of the modeling framework in the context of Harris County, Texas, as the study area and 2017 Hurricane Harvey as the flood event. Three categories of features are used for nowcasting the extent of flood inundation in different census tracts: (i) static features that capture spatial characteristics of various locations and influence their flood status similarity, (ii) physics-based dynamic features that capture changes in hydrodynamic variables, and (iii) heterogeneous human-sensed dynamic features that capture various aspects of residents’ activities that can provide information regarding flood status. Results indicate that the ASTGCN model provides superior performance for nowcasting of urban flood inundation at the census-tract level, with precision 0.808 and recall 0.891, which shows the model performs better compared with other state-of-the-art models. Moreover, ASTGCN model performance improves when heterogeneous dynamic features are added into the model that solely relies on physics-based features, which demonstrates the promise of using heterogenous human-sensed data for flood nowcasting. Given the results of the comparisons of the models, the proposed modeling framework has the potential to be more investigated when more data of historical events are available in order to develop a predictive tool to provide community responders with an enhanced prediction of the flood inundation during urban flood.

Learning spatial-temporal feature with graph product

Earlier works on dynamic spatial-temporal data modelling prefer using spatial-temporal factorized graph convolutional networks (GCNs), which are easy to interpret but fail to capture joint spatial-temporal correlations. Thus, lots of subsequent research focus on constructing a localized adjacent matrix to capture joint features from both spatial and temporal dimension simultaneously. However, their ways of building the adjacent matrices are usually heuristic, which makes the models difficult to interpret. Meanwhile, the lack of theoretical explanations hinders the model’s generalization. We introduce a general framework to model dynamic spatial-temporal graph data from the view of graph product. With the power of graph product, we propose a systematical way of constructing the spatial-temporal adjacent graphs, which can not only improve the model’s interpretability but increase the spatial-temporal receptive field. Under the novel framework, the existing methods can be taken as special cases of our model. Extensive experiments on multiple large-scale real-world datasets, NTU-RGB+D60, NTU-RGB+D120, UAV-Human, PEMS03, PEMS04, PEMS07, and PEMS08, demonstrate that the proposed model can generalize to most of the scenarios with a performance improvement in a significant margin compared to the state-of-the-art methods.

STGATE: Spatial-temporal graph attention network with a transformer encoder for EEG-based emotion recognition.

Electroencephalogram (EEG) is a crucial and widely utilized technique in neuroscience research. In this paper, we introduce a novel graph neural network called the spatial-temporal graph attention network with a transformer encoder (STGATE) to learn graph representations of emotion EEG signals and improve emotion recognition performance. In STGATE, a transformer-encoder is applied for capturing time-frequency features which are fed into a spatial-temporal graph attention for emotion classification. Using a dynamic adjacency matrix, the proposed STGATE adaptively learns intrinsic connections between different EEG channels. To evaluate the cross-subject emotion recognition performance, leave-one-subject-out experiments are carried out on three public emotion recognition datasets, i.e., SEED, SEED-IV, and DREAMER. The proposed STGATE model achieved a state-of-the-art EEG-based emotion recognition performance accuracy of 90.37% in SEED, 76.43% in SEED-IV, and 76.35% in DREAMER dataset, respectively. The experiments demonstrated the effectiveness of the proposed STGATE model for cross-subject EEG emotion recognition and its potential for graph-based neuroscience research.

Auto-STGCN: Autonomous Spatial-Temporal Graph Convolutional Network Search

In recent years, many spatial-temporal graph convolutional network (STGCN) models are proposed to deal with the spatial-temporal network data forecasting problem. These STGCN models have their own advantages, i.e., each of them puts forward many effective operations and achieves good prediction results in the real applications. If users can effectively utilize and combine these excellent operations integrating the advantages of existing models, then they may obtain more effective STGCN models thus create greater value using existing work. However, they fail to do so due to the lack of domain knowledge, and there is lack of automated system to help users to achieve this goal. In this article, we fill this gap and propose Auto-STGCN algorithm, which makes use of existing models to automatically explore high-performance STGCN model for specific scenarios. Specifically, we design Unified-STGCN framework, which summarizes the operations of existing architectures, and use parameters to control the usage and characteristic attributes of each operation, so as to realize the parameterized representation of the STGCN architecture and the reorganization and fusion of advantages. Then, we present Auto-STGCN, an optimization method based on reinforcement learning, to quickly search the parameter search space provided by Unified-STGCN, and generate optimal STGCN models automatically. Extensive experiments on real-world benchmark datasets show that our Auto-STGCN can find STGCN models superior to existing STGCN models used for search space construction, which demonstrates the effectiveness of our proposed method.

Multi-cue temporal modeling for skeleton-based sign language recognition.

Sign languages are visual languages used as the primary communication medium for the Deaf community. The signs comprise manual and non-manual articulators such as hand shapes, upper body movement, and facial expressions. Sign Language Recognition (SLR) aims to learn spatial and temporal representations from the videos of the signs. Most SLR studies focus on manual features often extracted from the shape of the dominant hand or the entire frame. However, facial expressions combined with hand and body gestures may also play a significant role in discriminating the context represented in the sign videos. In this study, we propose an isolated SLR framework based on Spatial-Temporal Graph Convolutional Networks (ST-GCNs) and Multi-Cue Long Short-Term Memorys (MC-LSTMs) to exploit multi-articulatory (e.g., body, hands, and face) information for recognizing sign glosses. We train an ST-GCN model for learning representations from the upper body and hands. Meanwhile, spatial embeddings of hand shape and facial expression cues are extracted from Convolutional Neural Networks (CNNs) pre-trained on large-scale hand and facial expression datasets. Thus, the proposed framework coupling ST-GCNs with MC-LSTMs for multi-articulatory temporal modeling can provide insights into the contribution of each visual Sign Language (SL) cue to recognition performance. To evaluate the proposed framework, we conducted extensive analyzes on two Turkish SL benchmark datasets with different linguistic properties, BosphorusSign22k and AUTSL. While we obtained comparable recognition performance with the skeleton-based state-of-the-art, we observe that incorporating multiple visual SL cues improves the recognition performance, especially in certain sign classes where multi-cue information is vital. The code is available at: https://github.com/ogulcanozdemir/multicue-slr.

Large-Scale Network Imputation and Prediction of Traffic Volume Based on Multi-Source Data Collection System

Although newly developed traffic detectors were actively deployed to improve the accuracy and coverage of collecting city-wise traffic state information, the rapid transition of the traffic management system caused the problems of massive data corruption. For the practical application of recovering the missing values, the deep learning-based imputation technique is used, which relies on prediction performance with the consideration of dynamic spatial and temporal characteristics in the traffic state information. However, the existing method requires an assumption that the given data comprise a complete dataset from a single source based on the experiments evaluated on a small scale or long stream of freeways. In this paper, we propose a multi-variable spatio-temporal learning technique based on multi-source traffic state information, which was realized by adopting Attention-based Spatial–Temporal Graph Convolutional Networks (ASTGCN). The proposed imputation method cooperatively aggregates spatial and temporal correlation from two different types of detectors into an integrated framework, which allows us to predict missing volume regardless of the missing rate. Moreover, the study was conducted on a large-scale network that contains the entire road characteristics. Daejeon city has served as a case study to demonstrate the performance, and the results show that the mean absolute error of the proposed method is under 12 vehicles/5 min. Our work indicates that multi-source traffic state information can be utilized to impute city-wide missing traffic volume.

STJA-GCN: A Multi-Branch Spatial–Temporal Joint Attention Graph Convolutional Network for Abnormal Gait Recognition

Early recognition of abnormal gait enables physicians to determine a prompt rehabilitation plan for patients for the most effective treatment and care. The Kinect depth sensor can easily collect skeleton data describing the position of joints in the human body. However, the default human skeleton model of Kinect includes an excessive number of many joints, which limits the accuracy of the gait recognition methods and increases the computational resources required. In this study, we propose an optimized human skeleton model for the Kinect system and streamline the joints using a center-of-mass calculation. We integrate several techniques to propose an end-to-end, spatial–temporal, joint attention graph convolutional network (STJA-GCN) architecture. We conducted experiments with a fivefold cross-validation on two common datasets of information on abnormal gaits to evaluate the performance of the proposed method. The results show that the STJA-GCN achieved 93.17 and 92.08% accuracy on the two datasets, and compared to the original spatial–temporal graph convolutional network (ST-GCN), the recognition accuracy increases by 9.22 and 20.65%, respectively. Overall, the results demonstrate that the STJA-GCN can accurately recognize abnormal gaits and, thus, can support low-cost rehabilitation assessments at community hospitals or in patients’ homes.

Attention-guided spatial–temporal graph relation network for video-based person re-identification

Attention-guided spatial–temporal graph relation network for video-based person re-identification

HIT-GCN: Spatial-Temporal Graph Convolutional Network Embedded with Heterogeneous Information of Road Network for Traffic Forecasting

In road networks, attribute information carried by road segment nodes, such as weather and points of interest (POI), exhibit strong heterogeneity and often involve one-to-many or many-to-one relationships. However, research on such heterogeneity in traffic prediction is relatively limited. Our research examines how varying the network propagation pattern based on the degree of node-to-node heterogeneity of information affects the model prediction performance. Specifically, at the node level, we use knowledge embedding to generate knowledge vectors that quantify the heterogeneity among the attribute information of a node. At the road network level, we calculate a homogeneity adjacency matrix that captures both the topological structure of the road network and the similarity of node heterogeneity. This adjacency matrix assigns different weights to neighbors based on their homogeneity, guiding the propagation of graph convolutional networks (GCN). Finally, we separate the representation of propagation into self-representation and neighbor representation to extract multi-attribute information, including self, homogeneity, and heterogeneity. Experiments on real datasets demonstrate that the incorporation of our homogeneity adjacency matrix leads to a significant improvement in the accuracy of short-term and long-term prediction compared with previous work on homogeneous and single-dimensional information. Furthermore, our approach maintains its performance advantage over baseline models under different embedding dimensions and parameter settings.

Forecasting of water consumption by integrating spatial and temporal characteristics of short-term water use in cities

Urban water consumption prediction is an important basis for urban water resource management and optimal scheduling of water supply. Most existing prediction methods disregard the spatial–temporal characteristics of water consumption data and utilize incomplete data mining, resulting in prediction models with low accuracy. To accurately predict short-term water demand, a data-cleaning algorithm based on statistical distributions and velocity constraints is proposed in this study. The long- and short-term time-series network (LSTNet), automated spatial–temporal graph prediction (AutoSTG), and attention-based spatial–temporal graphical convolutional neural network (ASTGCN) deep learning models were investigated for automatically capturing the spatial–temporal dynamic features. Additionally, the short-term predictions for 1, 3, 6, 12, and 24 h forecasting periods were investigated based on the water consumption data of water plants in Tianjin. The results revealed that the proposed data-cleaning algorithm can effectively manage single-point anomalies and continuous outliers, thus improving prediction accuracy. The LSTNet, AutoSTG, and ASTGCN deep learning models, particularly the ASTGCN model based on the spatial–temporal attention mechanism, significantly improved the prediction accuracy and adaptive spatial–temporal feature extraction. The final prediction model that combines data-cleaning algorithms with the ASTGCN deep learning method achieved root mean square error, mean absolute error, accuracy, and relative squared error values of 566.4 m3, 377.7 m3, 0.964, and 0.112, respectively. The proposed forecasting framework can improve short-term water use forecasting at multiple sites in the city and provide support for fine-grained water allocation.

Human Pose Estimation Based on a Spatial Temporal Graph Convolutional Network

To address the problem of poor detection and under-utilization of the spatial relationship between nodes in human pose estimation, a method based on an improved spatial temporal graph convolutional network (ST-GCN) model is proposed. Firstly, upsampling and segmented random sampling strategies are used to effectively solve the problems of class imbalance and the large sequence length of the dataset. Secondly, an improved detection transformer (DETR) structure is added to effectively suppress the generation of non-maximal suppression (NMS) and anchor points, a multi-head attention (M-ATT) module is introduced into each ST-GCN cell to capture richer feature information, and a residual module is introduced into the 9th ST-GCN cell to avoid possible network degradation in deep networks. In addition, strategies such as warmup, regularization, loss functions, and optimizers are configured to improve the model’s performance. The experimental results show that the average percentage of correct keypoints (PCK) of this method are 93.2% and 92.7% for the FSD and MPII datasets, respectively, which is 1.9% and 1.7% higher than the average PCK of the original ST-GCN method. Moreover, the confusion matrix corresponding to this method also indicated that the model has high recognition accuracy. In addition, comparison experiments with ST-GCN and other methods show that the computation of the model corresponding to this method is about 1.7 GFLOPs and the corresponding MACs are about 6.4 GMACs, which is a good performance.

MTGEA: A Multimodal Two-Stream GNN Framework for Efficient Point Cloud and Skeleton Data Alignment

Because of societal changes, human activity recognition, part of home care systems, has become increasingly important. Camera-based recognition is mainstream but has privacy concerns and is less accurate under dim lighting. In contrast, radar sensors do not record sensitive information, avoid the invasion of privacy, and work in poor lighting. However, the collected data are often sparse. To address this issue, we propose a novel Multimodal Two-stream GNN Framework for Efficient Point Cloud and Skeleton Data Alignment (MTGEA), which improves recognition accuracy through accurate skeletal features from Kinect models. We first collected two datasets using the mmWave radar and Kinect v4 sensors. Then, we used zero-padding, Gaussian Noise (GN), and Agglomerative Hierarchical Clustering (AHC) to increase the number of collected point clouds to 25 per frame to match the skeleton data. Second, we used Spatial Temporal Graph Convolutional Network (ST-GCN) architecture to acquire multimodal representations in the spatio-temporal domain focusing on skeletal features. Finally, we implemented an attention mechanism aligning the two multimodal features to capture the correlation between point clouds and skeleton data. The resulting model was evaluated empirically on human activity data and shown to improve human activity recognition with radar data only. All datasets and codes are available in our GitHub.

New machine learning application platform for spatial–temporal thermal error prediction and control with STFGCN for ball screw system

The big data platform, which has a high control accuracy and efficiency, is expected to realize the high-accuracy prediction and real-time control of the thermal error (TE) for the ball screw system. But the TE of a ball screw system has significant dynamic spatial–temporal behaviors. The TE model, which fully captures its dynamic spatial–temporal behaviors, has not been reported so far. Moreover, the embedding of the TE model into the big data platform is extremely difficult, and the deployment of the big data platform is time-consuming and relies on the expert knowledge, and the cloud computing used in the big data platform also faces a limited bandwidth pressure of the industrial Internet, leading to the failure of the real-time control. To overcome the above limitations, a novel machine learning application platform (MLAP) is designed based on the cloud-terminal architecture. Moreover, the spatial–temporal fusion graph convolutional network (STFGCN) is proposed for the first time to fully capture of the dynamic spatial–temporal behaviors, and the proposed STFGCN is embedded into the designed MLAP to expedite the training process of the STFGCN model and reduce the bandwidth pressure on the industrial Internet. The prediction performance and robustness of the proposed STFGCN model are compared with that of the traditional multiple linear regression (MLR), long short-term memory (LSTM), gated recurrent unit (GRU), convolutional neural network-long short-term memory (CNN-LSTM), temporal-graph convolutional network (T-GCN), hypergraph neural network (HGNN), spatial–temporal graph convolutional network (STGCN), and attention-based spatial–temporal graph convolutional network (ASTGCN). The results suggest that the evaluation metrics R2 of the MLR, LSTM, GRU, CNN-LSTM, T-GCN, HGNN, STGCN, ASTGCN, and STFGCN are 0.8039, 0.9500, 0.9441, 0.9414, 0.9571, 0.9512, 0.9689, 0.9712, and 0.9812 to show the prediction performance, respectively. Moreover, the training rate is improved by adding the number of the virtual machine nodes on the MLAP. The reduction rate of the machining error is the ratio of the machining error reduction to the original machining error. The positioning error and its fluctuation range of the ball screw system are reduced effectively, and the machining errors with the implementation of the MLAP are decreased by more than 33% and 72% compared with that with the pitch error control and without the TE control, respectively.

Graph collaborative filtering-based bug triaging

Issue tracking systems are widely used for collecting bug reports. A target of intelligent software engineering is to automate assigning bugs to appropriate developers. Recently, the momentum of artificial intelligence has brought many successful studies that triage bugs by classifying their reports with NLP-based methods. Some studies also try to introduce context information to represent developers. Nevertheless, they take a fundamental assumption that developers and bugs, closely related entities in real-world scenarios, should be modeled independently.To capture the bug-developer correlations in bug triaging activities, we propose a Graph Collaborative filtering-based Bug Triaging framework: (1) bug-developer correlations are modeled as a bipartite graph; (2) natural language processing-based pre-training is implemented on bug reports to initialize bug nodes; (3) spatial–temporal graph convolution strategy is designed to learn the representation of developer nodes; (4) information retrieval-based classifier is proposed to match bugs and developers. Extensive experiments across mainstream datasets show the competence of our GCBT. Moreover, We believe that GCBT could generally benefit the modeling of correlations in other software engineering scenarios.

Spatial–temporal graph attention network for video anomaly detection

Spatial–temporal graph attention network for video anomaly detection

Explainable fMRI-based brain decoding via spatial temporal-pyramid graph convolutional network.

Brain decoding, aiming to identify the brain states using neural activity, is important for cognitive neuroscience and neural engineering. However, existing machine learning methods for fMRI-based brain decoding either suffer from low classification performance or poor explainability. Here, we address this issue by proposing a biologically inspired architecture, Spatial Temporal-pyramid Graph Convolutional Network (STpGCN), to capture the spatial-temporal graph representation of functional brain activities. By designing multi-scale spatial-temporal pathways and bottom-up pathways that mimic the information process and temporal integration in the brain, STpGCN is capable of explicitly utilizing the multi-scale temporal dependency of brain activities via graph, thereby achieving high brain decoding performance. Additionally, we propose a sensitivity analysis method called BrainNetX to better explain the decoding results by automatically annotating task-related brain regions from the brain-network standpoint. We conduct extensive experiments on fMRI data under 23 cognitive tasks from Human Connectome Project (HCP) S1200. The results show that STpGCN significantly improves brain-decoding performance compared to competing baseline models; BrainNetX successfully annotates task-relevant brain regions. Post hoc analysis based on these regions further validates that the hierarchical structure in STpGCN significantly contributes to the explainability, robustness and generalization of the model. Our methods not only provide insights into information representation in the brain under multiple cognitive tasks but also indicate a bright future for fMRI-based brain decoding.

HAR-time: human action recognition with time factor analysis on worker operating time

ABSTRACT This research proposed a data analysis framework applying human action recognition (HAR) with filtering processing to recognize human workers’ actions and further calculate the operating time of each action. When applying HAR to recognize the worker’s operation actions on a manufacturing site, the repeated and reciprocating actions easily cause misrecognition. Without the separation of actions in advance, the evaluation of HAR in real-world manufacturing is different from the evaluation based on the open data in the literature. To resolve the practical issues of applying HAR on a manufacturing site, first, the spatial-temporal skeleton coordinates of human joints were generated as the input of the skeleton-based spatial-temporal graph convolutional neural network (ST-GCN) for training the recognition model. Once the human action is recognized in each frame, the filtering processing considering the reference correction and accumulative moving mode (AMM) was proposed to improve the recognition accuracy. The real-world case study of the forging industry was conducted, and the empirical result shows the proposed framework was able to detect human workers’ repeated actions with 89% accuracy. Also, the time analysis based on HAR-Time recognition can correctly detect the associated time features that have highly correlated to the low quality of the product.

Skeleton-Based Fall Detection with Multiple Inertial Sensors Using Spatial-Temporal Graph Convolutional Networks.

The application of wearable devices for fall detection has been the focus of much research over the past few years. One of the most common problems in established fall detection systems is the large number of false positives in the recognition schemes. In this paper, to make full use of the dependence between human joints and improve the accuracy and reliability of fall detection, a fall-recognition method based on the skeleton and spatial-temporal graph convolutional networks (ST-GCN) was proposed, using the human motion data of body joints acquired by inertial measurement units (IMUs). Firstly, the motion data of five inertial sensors were extracted from the UP-Fall dataset and a human skeleton model for fall detection was established through the natural connection relationship of body joints; after that, the ST-GCN-based fall-detection model was established to extract the motion features of human falls and the activities of daily living (ADLs) at the spatial and temporal scales for fall detection; then, the influence of two hyperparameters and window size on the algorithm performance was discussed; finally, the recognition results of ST-GCN were also compared with those of MLP, CNN, RNN, LSTM, TCN, TST, and MiniRocket. The experimental results showed that the ST-GCN fall-detection model outperformed the other seven algorithms in terms of accuracy, precision, recall, and F1-score. This study provides a new method for IMU-based fall detection, which has the reference significance for improving the accuracy and robustness of fall detection.