Spatio-temporal Graph Convolutional Network Research Articles

Biomechanical research based on the recognition and detection of the strength of table tennis hitting action

The reason why China’s table tennis can continue to this day is because it is an important heritage sports event in the country. In order to ensure the high-level performance of the Chinese table tennis team in the game, the identification and detection of its sports characteristics is a very meaningful work. With the development of computer technology, the use of multimedia intelligence technology to conduct research on competitive tactics has become a general consensus in the sports world. This paper explores the recognition and detection technology of the strength of table tennis hitting movements from the perspective of biomechanics, aiming to provide scientific basis and technical support for improving the performance of athletes. Based on the principles of biomechanics and combined with multimedia intelligent algorithms, this study developed a spatiotemporal graph convolutional network (STGCN) and a motion detection method based on Kinect technology to identify and quantify the hitting strength of table tennis players. The experimental results of this paper show that in the recognition of different types of movements based on the STGCN method, the correct recognition rate of 100 groups of movement strength is 84%, and the correct recognition rate of 500 groups of movement strength is 91.6%; in the recognition of different types of movement strength based on Kinect, the correct recognition rate of 100 groups of movement strength is 97%, and the correct recognition rate of 500 groups of movement strength is 99.6%; it can be seen that no matter how many groups of hitting movements are made, the correct recognition rate of strength based on Kinect is higher than that of STGCN.

Federated Learning-Based Traffic Flow Prediction Model in Intelligent Transportation Systems

The existing Intelligent Transportation System (ITS) achieves high success. As an essential component of ITS, Traffic Flow Prediction (TFP) has attracted tremendous attention. It is a critical but challenging task to improve the robust convergence and high accuracy of TFP in real-world scenarios. This study presents an optimized Federated Learning (FL)-based ChebNet model, FedproxChebNet, to realize the highly effective and accurate TFP. By selecting the best penalty constant in the proximal term to optimize the objective function, this model can achieve fast and stable convergence. Using the ChebNet model to aggregate neighbor nodes’ characteristics, more hidden information underlying the spatio-temporal traffic data can be taken into consideration for training the global and local models. All the superiority of the FedproxChebNet model makes it outperform other FL models with the Graph Convolutional Network (GCN), Graph Attention Network (GAT) and Spatio-Temporal Graph Convolutional Network (STGCN). We designed a series of experiments on various FL-based GNN model comparisons, parameter sensitivity tests, and on verifying the performance of FedproxChebNet with different heterogeneous systems with [Formula: see text], [Formula: see text] and [Formula: see text]. Based on four real-world data sets from the cognitive network, the experimental results demonstrate that the presented FedproxChebNet provides the best convergence and the highest accuracy in TFP, and achieves the best performance in a highly heterogeneous system ([Formula: see text]). Specifically, the accuracy of FedproxChebNet is at least improved by [Formula: see text] on PeMS07 than other FL-based GNN models. This proposed FedproxChebNet model may be preferable for different scenarios in ITS such as route planning, traffic congestion control and reversible lanes.

A traffic speed prediction algorithm for dynamic spatio-temporal graph convolutional networks based on attention mechanism

A traffic speed prediction algorithm for dynamic spatio-temporal graph convolutional networks based on attention mechanism

A bike-sharing demand prediction model based on Spatio-Temporal Graph Convolutional Networks

Shared bikes, as an eco-friendly transport mode, facilitate short commutes for urban dwellers and help alleviate traffic. However, the prevalent station-based strategy for bike placements often overlooks urban zones, cycling patterns, and more, resulting in underutilized bikes. To address this, we introduce the Spatio-Temporal Bike-sharing Demand Prediction (ST-BDP) model, leveraging multi-source data and Spatio-Temporal Graph Convolutional Networks (STGCN). This model predicts spatial user demand for bikes between stations by constructing a spatial demand graph, accounting for geographical influences. For precision, ST-BDP integrates an attention-based graph convolutional network for station demand graph’s temporal-spatial features, and a sequential convolutional network for multi-source data (e.g., weather, time). In real dataset, experimental results show that ST-BDP has excellent performance with mean absolute error (MAE) = 1.62, mean absolute percentage error (MAPE) = 15.82%, symmetric mean absolute percentage error (SMAPE) = 16.14%, and root mean square error (RMSE) = 2.36, outperforming the baseline techniques. This highlights its predictive accuracy and potential to guide future bike-sharing policies.

Transient Stability Assessment in Power Systems: A Spatiotemporal Graph Convolutional Network Approach with Graph Simplification

Accurate and fast transient stability assessment (TSA) of power systems is crucial for safe operation. However, deep learning-based methods require long training and fail to simultaneously extract the spatiotemporal characteristics of the transient process in power systems, limiting their performance in prediction. This paper proposes a novel TSA method based on a spatiotemporal graph convolutional network with graph simplification. First, based on the topology and node information entropy of power grids, as well as the power flow of each node, the input characteristic matrix is compressed to accelerate evaluation. Then, a high-performance TSA model combining a graph convolutional network and a Gated Convolutional Network is constructed to extract the spatial features of the power grid and the temporal features of the transient process. This model establishes a mapping relationship between spatiotemporal features and their transient stability. Finally, the focal loss function has been improved to dynamically adjust the influence of samples with different levels of difficulty on model training, adaptively addressing the challenge of sample imbalance. This improvement reduces misclassification rates and enhances overall accuracy. Case studies on the IEEE 39-bus system demonstrate that the proposed method is rapid, reliable, and generalizable.

Domain Transfer-Based Hypergraph Convolutional Network for Posture Anomaly Detection in Physical Education Teaching

Human skeleton-based posture anomaly detection has been widely applied in the field of physical education teaching. The existing spatio-temporal graph convolutional networks (ST-GCN) can fully utilize the local and global information of the human skeleton for action recognition, but the entire model requires a large amount of computation and the modeling of high-order relationships between joint points of the human skeleton is insufficient. To this end, this paper proposes a novel domain adaptive hypergraph convolutional network for basketball posture anomaly analysis by exploiting 2D skeleton information. First, we designed an effective hypergraph convolution feature extraction network to improve the high-order dependency modeling. To further improve the performance of the hypergraph convolutional network, we introduce domain adaptive learning technology to supervise the feature extraction learning of the target domain (2D skeleton) through the source domain (3D skeleton). At last, we construct a basketball action teaching analysis dataset for model evaluation. We conducted a large number of comparative experiments on the public dataset NTU RGB+D and our self-built dataset. All the results showed that our proposed hypergraph convolutional model effectively extracts features of 2D human skeletons, and by introducing domain adaptive learning, the performance of basketball anomaly detection is further improved.

SRSGCN: A novel multi-sensor fault diagnosis method for hydraulic axial piston pump with limited data

Deep learning has immense potential in ensuring the safe operation of hydraulic axial piston pumps (HAPP). However, the complex operating environment and high cost of labeling have resulted in a scarcity of labeled fault samples. This paper proposes a novel method called Siamese Random Spatiotemporal Graph Convolutional Network (SRSGCN). Firstly, based on graph convolutional networks, a Random Spatiotemporal Graph (RSG) is designed to aggregate multi-sensor information at different time stamps, fully exploiting the spatiotemporal features of the original data. Secondly, the Siamese Neural Network (SNN) is improved by retaining the twin subnetwork structure and removing the similarity output part. While preserving feature extraction capabilities, it is endowed with classification ability. Based on its strong feature mining capability, SRSGCN can fully utilize the scarce sample information to improve diagnostic accuracy. Finally, a case study was conducted on our HAPP experimental platform. The experiments show that compared with other existing methods, this method has higher diagnostic accuracy and can effectively solve the problem of HAPP fault diagnosis under limited data conditions.

Pose-oriented scene-adaptive matching for abnormal event detection

For intelligent surveillance systems, abnormal event detection automatically analyses surveillance video sequences and detects abnormal objects or unusual human actions at the frame level. Due to the lack of labelled data, most approaches are semi-supervised based on reconstruction or prediction methods. However, these methods may not generalize well to unseen scene contexts. To address this issue, we present a novel self and mutual scene-adaptive matching method for abnormal event detection. In the framework, we propose synergistic pose estimation and object detection, which effectively integrates human pose and object detection information to improve pose estimation accuracy. Then, the poses are resized to reduce the spatial distance between the source and target domains. The improved pose sequences are further fed into a spatio-temporal graph convolutional network to extract the geometric features. Finally, the features are embedded in a clustering layer to classify action types and compute normality scores. The training data is taken from the training part of common video anomaly detection datasets: UCSD PED1 & PED2, CHUK Avenue, and ShanghaiTech Campus. The proposed framework is evaluated on video sequences with unseen scene contexts in the UCSD PED2 and ShanghaiTech Campus datasets. The detection accuracy and efficiency are also evaluated in detail, and the proposed method for abnormal event detection achieves the highest AUC performance, 84.6%, on the ShanghaiTech Campus dataset and relatively high AUC performance, 96.9% and 74.8%, on UCSD PED2 & PED1 datasets. Compared with other state-of-the-art works, the performance analysis and results confirm the robustness and effectiveness of our proposed framework for cross-scene abnormal event detection.

Dynamic mode decomposition and short-time prediction of PM2.5 using the graph Neural Koopman network

Analyzing and accurately predicting the spatiotemporal dynamics of PM2.5 remain challenging. The existing spatiotemporal prediction approaches are associated with high model complexity and limited interpretability. Conventional methods combining Koopman theory and deep learning often neglect spatial correlations in spatiotemporal data. This study used the hourly PM2.5 dataset of the Beijing-Tianjin-Hebei region to reveal its spatiotemporal hierarchy using Koopman mode decomposition to identify the key dynamic modes. Furthermore, a Spatial Physics Constrained Learning (SPCL) model utilizing a graph representation learning method was proposed to combine the graph topological information of the PM2.5 spatial features with the Koopman feature function. The results showed that PM2.5 has growth, decay, and oscillation modes as well as daily, weekly, monthly, and yearly periods. SPCL achieved mean absolute error, root mean square error (RMSE), correlation r, and index of agreement values of 9.678, 13.922, 0.864, and 0.921, respectively. The average RMSE at 12 h improved by 16.1%, 12.7%, 0.9%, and 3.5% compared with using Long short-term Memory, Graph Convolutional Networks and Long Short-Term Memory Networks, Spatio-Temporal Graph Convolutional Networks, and Dynamic Spatiotemporal Graph Convolution Network, respectively. By discretizing the neural network hidden layers, the explanatory key of PM2.5 modes was elucidated, which demonstrated enhanced stability.

Deep learning algorithms for predicting routes of snowplows in European cities

Most cities are faced with annual snowfall but are finding it difficult to deal with their snow plowing activities. Even though winter road maintenance has been studied for many decades, most of the papers do not present models that can be scaled up to allow for incorporation of side constraints that are often met in practical applications. Subject of the paper: Comparison of deep learning algorithms for forecasting snowplow routes in European cities in terms of spatial analysis, routing and traffic flow. The study focuses on extending neural networks including Recurrent Neural Network (RNN), Convolutional Neural Network (CNN) and Graph Neural Network (GNN) to overcome the problematics of snowplow in urban setting. Temporal and spatial data are incorporated using sophisticated models such as Spatio-Temporal Graph Convolutional Networks (STGCNs) using examples that capture the capacity to control the changes in the snowplow routes in reaction to the real-time traffic flow and weather information. The report also looks at how transformer models among other emerging technologies could be harnessed to improve predictive accuracy as well as efficiency. Comparisons made to these other methods evaluating deep learning methods’ benefits and drawbacks and their application to urban infrastructure and traffic flow. The results, therefore, point out that integrating these complicated formulas can go a long way in enhancing the efficiency and safety of snowplowing in the European cities to create better lit and safer transport networks

Hierarchical Dynamic Spatio-Temporal Graph Convolutional Networks with Self-Supervised Learning for Traffic Flow Forecasting

Hierarchical Dynamic Spatio-Temporal Graph Convolutional Networks with Self-Supervised Learning for Traffic Flow Forecasting

Spatio-temporal deep learning framework for pedestrian intention prediction in urban traffic scenes

Pedestrian intent prediction is an essential task for ensuring the safety of pedestrians and vehicles on the road. This task involves predicting whether a pedestrian intends to cross a road or not based on their behavior and surrounding environment. Previous studies have explored feature-based machine learning and vision-based deep learning models for this task but these methods have limitations in capturing the global spatio-temporal context and fusing different features of data effectively. To address these issues, we propose a novel hybrid framework HSTGCN for pedestrian intent prediction that combines spatio-temporal graph convolutional neural networks (STGCN) and long short-term memory (LSTM) networks. The proposed framework utilizes the strengths of both models by fusing multiple features, including skeleton pose, trajectory, height, orientation, and ego-vehicle speed, to predict their intentions accurately. The framework’s performance have been evaluated on the JAAD benchmark dataset and the results show that it outperforms the state-of-the-art methods. The proposed framework has potential applications in developing intelligent transportation systems, autonomous vehicles, and pedestrian safety technologies. The utilization of multiple features can significantly improve the performance of the pedestrian intent prediction task.

ABNet: AI-Empowered Abnormal Action Recognition Method for Laboratory Mouse Behavior.

The automatic recognition and quantitative analysis of abnormal behavior in mice play a crucial role in behavioral observation experiments in neuroscience, pharmacology, and toxicology. Due to the challenging definition of abnormal behavior and difficulty in collecting training samples, directly applying behavior recognition methods to identify abnormal behavior is often infeasible. This paper proposes ABNet, an AI-empowered abnormal action recognition approach for mice. ABNet utilizes an enhanced Spatio-Temporal Graph Convolutional Network (ST-GCN) as an encoder; ST-GCN combines graph convolution and temporal convolution to efficiently capture and analyze spatio-temporal dynamic features in graph-structured data, making it suitable for complex tasks such as action recognition and traffic prediction. ABNet trains the encoding network with normal behavior samples, then employs unsupervised clustering to identify abnormal behavior in mice. Compared to the original ST-GCN network, the method significantly enhances the capabilities of feature extraction and encoding. We conduct comprehensive experiments on the Kinetics-Skeleton dataset and the mouse behavior dataset to evaluate and validate the performance of ABNet in behavior recognition and abnormal motion detection. In the behavior recognition experiments conducted on the Kinetics-Skeleton dataset, ABNet achieves an accuracy of 32.7% for the top one and 55.2% for the top five. Moreover, in the abnormal behavior analysis experiments conducted on the mouse behavior dataset, ABNet achieves an average accuracy of 83.1%.

An Adaptive Spatiotemporal Decouple Graph Convolutional Network Based Quality‐Related Fault Detection Method for Complex Industrial Processes

ABSTRACTWith the rapid development of industrial technology, industrial processes become increasingly complex, presenting characteristics of large‐scale and multi‐unit collaboration. However, most current fault detection methods focus on nonlinearity, dynamics, and other characteristics, while neglecting spatiotemporal information. To address this issue, an adaptive spatiotemporal decouple graph convolutional network based quality‐related fault detection method is proposed in this article. First, the temporal graph convolutional network and spatial graph convolutional network are combined organically in the form of joint training. Second, considering that fixed graph structures cannot reflect the dynamic relationships among nodes, we proposed an adaptive weighted mask mechanism to construct dynamic correlation graph embedded with priori knowledge. Multi‐attention mechanism is used to integrate spatiotemporal information, besides, we designed a decoupling layer to avoid information redundancy. Finally, the proposed spatiotemporal graph convolutional network is used to establish a regression model, the quality‐related latent variables are extracted by the decoupling layer, and the statistic is constructed based on Kullback–Leibler divergence. The effectiveness and feasibility of the proposed method are illustrated with the hot strip mill process and the Tennessee Eastman process.

Collaborative energy management of interconnected regional integrated energy systems considering spatio-temporal characteristics

Multi-regional integrated energy systems (MRIES), which encompass renewable distributed generation (RDG) and electric vehicles (EV) dispersed across multiple regions, are pivotal for promoting low-carbon energy supplies, offering both economic and environmental benefits. However, the complexity introduced by diverse energy conversions and spatiotemporal coupling in MRIES poses significant challenges for energy management. This paper introduces a novel low-carbon energy management framework (EMF) for multi-regional IES, based on spatiotemporal correlations and considering three-dimensional (3D) integrated demand-side response (IDR). The proposed framework employs a novel Cross-Gated Spatio-Temporal Graph Convolutional Network (CGSTG) to achieve high-precision joint power forecasting separately for the RDG and multi-type loads. In the day-ahead stage, 3D IDR involving flexible loads, e.g., EVs, is considered with multi-regional IES sharing electricity and heat. By engaging in IDR across the temporal (horizontal), spatial (depth), and energy conversion (vertical) dimensions, each IES can enhance its coordinated scheduling capability and operational economy. During the intra-day stage, rolling optimization is performed based on the model predictive control (MPC) framework, incorporating a ladder-type carbon trading mechanism (CTM). Comparative experiments demonstrate that the proposed method can effectively reduce operational costs and carbon emissions with significant improvement of coordinated scheduling capability, leading to economic and environmental benefits of MRIES.

Solar energy prediction in IoT system based optimized complex-valued spatio-temporal graph convolutional neural network

The accurate prediction of solar energy generation is significant for efficient energy management in Internet of Things (IoT) devices. However, current forecasting models frequently fail to account for the dynamic nature of weather conditions. Also, traditional methods often have limited accuracy and scalability. This paper proposes Solar Energy Prediction in IoT system based optimized Complex-Valued Spatio-Temporal Graph Convolutional Neural Network (SEP-CVSGCNN-IoT) to overcome the limitations of existing models. Initially, the data are collected from solar panel and weather forecast. The collected data is given to the pre-processing using Data-Adaptive Gaussian Average Filtering (DAGAF) to remove the unwanted data and replace missing data. The pre-processed data is given into Nutcracker Optimization (NCO) algorithm for selecting optimal features. Then, the selected features are given to the Complex-Valued Spatio-Temporal Graph Convolutional Neural Network (CVSGCNN) for solar energy prediction. Finally, Dipper Throated Optimization Algorithm (DTOA) is proposed to enhance the weight parameter of CVSGCNN classifier, which precisely predicts solar energy in IoT. The proposed SEP-CVSGCNN-IoT method provides 18.46%, 26.34, 15.69 and 20.84% higher accuracy and 18.24%, 23.77, 24.34 and 16.29% lower mean absolute error when analyzed with existing techniques, such as deep learning enhanced solar energy prediction and AI-driven IoT (DL-ESEF-AI), towards efficient renewable energy prediction using deep learning (TEE-REP-DL), a new deep learning method for effectual forecasting of short-term PV energy production (DL-EF-SPEP) and metaheuristic-dependent hyper parameter tuning for recurrent deep learning: application to the solar energy generation prediction (HT-RDL-PSEG) respectively.

GSTGM: Graph, spatial–temporal attention and generative based model for pedestrian multi-path prediction

Pedestrian trajectory prediction in urban environments has emerged as a critical research area with extensive applications across various domains. Accurate prediction of pedestrian trajectories is essential for the safe navigation of autonomous vehicles and robots in pedestrian-populated environments. Effective prediction models must capture both the spatial interactions among pedestrians and the temporal dependencies governing their movements. Existing research primarily focuses on forecasting a single trajectory per pedestrian, limiting its applicability in real-world scenarios characterised by diverse and unpredictable pedestrian behaviours. To address these challenges, this paper introduces the Graph Convolutional Network, Spatial–Temporal Attention, and Generative Model (GSTGM) for pedestrian trajectory prediction. GSTGM employs a spatiotemporal graph convolutional network to effectively capture complex interactions between pedestrians and their environment. Additionally, it integrates a spatial–temporal attention mechanism to prioritise relevant information during the prediction process. By incorporating a time-dependent prior within the latent space and utilising a computationally efficient generative model, GSTGM facilitates the generation of diverse and realistic future trajectories. The effectiveness of GSTGM is validated through experiments on real-world scenario datasets. Compared to the state-of-the-art models on benchmark datasets such as ETH/UCY, GSTGM demonstrates superior performance in accurately predicting multiple potential trajectories for individual pedestrians. This superiority is measured using metrics such as Final Displacement Error (FDE) and Average Displacement Error (ADE). Moreover, GSTGM achieves these results with significantly faster processing speeds.

LEST: Large language models and spatio-temporal data analysis for enhanced Sino-US exchange rate forecasting

Exchange rate forecasting has a significant impact on a country’s balance of international payments, financial security and stability. Under the context of Sino-US trade war, a series of news events have happened thus increasing the abnormal fluctuations in the RMB exchange rate, and these fluctuations have made it more difficult to forecast the exchange rate. The analysis of news events’ impact on exchange rate fluctuations focused on assigning emotional labels to different economic news and political events by building an emotional lexicon. However, the previous method of using an offline emotional lexicon to analyze events could not be updated promptly making it difficult to better exploit implicit information of news events. Also, the offline emotional lexicon requires manual annotation of events which is unavoidable and subjective and therefore influences the sentiment analysis of the event. Therefore, we build a fusion forecasting framework of spatio-temporal and emotional information for accurate, real-time exchange rate forecasting. First, in the preprocessing phase, we leveraged the situational learning capabilities of Large Language Models (LLMs), specifically ChatGPT, combined with prompt engineering to enable ChatGPT to provide real-time, online, and precise automated sentiment analyses of Sino-US trade events. Second, we proposed the SG-TL fusion model, which integrates Spatio-Temporal Graph Convolutional Networks (STGCN) and Gated Recurrent Unit (GRU) models in the feature processing approach to accurately extract exchange rate features in spatial and temporal dimensions. We also employed Transformer and Long Short-Term Memory (LSTM) models in the forecasting phase to forecast the Sino-US exchange rates. Finally, Experiment results demonstrate that the exchange rate forecasting framework LEST can accurately capture the exchange rate fluctuations during the period of Sino-US trade war. Furthermore, its forecasting performance is more precise, efficient, and stable than existing works. The source code of our developed method is publicly available at https://gitee.com/andyham_andy.ham/lest-forecasting-framework.

Evaluating the effects of safety incentives on worker safety behavior control through image-based activity classification.

Construction worker safety remains a major concern even as task automation increases. Although safety incentives have been introduced to encourage safety compliance, it is still difficult to accurately measure the effectiveness of these measures. A simple count of accident rates and lower numbers do not necessarily mean that workers are properly complying with safety regulations. To address this problem, this study proposes an image-based approach to monitor moment-by-moment worker safety behavior and evaluate the effects of different safety incentive scenarios. By capturing workers' safety behaviors using a model integrated with OpenPose and spatiotemporal graph convolutional network, this study evaluated the effects of safety-incentive scenarios on workers' compliance with rules while on the job. The safety incentive scenarios in this study were designed as 1) varying the type (i.e., providing rewards and penalties) of incentives and 2) varying the frequency of feedback about ones' own compliance status during tasks. The effects of the scenarios were compared to the average compliance rates of three safety regulations (i.e., personal protective equipment self-monitoring hazard avoidance, and arranging the safety hook) for each scenario. The results show that 1) rewarding a good-compliance is more effective when there is no feedback on compliance status, and 2) penalizing non-compliance is more effective when there are three feedbacks during the tasks. This study provides a more accurate assessment of safety incentives and their effectiveness by focusing on safe behaviors to promote safety compliance among construction workers.

Multiscale graph based spatio-temporal graph convolutional network for energy consumption prediction of natural gas transmission process

Energy consumption prediction is crucial for planning and managing natural gas networks, enhancing transmission efficiency. However, most of relevant studies fail to consider the topology of natural gas network. Therefore, a multiscale graph based spatio-temporal graph convolutional network model (MG-STGCN) is proposed, which can extract the multi-scale relationship of data. Based on graph convolutional and temporal gated convolution network, data correlation, spatial dependence and time dependence are considered to improve the prediction effect. In order to verify the effect of the model, the model is tested based on two natural gas networks in different region of China. The single step prediction results of MG-STGCN are significantly better than spatio-temporal graph convolutional network (STGCN), long short-term memory (LSTM), gate recurrent unit (GRU) and temporal convolutional network (TCN), which proves the effectiveness and generalization ability of MG-STGCN. The training time of MG-STGCN is reduced by 20.8 % compared with STGCN. In different prediction time step, the average prediction effect of MG-STGCN is better than that of other methods, and the performance improvement rate is not less than 8.6 %. MG-STGCN can well capture the parameter correlation, spatial and temporal dependence of data, and achieve effective prediction of pipeline energy consumption.