MV-STGCN: Multi-view Spatial-Temporal Graph Convolutional Networks for On-Street Parking Availability Prediction

Skeleton‐based human action recognition is gaining significant attention and finding widespread application in various fields, such as virtual reality and human‐computer interaction systems. Recent studies have highlighted the effectiveness of graph convolutional network (GCN) based methods in this task, leading to a remarkable improvement in prediction accuracy. However, most GCN‐based methods overlook the varying contributions of self, centripetal and centrifugal subsets. Besides, only a single‐scale temporal feature is adopted, and the multi‐temporal scale information is ignored. To this end, firstly, in order to differentiate the importance of different skeleton subsets, we develop a refinement graph convolution, which can adaptively learn a weight for each subset feature. Secondly, a multi‐temporal scale aggregation module is proposed to extract more discriminative temporal dynamic information. Furthermore, a multi‐temporal scale aggregation refinement graph convolutional network (MTSA‐RGCN) is proposed, and four‐stream structure is also adopted in this paper, which can comprehensively model complementary features and eventually achieves a significant performance boost. In the empirical experiments, the performance of our approach has been greatly improved on both NTU‐RGB+D 60 and NTU‐RGB+D 120 datasets, compared to other state‐of‐the‐art methods.

In smart cities, on-street parking space prediction is the key yet difficult point in smart parking system. However, conventional prediction methods generally neglect spatial and temporal dependencies and cannot predict long-term parking events accurately. To this end, we propose a parking space prediction scheme based on the spatial-temporal graph convolution networks (STGCN). We first consider the instantaneous status of the parking to calculate the on-street parking occupancy rate (POR). Then, based on the POR, we exploit a time convolution module and a graph convolution module to extract spatial and temporal dependencies of the parking spaces, respectively. Next, we design the parameters of the STGCN to predict the POR of all the parking spaces based on the spatial and temporal dependencies. Finally, based on the real-world data sets, we compare the proposed scheme with the benchmark models. The experimental results show that the proposed scheme has the best performance in predicting the POR.

Brain computer interface (BCI) could be useful in improving the quality of life for paralyzed patients. Motor imagery classification has recently been a center of research interest in the BCI-based rehabilitation. As of current, spatial features and spectral features were often used independently for motor imagery classification. While few studies attempted to combine the information from varying domains including spectral, spatial and temporal feature, the attempts employed simplistic linear models. In this study, a novel feature extraction method for including spatial and temporal information is proposed. The method uses recurrent convolutional neural network (RCNN) which excels in temporal and spatial classification. The method was tested for classifying wrist twisting-related task classification during manipulation of robotic arm via electroencephalography, and the performance of the method was compared to the conventional motor imagery classifiers with common spatial pattern (CSP) filter. The proposed method showed 73.9% accuracy in the classification of three types of tasks, whereas the highest accuracy achieved by conventional models was 59.5%. Overall, the performance of the proposed RCNN model was greater than the conventional models using the CSP as input features. The findings warrant further application of the proposed methods in varying BCI environment.

Previous research has primarily employed deep learning models such as Convolutional Neural Networks (CNNs), and Recurrent Neural Networks (RNNs) for decoding imagined character signals. These approaches have treated the temporal and spatial features of the signals in a sequential, parallel, or single-feature manner. However, there has been limited research on the cross-relationships between temporal and spatial features, despite the inherent association between channels and sampling points in Brain-Computer Interface (BCI) signal acquisition, which holds significant information about brain activity. To address the limited research on the relationships between temporal and spatial features, we proposed a Temporal-Spatial Cross-Attention Network model, named TSCA-Net. The TSCA-Net is comprised of four modules: the Temporal Feature (TF), the Spatial Feature (SF), the Temporal-Spatial Cross (TSCross), and the Classifier. The TF combines LSTM and Transformer to extract temporal features from BCI signals, while the SF captures spatial features. The TSCross is introduced to learn the correlations between the temporal and spatial features. The Classifier predicts the label of BCI data based on its characteristics. We validated the TSCA-Net model using publicly available datasets of handwritten characters, which recorded the spiking activity from two micro-electrode arrays (MEAs). The results showed that our proposed TSCA-Net outperformed other comparison models (EEG-Net, EEG-TCNet, S3T, GRU, LSTM, R-Transformer, and ViT) in terms of accuracy, precision, recall, and F1 score, achieving 92.66%\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\%$$\\end{document}, 92.77%\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\%$$\\end{document}, 92.70%\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\%$$\\end{document}, and 92.58%\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\%$$\\end{document}, respectively. The TSCA-Net model demonstrated a 3.65%\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\%$$\\end{document} to 7.49%\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\%$$\\end{document} improvement in accuracy over the comparison models.

The ability to predict city-wide parking availability is crucial for the successful development of Parking Guidance and Information (PGI) systems. Indeed, the effective prediction of city-wide parking availability can improve parking efficiency, help urban planning, and ultimately alleviate city congestion. However, it is a non-trivial task for predicting city-wide parking availability because of three major challenges: 1) the non-Euclidean spatial autocorrelation among parking lots, 2) the dynamic temporal autocorrelation inside of and between parking lots, and 3) the scarcity of information about real-time parking availability obtained from real-time sensors (e.g., camera, ultrasonic sensor, and GPS). To this end, we propose Semi-supervised Hierarchical Recurrent Graph Neural Network (SHARE) for predicting city-wide parking availability. Specifically, we first propose a hierarchical graph convolution structure to model non-Euclidean spatial autocorrelation among parking lots. Along this line, a contextual graph convolution block and a soft clustering graph convolution block are respectively proposed to capture local and global spatial dependencies between parking lots. Additionally, we adopt a recurrent neural network to incorporate dynamic temporal dependencies of parking lots. Moreover, we propose a parking availability approximation module to estimate missing real-time parking availabilities from both spatial and temporal domain. Finally, experiments on two real-world datasets demonstrate the prediction performance of \hmgnn outperforms seven state-of-the-art baselines.

As the foundation of route planning and applications for intelligent transportation systems, accurate spatio-temporal traffic forecasting plays an essential role in improving both road utilization and traffic safety. Recently, the graph convolution network (GCN), recurrent neural network (RNN) and many other deep-learning based methods have been adopted for traffic flow forecasting and performed much better than the conventional statistical approaches. However, some node information may be lost during the propagation in graph convolutional layers, and the existing methods are insufficient to model the temporal dependencies especially for long-range sequences. In order to address these deficiencies, we innovatively come up with a graph convolutional stacked temporal attention neural network (GSTA), which can simultaneously extract the spatial and temporal features to forecast the traffic flow with higher accuracy. Specifically, our proposed framework uses a mix-hop GCN to better capture the spatial dependencies by preserving more useful information compared with the traditional GCN. Moreover, to identify the relations among traffic flow data over different time steps, we adopt an attention mechanism and introduce the temporal feature through embedding technology to capture the temporal regularity. We evaluate the proposed GSTA on two real-world traffic datasets, and the experimental results demonstrate the performance of our proposed model is significantly superior to several existing methods.

Background Genomic prediction using high-density (HD) marker genotypes is expected to lead to higher prediction accuracy, particularly for more heterogeneous multi-breed and crossbred populations such as those in sheep and beef cattle, due to providing stronger linkage disequilibrium between single nucleotide polymorphisms and quantitative trait loci controlling a trait. The objective of this study was to evaluate a possible improvement in genomic prediction accuracy of production traits in Australian sheep breeds based on HD genotypes (600k, both observed and imputed) compared to prediction based on 50k marker genotypes. In particular, we compared improvement in prediction accuracy of animals that are more distantly related to the reference population and across sheep breeds.MethodsGenomic best linear unbiased prediction (GBLUP) and a Bayesian approach (BayesR) were used as prediction methods using whole or subsets of a large multi-breed/crossbred sheep reference set. Empirical prediction accuracy was evaluated for purebred Merino, Border Leicester, Poll Dorset and White Suffolk sire breeds according to the Pearson correlation coefficient between genomic estimated breeding values and breeding values estimated based on a progeny test in a separate dataset.ResultsResults showed a small absolute improvement (0.0 to 8.0% and on average 2.2% across all traits) in prediction accuracy of purebred animals from HD genotypes when prediction was based on the whole dataset. Greater improvement in prediction accuracy (1.0 to 12.0% and on average 5.2%) was observed for animals that were genetically lowly related to the reference set while it ranged from 0.0 to 5.0% for across-breed prediction. On average, no significant advantage was observed with BayesR compared to GBLUP.

Urban water demand forecasting is the key component of smart water, which plays an important role in building a smart city. Although various methods have been proposed to improve forecast accuracy, most of these methods lack the ability to model spatio-temporal correlations. When dealing with the rich water demand monitoring data currently, it is difficult to achieve the desired prediction results. To address this issue from the perspective of improving the ability to extract temporal and spatial features, we propose a dynamic graph convolution-based spatio-temporal feature network (DG-STFN) model. Our model contains two major components, one is the dynamic graph generation module, which builds the dynamic graph structure based on the attention mechanism, and the other is the spatio-temporal feature block, which extracts the spatial and temporal features through graph convolution and conventional convolution. Based on the Shenzhen urban water supply dataset, five models SARIMAX, LSTM, STGCN, DCRNN, and ASTGCN are used to compare with DG-STFN proposed. The results show that DG-STFN outperforms the other models.

Wind speed forecasting is still a challenge due to the stochastic and highly varying characteristics of wind. In this paper, a graph deep learning model is proposed to learn the powerful spatio-temporal features from the wind speed and wind direction data in neighboring wind farms. The underlying wind farms are modeled by an undirected graph, where each node corresponds to a wind site. For each node, temporal features are extracted using a long short-term memory Network. A scalable graph convolutional deep learning architecture (GCDLA), motivated by the localized first-order approximation of spectral graph convolutions, leverages the extracted temporal features to forecast the wind-speed time series of the whole graph nodes. The proposed GCDLA captures spatial wind features as well as deep temporal features of the wind data at each wind site. To further improve the prediction accuracy and capture robust latent representations, the rough set theory is incorporated with the proposed graph deep network by introducing upper and lower bound parameter approximations in the model. Simulation results show the advantages of capturing deep spatial and temporal interval features in the proposed framework compared to the state-of-the-art deep learning models as well as shallow architectures in the recent literature.

Online Action Detection (OAD) has attracted more and more attention in recent years. A network for OAD generally consists of three parts: a frame-level feature extractor, a temporal modeling module, and an action classifier. Most recent OAD networks use a single-channel Recurrent Neural Network (RNN) to capture long-term history information, with spatial and temporal features concatenated as network input. In OAD, spatial features describe object appearance and scene configuration within each frame while temporal features capture motion cues over time. It is crucial to effectively fuse both spatial and temporal features. In this paper, we propose a new framework named TwinLSTM based on two-channel Long Short-Term Memory (LSTM) network for OAD, in which each channel is used to extract and handle either spatial features or temporal features. To more effectively fuse both spatial and temporal features, we design a prediction fusion module (PFM) to utilize hidden states of both channels to obtain more action content, including information interaction and future context prediction. We evaluate TwinLSTM on two challenging datasets: THUMOS14 and HDD. Experiments show that TwinLSTM outperforms existing single-channel models by a significant margin. We also show the effectiveness of PFM through comprehensive ablation studies.

In skeleton-based human action recognition, spatial-temporal graph convolution networks (ST-GCNs) have achieved remarkable performances recently. However, how to explore more discriminative spatial and temporal features is still an open problem. The temporal graph convolution of the traditional ST-GCNs utilizes only one fixed kernel which cannot completely cover all the important stages of each action execution. Besides, the spatial and temporal graph convolution layers (GCLs) are serial connected, which mixes information of different domains and limits the feature extraction capability. In addition, the input features like joints, bones, and their motions are modeled in existing methods, but more input features are needed for better performance. To this end, this article proposes a novel multi-stream and enhanced spatial-temporal graph convolution network (MS-ESTGCN). For each basic block of MS-ESTGCN, densely connected multiple temporal GCLs with different kernel sizes are employed to aggregate more temporal features. To eliminate the adverse impact of information mixing, an additional spatial GCL branch is added to the block and the spatial features can be enhanced. Furthermore, we extend the input features by employing relative positions of joints and bones. Consequently, there are totally six data modalities (joints, bones, their motions and relative positions) that can be fed into the network independently with a six-stream paradigm. The proposed method is evaluated on two large scale datasets: NTU-RGB+D and Kinetics-Skeleton. The experimental results show that our method using only two data modalities delivers state-of-the-art performance, and our methods using four and six data modalities further exceed other methods with a significant margin.

Timely and accurate temperature prediction is crucial to human production and life. However, due to the highly nonlinear nature of temperature prediction, traditional methods cannot meet the medium- and long-term temperature prediction tasks. To address the problems of large prediction errors and inadequate extraction of spatio-temporal features in existing temperature prediction algorithms, an improved deep learning framework: graph convolutional recurrent neural network (GCRNN) is proposed to solve the time series prediction problem in the field of temperature prediction. Specifically, GCRNN uses graph convolution to capture spatial correlation, and uses Encoder-Decoder architecture to capture temporal correlation. In the specific implementation process, the matrix multiplication in recurrent neural network is replaced by graph convolution operator, so as to realize the fusion and extraction of spatio- temporal features. The model is evaluated on three real temperature datasets, and the results show that GCRNN is able to effectively capture the spatio-temporal correlations of real-time temperature datasets, Compared with the existing baseline model, the prediction effect of GCRNN has achieved better results.

Vehicle trajectory prediction is an important research basis for the decision making and path planning of the intelligent and connected vehicle. In the connected vehicle environment, vehicles share information and drive cooperatively, and the intelligent and connected vehicles are able to obtain more accurate and rich perception information, which provides a data basis for accurate prediction of vehicle trajectories. However, attaining accurate and effective vehicle trajectory predictions poses technical challenges due to insufficient extraction of vehicular spatial–temporal interaction features. In this paper, we propose a vehicle trajectory prediction model based on graph convolutional neural network (GCN) in a connected vehicle environment. Specifically, using the driving scene information obtained by the intelligent and connected vehicle, the spatial graph and temporal graph are constructed based on the spatial interaction coefficient (SIC) and self-attention mechanism, respectively. Then, the graph data are entered into the interaction extraction module, and the spatial interaction features and temporal interaction features are extracted separately using the graph convolutional networks, which are fused to obtain the spatial–temporal interaction information. Finally, the interaction features are learned based on the convolutional neural networks to output the future trajectory information of all vehicles in the scene by one forward operation rather than a step-by-step process. The ablation experiment results show that the method proposed in this study to model the spatiotemporal interaction among vehicles based on SIC and self-attention mechanism reduces the prediction error by 5% and 12%, respectively. The results from the model comparison experiment show that the proposed method engenders an 8% improvement in prediction accuracy over the state-of-the-art solution, providing technical and theoretical support for trajectory prediction research of intelligent and connected vehicles.

Sketch recognition relies on two types of information, namely, spatial contexts like the local structures in images and temporal contexts like the orders of strokes. Existing methods usually adopt convolutional neural networks (CNNs) to model spatial contexts, and recurrent neural networks (RNNs) for temporal contexts. However, most of them combine spatial and temporal features with late fusion or single-stage transformation, which is prone to losing the informative details in sketches. To tackle this problem, we propose a novel framework that aims at the multi-stage interactions and refinements of spatial and temporal features. Specifically, given a sketch represented by a stroke array, we first generate a temporal-enriched image (TEI), which is a pseudo-color image retaining the temporal order of strokes, to overcome the difficulty of CNNs in leveraging temporal information. We then construct a dual-branch network, in which a CNN branch and a RNN branch are adopted to process the stroke array and the TEI respectively. In the early stages of our network, considering the limited ability of RNNs in capturing spatial structures, we utilize multiple enhancement modules to enhance the stroke features with the TEI features. While in the last stage of our network, we propose a spatio-temporal enhancement module that refines stroke features and TEI features in a joint feature space. Furthermore, a bidirectional temporal-compatible unit that adaptively merges features in opposite temporal orders, is proposed to help RNNs tackle abrupt strokes. Comprehensive experimental results on QuickDraw and TU-Berlin demonstrate that the proposed method is a robust and efficient solution for sketch recognition.

Research in deep learning models to forecast traffic intensities has gained great attention in recent years due to their capability to capture the complex spatio-temporal relationships within the traffic data. However, most state-of-the-art approaches have designed spatial-only (e.g. Graph Neural Networks) and temporal-only (e.g. Recurrent Neural Networks) modules to separately extract spatial and temporal features. However, we argue that it is less effective to extract the complex spatiotemporal relationship with such factorized modules. Besides, most existing works predict the traffic intensity of a particular time interval only based on the traffic data of the previous one hour of that day. And thereby ignores the repetitive daily/weekly pattern that may exist in the last hour of data. Therefore, we propose a Unified Spatio-Temporal Graph Convolution Network (USTGCN) for traffic forecasting that performs both spatial and temporal aggregation through direct information propagation across different timestamp nodes with the help of spectral graph convolution on a spatio-temporal graph. Furthermore, it captures historical daily patterns in previous days and current-day patterns in current-day traffic data. Finally, we validate our work's effectiveness through experimental analysis <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> Code is available at github.com/AmitRoy7781/USTGCN, which shows that our model USTGCN can outperform state-of-the-art performances in three popular benchmark datasets from the Performance Measurement System (PeMS). Moreover, the training time is reduced significantly with our proposed USTGCN model.