Spatiotemporal Convolution Research Articles

Multi-area short-term load forecasting based on spatiotemporal graph neural network

Short term power load forecasting can accurately evaluate the overall power load changes and provide accurate reference for power system operation decision-making. To address the limitations of traditional load forecasting methods, which are unable to capture spatial correlations and simultaneously predict load changes in multiple areas, this paper proposes a load forecasting model based on the spatiotemporal attention convolutional mechanism. The proposed model is composed of two key components: a spatiotemporal attention module and a spatiotemporal convolution module. First, the dynamic spatiotemporal correlations between different electricity load areas are captured and analyzed by using the spatiotemporal attention mechanism. Secondly, the spatial pattern and temporal features of the load data sequence are effectively obtained by spatiotemporal convolutional layers. Finally, this paper verifies the accuracy and effectiveness of the proposed method through a real electricity consumption load dataset. The experimental results demonstrate that, in comparison to various benchmark prediction methods, the proposed method can fully explore and utilize the spatiotemporal correlation between different electricity load areas, thereby improving the accuracy of load prediction.

Parking Lot Traffic Prediction Based on Fusion of Multifaceted Spatio-Temporal Features.

With the rapid growth of population and vehicles, issues such as traffic congestion are becoming increasingly apparent. Parking guidance and information (PGI) systems are becoming more critical, with one of the most important tasks being the prediction of traffic flow in parking lots. Predicting parking traffic can effectively improve parking efficiency and alleviate traffic congestion, traffic accidents, and other problems. However, due to the complex characteristics of parking spatio-temporal data, high levels of noise, and the intricate influence of external factors, there are three challenges to predicting parking traffic in a city effectively: (1) how to better model the nonlinear, asymmetric, and complex spatial relationships among parking lots; (2) how to model the temporal autocorrelation of parking flow more accurately for each parking lot, whether periodic or aperiodic; and (3) how to model the correlation between external influences, such as holiday weekends, POIs (points of interest), and weather factors. In this context, this paper proposes a parking lot traffic prediction model based on the fusion of multifaceted spatio-temporal features (MFF-STGCN). The model consists of a feature embedding module, a spatio-temporal attention mechanism module, and a spatio-temporal convolution module. The feature embedding module embeds external features such as weekend holidays, geographic POIs, and weather features into the time series, the spatio-temporal attention mechanism module captures the dynamic spatio-temporal correlation of parking traffic, and the spatio-temporal convolution module captures the spatio-temporal features by using graph convolution and gated recursion units. Finally, the outputs of adjacent time series, daily series, and weekly series are weighted and fused to obtain the final prediction results, thus predicting the parking lot traffic flow more accurately and effectively. Results on real datasets demonstrate that the proposed model enhances prediction performance.

An auto-segmented multi-time window dual-scale neural network for brain-computer interfaces based on event-related potentials

Objective. Event-related potentials (ERPs) are cerebral responses to cognitive processes, also referred to as cognitive potentials. Accurately decoding ERPs can help to advance research on brain-computer interfaces (BCIs). The spatial pattern of ERP varies with time. In recent years, convolutional neural networks (CNNs) have shown promising results in electroencephalography (EEG) classification, specifically for ERP-based BCIs. Approach. This study proposes an auto-segmented multi-time window dual-scale neural network (AWDSNet). The combination of a multi-window design and a lightweight base network gives AWDSNet good performance at an acceptable cost of computing. For each individual, we create a time window set by calculating the correlation of signed R-squared values, which enables us to determine the length and number of windows automatically. The signal data are segmented based on the obtained window sets in sub-plus-global mode. Then, the multi-window data are fed into a dual-scale CNN model, where the sizes of the convolution kernels are determined by the window sizes. The use of dual-scale spatiotemporal convolution focuses on feature details while also having a large enough receptive length, and the grouping parallelism undermines the increase in the number of parameters that come with dual scaling. Main results. We evaluated the performance of AWDSNet on a public dataset and a self-collected dataset. A comparison was made with four popular methods including EEGNet, DeepConvNet, EEG-Inception, and PPNN. The experimental results show that AWDSNet has excellent classification performance with acceptable computational complexity. Significance. These results indicate that AWDSNet has great potential for applications in ERP decoding.

STC-PSSA: A New Model of Traffic Flow Forecasting Based on Spatiotemporal Convolution and Probabilistic Sparse Self-Attention

Traffic flow forecasting is the foundation of the dynamic control and application of intelligent transportation systems (ITS). It is also of significant practical value in alleviating road congestion. Given the periodic and dynamic changes in traffic flow and the spatiotemporal coupling interaction of complex road networks, traffic flow forecasting is challenging and rarely yields satisfactory prediction results. To capture the dynamic spatiotemporal characteristics of traffic flow, a new model of traffic flow forecasting based on spatiotemporal convolution and probabilistic sparse self-attention (STC-PSSA) is proposed. It consists of a spatiotemporal graph convolution network (ST-GCN) module, a spatiotemporal convolution module (ST-Conv), and a probabilistic sparse attention module (PSSA). ST-GCN consists of the gated temporal convolutional network (G-TCN) and the graph convolution network (GCN), which are used to capture the temporal dependence and spatial correlation of the traffic flow, respectively. Multiple ST-GCNs are stacked to handle spatial features at various time levels. The ST-Conv captures intricate temporal dependence at the same location and dynamic spatial features at neighboring locations simultaneously. The PSSA combines dynamic spatiotemporal features and performs long-term forecasting efficiently. The experimental results demonstrate that the STC-PSSA model can accurately extract the dynamic spatiotemporal characteristics of traffic flow and outperforms the popular baseline methods in forecasting accuracy.

LipNet: End-to-End Lipreading

Lipreading is the task of decoding text from the movement of a speaker’s mouth. This research presents the development of an advanced end-to-end lipreading system. Leveraging deep learning architectures and multimodal fusion techniques, the proposed system interprets spoken language solely from visual cues, such as lip movements. Through meticulous data collection, annotation, preprocessing, model development, and evaluation, diverse datasets encompassing various speakers, accents, languages, and environmental conditions are curated to ensure robustness and generalization. Conventional methods divided the task into two phases: prediction and designing or learning visual characteristics. Most deep lipreading methods are trainable from end to end. In the past, lipreading has been tackled using tedious and sometimes unsatisfactory techniques that break down speech into smaller units like phonemes or visemes. But these methods often fail when faced with real-world problems, such contextual factors, accents, and differences in speech patterns. Nevertheless, current research on end-to-end trained models only carries out word classification; sentence-level sequence prediction is not included. LipNet is an end-to-end trained model that uses spatiotemporal convolutions, a recurrent network, and the connectionist temporal classification loss to translate a variable-length sequence of video frames to text. LipNet breaks from this traditional paradigm by using an all-encompassing, end-to-end approach supported by deep learning algorithms, Convolutional neural networks (CNNs) and recurrent neural networks (RNNs), which are skilled at processing sequential data and extracting high-level representations, are fundamental to LipNet's architecture.LipNet achieves 95.2% accuracy in sentence-level on the GRID corpus, overlapped speaker split task, outperforming experienced human lipreaders and the previous 86.4% word-level state-of-the-art accuracy.The results underscore the transformative potential of the lipreading system in real-world applications, particularly in domains such as assistive technology and human-computer interaction, where it can significantly improve communication accessibility and inclusivity for individuals with hearing impairments.

EEG Emotion Recognition Network Based on Attention and Spatiotemporal Convolution.

Human emotions are complex psychological and physiological responses to external stimuli. Correctly identifying and providing feedback on emotions is an important goal in human-computer interaction research. Compared to facial expressions, speech, or other physiological signals, using electroencephalogram (EEG) signals for the task of emotion recognition has advantages in terms of authenticity, objectivity, and high reliability; thus, it is attracting increasing attention from researchers. However, the current methods have significant room for improvement in terms of the combination of information exchange between different brain regions and time-frequency feature extraction. Therefore, this paper proposes an EEG emotion recognition network, namely, self-organized graph pesudo-3D convolution (SOGPCN), based on attention and spatiotemporal convolution. Unlike previous methods that directly construct graph structures for brain channels, the proposed SOGPCN method considers that the spatial relationships between electrodes in each frequency band differ. First, a self-organizing map is constructed for each channel in each frequency band to obtain the 10 most relevant channels to the current channel, and graph convolution is employed to capture the spatial relationships between all channels in the self-organizing map constructed for each channel in each frequency band. Then, pseudo-three-dimensional convolution combined with partial dot product attention is implemented to extract the temporal features of the EEG sequence. Finally, LSTM is employed to learn the contextual information between adjacent time-series data. Subject-dependent and subject-independent experiments are conducted on the SEED dataset to evaluate the performance of the proposed SOGPCN method, which achieves recognition accuracies of 95.26% and 94.22%, respectively, indicating that the proposed method outperforms several baseline methods.

Advancing urban traffic accident forecasting through sparse spatio-temporal dynamic learning

Traffic accidents have emerged as one of the most public health safety matters, raising concerns from both the public and urban administrators. The ability to accurately predict traffic accident not only supports the governmental decision-making in advance but also enhances public confidence in safety measures. However, the efficacy of traditional spatio-temporal prediction models are compromised by the skewed distributions and sparse labeling of accident data. To this end, we propose a Sparse Spatio-Temporal Dynamic Hypergraph Learning (SST-DHL) framework that captures higher-order dependencies in sparse traffic accidents by combining hypergraph learning and self-supervised learning. The SST-DHL model incorporates a multi-view spatiotemporal convolution block to capture local correlations and semantics of traffic accidents, a cross-regional dynamic hypergraph learning model to identify global spatiotemporal dependencies, and a two-supervised self-learning paradigm to capture both local and global spatiotemporal patterns. Through experimentation on New York City and London accident datasets, we demonstrate that our proposed SST-DHL exhibits significant improvements compared to optimal baseline models at different sparsity levels. Additionally, it offers enhanced interpretability of results by elucidating complex spatio-temporal dependencies among various traffic accident instances. Our study demonstrates the effectiveness of the SST-DHL framework in accurately predicting traffic accidents, thereby enhancing public safety and trust.

Machine Learning-Based Traffic Flow Prediction and Intelligent Traffic Management

With the rapid development of information technology, multiple time series forecasting, which is typical of traffic flow forecasting, has become increasingly important in big data analysis. As the cornerstone of intelligent transportation system, traffic flow forecasting has important scientific research value and practical application value for urban traffic operation scheduling, quality and efficiency improvement of logistics transportation industry and public travel planning. Traffic flow prediction is always an important task of intelligent transportation system. Due to the complex temporal and spatial dependence of traffic flow sequence, it is very challenging to construct accurate traffic flow prediction in its ring neural network, graph network and Transformer model. Much of the existing work is based on very good models. Considering the advantages of convolutional networks, such as high computational efficiency and strong feature extraction ability, a traffic flow prediction model based on multi-view spatiotemporal convolution is proposed.

A Spatiotemporal Deep Learning-Based Smart Discovery Approach for Marine Pollution Incidents from the Data-Driven Perspective

Marine pollution incidents (MPI) are often a dynamic process of time and space interaction. Currently, the monitoring of MPI is basically realized by manual analysis from expert experience. Such working mode has an obvious time lag, and is not useful for timely disposal. As a result, intelligent algorithms that can make quick discovery for MPI from massive monitoring data remain a practical demand in this field. Considering that monitoring elements generally have multi-dimensional characteristics and spatiotemporal causal relationships, this work develops a spatiotemporal deep learning-based smart discovery approach for MPI from the data-driven perspective. In particular, a systematic preprocessing workflow is developed for the spatiotemporal monitoring data, which facilitates following feature extraction. Then, a spatiotemporal convolution neural network structure is developed to extract features from original spatiotemporal monitoring data. On this basis, the discovery results of MPI can be output via neural computing structures. Taking the polluting marine oil spill incident in the Bohai Sea in eastern China as a case study, this work carries out a simulation application and its result analysis. The obtained simulation results can reveal the proper performance of the proposal.

Phase Aberration Correction for In Vivo Ultrasound Localization Microscopy Using a Spatiotemporal Complex-Valued Neural Network.

Ultrasound Localization Microscopy (ULM) can map microvessels at a resolution of a few micrometers ( [Formula: see text]). Transcranial ULM remains challenging in presence of aberrations caused by the skull, which lead to localization errors. Herein, we propose a deep learning approach based on recently introduced complex-valued convolutional neural networks (CV-CNNs) to retrieve the aberration function, which can then be used to form enhanced images using standard delay-and-sum beamforming. CV-CNNs were selected as they can apply time delays through multiplication with in-phase quadrature input data. Predicting the aberration function rather than corrected images also confers enhanced explainability to the network. In addition, 3D spatiotemporal convolutions were used for the network to leverage entire microbubble tracks. For training and validation, we used an anatomically and hemodynamically realistic mouse brain microvascular network model to simulate the flow of microbubbles in presence of aberration. The proposed CV-CNN performance was compared to the coherence-based method by using microbubble tracks. We then confirmed the capability of the proposed network to generalize to transcranial in vivo data in the mouse brain (n=3). Vascular reconstructions using a locally predicted aberration function included additional and sharper vessels. The CV-CNN was more robust than the coherence-based method and could perform aberration correction in a 6-month-old mouse. After correction, we measured a resolution of [Formula: see text] for younger mice, representing an improvement of 25.8%, while the resolution was improved by 13.9% for the 6-month-old mouse. This work leads to different applications for complex-valued convolutions in biomedical imaging and strategies to perform transcranial ULM.

PreSTNet: Pre-trained Spatio-Temporal Network for traffic forecasting

Traffic forecasting stands as a cornerstone in urban planning, yet existing methods mainly fall short in capturing long-term spatio-temporal patterns. To be specific, various exquisite modules are designed in current works, while bringing heavy computing costs with the length of historical data. Furthermore, any replacement or reintegration of modules could weaken final performance, and the whole model needs to be trained from scratch on different datasets. To address the above shortcomings, inspired by pre-training methodologies, we introduce a novel Pre-trained Spatio-Temporal Network (PreSTNet), a two-phase framework comprising pre-training and fine-tuning. In the first pre-training phase, PreSTNet undertakes a data masking and recovery task, which is facilitated through a dedicated long sequence embedding module and an encoder–decoder structure with temporal attention layers and graph convolution operators. The recovery task involves masking several data, forcing the model to capture correlations between unmasked data and finally recover the masked part, which facilitates the extraction of long-term spatio-temporal features from extensive data. The second phase involves freezing the learned parameters and replacing the decoder with a forecasting header. This header, designed as a meta-learning fusion module and spatio-temporal convolution layers, can integrate long-term and short-term traffic data, and is trained in the manner of supervised learning to suit the target task. Rigorous experiments on real-world datasets underscore the superiority of PreSTNet. Further case studies reveal that the pre-trained encoder, coupled with an intuitive linear regression, achieves comparable performance to advanced methods. These findings affirm the robust capabilities of the proposed PreSTNet in addressing the complexities of traffic forecasting.

Robust Remote Photoplethysmography Estimation With Environmental Noise Disentanglement.

Remote Photoplethysmography (rPPG) has been attracting increasing attention due to its potential in a wide range of application scenarios such as physical training, clinical monitoring, and face anti-spoofing. On top of conventional solutions, deep-learning approach starts to dominate in rPPG estimation and achieves top-level performance. However, most of them try to integrate preprocessing steps such as the ROI selection into an end-to-end network, which may diverge the attention and also limit the generalization in other scenarios with different input skin regions. In this work, we focus on learning the intrinsic rPPG feature and design a lightweight but effective rPPG estimation network based on spatiotemporal convolution. To further improve the robustness, on top of the basic design we propose the Noise-Disentangled DeeprPPG (ND-DeeprPPG) by disentangling the environmental noise from the raw rPPG feature with an adversarial canonical correlation analysis learning strategy. Background regions are employed as a reference to guide the noise disentangling in a self-supervised manner. Extensive experiments show that our ND-DeeprPPG not only outperforms the state-of-the-arts on heart rate estimation but also exhibits promising robustness in cross-skin-region, cross-dataset scenarios and other rPPG-based tasks.

ADT Network: A Novel Nonlinear Method for Decoding Speech Envelopes From EEG Signals.

Decoding speech envelopes from electroencephalogram (EEG) signals holds potential as a research tool for objectively assessing auditory processing, which could contribute to future developments in hearing loss diagnosis. However, current methods struggle to meet both high accuracy and interpretability. We propose a deep learning model called the auditory decoding transformer (ADT) network for speech envelope reconstruction from EEG signals to address these issues. The ADT network uses spatio-temporal convolution for feature extraction, followed by a transformer decoder to decode the speech envelopes. Through anticausal masking, the ADT considers only the current and future EEG features to match the natural relationship of speech and EEG. Performance evaluation shows that the ADT network achieves average reconstruction scores of 0.168 and 0.167 on the SparrKULee and DTU datasets, respectively, rivaling those of other nonlinear models. Furthermore, by visualizing the weights of the spatio-temporal convolution layer as time-domain filters and brain topographies, combined with an ablation study of the temporal convolution kernels, we analyze the behavioral patterns of the ADT network in decoding speech envelopes. The results indicate that low- (0.5-8 Hz) and high-frequency (14-32 Hz) EEG signals are more critical for envelope reconstruction and that the active brain regions are primarily distributed bilaterally in the auditory cortex, consistent with previous research. Visualization of attention scores further validated previous research. In summary, the ADT network balances high performance and interpretability, making it a promising tool for studying neural speech envelope tracking.

Multi-Graph Spatio-Temporal Convolution for Traffic Flow Prediction Focusing on Edge Derived Imbalanced Data From Highway Electronics

Multi-Graph Spatio-Temporal Convolution for Traffic Flow Prediction Focusing on Edge Derived Imbalanced Data From Highway Electronics

Lightweight video super-resolution based on hybrid spatio-temporal convolution

Lightweight video super-resolution based on hybrid spatio-temporal convolution

A real-time basketball referee gesture recognition system based on multi-scale spatio-temporal features

Abstract In this paper, the multi-scale spatio-temporal convolution module and the channel attention mechanism of global information synchronization have excellent performance in the field of sports gesture action recognition; therefore, combining the multi-scale spatio-temporal convolution and the channel attention mechanism to construct the dynamic gesture recognition model. The trained, dynamic gesture recognition model is deployed to cell phones and web pages through Java computational programming language, and the application scenarios of the basketball referee gesture recognition system are explained in detail. The real-time basketball referee gesture recognition system processes are analyzed using simulation analysis after determining the dataset. The basketball referee gesture recognition system’s video cropping frame analysis has an accuracy of 87.64% and 71.25%, with a loss value of 6.24 and 5.59, as shown in the results. When compared to the single-resolution features on the basketball referee gesture recognition system, the fused features exhibit some improvement, and the average recognition rate is 92.20%. This study is applicable to the referee gesture tracking method in complex basketball sports scenarios and lays the foundation for higher semantic level research in the field of sports video analytics, such as action recognition, event detection, and content understanding.

Spatio-Temporal Heterogeneous Graph-Based Convolutional Networks for Traffic Flow Forecasting

Traffic flow forecasting plays a crucial role in the construction of intelligent transportation. The aims of this paper are to fully exploit the spatial correlation between nodes in a traffic network and to compensate for the inability of graph-based deep learning methods to model multiple relationship types, resulting in inadequate extraction of spatially correlated information about the traffic network. In this paper, we propose a deep spatio-temporal recurrent evolution network based on the graph convolution network (STREGCN) for heterogeneous graphs. Specifically, we transform the traffic network into a multi-relational heterogeneous graph to improve the information representation of the graph. This allows our model to capture multiple types of spatially relevant information. In the temporal dimension, we use one-dimensional causal convolution based on the gated linear unit to extract the temporal correlation information of the traffic flow. In addition, we designed the output of the spatio-temporal convolution module to obtain the final traffic flow predictions after a fully connected layer. Experiments on real datasets illustrate the effectiveness of the proposed STREGCN model and show the importance of representing information through heterogeneous graphs for the task of traffic flow prediction.

Spatiotemporal convolution sleep network based on graph attention mechanism with automatic feature extraction

Background and objectiveGraph neural networks (GNNs) are widely used for automatic sleep staging. However, the majority of GNNs are based on spectral approaches, as far as we know, which heavily depend on the Laplacian eigenbasis determined by the graph structure with a large computing cost. MethodsWe introduced a non-spectral approach named graph attention networks v2 (GATv2) as the core of our network to extract spatial information (S-GATv2 in our work), which is more flexible and intuitive than the routined spectral method. Meanwhile, to resolve the issue of weak generalization of using traditional feature extraction, the multi-convolutional layers are implemented to automatically extract features. In this work, the proposed spatiotemporal convolution sleep network (ST-GATv2) consists of multi-convolution layers and a GATv2 block. Of note, the graph attention technique to the time domain was applied to construct temporal GATv2 (T-GATv2), which intends to capture the connection between two channels in the adjacent sleep stages. Besides, the modified function is further proposed to capture the hidden changing trend information by the difference in the feature's value of the two adjacent stages. ResultsIn our experiment, we used the SS3 datasets in the MASS as our test datasets to compare with other advanced models. Our result reveals our model achieves the highest accuracy at 89.0 %. Besides, the proposed T-GATv2 block and modified function bring an approximate 0.5 % improvement in Kappa and F1-score. ConclusionsOur results support the potential of graph attention mechanisms and creative blocks (T-GATv2 and modified function) in sleep classification. We suggest the proposed ST-GATv2 model as an effective tool in sleep staging in either healthy or diseased states.

Sharpen data-driven prediction rules of individual large earthquakes with aid of Fourier and Gauss

Predicting individual large earthquakes (EQs)’ locations, magnitudes, and timing remains unreachable. The author’s prior study shows that individual large EQs have unique signatures obtained from multi-layered data transformations. Via spatio-temporal convolutions, decades-long EQ catalog data are transformed into pseudo-physics quantities (e.g., energy, power, vorticity, and Laplacian), which turn into surface-like information via Gauss curvatures. Using these new features, a rule-learning machine learning approach unravels promising prediction rules. This paper suggests further data transformation via Fourier transformation (FT). Results show that FT-based new feature can help sharpen the prediction rules. Feasibility tests of large EQs (Mge 6.5) over the past 40 years in the western U.S. show promise, shedding light on data-driven prediction of individual large EQs. The handshake among ML methods, Fourier, and Gauss may help answer the long-standing enigma of seismogenesis.

A novel hybrid framework based on temporal convolution network and transformer for network traffic prediction.

Accurately predicting mobile network traffic can help mobile network operators allocate resources more rationally and can facilitate stable and fast network services to users. However, due to burstiness and uncertainty, it is difficult to accurately predict network traffic. Considering the spatio-temporal correlation of network traffic, we proposed a deep-learning model, Convolutional Block Attention Module (CBAM) Spatio-Temporal Convolution Network-Transformer, for time-series prediction based on a CBAM attention mechanism, a Temporal Convolutional Network (TCN), and Transformer with a sparse self-attention mechanism. The model can be used to extract the spatio-temporal features of network traffic for prediction. First, we used the improved TCN for spatial information and added the CBAM attention mechanism, which we named CSTCN. This model dealt with important temporal and spatial features in network traffic. Second, Transformer was used to extract spatio-temporal features based on the sparse self-attention mechanism. The experiments in comparison with the baseline showed that the above work helped significantly to improve the prediction accuracy. We conducted experiments on a real network traffic dataset in the city of Milan. The results showed that CSTCN-Transformer reduced the mean square error and the mean average error of prediction results by 65.16%, 64.97%, and 60.26%, and by 51.36%, 53.10%, and 38.24%, respectively, compared to CSTCN, a Long Short-Term Memory network, and Transformer on test sets, which justified the model design in this paper.