Spatio-temporal Neural Network Research Articles

Multi-scale persistent spatiotemporal transformer for long-term urban traffic flow prediction

Long-term urban traffic flow prediction is an important task in the field of intelligent transportation, as it can help optimize traffic management and improve travel efficiency. To improve prediction accuracy, a crucial issue is how to model spatiotemporal dependency in urban traffic data. In recent years, many studies have adopted spatiotemporal neural networks to extract key information from traffic data. However, most models ignore the semantic spatial similarity between long-distance areas when mining spatial dependency. They also ignore the impact of predicted time steps on the next unpredicted time step for making long-term predictions. Moreover, these models lack a comprehensive data embedding process to represent complex spatiotemporal dependency. This paper proposes a multi-scale persistent spatiotemporal transformer (MSPSTT) model to perform accurate long-term traffic flow prediction in cities. MSPSTT adopts an encoder-decoder structure and incorporates temporal, periodic, and spatial features to fully embed urban traffic data to address these issues. The model consists of a spatiotemporal encoder and a spatiotemporal decoder, which rely on temporal, geospatial, and semantic space multi-head attention modules to dynamically extract temporal, geospatial, and semantic characteristics. The spatiotemporal decoder combines the context information provided by the encoder, integrates the predicted time step information, and is iteratively updated to learn the correlation between different time steps in the broader time range to improve the model's accuracy for long-term prediction. Experiments on four public transportation datasets demonstrate that MSPSTT outperforms the existing models by up to 9.5% on three common metrics.

Spatio-temporal neural network with handcrafted features for skeleton-based action recognition

The task of human action recognition (HAR) can be found in many computer vision practical applications. Various data modalities have been considered for solving this task, including joint-based skeletal representations which are suitable for real-time applications on platforms with limited computational resources. We propose a spatio-temporal neural network that uses handcrafted geometric features to classify human actions from video data. The proposed deep neural network architecture combines graph convolutional and temporal convolutional layers. The experiments performed on public HAR datasets show that our model obtains results similar to other state-of-the-art methods but has a lower inference time while offering the possibility to obtain an explanation for the classified action.

SME-MFP: A novel spatiotemporal neural network with multiangle initialization embedding toward multifunctional peptides prediction

As a promising alternative to conventional antibiotic drugs in the biomedical field, functional peptide has been widely used in disease treatment owing to its low toxicity, high absorption rate, and biological activity. Recently, several machine learning methods have been developed for functional peptide prediction. However, the main research heavily relies on statistical features and few consider multifunctional peptide identification. So, we propose SME-MFP, a novel predictor in the imbalanced multi-label functional peptide datasets. First, we employ physicochemical and evolutionary information to represent the peptide sequence’s initialization features from multiple perspectives. Second, the features are fused and then put into spatial feature extractors, where the residual connection and multiscale convolutional neural network extract more discriminative features of different lengths’ peptide sequences. Besides, we also design AFT-based temporal feature extractors to fully capture the global interactions of the sequences. Finally, devising a new loss to replace the traditional cross entropy loss to settle the class imbalance problems. The results show that our framework not only enhances the model's ability to capture sequence features effectively, but also accuracy improves by 3.89% over existing methods on public peptide datasets.

A Hybrid Control Path Planning Architecture Based on Traffic Equilibrium Assignment for Emergency

Unconventional events exacerbate the imbalance between regional transportation demand and limited road network resources. Scientific and efficient path planning serves as the foundation for rapidly restoring equilibrium to the road network. In real large-scale road networks, especially during emergencies, it is usually difficult to obtain or predict accurate dynamic traffic network flows in real-time, which is used to support equilibrium path planning. Moreover, the traditional iterative methods cannot meet the real-time demand of emergency equilibrium path planning decision generation. To solve the above problems, this paper proposes a hybrid control architecture for path planning based on equilibrium traffic assignment theory. The architecture introduces the travelers’ real-time travel data and constructs a spatio-temporal neural network, which captures the evolution of traffic network loads. Adaptive multi-graph fusion technology is used to mix the background traffic flow data and the traveler’s real-time Origin–Destination (OD) data, to mine the dynamic correlation between the traffic state and the travelers’ travel. Based on the real-time prediction of dynamic network states, equilibrium mapping learning is carried out to pre-allocate potential travel demands and construct equilibrium traffic graphs based on system optimization traffic assignment. Finally, individual evacuation path strategies are generated online in a data-driven manner in real time to achieve improved resilience in the transportation system.

LSTENet: Cement productivity prediction using a self-attention spatio-temporal variational autoencoder

In the advent of the Industry 4.0 paradigm, intelligent manufacturing has gained prominence with the integration of advanced Artificial Intelligence (AI) technologies aimed at augmenting production efficiency and mitigating operational costs. Particularly, in process industries such as cement manufacturing, the efficacy of production is intrinsically tied to the manifold process elements involved. The underlying complexity is further accentuated by the fact that the production chain is typified by a multi-variate, time-variant, and non-linear system. Such characteristics exacerbate the challenges associated with predicting the quality of final goods, especially considering the intricate physical–chemical reactions that persist even in steady states, which will reflect to the data representation. Additionally, prevailing prediction systems are impeded by their inability to assimilate ancillary spatiotemporal information, rendering their predictive accuracies suboptimal for practical production demands. Addressing these issues, this study introduces an innovative approach for online prediction of final production quality by employing a Spatiotemporal Neural Network. The core attributes of this technique encompass the extraction of latent spatial information and the effective handling of extensive temporal sequences within two main components by a variational autoencoder framework. In the variational autoencoder, it is achieved through the strategic application of learnable convolutional neural layers, supplemented by gated recurrent layers with self-attention. In a novel approach, these two fundamental components are seamlessly integrated within a single end-to-end optimization framework. Empirical evidence derived from a real-world dataset serves to substantiate the superior performance of our proposed methodology in contrast to extant machine learning algorithms. The findings are indicative of the potential that deep learning architectures harbor in addressing pragmatic challenges within industrial contexts.

Interpretable Multi-Branch Architecture for Spatiotemporal Neural Networks and Its Application in Seizure Prediction.

Currently, spatiotemporal convolutional neural networks (CNNs) for electroencephalogram (EEG) signals have emerged as promising tools for seizure prediction (SP), which explore the spatiotemporal biomarkers in an epileptic brain. Generally, these CNNs capture spatiotemporal features at single spectral resolution. However, epileptiform EEG signals contain irregular neural oscillations of different frequencies in different brain regions. Therefore, it may be underperforming and uninterpretable for the CNNs without capturing complex spectral properties sufficiently. This study proposed a novel interpretable multi-branch architecture for spatiotemporal CNNs, namely MultiSincNet. On the one hand, the MultiSincNet could directly show the frequency boundaries using the interpretable sinc-convolution layers. On the other hand, it could extract and integrate multiple spatiotemporal features across varying spectral resolutions using parallel branches. Moreover, we also constructed a post-hoc explanation technique for multi-branch CNNs, using the first-order Taylor expansion and chain rule based on the multivariate composite function, which demonstrates the crucial spatiotemporal features learned by the proposed multi-branch spatiotemporal CNN. When combined with the optimal MultiSincNet, ShallowConvNet, DeepConvNet, and EEGWaveNet had significantly improved the subject-specific performance on most metrics. Specifically, the optimal MultiSincNet significantly increased the average accuracy, sensitivity, specificity, binary F1-score, weighted F1-score, and AUC of EEGWaveNet by about 7%, 8%, 7%, 8%, 7%, and 7%, respectively. Besides, the visualization results showed that the optimal model mainly extracts the spectral energy difference from the high gamma band focalized to specific spatial areas as the dominant spatiotemporal EEG feature.

VRSTNN: Visual-Relational Spatio-Temporal Neural Network for Early Hazardous Event Detection in Automated Driving Systems

VRSTNN: Visual-Relational Spatio-Temporal Neural Network for Early Hazardous Event Detection in Automated Driving Systems

Analysis and Prediction of Dockless Shared Bike Demand Evolving Around Urban Rail Transit Stations: Case Study in Shenzhen, China

The emergence of dockless shared bikes (DSB) has led to their use as an important transfer mode to urban rail transit (URT) stations. However, in highly populated areas such as subway stations in peak hours, there is increasing concern about the imbalance between the demand and supply of shared bikes. To promote smoother subway transfer trips using shared bikes, it is very important to estimate the DSB demand, especially the disparity in the volume of bike pick-up and drop-off demand around subway stations. This research first utilizes the Shenzhen metro usage data and DSB usage data, analyzes data regarding subway and shared bike usage, discusses their potential transfer uses, and finds great disparity in DSB demand between different subway stations. The catchment area method is used to estimate bike usage as a potential transfer mode to the subway, where the catchment area is defined as a radius of 150 m from the subway station center. The DSB trip demand is categorized into two types: pick-up and drop-off. The most recent deep learning method, adaptive graph convolutional recurrent network (AGCRN), is used to predict the DSB demand more accurately because of its ability in enabling the modeling of relationships between entities in a self-adapted graph, and the prediction is compared with long short-term memory (LSTM), spatiotemporal neural network (STNN), diffusion convolutional recurrent neural network (DCRNN), and Graph WaveNet. Results show that methods with graphs (STNN, DCRNN, Graph WaveNet, and AGCRN) perform better than LSTM, and methods with adaptive graphs (Graph WaveNet and AGCRN) outperform methods with static graphs in terms of mean absolute error (MAE), root-mean-square error (RMSE), and mean absolute percentage error (MAPE). DSB prediction results show that AGCRN performs the best in this study. More data, particularly land use data and URT station volume data, are expected to improve the predictive accuracy of the method due to potentially improved graph representation of station characteristics and subway station volume correlations. And with more accurate prediction results, it will be possible to achieve a better balancing strategy for bike operation optimization for better bike usage, and thus for a higher transfer rate of DSB to subway.

Temporal segment dropout for human action video recognition

Temporal information is important for human action video recognition. With the widely used spatio-temporal neural networks, researchers have found that the learned high-level features preserve overfitted spatial information and limited temporal information, leading to inferior performance. This is because existing networks lack efficient regularization for the temporal structure. To learn more robust temporal features, we propose a temporal regularization method named Temporal Segment Dropout (TSD). TSD drops the most salient spatial features in order to enhance the temporal features in a clip of temporal segments. Without learning from complex examples, TSD can be easily deployed in existing networks. In the experiment, TSD is extensively evaluated on benchmark action recognition datasets, which brings consistent improvements over the baselines, especially for the action-centric classes.

A macro–micro spatio-temporal neural network for traffic prediction

Accurate traffic prediction is crucial for planning, management and control of intelligent transportation systems. Most state-of-the-art methods for traffic prediction effectively capture complex traffic patterns (e.g. spatial and temporal correlations of traffic data) by employing spatio-temporal neural networks as prediction models, together with graph convolution networks to learn spatial correlations of prediction objects (e.g. traffic states of road segments, as in this study). Such spatial correlations can be regarded as micro correlations. However, there are also macro correlations between regions, each of which is composed of multiple road segments or artificially partitioned areas. Macro correlations represent another type of interaction within road segments, and should be carefully considered when predicting traffic. The diversity of micro spatial correlations and corresponding macro spatial correlations (e.g. correlations based on physical proximity or traffic pattern similarity) further increases the complexity of traffic prediction. We overcome these challenges by developing a macro–micro spatio-temporal neural network model, denoted ‘MMSTNet’. MMSTNet captures spatio-temporal patterns by (a) utilizing a graph convolution network and a spatial attention network to capture micro and macro spatial correlations, respectively; (b) employing a temporal convolution network and a temporal attention network to learn temporal patterns; and (c) integrating hierarchically learned representations based on designed attention mechanisms. We perform evaluations on two real-world datasets and thereby demonstrate that MMSTNet outperforms state-of-the-art models in traffic prediction tasks.

Adaptive spatiotemporal neural networks based on machine learning for missing well-log prediction

The well log is the basis for understanding a geologic structure and evaluating petroleum reservoirs. It inevitably leads to missing logs due to borehole conditions and equipment failures. Existing machine-learning methods introduce convolutional neural networks (CNNs) in temporal networks to learn local morphological (spatial) features for improving prediction. However, they ignore the temporal background information of the logs and the differences in spatial features at different depth points. An adaptive spatiotemporal transformer (ASTT) is developed to effectively overcome these challenges, which consists of a temporal background encoder (TBE), a spatial encoder (SE), and a spatiotemporal decoder (STD). TBE introduces average pooling in the transformer encoder to learn the hidden geologic information in the extracted longitudinal temporal features. SE combines CNN and an attention mechanism to learn the spatial features of each depth point differentially. STD maps the extracted spatiotemporal features to the missing logs. Experimental results on real oilfield data indicate that our ASTT achieves excellent performance in terms of fitting degree and test error. The results in the cross-logs and crosswell cases demonstrate the generalization of ASTT.

Accelerated respiratory-resolved 4D-MRI with separable spatio-temporal neural networks.

Respiratory-resolved four-dimensional magnetic resonance imaging (4D-MRI) provides essential motion information for accurate radiation treatments of mobile tumors. However, obtaining high-quality 4D-MRI suffers from long acquisition and reconstruction times. To develop a deep learning architecture to quickly acquire and reconstruct high-quality 4D-MRI, enabling accurate motion quantification for MRI-guided radiotherapy (MRIgRT). A small convolutional neural network called MODEST is proposed to reconstruct 4D-MRI by performing a spatial and temporal decomposition, omitting the need for 4D convolutions to use all the spatio-temporal information present in 4D-MRI. This network is trained on undersampled 4D-MRI after respiratory binning to reconstruct high-quality 4D-MRI obtained by compressed sensing reconstruction. The network is trained, validated, and tested on 4D-MRI of 28 lung cancer patients acquired with a T1-weighted golden-angle radial stack-of-stars (GA-SOS) sequence. The 4D-MRI of 18, 5, and 5 patients were used for training, validation, and testing. Network performances are evaluated on image quality measured by the structural similarity index (SSIM) and motion consistency by comparing the position of the lung-liver interface on undersampled 4D-MRI before and after respiratory binning. The network is compared to conventional architectures such as a U-Net, which has 30 times more trainableparameters. MODEST can reconstruct high-quality 4D-MRI with higher image quality than a U-Net, despite a thirty-fold reduction in trainable parameters. High-quality 4D-MRI can be obtained using MODEST in approximately 2.5 min, including acquisition, processing, andreconstruction. High-quality accelerated 4D-MRI can be obtained using MODEST, which is particularly interesting forMRIgRT.

A Deep Learning Approach to Increase the Value of Satellite Data for PM2.5 Monitoring in China

Limitations in the current capability of monitoring PM2.5 adversely impact air quality management and health risk assessment of PM2.5 exposure. Commonly, ground-based monitoring networks are established to measure the PM2.5 concentrations in highly populated regions and protected areas such as national parks, yet large gaps exist in spatial coverage. Satellite-derived aerosol optical properties serve to complement the missing spatial information of ground-based monitoring networks. However, satellite remote sensing AODs are hampered under cloudy/hazy conditions or during nighttime. Here we strive to overcome the long-standing restriction that surface PM2.5 cannot be obtained with satellite remote sensing under cloudy/hazy conditions or during nighttime. In this work, we introduce a deep spatiotemporal neural network (ST-NN) and demonstrate that it can artfully fill these observational gaps. We quantified the quantitative impact of input variables on the results using sensitivity and visual analysis of the model. This technique provides ground-level PM2.5 concentrations with a high spatial resolution (0.01°) and 24-h temporal coverage, hour-by-hour, complete coverage. In central and eastern China, the 10-fold cross-validation results show that R2 is between 0.8 and 0.9, and RMSE is between 6 and 26 (µg m−3). The relative error varies in different concentration ranges and is generally less than 20%. Better constrained spatiotemporal distributions of PM2.5 concentrations will contribute to improving health effects studies, atmospheric emission estimates, and air quality predictions.

Assessing the impacts of past and ongoing deforestation on rainfall patterns in South America.

Despite recent advances in modeling forest-rainfall relationships, the current understanding of changes in observed rainfall patterns resulting from historical deforestation remains limited. To address this knowledge gap, we analyzed how 40 years of deforestation has altered rainfall patterns in South America as well as how current Amazonian forest cover sustains rainfall. First, we develop a spatiotemporal neural network model to simulate rainfall as a function of vegetation and climate inputs in South America; second, we assess the rainfall effects of observed deforestation in South America during the periods 1982-2020 and 2000-2020; third, we assess the potential rainfall changes in the Amazon biome under two deforestation scenarios. We find that, on average, cumulative deforestation in South America from 1982 to 2020 has reduced rainfall over the period 2016-2020 by 18% over deforested areas, and by 9% over non-deforested areas across South America. We also find that more recent deforestation, that is, from 2000 to 2020, has reduced rainfall over the period 2016-2020 by 10% over deforested areas and by 5% over non-deforested areas. Deforestation between 1982 and 2020 has led to a doubling in the area experiencing a minimum dry season of 4 months in the Amazon biome. Similarly, in the Cerrado region, there has been a corresponding doubling in the area with a minimum dry season of 7 months. These changes are compared to a hypothetical scenario where no deforestation occurred. Complete conversion of all Amazon forest land outside protected areas would reduce average annual rainfall in the Amazon by 36% and complete deforestation of all forest cover including protected areas would reduce average annual rainfall in the Amazon by 68%. Our findings emphasize the urgent need for effective conservation measures to safeguard both forest ecosystems and sustainable agricultural practices.

An adaptive spatio-temporal neural network for PM2.5 concentration forecasting

Accurate PM2.5 concentration prediction is essential for environmental control management, therefore numerous air quality monitoring stations have been established, which generate multiple time series with spatio-temporal correlation. However, the statistical distribution of data from different monitoring stations varies widely, which needs to provide higher flexibility in the feature extraction stage. Moreover, the spatio-temporal correlation and mutation characteristics of the time series are difficult to capture. To this end, an adaptive spatio-temporal prediction network (ASTP-NET) is proposed, in which the encoder adaptively extracts the input data features, then captures the spatio-temporal dependencies and dynamic changes of the time series, the decoder part maps the encoded features into a predicted future time series representation, while an objective function is proposed to measure the overall fluctuations of the model’s multi-step prediction. In this paper, ASTP-NET is evaluated based on the Xi'an air quality dataset, and the results show that the model outperforms other baseline methods.

Field–Road Operation Classification of Agricultural Machine GNSS Trajectories Using Spatio-Temporal Neural Network

The classification that distinguishes whether machines are driving on roads or working in fields based on their global navigation satellite system (GNSS) trajectories is essential for effective management of cross-regional agricultural machinery services in China. In this paper, a novel field–road classification method utilizing multiple deep neural networks (MultiDNN) is proposed to enhance the accuracy of field and road point classification. The MultiDNN model incorporates a bi-directional long short-term memory network (BiLSTM), a topology adaptive graph convolution network (TAG), and a self-attention network (ATT) to effectively extract spatio-temporal features for field–road classification. The BiLSTM is used to capture temporal relationships along the time axis of a trajectory, providing global contextual information for each point. Then, the TAG network is used to obtain the spatio-temporal relationships between adjacent points in a trajectory, offering local contextual information for each point. Finally, the ATT network assigns varying weights to features to emphasize important characteristics. The performance of the MultiDNN model was evaluated using a wheat harvesting trajectory dataset, and the results showed that it achieved a high degree of accuracy, up to 89.75%, outperforming the best baseline method (GCN) by 2.79%.

Available power estimation of wind farms based on deep spatio-temporal neural networks

With the development of advanced digital infrastructure in new wind power plants in China, the individual wind-turbine level data are available to power operators and can potentially provide more accurate available wind power estimations. In this paper, considering the state of the wind turbine and the loss in the station, a four-layer spatio-temporal neural network is proposed to compute the available power of wind farms. Specifically, the long short-term memory (LSTM) network is built for each wind turbine to extract the time-series correlations in historical data. In addition, the graph convolution network (GCN) is employed to extract the spatial relationship between neighboring wind turbines based on the topology and patterns of historical data. The case studies are performed using actual data from a wind farm in northern China. The study results indicate that the computation error using the proposed model is lower than that using the conventional physics-based methods and is also lower than that using other artificial intelligence methods.

A Spatio-Temporal Graph Convolutional Network for Air Quality Prediction

Air pollution is a pressing issue that poses significant threats to human health and the ecological environment. The accurate prediction of air quality is crucial to enable management authorities and vulnerable populations to take measures to minimize their exposure to hazardous pollutants. Although many methods have been developed to predict air quality data, the spatio-temporal correlation of air quality data is complex and nonstationary, which makes air quality prediction still challenging. To address this, we propose a novel spatio-temporal neural network, GCNInformer, that combines the graph convolution network with Informer to predict air quality data. GCNInformer incorporates information about the spatial correlations among different monitoring sites through GCN layers and acquires both short-term and long-term temporal information in air quality data through Informer layers. Moreover, GCNInformer uses MLP layers to learn low-dimensional representations from meteorological and air quality data. These designs give GCNInformer the ability to capture the complex and nonstationary relationships between air pollutants and their surrounding environment, allowing for more accurate predictions. The experimental results demonstrate that GCNInformer outperforms other methods in predicting both short-term and long-term air quality data. Thus, the use of GCNInformer can provide useful information for air pollutant prevention and management, which can greatly improve public health by alerting individuals and communities to potential air quality hazards.

Fixed-time stabilization of discontinuous spatiotemporal neural networks with time-varying coefficients via aperiodically switching control

Fixed-time stabilization of discontinuous spatiotemporal neural networks with time-varying coefficients via aperiodically switching control

Biceph-Net: A Robust and Lightweight Framework for the Diagnosis of Alzheimer's Disease Using 2D-MRI Scans and Deep Similarity Learning.

Alzheimer's Disease (AD) is a neurodegenerative disease that is one of the significant causes of death in the elderly population. Many deep learning techniques have been proposed to diagnose AD using Magnetic Resonance Imaging (MRI) scans. Predicting AD using 2D slices extracted from 3D MRI scans is challenging as the inter-slice information gets lost. To this end, we propose a novel and lightweight framework termed 'Biceph-net' for AD diagnosis using 2D MRI scans that models both the intra-slice and inter-slice information. 'Biceph-net' has been experimentally shown to perform similar to other Spatio-temporal neural networks while being computationally more efficient. Biceph-net is also superior in performance compared to vanilla 2D convolutional neural networks (CNN) for AD diagnosis using 2D MRI slices. Biceph-net also has an inbuilt neighbourhood-based model interpretation feature that can be exploited to understand the classification decision taken by the network. Biceph-net experimentally achieves a test accuracy of 100% in the classification of Cognitively Normal (CN) vs AD, 98.16% for Mild Cognitive Impairment (MCI) vs AD, and 97.80% for CN vs MCI vs AD.