Spatial-temporal Learning Research Articles

Improving AQI Prediction and Optimization Using Graph Convolutional Networks and Transformer Models with Spatial Embedding Operations

State-of-the-art models do not quite capture complex spatial-temporal dependencies driving variations in the AQI across different urban zones, particularly in areas characterized by heavy traffic and industrial emissions. Accurate and timely prediction of the AQI is critical since it could mitigate the health and environmental impacts of air pollution, especially in metropolitan areas like Delhi that frequent hazardous levels of pollution. They rely primarily on grid-based or very naive temporal schemes that hardly capture the real nature of an urban region. To overcome these shortcomings, this paper suggests three advanced paradigms: (1) Graph Convolutional Networks with Temporal Attention Mechanism, (2) Transformer-based AQI Prediction with Spatial Embedding, and (3) Multi-Agent Reinforcement Learning with Nash Equilibrium for AQI optimization. This model captures the complex spatial interdependencies among different monitoring stations and directs attention in time for emphasizing related important features in different time domains. Therefore, the prediction accuracy improves up to 20% with an average error of ~2.8 units in high-risk zones like Wazipur and Okhla. The Transformer model introduces spatial embeddings that improve the self-attention mechanism to accurately predict spikes in AQI during polluted episodes and provides an estimated 25% improvement with an average error at around ~2.6 units. Conclusion In conclusion, the MARL model improves optimizations of interventions applied in pollution control by representing several regions or sources via agents, achieving a 20% decline in high-pollution episodes. This work effectively improved upon spatial-temporal learning by great enhancements in prediction and optimization of AQI, leading towards a more accurate forecast and practical control strategy for pollution: an objective that lends environmental management improved as well as greater public health benefits for urban and industrial areas.

Variational spatial–temporal graph attention network for state monitoring and forecasting

With the popularity of smart devices, there has been extensive collection of data structured in graphs. Spatial graphs, in particular, have garnered significant interest owing to their diverse range of applications. In this paper, we focus on applying spatial graph model for state monitoring and forecasting, with special attention to transportation systems. In current spatial modeling approaches, understanding the structure of a spatial graph usually relies on the analysis of gathered spatial–temporal data. However, spatial graph data in real world often contains temporary factors that are not easily detected. In this paper, we propose a novel variational inference-based model VISTG, which integrates such dynamics into spatial–temporal learning of spatial graph modeling. Specifically, VISTG is primarily composed of several spatial–temporal learning blocks, each encompassing both temporal and spatial learning layers. The temporal learning layer is crafted to characterize the distributions of latent factors using a variational inference-based model, aiming to capture the dynamics within the data. Subsequently, the spatial learning layer utilizes graph attention networks to describe the correlation among nodes. Additionally, an adaptive fusion module is implemented to equalize the impact of diverse temporal patterns. Finally, comprehensive experiments are carried out on two real-world datasets. The results affirm the efficacy of our model.

Emotion recognition using hierarchical spatial–temporal learning transformer from regional to global brain

Emotion recognition is an essential but challenging task in human–computer interaction systems due to the distinctive spatial structures and dynamic temporal dependencies associated with each emotion. However, current approaches fail to accurately capture the intricate effects of electroencephalogram (EEG) signals across different brain regions on emotion recognition. Therefore, this paper designs a transformer-based method, denoted by R2G-STLT, which relies on a spatial–temporal transformer encoder with regional to global hierarchical learning that learns the representative spatiotemporal features from the electrode level to the brain-region level. The regional spatial–temporal transformer (RST-Trans) encoder is designed to obtain spatial information and context dependence at the electrode level aiming to learn the regional spatiotemporal features. Then, the global spatial–temporal transformer (GST-Trans) encoder is utilized to extract reliable global spatiotemporal features, reflecting the impact of various brain regions on emotion recognition tasks. Moreover, the multi-head attention mechanism is placed into the GST-Trans encoder to empower it to capture the long-range spatial–temporal information among the brain regions. Finally, subject-independent experiments are conducted on each frequency band of the DEAP, SEED, and SEED-IV datasets to assess the performance of the proposed model. Results indicate that the R2G-STLT model surpasses several state-of-the-art approaches.

Design of an Iterative Method for Crop Disease Analysis Incorporating Graph Attention with Spatial-Temporal Learning and Deep Q-Networks

Design of an Iterative Method for Crop Disease Analysis Incorporating Graph Attention with Spatial-Temporal Learning and Deep Q-Networks

Fault detection of complicated processes based on an enhanced transformer network with graph attention mechanism

Recently, deep learning becomes increasingly popular in the industrial data analysis field due to its distinguished feature representation capability. As an emerging deep learning technology, transformer network has attracted extensive attention, but its application in industrial fault detection is still not sufficiently exploited. Furthermore, the traditional transformer network mainly focuses on the time series information of the data, while ignoring the spatial characteristic of the data. For this problem, an improved transformer network model with graph attention mechanism (GA-Tran) is proposed and used for complicated process fault detection. Considering the strong coupling property of process variables, the presented method performs joint spatial-temporal learning by mining the data information in views of both time-dimension and variable-dimension. On the one hand, a transformer encoder module equipped with a self-attention mechanism is utilized to capture long-term temporal dependencies, while multi-scale convolution is integrated to abstract short-term time dependencies. On the other hand, a graph attention network is introduced into the transformer network to meticulously analyze the spatial relationships among variables. Considering the topology of the industrial process variables, we use spectral clustering to infer prior topology information, and remove intelligently irrelevant spatial information. Different from the existing transformer models with the anomaly scores as the monitoring indicator, two monitoring statistics based on encoder features and reconstruction errors are constructed for process status monitoring. Studies on the Tennessee Eastman chemical process demonstrate the effectiveness of the proposed GA-Tran method.

FAST-CA: Fusion-based Adaptive Spatial–Temporal Learning with Coupled Attention for airport network delay propagation prediction

The issue of delay propagation prediction in airport networks has garnered increasing global attention, particularly due to its profound impact on operational efficiency and passenger satisfaction in modern air transportation systems. Despite research advancements in this domain, existing methodologies often fall short of comprehensively addressing the challenges associated with predicting delay propagation in airport networks, especially in terms of handling complex spatial–temporal dependencies and sequence couplings. In response to the complex challenge of predicting delay propagation in airport networks, we introduce the Fusion-based Adaptive Spatial–Temporal Learning with Coupled Attention (FAST-CA) framework. FAST-CA is an innovative model that integrates dynamic and adaptive graph learning, coupled attention mechanisms, periodicity feature extraction, and multifaceted information fusion modules. This holistic approach enables a thorough analysis of the interplay between flight departure and arrival delays and the spatial–temporal correlations within airport networks. Rigorously evaluated on two extensive real-world datasets, our model consistently outperforms current state-of-the-art baseline models, showcasing superior predictive performance and the effective learning capabilities of its intricately designed modules. Our research highlights the criticality of analyzing spatial–temporal relationships and the dynamics of flight coupling, offering significant theoretical and practical contributions to the advancement and management of air transportation systems.

Flow-Time Minimization for Timely Data Stream Processing in UAV-Aided Mobile Edge Computing

Unmanned Aerial Vehicle (UAV) has gained increasing attentions by both academic and industrial communities, due to its flexible deployment and efficient line-of-sight communication. Recently, UAVs equipped with base stations have been envisioned as a key technology to provide 5G network services for mobile users. In this paper, we provide timely services on the data streams of mobile users in a UAV-aided Mobile Edge Computing (MEC) network, in which each UAV is equipped with a 5G small-cell base station for communication and data processing. Specifically, we first formulate a flow-time minimization problem by jointly caching services and offloading tasks of mobile users to the UAV-aided MEC with the aim to minimize the flow-time, where the flow-time of a user request is referred to the time duration from the request issuing time point to its completion point, subject to resource and energy capacity on each UAV. We then propose a spatial-temporal learning optimization framework. We also devise an online algorithm with a competitive ratio for the problem based upon the framework, by leveraging the round-robin scheduling and dual fitting techniques. Finally, we evaluate the performance of the proposed algorithms through experimental simulation. The simulation results demonstrated that the proposed algorithms outperform their comparison counterparts, by reducing the flow-time no less than 19% on average.

Dual-track spatio-temporal learning for urban flow prediction with adaptive normalization

Robust urban flow prediction is crucial for transportation planning and management in urban areas. Although recent advances in modeling spatio-temporal correlations have shown potential, most models fail to adequately consider the complex spatio-temporal semantic information present in real-world scenarios. We summarize the following three primary limitations in existing models: a) The majority of existing models project overall time periods into the same latent space, neglecting the diverse temporal semantics between different time intervals. b) Existing models tend to capture spatial dependencies from a locale perspective such as surroundings but do not pay attention to the global influence factors. c) Beyond the spatio-temporal properties, the dynamics and instability of the data sequences introduce perturbations to the prediction results, potentially leading to model degradation. To address these issues, we propose a dual-track spatial-temporal learning module named DualST for accurate urban flow inference. To more effectively differentiate semantic information in the time dimension, we assign the overall time scales into closeness and periodicity. The dual-track module, which includes temporal causality inference and temporal contextual inference, simultaneously exploits the dynamic evolutionary trends and periodic traffic patterns, respectively. The proposed DualST captures global spatial features in a self-supervised manner which not only enriches the spatial semantics but also avoids introducing additional prior knowledge. To eliminate the instability caused by dynamics, we first adopt spatio-temporal adaptive normalization to learn appropriate data sequence normalization. We evaluate the proposed DualST on two typical urban flow datasets. The experiment results show that our model not only exhibits a consistent superiority over various state-of-the-art baselines but also has remarkable generalization capability.

Fine-grained video super-resolution via spatial-temporal learning and image detail enhancement

This paper addresses the problem for fine-grained video super-resolution (FGVSR) to suppress temporal flickering caused by separately processed consecutive frames and enhance the quality of restored video frame details when upsizing videos. Some existing video SR methods fail to sufficiently utilize spatial-temporal information from input low-resolution (LR) videos, while others may generate undesirable artifacts or cannot well reconstruct image details. To overcome these problems, we present a novel deep learning framework for FGVSR, which takes a set of consecutive LR video frames and generate the corresponding super-resolved frames. Our deep FGVSR framework focuses on reconstructing missing information from the LR sources based on the proposed multi-frame alignment and refinement strategies. More specifically, we propose an alignment module, where multiple frames are aligned at feature level, to prevent the output videos from flickering. Then, we introduce a feature fusion module, where aligned features generated from our alignment module are fused and refined in a multi-scale manner. Finally, the proposed refinement module is used to reconstruct missing information based on the fused features. In addition, we also embed an image enhancement module on the skip connection from the input layer to the output layer of our network for further enhancing the SR results. Experimental results show that the proposed deep FGVSR, compared with existing deep learning-based VSR methods, achieves state-of-the-art performances on the three well-known benchmarks, including REDS, Vid4, and Vimeo90k. More specifically, compared with the state-of-the-art VSR methods in our experiments, our FGVSR achieves quantitative improvements from 0.70 dB to 9.54 dB in PSNR. On the other hand, our method has also been shown to be efficient to other image restoration tasks, such as image inpainting.

Fault Detection for UAVs With Spatial-Temporal Learning on Multivariate Flight Data

Fault Detection for UAVs With Spatial-Temporal Learning on Multivariate Flight Data

DarLoc: Deep learning and data-feature augmentation based robust magnetic indoor localization

Magnetic-based indoor localization has attracted considerable attention due to the pervasiveness of geomagnetic fields and is free of extra infrastructures. However, existing approaches still face heterogeneity problems caused by the type of devices, the holding attitudes, and the moving speeds of pedestrians. To cope with it, we propose a novel deep learning and data-feature augmentation based magnetic localization framework (DarLoc). First, we invoke orientation-insensitive magnetic signal extraction to remove the Direct Current (DC) component of the sequence and to eliminate the impact caused by different holding attitudes and various mobile devices. Second, we propose novel data augmentation and feature augmentation methods to extract features of speed information, thus tackling the multi-scale sequence problem caused by different moving speeds. Finally, we propose a deep multi-scale spatial–temporal learning model to extract both spatial and temporal features of the augmented sequences and to provide a robust localization for people with various moving speeds and attitudes. Extensive experiments, including a total of 189 volunteers with 4 different mobile devices and multiple moving speeds, are employed to evaluate the performance of DarLoc over 14 months. Experiment results reveal that: (1) DarLoc obtains an average localization accuracy of 0.47 m and 0.56 m under constant and variable moving speeds scenarios, respectively; (2) DarLoc obtains about 41% and 60% localization accuracy improvement compared with the state-of-the-art localization methods.

Enhanced spatial-temporal learning network for dynamic facial expression recognition

The recognition of dynamic facial expressions has received increasing attention since they can better reflect the real expression process of emotion than a static image. However, due to various factors such as subtle variation differences, pose, occlusion, and illumination, it has been a challenging vision task to obtain discriminative expression features in dynamic facial expression recognition. Traditional CNN-based deep learning networks lack global and temporal contextual expression understanding, which tends to affect the final recognition of dynamic expressions. Therefore, we propose an enhanced spatial–temporal learning network (ESTLNet) for more robust dynamic facial expression recognition, which consists of a spatial fusion learning module (SFLM) and a temporal transformer enhancement module (TTEM). First, the SFLM obtains a more expressive spatial feature representation through a dual-channel feature fusion learning module. Then, the TTEM extracts more valid temporal contextual expression features based on the above spatial features through an encoder constructed by cascading a self-attention learning network and an effective gated feed-forward network. Finally, the co-enhanced spatial–temporal model approach is assessed on the four broadly used dynamic expression datasets (DFEW, AFEW, CK+, and Oulu-CASIA). Extensive experimental outcomes demonstrate that our approach surpasses several existing state-of-the-art methods, leading to notable enhancements in performance.

Enhancing Dynamic GCN for Node Attribute Forecasting with Meta Spatial-Temporal Learning (Student Abstract)

Node attribute forecasting has recently attracted considerable attention. Recent attempts have thus far utilize dynamic graph convolutional network (GCN) to predict future node attributes. However, few prior works have notice that the complex spatial and temporal interaction between nodes, which will hamper the performance of dynamic GCN. In this paper, we propose a new dynamic GCN model named meta-DGCN, leveraging meta spatial-temporal tasks to enhance the ability of dynamic GCN for better capturing node attributes in the future. Experiments show that meta-DGCN effectively modeling comprehensive spatio-temporal correlations between nodes and outperforms state-of-the-art baselines on various real-world datasets.

Typical Facial Expression Network Using a Facial Feature Decoupler and Spatial-Temporal Learning

Facial expression recognition (FER) accuracy is often affected by an individuals unique facial characteristics. Recognition performance can be improved if the influence from these physical characteristics is minimized. Using video instead of single image for FER provides better results but requires extracting temporal features and the spatial structure of facial expressions in an integrated manner. We propose a new network called Typical Facial Expression Network (TFEN) to address both challenges. TFEN uses two deep two-dimensional (2D) convolutional neural networks (CNNs) to extract facial and expression features from input video. A facial feature decoupler decouples facial features from expression features to minimize the influence from inter-subject face variations. These networks combine with a 3D CNN and form a spatial-temporal learning network to jointly explore the spatial-temporal features in a video. A facial recognition network works as an adversarial network to refine the facial feature decoupler and the network performance by minimizing the residual influence of facial features after decoupling. The whole network is trained with an adversarial algorithm to improve FER performance. TFEN was evaluated on four popular dynamic FER datasets. Experimental results show TFEN achieves or outperforms the recognition accuracy of state-of-the-art approaches.

Dynamic Multi-View Graph Neural Networks for Citywide Traffic Inference

Accurate citywide traffic inference is critical for improving intelligent transportation systems with smart city applications. However, this task is very challenging given the limited training data, due to the high cost of sensor installment and maintenance across the entire urban space. A more practical scenario to study the citywide traffic inference is effectively modeling the spatial and temporal traffic patterns with limited historical traffic observations. In this work, we propose a dynamic multi-view graph neural network for citywide traffic inference with the method CTVI+. Specifically, for the temporal dimension, we propose a temporal self-attention mechanism that is capable of learning the dynamics of traffic data with the time-evolving traffic volume variations. For spatial dimension, we build a multi-view graph neural network, employing the road-wise message passing scheme to capture the region dependencies. With the designed spatial-temporal learning paradigms, we enable our traffic inference model to encode the dynamism from both spatial and temporal traffic patterns, which is reflective of intra- and inter-road traffic correlations. In our evaluation, CTVI+ achieves consistent better performance compared with different baselines on real-world traffic volume datasets. Further ablation study validates the effectiveness of key components in CTVI+. We release the model implementation at https://github.com/dsj96/TKDD.

A graph-attention based spatial-temporal learning framework for tourism demand forecasting

Accurate tourism demand forecasting can improve tourism experiences and realize smart tourism. Existing spatial–temporal tourism demand forecasting models only explore pre-specified and static spatial connections across regions without considering multiple or dynamic spatial connections; however, this is not sufficient for modeling actual tourism demand. In this paper, we propose a graph-attention based spatial–temporal learning framework for tourism demand forecasting. A weight-dynamic multi-dimensional graph is organized to embed multiple explicit dynamic spatial connections and provide a node attribute sequence for learning implicit dynamic spatial connections. We further propose a heterogeneous spatial–temporal graph-attention network (called HSTGANet), which is effective in handling both explicit and implicit dynamic spatial connections, learning high-dimensional spatial–temporal features, and forecasting tourism demand. Experimental results demonstrate the effectiveness of the proposed model over baseline models in forecasting the tourism demand for six regions of Wanshan Archipelago in Zhuhai, China, and indicate that the proposed spatial–temporal learning framework may provide useful insights for developing more effective models for other spatial–temporal forecasting problems.

A Novel Unsupervised Spatial–Temporal Learning Mechanism in a Bio-inspired Spiking Neural Network

A Novel Unsupervised Spatial–Temporal Learning Mechanism in a Bio-inspired Spiking Neural Network

Multiplex parallel GAT-ALSTM: A novel spatial-temporal learning model for multi-sites wind power collaborative forecasting

In order to improve the accuracy of wind power output forecasting and ensure reliability of the power grid, multiplex parallel GAT-ALSTM, a spatial-temporal learning model for multi-sites wind power collaborative forecasting is proposed in this study. Topography was generated by using geographic information (longitude and latitude) obtained from the wind power generation sites. The GAT layer was used to capture the spatial correlation characteristics of multi-sites wind power. Feature dimension enhancement of each wind power generation site was achieved by aggregating the information from the adjacent sites. The ALSTM layer was used to capture the temporal correlation of each power output time series. The multiplex parallel structure of the model is designed to provide fast prediction of large-scale distributed wind power generation. The validity of the proposed multiplex parallel GAT-ALSTM was confirmed by comparison with the forecast results obtained by RNN, LSTM, ALSTM, and GNN-ALSTM. The testing results showed that, compared to RNN, LSTM, ALSTM, and GNN-ALSTM, the forecast results of the multiplex parallel GAT-ALSTM had the lowest mean absolute value error and the highest accuracy.

Deep Transfer Learning Across Cities for Mobile Traffic Prediction

Precise citywide mobile traffic prediction is of great significance for intelligent network planning and proactive service provisioning. Current traffic prediction approaches mainly focus on training a well-performed model for the cities with a large amount of mobile traffic data. However, for the cities with scarce data, the prediction performance will be greatly limited. To tackle this problem, in this paper we propose a novel cross-city deep transfer learning framework named CCTP for citywide mobile traffic prediction in cities with data scarcity. Specifically, we first present a novel spatial-temporal learning model and pre-train the model by abundant data of a source city to obtain prior knowledge of mobile traffic dynamics. We then devise an efficient generative adversarial network (GAN) based cross-domain adapter for distribution alignment between target data and source data. To deal with data scarcity issue in some clusters of target city, we further design an inter-cluster transfer learning strategy for performance enhancement. Extensive experiments conducted on real-world mobile traffic datasets demonstrate that our proposed CCTP framework can achieve superior performance in citywide mobile traffic prediction with data scarcity.

A spatial–temporal graph attention network approach for air temperature forecasting

Air temperature prediction is a significant task for researchers and forecasters in the field of meteorology. In this paper, we present an innovative, deep spatial–temporal learning air temperature forecasting framework based on graph attention network and gated recurrent unit. Particularly, historical observations containing multiple environmental variables at different stations are constructed as graph signals. The original stations’ conditions and the learned attention information are all included in our model, which overcomes the flaw of the conventional graph network approach. Results of experiments on a real-world dataset demonstrate that, compared to the state-of-the-art baselines, our model achieves the best performance in terms of short-, middle- and long-term air temperature predictions.