Spatiotemporal Dependence Research Articles

Prior-free 3D human pose estimation in a video using limb-vectors

Estimating accurate 3D human poses from a monocular video is fundamental to various computer vision tasks. Existing methods exploit 2D-to-3D pose lifting, multiview images, and depth sensors to model spatio-temporal dependencies. However, depth ambiguities, occlusions, and larger temporal receptive fields pose challenges to these approaches. To address this, we propose a novel prior-free DCNN-based 3D human pose estimation method for monocular image sequences using limb vectors. Our method comprises two subnetworks: a limb direction estimator and a limb length estimator. The limb direction estimator utilizes a fully convolutional network to model limb direction vectors across a temporal window. We show that network complexity can be significantly reduced by utilizing dilated convolutional operations and a relatively smaller receptive field while maintaining estimation accuracy. Moreover, the limb length estimator captures stable limb length estimations from a reliable frame set. Our model has shown superior performance compared to existing methods on the Human3.6M and MPI-INF-3DHP datasets.

ESMformer: Error-aware self-supervised transformer for multi-view 3D human pose estimation

Multi-view 3D human pose estimation (HPE) currently faces several key challenges. Information from different viewpoints exhibits high variability due to complex environmental factors, posing difficulties in cross-view feature extraction and fusion. Additionally, multi-view 3D labeled pose data is rather scarce, and the impact of input 2D poses on 3D HPE accuracy has received little attention. To address these issues, we propose an Error-aware Self-supervised transformer framework for Multi-view 3D HPE (ESMformer). Firstly, we introduce a single-view multi-level feature extraction module to enhance pose features in individual viewpoints, which incorporates a novel relative attention mechanism for representative feature extraction at different levels. Subsequently, we develop multi-view intra-level and cross-level fusion modules to exploit spatio-temporal feature dependencies among human joints, and progressively fuse pose information from all views and levels. Furthermore, we explore an error-aware self-supervised learning strategy to reduce the model’s reliance on 3D pose annotations and mitigate the impact of incorrect 2D poses. This strategy adaptively selects reliable input 2D poses based on 3D pose prediction errors. Experiments on three popular benchmarks show that ESMformer achieves state-of-the-art results and maintains cost-effective computational complexity. Notably, ESMformer does not rely on any 3D pose annotations or prior human body knowledge, making it highly versatile and adaptable in practical applications.11The code and models are available at https://github.com/z0911k/ESMformer.

Learning integral operators via neural integral equations

Nonlinear operators with long-distance spatiotemporal dependencies are fundamental in modelling complex systems across sciences; yet, learning these non-local operators remains challenging in machine learning. Integral equations, which model such non-local systems, have wide-ranging applications in physics, chemistry, biology and engineering. We introduce the neural integral equation, a method for learning unknown integral operators from data using an integral equation solver. To improve scalability and model capacity, we also present the attentional neural integral equation, which replaces the integral with self-attention. Both models are grounded in the theory of second-kind integral equations, where the indeterminate appears both inside and outside the integral operator. We provide a theoretical analysis showing how self-attention can approximate integral operators under mild regularity assumptions, further deepening previously reported connections between transformers and integration, as well as deriving corresponding approximation results for integral operators. Through numerical benchmarks on synthetic and real-world data, including Lotka–Volterra, Navier–Stokes and Burgers’ equations, as well as brain dynamics and integral equations, we showcase the models’ capabilities and their ability to derive interpretable dynamics embeddings. Our experiments demonstrate that attentional neural integral equations outperform existing methods, especially for longer time intervals and higher-dimensional problems. Our work addresses a critical gap in machine learning for non-local operators and offers a powerful tool for studying unknown complex systems with long-range dependencies.

Multi-Scale Spatio-Temporal Attention Networks for Network-Scale Traffic Learning and Forecasting.

Accurate and timely forecasting of traffic on local road networks is crucial for deploying effective dynamic traffic control, advanced route planning, and navigation services. This task is particularly challenging due to complex spatio-temporal dependencies arising from non-Euclidean spatial relations in road networks and non-linear temporal dynamics influenced by changing road conditions. This paper introduces the spatio-temporal network embedding (STNE) model, a novel deep learning framework tailored for learning and forecasting graph-structured traffic data over extended input sequences. Unlike traditional convolutional neural networks (CNNs), the model employs graph convolutional networks (GCNs) to capture the spatial characteristics of local road network topologies. Moreover, the segmentation of very long input traffic data into multiple sub-sequences, based on significant temporal properties such as closeness, periodicity, and trend, is performed. Multi-dimensional long short-term memory neural networks (MDLSTM) are utilized to flexibly access multi-dimensional context. Experimental results demonstrate that the STNE model surpasses state-of-the-art traffic forecasting benchmarks on two large-scale real-world traffic datasets.

Spatio-temporal feature extraction network based multi-performance indicators synergetic monitoring method for complex industrial processes

In the context of smart manufacturing, modern industrial processes are becoming increasingly complex in terms of process flows, product varieties, and performance indicators (PIs). Performance-driven process monitoring attracts extensive attentions in recent years. However, most methods require spatio-temporal correspondence between process variables and PIs, and rarely consider the correlation among various PIs. In this paper, a spatio-temporal feature extraction network-based multi-performance indicators synergetic monitoring framework is presented. Firstly, considering the missing data in PI measurements, a weighted sum of tensor nuclear norm (WSTNN) based batch-process data completion approach is developed, which can adeptly handle local missing and incomplete data issues and establish the spatio-temporal correspondence for subsequent modeling. Secondly, for a specific PI, a new canonical variate analysis embedded spatio-temporal convolutional network (CVA-STCN) is designed to extract the PI-related features with spatio-temporal dependency. Thirdly, considering the dynamic interaction of multiple PIs, a third-order feature tensor is established to perform the future fusion, and the correlations among various PI-related features are explored via tensor decomposition. Finally, a hierarchical multi-performance indicators synergetic monitoring model is developed over several subspaces. The proposed method is verified on Tennessee Eastman process and an actual hot strip mill process. Overall, the monitoring performance of the proposed method outperforms the traditional ones, in terms of higher fault detection rates and lower false alarm rates. Moreover, the information provided by the multi-subspace synergetic monitoring charts can offer valuable guidance to field engineers.

Short-Term Precipitation Radar Echo Extrapolation Method Based on the MS-DD3D-RSTN Network and STLoss Function.

Short-term precipitation forecasting is essential for agriculture, transportation, urban management, and tourism. The radar echo extrapolation method is widely used in precipitation forecasting. To address issues like forecast degradation, insufficient capture of spatiotemporal dependencies, and low accuracy in radar echo extrapolation, we propose a new model: MS-DD3D-RSTN. This model employs spatiotemporal convolutional blocks (STCBs) as spatiotemporal feature extractors and uses the spatial-temporal loss (STLoss) function to learn intra-frame and inter-frame changes for end-to-end training, thereby capturing the spatiotemporal dependencies in radar echo signals. Experiments on the Sichuan dataset and the HKO-7 dataset show that the proposed model outperforms advanced models in terms of CSI and POD evaluation metrics. For 2 h forecasts with 20 dBZ and 30 dBZ reflectivity thresholds, the CSI metrics reached 0.538, 0.386, 0.485, and 0.198, respectively, representing the best levels among existing methods. The experiments demonstrate that the MS-DD3D-RSTN model enhances the ability to capture spatiotemporal dependencies, mitigates forecast degradation, and further improves radar echo prediction performance.

A multi-scale attributes fusion model for travel mode identification using GPS trajectories

ABSTRACT Travel mode recognition is a key issue in urban planning and transportation research. While traditional travel surveys use manual data collection and have limited coverage, poor timeliness, and insufficient sample capacity, recent advancements in Global Positioning System (GPS) technology allow large-scale data collection and offer novel opportunities to enhance travel mode recognition. However, existing studies often neglect regular differences and changes in motion states across different travel modes and fail to fully integrate multi-scale spatio-temporal features, which limits the accurate classification of travel modes. To fill this gap, this study proposes a multi-scale spatio-temporal attribute fusion (MSAF) model for precise travel mode identification using solely GPS trajectories without altering their sampling rate. The MSAF model segments GPS trajectories into various temporal and spatial scales, extracting local motion states and spatial features at multiple scales. The spatio-temporal feature extraction module is constructed to extract local motion states and capture spatio-temporal dependencies. Additionally, the model incorporates a multi-scale feature fusion module, which effectively combines features of various scales through a series of fusion techniques to obtain a comprehensive representation, enabling automatic and accurate travel mode identification. Experiments on real-world datasets, including the GeoLife Trajectories dataset and the Sussex-Huawei Locomotion-Transportation (SHL) dataset, demonstrate the effectiveness of the MSAF model, achieving a competitive accuracy of 95.16% and 91.70%. This represents an improvement of 2.50% to 7.95% and 0.8% to 6.62% over several state-of-the-art baselines, effectively addressing sample imbalance challenges. Moreover, the experiments demonstrate the significant role of multiscale feature fusion in improving model performance.

Remaining useful life prediction of Lithium-ion batteries using spatio-temporal multimodal attention networks

Lithium-ion batteries are widely used in various applications, including electric vehicles and renewable energy storage. The prediction of the remaining useful life (RUL) of batteries is crucial for ensuring reliable and efficient operation, as well as reducing maintenance costs. However, determining the life cycle of batteries in real-world scenarios is challenging, and existing methods have limitations in predicting the number of cycles iteratively. In addition, existing works often oversimplify the datasets, neglecting important features of the batteries such as temperature, internal resistance, and material type. To address these limitations, this paper proposes a two-stage RUL prediction scheme for Lithium-ion batteries using a spatio-temporal multimodal attention network (ST-MAN). The proposed ST-MAN is to capture the complex spatio-temporal dependencies in the battery data, including the features that are often neglected in existing works. Despite operating without prior knowledge of end-of-life (EOL) events, our method consistently achieves lower error rates, boasting mean absolute error (MAE) and mean square error (MSE) of 0.0275 and 0.0014, respectively, compared to existing convolutional neural networks (CNN) and long short-term memory (LSTM)-based methods. The proposed method has the potential to improve the reliability and efficiency of battery operations and is applicable in various industries.

Detection of spatiotemporal changepoints: a generalised additive model approach

The detection of changepoints in spatio-temporal datasets has been receiving increased focus in recent years and is utilised in a wide range of fields. With temporal data observed at different spatial locations, the current approach is typically to use univariate changepoint methods in a marginal sense with the detected changepoint being representative of a single location only. We present a spatio-temporal changepoint method that utilises a generalised additive model (GAM) dependent on the 2D spatial location and the observation time to account for the underlying spatio-temporal process. We use the full likelihood of the GAM in conjunction with the pruned linear exact time (PELT) changepoint search algorithm to detect multiple changepoints across spatial locations in a computationally efficient manner. When compared to a univariate marginal approach our method is shown to perform more efficiently in simulation studies at detecting true changepoints and demonstrates less evidence of overfitting. Furthermore, as the approach explicitly models spatio-temporal dependencies between spatial locations, any changepoints detected are common across the locations. We demonstrate an application of the method to an air quality dataset covering the COVID-19 lockdown in the United Kingdom.

Unraveling the propagation dynamics of passenger congestion in public transportation systems using big smart card data

We propose a comprehensive framework to explore propagation of passenger crowdedness in public transportation systems using 74.16 million trip chains. A deep learning method HSTGCNT is adopted which firstly models normal spatio-temporal dependencies in time series data, and assigns an anomaly score to new data based on its prediction error. The derived discrete anomaly events are further aggregated into groups based on their spatio-temporal proximity by using a clustering algorithm that binds shared nearest neighbors together based on pairwise correlations. The clustering results uncover several key stages located in a propagation chain, such as a core event, adjacent events with spatio-temporal proximity, or a broadcasting group. Weibo data serves as a third-party dataset to verify the above anomaly clusters. Results show that the framework is effective to detect outlier clusters, with recall value being 0.86. Findings may assist public safety departments in taking pro-active actions to interrupt chaos from broadcasting.

Predicting urban signal-controlled intersection congestion events using spatio-temporal neural point process

ABSTRACT The urban traffic signal-controlled intersections are of great significance for solving the problem of urban road congestion. Previous research on congestion prediction mainly aggregated data at the level of road segments or traffic flow at a coarse regulated time interval. Fine-grained prediction of congestion events at the lane-level and cycle-level enables detailed a understanding of spatio-temporal dependencies, leading to congestion reduction, improved efficiency. This paper presents a Spatio-Temporal Neural Point Process (STNPP) model that combines Graph Neural Networks and Neural Temporal Point Process to predict congestion events at urban intersections. The proposed model allows for complete prediction of congestion events, including their occurrence, development, dissipation. In the process of spatial correlation modeling, graph neural networks are used to model the spatial relationships between both region and intersections. The current intersection and its upstream/downstream areas are modeled separately. To model the temporal correlations at individual intersections, we focus on a specific lane and capture the evolution of congestion events using the Neural Point Process Gated Recurrent Unit (NPPGRU), which captures the temporal granularity changes of signal-controlled cycles in congestion events. Using actual traffic speed and signal-controlled data from Hangzhou city, we validate that the proposed method achieves stable predictive performance.

Long-term wind power forecasting with series decomposition and spatio-temporal graph neural network

ABSTRACT The stable operation of the power grid requires accurate predictions of wind power generation and the stabilization of its fluctuations through the integration of other energy sources. An increasing number of deep learning methods are now being employed in the field. However, due to the instability of wind power data, existing methods struggle to uncover deep spatiotemporal dependencies. We propose a novel method named SD-STGNN (Series Decomposition and Spatio-Temporal Graph Neural Network). SD-STGNN first decomposes unstable wind power data into seasonal and trend components. For the seasonal data reflecting short-term fluctuation patterns, we employ a Gated Temporal Convolutional Network and Graph Convolutional Network to capture spatiotemporal relationships. Additionally, for trend data reflecting long-term fluctuation patterns, we introduce a Temporal-Feature Enhancement module, utilizing Multi-Layer Perceptrons to extract deep information along both temporal and feature dimensions. Extensive experiments were conducted on the SDWPF public dataset. Compared to existing state-of-the-art baseline methods, our proposed SD-STGNN model achieves a notable average reduction in Mean Absolute Error by approximately 6.26%, in Root Mean Squared Error by 7.55%, and in Mean Absolute Percentage Error by 2.65%. Additionally, there is an average improvement of about 4.74% in the coefficient of determination.

Short-Term Origin-Destination Demand Prediction Based on Spatiotemporal Encoder-Decoder Network with a Residual Feature Extractor

Online ride-hailing services play a crucial role in daily transportation, However, challenges persist in certain regions with limited access, and drivers encounter difficulties in receiving orders. Accurate prediction of short-term origin-destination (OD) demand is crucial for addressing these issues. This study leverages recent advancements in artificial intelligence and big data to introduce a spatiotemporal encoder-decoder network with a residual feature extractor (RF-STED) for short-term OD demand prediction in online ride-hailing services. The RF-STED model, built on deep learning models such as graph convolutional networks and convolutional long short-term memory (Conv-LSTM), includes spatiotemporal networks, encoding layers, and a residual feature extractor. The spatiotemporal network has two branches: branch one processes multi-pattern OD data using a multi-pattern temporal feature extraction module, utilizing a multi-channel Conv-LSTM to capture temporal correlations. Branch two utilizes a multi-spatial feature extraction module to convert OD pair associations into a spatial topology, extracting multi-spatial correlations. The encoding layer captures spatiotemporal dependencies, while the residual feature extractor decodes compressed vectors back into an OD graph for forecasting future demand. Experiments with a Manhattan taxi dataset in the U.S. show the RF-STED model outperforms 10 baseline models and four ablation models. The results emphasize the model’s strength and robustness in short-term OD flow prediction.

Scientific logic and spatio-temporal dependence in analyzing extreme-precipitation frequency: negligible or neglected?

Abstract. Statistics is often misused in hydro-climatology, thus causing research to get stuck on unscientific concepts that hinder scientific advances. In particular, neglecting the scientific rationale of statistical inference results in logical and operational fallacies that prevent the discernment of facts, assumptions, and models, thus leading to systematic misinterpretations of the output of data analysis. This study discusses how epistemological principles are not just philosophical concepts but also have very practical effects. To this aim, we focus on the iterated underestimation and misinterpretation of the role of spatio-temporal dependence in statistical analysis of hydro-climatic processes by analyzing the occurrence process of extreme precipitation (P) derived from 100-year daily time series recorded at 1106 worldwide gauges of the Global Historical Climatology Network. The analysis contrasts a model-based approach that is compliant with the well-devised but often neglected logic of statistical inference and a widespread but theoretically problematic test-based approach relying on statistical hypothesis tests applied to unrepeatable hydro-climatic records. The model-based approach highlights the actual impact of spatio-temporal dependence and a finite sample size on statistical inference, resulting in over-dispersed marginal distributions and biased estimates of dependence properties, such as autocorrelation and power spectrum density. These issues also affect the outcome and interpretation of statistical tests for trend detection. Overall, the model-based approach results in a theoretically coherent modeling framework where stationary stochastic processes incorporating the empirical spatio-temporal correlation and its effects provide a faithful description of the occurrence process of extreme P at various spatio-temporal scales. On the other hand, the test-based approach leads to theoretically unsubstantiated results and interpretations, along with logically contradictory conclusions such as the simultaneous equi-dispersion and over-dispersion of extreme P. Therefore, accounting for the effect of dependence in the analysis of the frequency of extreme P has a huge impact that cannot be ignored, and, more importantly, any data analysis can be scientifically meaningful only if it considers the epistemological principles of statistical inference such as the asymmetry between confirmatory and disconfirmatory empiricism, the inverse-probability problem affecting statistical tests, and the difference between assumptions and models.

Spatio-Temporal Parallel Transformer based model for Traffic Prediction

Traffic forecasting problems involve jointly modeling the non-linear spatio-temporal dependencies at different scales. While Graph Neural Network models have been effectively used to capture the non-linear spatial dependencies, capturing the dynamic spatial dependencies between the locations remains a major challenge. The errors in capturing such dependencies propagate in modeling the temporal dependencies between the locations, thereby severely affecting the performance of long-term predictions. While transformer-based mechanisms have been recently proposed for capturing the dynamic spatial dependencies, these methods are susceptible to fluctuations in data brought on by unforeseen events like traffic congestion and accidents. To mitigate these issues we propose an improvised Spatio-temporal parallel transformer (STPT) based model for traffic prediction that uses multiple adjacency graphs passed through a pair of coupled graph transformer-convolution network units, operating in parallel, to generate more noise-resilient embeddings. We conduct extensive experiments on 4 real-world traffic datasets and compare the performance of STPT with several state-of-the-art baselines, in terms of measures like RMSE, MAE, and MAPE. We find that using STPT improves the performance by around \(10-34\%\) as compared to the baselines. We also investigate the applicability of the model on other spatio-temporal data in other domains. We use a covid-19 dataset to predict the number of future occurrences in different regions from a given set of historical occurrences. The results demonstrate the superiority of our model for such datasets.

DPP: A Novel Disease Progression Prediction Method for Ginkgo Leaf Disease Based on Image Sequences

Ginkgo leaf disease poses a grave threat to Ginkgo biloba. The current management of Ginkgo leaf disease lacks precision guidance and intelligent technologies. To provide precision guidance for disease management and to evaluate the effectiveness of the implemented measures, the present study proposes a novel disease progression prediction (DPP) method for Ginkgo leaf blight with a multi-level feature translation architecture and enhanced spatiotemporal attention module (eSTA). The proposed DPP method is capable of capturing key spatiotemporal dependencies of disease symptoms at various feature levels. Experiments demonstrated that the DPP method achieves state-of-the-art prediction performance in disease progression prediction. Compared to the top-performing spatiotemporal predictive learning method (SimVP + TAU), our method significantly reduced the mean absolute error (MAE) by 19.95% and the mean square error (MSE) by 25.35%. Moreover, it achieved a higher structure similarity index measure (SSIM) of 0.970 and superior peak signal-to-noise ratio (PSNR) of 37.746 dB. The proposed method can accurately forecast the progression of Ginkgo leaf blight to a large extent, which is expected to provide valuable insights for precision and intelligent disease management. Additionally, this study presents a novel perspective for the extensive research on plant disease prediction.

Convolutional Non-Homogeneous Poisson Process and its Application to Wildfire Ignition Risk Quantification for Power Delivery Networks

To quantify wildfire ignition risks on power delivery networks, the current practice predominantly relies on the empirically calculated fire danger indices, which may not well capture the effects of dynamically changing environmental factors. This article proposes a spatio-temporal point process model, known as the Convolutional Non-homogeneous Poisson Process (cNHPP), and applies the model to quantify wildfire ignition risks for power delivery networks. The proposed model captures both the current (i.e., instantaneous) and cumulative (i.e., historical) effects of key environmental processes (i.e., covariates) on wildfire risks, as well as the spatio-temporal dependency among different segments of the power delivery network. The computation and interpretation of the intensity function are thoroughly investigated. We apply the proposed approach to estimate wildfire ignition risks on major transmission lines in California, using historical fire data, meteorological and vegetation data obtained from the National Oceanic and Atmospheric Administration and National Aeronautics and Space Administration. A comprehensive comparison study is performed to show the applicability and predictive capability of the proposed approach.

Traffic Congestion Prediction Using Graph Convolution Networks

Traffic congestion is a critical issue in urban transportation systems, leading to increased travel times, fuel consumption, and air pollution. Furthermore, accurate prediction of traffic conditions is essential for effective traffic management and route planning. However, traditional approaches often fail to capture the complex spatiotemporal dependencies inherent in road networks. This study compares the performance of Graph Convolutional Networks (GCNs) for traffic congestion prediction using Amsterdam sensor location datasets from 13 locations (01-10-2023 to 31-10-2023) and 18 locations (01-01-2024 to 26-01-2024). The GCN model achieves an accuracy above 0.5, with a peak accuracy of 0.6 for the 18-location dataset and 0.55 for the 13-location dataset. Precision ranges from 0.5 to 0.8, while recall oscillates between 0.5 and 0.6. Also, the F1-score reaches 0.6 for the 18-location dataset and remains above 0.4 for the 13-location dataset. The results demonstrate the GCN's effectiveness in capturing spatial dependencies and achieving high-performance metrics, with better performance observed for larger datasets. Moreover, the findings contribute to the development of intelligent schemes for GCNs and the Internet of Vehicles in Intelligent Transportation Systems (ITS), advancing traffic congestion prediction capabilities.

Dynamic Spatio-Temporal Adaptive Graph Convolutional Recurrent Networks for Vacant Parking Space Prediction

The prediction of vacant parking spaces (VPSs) can reduce the time drivers spend searching for parking, thus alleviating traffic congestion. However, previous studies have mostly focused on modeling the temporal features of VPSs using historical data, neglecting the complex and extensive spatial characteristics of different parking lots within the transportation network. This is mainly due to the lack of direct physical connections between parking lots, making it challenging to quantify the spatio-temporal features among them. To address this issue, we propose a dynamic spatio-temporal adaptive graph convolutional recursive network (DSTAGCRN) for VPS prediction. Specifically, DSTAGCRN divides VPS data into seasonal and periodic trend components and combines daily and weekly information with node embeddings using the dynamic parameter-learning module (DPLM) to generate dynamic graphs. Then, by integrating gated recurrent units (GRUs) with the parameter-learning graph convolutional recursive module (PLGCRM) of DPLM, we infer the spatio-temporal dependencies for each time step. Furthermore, we introduce a multihead attention mechanism to effectively capture and fuse the spatio-temporal dependencies and dynamic changes in the VPS data, thereby enhancing the prediction performance. Finally, we evaluate the proposed DSTAGCRN on three real parking datasets. Extensive experiments and analyses demonstrate that the DSTAGCRN model proposed in this study not only improves the prediction accuracy but can also better extract the dynamic spatio-temporal characteristics of available parking space data in multiple parking lots.

Vehicle Behavior Discovery and Three-Dimensional Object Detection and Tracking Based on Spatio-Temporal Dependency Knowledge and Artificial Fish Swarm Algorithm.

In complex traffic environments, 3D target tracking and detection are often occluded by various stationary and moving objects. When the target is occluded, its apparent characteristics change, resulting in a decrease in the accuracy of tracking and detection. In order to solve this problem, we propose to learn the vehicle behavior from the driving data, predict and calibrate the vehicle trajectory, and finally use the artificial fish swarm algorithm to optimize the tracking results. The experiments show that compared with the CenterTrack method, the proposed method improves the key indicators of MOTA (Multi-Object Tracking Accuracy) in 3D object detection and tracking on the nuScenes dataset, and the frame rate is 26 fps.