Spatial-temporal Modeling Research Articles

Neural dynamical operator: Continuous spatial-temporal model with gradient-based and derivative-free optimization methods

Data-driven modeling techniques have been explored in the spatial-temporal modeling of complex dynamical systems for many engineering applications. However, a systematic approach is still lacking to leverage the information from different types of data, e.g., with different spatial and temporal resolutions, and the combined use of short-term trajectories and long-term statistics. In this work, we build on the recent progress of neural operator and present a data-driven modeling framework called neural dynamical operator that is continuous in both space and time. A key feature of the neural dynamical operator is the resolution-invariance with respect to both spatial and temporal discretizations, without demanding abundant training data in different temporal resolutions. To improve the long-term performance of the calibrated model, we further propose a hybrid optimization scheme that leverages both gradient-based and derivative-free optimization methods and efficiently trains on both short-term time series and long-term statistics. We investigate the performance of the neural dynamical operator with three numerical examples, including the viscous Burgers' equation, the Navier–Stokes equations, and the Kuramoto–Sivashinsky equation. The results confirm the resolution-invariance of the proposed modeling framework and also demonstrate stable long-term simulations with only short-term time series data. In addition, we show that the proposed model can better predict long-term statistics via the hybrid optimization scheme with a combined use of short-term and long-term data.

Hotel demand forecasting with multi-scale spatiotemporal features

Accurate demand forecasting is critical to hotel revenue management and related decision-making. Considering the heterogeneity and dynamics of spatial effects across different time scales, this study introduces a novel model which can deeply extract these features to improve the forecasting performance of hotel demand. Specifically, the model constructs input variables with different periodicities and then integrates a Transformer neural network and long short-term memory to extract multi-scale and dynamic spatiotemporal features to generate accurate forecasts. The effectiveness of the model is verified through an empirical case in Xiamen, China. Results suggest our model significantly outperforms benchmarks in terms of accuracy and robustness. The findings extend the application of spatial-temporal modeling in hotel demand forecasting. Hotel managers can use our forecasts to optimize operations, improve revenues, and control risks. The extracted spatiotemporal features can also help managers examine cooperation and competition relationships with neighbor hotels.

Edit Temporal-Consistent Videos with Image Diffusion Model

Large-scale text-to-image (T2I) diffusion models have been extended for text-guided video editing, yielding impressive zero-shot video editing performance. Nonetheless, the generated videos usually show spatial irregularities and temporal inconsistencies as the temporal characteristics of videos have not been faithfully modeled. In this paper, we propose an elegant yet effective Temporal-Consistent Video Editing (TCVE) method to mitigate the temporal inconsistency challenge for robust text-guided video editing. In addition to the utilization of a pretrained T2I 2D Unet for spatial content manipulation, we establish a dedicated temporal Unet architecture to faithfully capture the temporal coherence of the input video sequences. Furthermore, to establish coherence and interrelation between the spatial-focused and temporal-focused components, a cohesive spatial-temporal modeling unit is formulated. This unit effectively interconnects the temporal Unet with the pretrained 2D Unet, thereby enhancing the temporal consistency of the generated videos while preserving the capacity for video content manipulation. Quantitative experimental results and visualization results demonstrate that TCVE achieves state-of-the-art performance in both video temporal consistency and video editing capability, surpassing existing benchmarks in the field. Codes are released at https://github.com/mdswyz/TCVE .

Prediction of novel malware using hybrid convolution neural network and long short-term memory approach

The rapid evolution of network communication technologies has led to the emergence of new forms of malware and cybercrimes, posing significant threats to user safety, network infrastructure integrity, and data privacy. Despite efforts to develop advanced algorithms for detecting malicious activity, constructing models that are both accurate and reliable remains a challenge, especially in handling vast and dynamically shifting data patterns. The prevalent bag-of-words (BOW) method, while widely used, falls short in capturing crucial spatial and sequence information vital for detecting malware patterns. To address this challenge, the work presented in this paper proposes hybrid convolution neural network-long short-term memory network (CNN-LSTM) combination models, leveraging CNN's spatial information extraction and LSTM's temporal modeling capabilities. Focused on predicting the infiltration of malicious software into personal computers, the proposed hybrid CNN-LSTM model considers factors such as location, firmware version, operating system, and anti-virus software. The proposed models undergo training and evaluation using Microsoft's malware dataset, demonstrating superior performance compared to traditional CNN and LSTM models. The CNN-LSTM model achieves an impressive accuracy of 95% on the Microsoft malware dataset, highlighting its effectiveness in malware detection.

Adaptive Scheduling of High-Availability Drone Swarms for Congestion Alleviation in Connected Automated Vehicles

The Intelligent Transportation System (ITS) serves as a pivotal element within urban networks, offering decision support to users and connected automated vehicles (CAVs) through comprehensive information gathering, sensing, device control, and data processing. Presently, ITS predominantly relies on sensors embedded in fixed infrastructure, notably Roadside Units (RSUs). However, RSUs are confined by coverage limitations and may encounter challenges in prompt emergency responses. On-demand resources, such as drones, present a viable option to supplement these deficiencies effectively. This paper introduces an approach where Software-Defined Networking (SDN) and Mobile Edge Computing (MEC) technologies are integrated to formulate a high-availability drone swarm control and communication infrastructure framework, comprising the cloud layer, edge layer, and device layer. Drones confront limitations in flight duration attributed to battery limitations, posing a challenge in sustaining continuous monitoring of road conditions over extended periods. Effective drone scheduling stands as a promising solution to overcome these constraints. To tackle this issue, we initially utilized Graph WaveNet, a specialized graph neural network structure tailored for spatial-temporal graph modeling, for training a congestion prediction model using real-world dataset inputs. Building upon this, we further propose an algorithm for drone scheduling based on congestion prediction. Our simulation experiments using real-world data demonstrate that, compared to the baseline method, the proposed scheduling algorithm not only yielded superior scheduling gains but also mitigated drone idle rates.

Spatial-temporal modeling of oil condition monitoring: A review

Lubricating oil plays a vital role as the information carrier for equipment tribological performance. Therefore, oil condition monitoring (OCM) serves as a crucial technology for assessing and predicting state degradation, providing the first-line defense against functional failure. However, limited by random fluctuation and insufficient knowledge, the modeling of OCM has suffered from uncertainty problems, leading to poor applicability and insufficient generalizability. The existing reviews are less elaborated on the information uncertainty for description, characterization, and treatment, which expresses the essence of modeling constraints. To bridge this gap, the paper provides a comprehensive review of the oil state modeling in terms of uncertainty. The existing methods and metrics are reviewed from the perspective of the spatial oil state assessment (OCA) and temporal remaining useful life (RUL) prediction modeling. Further, the existing methods are analyzed to solve the uncertainty problem, and then the solutions in oil state modeling considering uncertainty are discussed. Finally, targeting the challenges of the monitoring technology, the future trends of OCM are presented.

Spatial-temporal modeling of the relationship between surface temperature and air temperature in metropolitan urban systems

Research about urban local climate and urban heat island often relies on land surface temperature (LST) data to characterize the distribution of temperature near the surface. Although using remotely sensed data for such work has the advantage of continuous spatial coverage at regular temporal intervals, it is recognized that surface temperature is not an ideal proxy for air temperature (AT). This study's goal is to develop a spatiotemporal model revealing the relationship between LST and AT within the complexities of the urban environment. A mobile weather monitoring unit was used to collect spatially-explicit fine-scale AT data while Landsat 8 and 9 passed overhead collecting LST data. A spatiotemporal model of the relationship between LST and AT in Philadelphia was constructed with this data utilizing basis functions to account for spatial and temporal autocorrelation. The spatiotemporal model results show a strong relationship between LST and AT and indicate that it is possible to predict fine scale AT (120 m) using remotely sensed LST in an urban context (r-squared = 0.99, RMSE = 0.89 °C). The spatiotemporal model outperforms models that do not account for spatial and temporal autocorrelation, highlighting the importance of considering these dependencies in temperature modeling. City-wide AT predictions were generated for Philadelphia demonstrating the ability of the model to improve understanding of local urban climate.

Spatial-Temporal Causality Modeling for Industrial Processes With a Knowledge-Data Guided Reinforcement Learning

Spatial-Temporal Causality Modeling for Industrial Processes With a Knowledge-Data Guided Reinforcement Learning

Motion sensitive network for action recognition in control and decision-making of autonomous systems.

Spatial-temporal modeling is crucial for action recognition in videos within the field of artificial intelligence. However, robustly extracting motion information remains a primary challenge due to temporal deformations of appearances and variations in motion frequencies between different actions. In order to address these issues, we propose an innovative and effective method called the Motion Sensitive Network (MSN), incorporating the theories of artificial neural networks and key concepts of autonomous system control and decision-making. Specifically, we employ an approach known as Spatial-Temporal Pyramid Motion Extraction (STP-ME) module, adjusting convolution kernel sizes and time intervals synchronously to gather motion information at different temporal scales, aligning with the learning and prediction characteristics of artificial neural networks. Additionally, we introduce a new module called Variable Scale Motion Excitation (DS-ME), utilizing a differential model to capture motion information in resonance with the flexibility of autonomous system control. Particularly, we employ a multi-scale deformable convolutional network to alter the motion scale of the target object before computing temporal differences across consecutive frames, providing theoretical support for the flexibility of autonomous systems. Temporal modeling is a crucial step in understanding environmental changes and actions within autonomous systems, and MSN, by integrating the advantages of Artificial Neural Networks (ANN) in this task, provides an effective framework for the future utilization of artificial neural networks in autonomous systems. We evaluate our proposed method on three challenging action recognition datasets (Kinetics-400, Something-Something V1, and Something-Something V2). The results indicate an improvement in accuracy ranging from 1.1% to 2.2% on the test set. When compared with state-of-the-art (SOTA) methods, the proposed approach achieves a maximum performance of 89.90%. In ablation experiments, the performance gain of this module also shows an increase ranging from 2% to 5.3%. The introduced Motion Sensitive Network (MSN) demonstrates significant potential in various challenging scenarios, providing an initial exploration into integrating artificial neural networks into the domain of autonomous systems.

Cell4D: a general purpose spatial stochastic simulator for cellular pathways

BackgroundWith the generation of vast compendia of biological datasets, the challenge is how best to interpret ‘omics data alongside biochemical and other small-scale experiments to gain meaningful biological insights. Key to this challenge are computational methods that enable domain-users to generate novel hypotheses that can be used to guide future experiments. Of particular interest are flexible modeling platforms, capable of simulating a diverse range of biological systems with low barriers of adoption to those with limited computational expertise.ResultsWe introduce Cell4D, a spatial-temporal modeling platform combining a robust simulation engine with integrated graphics visualization, a model design editor, and an underlying XML data model capable of capturing a variety of cellular functions. Cell4D provides an interactive visualization mode, allowing intuitive feedback on model behavior and exploration of novel hypotheses, together with a non-graphics mode, compatible with high performance cloud compute solutions, to facilitate generation of statistical data. To demonstrate the flexibility and effectiveness of Cell4D, we investigate the dynamics of CEACAM1 localization in T-cell activation. We confirm the importance of Ca2+ microdomains in activating calmodulin and highlight a key role of activated calmodulin on the surface expression of CEACAM1. We further show how lymphocyte-specific protein tyrosine kinase can help regulate this cell surface expression and exploit spatial modeling features of Cell4D to test the hypothesis that lipid rafts regulate clustering of CEACAM1 to promote trans-binding to neighbouring cells.ConclusionsThrough demonstrating its ability to test and generate hypotheses, Cell4D represents an effective tool to help integrate knowledge across diverse, large and small-scale datasets.

Video-MLP: Convolution-free, attention-free architecture for video classification

Convolutional neural networks (CNNs) and Transformer architectures have traditionally been recognized as the preferred models for addressing computer vision tasks. However, there has been a recent surge in the popularity of networks based on multi-layer perceptron (MLP) structures that do not rely on convolution or attention mechanisms. These MLP architectures have demonstrated exceptional performance in image classification tasks, exhibiting lower time complexity while maintaining high accuracy. In contrast, video classification tasks involve larger amounts of data and necessitate more intricate feature extraction, resulting in increased time and resource consumption. To enhance computational efficiency and minimize resource utilization, we propose a convolution-free and Transformer-free architecture for video classification called Video-MLP for video classification. Video-MLP utilizes a simple MLP structure to learn video features. Specifically, it comprises two types of layers: Spatial-Mixer and Temporal-Mixer, which respectively capture spatial and temporal information. The Spatial-Mixer extracts spatial information from each frame along the height and width dimensions, while the Temporal-Mixer models temporal information for the same spatial positions across frames. To improve the efficiency of spatial-temporal modeling in our network, we used a spatial-temporal information fusion approach to integrate information at different scales. Additionally, we grouped the input data along the time dimension and designed three different grouping schemes when extracting temporal information. The experimental results indicate that Video-MLP achieved accuracy rates of 87.2% on the Kinetics-400 dataset and 75.3% on the SomethingV2 dataset, outperforming models with equivalent computational complexity. Notably, Video-MLP achieved these results without using convolution and attention mechanisms, and without pre-training on large-scale image and video datasets.

Dynamic multiple-graph spatial-temporal synchronous aggregation framework for traffic prediction in intelligent transportation systems.

Accurate traffic prediction contributes significantly to the success of intelligent transportation systems (ITS), which enables ITS to rationally deploy road resources and enhance the utilization efficiency of road networks. Improvements in prediction performance are evident by utilizing synchronized rather than stepwise components to model spatial-temporal correlations. Some existing studies have designed graph structures containing spatial and temporal attributes to achieve spatial-temporal synchronous learning. However, two challenges remain due to the intricate dynamics: (a) Accounting for the impact of external factors in spatial-temporal synchronous modeling. (b) Multiple perspectives in constructing spatial-temporal synchronous graphs. To address the mentioned limitations, a novel model named dynamic multiple-graph spatial-temporal synchronous aggregation framework (DMSTSAF) for traffic prediction is proposed. Specifically, DMSTSAF utilizes a feature augmentation module (FAM) to adaptively incorporate traffic data with external factors and generate fused features as inputs to subsequent modules. Moreover, DMSTSAF introduces diverse spatial and temporal graphs according to different spatial-temporal relationships. Based on this, two types of spatial-temporal synchronous graphs and the corresponding synchronous aggregation modules are designed to simultaneously extract hidden features from various aspects. Extensive experiments constructed on four real-world datasets indicate that our model improves by 3.68-8.54% compared to the state-of-the-art baseline.

Map-Free Trajectory Prediction in Traffic With Multi-Level Spatial-Temporal Modeling

Map-Free Trajectory Prediction in Traffic With Multi-Level Spatial-Temporal Modeling

The price of clean air – quantifying air pollution exposure in real estate decisions

PurposeThe purpose of this paper is to explore the significance impact of air quality as a contributing factor on residential property rents by applying geo-informatics to economic issues. Since air pollution poses a severe health threat, city residents should have a right to know about the (invisible) hazards they are exposed to.Design/methodology/approachWithin spatial-temporal modeling of air pollutants in Berlin, Germany, three interpolation techniques are tested. The most suitable one is selected to create seasonal maps for 2018 and 2021 with pollution concentrations for particulate matter values and nitrogen dioxide for each 1,000 m2 cell within the administrative boundaries. Based on the evaluated pollution particulate matter values, which are used as additional variables for semi-parametric regressions the impact of the air quality on rents is estimated.FindingsThe findings reveal a compelling association between air quality and the economic aspect of the residential real estate market, with noteworthy implications for both tenants and property investors. The relationship between air pollution variables and rents is statistically significant. However, there is only a “willingness-to- pay” for low particulate matter values, but not for nitrogen dioxide concentrations. With good air quality, residents in Berlin are willing to pay a higher rent (3%).Practical implicationsThese results suggest that a “marginal willingness-to-pay” occurs in a German city. The research underscores the multifaceted impact of air quality on the residential rental market in Berlin. The evidence supports the notion that a cleaner environment not only benefits human health and the planet but also contributes significantly to the economic bottom line of property investors.Originality/valueThe paper has a unique data engineering approach. It collects spatiotemporal data from network of state-certified measuring sites to create an index of air pollution. This spatial information is merged with residential listings. Afterward non-linear regression models are estimated.

GMHANN: A Novel Traffic Flow Prediction Method for Transportation Management Based on Spatial-Temporal Graph Modeling

GMHANN: A Novel Traffic Flow Prediction Method for Transportation Management Based on Spatial-Temporal Graph Modeling

Graph-DETR4D: Spatio-Temporal Graph Modeling for Multi-View 3D Object Detection.

Multi-View 3D object detection (MV3D) has made tremendous progress by leveraging multiple perspective features through surrounding cameras. Despite demonstrating promising prospects in various applications, accurately detecting objects through camera view in the 3D space is extremely difficult due to the ill-posed issue in monocular depth estimation. Recently, Graph-DETR3D presents a novel graph-based 3D-2D query paradigm in aggregating multi-view images for 3D object detection and achieves competitive performance. Although it enriches the query representations with 2D image features through a learnable 3D graph, it still suffers from limited depth and velocity estimation abilities due to the adoption of a single-frame input setting. To solve this problem, we introduce a unified spatial-temporal graph modeling framework to fully leverage the multi-view imagery cues under the multi-frame inputs setting. Thanks to the flexibility and sparsity of the dynamic graph architecture, we lift the original 3D graph into the 4D space with an effective attention mechanism to automatically perceive imagery information at both spatial and temporal levels. Moreover, considering the main latency bottleneck lies in the image backbone, we propose a novel dense-sparse distillation framework for multi-view 3D object detection, to reduce the computational budget while sacrificing no detection accuracy, making it more suitable for real-world deployment. To this end, we propose Graph-DETR4D, a faster and stronger multi-view 3D object detection framework, built on top of Graph-DETR3D. Extensive experiments on nuScenes and Waymo benchmarks demonstrate the effectiveness and efficiency of Graph-DETR4D. Notably, our best model achieves 62.0% NDS on nuScenes test leaderboard. Code is available at https://github.com/zehuichen123/Graph-DETR4D.

Tagged-to-Cine MRI Sequence Synthesis via Light Spatial-Temporal Transformer.

Tagged magnetic resonance imaging (MRI) has been successfully used to track the motion of internal tissue points within moving organs. Typically, to analyze motion using tagged MRI, cine MRI data in the same coordinate system are acquired, incurring additional time and costs. Consequently, tagged-to-cine MR synthesis holds the potential to reduce the extra acquisition time and costs associated with cine MRI, without disrupting downstream motion analysis tasks. Previous approaches have processed each frame independently, thereby overlooking the fact that complementary information from occluded regions of the tag patterns could be present in neighboring frames exhibiting motion. Furthermore, the inconsistent visual appearance, e.g., tag fading, across frames can reduce synthesis performance. To address this, we propose an efficient framework for tagged-to-cine MR sequence synthesis, leveraging both spatial and temporal information with relatively limited data. Specifically, we follow a split-and-integral protocol to balance spatialtemporal modeling efficiency and consistency. The light spatial-temporal transformer (LiST2) is designed to exploit the local and global attention in motion sequence with relatively lightweight training parameters. The directional product relative position-time bias is adapted to make the model aware of the spatial-temporal correlation, while the shifted window is used for motion alignment. Then, a recurrent sliding fine-tuning (ReST) scheme is applied to further enhance the temporal consistency. Our framework is evaluated on paired tagged and cine MRI sequences, demonstrating superior performance over comparison methods.

Nonlinear modeling of temperature-induced bearing displacement of long-span single-pier rigid frame bridge based on DCNN-LSTM

Long-span single-pier rigid frame bridge may experience excessive bearing displacement under temperature variation, which can result in structural deformation or instability, causing significant engineering accidents and immeasurable losses on society. This paper proposes a spatial-temporal nonlinear modeling method for temperature and temperature-induced bearing displacement (TIBD) of long-span single-pier rigid frame bridge based on DCNN-LSTM network with elastic modulus fusion, relying on the monitoring data of the Second Yangtze River Bridge in Wuhan. This framework introduces a dynamic autoregressive method for model inference to improve the prediction accuracy of the model. This study addresses three major issues: Insufficient rational selection of temperature characteristic values; Inadequate research on correlation between structural temperature and TIBD for the bridges; Poor prediction accuracy of temperature-TIBD regression models. The method proposed in this study achieves prediction accuracy of up to 99.8 % for TIBD, and this model is not affected by seasonal variation.

A Novel Spatial–Temporal Network for Gait Recognition Using Millimeter-Wave Radar Point Cloud Videos

Gait recognition is a behavioral biometric technology that aims to identify individuals through their manner of walking. Compared with vision and wearable solutions, millimeter-wave (mmWave)-radar-based gait recognition has drawn attention because radar sensing is privacy-preserving and non-contact. However, it is challenging to capture the motion dynamics of walking people from mmWave radar signals, which is crucial for robust gait recognition. In this study, a novel spatial–temporal gait recognition network based on mmWave radar is proposed to address this problem. First, a four-dimensional (4D) radar point cloud video (RPCV) was introduced to characterize human walking patterns. Then, a PointNet block was utilized to extract spatial features from the radar point clouds in each frame. Finally, a Transformer layer was applied for the spatial–temporal modeling of the 4D RPCVs, capturing walking motion information, followed by fully connected layers to output the identification results. The experimental results demonstrated the superiority of the proposed network over mainstream networks, which achieved the best human identification performance on a dataset of 15 volunteers.

Examining associations between social vulnerability indices and COVID-19 incidence and mortality with spatial-temporal Bayesian modeling

This study compares two social vulnerability indices, the U.S. CDC SVI and SoVI (the Social Vulnerability Index developed at the Hazards Vulnerability & Resilience Institute at the University of South Carolina), on their ability to predict the risk of COVID-19 cases and deaths. We utilize COVID-19 cases and deaths data for the state of Indiana from the Regenstrief Institute in Indianapolis, Indiana, from March 1, 2020, to March 31, 2021. We then aggregate the COVID-19 data to the census tract level, obtain the input variables, domains (components), and composite measures of both CDC SVI and SoVI data to create a Bayesian spatial-temporal ecological regression model. We compare the resulting spatial-temporal patterns and relative risk (RR) of SARS-CoV-2 infection (COVID-19 cases) and associated death. Results show there are discernable spatial-temporal patterns for SARS-CoV-2 infections and deaths with the largest contiguous hotspot for SARS-CoV-2 infections found in the southwest of the Indianapolis metropolitan area. We also observed one large contiguous hotspot for deaths that stretches across Indiana from the Cincinnati area in the southeast to just east and north of Terre Haute (southeast to west central). The spatial-temporal Bayesian model shows that a 1-percentile increase in CDC SVI was significantly (p ≤ 0.05) associated with an increased risk of SARS-CoV-2 infection by 6 % (RR = 1.06, 95 %CI = 1.04 -1.08). Whereas a 1-percentile increase in SoVI was significantly predicted to increase the risk of COVID-19 death by 45 % (RR = 1.45, 95 %CI =1.38 – 1.53). Domain-specific variables related to socioeconomic status, age, and race/ethnicity were shown to increase the risk of SARS-CoV-2 infections and deaths. There were notable differences in the relative risk estimates for SARS-CoV-2 infections and deaths when each of the two indices were incorporated in the model. Observed differences between the two social vulnerability indices and infection and death are likely due to alternative methodologies of formation and differences in input variables. The findings add to the growing literature on the relationship between social vulnerability and COVID-19 and further the development of COVID-19-specific vulnerability indices by illustrating the utility of local spatial-temporal analysis.