Spatiotemporal Learning Research Articles

A spatiotemporal learning approach to safety‐oriented individualized driving risk assessment in a vehicle‐to‐everything (V2X) environment

A spatiotemporal learning approach to safety‐oriented individualized driving risk assessment in a vehicle‐to‐everything (V2X) environment

Asymmetric Spatiotemporal Online Learning for Deep Spiking Neural Networks

Although the precise symmetric forward and feedback connections between neurons are thought to be impossible in the brain, most existing deep spiking neural networks (SNNs) spatio-temporal learning algorithms still require such a strong architectural constraint to complete tasks. Besides that, as an expected feature for specialized neuromorphic hardware to be deployed, effective online learning of deep SNNs for training spatio-temporal data are still lacking. To address these issues, an effective biologically plausible asymmetric spatio-temporal online learning algorithm called ASTOL is proposed for training deep SNNs in real-time. ASTOL could learn the spatial and temporal features simultaneously in real-time, without the precise weights symmetric backpropagation and the need to know the duration of spatio-temporal data in advance. Experimental results show that the proposed ASTOL algorithm achieves comparable performance in real-time learning on the rate coding MNIST dataset and the temporal coding music MedleyDB dataset and speech TIDIGITS dataset with other state-of-the-art algorithms with simpler learning mechanism way as it avoids transport of synaptic weights. Besides that, the experiment on MNIST datasets also shows that ASTOL can use simpler network structures to achieve good performance faster.

GNSS positioning enhancement based on NLOS signal detection using spatio-temporal learning in urban canyons

GNSS positioning enhancement based on NLOS signal detection using spatio-temporal learning in urban canyons

Flexible optoelectronic N-I-P synaptic device with visible spectrum perception for energy-efficient artificial vision and efferent neuromuscular system

We have designed a flexible photoelectric artificial synapse with an oxide/mixed perovskite/polymer N-I-P structure that exhibits essential synaptic plasticity. Formamidinium lead triiodide FAPbI3 perovskite doped with bromine and methylammonium (FAxMA1−xPbI2Br) is employed as the intrinsic layer to improve the optical properties of devices. Without requiring a power source in reaction to outside optical spikes, multiple pulse-dependent plasticity is reproduced on the synaptic devices, and the image's edges are sharpened using high-pass filtering. Additionally, the classical conditioning and spatiotemporal learning are copied under the electric pulse excitation. Significant negative differential resistance is evident, even after 1500 flex/flat mechanical operation. The recognition rate of letters in the visual system is as high as 92%, and the walking distance in the efferent neuromuscular system is controllable. The flexible optoelectronic N-I-P synaptic device is designed to facilitate energy-efficient information processing for neuromorphic computing.

Spatio-temporal learning and explaining for dynamic functional connectivity analysis: Application to depression

BackgroundFunctional connectivity has been shown to fluctuate over time. The present study aimed to identifying major depressive disorders (MDD) with dynamic functional connectivity (dFC) from resting-state fMRI data, which would be helpful to produce tools of early depression diagnosis and enhance our understanding of depressive etiology. MethodsThe resting-state fMRI data of 178 subjects were collected, including 89 MDD and 89 healthy controls. We propose a spatio-temporal learning and explaining framework for dFC analysis. A yet effective spatio-temporal model is developed to classifying MDD from healthy controls with dFCs. The model is a stacking neural network model, which learns network structure information by a multi-layer perceptron based spatial encoder, and learns time-varying patterns by a Transformer based temporal encoder. We propose to explain the spatio-temporal model with a two-stage explanation method of importance feature extracting and disorder-relevant pattern exploring. The layer-wise relevance propagation (LRP) method is introduced to extract the most relevant input features in the model, and the attention mechanism with LRP is applied to extract the important time steps of dFCs. The disorder-relevant functional connections, brain regions, and brain states in the model are further explored and identified. ResultsWe achieved the best classification performance in identifying MDD from healthy controls with dFC data. The top important functional connectivity, brain regions, and dynamic states closely related to MDD have been identified. LimitationsThe data preprocessing may affect the classification performance of the model, and this study needs further validation in a larger patient population. ConclusionsThe experimental results demonstrate that the proposed spatio-temporal model could effectively classify MDD, and uncover structural and temporal patterns of dFCs in depression.

Symmetric spatiotemporal learning network with sparse meter graph for short-term energy-consumption prediction in manufacturing systems

Short-term energy-consumption prediction is the basis of anomaly detection, real-time scheduling, and energy-saving control in manufacturing systems. Most existing methods focus on single-node energy-consumption prediction and suffer from difficult parameter collection and modelling. Although several methods have been presented for multinode energy-consumption prediction, their prediction performance needs to be improved owing to a lack of appropriate knowledge guidance and learning networks for complex spatiotemporal relationships. This study presents a symmetric spatiotemporal learning network (SSTLN) with a sparse meter graph (SMG) (SSTLN-SMG) that aims to predict multiple nodes based on energy-consumption time series and general process knowledge. The SMG expresses process knowledge by abstracting production nodes, material flows, and energy usage, and provides initial guidance for the SSTLN to extract spatial features. SSTLN, a symmetrical stack of graph convolutional networks (GCN) and gated linear units (GLU), is devised to achieve a trade-off not only between spatial and temporal feature extraction but also between detail capture and noise suppression. Extensive experiments were performed using datasets from an aluminium profile plant. The experimental results demonstrate that the proposed method allows multinode energy-consumption prediction with less prediction error than state-of-the-art methods, methods with deformed meter graphs, and methods with deformed learning networks.

Incidental learning of temporal and spatial associations in hybrid search

ABSTRACT The present study investigated whether incidental learning of spatial and temporal associations in hybrid visual and memory search enable observers to predict targets in space and time. In three experiments, observers looked for four, previously memorized, target items across many trials. We examined effects of learning target item sequences (e.g., the butterfly always follows the paint-box), target item-location associations (the butterfly is always in the right corner), and target item-location sequences (the butterfly in the right corner always follows the paint-box in the lower middle-left). Learning effects for the sequences of target items alone were weak. By contrast, we found good learning of target item-location associations and a reliable effect of sequence learning for target item-location associations. These findings suggest that spatiotemporal learning in hybrid search is hierarchical: Only if spatial and non-spatial target features are bound, temporal associations bias attention dynamically to the target expected to occur next.

Contaminant Transport Modeling and Source Attribution With Attention‐Based Graph Neural Network

AbstractGroundwater contamination induced by anthropogenic activities has long been a global issue. Characterizing and modeling contaminant transport processes is crucial to groundwater protection and management. However, challenges still exist in process complexity, data constraint, and computational cost. In the era of big data, the growth of machine learning has led to new opportunities in studying contaminant transport in groundwater systems. In this work, we introduce a new attention‐based graph neural network (aGNN) for modeling contaminant transport with limited monitoring data and quantifying causal connections between contaminant sources (drivers) and their spreading (outcomes). In five synthetic case studies that involve varying monitoring networks in heterogeneous aquifers, aGNN is shown to outperform LSTM‐based (long‐short term memory) and CNN‐ based (convolutional neural network) methods in multistep predictions (i.e., transductive learning). It also demonstrates a high level of applicability in inferring observations for unmonitored sites (i.e., inductive learning). Furthermore, an explanatory analysis based on aGNN quantifies the influence of each contaminant source, which has been validated by a physics‐based model with consistent outcomes with an R2 value exceeding 92%. The major advantage of aGNN is that it not only has a high level of predictive power in multiple scenario evaluations but also substantially reduces computational cost. Overall, this study shows that aGNN is efficient and robust for highly nonlinear spatiotemporal learning in subsurface contaminant transport, and provides a promising tool for groundwater management involving contaminant source attribution.

Multi-scale feature enhanced spatio-temporal learning for traffic flow forecasting

Traffic flow forecasting is a critical task within Intelligent Transportation Systems (ITS). Its main challenge lies in effectively modeling the complex traffic related big data, including intricate intra-channel and inter-channel correlations, as well as dynamic spatio-temporal dependencies. Furthermore, current methods continue to encounter bottlenecks in extracting and learning complex and dynamic spatio-temporal features from the original traffic data for long-term prediction, resulting in challenges related to model robustness and generalization. In response to these, we introduce a novel deep learning model with multi-scale feature enhancement for traffic flow forecasting, which is based on the attention mechanism and the graph convolution learning framework. We first introduce the integration design of the spatio-temporal dependency features enhancement module with the base attention learning block through a memory embedding layer. Then we propose a traffic network topology features enhancement module with the spatial attention layer, enabling dynamic enhanced learning of spatio-temporal dependency features. This comprehensive approach enables the model to effectively learn complex and dynamic spatio-temporal dependencies, capturing key patterns in traffic flow data. Through extensive experimental evaluations using traffic flow forecasting benchmarks, we have validated the superior performance of the proposed model over the state-of-the-art.

A Two-Layer Self-Organizing Map with Vector Symbolic Architecture for Spatiotemporal Sequence Learning and Prediction.

We propose a new nature- and neuro-science-inspired algorithm for spatiotemporal learning and prediction based on sequential recall and vector symbolic architecture. A key novelty is the learning of spatial and temporal patterns as decoupled concepts where the temporal pattern sequences are constructed using the learned spatial patterns as an alphabet of elements. The decoupling, motivated by cognitive neuroscience research, provides the flexibility for fast and adaptive learning with dynamic changes to data and concept drift and as such is better suited for real-time learning and prediction. The algorithm further addresses several key computational requirements for predicting the next occurrences based on real-life spatiotemporal data, which have been found to be challenging with current state-of-the-art algorithms. Firstly, spatial and temporal patterns are detected using unsupervised learning from unlabeled data streams in changing environments; secondly, vector symbolic architecture (VSA) is used to manage variable-length sequences; and thirdly, hyper dimensional (HD) computing-based associative memory is used to facilitate the continuous prediction of the next occurrences in sequential patterns. The algorithm has been empirically evaluated using two benchmark and three time-series datasets to demonstrate its advantages compared to the state-of-the-art in spatiotemporal unsupervised sequence learning where the proposed ST-SOM algorithm is able to achieve 45% error reduction compared to HTM algorithm.

Dual-track spatio-temporal learning for urban flow prediction with adaptive normalization

Robust urban flow prediction is crucial for transportation planning and management in urban areas. Although recent advances in modeling spatio-temporal correlations have shown potential, most models fail to adequately consider the complex spatio-temporal semantic information present in real-world scenarios. We summarize the following three primary limitations in existing models: a) The majority of existing models project overall time periods into the same latent space, neglecting the diverse temporal semantics between different time intervals. b) Existing models tend to capture spatial dependencies from a locale perspective such as surroundings but do not pay attention to the global influence factors. c) Beyond the spatio-temporal properties, the dynamics and instability of the data sequences introduce perturbations to the prediction results, potentially leading to model degradation. To address these issues, we propose a dual-track spatial-temporal learning module named DualST for accurate urban flow inference. To more effectively differentiate semantic information in the time dimension, we assign the overall time scales into closeness and periodicity. The dual-track module, which includes temporal causality inference and temporal contextual inference, simultaneously exploits the dynamic evolutionary trends and periodic traffic patterns, respectively. The proposed DualST captures global spatial features in a self-supervised manner which not only enriches the spatial semantics but also avoids introducing additional prior knowledge. To eliminate the instability caused by dynamics, we first adopt spatio-temporal adaptive normalization to learn appropriate data sequence normalization. We evaluate the proposed DualST on two typical urban flow datasets. The experiment results show that our model not only exhibits a consistent superiority over various state-of-the-art baselines but also has remarkable generalization capability.

Deep generative spatiotemporal learning for integrating fracture mechanics in composite materials: inverse design, discovery, and optimization.

The trade-off between strength and toughness presents a fundamental challenge in engineering material design. Composite materials (CMs) can strategically arrange different materials to enhance both strength and toughness by optimizing the distribution of loads and increasing resistance to crack propagation. However, current data-driven computational modeling approaches for CM configuration optimization suffer from limitations of "substantial computational cost" and "poor predictive power over extrapolation spaces", making it difficult to integrate with global optimization algorithms, and ultimately limiting the discovery of materials with optimal tradeoffs. As a breakthrough, we propose a data-driven design framework with a multi-task DL architecture capable of accurately predicting local fields' spatiotemporal behavior, including stress evolution and crack propagation, alongside homogenized mechanical properties. Our model, trained on datasets generated from crack phase fields simulations of random configurations, demonstrated exceptional predictive performance even for unseen configurations with well organized patterns exploiting nature-inspired morphological features. Importantly, solely from composite material (CM) configurations, our model effectively predicts long-term spatiotemporal fields with an accuracy comparable to FEM but with a substantial reduction in computational time. By coupling the model's predictive power with genetic optimization algorithms, we demonstrated the framework's applicability in two representative inverse design tasks: devising CM configurations with mechanical properties beyond the training set and guiding desired crack pattern formation. Our research highlights the potential of artificial intelligence as a feasible alternative to conventional computational approaches for straightforward configurational and structural optimization.

PM2.5 forecasting under distribution shift: A graph learning approach

We present a new benchmark task for graph-based machine learning, aiming to predict future air quality (PM2.5 concentration) observed by a geographically distributed network of environmental sensors. While prior work has successfully applied Graph Neural Networks (GNNs) on a wide family of spatio-temporal prediction tasks, the new benchmark task introduced here brings a technical challenge that has been less studied in the context of graph-based spatio-temporal learning: distribution shift across a long period of time. An important goal of this paper is to understand the behavior of spatio-temporal GNNs under distribution shift. We conduct a comprehensive comparative study of both graph-based and non-graph-based machine learning models under two data split methods, one results in distribution shift and one does not. Our empirical results suggest that GNN models tend to suffer more from distribution shift compared to non-graph-based models, which calls for special attention when deploying spatio-temporal GNNs in practice.

Chronic Inhibition of Nitric Oxide Synthases Impairs Spatiotemporal Learning and Memory to a Similar Extent in C57BL/6 and hAPP23+/− Mice

Due to global population growth, age-related disorders like cardiovascular disease and dementia are anticipated to increase. Recent data suggests a connection between cardiovascular disease and neurodegeneration, especially focusing on arterial stiffness (AS) and Alzheimer’s disease (AD). In light of this, we conducted a study to explore the impact of long-term nitric oxide synthase (NOS) isoform inhibition, which leads to AS, on neurobehavioral performance. We also compared these effects in an AD model and control mice. C57BL/6 and hAPP23+/− mice (an established AD model) were given 0.5 mg/mL N(G)-Nitro-L-Arginine Methyl Ester (L-NAME) in their drinking water for 16 weeks. Our findings indicate that chronic non-selective NOS inhibition increased AS and reduced spatiotemporal learning and memory in both C57BL/6 and hAPP23+/− mice. These effects were consistent across both groups, emphasizing the role of neuronal NOS (nNOS) in cognitive aging, regardless of genetic predisposition to AD.

Resource efficient aortic distensibility calculation by end to end spatiotemporal learning of aortic lumen from multicentre multivendor multidisease CMR images

Aortic distensibility (AD) is important for the prognosis of multiple cardiovascular diseases. We propose a novel resource-efficient deep learning (DL) model, inspired by the bi-directional ConvLSTM U-Net with densely connected convolutions, to perform end-to-end hierarchical learning of the aorta from cine cardiovascular MRI towards streamlining AD quantification. Unlike current DL aortic segmentation approaches, our pipeline: (i) performs simultaneous spatio-temporal learning of the video input, (ii) combines the feature maps from the encoder and decoder using non-linear functions, and (iii) takes into account the high class imbalance. By using multi-centre multi-vendor data from a highly heterogeneous patient cohort, we demonstrate that the proposed method outperforms the state-of-the-art method in terms of accuracy and at the same time it consumes ∼\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\sim$$\\end{document} 3.9 times less fuel and generates ∼\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\sim$$\\end{document} 2.8 less carbon emissions. Our model could provide a valuable tool for exploring genome-wide associations of the AD with the cognitive performance in large-scale biomedical databases. By making energy usage and carbon emissions explicit, the presented work aligns with efforts to keep DL’s energy requirements and carbon cost in check. The improved resource efficiency of our pipeline might open up the more systematic DL-powered evaluation of the MRI-derived aortic stiffness.

Multi-view spatiotemporal learning for traffic forecasting

The spatiotemporal learning has been one of the most burning research topics in traffic data analytics for traffic forecasting, which is instrumental in developing the intelligent transportation systems. Capturing accurately and efficiently complex spatiotemporal dependencies of the road network is a critical prerequisite of the traffic spatiotemporal forecasting. This study proposes a multi-view spatiotemporal learning (MVSTL) framework, which combines the fast parallel learning (FPL) and the serial learning (SL) to juggle the receptive field with the fitting capability of the model. The FPL is proposed to capture the spatiotemporal correlations synchronously and efficiently by reducing the number of parameters significantly. As a supplement to the FPL, the SL, which extracts the spatial and temporal features serially, can expand the spatial/temporal receptive fields and avoid information overwriting. The evaluation of the MVSTL is carried out by employing four sets of traffic data from the LA freeway system, and the experiment results reveal that the MVSTL is able to perform better than the advanced methods in prediction accuracy.

Online Spatio-Temporal Learning in Deep Neural Networks.

Biological neural networks are equipped with an inherent capability to continuously adapt through online learning. This aspect remains in stark contrast to learning with error backpropagation through time (BPTT) that involves offline computation of the gradients due to the need to unroll the network through time. Here, we present an alternative online learning algorithm ic framework for deep recurrent neural networks (RNNs) and spiking neural networks (SNNs), called online spatio-temporal learning (OSTL). It is based on insights from biology and proposes the clear separation of spatial and temporal gradient components. For shallow SNNs, OSTL is gradient equivalent to BPTT enabling for the first time online training of SNNs with BPTT-equivalent gradients. In addition, the proposed formulation unveils a class of SNN architectures trainable online at low time complexity. Moreover, we extend OSTL to a generic form, applicable to a wide range of network architectures, including networks comprising long short-term memory (LSTM) and gated recurrent units (GRUs). We demonstrate the operation of our algorithm ic framework on various tasks from language modeling to speech recognition and obtain results on par with the BPTT baselines.

Vertex position estimation with spatial–temporal transformer for 3D human reconstruction

Reconstructing 3D human pose and body shape from monocular images or videos is a fundamental task for comprehending human dynamics. Frame-based methods can be broadly categorized into two fashions: those regressing parametric model parameters (e.g., SMPL) and those exploring alternative representations (e.g., volumetric shapes, 3D coordinates). Non-parametric representations have demonstrated superior performance due to their enhanced flexibility. However, when applied to video data, these non-parametric frame-based methods tend to generate inconsistent and unsmooth results. To this end, we present a novel approach that directly regresses the 3D coordinates of the mesh vertices and body joints with a spatial–temporal Transformer. In our method, we introduce a SpatioTemporal Learning Block (STLB) with Spatial Learning Module (SLM) and Temporal Learning Module (TLM), which leverages spatial and temporal information to model interactions at a finer granularity, specifically at the body token level. Our method outperforms previous state-of-the-art approaches on Human3.6M and 3DPW benchmark datasets.

Face Positioned Driver Drowsiness Detection Using Multistage Adaptive 3D Convolutional Neural Network

Accidents due to driver drowsiness are observed to be increasing at an alarming rate across all countries and it becomes necessary to identify driver drowsiness to reduce accident rates. Researchers handled many machine learning and deep learning techniques especially many CNN variants created for drowsiness detection, but it is dangerous to use in real time, as the design fails due to high computational complexity, low evaluation accuracies and low reliability. In this article, we introduce a multistage adaptive 3D-CNN model with multi-expressive features for Driver Drowsiness Detection (DDD) with special attention to system complexity and performance. The proposed architecture is divided into five cascaded stages: (1) A three level Convolutional Neural Network (CNN) for driver face positioning (2) 3D-CNN based Spatio-Temporal (ST) Learning to extract 3D features from face positioned stacked samples. (3) State Understanding (SU) to train 3D-CNN based drowsiness models (4) Feature fusion using ST and SU stages (5) Drowsiness Detection stage. The Proposed system extract ST values from the face positioned images and then merges it with SU results from each state understanding sub models to create conditional driver facial features for final Drowsiness Detection (DD) model. Final DD Model is trained offline and implemented in online, results show the developed model performs well when compared to others and additionally capable of handling Indian conditions. This method is applied (Trained and Evaluated) using two different datasets, Kongu Engineering College Driver Drowsiness Detection (KEC-DDD) own dataset and National Tsing Hua University Driver Drowsiness Detection (NTHU-DDD) Benchmark Dataset. The proposed system trained with KEC-DDD dataset produces accuracy of 77.45% and 75.91% using evaluation set of KEC-DDD and NTHU-DDD dataset and capable to detect driver drowsiness from 256×256 resolution images at 39.6 fps at an average of 400 execution seconds.

Spatiotemporal Learning Transformer for Video-Based Human Pose Estimation

Multi-frame human pose estimation has long been an appealing and fundamental issue in visual perception. Owing to the frequent rapid motion and pose occlusion in videos, this task is extremely challenging. Current state-of-the-art methods seek to model spatiotemporal features by equally fusing each frame in the local sequence, which weakens the target frame information. In addition, existing approaches usually emphasize more on deep features while ignoring the detailed information implied in the shallow feature maps, resulting in the dropping of crucial features. To address the above problems, we propose an effective framework, namely spatiotemporal learning transformer for video-based human pose estimation (SLT-Pose), which consists of a Personalized Feature Extraction Module (PFEM), Self-feature Refinement Module (SRM), Cross-frame Temporal Learning Module (CTLM) and Disentangled Keypoint Detector (DKD). To be specific, we propose PFEM which extracts and modulates the individual frame features to adapt to the varying human shape, and integrates single-frame features to obtain the spatiotemporal features. We further present SRM to establish global correlation spatial cues on the target frame to attain the refinement feature. Then, a CTLM is designed to search for the information most closely related to the target frame from the spatiotemporal features to intensify the interaction between the target frame and the local sequence, using both the shallow detailed and the deep semantic representations. Finally, we employ DKD to extract the disentangled characteristics of each joint and encode the articulated joint pairs in the human body, promoting the model to reasonably and accurately predict the keypoint heatmaps. Extensive experiments on three huamn motion benchmarks, including PoseTrack2017, PoseTrack2018, and Sub-JHMDB dataset, demonstrate that SLT-Pose plays favorably against state-of-the-art approaches in terms of both objective evaluation and subjective visual performance.