Long-term Temporal Dependencies Research Articles

LSTGCN: Inductive Spatial Temporal Imputation Using Long Short Term Dependencies

Spatial temporal forecasting of urban sensors is essentially important for many urban systems, such as intelligent transportation and smart cities. However, due to the problem of hardware failure or network failure, there are some missing values or missing monitoring sensors that need to be interpolated. Recent research on deep learning has made substantial progress on imputation problem, especially temporal aspect (i.e., time series imputation), while little attention has been paid to spatial aspect (both dynamic and static) and long term temporal dependencies. In this paper, we proposed a spatial temporal imputation model, named Long Short Term Graph Convolution Networks (LSTGCN), which includes gated temporal extraction (GTE) module, multi-head attention based temporal capture (MHAT) module, long term periodic temporal encoding (LPTE) module and bidirectional spatial graph convolution (BSGC) module. The GTE adopts a gated mechanism to filter short-term temporal information, while the MHAT utilizes position encoding to enhance the difference of each time stamps, then use multi-head attention to capture short term temporal dependency. The BSGC is adopted to handle with spatial relationships between sensor nodes. And we design a periodic encoding technique to process long term temporal dependencies. The BSGC handles spatial relationships between sensor nodes, and a periodic encoding technique is used to process long-term temporal dependencies. Our experimental analysis includes completion and forecasting tasks, as well as transfer and ablation analyses. The results show that our proposed model outperforms state-of-the-art baselines on real-world datasets.

Delving into CLIP latent space for Video Anomaly Recognition

We tackle the complex problem of detecting and recognising anomalies in surveillance videos at the frame level, utilising only video-level supervision. We introduce the novel method AnomalyCLIP, the first to combine Vision and Language Models (VLMs), such as CLIP, with multiple instance learning for joint video anomaly detection and classification. Our approach specifically involves manipulating the latent CLIP feature space to identify the normal event subspace, which in turn allows us to effectively learn text-driven directions for abnormal events. When anomalous frames are projected onto these directions, they exhibit a large feature magnitude if they belong to a particular class. We also leverage a computationally efficient Transformer architecture to model short- and long-term temporal dependencies between frames, ultimately producing the final anomaly score and class prediction probabilities. We compare AnomalyCLIP against state-of-the-art methods considering three major anomaly detection benchmarks, i.e. ShanghaiTech, UCF-Crime, and XD-Violence, and empirically show that it outperforms baselines in recognising video anomalies. Project website and code are available at https://lucazanella.github.io/AnomalyCLIP/.

TADGCN: A Time-Aware Dynamic Graph Convolution Network for long-term traffic flow prediction

Traffic flow prediction is a crucial component of the Intelligent Transportation System (ITS) with substantial implications for traffic management and daily travel. While short-term traffic forecasting has shown promise, long-term prediction presents unique challenges. The existing short-term approaches fall short in long-term scenarios due to two primary issues: (1) ignoring modeling the interaction relationships among dynamic spatial dependencies, and (2) extracting long-term temporal dependencies without capturing the time information in-depth. To mitigate the above issues, we propose a novel model called TADGCN, a Time-Aware Dynamic Graph Convolution Network, to effectively perform long-term traffic flow prediction. TADGCN captures latent interactions among dynamic spatial dependencies using Attention-Driven Dynamic Adaptive Graph Convolution Network (ADAGCN) modules. The time-aware capability is effectively enhanced using Time-Aware Joint Multi-View Temporal Encoder (TJMTE) modules due to the construction of a time-aware matrix. We conduct diversified experiments on three real-world datasets (England, PEMS04, and PEMS08) to evaluate the performance of our proposed method TADGCN. The results reveal that TADGCN achieves the best accuracy in all cases against 16 competitive state-of-the-art baselines.

Self-attention with temporal prior: can we learn more from the arrow of time?

Many diverse phenomena in nature often inherently encode both short- and long-term temporal dependencies, which especially result from the direction of the flow of time. In this respect, we discovered experimental evidence suggesting that interrelations of these events are higher for closer time stamps. However, to be able for attention-based models to learn these regularities in short-term dependencies, it requires large amounts of data, which are often infeasible. This is because, while they are good at learning piece-wise temporal dependencies, attention-based models lack structures that encode biases in time series. As a resolution, we propose a simple and efficient method that enables attention layers to better encode the short-term temporal bias of these data sets by applying learnable, adaptive kernels directly to the attention matrices. We chose various prediction tasks for the experiments using Electronic Health Records (EHR) data sets since they are great examples with underlying long- and short-term temporal dependencies. Our experiments show exceptional classification results compared to best-performing models on most tasks and data sets.

Deep spatio-temporal feature fusion learning for multi-step building cooling load forecasting

Accurate forecasting of building cooling load is essential for optimizing control strategies and energy efficiency in heating, ventilation, and air conditioning (HVAC) systems. However, existing machine learning and deep learning algorithms encounter challenges in addressing the dynamics and uncertain nature of cooling load demand, as they solely focus on temporal analysis. To tackle this issue, the paper proposes CRG-Informer, a deep learning approach that integrates spatio-temporal features to enhance multi-step predictive performance. Firstly, a time series spatial control relationship graph set is constructed based on the interconnected structure, static attributes, and control characteristics of the cooling system. Then, a graph neural network (GNN) is designed for graph embedding to extract spatial correlation information at each instance. Additionally, the Informer model with the ProbSparse self-attention mechanism is utilized to capture long-term temporal dependencies among input sequences. This integration of components enables feature fusion, and the proposed approach is evaluated using data over an entire cooling season from an office building in Hangzhou, China. Extensive experiments were conducted to compare our method with five superior baseline models for 6, 12, 24, 72, and 144-step cooling load forecasting across four types of data acquisition intervals. The results demonstrate that the CRG-Informer, through spatio-temporal feature fusion, outperforms existing methods and provides a novel solution for improving cooling load forecasting performance.

Progressive prediction algorithm by multi-interval data sampling in multi-task learning for real-time gas identification

Effective surveillance of harmful gases is paramount in the safeguard of human health and the protection of environmental air quality, thereby emphasizing the imperative for robust detection strategies, such as electronic noses. While the integration of intelligent algorithms has significantly improved the detection capabilities of electronic noses, most existing models focus on enhancing accuracy but usually ignore the crucial need for detection speed. To address this issue, the progressive prediction algorithm (PPA) is proposed to accomplish real-time gas recognition. The implementation of PPA incorporates time correction and multi-interval data sampling for gas analysis. Leveraging multi-task networks, this method proposes a novel network architecture that predicts gas type and concentration. The architecture synergizes the flexible receptive field size of time convolutional networks (TCN) with the long-term temporal dependency capture of gated recurrent units (GRU) to optimize overall model performance. The PPA achieves a classification accuracy of 99.3 % and a regression R2 score of 0.927 within half of the response time. For practical applications, this work enables the prediction of gas type and concentration during the early stages of sensor response, facilitating rapid detection of hazardous gases.

A Residual Deep Learning Method for Accurate and Efficient Recognition of Gym Exercise Activities Using Electromyography and IMU Sensors

The accurate and efficient recognition of gym workout activities using wearable sensors holds significant implications for assessing fitness levels, tailoring personalized training regimens, and overseeing rehabilitation progress. This study introduces CNN-ResBiGRU, a novel deep learning architecture that amalgamates residual and hybrid methodologies, aiming to precisely categorize gym exercises based on multimodal sensor data. The primary goal of this model is to effectively identify various gym workouts by integrating convolutional neural networks, residual connections, and bidirectional gated recurrent units. Raw electromyography and inertial measurement unit data collected from wearable sensors worn by individuals during strength training and gym sessions serve as inputs for the CNN-ResBiGRU model. Initially, convolutional neural network layers are employed to extract unique features in both temporal and spatial dimensions, capturing localized patterns within the sensor outputs. Subsequently, the extracted features are fed into the ResBiGRU component, leveraging residual connections and bidirectional processing to capture the exercise activities’ long-term temporal dependencies and contextual information. The performance of the proposed model is evaluated using the Myogym dataset, comprising data from 10 participants engaged in 30 distinct gym activities. The model achieves a classification accuracy of 97.29% and an F1-score of 92.68%. Ablation studies confirm the effectiveness of the convolutional neural network and ResBiGRU components. The proposed hybrid model uses wearable multimodal sensor data to accurately and efficiently recognize gym exercise activity.

A generative adversarial network based on an efficient transformer for high-fidelity flow field reconstruction

The reconstruction of high-fidelity flow fields from low-fidelity data has attracted considerable attention in fluid dynamics but poses many challenges to existing deep learning methods due to the spatiotemporal complexity of flows and the lack of standardized benchmark datasets. In this study, we generate a low- and high-fidelity dataset containing 25 600 snapshots of four representative flow dynamics simulations using eight different numerical-precision and grid-resolution configurations. Using this dataset, we develop a physics-guided transformer-based generative adversarial network (PgTransGAN) for concurrently handling numerical-precision and grid-resolution enhancement. PgTransGAN leverages a dual-discriminator-based generative adversarial network for capturing continuous spatial and temporal dynamics of flows and applies a soft-constraint approach to enforce physical consistency in the reconstructed data using gradient information. An efficient transformer model is also developed to obtain the long-term temporal dependencies and further alleviate storage constraints. We compare the performance of PgTransGAN against standard linear interpolation and solutions based solely on convolutional neural networks or generative adversarial networks, and demonstrate that our method achieves better reconstruction quality at the data, image, and physics levels with an upscaling factor of 4 or even 8 in each grid dimension.

MDSTF: a multi-dimensional spatio-temporal feature fusion trajectory prediction model for autonomous driving

In the field of autonomous driving, trajectory prediction of traffic agents is an important and challenging problem. Fully capturing the complex spatio-temporal features in trajectory data is crucial for accurate trajectory prediction. This paper proposes a trajectory prediction model called multi-dimensional spatio-temporal feature fusion (MDSTF), which integrates multi-dimensional spatio-temporal features to model the trajectory information of traffic agents. In the spatial dimension, we employ graph convolutional networks (GCN) to capture the local spatial features of traffic agents, spatial attention mechanism to capture the global spatial features, and LSTM combined with spatial attention to capture the full-process spatial features of traffic agents. Subsequently, these three spatial features are fused using a gate fusion mechanism. Moreover, during the modeling of the full-process spatial features, LSTM is capable of capturing short-term temporal dependencies in the trajectory information of traffic agents. In the temporal dimension, we utilize a Transformer-based encoder to extract long-term temporal dependencies in the trajectory information of traffic agents, which are then fused with the short-term temporal dependencies captured by LSTM. Finally, we employ two temporal convolutional networks (TCN) to predict trajectories based on the fused spatio-temporal features. Experimental results on the ApolloScape trajectory dataset demonstrate that our proposed method outperforms state-of-the-art methods in terms of weighted sum of average displacement error (WSADE) and weighted sum of final displacement error (WSFDE) metrics. Compared to the best baseline model (S2TNet), our method achieves reductions of 4.37% and 6.23% respectively in these metrics.

Spatial-temporal memory enhanced multi-level attention network for origin-destination demand prediction

Origin-destination demand prediction is a critical task in the field of intelligent transportation systems. However, accurately modeling the complex spatial-temporal dependencies presents significant challenges, which arises from various factors, including spatial, temporal, and external influences such as geographical features, weather conditions, and traffic incidents. Moreover, capturing multi-scale dependencies of local and global spatial dependencies, as well as short and long-term temporal dependencies, further complicates the task. To address these challenges, a novel framework called the Spatial-Temporal Memory Enhanced Multi-Level Attention Network (ST-MEN) is proposed. The framework consists of several key components. Firstly, an external attention mechanism is incorporated to efficiently process external factors into the prediction process. Secondly, a dynamic spatial feature extraction module is designed that effectively captures the spatial dependencies among nodes. By incorporating two skip-connections, this module preserves the original node information while aggregating information from other nodes. Finally, a temporal feature extraction module is proposed that captures both continuous and discrete temporal dependencies using a hierarchical memory network. In addition, multi-scale features cascade fusion is incorporated to enhance the performance of the proposed model. To evaluate the effectiveness of the proposed model, extensively experiments are conducted on two real-world datasets. The experimental results demonstrate that the ST-MEN model achieves excellent prediction accuracy, where the maximum improvement can reach to 19.1%.

Amplitude-Time Dual-View Fused EEG Temporal Feature Learning for Automatic Sleep Staging.

Electroencephalogram (EEG) plays an important role in studying brain function and human cognitive performance, and the recognition of EEG signals is vital to develop an automatic sleep staging system. However, due to the complex nonstationary characteristics and the individual difference between subjects, how to obtain the effective signal features of the EEG for practical application is still a challenging task. In this article, we investigate the EEG feature learning problem and propose a novel temporal feature learning method based on amplitude-time dual-view fusion for automatic sleep staging. First, we explore the feature extraction ability of convolutional neural networks for the EEG signal from the perspective of interpretability and construct two new representation signals for the raw EEG from the views of amplitude and time. Then, we extract the amplitude-time signal features that reflect the transformation between different sleep stages from the obtained representation signals by using conventional 1-D CNNs. Furthermore, a hybrid dilation convolution module is used to learn the long-term temporal dependency features of EEG signals, which can overcome the shortcoming that the small-scale convolution kernel can only learn the local signal variation information. Finally, we conduct attention-based feature fusion for the learned dual-view signal features to further improve sleep staging performance. To evaluate the performance of the proposed method, we test 30-s-epoch EEG signal samples for healthy subjects and subjects with mild sleep disorders. The experimental results from the most commonly used datasets show that the proposed method has better sleep staging performance and has the potential for the development and application of an EEG-based automatic sleep staging system.

Nonlinear slow-varying dynamics-assisted temporal graph transformer network for remaining useful life prediction

Remaining useful life (RUL) plays an important role in the prognostics and health management of mechanical systems. Recently, deep learning-based methods have been widely applied in the field of RUL prediction. However, there still suffer from two limitations. One is that the existing RUL prediction methods cannot capture spatial dependencies and long-term temporal dependencies. The other is that nonlinear slow-varying dynamics related to the degradation behavior have not been explored in the RUL prediction. To break these limitations, a nonlinear slow-varying dynamics-assisted temporal graph Transformer network (NSD-TGTN) is proposed in this paper for RUL prediction. NSD-TGTN can simultaneously capture and model spatiotemporal graphs and nonlinear slow-varying dynamics to achieve RUL prediction. Herein, the TGTN is developed to mine both spatial and long-term temporal dependencies for constructing the spatiotemporal features. And, nonlinear slow-varying features are built and introduced into the TGTN to enhance the RUL prediction capacity. Two datasets are utilized to validate the effectiveness and superiority of the proposed method. Compared with existing advanced methods, the average prediction accuracies of the NSD-TGTN on the C-MAPSS dataset and the wear dataset are improved by 1.70 % and 8.22 %, respectively.

PV Power Forecasting in the Hexi Region of Gansu Province Based on AP Clustering and LSTNet

Accurate PV power forecasting is becoming a mandatory task to integrate the PV system into the power grid, schedule it, and ensure the safety of the power grid. In this paper, a novel model for PV power prediction using AP-LSTNet has been proposed. It consists of a combination of affinity propagation clustering and long-term and short-term time series network models. First, the affinity propagation algorithm is used to divide the regionally distributed photovoltaic station clusters into different seasons. The Pearson correlation coefficient is used to determine the strong correlation between meteorological factors of photovoltaic power, and the bilinear interpolation method is used to encrypt the meteorological data of the corresponding photovoltaic station cluster. Furthermore, LSTNet is used to mine the long-term and short-term temporal and spatial dependence of photovoltaic power, and meteorological factor series and linear components of auto-regression are superimposed to realize the simultaneous prediction of multiple photovoltaic stations in the group. Finally, PV power plants in five cities, Wuwei, Jinchang, Zhangye, Jiuquan, and Jiayuguan in the Hexi region of Gansu Province, China, will be selected to test the proposed model. The experimental comparison shows that the prediction model achieves high prediction accuracy and robustness.

Predicting dust pollution from dry bulk ports in coastal cities: A hybrid approach based on data decomposition and deep learning

Dust pollution from storage and handling of materials in dry bulk ports seriously affects air quality and public health in coastal cities. Accurate prediction of dust pollution helps identify risks early and take preventive measures. However, there remain challenges in solving non-stationary time series and selecting relevant features. Besides, existing studies rarely consider impacts of port operations on dust pollution. Therefore, a hybrid approach based on data decomposition and deep learning is proposed to predict dust pollution from dry bulk ports. Port operational data is specially integrated into input features. A secondary decomposition and recombination (SDR) strategy is presented to reduce data non-stationarity. A dual-stage attention-based sequence-to-sequence (DA-Seq2Seq) model is employed to adaptively select the most relevant features at each time step, as well as capture long-term temporal dependencies. This approach is compared with baseline models on a dataset from a dry bulk port in northern China. The results reveal the advantages of SDR strategy and integrating operational data and show that this approach has higher accuracy than baseline models. The proposed approach can mitigate adverse effects of dust pollution from dry bulk ports on urban residents and help port authorities control dust pollution.

Fault detection of complicated processes based on an enhanced transformer network with graph attention mechanism

Recently, deep learning becomes increasingly popular in the industrial data analysis field due to its distinguished feature representation capability. As an emerging deep learning technology, transformer network has attracted extensive attention, but its application in industrial fault detection is still not sufficiently exploited. Furthermore, the traditional transformer network mainly focuses on the time series information of the data, while ignoring the spatial characteristic of the data. For this problem, an improved transformer network model with graph attention mechanism (GA-Tran) is proposed and used for complicated process fault detection. Considering the strong coupling property of process variables, the presented method performs joint spatial-temporal learning by mining the data information in views of both time-dimension and variable-dimension. On the one hand, a transformer encoder module equipped with a self-attention mechanism is utilized to capture long-term temporal dependencies, while multi-scale convolution is integrated to abstract short-term time dependencies. On the other hand, a graph attention network is introduced into the transformer network to meticulously analyze the spatial relationships among variables. Considering the topology of the industrial process variables, we use spectral clustering to infer prior topology information, and remove intelligently irrelevant spatial information. Different from the existing transformer models with the anomaly scores as the monitoring indicator, two monitoring statistics based on encoder features and reconstruction errors are constructed for process status monitoring. Studies on the Tennessee Eastman chemical process demonstrate the effectiveness of the proposed GA-Tran method.

Spatial-temporal graph neural ODE networks for skeleton-based action recognition.

In the field of skeleton-based action recognition, accurately recognizing human actions is crucial for applications such as virtual reality and motion analysis. However, this task faces challenges such intraindividual action differences and long-term temporal dependencies. To address these challenges, we propose an innovative model called spatial-temporal graph neural ordinary differential equations (STG-NODE). First, in the data preprocessing stage, the dynamic time warping (DTW) algorithm is used to normalize and calculate 3D skeleton data to facilitate the derivation of customized adjacency matrices for improving the influence of intraindividual action differences. Secondly, a custom ordinary differential equation (ODE) integrator is applied based on the initial conditions of the temporal features, producing a solution function that simulates the dynamic evolution trend of the events of interest. Finally, the outstanding ODE solver is used to numerically solve the time features based on the solution function to increase the influence of long-term dependencies on the recognition accuracy of the model and provide it with a more powerful temporal modeling ability. Through extensive experiments conducted on the NTU RGB+D 60 and Kinetics Skeleton 400 benchmark datasets, we demonstrate the superior performance of STG-NODE in the action recognition domain. The success of the STG-NODE model also provides new ideas and methods for the future development of the action recognition field.

Short-term wind power forecasting through stacked and bi directional LSTM techniques.

Computational intelligence (CI) based prediction models increase the efficient and effective utilization of resources for wind prediction. However, the traditional recurrent neural networks (RNN) are difficult to train on data having long-term temporal dependencies, thus susceptible to an inherent problem of vanishing gradient. This work proposed a method based on an advanced version of RNN known as long short-term memory (LSTM) architecture, which updates recurrent weights to overcome the vanishing gradient problem. This, in turn, improves training performance. The RNN model is developed based on stack LSTM and bidirectional LSTM. The parameters like mean absolute error (MAE), standard deviation error (SDE), and root mean squared error (RMSE) are utilized as performance measures for comparison with recent state-of-the-art techniques. Results showed that the proposed technique outperformed the existing techniques in terms of RMSE and MAE against all the used wind farm datasets. Whereas, a reduction in SDE is observed for larger wind farm datasets. The proposed RNN approach performed better than the existing models despite fewer parameters. In addition, the approach requires minimum processing power to achieve compatible results.

Mobile_ViT: Underwater Acoustic Target Recognition Method Based on Local–Global Feature Fusion

To overcome the challenges of inadequate representation and ineffective information exchange stemming from feature homogenization in underwater acoustic target recognition, we introduce a hybrid network named Mobile_ViT, which synergizes MobileNet and Transformer architectures. The network begins with a convolutional backbone incorporating an embedded coordinate attention mechanism to enhance the local details of inputs. This mechanism captures the long-term temporal dependencies and precise frequency–domain relationships of signals, focusing the features on the time–frequency positions. Subsequently, the Transformer’s Encoder is integrated at the end of the backbone to facilitate global characterization, thus effectively overcoming the convolutional neural network’s shortcomings in capturing long-range feature dependencies. Evaluation on the Shipsear and DeepShip datasets yields accuracies of 98.50% and 94.57%, respectively, marking a substantial improvement over the baseline. Notably, the proposed method also demonstrates obvious separation coefficients, signifying enhanced clustering effectiveness, and is lighter than other Transformers.

Joint learning of video scene detection and annotation via multi-modal adaptive context network

The tasks of scene detection and annotation have gained impressive attention for understanding video content. The main challenges lie in mitigating the error propagation of shot detection, recognizing cuts and gradual transitions, fusing hierarchical multi-modal cues, and solving these two tasks simultaneously. To address these challenges, we propose the Multi-modal Adaptive Context Network (MACN) to jointly learn scene detection and annotation from a window partitioning perspective. As a shared task-agnostic part, we perform Window-based Cross-modal Representation (WCR) to distill complex semantic correlations from multi-modal sources for each window. Considering the long-term temporal dependency of variable-length scenes, we further develop Adaptive Context-aware Representation (ACR) to improve the performance for specific tasks. Different from previous works, scene detection is formulated as locating the starting window and its associated location offset and transition duration. Meanwhile, we assemble two multi-label sub-classifiers in different levels to predict the labels for each scene candidate. Experimental comparisons to state-of-the-art algorithms on the TAVS and ClipShots indicate that the proposed method yields promising performance in both tasks. Our code and test sample videos are released at MACN.

TC-LIF: A Two-Compartment Spiking Neuron Model for Long-Term Sequential Modelling

The identification of sensory cues associated with potential opportunities and dangers is frequently complicated by unrelated events that separate useful cues by long delays. As a result, it remains a challenging task for state-of-the-art spiking neural networks (SNNs) to establish long-term temporal dependency between distant cues. To address this challenge, we propose a novel biologically inspired Two-Compartment Leaky Integrate-and-Fire spiking neuron model, dubbed TC-LIF. The proposed model incorporates carefully designed somatic and dendritic compartments that are tailored to facilitate learning long-term temporal dependencies. Furthermore, the theoretical analysis is provided to validate the effectiveness of TC-LIF in propagating error gradients over an extended temporal duration. Our experimental results, on a diverse range of temporal classification tasks, demonstrate superior temporal classification capability, rapid training convergence, and high energy efficiency of the proposed TC-LIF model. Therefore, this work opens up a myriad of opportunities for solving challenging temporal processing tasks on emerging neuromorphic computing systems. Our code is publicly available at https://github.com/ZhangShimin1/TC-LIF.