Long-term Temporal Dependencies Research Articles

Self-attention with temporal prior: can we learn more from the arrow of time?

Many diverse phenomena in nature often inherently encode both short- and long-term temporal dependencies, which especially result from the direction of the flow of time. In this respect, we discovered experimental evidence suggesting that interrelations of these events are higher for closer time stamps. However, to be able for attention-based models to learn these regularities in short-term dependencies, it requires large amounts of data, which are often infeasible. This is because, while they are good at learning piece-wise temporal dependencies, attention-based models lack structures that encode biases in time series. As a resolution, we propose a simple and efficient method that enables attention layers to better encode the short-term temporal bias of these data sets by applying learnable, adaptive kernels directly to the attention matrices. We chose various prediction tasks for the experiments using Electronic Health Records (EHR) data sets since they are great examples with underlying long- and short-term temporal dependencies. Our experiments show exceptional classification results compared to best-performing models on most tasks and data sets.

Progressive prediction algorithm by multi-interval data sampling in multi-task learning for real-time gas identification

Effective surveillance of harmful gases is paramount in the safeguard of human health and the protection of environmental air quality, thereby emphasizing the imperative for robust detection strategies, such as electronic noses. While the integration of intelligent algorithms has significantly improved the detection capabilities of electronic noses, most existing models focus on enhancing accuracy but usually ignore the crucial need for detection speed. To address this issue, the progressive prediction algorithm (PPA) is proposed to accomplish real-time gas recognition. The implementation of PPA incorporates time correction and multi-interval data sampling for gas analysis. Leveraging multi-task networks, this method proposes a novel network architecture that predicts gas type and concentration. The architecture synergizes the flexible receptive field size of time convolutional networks (TCN) with the long-term temporal dependency capture of gated recurrent units (GRU) to optimize overall model performance. The PPA achieves a classification accuracy of 99.3 % and a regression R2 score of 0.927 within half of the response time. For practical applications, this work enables the prediction of gas type and concentration during the early stages of sensor response, facilitating rapid detection of hazardous gases.

A Residual Deep Learning Method for Accurate and Efficient Recognition of Gym Exercise Activities Using Electromyography and IMU Sensors

The accurate and efficient recognition of gym workout activities using wearable sensors holds significant implications for assessing fitness levels, tailoring personalized training regimens, and overseeing rehabilitation progress. This study introduces CNN-ResBiGRU, a novel deep learning architecture that amalgamates residual and hybrid methodologies, aiming to precisely categorize gym exercises based on multimodal sensor data. The primary goal of this model is to effectively identify various gym workouts by integrating convolutional neural networks, residual connections, and bidirectional gated recurrent units. Raw electromyography and inertial measurement unit data collected from wearable sensors worn by individuals during strength training and gym sessions serve as inputs for the CNN-ResBiGRU model. Initially, convolutional neural network layers are employed to extract unique features in both temporal and spatial dimensions, capturing localized patterns within the sensor outputs. Subsequently, the extracted features are fed into the ResBiGRU component, leveraging residual connections and bidirectional processing to capture the exercise activities’ long-term temporal dependencies and contextual information. The performance of the proposed model is evaluated using the Myogym dataset, comprising data from 10 participants engaged in 30 distinct gym activities. The model achieves a classification accuracy of 97.29% and an F1-score of 92.68%. Ablation studies confirm the effectiveness of the convolutional neural network and ResBiGRU components. The proposed hybrid model uses wearable multimodal sensor data to accurately and efficiently recognize gym exercise activity.

A generative adversarial network based on an efficient transformer for high-fidelity flow field reconstruction

The reconstruction of high-fidelity flow fields from low-fidelity data has attracted considerable attention in fluid dynamics but poses many challenges to existing deep learning methods due to the spatiotemporal complexity of flows and the lack of standardized benchmark datasets. In this study, we generate a low- and high-fidelity dataset containing 25 600 snapshots of four representative flow dynamics simulations using eight different numerical-precision and grid-resolution configurations. Using this dataset, we develop a physics-guided transformer-based generative adversarial network (PgTransGAN) for concurrently handling numerical-precision and grid-resolution enhancement. PgTransGAN leverages a dual-discriminator-based generative adversarial network for capturing continuous spatial and temporal dynamics of flows and applies a soft-constraint approach to enforce physical consistency in the reconstructed data using gradient information. An efficient transformer model is also developed to obtain the long-term temporal dependencies and further alleviate storage constraints. We compare the performance of PgTransGAN against standard linear interpolation and solutions based solely on convolutional neural networks or generative adversarial networks, and demonstrate that our method achieves better reconstruction quality at the data, image, and physics levels with an upscaling factor of 4 or even 8 in each grid dimension.

MDSTF: a multi-dimensional spatio-temporal feature fusion trajectory prediction model for autonomous driving

In the field of autonomous driving, trajectory prediction of traffic agents is an important and challenging problem. Fully capturing the complex spatio-temporal features in trajectory data is crucial for accurate trajectory prediction. This paper proposes a trajectory prediction model called multi-dimensional spatio-temporal feature fusion (MDSTF), which integrates multi-dimensional spatio-temporal features to model the trajectory information of traffic agents. In the spatial dimension, we employ graph convolutional networks (GCN) to capture the local spatial features of traffic agents, spatial attention mechanism to capture the global spatial features, and LSTM combined with spatial attention to capture the full-process spatial features of traffic agents. Subsequently, these three spatial features are fused using a gate fusion mechanism. Moreover, during the modeling of the full-process spatial features, LSTM is capable of capturing short-term temporal dependencies in the trajectory information of traffic agents. In the temporal dimension, we utilize a Transformer-based encoder to extract long-term temporal dependencies in the trajectory information of traffic agents, which are then fused with the short-term temporal dependencies captured by LSTM. Finally, we employ two temporal convolutional networks (TCN) to predict trajectories based on the fused spatio-temporal features. Experimental results on the ApolloScape trajectory dataset demonstrate that our proposed method outperforms state-of-the-art methods in terms of weighted sum of average displacement error (WSADE) and weighted sum of final displacement error (WSFDE) metrics. Compared to the best baseline model (S2TNet), our method achieves reductions of 4.37% and 6.23% respectively in these metrics.

Spatial-temporal memory enhanced multi-level attention network for origin-destination demand prediction

Origin-destination demand prediction is a critical task in the field of intelligent transportation systems. However, accurately modeling the complex spatial-temporal dependencies presents significant challenges, which arises from various factors, including spatial, temporal, and external influences such as geographical features, weather conditions, and traffic incidents. Moreover, capturing multi-scale dependencies of local and global spatial dependencies, as well as short and long-term temporal dependencies, further complicates the task. To address these challenges, a novel framework called the Spatial-Temporal Memory Enhanced Multi-Level Attention Network (ST-MEN) is proposed. The framework consists of several key components. Firstly, an external attention mechanism is incorporated to efficiently process external factors into the prediction process. Secondly, a dynamic spatial feature extraction module is designed that effectively captures the spatial dependencies among nodes. By incorporating two skip-connections, this module preserves the original node information while aggregating information from other nodes. Finally, a temporal feature extraction module is proposed that captures both continuous and discrete temporal dependencies using a hierarchical memory network. In addition, multi-scale features cascade fusion is incorporated to enhance the performance of the proposed model. To evaluate the effectiveness of the proposed model, extensively experiments are conducted on two real-world datasets. The experimental results demonstrate that the ST-MEN model achieves excellent prediction accuracy, where the maximum improvement can reach to 19.1%.

Amplitude-Time Dual-View Fused EEG Temporal Feature Learning for Automatic Sleep Staging.

Electroencephalogram (EEG) plays an important role in studying brain function and human cognitive performance, and the recognition of EEG signals is vital to develop an automatic sleep staging system. However, due to the complex nonstationary characteristics and the individual difference between subjects, how to obtain the effective signal features of the EEG for practical application is still a challenging task. In this article, we investigate the EEG feature learning problem and propose a novel temporal feature learning method based on amplitude-time dual-view fusion for automatic sleep staging. First, we explore the feature extraction ability of convolutional neural networks for the EEG signal from the perspective of interpretability and construct two new representation signals for the raw EEG from the views of amplitude and time. Then, we extract the amplitude-time signal features that reflect the transformation between different sleep stages from the obtained representation signals by using conventional 1-D CNNs. Furthermore, a hybrid dilation convolution module is used to learn the long-term temporal dependency features of EEG signals, which can overcome the shortcoming that the small-scale convolution kernel can only learn the local signal variation information. Finally, we conduct attention-based feature fusion for the learned dual-view signal features to further improve sleep staging performance. To evaluate the performance of the proposed method, we test 30-s-epoch EEG signal samples for healthy subjects and subjects with mild sleep disorders. The experimental results from the most commonly used datasets show that the proposed method has better sleep staging performance and has the potential for the development and application of an EEG-based automatic sleep staging system.

Nonlinear slow-varying dynamics-assisted temporal graph transformer network for remaining useful life prediction

Remaining useful life (RUL) plays an important role in the prognostics and health management of mechanical systems. Recently, deep learning-based methods have been widely applied in the field of RUL prediction. However, there still suffer from two limitations. One is that the existing RUL prediction methods cannot capture spatial dependencies and long-term temporal dependencies. The other is that nonlinear slow-varying dynamics related to the degradation behavior have not been explored in the RUL prediction. To break these limitations, a nonlinear slow-varying dynamics-assisted temporal graph Transformer network (NSD-TGTN) is proposed in this paper for RUL prediction. NSD-TGTN can simultaneously capture and model spatiotemporal graphs and nonlinear slow-varying dynamics to achieve RUL prediction. Herein, the TGTN is developed to mine both spatial and long-term temporal dependencies for constructing the spatiotemporal features. And, nonlinear slow-varying features are built and introduced into the TGTN to enhance the RUL prediction capacity. Two datasets are utilized to validate the effectiveness and superiority of the proposed method. Compared with existing advanced methods, the average prediction accuracies of the NSD-TGTN on the C-MAPSS dataset and the wear dataset are improved by 1.70 % and 8.22 %, respectively.

PV Power Forecasting in the Hexi Region of Gansu Province Based on AP Clustering and LSTNet

Accurate PV power forecasting is becoming a mandatory task to integrate the PV system into the power grid, schedule it, and ensure the safety of the power grid. In this paper, a novel model for PV power prediction using AP-LSTNet has been proposed. It consists of a combination of affinity propagation clustering and long-term and short-term time series network models. First, the affinity propagation algorithm is used to divide the regionally distributed photovoltaic station clusters into different seasons. The Pearson correlation coefficient is used to determine the strong correlation between meteorological factors of photovoltaic power, and the bilinear interpolation method is used to encrypt the meteorological data of the corresponding photovoltaic station cluster. Furthermore, LSTNet is used to mine the long-term and short-term temporal and spatial dependence of photovoltaic power, and meteorological factor series and linear components of auto-regression are superimposed to realize the simultaneous prediction of multiple photovoltaic stations in the group. Finally, PV power plants in five cities, Wuwei, Jinchang, Zhangye, Jiuquan, and Jiayuguan in the Hexi region of Gansu Province, China, will be selected to test the proposed model. The experimental comparison shows that the prediction model achieves high prediction accuracy and robustness.

Predicting dust pollution from dry bulk ports in coastal cities: A hybrid approach based on data decomposition and deep learning

Dust pollution from storage and handling of materials in dry bulk ports seriously affects air quality and public health in coastal cities. Accurate prediction of dust pollution helps identify risks early and take preventive measures. However, there remain challenges in solving non-stationary time series and selecting relevant features. Besides, existing studies rarely consider impacts of port operations on dust pollution. Therefore, a hybrid approach based on data decomposition and deep learning is proposed to predict dust pollution from dry bulk ports. Port operational data is specially integrated into input features. A secondary decomposition and recombination (SDR) strategy is presented to reduce data non-stationarity. A dual-stage attention-based sequence-to-sequence (DA-Seq2Seq) model is employed to adaptively select the most relevant features at each time step, as well as capture long-term temporal dependencies. This approach is compared with baseline models on a dataset from a dry bulk port in northern China. The results reveal the advantages of SDR strategy and integrating operational data and show that this approach has higher accuracy than baseline models. The proposed approach can mitigate adverse effects of dust pollution from dry bulk ports on urban residents and help port authorities control dust pollution.

Fault detection of complicated processes based on an enhanced transformer network with graph attention mechanism

Recently, deep learning becomes increasingly popular in the industrial data analysis field due to its distinguished feature representation capability. As an emerging deep learning technology, transformer network has attracted extensive attention, but its application in industrial fault detection is still not sufficiently exploited. Furthermore, the traditional transformer network mainly focuses on the time series information of the data, while ignoring the spatial characteristic of the data. For this problem, an improved transformer network model with graph attention mechanism (GA-Tran) is proposed and used for complicated process fault detection. Considering the strong coupling property of process variables, the presented method performs joint spatial-temporal learning by mining the data information in views of both time-dimension and variable-dimension. On the one hand, a transformer encoder module equipped with a self-attention mechanism is utilized to capture long-term temporal dependencies, while multi-scale convolution is integrated to abstract short-term time dependencies. On the other hand, a graph attention network is introduced into the transformer network to meticulously analyze the spatial relationships among variables. Considering the topology of the industrial process variables, we use spectral clustering to infer prior topology information, and remove intelligently irrelevant spatial information. Different from the existing transformer models with the anomaly scores as the monitoring indicator, two monitoring statistics based on encoder features and reconstruction errors are constructed for process status monitoring. Studies on the Tennessee Eastman chemical process demonstrate the effectiveness of the proposed GA-Tran method.

Spatial-temporal graph neural ODE networks for skeleton-based action recognition.

In the field of skeleton-based action recognition, accurately recognizing human actions is crucial for applications such as virtual reality and motion analysis. However, this task faces challenges such intraindividual action differences and long-term temporal dependencies. To address these challenges, we propose an innovative model called spatial-temporal graph neural ordinary differential equations (STG-NODE). First, in the data preprocessing stage, the dynamic time warping (DTW) algorithm is used to normalize and calculate 3D skeleton data to facilitate the derivation of customized adjacency matrices for improving the influence of intraindividual action differences. Secondly, a custom ordinary differential equation (ODE) integrator is applied based on the initial conditions of the temporal features, producing a solution function that simulates the dynamic evolution trend of the events of interest. Finally, the outstanding ODE solver is used to numerically solve the time features based on the solution function to increase the influence of long-term dependencies on the recognition accuracy of the model and provide it with a more powerful temporal modeling ability. Through extensive experiments conducted on the NTU RGB+D 60 and Kinetics Skeleton 400 benchmark datasets, we demonstrate the superior performance of STG-NODE in the action recognition domain. The success of the STG-NODE model also provides new ideas and methods for the future development of the action recognition field.

Short-term wind power forecasting through stacked and bi directional LSTM techniques.

Computational intelligence (CI) based prediction models increase the efficient and effective utilization of resources for wind prediction. However, the traditional recurrent neural networks (RNN) are difficult to train on data having long-term temporal dependencies, thus susceptible to an inherent problem of vanishing gradient. This work proposed a method based on an advanced version of RNN known as long short-term memory (LSTM) architecture, which updates recurrent weights to overcome the vanishing gradient problem. This, in turn, improves training performance. The RNN model is developed based on stack LSTM and bidirectional LSTM. The parameters like mean absolute error (MAE), standard deviation error (SDE), and root mean squared error (RMSE) are utilized as performance measures for comparison with recent state-of-the-art techniques. Results showed that the proposed technique outperformed the existing techniques in terms of RMSE and MAE against all the used wind farm datasets. Whereas, a reduction in SDE is observed for larger wind farm datasets. The proposed RNN approach performed better than the existing models despite fewer parameters. In addition, the approach requires minimum processing power to achieve compatible results.

Mobile_ViT: Underwater Acoustic Target Recognition Method Based on Local–Global Feature Fusion

To overcome the challenges of inadequate representation and ineffective information exchange stemming from feature homogenization in underwater acoustic target recognition, we introduce a hybrid network named Mobile_ViT, which synergizes MobileNet and Transformer architectures. The network begins with a convolutional backbone incorporating an embedded coordinate attention mechanism to enhance the local details of inputs. This mechanism captures the long-term temporal dependencies and precise frequency–domain relationships of signals, focusing the features on the time–frequency positions. Subsequently, the Transformer’s Encoder is integrated at the end of the backbone to facilitate global characterization, thus effectively overcoming the convolutional neural network’s shortcomings in capturing long-range feature dependencies. Evaluation on the Shipsear and DeepShip datasets yields accuracies of 98.50% and 94.57%, respectively, marking a substantial improvement over the baseline. Notably, the proposed method also demonstrates obvious separation coefficients, signifying enhanced clustering effectiveness, and is lighter than other Transformers.

Joint learning of video scene detection and annotation via multi-modal adaptive context network

The tasks of scene detection and annotation have gained impressive attention for understanding video content. The main challenges lie in mitigating the error propagation of shot detection, recognizing cuts and gradual transitions, fusing hierarchical multi-modal cues, and solving these two tasks simultaneously. To address these challenges, we propose the Multi-modal Adaptive Context Network (MACN) to jointly learn scene detection and annotation from a window partitioning perspective. As a shared task-agnostic part, we perform Window-based Cross-modal Representation (WCR) to distill complex semantic correlations from multi-modal sources for each window. Considering the long-term temporal dependency of variable-length scenes, we further develop Adaptive Context-aware Representation (ACR) to improve the performance for specific tasks. Different from previous works, scene detection is formulated as locating the starting window and its associated location offset and transition duration. Meanwhile, we assemble two multi-label sub-classifiers in different levels to predict the labels for each scene candidate. Experimental comparisons to state-of-the-art algorithms on the TAVS and ClipShots indicate that the proposed method yields promising performance in both tasks. Our code and test sample videos are released at MACN.

TC-LIF: A Two-Compartment Spiking Neuron Model for Long-Term Sequential Modelling

The identification of sensory cues associated with potential opportunities and dangers is frequently complicated by unrelated events that separate useful cues by long delays. As a result, it remains a challenging task for state-of-the-art spiking neural networks (SNNs) to establish long-term temporal dependency between distant cues. To address this challenge, we propose a novel biologically inspired Two-Compartment Leaky Integrate-and-Fire spiking neuron model, dubbed TC-LIF. The proposed model incorporates carefully designed somatic and dendritic compartments that are tailored to facilitate learning long-term temporal dependencies. Furthermore, the theoretical analysis is provided to validate the effectiveness of TC-LIF in propagating error gradients over an extended temporal duration. Our experimental results, on a diverse range of temporal classification tasks, demonstrate superior temporal classification capability, rapid training convergence, and high energy efficiency of the proposed TC-LIF model. Therefore, this work opens up a myriad of opportunities for solving challenging temporal processing tasks on emerging neuromorphic computing systems. Our code is publicly available at https://github.com/ZhangShimin1/TC-LIF.

Goal-Guided Graph Attention Network with Interactive State Refinement for Multi-Agent Trajectory Prediction.

The accurate prediction of the future trajectories of traffic participants is crucial for enhancing the safety and decision-making capabilities of autonomous vehicles. Modeling social interactions among agents and revealing the inherent relationships is crucial for accurate trajectory prediction. In this context, we propose a goal-guided and interaction-aware state refinement graph attention network (SRGAT) for multi-agent trajectory prediction. This model effectively integrates high-precision map data and dynamic traffic states and captures long-term temporal dependencies through the Transformer network. Based on these dependencies, it generates multiple potential goals and Points of Interest (POIs). Through its dual-branch, multimodal prediction approach, the model not only proposes various plausible future trajectories associated with these POIs, but also rigorously assesses the confidence levels of each trajectory. This goal-oriented strategy enables SRGAT to accurately predict the future movement trajectories of other vehicles in complex traffic scenarios. Tested on the Argoverse and nuScenes datasets, SRGAT surpasses existing algorithms in key performance metrics by adeptly integrating past trajectories and current context. This goal-guided approach not only enhances long-term prediction accuracy, but also ensures its reliability, demonstrating a significant advancement in trajectory forecasting.

Local-Global Spatial-Temporal Graph Convolutional Network for Traffic Flow Forecasting

Traffic forecasting’s key challenge is to extract dynamic spatial-temporal features within intricate traffic systems. This paper introduces a novel framework for traffic prediction, named Local-Global Spatial-Temporal Graph Convolutional Network (LGSTGCN). The framework consists of three core components. Firstly, a graph attention residual network layer is proposed to capture global spatial dependencies by evaluating traffic mode correlations between different nodes. The context information added in the residual connection can improve the generalization ability of the model. Secondly, a T-GCN module, combining a Graph Convolution Network (GCN) with a Gated Recurrent Unit (GRU), is introduced to capture real-time local spatial-temporal dependencies. Finally, a transformer layer is designed to extract long-term temporal dependence and to identify the sequence characteristics of traffic data through positional encoding. Experiments conducted on four real traffic datasets validate the forecasting performance of the LGSTGCN model. The results demonstrate that LGSTGCN can achieve good performance and be applicable to traffic forecasting tasks.

Spatial–Temporal Feature Network for Speech-Based Depression Recognition

Depression is a serious mental disorder that has received increased attention from society. Due to the advantage of easy acquisition of speech, researchers have tried to propose various automatic depression recognition algorithms based on speech. Feature selection and algorithm design are the main difficulties in speech-based depression recognition. In our work, we propose the spatial-temporal feature network (STFN) for depression recognition, which can capture the long-term temporal dependence of audio sequences. First, to obtain a better feature representation for depression analysis, we develop a self-supervised learning framework, called Vector Quantized Wav2vec Transformer Net (VQWTNet) to map speech features and phonemes with transfer learning. Second, the stacked gated residual blocks in SFNet are used in the model to integrate causal and dilated convolutions to capture multiscale contextual information by continuously expanding the receptive field. In addition, instead of LSTM, our method employs the hierarchical contrastive predictive coding (HCPC) loss in HCPCNet to capture the long-term temporal dependencies of speech, reducing the number of parameters while making the network easier to train. Finally, experimental results on DAIC-WOZ (AVEC 2017) and E-DAIC (AVEC 2019) show that the proposed model significantly improves the accuracy of depression recognition. On both datasets, the performance of our method far exceeds the baseline and achieves competitive results compared to state-of-the-art methods.

LEISN: A long explicit–implicit spatio-temporal network for traffic flow forecasting

Recent studies have shown that it is necessary to further improve the prediction accuracy of complex and dynamic traffic flow from two angles: temporal dependence and spatial dependence. Although many spatio-temporal networks are extracting features from both temporal and spatial dependencies, there are still two problems that need to be addressed. Firstly, existing methods often focus on modeling local spatial dependence relationship between adjacent nodes, but neglect non-local spatial dependence relationship between distant nodes. Secondly, while recent work has explored various methods such as convolutional and recurrent neural networks for modeling temporal dependence, they may not fully capture the long-term temporal dependence in traffic data. Therefore, addressing the challenges of long-term dependency, explicit spatial dependency, and implicit spatial dependency, this paper proposes a Long-term Explicit–Implicit Spatio-Temporal Network (LEISN) for traffic flow prediction. A Long-term Dependency Module has been designed to store hidden states generated from multiple previous time steps, facilitating the transmission of long-term features. Based on this, two graph convolution-based spatial feature extraction branches are designed to extract explicit spatial features based on the adjacency matrix generated by spatial topology and implicit spatial features based on the implicit adjacency matrix generated by trend similarity, respectively. Then all spatio-temporal features are fused to produce the next state. Additionally, a new encoder framework, LENSI-ED, was proposed based on LEISN. Comparative experiments are conducted on four datasets, and the results showed that our model has advantages over existing methods. The proposed model addresses the issues of local and non-local spatial dependence relationships and long-term temporal dependence in traffic flow forecasting.