Temporal Attention Research Articles

Multi-Attention Recurrent Neural Network for Multi-Step Prediction of Chlorophyll Concentration

Abstract: Chlorophyll prediction facilitates the comprehension of red tide characteristics and enables early warning. In practice, it is formulated as a multivariate time series forecasting problem aimed at forecasting future chlorophyll concentrations by considering both exogenous factors and chlorophyll. However, the multi-step prediction of chlorophyll concentration poses a formidable challenge due to the intricate interaction between factors and the long temporal dependence between input sequences. In this work, we propose a Multi-attention Recurrent Neural Network (MaRNN) for the multi-step prediction of chlorophyll concentration. The MaRNN comprises an encoder incorporating two-stage spatial attention and a decoder employing temporal attention. The encoder first learns the significance of exogenous factors for prediction in the first phase, and subsequently captures the spatial correlation between the exogenous sequence and chlorophyll sequence in the second phase. The decoder further excavates input sequences that exhibit a strong correlation with the task through temporal attention module, thereby enhancing the prediction accuracy of the model. Experiments conducted on two real-world datasets reveal that MaRNN not only surpasses state-of-the-art methods in performance, but also offers interpretability for chlorophyll prediction.

Distributed Photovoltaic Communication Anomaly Detection Based on Spatiotemporal Feature Collaborative Modeling

As distributed photovoltaic (PV) technology rapidly develops and is widely applied, the methods of cyberattacks are continuously evolving, posing increasingly severe threats to the communication networks of distributed PV systems. Recent studies have shown that the Transformer model, which effectively integrates global information and handles long-distance dependencies, has garnered significant attention. Based on this, our research proposes a model named STformer, which is applied to the task of attack detection in distributed PV communication. Specifically, we propose a temporal attention mechanism and a variable attention mechanism. The temporal attention mechanism focuses on capturing subtle changes and trends in data sequences over time, ensuring a highly sensitive recognition of patterns inherent in time-series data. In contrast, the variable attention mechanism analyzes the intrinsic relationships and interactions between different variables, uncovering critical correlations that may indicate abnormal behavior or potential attacks. Additionally, we incorporate the Uniform Manifold Approximation and Projection (UMAP) dimensionality reduction technique. This technique not only helps reduce computational complexity but, in certain cases, can enhance anomaly detection performance. Finally, compared to classical and advanced methods, STformer demonstrates satisfactory performance in simulation experiments.

A Two-Stream Method for Human Action Recognition Using Facial Action Cues

Human action recognition (HAR) is a critical area in computer vision with wide-ranging applications, including video surveillance, healthcare monitoring, and abnormal behavior detection. Current HAR methods predominantly rely on full-body data, which can limit their effectiveness in real-world scenarios where occlusion is common. In such situations, the face often remains visible, providing valuable cues for action recognition. This paper introduces Face in Action (FIA), a novel two-stream method that leverages facial action cues for robust action recognition under conditions of significant occlusion. FIA consists of an RGB stream and a landmark stream. The RGB stream processes facial image sequences using a fine-spatio-multitemporal (FSM) 3D convolution module, which employs smaller spatial receptive fields to capture detailed local facial movements and larger temporal receptive fields to model broader temporal dynamics. The landmark stream processes facial landmark sequences using a normalized temporal attention (NTA) module within an NTA-GCN block, enhancing the detection of key facial frames and improving overall recognition accuracy. We validate the effectiveness of FIA using the NTU RGB+D and NTU RGB+D 120 datasets, focusing on action categories related to medical conditions. Our experiments demonstrate that FIA significantly outperforms existing methods in scenarios with extensive occlusion, highlighting its potential for practical applications in surveillance and healthcare settings.

Anticipatory and evoked visual cortical dynamics of voluntary temporal attention.

We can often anticipate the precise moment when a stimulus will be relevant for our behavioral goals. Voluntary temporal attention, the prioritization of sensory information at task-relevant time points, enhances visual perception. However, the neural mechanisms of voluntary temporal attention have not been isolated from those of temporal expectation, which reflects timing predictability rather than relevance. Here we use time-resolved steady-state visual evoked responses (SSVER) to investigate how temporal attention dynamically modulates visual activity when temporal expectation is controlled. We recorded magnetoencephalography while participants directed temporal attention to one of two sequential grating targets with predictable timing. Meanwhile, a co-localized SSVER probe continuously tracked visual cortical modulations both before and after the target stimuli. We find that in the pre-target period, the SSVER gradually ramps up as the targets approach, reflecting temporal expectation. Furthermore, we find a low-frequency modulation of the SSVER, which shifts approximately half a cycle in phase according to which target is attended. In the post-target period, temporal attention to the first target transiently modulates the SSVER shortly after target onset. Thus, temporal attention dynamically modulates visual cortical responses via both periodic pre-target and transient post-target mechanisms to prioritize sensory information at precise moments.

Rhythm-based Temporal Expectations: Unique Contributions of Predictability and Periodicity.

Anticipating events and focusing attention accordingly are crucial for navigating our dynamic environment. Rhythmic patterns of sensory input offer valuable cues for temporal expectations and facilitate perceptual processing. Rhythm-based temporal expectations may rely on oscillatory entrainment, where neural activity and perceptual sensitivity synchronize with periodic stimuli. However, whether entrainment models can account for aperiodic predictable rhythms remains unclear. Our study aimed to delineate the distinct roles of predictability and periodicity in rhythm-based expectations. Participants performed a pitch-identification task preceded by periodic predictable, aperiodic predictable, or aperiodic unpredictable temporal sequences. By manipulating the temporal position of the target sound, we observed how auditory perceptual performance was modulated by the target position's relative phase relationship to the preceding sequences. Results revealed a significant performance advantage for predictable sequences, both periodic and aperiodic, compared with unpredictable ones. However, only the periodic sequence induced an entrained modulation pattern, with performance peaking in synchrony with the inherent sequence continuation. Event-related brain potentials corroborated these findings. The target-evoked P3b, possibly a neural marker of attention allocation, mirrored the behavioral performance patterns. This supports our hypothesis that temporal attention guided by rhythm-based expectations modulates perceptual performance. Furthermore, the predictive sequences were associated with enhanced target-preceding negativity (akin to the contingent negative variation), indicating enhanced target preparation. The periodic-specific modulation likely reflects more precise temporal expectations, potentially involving neural entrainment and/or more focused attention. Our findings suggest that predictability and periodicity influence perception through distinct mechanisms.

Dynamic Spatio-Temporal Modeling for Enhanced Air Quality Prediction: Implications for Information Management and Public Health Decision Support Systems

Air quality forecasting has significant implications for environmental monitoring, public health, and information management systems. This study proposes three advanced deep learning models: Graph Neural Networks with Dynamic Spatio-Temporal Attention (GNN-DSTA), Multi-Resolution Convolutional Recurrent Neural Networks (MRC-RNN), and Variational Autoencoders with Spatio-Temporal Latent Embeddings (VAE-STLE). These models address the challenges of capturing complex spatio-temporal dependencies in environments with sparse data samples. The GNN-DSTA model introduces a temporal attention mechanism that dynamically captures evolving spatial-temporal dependencies. MRC-RNN combines CNN's spatial pattern recognition with RNN's temporal modeling across multiple spatial resolutions. VAE-STLE provides a probabilistic framework for robust and interpretable forecasting. Experimental results demonstrate significant improvements in prediction accuracy: GNN-DSTA reduces RMSE by 15-20%, MRC-RNN improves accuracy by 12-15%, and VAE-STLE shows a 10-12% improvement with enhanced uncertainty estimation. These models advance AQI predictions through dynamic attention mechanisms, multi-resolution analysis, and probabilistic forecasting. Furthermore, this study explores the implications of these advanced predictions for information management and public health decision support systems. We discuss the integration of real-time data, scalability considerations for large-scale deployments, user interface design for effective communication of predictions, and ethical considerations in using AI-driven models for public health decision-making. The proposed approach not only enhances the accuracy and reliability of AQI predictions but also provides a framework for developing more effective and responsive public health interventions and environmental policies.

Deep video steganography using temporal-attention-based frame selection and spatial sparse adversarial attack

With the development of deep learning-based steganalysis, video steganography is facing with great challenges. To address the insufficient security against steganalysis of existing deep video steganography, given that the video has both spatial and temporal dimensions, this paper proposes a deep video steganography method using temporal frame selection and spatial sparse adversarial attack. In temporal dimension, a stego frame selection module based on temporal attention is designed to calculate the weight of each frame and selects frames with high weights for message and sparse perturbation embedding. In spatial dimension, sparse adversarial perturbations are performed in the selected frames to improve the ability of resisting steganalysis. Moreover, to control the adversarial perturbations’ sparsity flexibly, an intra-frame dynamic sparsity threshold mechanism is designed by using percentile. Experimental results demonstrate that the proposed method effectively enhances the visual quality and security against steganalysis of video steganography and has controllable sparsity of adversarial perturbations.

End‐to‐end speech‐denoising deep neural network based on residual‐attention gated linear units

AbstractIn this letter, an improved gated linear unit (GLU) structure for end‐to‐end (E2E) speech enhancement is proposed. In the U‐Net structure, which is widely used as the foundational architecture for E2E deep neural network‐based speech denoising, the input noisy speech signal undergoes multiple layers of encoding and is compressed into essential potential representative information at the bottleneck. The latent information is then transmitted to the decoder stage for the restoration of the target clean speech. Among these approaches, CleanUNet, a prominent state‐of‐the‐art (SOTA) method, enhances temporal attention in latent space by employing multi‐head self‐attention. However, unlike the approach of applying the attention mechanism to the potentially compressed representative information of the bottleneck layer, the proposed method instead assigns the attention module to the GLU of each encoder/decoder block layer. The proposed method is validated by measuring short‐term objective speech intelligibility and sound quality. The objective evaluation results indicated that the proposed method using residual‐attention GLU outperformed existing methods using SOTA models such as FAIR‐denoiser and CleanUNet across signal‐to‐noise ratios ranging from 0 to 15 dB.

Local Weather and Global Climate Data-Driven Long-Term Runoff Forecasting Based on Local–Global–Temporal Attention Mechanisms and Graph Attention Networks

The accuracy of long-term runoff models can be increased through the input of local weather variables and global climate indices. However, existing methods do not effectively extract important information from complex input factors across various temporal and spatial dimensions, thereby contributing to inaccurate predictions of long-term runoff. In this study, local–global–temporal attention mechanisms (LGTA) were proposed for capturing crucial information on global climate indices on monthly, annual, and interannual time scales. The graph attention network (GAT) was employed to extract geographical topological information of meteorological stations, based on remotely sensed elevation data. A long-term runoff prediction model was established based on long-short-term memory (LSTM) integrated with GAT and LGTA, referred to as GAT–LGTA–LSTM. The proposed model was compared to five comparative models (LGTA–LSTM, GAT–GTA–LSTM, GTA–LSTM, GAT–GA–LSTM, GA–LSTM). The models were applied to forecast the long-term runoff at Luning and Pingshan stations in China. The results indicated that the GAT–LGTA–LSTM model demonstrated the best forecasting performance among the comparative models. The Nash–Sutcliffe Efficiency (NSE) of GAT–LGTA–LSTM at the Luning and Pingshan stations reached 0.87 and 0.89, respectively. Compared to the GA–LSTM benchmark model, the GAT–LGTA–LSTM model demonstrated an average increase in NSE of 0.07, an average increase in Kling–Gupta Efficiency (KGE) of 0.08, and an average reduction in mean absolute percent error (MAPE) of 0.12. The excellent performance of the proposed model is attributed to the following: (1) local attention mechanism assigns a higher weight to key global climate indices at a monthly scale, enhancing the ability of global and temporal attention mechanisms to capture the critical information at annual and interannual scales and (2) the global attention mechanism integrated with GAT effectively extracts crucial temporal and spatial information from precipitation and remotely-sensed elevation data. Furthermore, attention visualization reveals that various global climate indices contribute differently to runoff predictions across distinct months. The global climate indices corresponding to specific seasons or months should be selected to forecast the respective monthly runoff.

Spiking Neural Networks for event-based action recognition: A new task to understand their advantage

Spiking Neural Networks (SNN) are characterised by their unique temporal dynamics, but the properties and advantages of such computations are still not well understood. In order to provide answers, in this work we demonstrate how Spiking neurons can enable temporal feature extraction in feed-forward neural networks without the need for recurrent synapses, and how recurrent SNNs can achieve comparable results to LSTM with a smaller number of parameters. This shows how their bio-inspired computing principles can be successfully exploited beyond energy efficiency gains and evidences their differences with respect to conventional artificial neural networks. These results are obtained through a new task, DVS-Gesture-Chain (DVS-GC), which allows, for the first time, to evaluate the perception of temporal dependencies in a real event-based action recognition dataset. Our study proves how the widely used DVS Gesture benchmark can be solved by networks without temporal feature extraction when its events are accumulated in frames, unlike the new DVS-GC which demands an understanding of the order in which events happen. Furthermore, this setup allowed us to reveal the role of the leakage rate in spiking neurons for temporal processing tasks and demonstrated the benefits of ”hard reset” mechanisms. Additionally, we also show how time-dependent weights and normalisation can lead to understanding order by means of temporal attention. Code for the DVS-GC task is available.

TKSTAGNet: A Top-K Spatio-Temporal Attention Gating Network for air pollution prediction

Accurate air pollution prediction is of great significance for human health, travel, and environmental management. As an emerging artificial intelligence technology, Transformer has already shown tremendous potential in air pollution prediction. Unfortunately, existing Transformer-based methods are challenging to accurately predict air pollution due to two limitations: (1) they neglect irrelevant spatial and temporal neighbors (e.g., neighboring sensors and timesteps) when modeling spatio-temporal correlations, and (2) they lack an adaptive mechanism to mine different effects caused by various external factors (e.g., meteorological conditions). Jointly taking these limitations into account, we propose a novel Transformer-based architecture for air pollution prediction, named Top-K Spatio-Temporal Attention Gating Network (TKSTAGNet). In this architecture, to effectively extract spatio-temporal correlations, we first design a TopKpooling Spatial Attention (TopK-SA) and a TopKpooling Temporal Attention (TopK-TA). In TopK-SA, the TopKpooling mechanism is introduced into spatial attention to filter out irrelevant spatial neighbors, accurately capturing spatial dependencies. In TopK-TA, the TopKpooling mechanism is integrated with temporal attention to identify and exclude irrelevant temporal neighbors, precisely extracting temporal dependencies. To adaptively fuse various external factors, a gating fusion module is then proposed. We conduct experiments on three real-world air pollution datasets from urban regions in Beijing, Shanghai, and Chongqing, respectively. Experimental results demonstrate that TKSTAGNet outperforms state-of-the-art baselines in prediction precision and fitting the variation trends of air pollution.

Trajectory Privacy-Protection Mechanism Based on Multidimensional Spatial–Temporal Prediction

The popularity of global GPS location services and location-enabled personal terminal applications has contributed to the rapid growth of location-based social networks. Users can access social networks at anytime and anywhere to obtain services in the relevant location. While accessing services is convenient, there is a potential risk of leaking users’ private information. In data processing, the discovery of issues and the generation of optimal solutions constitute a symmetrical process. Therefore, this paper proposes a symmetry–trajectory differential privacy-protection mechanism based on multi-dimensional prediction (TPPM-MP). Firstly, the temporal attention mechanism is designed to extract spatiotemporal features of trajectories from different spatiotemporal dimensions and perform trajectory-sensitive prediction. Secondly, class-prevalence-based weights are assigned to sensitive regions. Finally, the privacy budget is assigned based on the sensitive weights, and noise conforming to localized differential privacy is added. Validated on real datasets, the proposed method in this paper enhanced usability by 22% and 37% on the same dataset compared with other methods mentioned, while providing equivalent privacy protection.

Temporal attention and expectation jointly modulate microsaccades

Temporal attention and expectation jointly modulate microsaccades

Temporal attention and expectation interact regardless of expectation’s trial sequence

Temporal attention and expectation interact regardless of expectation’s trial sequence

Isolating neural mechanisms of voluntary temporal attention

Isolating neural mechanisms of voluntary temporal attention

Evidence of Rhythmic Environmental Sampling in a Cued Temporal Attention Paradigm

Evidence of Rhythmic Environmental Sampling in a Cued Temporal Attention Paradigm

Efficient prediction framework for large-scale nonlinear petrochemical process based on feature selection and temporal-attention LSTM: Applied to fluid catalytic cracking

Data-driven modeling plays a vital role in petrochemical industry, especially fluid catalytic cracking (FCC). However, the long operational cycles, large-scale measurements, multivariate data, and intricate temporal correlations in FCC units may lead to the problem of low prediction accuracy when only use a single time series data-driven modeling neural network such as long short-term memory (LSTM) network. To address these challenges, an effective prediction framework is proposed that integrates LSTM network with extreme gradient boosting (XGBOOST)-based feature selection and temporal-attention (TA) mechanisms. XGBOOST is applied to filter features related to the predictive variables in order to eliminate redundant variables. TA mechanisms within an LSTM network is used to capture the relevant historical time steps of the current moment. The results of our efficient prediction framework, applied to FCC process and three additional petrochemical processes, have proven to be superior to other methods.

A time patch dynamic attention transformer for enhanced well production forecasting in complex oilfield operations

In the field of oil production, accurately predicting well production is of great significance for optimizing production processes and enhancing economic benefits. However, oil production data generally exhibit characteristics such as high noise, complex distribution, and severe temporal fluctuations, which pose significant challenges to traditional prediction models, thereby limiting their accuracy. To address this issue, this study proposes an innovative Time Patch Dynamic Attention Transformer (TPDAT) model. This model segments the time series data into multiple time patches, extracts local features, and integrates a dynamically fused causal temporal attention mechanism to enhance the recognition of transient events and short-term fluctuations. The TPDAT model comprises the Temporal Patch Feature Extraction (TPF) module and the Dynamic Fusion Causal Temporal Attention (DFCTA) mechanism, effectively capturing multi-scale temporal features and enhancing the modeling of causal relationships during oilfield production. Experiments on the North Sea Volve oilfield dataset show that the TPDAT model significantly outperforms traditional deep learning models in well production prediction, achieves the best performance in model evaluation metrics, and demonstrates strong adaptability to complex data. The results confirm that the TPDAT model improves the accuracy and stability of well production prediction in complex oilfield environments, providing reliable support for oilfield production optimization.

A Semantic and Motion-Aware Spatiotemporal Transformer Network for Action Detection.

This paper presents a novel spatiotemporal transformer network that introduces several original components to detect actions in untrimmed videos. First, the multi-feature selective semantic attention model calculates the correlations between spatial and motion features to model spatiotemporal interactions between different action semantics properly. Second, the motion-aware network encodes the locations of action semantics in video frames utilizing the motion-aware 2D positional encoding algorithm. Such a motion-aware mechanism memorizes the dynamic spatiotemporal variations in action frames that current methods cannot exploit. Third, the sequence-based temporal attention model captures the heterogeneous temporal dependencies in action frames. In contrast to standard temporal attention used in natural language processing, primarily aimed at finding similarities between linguistic words, the proposed sequence-based temporal attention is designed to determine both the differences and similarities between video frames that jointly define the meaning of actions. The proposed approach outperforms the state-of-the-art solutions on four spatiotemporal action datasets: AVA 2.2, AVA 2.1, UCF101-24, and EPIC-Kitchens.

Temporal attention network for CNNE model of variable-length ECG signals in early arrhythmia detection

<p>Cardiac arrhythmia identification and categorization are crucial for prompt treatment and better patient outcomes. Arrhythmia identification is the main focus of this study's temporal attention network (TAN)-based multiclass categorization of varied-length electrocardiogram (ECG) data. The suggested TAN is designed to handle variable-duration ECG signals, making it ideal for real-time monitoring. The TAN uses a dynamic snippet extraction approach to choose meaningful ECG segments to ensure the model captures essential properties despite the constraints of processing such heterogeneous data. Training and assessment use a large dataset of atrial fibrillation, ventricular, and supraventricular arrhythmias. The TAN outperforms current approaches in multiclass early arrhythmia classification and is very accurate. Concatenating EfficientNet with CNN layer helped overcome different data and variable-length signals. High accuracy: 98% of normal, 97.1% of atrial fibrillation (AF), 98% of other, and 98% of noisy using the proposed CEEC model. Early arrhythmia diagnosis has improved due to the TAN's ability to effectively identify varied-length ECG data and give interpretability. It enables quicker interventions, personalised treatment plans, and improved arrhythmia control, which can greatly benefit patient care.</p>