Recurrent Architecture Research Articles

A new deep learning-based model for reconstructing high-quality NDVI time-series data in heavily cloudy areas: fusion of Sentinel 1 and 2 data

ABSTRACT Reconstructing high-quality Normalized Difference Vegetation Index time series data is essential for ecological and agricultural applications but remains challenging in heavily cloudy areas. Fusing Sentinel SAR and optical data with deep learning could be helpful but is also challenging for stable models due to unstable SAR-NDVI relationships caused by imaging mechanism differences and environmental complexities. In this study, we developed a new Bidirectional Recurrent Imputation for Optical-SAR fusion (BRIOS) model to reconstruct high-quality Sentinel-2 NDVI time series data. BRIOS designs a two-layer recurrent architecture that integrates the autocorrelation of discrete, cloud-free NDVI observations into the model for a more stable SAR-NDVI relationship. Evaluating BRIOS against three baseline methods (GF-SG spatiotemporal fusion, Harmonic regression interpolation, and MCNN-Seq deep learning) across three full Sentinel-2 tiles in reconstructing 8-day NDVI time series, BRIOS consistently outperformed in scenarios of either random or continuously missing data, as evidenced by lower RMSE values (e.g. 0.075 for BRIOS vs. 0.108 for GF-SG vs. 0.143 for Harmonic regression vs. 0.303 for MCNN-Seq), better Edge index, and high linear correlation coefficients (R values up to 0.97). Further ablation experiments revealed that deep integration of NDVI autocorrelation features and SAR temporal change patterns has improved the stability and generalization of BRIOS. Discussions on the model's scalability across various cloud sizes and training dataset sizes affirm its practicality for broad-scale application in vegetation monitoring under challenging cloudy conditions.

Recognition of Real-Time Video Activities Using Stacked Bi-GRU with Fusion-based Deep Architecture

Recognizing and understanding human activities in real-time videos is a challenging task due to the complex nature of video data and the need for efficient and accurate analysis. This research pioneers a breakthrough in video activity recognition by introducing a robust framework leveraging the power of a stacked Bidirectional Long Short-Term Memory (Bi-LSTM) and Gated Recurrent Unit (GRU) architecture, harmonized within a fusion-based deep model. The stacked Bi-LSTM-GRU model capitalizes on its dual recurrent architecture, capturing nuanced temporal dependencies within video sequences. The fusion-based deep architecture synergizes spatial and temporal features, enabling the model to discern intricate patterns in human activities. To further enhance the discriminative power of the model, we introduce a fusion module in the proposed deep architecture. The fusion module integrates multi-modal features extracted from different levels of the network hierarchy, allowing for a more comprehensive representation of video activities. We demonstrate the efficacy of our approach through rigorous experimentation on UCF50, UCF101, and HMDB51 datasets. In experiments on the UCF50 dataset, our model achieves an accuracy of 97.01% and 95.86% on training and validation sets respectively, showcasing its proficiency in discerning activities across a diverse range of scenarios. The evaluation extends to the UCF101 dataset, where the proposed approach achieves a competitive accuracy of 97.62% and 96.93% on training and validation sets, surpassing previous benchmarks by a margin of approx 1%. Further-more, on the challenging HMDB51 dataset, the model demonstrates a robust accuracy of 89.71%and 88.88% on training and validation sets, solidifying its efficacy in intricate action recognition tasks.

Automated cardiac coverage assessment in cardiovascular magnetic resonance imaging using an explainable recurrent 3D dual-domain convolutional network.

Cardiovascular magnetic resonance (CMR) imaging has become a modality with superior power for the diagnosis and prognosis of cardiovascular diseases. One of the essential quality controls of CMR images is to investigate the complete cardiac coverage, which is necessary for the volumetric and functional assessment. This study examines the full cardiac coverage using a 3D dual-domain convolutional model and then improves this model using an innovative explainable salient region detection model and a recurrent architecture. Salient regions are extracted from the short-axis cine CMR stacks using a three-step proposed algorithm. Changing the architecture of the 3D dual-domain convolutional model to a recurrent one and taking advantage of the salient region detection model creates a kind of attention mechanism that leads to improved results. The results obtained from the images of over 6200 participants of the UK Biobank population cohort study show the superiority of the proposed model over the previous studies. The dataset is the largest regarding the number of participants to control the cardiac coverage. The accuracies of the proposed model in identifying the presence/absence of basal/apical slices are 96.22% and 95.42%, respectively. The proposed recurrent architecture of the 3D dual-domain convolutional model can force the model to focus on the most informative areas of the images using the extracted salient regions, which can help the model improve accuracy. The performance of the proposed fully automated model indicates that it can be used for image quality control in population cohort datasets and real-time post-imaging quality assessments. Codes are available at https://github.com/mohammadhashemii/CMR_Cardiac_Coverage_Control.

Learning long sequences in spiking neural networks

Spiking neural networks (SNNs) take inspiration from the brain to enable energy-efficient computations. Since the advent of Transformers, SNNs have struggled to compete with artificial networks on modern sequential tasks, as they inherit limitations from recurrent neural networks (RNNs), with the added challenge of training with non-differentiable binary spiking activations. However, a recent renewed interest in efficient alternatives to Transformers has given rise to state-of-the-art recurrent architectures named state space models (SSMs). This work systematically investigates, for the first time, the intersection of state-of-the-art SSMs with SNNs for long-range sequence modelling. Results suggest that SSM-based SNNs can outperform the Transformer on all tasks of a well-established long-range sequence modelling benchmark. It is also shown that SSM-based SNNs can outperform current state-of-the-art SNNs with fewer parameters on sequential image classification. Finally, a novel feature mixing layer is introduced, improving SNN accuracy while challenging assumptions about the role of binary activations in SNNs. This work paves the way for deploying powerful SSM-based architectures, such as large language models, to neuromorphic hardware for energy-efficient long-range sequence modelling.

XLSTMTime: Long-Term Time Series Forecasting with xLSTM

In recent years, transformer-based models have gained prominence in multivariate long-term time series forecasting (LTSF), demonstrating significant advancements despite facing challenges such as high computational demands, difficulty in capturing temporal dynamics, and managing long-term dependencies. The emergence of LTSF-Linear, with its straightforward linear architecture, has notably outperformed transformer-based counterparts, prompting a reevaluation of the transformer’s utility in time series forecasting. In response, this paper presents an adaptation of a recent architecture, termed extended LSTM (xLSTM), for LTSF. xLSTM incorporates exponential gating and a revised memory structure with higher capacity that has good potential for LTSF. Our adopted architecture for LTSF, termed xLSTMTime, surpasses current approaches. We compare xLSTMTime’s performance against various state-of-the-art models across multiple real-world datasets, demonstrating superior forecasting capabilities. Our findings suggest that refined recurrent architectures can offer competitive alternatives to transformer-based models in LTSF tasks, potentially redefining the landscape of time series forecasting.

Compensation adaptive robust control for a linear motor–driven stage system with state and input constraints based on gated recurrent unit architecture

Compensation adaptive robust control for a linear motor–driven stage system with state and input constraints based on gated recurrent unit architecture

Multi-Directional Long-Term Recurrent Convolutional Network for Road Situation Recognition.

Understanding road conditions is essential for implementing effective road safety measures and driving solutions. Road situations encompass the day-to-day conditions of roads, including the presence of vehicles and pedestrians. Surveillance cameras strategically placed along streets have been instrumental in monitoring road situations and providing valuable information on pedestrians, moving vehicles, and objects within road environments. However, these video data and information are stored in large volumes, making analysis tedious and time-consuming. Deep learning models are increasingly utilized to monitor vehicles and identify and evaluate road and driving comfort situations. However, the current neural network model requires the recognition of situations using time-series video data. In this paper, we introduced a multi-directional detection model for road situations to uphold high accuracy. Deep learning methods often integrate long short-term memory (LSTM) into long-term recurrent network architectures. This approach effectively combines recurrent neural networks to capture temporal dependencies and convolutional neural networks (CNNs) to extract features from extensive video data. In our proposed method, we form a multi-directional long-term recurrent convolutional network approach with two groups equipped with CNN and two layers of LSTM. Additionally, we compare road situation recognition using convolutional neural networks, long short-term networks, and long-term recurrent convolutional networks. The paper presents a method for detecting and recognizing multi-directional road contexts using a modified LRCN. After balancing the dataset through data augmentation, the number of video files increased, resulting in our model achieving 91% accuracy, a significant improvement from the original dataset.

Modeling the Emergence of Circuit Organization and Function during Development.

Developing neural circuits show unique patterns of spontaneous activity and structured network connectivity shaped by diverse activity-dependent plasticity mechanisms. Based on extensive experimental work characterizing patterns of spontaneous activity in different brain regions over development, theoretical and computational models have played an important role in delineating the generation and function of individual features of spontaneous activity and their role in the plasticity-driven formation of circuit connectivity. Here, we review recent modeling efforts that explore how the developing cortex and hippocampus generate spontaneous activity, focusing on specific connectivity profiles and the gradual strengthening of inhibition as the key drivers behind the observed developmental changes in spontaneous activity. We then discuss computational models that mechanistically explore how different plasticity mechanisms use this spontaneous activity to instruct the formation and refinement of circuit connectivity, from the formation of single neuron receptive fields to sensory feature maps and recurrent architectures. We end by highlighting several open challenges regarding the functional implications of the discussed circuit changes, wherein models could provide the missing step linking immature developmental and mature adult information processing capabilities.

Sparse autoregressive neural networks for classical spin systems

Efficient sampling and approximation of Boltzmann distributions involving large sets of binary variables, or spins, are pivotal in diverse scientific fields even beyond physics. Recent advances in generative neural networks have significantly impacted this domain. However, these neural networks are often treated as black boxes, with architectures primarily influenced by data-driven problems in computational science. Addressing this gap, we introduce a novel autoregressive neural network architecture named TwoBo, specifically designed for sparse two-body interacting spin systems. We directly incorporate the Boltzmann distribution into its architecture and parameters, resulting in enhanced convergence speed, superior free energy accuracy, and reduced trainable parameters. We perform numerical experiments on disordered, frustrated systems with more than 1000 spins on grids and random graphs, and demonstrate its advantages compared to previous autoregressive and recurrent architectures. Our findings validate a physically informed approach and suggest potential extensions to multivalued variables and many-body interaction systems, paving the way for broader applications in scientific research.

Excitatory and inhibitory neuronal synapse unit: A novel recurrent cell for time series prediction

Time series prediction is one of the most challenging topics in real-world applications. Most studies undertook advanced mathematics and computer techniques above benchmark methods but often unable to guarantee forecasting models’ generalization and reliability. This paper proposed a biologically motivated recurrent unit based on neuronal synaptic activity mechanism and chaotic behaviors in deep learning domain called Excitatory and Inhibitory Neural Synapse unit (EINS). The major contribution of this research is to create a flexible and robust recurrent unit based on neuroscience theory. It conducted experiments on three real-world time series forecast examples including finance, household power, weather for generalization and robustness evaluation using eight state-of-the-art recurrent architecture units as baseline models to compare convergence rate and forecasting accuracy. Experimental results showed that EINS achieved satisfactory performances in time series prediction reliability and effectiveness.

Error-feedback three-phase optimization to configurable convolutional echo state network for time series forecasting

Time series forecasting is critical for many real-world applications. Convolutional echo state networks (CESNs) have shown intriguing time series modeling efficacy by combining convolutional neural network (CNN) and echo state network (ESN). However, current CESN models are tailored for the classification tasks and rely on elaborately designed neural architectures. To this end, we propose a novel configurable convolutional echo state network (CCESN) with an innovative error-feedback three-phase optimization (ETO) strategy for time series forecasting. The network is progressively constructed with heterogeneous modular subnetworks, including ESN, CNN, CESN, and reversed CESN modules. This scheme leverages the complementary feature extraction capabilities of convolutional and recurrent neural architectures. To adaptively evolve the CCESN, we propose a novel error-feedback three-phase optimization (ETO) strategy by selecting optimal subnetwork modules while step-wise tuning parameters. Comprehensive experiments are conducted on representative simulated and real-world datasets. The results indicate that ETO-CCESN can adaptively select and evolve heterogeneous subnetworks to acclimatize to varied scenarios, and thus demonstrate significant performance improvements, achieving a 45.69% average enhancement in forecasting accuracy compared to the existing CESN model, and surpassing the best baseline by 8.79% in terms of symmetric mean absolute percentage error across diverse forecasting tasks.

Towards Efficient Recurrent Architectures: A Deep LSTM Neural Network Applied to Speech Enhancement and Recognition

Towards Efficient Recurrent Architectures: A Deep LSTM Neural Network Applied to Speech Enhancement and Recognition

Systematic analysis of speech transcription modeling for reliable assessment of depression severity

For depression severity assessment, we systematically analyze a modular deep learning pipeline that uses speech transcriptions as input for depression severity prediction. Through our pipeline, we investigate the role of popular deep learning architectures in creating representations for depression assessment. Evaluation of the proposed architectures is performed on the publicly available Extended Distress Analysis Interview Corpus dataset (E-DAIC). Through the results and discussions, we show that informative representations for depression assessment can be obtained without exploiting the temporal dynamics between descriptive text representations. More specifically, temporal pooling of latent representations outperforms the state of the art, which employs recurrent architectures, by 8.8% in terms of Concordance Correlation Coefficient (CCC).

Peringkasan Teks Otomatis Abstraktif Menggunakan Transformer Pada Teks Bahasa Indonesia

Abstractive text summarization is used to generate summaries that are similar to human-written summaries. To achieve that capability, recurrent deep learning architectures were usually applied, such as RNN, LSTM, and GRU. In previous studies about abstractive text summarization in Indonesian, recurrent models were widely used and there were cohesion and grammar errors that appeared in the generated summaries—this could have an impact on performance. Currently, there is Transformer, a relatively new architecture that relies on the attention mechanism entirely. Due to its non-recurrent nature, Transformer overcomes the problem of dependency the on hidden states that occurs in recurrent models and it can retain information on all input sequences. In this study, we use Transformer to evaluate how good it is at abstractive text summarization in Indonesian. The training was conducted using the pre-trained T5 model with IndoSum dataset which contains around 19K news-summary pairs. We achieved evaluation scores of 0.61 ROUGE-1 and 0.51 ROUGE-2.

Real-Time Multi-Person Video Synthesis with Controllable Prior-Guided Matting.

In order to enhance the matting performance in multi-person dynamic scenarios, we introduce a robust, real-time, high-resolution, and controllable human video matting method that achieves state of the art on all metrics. Unlike most existing methods that perform video matting frame by frame as independent images, we design a unified architecture using a controllable generation model to solve the problem of the lack of overall semantic information in multi-person video. Our method, called ControlMatting, uses an independent recurrent architecture to exploit temporal information in videos and achieves significant improvements in temporal coherence and detailed matting quality. ControlMatting adopts a mixed training strategy comprised of matting and a semantic segmentation dataset, which effectively improves the semantic understanding ability of the model. Furthermore, we propose a novel deep learning-based image filter algorithm that enforces our detailed augmentation ability on both matting and segmentation objectives. Our experiments have proved that prior information about the human body from the image itself can effectively combat the defect masking problem caused by complex dynamic scenarios with multiple people.

DenRAM: neuromorphic dendritic architecture with RRAM for efficient temporal processing with delays

Neuroscience findings emphasize the role of dendritic branching in neocortical pyramidal neurons for non-linear computations and signal processing. Dendritic branches facilitate temporal feature detection via synaptic delays that enable coincidence detection (CD) mechanisms. Spiking neural networks highlight the significance of delays for spatio-temporal pattern recognition in feed-forward networks, eliminating the need for recurrent structures. Here, we introduce DenRAM, a novel analog electronic feed-forward spiking neural network with dendritic compartments. Utilizing 130 nm technology integrated with resistive RAM (RRAM), DenRAM incorporates both delays and synaptic weights. By configuring RRAMs to emulate bio-realistic delays and exploiting their heterogeneity, DenRAM mimics synaptic delays and efficiently performs CD for pattern recognition. Hardware-aware simulations on temporal benchmarks show DenRAM’s robustness against hardware noise, and its higher accuracy over recurrent networks. DenRAM advances temporal processing in neuromorphic computing, optimizes memory usage, and marks progress in low-power, real-time signal processing

Heterogeneous Forgetting Rates and Greedy Allocation in Slot-Based Memory Networks Promotes Signal Retention.

A key question in the neuroscience of memory encoding pertains to the mechanisms by which afferent stimuli are allocated within memory networks. This issue is especially pronounced in the domain of working memory, where capacity is finite. Presumably the brain must embed some "policy" by which to allocate these mnemonic resources in an online manner in order to maximally represent and store afferent information for as long as possible and without interference from subsequent stimuli. Here, we engage this question through a top-down theoretical modeling framework. We formally optimize a gating mechanism that projects afferent stimuli onto a finite number of memory slots within a recurrent network architecture. In the absence of external input, the activity in each slot attenuates over time (i.e., a process of gradual forgetting). It turns out that the optimal gating policy consists of a direct projection from sensory activity to memory slots, alongside an activity-dependent lateral inhibition. Interestingly, allocating resources myopically (greedily with respect to the current stimulus) leads to efficient utilization of slots over time. In other words, later-arriving stimuli are distributed across slots in such a way that the network state is minimally shifted and so prior signals are minimally "overwritten." Further, networks with heterogeneity in the timescales of their forgetting rates retain stimuli better than those that are more homogeneous. Our results suggest how online, recurrent networks working on temporally localized objectives without high-level supervision can nonetheless implement efficient allocation of memory resources over time.

Explainability in AI-based behavioral malware detection systems

Nowadays, our security and privacy are strongly threatened by malware programs which aim to steal our confidential data and make our systems out of service, among other things. While traditional signature-based malware detection methods or statistical analysis have proven to be ineffective and time-consuming, recently data-driven Artificial Intelligence (AI) techniques, i.e. Machine Learning (ML) and Deep Learning (DL) approaches, have been successfully applied leveraging the behavior of malware in terms of API calls, and achieving promising performances. However, their black-box behavior leads to a lack of explainability thus preventing their application in real world scenarios. In light of this, eXplainable Artificial Intelligence (XAI) methodologies and tools can be effectively embedded within an AI-based malware detection process in order to make more understandable the produced results. In this paper, we propose a XAI framework for behavioral malware detection problems and evaluate the usefulness of four XAI methods (SHAP, LIME, LRP and Attention mechanism) on three datasets with different size, sequence length and number of classes, by which we could evaluate the strengths and weaknesses – from effectiveness and efficiency point of views – of recurrent deep architectures (i.e. Long-Short Term Memory (LSTM) and Gated Recurrent Unit (GRU) models), and their applicability in the modern Cyber Security (CS) scenarios.

Neural Oscillators for Generalization of Physics-Informed Machine Learning

A primary challenge of physics-informed machine learning (PIML) is its generalization beyond the training domain, especially when dealing with complex physical problems represented by partial differential equations (PDEs). This paper aims to enhance the generalization capabilities of PIML, facilitating practical, real-world applications where accurate predictions in unexplored regions are crucial. We leverage the inherent causality and temporal sequential characteristics of PDE solutions to fuse PIML models with recurrent neural architectures based on systems of ordinary differential equations, referred to as neural oscillators. Through effectively capturing long-time dependencies and mitigating the exploding and vanishing gradient problem, neural oscillators foster improved generalization in PIML tasks. Extensive experimentation involving time-dependent nonlinear PDEs and biharmonic beam equations demonstrates the efficacy of the proposed approach. Incorporating neural oscillators outperforms existing state-of-the-art methods on benchmark problems across various metrics. Consequently, the proposed method improves the generalization capabilities of PIML, providing accurate solutions for extrapolation and prediction beyond the training data.

FC-TFS-CGRU: A Temporal-Frequency-Spatial Electroencephalography Emotion Recognition Model Based on Functional Connectivity and a Convolutional Gated Recurrent Unit Hybrid Architecture.

The gated recurrent unit (GRU) network can effectively capture temporal information for 1D signals, such as electroencephalography and event-related brain potential, and it has been widely used in the field of EEG emotion recognition. However, multi-domain features, including the spatial, frequency, and temporal features of EEG signals, contribute to emotion recognition, while GRUs show some limitations in capturing frequency-spatial features. Thus, we proposed a hybrid architecture of convolutional neural networks and GRUs (CGRU) to effectively capture the complementary temporal features and spatial-frequency features hidden in signal channels. In addition, to investigate the interactions among different brain regions during emotional information processing, we considered the functional connectivity relationship of the brain by introducing a phase-locking value to calculate the phase difference between the EEG channels to gain spatial information based on functional connectivity. Then, in the classification module, we incorporated attention constraints to address the issue of the uneven recognition contribution of EEG signal features. Finally, we conducted experiments on the DEAP and DREAMER databases. The results demonstrated that our model outperforms the other models with remarkable recognition accuracy of 99.51%, 99.60%, and 99.59% (58.67%, 65.74%, and 67.05%) on DEAP and 98.63%, 98.7%, and 98.71% (75.65%, 75.89%, and 71.71%) on DREAMER in a subject-dependent experiment (subject-independent experiment) for arousal, valence, and dominance.