Recurrent Connections Research Articles

Cholinergic modulation supports dynamic switching of resting state networks through selective DMN suppression.

Brain activity during the resting state is widely used to examine brain organization, cognition and alterations in disease states. While it is known that neuromodulation and the state of alertness impact resting-state activity, neural mechanisms behind such modulation of resting-state activity are unknown. In this work, we used a computational model to demonstrate that change in excitability and recurrent connections, due to cholinergic modulation, impacts resting-state activity. The results of such modulation in the model match closely with experimental work on direct cholinergic modulation of Default Mode Network (DMN) in rodents. We further extended our study to the human connectome derived from diffusion-weighted MRI. In human resting-state simulations, an increase in cholinergic input resulted in a brain-wide reduction of functional connectivity. Furthermore, selective cholinergic modulation of DMN closely captured experimentally observed transitions between the baseline resting state and states with suppressed DMN fluctuations associated with attention to external tasks. Our study thus provides insight into potential neural mechanisms for the effects of cholinergic neuromodulation on resting-state activity and its dynamics.

Unveiling the neural dynamics of conscious perception in rapid object recognition

Our brain excels at recognizing objects, even when they flash by in a rapid sequence. However, the neural processes determining whether a target image in a rapid sequence can be recognized or not remains elusive. We used electroencephalography (EEG) to investigate the temporal dynamics of brain processes that shape perceptual outcomes in these challenging viewing conditions. Using naturalistic images and advanced multivariate pattern analysis (MVPA) techniques, we probed the brain dynamics governing conscious object recognition. Our results show that although initially similar, the processes for when an object can or cannot be recognized diverge around 180 ms post-appearance, coinciding with feedback neural processes. Decoding analyses indicate that gist perception (partial conscious perception) can occur at ∼120 ms through feedforward mechanisms. In contrast, object identification (full conscious perception of the image) is resolved at ∼190 ms after target onset, suggesting involvement of recurrent processing. These findings underscore the importance of recurrent neural connections in object recognition and awareness in rapid visual presentations.

Object detection on low-resolution images with two-stage enhancement

Although deep learning-based object detection methods have achieved superior performance on conventional benchmark datasets, it is still difficult to detect objects from low-resolution (LR) images under diverse degradation conditions. To this end, a two-stage enhancement method for the LR image object detection (TELOD) framework is proposed. In the first stage, an extremely lightweight task disentanglement enhancement network (TDEN) is developed as a super-resolution (SR) sub-network before the detector. In the TDEN, the SR images can be obtained by applying the recurrent connection manner between an image restoration branch (IRB) and a resolution enhancement branch (REB) to enhance the input LR images. Specifically, the TDEN reduces the difficulty of image reconstruction by dividing the total image enhancement task into two sub-tasks, which are accomplished by the IRB and REB, respectively. Furthermore, a shared feature extractor is applied across two sub-tasks to explore common and accurate feature representations. In the second stage, an auxiliary feature enhancement head (AFEH) driven by high-resolution (HR) image priors is designed to improve the task-specific features produced by the detection Neck without any extra inference costs. In particular, the feature interaction module is built into the AFEH to integrate the features from the enhancement and detection phases to learn comprehensive information for detection. Extensive experiments show that the proposed TELOD significantly outperforms other methods. Specifically, the TELOD achieves mAP improvements of 1.8% and 3.3% over the second best method AERIS on degraded VOC and COCO datasets, respectively.

Optimising air quality prediction in smart cities with hybrid particle swarm optimization‐long‐short term memory‐recurrent neural network model

AbstractIn smart cities, air pollution is a critical issue that affects individual health and harms the environment. The air pollution prediction can supply important information to all relevant parties to take appropriate initiatives. Air quality prediction is a hot area of research. The existing research encounters several challenges that is, poor accuracy and incorrect real‐time updates. This research presents a hybrid model based on long‐short term memory (LSTM), recurrent neural network (RNN), and Curiosity‐based Motivation method. The proposed model extracts a feature set from the training dataset using an RNN layer and achieves sequencing learning by applying an LSTM layer. Also, to deal with the overfitting issues in LSTM, the proposed model utilises a dropout strategy. In the proposed model, input and recurrent connections can be dropped from activation and weight updates using the dropout regularisation approach, and it utilises a Curiosity‐based Motivation model to construct a novel motivational model, which helps in the reconstruction of long short‐term memory recurrent neural network. To minimise the prediction error, particle swarm optimisation is implemented to optimise the LSTM neural network's weights. The authors utilise an online Air Pollution Monitoring dataset from Salt Lake City, USA with five air quality indicators for comparison, that is, SO2, CO, O3, and NO2, to predict air quality. The proposed model is compared with existing Gradient Boosted Tree Regression, Existing LSTM, and Support Vector Machine based Regression Model. Experimental analysis shows that the proposed method has 0.0184 (Root Mean Square Error (RMSE)), 0.0082 (Mean Absolute Error), 2002*109 (Mean Absolute Percentage Error), and 0.122 (R2‐Score). The experimental findings demonstrate that the proposed LSTM model had RMSE performance in the prescribed dataset and statistically significant superior outcomes compared to existing methods.

Hue selectivity from recurrent circuitry in Drosophila

In the perception of color, wavelengths of light reflected off objects are transformed into the derived quantities of brightness, saturation and hue. Neurons responding selectively to hue have been reported in primate cortex, but it is unknown how their narrow tuning in color space is produced by upstream circuit mechanisms. We report the discovery of neurons in the Drosophila optic lobe with hue-selective properties, which enables circuit-level analysis of color processing. From our analysis of an electron microscopy volume of a whole Drosophila brain, we construct a connectomics-constrained circuit model that accounts for this hue selectivity. Our model predicts that recurrent connections in the circuit are critical for generating hue selectivity. Experiments using genetic manipulations to perturb recurrence in adult flies confirm this prediction. Our findings reveal a circuit basis for hue selectivity in color vision.

A predictor-informed multi-subject bayesian approach for dynamic functional connectivity.

Dynamic functional connectivity investigates how the interactions among brain regions vary over the course of an fMRI experiment. Such transitions between different individual connectivity states can be modulated by changes in underlying physiological mechanisms that drive functional network dynamics, e.g., changes in attention or cognitive effort. In this paper, we develop a multi-subject Bayesian framework where the estimation of dynamic functional networks is informed by time-varying exogenous physiological covariates that are simultaneously recorded in each subject during the fMRI experiment. More specifically, we consider a dynamic Gaussian graphical model approach where a non-homogeneous hidden Markov model is employed to classify the fMRI time series into latent neurological states. We assume the state-transition probabilities to vary over time and across subjects as a function of the underlying covariates, allowing for the estimation of recurrent connectivity patterns and the sharing of networks among the subjects. We further assume sparsity in the network structures via shrinkage priors, and achieve edge selection in the estimated graph structures by introducing a multi-comparison procedure for shrinkage-based inferences with Bayesian false discovery rate control. We evaluate the performances of our method vs alternative approaches on synthetic data. We apply our modeling framework on a resting-state experiment where fMRI data have been collected concurrently with pupillometry measurements, as a proxy of cognitive processing, and assess the heterogeneity of the effects of changes in pupil dilation on the subjects' propensity to change connectivity states. The heterogeneity of state occupancy across subjects provides an understanding of the relationship between increased pupil dilation and transitions toward different cognitive states.

Fast learning without synaptic plasticity in spiking neural networks

Spiking neural networks are of high current interest, both from the perspective of modelling neural networks of the brain and for porting their fast learning capability and energy efficiency into neuromorphic hardware. But so far we have not been able to reproduce fast learning capabilities of the brain in spiking neural networks. Biological data suggest that a synergy of synaptic plasticity on a slow time scale with network dynamics on a faster time scale is responsible for fast learning capabilities of the brain. We show here that a suitable orchestration of this synergy between synaptic plasticity and network dynamics does in fact reproduce fast learning capabilities of generic recurrent networks of spiking neurons. This points to the important role of recurrent connections in spiking networks, since these are necessary for enabling salient network dynamics. We show more specifically that the proposed synergy enables synaptic weights to encode more general information such as priors and task structures, since moment-to-moment processing of new information can be delegated to the network dynamics.

Script identification in handwritten and printed documents using convolutional recurrent connection

Script identification in handwritten and printed documents using convolutional recurrent connection

An encoder-decoder ConvLSTM surrogate model for simulating geological CO2 sequestration with dynamic well controls

In Geological Carbon Sequestration (GCS), effectively managing the project requires predicting state variables such as pressure and saturation. However, numerical simulation of multiphase flow in subsurface porous media involves solving large linear algebra systems, resulting in a substantial computational burden. This can make it impractical for real-time history matching or optimization. Meanwhile, deep learning-based surrogate models are emerging as fast and accurate approximators. This study delves into the application of the Encoder-Decoder Convolutional Long Short-Term Memory (ED-ConvLSTM) neural network for predicting the complex evolution of state variables under dynamic CO2 injection schemes. Inception blocks enhanced with light-weighted attention modules are introduced in the encoder to extract high-dimensional input features. ConvLSTM is employed to propagate spatial temporal information in the low-dimensional latent space. Further, progressive upsampling blocks are used to reconstruct the latent features for the desired output. Instead of taking discrete time steps as an input feature, the proposed network captures the dynamic dependencies with the inherent ConvLSTM cell. The network has access to data at only portion of the initial time steps during training stage, while it is used to predict the state variables at unseen time steps during testing stage. Results show that the network can produce excellent predictions for both pressure and saturation, even at unseen future time steps. The remarkable generalizability to different geological permeability fields is also evaluated. ED-ConvLSTM outperforms the standard U-Net by far, especially when predicting beyond the training time period. These numerical experiments demonstrate the advantages of ED-ConvLSTM in terms of prediction accuracy, extrapolability and generalizability. This study highlights the importance of incorporating recurrent connections into the deep neural networks for simulating time-dependent multiphase flow problems. The proposed methodology has great potential in GCS surrogate modeling and offers a possible approach for real-time optimization of CO2 injection.

Circuits in the motor cortex explain oscillatory responses to transcranial magnetic stimulation.

Transcranial magnetic stimulation (TMS) is a popular method used to investigate brain function. Stimulation over the motor cortex evokes muscle contractions known as motor evoked potentials (MEPs) and also high-frequency volleys of electrical activity measured in the cervical spinal cord. The physiological mechanisms of these experimentally derived responses remain unclear, but it is thought that the connections between circuits of excitatory and inhibitory neurons play a vital role. Using a spiking neural network model of the motor cortex, we explained the generation of waves of activity, so called 'I-waves', following cortical stimulation. The model reproduces a number of experimentally known responses including direction of TMS, increased inhibition, and changes in strength. Using populations of thousands of neurons in a model of cortical circuitry we showed that the cortex generated transient oscillatory responses without any tuning, and that neuron parameters such as refractory period and delays influenced the pattern and timing of those oscillations. By comparing our network with simpler, previously proposed circuits, we explored the contributions of specific connections and found that recurrent inhibitory connections are vital in producing later waves that significantly impact the production of motor evoked potentials in downstream muscles (Thickbroom, 2011). This model builds on previous work to increase our understanding of how complex circuitry of the cortex is involved in the generation of I-waves.

Modeling and contractivity of neural-synaptic networks with Hebbian learning

This paper is concerned with the modeling and analysis of two of the most commonly used recurrent neural network models (i.e., Hopfield neural network and firing-rate neural network) with dynamic recurrent connections undergoing Hebbian learning rules. To capture the synaptic sparsity of neural circuits we propose a low dimensional formulation. We then characterize certain key dynamical properties. First, we give biologically-inspired forward invariance results. Then, we give sufficient conditions for the non-Euclidean contractivity of the models. Our contraction analysis leads to stability and robustness of time-varying trajectories — for networks with both excitatory and inhibitory synapses governed by both Hebbian and anti-Hebbian rules. For each model, we propose a contractivity test based upon biologically meaningful quantities, e.g., neural and synaptic decay rate, maximum in-degree, and the maximum synaptic strength. Then, we show that the models satisfy Dale’s Principle. Finally, we illustrate the effectiveness of our results via a numerical example.

Recurrent feature propagation and edge skip-connections for automatic abdominal organ segmentation

Automatic segmentation of abdominal organs in computed tomography (CT) images can support radiation therapy and image-guided surgery workflows. Development of such automatic solutions remains challenging mainly owing to complex organ interactions and blurry boundaries in CT images. To address these issues, we focus on effective spatial context modeling and explicit edge segmentation priors. Accordingly, we propose a 3D network with four main components trained end-to-end including shared encoder, edge detector, decoder with edge skip-connections (ESCs) and recurrent feature propagation head (RFP-Head). To capture wide-range spatial dependencies, the RFP-Head propagates and harvests local features through directed acyclic graphs (DAGs) formulated with recurrent connections in an efficient slice-wise manner, with regard to spatial arrangement of image units. To leverage edge information, the edge detector learns edge prior knowledge specifically tuned for semantic segmentation by exploiting intermediate features from the encoder with the edge supervision. The ESCs then aggregate the edge knowledge with multi-level decoder features to learn a hierarchy of discriminative features explicitly modeling complementarity between organs’ interiors and edges for segmentation. We conduct extensive experiments on challenging abdominal CT datasets with eight annotated organs. Experimental results show that the proposed network outperforms several state-of-the-art models, especially for the segmentation of small and complicated structures (gallbladder, esophagus, stomach, pancreas and duodenum).

CA3 Circuit Model Compressing Sequential Information in Theta Oscillation and Replay.

The hippocampus plays a critical role in the compression and retrieval of sequential information. During wakefulness, it achieves this through theta phase precession and theta sequences. Subsequently, during periods of sleep or rest, the compressed information reactivates through sharp-wave ripple events, manifesting as memory replay. However, how these sequential neuronal activities are generated and how they store information about the external environment remain unknown. We developed a hippocampal cornu ammonis 3 (CA3) computational model based on anatomical and electrophysiological evidence from the biological CA3 circuit to address these questions. The model comprises theta rhythm inhibition, place input, and CA3-CA3 plastic recurrent connection. The model can compress the sequence of the external inputs, reproduce theta phase precession and replay, learn additional sequences, and reorganize previously learned sequences. A gradual increase in synaptic inputs, controlled by interactions between theta-paced inhibition and place inputs, explained the mechanism of sequence acquisition. This model highlights the crucial role of plasticity in the CA3 recurrent connection and theta oscillational dynamics and hypothesizes how the CA3 circuit acquires, compresses, and replays sequential information.

Cortical cell assemblies and their underlying connectivity: An in silico study.

Recent developments in experimental techniques have enabled simultaneous recordings from thousands of neurons, enabling the study of functional cell assemblies. However, determining the patterns of synaptic connectivity giving rise to these assemblies remains challenging. To address this, we developed a complementary, simulation-based approach, using a detailed, large-scale cortical network model. Using a combination of established methods we detected functional cell assemblies from the stimulus-evoked spiking activity of 186,665 neurons. We studied how the structure of synaptic connectivity underlies assembly composition, quantifying the effects of thalamic innervation, recurrent connectivity, and the spatial arrangement of synapses on dendrites. We determined that these features reduce up to 30%, 22%, and 10% of the uncertainty of a neuron belonging to an assembly. The detected assemblies were activated in a stimulus-specific sequence and were grouped based on their position in the sequence. We found that the different groups were affected to different degrees by the structural features we considered. Additionally, connectivity was more predictive of assembly membership if its direction aligned with the temporal order of assembly activation, if it originated from strongly interconnected populations, and if synapses clustered on dendritic branches. In summary, reversing Hebb's postulate, we showed how cells that are wired together, fire together, quantifying how connectivity patterns interact to shape the emergence of assemblies. This includes a qualitative aspect of connectivity: not just the amount, but also the local structure matters; from the subcellular level in the form of dendritic clustering to the presence of specific network motifs.

Shared input and recurrency in neural networks for metabolically efficient information transmission.

Shared input to a population of neurons induces noise correlations, which can decrease the information carried by a population activity. Inhibitory feedback in recurrent neural networks can reduce the noise correlations and thus increase the information carried by the population activity. However, the activity of inhibitory neurons is costly. This inhibitory feedback decreases the gain of the population. Thus, depolarization of its neurons requires stronger excitatory synaptic input, which is associated with higher ATP consumption. Given that the goal of neural populations is to transmit as much information as possible at minimal metabolic costs, it is unclear whether the increased information transmission reliability provided by inhibitory feedback compensates for the additional costs. We analyze this problem in a network of leaky integrate-and-fire neurons receiving correlated input. By maximizing mutual information with metabolic cost constraints, we show that there is an optimal strength of recurrent connections in the network, which maximizes the value of mutual information-per-cost. For higher values of input correlation, the mutual information-per-cost is higher for recurrent networks with inhibitory feedback compared to feedforward networks without any inhibitory neurons. Our results, therefore, show that the optimal synaptic strength of a recurrent network can be inferred from metabolically efficient coding arguments and that decorrelation of the input by inhibitory feedback compensates for the associated increased metabolic costs.

A consistent map in the medial entorhinal cortex supports spatial memory

The medial entorhinal cortex (MEC) is hypothesized to function as a cognitive map for memory-guided navigation. How this map develops during learning and influences memory remains unclear. By imaging MEC calcium dynamics while mice successfully learned a novel virtual environment over ten days, we discovered that the dynamics gradually became more spatially consistent and then stabilized. Additionally, grid cells in the MEC not only exhibited improved spatial tuning consistency, but also maintained stable phase relationships, suggesting a network mechanism involving synaptic plasticity and rigid recurrent connectivity to shape grid cell activity during learning. Increased c-Fos expression in the MEC in novel environments further supports the induction of synaptic plasticity. Unsuccessful learning lacked these activity features, indicating that a consistent map is specific for effective spatial memory. Finally, optogenetically disrupting spatial consistency of the map impaired memory-guided navigation in a well-learned environment. Thus, we demonstrate that the establishment of a spatially consistent MEC map across learning both correlates with, and is necessary for, successful spatial memory.

Prediction of treatment response in major depressive disorder using a hybrid of convolutional recurrent deep neural networks and effective connectivity based on EEG signal.

In this study, we have developed a novel method based on deep learning and brain effective connectivity to classify responders and non-responders to selective serotonin reuptake inhibitors (SSRIs) antidepressants in major depressive disorder (MDD) patients prior to the treatment using EEG signal. The effective connectivity of 30 MDD patients was determined by analyzing their pretreatment EEG signals, which were then concatenated into delta, theta, alpha, and beta bands and transformed into images. Using these images, we then fine tuned a hybrid Convolutional Neural Network that is enhanced with bidirectional Long Short-Term Memory cells based on transfer learning. The Inception-v3, ResNet18, DenseNet121, and EfficientNet-B0 models are implemented as base models. Finally, the models are followed by BiLSTM and dense layers in order to classify responders and non-responders to SSRI treatment. Results showed that the EfficiencyNet-B0 has the highest accuracy of 98.33, followed by DensNet121, ResNet18 and Inception-v3. Therefore, a new method was proposed in this study that uses deep learning models to extract both spatial and temporal features automatically, which will improve classification results. The proposed method provides accurate identification of MDD patients who are responding, thereby reducing the cost of medical facilities and patient care.

Deep Learning-Based Managing Disaster with Smart Education in Environmental Sustainability while Avoiding Fake News

Networking continues to increase day by day to invest increasingly our daily, creating a huge volume of various and precise data: it is not easy to collect content, especially in crisis times. We focus on Smart Education proposed as the primary tool of a hybrid of Deep Convolutional Neural Networks (CNN)-Long Short-Term Memory (LSTM)-based model to retrieve content efficiently: CNN is used to extract meaningful features from multiple sources, enabling to have qualitative and sure information, notably with an efficient fake news security, and LSTM is used to maintain long-term dependencies in the extracted features with recurrent connections. This model has been compared to previous approaches to the performance of a publicly available dataset to demonstrate its highly satisfactory performance. This new approach makes it possible to integrate artificial intelligence technologies, deep learning, social media, and detecting or avoiding fake news into the crisis management model. It is based on an extension of our previous approach, namely disaster management based on short-term memory and education: this experience constitutes a background for this model. It combines representation training with awareness and education while retrieving pattern information by combining various search results from multiple sources. We have extended it to improve our disaster management model and evaluate it in the case of COVID-19 while obtaining promising results, through past programs and experiences that have shown overwhelmingly positive effects of education for vulnerability reduction and disaster risk management, in the pursuit of Environmental Sustainability

EEG-based emotion recognition using MobileNet Recurrent Neural Network with time-frequency features

Despite the developments in deep learning, extracting different features from brain signals remains a crucial challenge in EEG-based emotion recognition. This study introduces a novel methodology to overcome the challenge of capturing temporal and channel-related features. The basic idea of the proposed method is to use scalograms to capture subtle emotional patterns in EEG signals, followed by MobileNet Recurrent Neural Network (MRNN). Unlike conventional methods, MRNN utilizes recurrent connections to learn sequential patterns and recognize complex temporal dependencies in the data, enhancing EEG-based emotion recognition. The model is trained and tested on two publicly available datasets, DEAP and DREAMER, and achieved better results at 85.95 % for 15 classes and 96.29 % for 9 classes accuracy on average. Our approach extends beyond binary or limited multi-class emotion classification and significantly improves accuracy compared to the state-of-the-art methods. This research can have potential applications in developing emotion recognition systems for real-world scenarios, such as mental health monitoring or human-computer interaction.

Macroscale intrinsic dynamics are associated with microcircuit function in focal and generalized epilepsies

Epilepsies are a group of neurological disorders characterized by abnormal spontaneous brain activity, involving multiscale changes in brain functional organizations. However, it is not clear to what extent the epilepsy-related perturbations of spontaneous brain activity affect macroscale intrinsic dynamics and microcircuit organizations, that supports their pathological relevance. We collect a sample of patients with temporal lobe epilepsy (TLE) and genetic generalized epilepsy with tonic-clonic seizure (GTCS), as well as healthy controls. We extract massive temporal features of fMRI BOLD time-series to characterize macroscale intrinsic dynamics, and simulate microcircuit neuronal dynamics used a large-scale biological model. Here we show whether macroscale intrinsic dynamics and microcircuit dysfunction are differed in epilepsies, and how these changes are linked. Differences in macroscale gradient of time-series features are prominent in the primary network and default mode network in TLE and GTCS. Biophysical simulations indicate reduced recurrent connection within somatomotor microcircuits in both subtypes, and even more reduced in GTCS. We further demonstrate strong spatial correlations between differences in the gradient of macroscale intrinsic dynamics and microcircuit dysfunction in epilepsies. These results emphasize the impact of abnormal neuronal activity on primary network and high-order networks, suggesting a systematic abnormality of brain hierarchical organization.