Recurrent Connections Research Articles

On spacetimes with a semi-symmetric recurrent metric connection

In this paper, we deal with spacetimes allowing a semi-symmetric connection whose metric tensor is recurrent. We investigate the impact of this connection on spacetimes and establish a nontrivial example. We physically explain the properties of spacetime that we obtained. Under a certain condition, we demostrate that a spacetime admitting such a connection becomes the GRW spacetime as well as the anti-de Sitter and represents the dark matter era. Finally, we scrutinize in case such a spacetime admits a Ricci soliton and generalized Ricci soliton.

Intrinsic dynamic shapes responses to external stimulation in the human brain.

Sensory stimulation of the brain reverberates in its recurrent neuronal networks. However, current computational models of brain activity do not separate immediate sensory responses from intrinsic recurrent dynamics. We apply a vector-autoregressive model with external input (VARX), combining the concepts of "functional connectivity" and "encoding models", to intracranial recordings in humans. We find that the recurrent connectivity during rest is largely unaltered during movie watching. The intrinsic recurrent dynamic enhances and prolongs the neural responses to scene cuts, eye movements, and sounds. Failing to account for these exogenous inputs, leads to spurious connections in the intrinsic "connectivity". The model shows that an external stimulus can reduce intrinsic noise. It also shows that sensory areas have mostly outward, whereas higher-order brain areas mostly incoming connections. We conclude that the response to an external audiovisual stimulus can largely be attributed to the intrinsic dynamic of the brain, already observed during rest.

Intrinsic dynamics of randomly clustered networks generate place fields and preplay of novel environments.

During both sleep and awake immobility, hippocampal place cells reactivate time-compressed versions of sequences representing recently experienced trajectories in a phenomenon known as replay. Intriguingly, spontaneous sequences can also correspond to forthcoming trajectories in novel environments experienced later, in a phenomenon known as preplay. Here, we present a model showing that sequences of spikes correlated with the place fields underlying spatial trajectories in both previously experienced and future novel environments can arise spontaneously in neural circuits with random, clustered connectivity rather than pre-configured spatial maps. Moreover, the realistic place fields themselves arise in the circuit from minimal, landmark-based inputs. We find that preplay quality depends on the network's balance of cluster isolation and overlap, with optimal preplay occurring in small-world regimes of high clustering yet short path lengths. We validate the results of our model by applying the same place field and preplay analyses to previously published rat hippocampal place cell data. Our results show that clustered recurrent connectivity can generate spontaneous preplay and immediate replay of novel environments. These findings support a framework whereby novel sensory experiences become associated with preexisting "pluripotent" internal neural activity patterns.

Intrinsic dynamics of randomly clustered networks generate place fields and preplay of novel environments

During both sleep and awake immobility, hippocampal place cells reactivate time-compressed versions of sequences representing recently experienced trajectories in a phenomenon known as replay. Intriguingly, spontaneous sequences can also correspond to forthcoming trajectories in novel environments experienced later, in a phenomenon known as preplay. Here, we present a model showing that sequences of spikes correlated with the place fields underlying spatial trajectories in both previously experienced and future novel environments can arise spontaneously in neural circuits with random, clustered connectivity rather than pre-configured spatial maps. Moreover, the realistic place fields themselves arise in the circuit from minimal, landmark-based inputs. We find that preplay quality depends on the network’s balance of cluster isolation and overlap, with optimal preplay occurring in small-world regimes of high clustering yet short path lengths. We validate the results of our model by applying the same place field and preplay analyses to previously published rat hippocampal place cell data. Our results show that clustered recurrent connectivity can generate spontaneous preplay and immediate replay of novel environments. These findings support a framework whereby novel sensory experiences become associated with preexisting “pluripotent” internal neural activity patterns.

Intrinsic study of generalized horizontal recurrent Finsler connection

Intrinsic study of generalized horizontal recurrent Finsler connection

Recurrent cortical networks encode natural sensory statistics via sequence filtering.

Recurrent neural networks can generate dynamics, but in sensory cortex it has been unclear if any dynamic processing is supported by the dense recurrent excitatory-excitatory network. Here we show a new role for recurrent connections in mouse visual cortex: they support powerful dynamical computations, but by filtering sequences of input instead of generating sequences. Using two-photon optogenetics, we measure neural responses to natural images and play them back, finding inputs are amplified when played back during the correct movie dynamic context- when the preceding sequence corresponds to natural vision. This sequence selectivity depends on a network mechanism: earlier input patterns produce responses in other local neurons, which interact with later input patterns. We confirm this mechanism by designing sequences of inputs that are amplified or suppressed by the network. These data suggest recurrent cortical connections perform predictive processing, encoding the statistics of the natural world in input-output transformations.

Exploring neural oscillations during speech perception via surrogate gradient spiking neural networks.

Understanding cognitive processes in the brain demands sophisticated models capable of replicating neural dynamics at large scales. We present a physiologically inspired speech recognition architecture, compatible and scalable with deep learning frameworks, and demonstrate that end-to-end gradient descent training leads to the emergence of neural oscillations in the central spiking neural network. Significant cross-frequency couplings, indicative of these oscillations, are measured within and across network layers during speech processing, whereas no such interactions are observed when handling background noise inputs. Furthermore, our findings highlight the crucial inhibitory role of feedback mechanisms, such as spike frequency adaptation and recurrent connections, in regulating and synchronizing neural activity to improve recognition performance. Overall, on top of developing our understanding of synchronization phenomena notably observed in the human auditory pathway, our architecture exhibits dynamic and efficient information processing, with relevance to neuromorphic technology.

Dynamical predictive coding with reservoir computing performs noise-robust multi-sensory speech recognition.

Multi-sensory integration is a perceptual process through which the brain synthesizes a unified perception by integrating inputs from multiple sensory modalities. A key issue is understanding how the brain performs multi-sensory integrations using a common neural basis in the cortex. A cortical model based on reservoir computing has been proposed to elucidate the role of recurrent connectivity among cortical neurons in this process. Reservoir computing is well-suited for time series processing, such as speech recognition. This inquiry focuses on extending a reservoir computing-based cortical model to encompass multi-sensory integration within the cortex. This research introduces a dynamical model of multi-sensory speech recognition, leveraging predictive coding combined with reservoir computing. Predictive coding offers a framework for the hierarchical structure of the cortex. The model integrates reliability weighting, derived from the computational theory of multi-sensory integration, to adapt to multi-sensory time series processing. The model addresses a multi-sensory speech recognition task, necessitating the management of complex time series. We observed that the reservoir effectively recognizes speech by extracting time-contextual information and weighting sensory inputs according to sensory noise. These findings indicate that the dynamic properties of recurrent networks are applicable to multi-sensory time series processing, positioning reservoir computing as a suitable model for multi-sensory integration.

Behavioral state and stimulus strength regulate the role of somatostatin interneurons in stabilizing network activity.

Inhibition stabilization enables cortical circuits to encode sensory signals across diverse contexts. Somatostatin-expressing (SST) interneurons are well-suited for this role through their strong recurrent connectivity with excitatory pyramidal cells. We developed a cortical circuit model predicting that SST cells become increasingly important for stabilization as sensory input strengthens. We tested this prediction in mouse primary visual cortex by manipulating excitatory input to SST cells, a key parameter for inhibition stabilization, with a novel cell-type specific pharmacological method to selectively block glutamatergic receptors on SST cells. Consistent with our model predictions, we find antagonizing glutamatergic receptors drives a paradoxical facilitation of SST cells with increasing stimulus contrast. In addition, we find even stronger engagement of SST-dependent stabilization when the mice are aroused. Thus, we reveal that the role of SST cells in cortical processing gradually switches as a function of both input strength and behavioral state.

Advancing image inpainting efficiency: An exploration of pixel and channel split operations

Deep learning-based image inpainting techniques have achieved unprecedented results using encoder–decoder structures to recover complex missing areas of an image. Recent inpainting models use additional information or networks (e.g., landmarks, edges, styles, and filters) to realize improved restoration performance, but at the cost of increased computational resources. To improve the relationship between inpainting performance and the number of model parameters, researchers have investigated efficient structural approaches such as recurrent and residual connection structures. However, these methods are difficult to apply in the general encoder–decoder structure. In this study, we explored the downsampling and upsampling operations associated with an encoder–decoder structure. We propose two novel split operations: the pixel-split operation (PSO) and channel-split operation (CSO). The proposed PSO transfers image features from high to low resolution with two dilation rate effects and a similar number of parameters as existing downsampling operations. Conversely, the proposed CSO increases the image resolution using only one-fourth the number of parameters of existing upsampling operations. The restoration performance and efficiency of the proposed model were evaluated in terms of five metrics on public datasets, e.g., the Places2 and CelebA datasets, to validate our proposed operations’ contribution to inpainting performance. We achieved state-of-the-art performance and reduced the size of the parameters by 20%. An ablation study was conducted to confirm the effect of each operation on the CelebA-HQ dataset. Results indicated that these split operations exhibit an advanced relationship between inpainting performance and optimization of the model parameters. The corresponding codes are available online (https://github.com/MrCAIcode/Split_operation_for_inpainting).

Inhibitory neurons marked by a connectivity molecule regulate memory precision.

The CA3 region is central to hippocampal function during learning and memory and has a unique connectivity. CA3 pyramidal neurons are the targets of huge, excitatory mossy fiber synapses from DG axons and have a high degree of excitatory recurrent connectivity. Thus, inhibition likely plays an outsized importance in constraining excitation and shaping CA3 ensembles during learning and memory. Here, we investigate the function of a never-before studied set of dendrite-targeting, GABAergic neurons defined by expression of the synaptic adhesion molecule, Kirrel3. We discovered that activating Kirrel3-expressing GABAergic neurons specifically impairs memory discrimination and inhibits CA3 pyramidal neurons in novel contexts. Kirrel3 is required for DG-to-GABA synapse formation and variants in Kirrel3 are strong risk factors for neurodevelopmental disorders. Thus, our work suggests that Kirrel3-GABA neurons are a critical source of feed-forward inhibition from DG to CA3 during the encoding and retrieval of contextual memories, a function which may be specifically disrupted in some brain disorders.

Time Makes Space: Emergence of Place Fields in Networks Encoding Temporally Continuous Sensory Experiences.

The vertebrate hippocampus is believed to use recurrent connectivity in area CA3 to support episodic memory recall from partial cues. This brain area also contains place cells, whose location-selective firing fields implement maps supporting spatial memory. Here we show that place cells emerge in networks trained to remember temporally continuous sensory episodes. We model CA3 as a recurrent autoencoder that recalls and reconstructs sensory experiences from noisy and partially occluded observations by agents traversing simulated arenas. The agents move in realistic trajectories modeled from rodents and environments are modeled as continuously varying, high-dimensional, sensory experience maps (spatially smoothed Gaussian random fields). Training our autoencoder to accurately pattern-complete and reconstruct sensory experiences with a constraint on total activity causes spatially localized firing fields, i.e., place cells, to emerge in the encoding layer. The emergent place fields reproduce key aspects of hippocampal phenomenology: a) remapping (maintenance of and reversion to distinct learned maps in different environments), implemented via repositioning of experience manifolds in the network's hidden layer, b) orthogonality of spatial representations in different arenas, c) robust place field emergence in differently shaped rooms, with single units showing multiple place fields in large or complex spaces, and d) slow representational drift of place fields. We argue that these results arise because continuous traversal of space makes sensory experience temporally continuous. We make testable predictions: a) rapidly changing sensory context will disrupt place fields, b) place fields will form even if recurrent connections are blocked, but reversion to previously learned representations upon remapping will be abolished, c) the dimension of temporally smooth experience sets the dimensionality of place fields, including during virtual navigation of abstract spaces.

A line attractor encoding a persistent internal state requires neuropeptide signaling

Internal states drive survival behaviors, but their neural implementation is poorly understood. Recently, we identified a line attractor in the ventromedial hypothalamus (VMH) that represents a state of aggressiveness. Line attractors can be implemented by recurrent connectivity or neuromodulatory signaling, but evidence for the latter is scant. Here, we demonstrate that neuropeptidergic signaling is necessary for line attractor dynamics in this system by using cell-type-specific CRISPR-Cas9-based gene editing combined with single-cell calcium imaging. Co-disruption of receptors for oxytocin and vasopressin in adult VMH Esr1+ neurons that control aggression diminished attack, reduced persistent neural activity, and eliminated line attractor dynamics while only slightly reducing overall neural activity and sex- or behavior-specific tuning. These data identify a requisite role for neuropeptidergic signaling in implementing a behaviorally relevant line attractor in mammals. Our approach should facilitate mechanistic studies in neuroscience that bridge different levels of biological function and abstraction.

Temporally Stable Metropolis Light Transport Denoising using Recurrent Transformer Blocks

Metropolis Light Transport (MLT) is a global illumination algorithm that is well-known for rendering challenging scenes with intricate light paths. However, MLT methods tend to produce unpredictable correlation artifacts in images, which can introduce visual inconsistencies for animation rendering. This drawback also makes it challenging to denoise MLT renderings while maintaining temporal stability. We tackle this issue with modern learning-based methods and build a sequence denoiser combining the recurrent connections with the cutting-edge vision transformer architecture. We demonstrate that our sophisticated denoiser can consistently improve the quality and temporal stability of MLT renderings with difficult light paths. Our method is efficient and scalable for complex scene renderings that require high sample counts.

Contributions of cortical neuron firing patterns, synaptic connectivity, and plasticity to task performance

Neuronal responses during behavior are diverse, ranging from highly reliable ‘classical’ responses to irregular ‘non-classically responsive’ firing. While a continuum of response properties is observed across neural systems, little is known about the synaptic origins and contributions of diverse responses to network function, perception, and behavior. To capture the heterogeneous responses measured from auditory cortex of rodents performing a frequency recognition task, we use a novel task-performing spiking recurrent neural network incorporating spike-timing-dependent plasticity. Reliable and irregular units contribute differentially to task performance via output and recurrent connections, respectively. Excitatory plasticity shifts the response distribution while inhibition constrains its diversity. Together both improve task performance with full network engagement. The same local patterns of synaptic inputs predict spiking response properties of network units and auditory cortical neurons from in vivo whole-cell recordings during behavior. Thus, diverse neural responses contribute to network function and emerge from synaptic plasticity rules.

Delineation of 12-Lead ECG Representative Beats Using Convolutional Encoder-Decoders with Residual and Recurrent Connections.

The aim of this study is to address the challenge of 12-lead ECG delineation by different encoder-decoder architectures of deep neural networks (DNNs). This study compares four concepts for encoder-decoders based on a fully convolutional architecture (CED-Net) and its modifications with a recurrent layer (CED-LSTM-Net), residual connections between symmetrical encoder and decoder feature maps (CED-U-Net), and sequential residual blocks (CED-Res-Net). All DNNs transform 12-lead representative beats to three diagnostic ECG intervals (P-wave, QRS-complex, QT-interval) used for the global delineation of the representative beat (P-onset, P-offset, QRS-onset, QRS-offset, T-offset). All DNNs were trained and optimized using the large PhysioNet ECG database (PTB-XL) under identical conditions, applying an advanced approach for machine-based supervised learning with a reference algorithm for ECG delineation (ETM, Schiller AG, Baar, Switzerland). The test results indicate that all DNN architectures are equally capable of reproducing the reference delineation algorithm's measurements in the diagnostic PTB database with an average P-wave detection accuracy (96.6%) and time and duration errors: mean values (-2.6 to 2.4 ms) and standard deviations (2.9 to 11.4 ms). The validation according to the standard-based evaluation practices of diagnostic electrocardiographs with the CSE database outlines a CED-Net model, which measures P-duration (2.6 ± 11.0 ms), PQ-interval (0.9 ± 5.8 ms), QRS-duration (-2.4 ± 5.4 ms), and QT-interval (-0.7 ± 10.3 ms), which meet all standard tolerances. Noise tests with high-frequency, low-frequency, and power-line frequency noise (50/60 Hz) confirm that CED-Net, CED-Res-Net, and CED-LSTM-Net are robust to all types of noise, mostly presenting a mean duration error < 2.5 ms when compared to measurements without noise. Reduced noise immunity is observed for the U-net architecture. Comparative analysis with other published studies scores this research within the lower range of time errors, highlighting its competitive performance.

Embryonically active piriform cortex neurons promote intracortical recurrent connectivity during development

Neuronal activity plays a critical role in the maturation of circuits that propagate sensory information into the brain. How widely does early activity regulate circuit maturation across the developing brain? Here, we used targeted recombination in active populations (TRAP) to perform a brain-wide survey for prenatally active neurons in mice and identified the piriform cortex as an abundantly TRAPed region. Whole-cell recordings in neonatal slices revealed preferential interconnectivity within embryonically TRAPed piriform neurons and their enhanced synaptic connectivity with other piriform neurons. Invivo Neuropixels recordings in neonates demonstrated that embryonically TRAPed piriform neurons exhibit broad functional connectivity within piriform and lead spontaneous synchronized population activity during a transient neonatal period, when recurrent connectivity is strengthening. Selectively activating or silencing these neurons in neonates enhanced or suppressed recurrent synaptic strength, respectively. Thus, embryonically TRAPed piriform neurons represent an interconnected hub-like population whose activity promotes recurrent connectivity in early development.

Intrinsic dynamics of randomly clustered networks generate place fields and preplay of novel environments.

During both sleep and awake immobility, hippocampal place cells reactivate time-compressed versions of sequences representing recently experienced trajectories in a phenomenon known as replay. Intriguingly, spontaneous sequences can also correspond to forthcoming trajectories in novel environments experienced later, in a phenomenon known as preplay. Here, we present a model showing that sequences of spikes correlated with the place fields underlying spatial trajectories in both previously experienced and future novel environments can arise spontaneously in neural circuits with random, clustered connectivity rather than pre-configured spatial maps. Moreover, the realistic place fields themselves arise in the circuit from minimal, landmark-based inputs. We find that preplay quality depends on the network's balance of cluster isolation and overlap, with optimal preplay occurring in small-world regimes of high clustering yet short path lengths. We validate the results of our model by applying the same place field and preplay analyses to previously published rat hippocampal place cell data. Our results show that clustered recurrent connectivity can generate spontaneous preplay and immediate replay of novel environments. These findings support a framework whereby novel sensory experiences become associated with preexisting "pluripotent" internal neural activity patterns.

Exploiting Signal Propagation Delays to Match Task Memory Requirements in Reservoir Computing.

Recurrent neural networks (RNNs) transmit information over time through recurrent connections. In contrast, biological neural networks use many other temporal processing mechanisms. One of these mechanisms is the inter-neuron delays caused by varying axon properties. Recently, this feature was implemented in echo state networks (ESNs), a type of RNN, by assigning spatial locations to neurons and introducing distance-dependent inter-neuron delays. These delays were shown to significantly improve ESN task performance. However, thus far, it is still unclear why distance-based delay networks (DDNs) perform better than ESNs. In this paper, we show that by optimizing inter-node delays, the memory capacity of the network matches the memory requirements of the task. As such, networks concentrate their memory capabilities to the points in the past which contain the most information for the task at hand. Moreover, we show that DDNs have a greater total linear memory capacity, with the same amount of non-linear processing power.

Neural network architectures in data sequence analysis: diagnosis of Reinke's oedema and Laryngeal polyps

In this review paper, we focus our attention on presenting selected neural network architectures dedicated to the analysis of sequential data, in particular to support the diagnosis of Reinke's oedema and laryngeal polyps. The research discussed here is located in the area of clinical computer decision support systems (CDS) based on the use of artificial recurrent neural networks (RNNs) for speech signal analysis. RNNs are able to predict time series due to their memory and local recurrent connections. In the experimental part, Elman-Jordan artificial neural networks are used, whose characteristics are speed and accuracy in pattern learning allowing real-time decision-making. In the review presented here, one important theme is the use of Bezier curves for preprocessing the speech signal, leading to data reduction and noise elimination. Elman-Jordan networks significantly speed up the learning process and show high classification accuracy in laryngopathy diagnosis.