Synaptic Weight Matrix Research Articles

Energy-information trade-off induces continuous and discontinuous phase transitions in lateral predictive coding

Lateral predictive coding is a recurrent neural network that creates energy-efficient internal representations by exploiting statistical regularity in sensory inputs. Here, we analytically investigate the trade-off between information robustness and energy in a linear model of lateral predictive coding and numerically minimize a free energy quantity. We observed several phase transitions in the synaptic weight matrix, particularly a continuous transition that breaks reciprocity and permutation symmetry and builds cyclic dominance and a discontinuous transition with the associated sudden emergence of tight balance between excitatory and inhibitory interactions. The optimal network follows an ideal gas law over an extended temperature range and saturates the efficiency upper bound of energy use. These results provide theoretical insights into the emergence and evolution of complex internal models in predictive processing systems.

Toward Optimal Synthesis of Discrete-Time Hopfield Neural Network.

In this research brief, the relationship between eigenvectors (with {+1, -1} components) of a synaptic weight matrix W and the stable/anti-stable states of discrete-time Hopfield associative memory (HAM) is established. Also, the synthesis of W with desired stable/anti-stable states using spectral representation of W in even/odd dimension is discussed when the threshold vector is a non-zero vector. Freedom in choice of eigenvalues is capitalized to improve the noise immunity of the Hopfield neural network (HNN). Also, the problem of optimal synthesis of Hopfield Associative memory is formulated.

Neuromorphic visual artificial synapse in-memory computing systems based on GeOx-coated MXene nanosheets

Artificial synapses with light signal perception capability offer the ability to neuromorphic visual signal processing system on demand. In light of the excellent optical and electrical characteristics, the low-dimensional materials have become one of the most favorable candidates of the key component for optoelectronic artificial synapses. Previously, our group originally proposed the synthesis of germanium oxide-coated MXene nanosheets. In this work, we further applied this technology into the optoelectronic synaptic thin-film transistors for the first time. The devices exhibited the adjustable postsynaptic current behaviors under the visible light inputs. Moreover, the potentiation and depression operation modes of the devices further improved the application potential of the devices in mimicking biological synapses. Regulated by the wavelength of incident lights, the proposed artificial synapse could effectively help detect the target area of the image. Eventually, we further showed the results of the devices in the projects of neural network computing task. The long-term potentiation/depression characteristics of the conductance were applied to the synaptic weight matrix for image identification and path recognition tasks. By adding knowledge transfer in the process of recognition, the epoch required for convergence has been greatly reduced. The result of high noise tolerance revealed the great potential of the proposed transistors in establishing high-efficiency and robustness hardware neuromorphic systems for in-memory computing.

A multi-class logistic regression algorithm to reliably infer network connectivity from cell membrane potentials

In neuroscience, the structural connectivity matrix of synaptic weights between neurons is one of the critical factors that determine the overall function of a network of neurons. The mechanisms of signal transduction have been intensively studied at different time and spatial scales and both the cellular and molecular levels. While a better understanding and knowledge of some basic processes of information handling by neurons has been achieved, little is known about the organization and function of complex neuronal networks. Experimental methods are now available to simultaneously monitor the electrical activity of a large number of neurons in real time. The analysis of the data related to the activities of individual neurons can become a very valuable tool for the study of the dynamics and architecture of neural networks. In particular, advances in optical imaging techniques allow us to record up to thousands of neurons nowadays. However, most of the efforts have been focused on calcium signals, that lack relevant aspects of cell activity. In recent years, progresses in the field of genetically encoded voltage indicators have shown that imaging signals could be well suited to record spiking and synaptic events from a large population of neurons. Here, we present a methodology to infer the connectivity of a population of neurons from their voltage traces. At first, putative synaptic events were detected. Then, a multi-class logistic regression was used to fit the putative events to the spiking activities and a penalization term was allowed to regulate the sparseness of the inferred network. The proposed Multi-Class Logistic Regression with L1 penalization (MCLRL) was benchmarked against data obtained from in silico network simulations. MCLRL properly inferred the connectivity of all tested networks, as indicated by the Matthew correlation coefficient (MCC). Importantly, MCLRL was accomplished to reconstruct the connectivity among subgroups of neurons sampled from the network. The robustness of MCLRL to noise was also assessed and the performances remained high (MCC>0.95) even in extremely high noise conditions (>95% noisy events). Finally, we devised a procedure to determine the optimal MCLRL regularization term, which allows us to envision its application to experimental data.

Permitted Sets and Convex Coding in Nonthreshold Linear Networks.

Hebbian theory proposes that ensembles of neurons form a basis for neural processing. It is possible to gain insight into the activity patterns of these neural ensembles through a binary analysis, regarding neurons as either active or inactive. The framework of permitted and forbidden sets, introduced by Hahnloser, Seung, and Slotine (2003), is a mathematical model of such a binary analysis: groups of coactive neurons can be permitted or forbidden depending on the network's structure. In order to widen the applicability of the framework of permitted sets, we extend the permitted set analysis from the original threshold-linear regime. Specifically, we generalize permitted sets to firing rate models in which Φ is a nonnegative continuous piecewise C1 activation function. In our framework, the focus is shifted from a neuron's firing rate to its responsiveness to inputs; if a neuron's firing rate is sufficiently sensitive to changes in its input, we say that the neuron is responsive. The algorithm for categorizing a neuron as responsive depends on thresholds that a user can select arbitrarily and that are independent of the dynamics. Given a synaptic weight matrix W, we say that a set of neurons is permitted if it is possible to find a stimulus where those neurons, and no others, remain responsive. The main coding property we establish about PΦ(W), the collection of all permitted sets of the network, is that PΦ(W) is a convex code when W is almost rank one. This means that PΦ(W) in the low-rank regime can be realized as a neural code resulting from the pattern of overlaps of receptive fields that are convex.

Geometric framework to predict structure from function in neural networks.

Neural computation in biological and artificial networks relies on the nonlinear summation of many inputs. The structural connectivity matrix of synaptic weights between neurons is a critical determinant of overall network function, but quantitative links between neural network structure and function are complex and subtle. For example, many networks can give rise to similar functional responses, and the same network can function differently depending on context. Whether certain patterns of synaptic connectivity are required to generate specific network-level computations is largely unknown. Here we introduce a geometric framework for identifying synaptic connections required by steady-state responses in recurrent networks of threshold-linear neurons. Assuming that the number of specified response patterns does not exceed the number of input synapses, we analytically calculate the solution space of all feedforward and recurrent connectivity matrices that can generate the specified responses from the network inputs. A generalization accounting for noise further reveals that the solution space geometry can undergo topological transitions as the allowed error increases, which could provide insight into both neuroscience and machine learning. We ultimately use this geometric characterization to derive certainty conditions guaranteeing a nonzero synapse between neurons. Our theoretical framework could thus be applied to neural activity data to make rigorous anatomical predictions that follow generally from the model architecture.

A Neural Network Model of the Entorhinal Cortex and Hippocampus for Event-Order Memory Processing

To solve a navigation task based on experiences, we need a mechanism to associate places with objects and recall them along the course of action. In a reward-oriented task, if the route to a reward location is simulated in mind after experiencing it once, it might be possible that the reward is gained efficiently. One way to solve this is to incorporate a biologically plausible mechanism. In this study, we propose a neural network that stores a sequence of events associated with a reward. The proposed network recalls the reward location by tracing them in its mind in order. We simulated a virtual mouse that explores a figure-eight maze and recalls the route to the reward location. During the learning period, a sequence of events related to firing along a passage was temporarily stored in the heteroassociative network, and the sequence of events is consolidated in the synaptic weight matrix when a reward is fed. For retrieval, an impetus input internally generates the sequential activation of conjunctive cue–place cells toward the reward location. In the figure-eight maze task, the location of the reward was estimated by mind travel, irrespective of whether the reward is in the counterclockwise or distant clockwise route. The mechanism of efficiently reaching the goal by mind travel in the brain based on experiences is beneficial for mobile service robots that perform autonomous navigation.

Local Homeostatic Regulation of the Spectral Radius of Echo-State Networks.

Recurrent cortical networks provide reservoirs of states that are thought to play a crucial role for sequential information processing in the brain. However, classical reservoir computing requires manual adjustments of global network parameters, particularly of the spectral radius of the recurrent synaptic weight matrix. It is hence not clear if the spectral radius is accessible to biological neural networks. Using random matrix theory, we show that the spectral radius is related to local properties of the neuronal dynamics whenever the overall dynamical state is only weakly correlated. This result allows us to introduce two local homeostatic synaptic scaling mechanisms, termed flow control and variance control, that implicitly drive the spectral radius toward the desired value. For both mechanisms the spectral radius is autonomously adapted while the network receives and processes inputs under working conditions. We demonstrate the effectiveness of the two adaptation mechanisms under different external input protocols. Moreover, we evaluated the network performance after adaptation by training the network to perform a time-delayed XOR operation on binary sequences. As our main result, we found that flow control reliably regulates the spectral radius for different types of input statistics. Precise tuning is however negatively affected when interneural correlations are substantial. Furthermore, we found a consistent task performance over a wide range of input strengths/variances. Variance control did however not yield the desired spectral radii with the same precision, being less consistent across different input strengths. Given the effectiveness and remarkably simple mathematical form of flow control, we conclude that self-consistent local control of the spectral radius via an implicit adaptation scheme is an interesting and biological plausible alternative to conventional methods using set point homeostatic feedback controls of neural firing.

A Pulse-Number-Adjustable MSPCNN and Its Image Enhancement Application

Pulse-coupled neural network (PCNN) aims to control neuronal firing state automatically and complete related image processing tasks. This paper presents a pulse-number-adjustable MSPCNN model (PNA-MSPCNN) that can automatically acquire the firing times and the firing frequency of each neuron. Hereinto, synaptic weight matrix Wijkl and decay factor α will generate an interaction value to determine the final calculation result of the internal activity U. Dynamic threshold amplitude V, step function Q, and auxiliary parameter P can precisely adjust the variation ranges of the dynamic threshold E. Additionally, we propose a low-light image enhancement method based on the above PNA-MSPCNN and a modified low-light image enhancement (LIME). The proposed LIME algorithm focuses mainly on the parameter setting method of weight matrix Wmq, which will bring further improvement of testing image contrast. Experimental results demonstrate that our proposed method achieves better low-light image enhancement performances, compared to prevalent image enhancement methods, including SSIM of 0.8725, AMBE of 0.0550, MSE of 0.0092, and PSNR of 45.7764.

Network response synchronization enhanced by synaptic plasticity

Synchronization of neural network response on spatially localized periodic stimulation was studied. The network consisted of synaptically coupled spiking neurons with spike-timing-dependent synaptic plasticity (STDP). Network connectivity was defined by time evolving matrix of synaptic weights. We found that the steady-state spatial pattern of the weights could be rearranged due to locally applied external periodic stimulation. A method for visualization of synaptic weights as vector field was introduced to monitor the evolving connectivity matrix. We demonstrated that changes in the vector field and associated weight rearrangements underlay an enhancement of synchronization range.

Emergence of Slow-Switching Assemblies in Structured Neuronal Networks.

Unraveling the interplay between connectivity and spatio-temporal dynamics in neuronal networks is a key step to advance our understanding of neuronal information processing. Here we investigate how particular features of network connectivity underpin the propensity of neural networks to generate slow-switching assembly (SSA) dynamics, i.e., sustained epochs of increased firing within assemblies of neurons which transition slowly between different assemblies throughout the network. We show that the emergence of SSA activity is linked to spectral properties of the asymmetric synaptic weight matrix. In particular, the leading eigenvalues that dictate the slow dynamics exhibit a gap with respect to the bulk of the spectrum, and the associated Schur vectors exhibit a measure of block-localization on groups of neurons, thus resulting in coherent dynamical activity on those groups. Through simple rate models, we gain analytical understanding of the origin and importance of the spectral gap, and use these insights to develop new network topologies with alternative connectivity paradigms which also display SSA activity. Specifically, SSA dynamics involving excitatory and inhibitory neurons can be achieved by modifying the connectivity patterns between both types of neurons. We also show that SSA activity can occur at multiple timescales reflecting a hierarchy in the connectivity, and demonstrate the emergence of SSA in small-world like networks. Our work provides a step towards understanding how network structure (uncovered through advancements in neuroanatomy and connectomics) can impact on spatio-temporal neural activity and constrain the resulting dynamics.

Self-organization of synchronous activity propagation in neuronal networks driven by local excitation.

Many experimental and theoretical studies have suggested that the reliable propagation of synchronous neural activity is crucial for neural information processing. The propagation of synchronous firing activity in so-called synfire chains has been studied extensively in feed-forward networks of spiking neurons. However, it remains unclear how such neural activity could emerge in recurrent neuronal networks through synaptic plasticity. In this study, we investigate whether local excitation, i.e., neurons that fire at a higher frequency than the other, spontaneously active neurons in the network, can shape a network to allow for synchronous activity propagation. We use two-dimensional, locally connected and heterogeneous neuronal networks with spike-timing dependent plasticity (STDP). We find that, in our model, local excitation drives profound network changes within seconds. In the emergent network, neural activity propagates synchronously through the network. This activity originates from the site of the local excitation and propagates through the network. The synchronous activity propagation persists, even when the local excitation is removed, since it derives from the synaptic weight matrix. Importantly, once this connectivity is established it remains stable even in the presence of spontaneous activity. Our results suggest that synfire-chain-like activity can emerge in a relatively simple way in realistic neural networks by locally exciting the desired origin of the neuronal sequence.

Fast approximate models of large networks

This work addresses two key challenges in modeling large neural systems such as the primate visual system. The first challenge is that large detailed models require prohibitively expensive computers and long simulation times. The second is that interpretation of simulation results is difficult due to the large number of model parameters. The Neural Engineering Framework (NEF) helps to address the second challenge by defining mappings between neural activity and higher-level behaviour [1,2]. In particular, NEF models behave approximately as systems of nonlinear ordinary differential equations in which the number of states is typically about two orders of magnitude less than the number of neurons. The present work consists of a computationally efficient model of this approximation, i.e. of the functional differences between NEF neural-circuit models and corresponding lower-dimensional dynamics. There are only two such functional differences. One is that the neurons behave as if the right-hand sides of the ODEs are somewhat distorted. High-frequency distortions over the state variables impose attractors on feedback networks. There is also compression at large magnitudes of the state variables. The other difference is that spikes introduce broadband noise-like current fluctuations. The present work approximates these differences in a computationally efficient manner. The procedure is to model a group of point neurons and their connections within a network, to approximate their aggregate behaviour as a surrogate “population model”, and then to simulate only the population model. These surrogate models have two benefits. The first is the practical benefit of a 1-2 orders-of-magnitude reduction in computation. This allows one to run much more complex simulations with the same computer. Efficient parallel implementations are currently in progress, but functionally realistic real-time simulation of 107-108 point neurons is anticipated on a single graphical processing unit (the number depending substantially on network connection statistics). The second benefit is that this model provides a simple and thorough description of the behaviour of complex networks and how it arises from neuron parameters. As an example, we found that fluctuations in post-synaptic current due to synaptic input shift to higher frequencies when the presynaptic spike rates are higher. The main goal of this work has been to accelerate and enhance insight into NEF models, but the method applies to any network of point neurons with linear synaptic integration. If there are about half as many large singular values in a synaptic weight matrix as there are post-synaptic neurons, the surrogate model will be about as efficient as the full neuron model. However, if there are few large singular values then the population model is much more efficient. Future work will address conductance models of synapses, synaptic plasticity, and further scale improvements. Figure 1 Simulations of 400 neurons that encode a one-dimensional value x, which ramps from -1 to 3. A. Weighted sum of spikes (weights chosen to optimally approximate x), filtered by a synapse model. This approximates synaptic drive from these neurons to another ...

A Modified Pulse Coupled Neural Network with Anisotropic Synaptic Weight Matrix for Image Edge Detection

A Modified Pulse Coupled Neural Network with Anisotropic Synaptic Weight Matrix for Image Edge Detection

Optimal architectures for fast-learning, flexible networks

Event Abstract Back to Event Optimal architectures for fast-learning, flexible networks Vladimir Itskov1*, Anda Degeratu2 and Carina Curto1 1 University of Nebraska-Lincoln, United States 2 Max Planck Institute, Germany New memories (patterns) in some brain areas, such as hippocampus, can be encoded quickly. Irrespective of the plasticity mechanism (or learning rule) used to encode patterns via changes in synaptic weights, rapid learning is perhaps most easily accomplished if new patterns can be learned via only small modifications of the initial synaptic weights. It may thus be desirable for fast-learning and flexible neural networks to have architectures which enable large numbers of patterns to be encoded by only small perturbations of the synaptic efficacies. What kinds of network architectures have this property? We define the perturbative capacity of a network to be the number of memory patterns that can be learned under small perturbations of the (effective) synaptic weights. We propose that candidate architectures for fast-learning, flexible networks should be networks with high perturbative capacity. What are optimal architectures that maximize a network’s perturbative capacity? We investigate this question for threshold-linear networks. Here the memory patterns encoded by the recurrent network are groups of neurons that co-fire at a stable fixed point for some external input. We prove that for an arbitrary matrix of effective synaptic weights, the network’s memory patterns correspond to stable submatrices. This enables us to study the perturbative capacity of a network by analyzing the eigenvalues of submatrices under small perturbations of the synaptic weights. In the case of symmetric threshold-linear networks, we find (analytically) the set of optimal network architectures that have maximal perturbative capacity. For these networks, any of the possible memory patterns can be selectively encoded via small perturbations of the synaptic weights. We show that these architectures correspond to a highly restricted set of possible sign patterns governing the effective interactions between principal neurons, and we completely describe these patterns. In particular, we find that at least one half of the effective weights must be inhibitory, and thus the optimal architectures reflect inhibition-stabilized networks with a significant level of inhibition. Finally, we study a larger class of threshold-linear networks (where the matrices are no longer assumed symmetric), and find that our qualitative results continue to hold. The optimal architectures we discover provide a benchmark for comparison to experimentally obtainable estimates of the effective interactions between pyramidal neurons in fast-learning networks. They also give clues as to the differences we might expect between recurrent networks in areas such as the hippocampus, and more rigid networks in areas such as primary sensory cortices, where it may be undesirable for the allowed response patterns to be sensitive to small perturbations in synaptic weights. Conference: Computational and Systems Neuroscience 2010, Salt Lake City, UT, United States, 25 Feb - 2 Mar, 2010. Presentation Type: Poster Presentation Topic: Poster session III Citation: Itskov V, Degeratu A and Curto C (2010). Optimal architectures for fast-learning, flexible networks. Front. Neurosci. Conference Abstract: Computational and Systems Neuroscience 2010. doi: 10.3389/conf.fnins.2010.03.00214 Copyright: The abstracts in this collection have not been subject to any Frontiers peer review or checks, and are not endorsed by Frontiers. They are made available through the Frontiers publishing platform as a service to conference organizers and presenters. The copyright in the individual abstracts is owned by the author of each abstract or his/her employer unless otherwise stated. Each abstract, as well as the collection of abstracts, are published under a Creative Commons CC-BY 4.0 (attribution) licence (https://creativecommons.org/licenses/by/4.0/) and may thus be reproduced, translated, adapted and be the subject of derivative works provided the authors and Frontiers are attributed. For Frontiers’ terms and conditions please see https://www.frontiersin.org/legal/terms-and-conditions. Received: 04 Mar 2010; Published Online: 04 Mar 2010. * Correspondence: Vladimir Itskov, University of Nebraska-Lincoln, Lincoln, United States, vitskov2@math.unl.edu Login Required This action requires you to be registered with Frontiers and logged in. To register or login click here. Abstract Info Abstract The Authors in Frontiers Vladimir Itskov Anda Degeratu Carina Curto Google Vladimir Itskov Anda Degeratu Carina Curto Google Scholar Vladimir Itskov Anda Degeratu Carina Curto PubMed Vladimir Itskov Anda Degeratu Carina Curto Related Article in Frontiers Google Scholar PubMed Abstract Close Back to top Javascript is disabled. Please enable Javascript in your browser settings in order to see all the content on this page.

The timing of external input controls the sign of plasticity at local synapses

The method by which local networks in the brain store information from extrinsic afferent inputs is not well understood. We found that the timing of afferent input can bidirectionally control the sign of spike timing-dependent plasticity at local synapses in rat hippocampus. This mechanism provides a means by which temporal information in external input can be encoded in the local matrix of synaptic weights.

Change-based inference for invariant discrimination

Under a conventional view of information processing in recurrently connected populations of neurons, computations consist in mapping inputs onto terminal attractor states of the dynamical interactions. However, there is evidence that substantial information representation and processing can occur over the course of the initial evolution of the dynamical states of such populations, a possibility that has attractive computational properties. Here, we suggest a model that explores one such property, namely, the invariance to an irrelevant feature dimension that arises from monitoring not the state of the population, but rather (a statistic of) the change in this state over time. We illustrate our proposal in the context of the bisection task, a paradigmatic example of perceptual learning for which an attractor-state recurrent model has previously been suggested. We show a change-based inference scheme that achieves near optimal performance in the task (with invariance to translation), is robust to high levels of dynamical noise and variations of the synaptic weight matrix, and indeed admits a computationally straightforward learning rule.

AN ASSOCIATIVE NETWORK WITH CHAOTIC NEURONS AND DYNAMIC SYNAPSES

Chaotic neurons are interconnected with each other through a matrix of dynamic synaptic weights to constitute an associative network. First, the dynamics of the two chaotic neurons connected with dynamic synapses is analyzed. A quick memory switching phenomenon in the Associative Network with Chaotic Neurons and Dynamic Synapses (ANCNDS) is then demonstrated. It is further discovered that the memory retrieval frequency of the network among stored patterns distributes more uniformly, and is much higher than that of a chaotic network without dynamic synapses. Moreover, the network acts as a peculiar pattern recognition system when applied with external stimulations. If the external stimulation is close to only one stored pattern, the network recalls the stored pattern exactly. On the other hand, if the external stimulation shares similarity with two or several stored patterns, the network transits among these related stored patterns. When the external stimulation does not resemble any stored patterns, the network wanders among all the stored patterns.

Least-Squares Design of FIR Filters Based on a Compacted Feedback Neural Network

The design of finite-impulse response (FIR) filters can be performed by using neural networks by formulating the objective function to a Lyapunov energy function. Focusing on this goal, the authors present an improved structure of a feedback neural network to implement the least-squares design of FIR filters. In addition to using the closed-form expressions for the synaptic weight matrix and the bias parameter of the Hopfield neural network (HNN), the proposed approach can achieve a notable reduction both in the amount of computation required and hardware complexity compared to the previous neural-based method. Simulation results indicate the effectiveness of the proposed approach

A Symmetric Probabilistic Encryption Scheme Based On CHNN Without Data Expansion

In 1999, Guo et al. proposed a new probabilistic symmetric probabilistic encryption scheme based on chaotic attractors of neural networks. The scheme is based on chaotic properties of the Overstoraged Hopfield Neural Network (OHNN). The approach bridges the relationship between neural network and cryptography. However, there are some problems in their scheme: (1) exhaustive search is needed to find all the attractors; (2) the data expansion in the paper is wrongly derived; (3) problem exists on creating the synaptic weight matrix. In this letter, we propose a symmetric probabilistic encryption scheme based on Clipped Hopfield Neural Network (CHNN), which solves the above mentioned problems. Furthermore, it keeps the length of the ciphertext equals to that of the plaintext.