Synaptic Efficacies Research Articles

Effects of refractory periods in the dynamics of a diluted neural network.

We propose a stochastic dynamics for a neural network which accounts for the effects of the refractory periods (absolute and relative) in the dynamics of a single neuron. The dynamics can be solved analytically in an extremely diluted network. We found a very rich scenario that presents retrieval phases and a period doubling route to chaos in the attractors of the overlap order parameter. Our model incorporates some characteristics that make it biologically appealing, such as asymmetric synaptic efficacies, dilution of the synaptic matrix, absolute and relative refractory periods, complex retrieval dynamics, and low levels of activity in the retrieval regime. \textcopyright{} 1996 The American Physical Society.

Redistribution of synaptic efficacy: A mechanism to generate infinite synaptic input diversity from a homogenous population of neurons without changing absolute synaptic efficacies

Changing the reliability of neurotransmitter release results in a change in the efficay of low frequency synaptic transmission and in the rate of high frequency synaptic depression thus it can not cause an uniform change in strength of synapses and instead results in a change in the dynamics of synaptic transmission referred to as ‘redistribution of synaptic efficacy’ (RSE). Since the change in synaptic transmission associated with RSE depends on the history of action potential activity it is concluded that RSE serves as a mechanism to generate a potentially infinite diversity of synaptic input.

An intracellular study of space and time representation in primary visual cortical receptive fields

In contrast with previous knowledge based on extracellular recordings, the recent development of intracellular techniques in vivo (sharp electrode or ‘blind patch’) ideally allows experimenters to analyze and dissect the contribution of feedforward and lateral connectivity in the functional expression of a synaptic ‘integration field’. We will present recent data which demonstrate that the visual receptive field of cortical neurons described at the level of subthreshold synaptic events extends over much larger regions of the visual field than previously thought, and that the capacity of cells to amplify subthreshold responses depends on the immediate past history of their membrane potential. Our data suggest that visual cortical receptive fields should not be considered as a fixed entity but more as a dynamic field of integration and association. Two types of dynamics can be argued for: 1) the spatial structure of the minimal discharge field (defined by suprathreshold activation of the cell) can be profoundly reorganized at least during development and most probably during selective phases of learning under the control of activity-dependent mechanisms. Adaptive changes in visual responses are thought to reflect long-lasting potentiation and/or depression of synaptic efficacies conveying ON- and OFF-center information; and 2) during sensory processing, reconfiguration of synaptic weights may be achieved on a much faster time-scale and linke to non-linear properties of the postsynaptic membrane as well as that of recruited networks. Association of information available in the central part of the receptive field (RF) and of input coming from the reputedly ‘unresponsive’ regions surrounding it, or arising simultaneously from different parts of the visual field, might be suppressive in certain cases and capable of boosting hidden responses in other cases, depending on the global stimulus configuration.

Electronic implementation of an analogue attractor neural network with stochastic learning

We describe and discuss an electronic implementation of an attractor neural network with plastic synapses. The network undergoes double dynamics, for the neurons as well as the synapses. Both dynamical processes are unsupervised. The synaptic dynamics is autonomous, in that it is driven exclusively and perpetually by neural activities. The latter follow the network activity via the developing synapses and the influence of external stimuli. Such a network self-organizes and is a device which converts the gross statistical characteristics of the stimulus input stream into a set of attractors (reverberations). To maintain for long time the acquired memory, the analog synaptic efficacies are discretized by a stochastic refresh mechanism. The discretized synaptic memory has indefinitely long lifetime in the absence of activity in the network. It is modified only by the arrival of new stimuli. The stochastic refresh mechanism produces transitions at low probability which ensures that transient stimuli do not create significant modifications and that the system has large palimpsestic memory. A change in the attractor structure represents a major, macroscopic change in the statistics of the input stream, which may deform attractors, may create new ones and may eliminate others.The electronic implementation is completely analogue, stochastic and asynchronous. The circuitry of the first prototype is discussed in detail as well as the tests performed on it. In carrying out the implementation we have been guided by biological considerations and by electronic constraints. Both are discussed and new insights and lessons for the learning process are proposed.

Population trends in the fine spatial re-organization of synaptic elements in forebrain regions of chicks 0.5 and 24 hours after passive avoidance training.

Two regions in the forebrain of domestic chicks (Gallus domesticus), the intermediate and medial hyperstriatum ventrale and the lobus parolfactorius, have previously been shown to be important centres of biochemical, pharmacological and physiological change following one-trial passive avoidance training. The purpose of the present study was to examine, at the electron microscopic level, the fine spatial re-arrangement of synaptic structures in the intermediate and medial hyperstriatum ventrale (at 30 min), and in the lobus parolfactorius (at 24 h), post-training using comprehensive biometrical designs, image analysis and stochastic approaches. In intermediate and medial hyperstriatum ventrale, no significant differences in the numerical density of synapses either between control and trained chicks, or between hemispheres, were revealed using the disector method. However, after training, a nested-ANOVA demonstrated an increase in the thickness of pre- and post-synaptic electron densities (estimated via image analysis) only in the left intermediate and medial hyperstriatum ventrale, whereas synaptic apposition zone profiles increased in length bilaterally. In presynaptic terminals from the intermediate and medial hyperstriatum ventrale, stochastic analysis revealed that training resulted in the re-distribution of synaptic vesicles between two spatial pools relative to synaptic apposition zones, in both hemispheres producing a large number of synaptic vesicles closer to synaptic apposition zones; a nearest neighbour analysis of synaptic apposition zone profiles indicated that the lateral shape of the synaptic apposition zone after training is more complex in both hemispheres. In the lobus parolfactorius at 24 h post-training the main changes in synaptic fine structure involved a shift of synaptic vesicles away from synaptic apposition zones in the right hemisphere with the distance between synaptic apposition zones decreasing; in the left lobus parolfactorius, synaptic apposition zones became more regular/round in shape with a greater distance between them after training. These data suggest that the initial acquisition of memory involves population changes in the fine spatial organization of synaptic vesicles and synaptic apposition zones in synapses in the intermediate and medial hyperstriatum ventrale, which indicate a possible tendency towards greater synaptic efficacies. These changes are as dynamics as the molecular changes which have hitherto been considered the preserve of short-term correlates of memory formation.

Learning internal representations in an attractor neural network with analogue neurons

A leaming amtor neural network (U) with a double dynamics of neual activities and synaptic efficacies, operating on two different timescales is studied by simulations in preparation for an electronic implementation. The present network includes several qnasi- realistic features: neurons me represented by their afferent currents and output spike rates; excitatory and inhibitory neurons are separated; amactor spike rates as well as coding levels in arriving stimuli are low; learning lakes place only between excitatory units. Synaptic dynamics is an unsupervised, analogue Hebbian process, but long term memory in the absence of neural activity is maintained by a refresh mechanism which on long timescales discretizes the synaptic values. converting leaning into asynchronous stochastic process induced by the stimuli on the synaptic efficacies. This network is intended to lean a set of amactors from the statistics of freely arriving stimuli, which are represented by extemal synaptic inputs injected into the excitatory neurons. In the simulations different types of sequences of many thousands of stimuli are presented to Ihe network, without distinguishing in the dynamics a leaming phase from retrieval. Stimulus sequences differ in pre-assigned global statistics (including time-dependent statistics); in orders of presentation of individual stimuli within a given statistics: in lengths of time intervals for each presentation and in the intervals separating one stimulus from another. We find that the network effectively learns a set of attractors representing the statistics of the stimuli. and is able to modify its amactors when the input staristics change. Moreover, as the global input statistics changes the network can also forget attracton related to stimulus classes no longer presented. Forgetting takes place only due to the @val of new stimuli. The performance of the network and the statistics of he amactors are studied as a function of the inmt statistics. Mast of the --scale characteristics of the leamina dynamics can be captured . -. theoretically. This model modifies a previous implementation of a LA composed of discrete neurons. in a nerwork of more redistic neurons. The different elements have been designed to facilitate their implementation in silicon.

Maximization of mutual information in a linear noisy network: a detailed study

We consider a linear, one-layer feedforward neural network performing a coding task. The goal of the network is to provide a statistical neural representation that conveys as much information as possible on the input stimuli in noisy conditions. We determine the family of synaptic couplings that maximizes the mutual information between input and output distribution. Optimization is performed under different constraints on the synaptic efficacies. We analyse the dependence of the solutions on input and output noises. This work goes beyond previous studies of the same problem in that: (i) we perform a detailed stability analysis in order to find the global maxima of the mutual information; (ii) we examine the properties of the optimal synaptic configurations under different constraints; (iii) and we do not assume translational invariance of the input data, as it is usually done when inputs are assumed to be visual stimuli.

Learning in neural networks with partially structured synaptic transitions

We show that stochastic learning of attractors can take place in a situation in which either only potentiation or only depression of synaptic efficacies is caused in a structured Hebbian way. In each case, the transition in the opposite sense takes place at random, but occurs only upon presentation of a stimulus. The outcome is an associative memory with the palimpsest property. It is shown that structured potentiation produces more effective learning than structured depression, i.e. it creates a network with a much higher number of retrievable memories.

Electronic implementation of an analogue attractor neural network with stochastic learning

We describe and discuss an electronic implementation of an attractor neural network with plastic synapses. The network undergoes double dynamics, for the neurons as well as the synapses. Both dynamical processes are unsupervised. The synaptic dynamics is autonomous, in that it is driven exclusively and perpetually by neural activities. The latter follow the network activity via the developing synapses and the influence of external stimuli. Such a network self-organizes and is a device which converts the gross statistical characteristics of the stimulus input stream into a set of attractors (reverberations). To maintain for long time the acquired memory, the analog synaptic efficacies are discretized by a stochastic refresh mechanism. The discretized synaptic memory has indefinitely long lifetime in the absence of activity in the network. It is modified only by the arrival of new stimuli. The stochastic refresh mechanism produces transitions at low probability which ensures that transient stimuli do not cr...

Nonlinear neurons in the low-noise limit: a factorial code maximizes information transfer

We investigate the consequences of maximizing information transfer in a simple neural network (one input layer, one output layer), focusing on the case of nonlinear transfer functions. We assume that both receptive fields (synaptic efficacies) and transfer functions can be adapted to the environment. The main result is that, for bounded and invertible transfer functions, in the case of a vanishing additive output noise, and no input noise, maximization of information (Linsker's infomax principle) leads to a factorial code-hence to the same solution as required by the redundancy-reduction principle of Barlow. We also show that this result is valid for linear and, more generally, unbounded, transfer functions, provided optimization is performed under an additive constraint, i.e. which can be written as a sum of terms, each one being specific to one output neuron. Finally, we study the effect of a non-zero input noise. We find that, to first order in the input noise, assumed to be small in comparison with the (small) output noise, the above results are still valid, provided the output noise is uncorrelated from one neuron to the other.

Theory of correlations in stochastic neural networks

One of the main experimental tools in probing the interactions between neurons has been the measurement of the correlations in their activity. In general, however, the interpretation of the observed correlations is difficult since the correlation between a pair of neurons is influenced not only by the direct interaction between them but also by the dynamic state of the entire network to which they belong. Thus a comparison between the observed correlations and the predictions from specific model networks is needed. In this paper we develop a theory of neuronal correlation functions in large networks comprising several highly connected subpopulations and obeying stochastic dynamic rules. When the networks are in asynchronous states, the cross correlations are relatively weak, i.e., their amplitude relative to that of the autocorrelations is of order of 1/N, N being the size of the interacting populations. Using the weakness of the cross correlations, general equations that express the matrix of cross correlations in terms of the mean neuronal activities and the effective interaction matrix are presented. The effective interactions are the synaptic efficacies multiplied by the gain of the postsynaptic neurons. The time-delayed cross-correlation matrix can be expressed as a sum of exponentially decaying modes that correspond to the (nonorthogonal) eigenvectors of the effective interaction matrix. The theory is extended to networks with random connectivity, such as randomly dilute networks. This allows for a comparison between the contribution from the internal common input and that from the direct interactions to the correlations of monosynaptically coupled pairs.A closely related quantity is the linear response of the neurons to external time-dependent perturbations. We derive the form of the dynamic linear response function of neurons in the above architecture in terms of the eigenmodes of the effective interaction matrix. The behavior of the correlations and the linear response when the system is near a bifurcation point is analyzed. Near a saddle-node bifurcation, the correlation matrix is dominated by a single slowly decaying critical mode. Near a Hopf bifurcation the correlations exhibit weakly damped sinusoidal oscillations. The general theory is applied to the case of a randomly dilute network consisting of excitatory and inhibitory subpopulations, using parameters that mimic the local circuit of 1 ${\mathrm{mm}}^{3}$ of the rat neocortex. Both the effect of dilution as well as the influence of a nearby bifurcation to an oscillatory state are demonstrated.

Detailed passive cable models of whole-cell recorded CA3 pyramidal neurons in rat hippocampal slices

Tight-seal whole-cell recordings were made from cleaned somata of CA3 pyramidal cells deep in hippocampal slices from 19-21-d-old rats. The cells were filled with biocytin, and their voltage responses to short current pulses were recorded. After washout of initial sag, responses scaled linearly with injected current and were stable over time. The dendritic and axonal arbors of four cells were reconstructed and measured using light microscopy. Dendritic spines and axonal boutons were counted and the additional membrane area was incorporated into the relevant segments. The morphology of each neuron was converted into a detailed branching cable model by assuming values for specific membrane capacitance Cm and resistance Rm, and cytoplasmic resistivity Ri. These parameters were optimized for each cell by directly matching the model's response to that of the real cell by means of a modified weighted least-squares fitting procedure. By comparing the deviations between model and experimental responses to control noise recordings, approximate 95% confidence intervals were established for each parameter. If a somatic shunt was allowed, a wide range of possible Rm values produced acceptable fits. With zero shunt, Cm was 0.7-0.8 microFcm-2, Ri was 170-340 omega cm, and Rm ranged between 120 and 200 k omega cm2. The electrotonic lengths of the basal and oblique dendrites were 0.2-0.3 space constants, and those of the apical tufts were 0.4-0.7 space constants. The steady-state electrical geometry of these cells was therefore compact; average dendritic tip/soma relative synaptic efficacies were > 93% for the basal and oblique dendrites, and > 81% for the tufts. With fast transient synaptic inputs, however, the models produced a wide range of postsynaptic potential shapes and marked filtering of voltage-clamp currents.

Nonlinear neurons in the low-noise limit: a factorial code maximizes information transfer

We investigate the consequences of maximizing information transfer in a simple neural network (one input layer, one output layer), focusing on the case of nonlinear transfer functions. We assume that both receptive fields (synaptic efficacies) and transfer functions can be adapted to the environment. The main result is that, for bounded and invertible transfer functions, in the case of a vanishing additive output noise, and no input noise, maximization of information (Linsker's infomax principle) leads to a factorial code-hence to the same solution as required by the redundancy-reduction principle of Barlow. We also show that this result is valid for linear and, more generally, unbounded, transfer functions, provided optimization is performed under an additive constraint, i.e. which can be written as a sum of terms, each one being specific to one output neuron. Finally, we study the effect of a non-zero input noise. We find that, to first order in the input noise, assumed to be small in comparison with th...

Significance of distributed representation in the output layer of a neural network in a pattern recognition task

In the cerebral cortex, it is assumed that information is represented by the activity pattern of an assembly of neurons and the synaptic efficacies among them. A distributed representation of pattern is incorporated in the output layer of a neural network with an error back-propagation algorithm, in order to study its technological merits. The network has three layers, which consist of a 32 x 32 array of units (1024) for the input layer, 6-25 units for the hidden layer and 12 units for the output layer. 12 triangular patterns with a variety of parameters are presented to the input layer. Three output-layer units are assigned to each input figure. After initial learning, the network responds to the learned pattern with high accuracy. In addition, it responds with high accuracy to similar but unpresented patterns, showing a generalisation for patterns. The network shows resistance to unit de-activation procedures. When the input layer is exposed to the learned pattern, the hidden-layer units show associative activation pattern. These results indicate that the organisation of information representation in the output layer in a neural network strongly influences both the performance of the whole network and information representation in the hidden layer.

Multifractality in forgetful memories

Learning rules of a forgetful memory generate their synaptic efficacies through iterative procedures that operate on the input data, random patterns. We analyse invariant distributions of the synaptic couplings as they arise asymptotically and show that they exhibit fractal or multifractal properties. We also discuss their dependence upon the learning rule and the parameters specifying it, and indicate how the nature of the invariant distribution is related to the network performance.

Adequate input for learning in attractor neural networks

In the context of learning in attractor neural networks (ANN) the authors discuss the issue of the constraints imposed by there requirements that the afferents arriving at the neurons in the attractor network from the stimulus, compete successfully with the afferents generated by the recurrent activity inside the network, in a situation in which both sets of synaptic efficacies are weak and approximately equal.We simulate and analyse a two-component network: one representing the stimulus, the other an ANN. They show that if stimuli art correlated with the receptive fields of neurons in the ANN, and are of sufficient contrast, the stimulus can provide the necessary information to the recurrent network to allow learning new stimulus, even in the very disfavoured situation of synaptic predominance in the recurrent part. Stimuli which are insufficiently correlated with the receptive fields, or are of insufficient contrast, are submerged by the recurrent activity.

Learning algorithms for perceptrons from statistical physics

Learning algorithms for perceptrons are deduced from statistical mechanics. Thermodynamical quantities are used as cost functions which may be extremal12ed by gradient dynamics to find the synaptic efficacies that store the learning set of patterns. The learning rules so obtained are classified in two categories, following the statistics used to derive the cost functions, namely, Bolt2mann statistics, and Fermi statistics. In the limits of zero or infinite temperatures some of the rules behave like already known algorithms, but new strategies for learning are obtained at finite temperatures, which minim12e the number of errors on the training set.

Constraints on learning in dynamic synapses

Hebbian-type learning is discussed in a network whose synapses are analogue, dynamic variables, whose values have to be periodically refreshed due to possible exponential decay, or other instability of continuous synaptic efficacies. It is shown that the end product of learning in such networks is very sensitive to the relation between the rate of presentation of patterns and the size of the refresh time interval. It is shown that in the limit of slow presentation, the network can learn at most O(In N) patterns in N neurons, and must learn each one in one shot, thus learning all errors present in a corrupt stimulus presented for retrieval.It is then shown that as the rate of presentation is increased the performance is increased rapidly. Another option we investigate is that in which the refresh mechanism is acting stochastically. In this case the rate of learning can be slowed down very significantly, but the number of stored patterns cannot surpass √N.

A sequential pseudo-inverse learning rule for networks of formal neurons

For networks of formal two-state neurons the sequential learning problem is considered: a set of patterns has been memorized by adjusting a matrix of the synaptic efficacies. Then an extra pattern is presented to the network and should be stored in addition to previous patterns in such a way that the latter are not used. This problem has been solved in the framework of the pseudo-inverse learning rules. The resulting synaptic matrix has the same value as for other variants of the pseudo-inverse learning rule if memorizing of the previous patterns was performed by some pseudo-inverse rule.

Associative memory in a network of ‘spiking’ neurons

The Hopfield network provides a simple model of an associative memory in a neuronal structure. It is, however, based on highly artificial assumptions, especially the use of formal two-state neurons or graded-response neurons. The authors address the question of what happens if formal neurons are replaced by a model of ‘spiking’ neurons. They do so in two steps. First, they show how to include refractoriness and noise into a simple threshold model of neuronal spiking. The spike trains resulting from such a model reproduce the distribution of interspike intervals and gain functions found in real neurons. In a second step they connect the model neurons so as to form a large associative memory system. The spike transmission is described by a synaptic kernel which includes axonal delays, ‘Hebbian’ synaptic efficacies, and a realistic postsynaptic response. The collective behaviour of the system is predicted by a set of dynamical equations which are exact in the limit of a large and fully connected network that...