Neuron Transfer Function Research Articles

A Mathematical Model of a Neuron with Synapses based on Physiology

AbstractThe neuron, when considered as a signal processing device, itsinputs are the frequency of pulses received at the synapses, and its output is the frequency of action potentials generated- in essence, a neuron is a pulse frequency signal processing device. In comparison, electrical devices use either digital or analog signals for communication or processing, and the mathematics behind these subjects is well understood. However, in regards to pulse frequency processing devices, there has not yet been a clear and persuasive mathematical model to describe the functions of neurons. It goes without saying that such a model is very important, not only for understanding neuron and neural system behavior, but also for undeveloped potential applications in industry. This paper proposes a method for obtaining the mathematical relationship between the input and output signals of a neuron based on physiological facts. The proposed method focuses on the currents across the postsynaptic membrane of each synapse, and the key is to recognize that the net charge across the whole membrane of a neuron over each action potential cycle must equal to zero. By analyzing the relationship between the input of a synapse and the currents across the postsynaptic membranes, a dynamic pulse frequency model of the neuron can be obtained. Here, we show that the transfer function of a neuron depends on the function of thepostsynaptic current of each synapse in resting state, which can be found by detecting the postsynaptic current when a pulse is received at the synapse. The transfer function of a typical neuron generally includes addition and subtraction of feedthrough terms and/or first order lag functions. To focus on the most basic characteristics of a neuron, accommodation, adaptation, learning, etc. are not discussed in this paper.

Retinal bipolar cells: Temporal filtering of signals from cone photoreceptors

The temporal dynamics of the response of neurons in the outer retina were investigated by intracellular recording from cones, bipolar, and horizontal cells in the intact, light-adapted retina of the tiger salamander (Ambystoma tigrinum), with special emphasis on comparing the two major classes of bipolars cells, the ON depolarizing bipolars (Bd) and the OFF hyperpolarizing bipolars (Bh). Transfer functions were computed from impulse responses evoked by a brief light flash on a steady background of 20 cd/m(2). Phase delays ranged from about 89 ms for cones to 170 ms for Bd cells, yielding delays relative to that of cones of about 49 ms for Bh cells and 81 ms for Bd cells. The difference between Bd and Bh cells, which may be due to a delay introduced by the second messenger G-protein pathway unique to Bd cells, was further quantified by latency measurements and responses to white noise. The amplitude transfer functions of the outer retinal neurons varied with light adaptation in qualitative agreement with results for other vertebrates and human vision. The transfer functions at 20 cd/m(2) were predominantly low pass with 10-fold attenuation at about 13, 14, 9.1, and 7.7 Hz for cones, horizontal, Bh, and Bd cells, respectively. The transfer function from the cone voltage to the bipolar voltage response, as computed from the above measurements, was low pass and approximated by a cascade of three low pass RC filters ("leaky integrators"). These results for cone-->bipolar transmission are surprisingly similar to recent results for rod-->bipolar transmission in salamander slice preparations. These and other findings suggest that the rate of vesicle replenishment rather than the rate of release may be a common factor shaping synaptic signal transmission from rods and cones to bipolar cells.

Spike-frequency adaptation generates intensity invariance in a primary auditory interneuron

Adaptation of the spike-frequency response to constant stimulation, as observed on various timescales in many neurons, reflects high-pass filter properties of a neuron's transfer function. Adaptation in general, however, is not sufficient to make a neuron's response independent of the mean intensity of a sensory stimulus, since low frequency components of the stimulus are still transmitted, although with reduced gain. We here show, based on an analytically tractable model, that the response of a neuron is intensity invariant, if the fully adapted steady-state spike-frequency response to constant stimuli is independent of stimulus intensity. Electrophysiological recordings from the AN1, a primary auditory interneuron of crickets, show that for intensities above 60 dB SPL (sound pressure level) the AN1 adapted with a time-constant of approximately 40 ms to a steady-state firing rate of approximately 100 Hz. Using identical random amplitude-modulation stimuli we verified that the AN1's spike-frequency response is indeed invariant to the stimulus' mean intensity above 60 dB SPL. The transfer function of the AN1 is a band pass, resulting from a high-pass filter (cutoff frequency at 4 Hz) due to adaptation and a low-pass filter (100 Hz) determined by the steady-state spike frequency. Thus, fast spike-frequency adaptation can generate intensity invariance already at the first level of neural processing.

The role of neuron transfer function in artificial neural networks

By analysis of local field distribution of the neurons in stationary state of associative memory neural networks, the role of the analog neuron transfer function in affecting the neural network performance is re-investigated. Different from the research done before, we find that the analog transfer function has no obvious advantages over the hard limit transfer function. Furthermore, analog transfer function sometimes produces a negative impact on certain functions of the network, such as the maximal storage capacity. We show that in pursuing the same performance a proper design rule is more essential than the choice of the transfer function.

Statistics of subthreshold neuronal voltage fluctuations due to conductance-based synaptic shot noise

Neurons in the central nervous system, and in the cortex in particular, are subject to a barrage of pulses from their presynaptic populations. These synaptic pulses are mediated by conductance changes and therefore lead to increases or decreases of the neuronal membrane potential with amplitudes that are dependent on the voltage: synaptic noise is multiplicative. The statistics of the membrane potential are of experimental interest because the measurement of a single subthreshold voltage can be used to probe the activity occurring across the presynaptic population. Though the interpulse interval is not always significantly smaller than the characteristic decay time of the pulses, and so the fluctuations have the nature of shot noise, the majority of results available in the literature have been calculated in the diffusion limit, which is valid for high-rate pulses. Here the effects that multiplicative conductance noise and shot noise have on the voltage fluctuations are examined. It is shown that both these aspects of synaptic drive sculpt high-order features of the subthreshold voltage distribution, such as the skew. It is further shown that the diffusion approximation can only capture the effects arising from the multiplicative conductance noise, predicting a negative voltage skew for excitatory drive. Exact results for the full dynamics are derived from a master-equation approach, predicting positively skewed distributions with long tails in voltage ranges typical for action potential generation. It is argued that, although the skew is a high-order feature of subthreshold voltage distributions, the increased probability of reaching firing threshold suggests a potential role for shot noise in shaping the neuronal transfer function.

Preference of Sensory Neural Coding for1/fSignals

We investigated the influences of different types of temporal correlations in the input signal on the functions and coding properties of neurons in the primary visual cortex (V1). We found that the temporal transfer functions of V1 neurons exhibit higher gain, and the spike responses exhibit higher coding efficiency and information transmission rates, for the 1/f (natural long-term correlation) signals than for 1/f(0) (no correlation) and 1/f(2) (stronger long-term correlation) signals. These results suggest that the intermediate long-term correlation ubiquitous to natural signals may play an important role in shaping and optimizing the machinery of neurons in their adaptation to the natural environment.

NNICE – a neural network aircraft icing algorithm

Although much is known about the meteorological conditions for significant aircraft icing, research studies to date have only been successful identifying conditions for general icing, i.e. clouds in the temperature range from 0 °C to about −20 °C. The aerodynamics of ice accumulation suggest three meteorological factors, cloud liquid water, droplet size, and air temperature. Only the latter is known or forecast with a significant degree of accuracy. The first two are partial functions of the atmosphere's vertical motion which is poorly known, especially in convective situations. However, favorable patterns of relative humidity and potential instability that indicate conditions for possible convection are readily discernable from sounding or numerical forecast model data. Two neural networks were taught to sort out these patterns with respect to icing intensity. Each uses a different neuron transfer function which gives each a “personality”. The “conservative” network diagnoses light icing well but has difficulty with moderate or greater icing. On the other hand, the “radical” network finds the higher intensity icing but is not as good at lower intensities. By combining the strengths of each, NNICE makes skillful icing forecasts of all intensities.

Dynamical mechanisms underlying contrast gain control in single neurons.

Recent neurophysiological experiments have revealed that the linear and nonlinear kernels of the transfer function in sensory neurons are not static. Rather, they are adaptive to the contrast or the variance of time-varying input stimuli, exhibiting a contrast gain control phenomenon. We investigated the underlying biophysical causes of this phenomenon by simulating and analyzing the leaky integrate-and-fire and the Hodgkin-Huxley neuronal models. Our findings indicate that contrast gain control may result from the synergistic cooperation of the nonlinear dynamics of spike generation and the statistical properties of the stimuli. The resulting statistics-dependent stimulus threshold is shown to be a key factor underlying the adaptation of frequency tuning and amplitude gain of a neuron's transfer function in different stimulus environments.

Adaptation of the transfer function of the Hodgkin–Huxley (HH) neuronal model

The transfer functions of sensory neurons are known to adapt to the statistics of the input signals. It is however unclear whether such adaptation arises from the nonlinear dynamics of the neuron or emerges from the collective interaction among neurons embedded in a network. We investigated the Hodgkin–Huxley (HH) neuronal model's response to Gaussian white noise signals of different variances and found that the recovered kernel adapt its preferred temporal frequency and its energy gains according to noise variance. This adaptation is likely a consequence of the cooperative interaction between the noises and the bifurcation dynamics of the neurons.

Population dynamics of interacting spiking neurons.

A dynamical equation is derived for the spike emission rate nu(t) of a homogeneous network of integrate-and-fire (IF) neurons in a mean-field theoretical framework, where the activity of the single cell depends both on the mean afferent current (the "field") and on its fluctuations. Finite-size effects are taken into account, by a stochastic extension of the dynamical equation for the nu; their effect on the collective activity is studied in detail. Conditions for the local stability of the collective activity are shown to be naturally and simply expressed in terms of (the slope of) the single neuron, static, current-to-rate transfer function. In the framework of the local analysis, we studied the spectral properties of the time-dependent collective activity of the finite network in an asynchronous state; finite-size fluctuations act as an ongoing self-stimulation, which probes the spectral structure of the system on a wide frequency range. The power spectrum of nu exhibits modes ranging from very high frequency (depending on spike transmission delays), which are responsible for instability, to oscillations at a few Hz, direct expression of the diffusion process describing the population dynamics. The latter "diffusion" slow modes do not contribute to the stability conditions. Their characteristic times govern the transient response of the network; these reaction times also exhibit a simple dependence on the slope of the neuron transfer function. We speculate on the possible relevance of our results for the change in the characteristic response time of a neural population during the learning process which shapes the synaptic couplings, thereby affecting the slope of the transfer function. There is remarkable agreement of the theoretical predictions with simulations of a network of IF neurons with a constant leakage term for the membrane potential.

Comparison of different forms of the Multi-layer Feed-Forward Neural Network method used for river flow forecasting

Abstract. The Multi-Layer Feed-Forward Neural Network (MLFFNN) is applied in the context of river flow forecast combination, where a number of rainfall-runoff models are used simultaneously to produce an overall combined river flow forecast. The operation of the MLFFNN depends not only on its neuron configuration but also on the choice of neuron transfer function adopted, which is non-linear for the hidden and output layers. These models, each having a different structure to simulate the perceived mechanisms of the runoff process, utilise the information carrying capacity of the model calibration data in different ways. Hence, in a discharge forecast combination procedure, the discharge forecasts of each model provide a source of information different from that of the other models used in the combination. In the present work, the significance of the choice of the transfer function type in the overall performance of the MLFFNN, when used in the river flow forecast combination context, is investigated critically. Five neuron transfer functions are used in this investigation, namely, the logistic function, the bipolar function, the hyperbolic tangent function, the arctan function and the scaled arctan function. The results indicate that the logistic function yields the best model forecast combination performance. Keywords: River flow forecast combination, multi-layer feed-forward neural network, neuron transfer functions, rainfall-runoff models

An alternative approach to infomax and independent component analysis

Infomax means maximization of information flow in a neural system. A nonlinear version of infomax has been shown to be connected to independent component analysis and the receptive fields of neurons in the visual cortex. Here we show a problem of nonrobustness of nonlinear infomax: it is very sensitive to the choice the nonlinear neuronal transfer function. We consider an alternative approach in which the system is linear, but the noise level depends on the mean of the signal, as in a Poisson neuron model. This gives similar predictions as the nonlinear infomax, but seems to be more robust.

Feature selection with neural networks

We present a neural network based approach for identifying salient features for classification in feedforward neural networks. Our approach involves neural network training with an augmented cross-entropy error function. The augmented error function forces the neural network to keep low derivatives of the transfer functions of neurons when learning a classification task. Such an approach reduces output sensitivity to the input changes. Feature selection is based on the reaction of the cross-validation data set classification error due to the removal of the individual features. We demonstrate the usefulness of the proposed approach on one artificial and three real-world classification problems. We compared the approach with five other feature selection methods, each of which banks on a different concept. The algorithm developed outperformed the other methods by achieving higher classification accuracy on all the problems tested.

A two-layer paradigm capable of forming arbitrary decision regions in input space

It is well known that a two-layer perceptron network with threshold neurons is incapable of forming arbitrary decision regions in input space, while a three-layer perceptron has that capability. The effect of replacing the output neuron in a two-layer perceptron with a bithreshold element is studied. The limitations of this modified two-layer perceptron are observed. Results on the separating capabilities of a pair of parallel hyperplanes are obtained. Based on these, a new two-layer neural paradigm based on increasing the dimensionality of the output of the first layer is proposed and is shown to be capable of forming any arbitrary decision region in input space. Then a type of logic called bithreshold logic, based on the bithreshold neuron transfer function, is studied. Results on the limits of switching function realizability using bithreshold gates are obtained.

Resonance of Spike Discharge Modulation in Neurons of the Guinea Pig Medial Vestibular Nucleus

The modulation of action potential discharge rates is an important aspect of neuronal information processing. In these experiments, we have attempted to determine how effectively spike discharge modulation reflects changes in the membrane potential in central vestibular neurons. We have measured how their spike discharge rate was modulated by various current inputs to obtain neuronal transfer functions. Differences in the modulation of spiking rates were observed between neurons with a single, prominent after hyperpolarization (AHP, type A neurons) and cells with more complex AHPs (type B neurons). The spike discharge modulation amplitudes increased with the frequency of the current stimulus, which was quantitatively described by a neuronal model that showed a resonance peak >10 Hz. Modeling of the resonance peak required two putative potassium conductances whose properties had to be markedly dependent on the level of the membrane potential. At low frequencies (< or =0.4 Hz), the gain or magnitude functions of type A and B discharge rates were similar relative to the current input. However, resting input resistances obtained from the ratio of the membrane potential and current were lower in type B compared with type A cells, presumably due to a higher level of active potassium conductances at rest. The lower input resistance of type B neurons was compensated by a twofold greater sensitivity of their firing rate to changes in membrane potential, which suggests that synaptic inputs on their dendritic processes would be more efficacious. This increased sensitivity is also reflected in a greater ability of type B neurons to synchronize with low-amplitude sinusoidal current inputs, and in addition, their responses to steep slope ramp stimulation are enhanced over the more linear behavior of type A neurons. This behavior suggests that the type B MVNn are moderately tuned active filters that promote high-frequency responses and that type A neurons are like low-pass filters that are well suited for the resting tonic activity of the vestibular system. However, the more sensitive and phasic type B neurons contribute to both low- and high-frequency control as well as signal detection and would amplify the contribution of both irregular and regular primary afferents at high frequencies.

Training neural networks by stochastic optimisation

We present a stochastic learning algorithm for neural networks. The algorithm does not make any assumptions about transfer functions of individual neurons and does not depend on a functional form of a performance measure. The algorithm uses a random step of varying size to adapt weights. The average size of the step decreases during learning. The large steps enable the algorithm to jump over local maxima/minima, while the small ones ensure convergence in a local area. We investigate convergence properties of the proposed algorithm as well as test the algorithm on four supervised and unsupervised learning problems. We have found a superiority of this algorithm compared to several known algorithms when testing them on generated as well as real data.

A Note on the Equivalence of NARX and RNN

This paper presents several aspects with regards the application of the NARX model and Recurrent Neural Network (RNN) model in system identification and control. We show that every RNN can be transformed to a first order NARX model, and vice versa, under the condition that the neuron transfer function is similar to the NARX transfer function. If the neuron transfer function is piecewise linear, that is f(x):=x if uxu , 1 and f(x):=sign(x) otherwise, we further show that every NARX model of order larger than one can be transformed into a RNN. According to these equivalence results, there are three advantages from which we can benefit: (i) if the output dimension of a NARX model is larger than the number of its hidden unit, training an equivalent RNN will be faster, i.e. the equivalent RNN is trained instead of the NARX model. Once the training is finished, the RNN is transformed back to an equivalent NARX model. On the other hand, (ii) if the output dimension of a RNN model is less than the number of its hidden units, the training of a RNN can be speeded up by using a similar method; (iii) the RNN pruning can be accomplished in a much simpler way, i.e. the equivalent NARX model is pruned instead of the RNN. After pruning, the NARX model is transformed back to the equivalent RNN.

Computing with the leaky integrate-and-fire neuron: logarithmic computation and multiplication.

The leaky integrate-and-fire (LIF) model of neuronal spiking (Stein 1967) provides an analytically tractable formalism of neuronal firing rate in terms of a neuron's membrane time constant, threshold, and refractory period. LIF neurons have mainly been used to model physiologically realistic spike trains, but little application of the LIF model appears to have been made in explicitly computational contexts. In this article, we show that the transfer function of a LIF neuron provides, over a wide-parameter range, a compressive nonlinearity sufficiently close to that of the logarithm so that LIF neurons can be used to multiply neural signals by mere addition of their outputs yielding the logarithm of the product. A simulation of the LIF multiplier shows that under a wide choice of parameters, a LIF neuron can log-multiply its inputs to within a 5% relative error.

Multiphasic neuronal transfer functions for representing temporal structure

Neural network models of timing have struggled to account for animal timing capabilities using the accepted connectionist assumptions, in most cases without postulating the existence of explicit neuronal time-keeping mechanisms. Current ethological and physiological data, however, suggest that cellular oscillators form the foundation for animals’ temporal capacities. We propose that these oscillators could be used as temporal filters that capture the temporal structure of the animal’s experience. A model is presented that accounts for a number of the salient features of the feeding anticipatory response with only a single circadian filter. This model goes beyond current entrainment models in that it correctly predicts the relationship between the feeding period and the anticipatory interval. An alternative approach using multiple filters is examined that can account for animals’ ability to correctly anticipate two daily feeding times.

Practical optoelectronic neural network realizations based on the fault tolerance of the backpropagation algorithm

This paper describes how the fault tolerance of the backpropagation algorithm can be used to accommodate the realistic (nonideal) transfer characteristics of the optical communication links used, between neural layers, in optoelectronic neural networks. In particular the authors demonstrate that networks, utilizing MSM (metal-semiconductor-metal) photodiodes (PDs) and either LED (light emitting diode) or MQW (multiple quantum well) laser transmitters within these intraneural links, are capable of performing satisfactorily even in the presence of such nonideal device phenomena as: 60% optical crosstalk, 50% optoelectronic device variation, or a thresholded (I(th)=0.5*I(max)) laser output characteristic. Subsequent to this, the authors then show how it is possible to use this fault tolerance to simplify the neuron architecture, to the extent that it consists only of MSM PDs a current amplifier, and an MQW laser. The overall neuron transfer function is then a first-order approximation to the original sigmoidal function.