General Activation Functions Research Articles

Exponential stability of discrete-time Hopfield neural networks

In this paper, some sufficient conditions for the local and global exponential stability of the discrete-time Hopfield neural networks with general activation functions are derived, which generalize those existing results. By means of M-matrix theory and some inequality analysis techniques, the exponential convergence rate of the neural networks to the equilibrium is estimated, and for the local exponential stability, the basin of attraction of the stable equilibrium is also characterized.

A fully adaptive normalized nonlinear gradient descent algorithm for complex-valued nonlinear adaptive filters

A fully adaptive normalized nonlinear complex-valued gradient descent (FANNCGD) learning algorithm for training nonlinear (neural) adaptive finite impulse response (FIR) filters is derived. First, a normalized nonlinear complex-valued gradient descent (NNCGD) algorithm is introduced. For rigour, the remainder of the Taylor series expansion of the instantaneous output error in the derivation of NNCGD is made adaptive at every discrete time instant using a gradient-based approach. This results in the fully adaptive normalized nonlinear complex-valued gradient descent learning algorithm that is suitable for nonlinear complex adaptive filtering with a general holomorphic activation function and is robust to the initial conditions. Convergence analysis of the proposed algorithm is provided both analytically and experimentally. Experimental results on the prediction of colored and nonlinear inputs show the FANNCGD outperforming other algorithms of this kind.

Mechanisms of Electrical Stimulation with Neural Prostheses

Individual electric and geometric characteristics of neural substructures can have surprising effects on artificially controlled neural signaling. A rule of thumb approved for the stimulation of long peripheral axons may not hold when the central nervous system is involved. This is demonstrated here with a comparison of results from the electrically stimulated cochlea, retina, and spinal cord. A generalized form of the activating function together with accurate modeling of the neural membrane dynamics are the tools to analyze the excitation mechanisms initiated by neural prostheses. Analysis is sometimes possible with a linear theory, in other cases, simulation of internal calcium concentration or ion channel current fluctuations is needed to see irregularities in spike trains. Spike initiation site can easily change within a single target neuron under constant stimulation conditions of a cochlear implant. Poor myelinization in the soma region of the human cochlear neurons causes firing characteristics different from any animal data. Retinal ganglion cells also generate propagating spikes within the dendritic tree. Bipolar cells in the retina are expected to respond with neurotransmitter release before a spike is generated in the ganglion cell, even when they are far away from the electrode. Epidural stimulation of the lumbar spinal cord predominantly stimulates large sensory axons in the dorsal roots which induce muscle reflex responses. Analysis with the generalized activating function, computer simulations of the nonlinear neural membrane behavior together with experimental and clinical data analysis enlighten our understanding of artificial firing patterns influenced by neural prostheses.

Data-Reusing Recurrent Neural Adaptive Filters

A class of data-reusing learning algorithms for real-time recurrent neural networks (RNNs) is analyzed. The analysis is undertaken for a general sigmoid nonlinear activation function of a neuron for the real time recurrent learning training algorithm. Error bounds and convergence conditions for such data-reusing algorithms are provided for both contractive and expansive activation functions. The analysis is undertaken for various configurations that are generalizations of a linear structure infinite impulse response adaptive filter.

A model of the electrically excited human cochlear neuron: I. Contribution of neural substructures to the generation and propagation of spikes

Differences in neural geometry and the fact that the soma of the human cochlear neuron typically is not myelinated are reasons for disagreements between single fiber recordings in animals and the neural code evoked in cochlear implant patients. We introduce a compartment model of the human cochlear neuron to study the excitation and propagation process of action potentials. The model can be used to predict (i) the points of spike generation, (ii) the time difference between stimulation and the arrival of a spike at the proximal end of the central axon, (iii) the vanishing of peripherally evoked spikes at the soma region under specific conditions, (iv) the influence of electrode positions on spiking behavior, and (v) consequences of the loss of the peripheral axon. Every subunit of the cochlear neuron is separately modeled. Ion channel dynamics are described by a modified Hodgkin–Huxley model. Influence of membrane noise is taken into account. Additionally, the generalized activating function is introduced as a tool to give an envision of the origin of spikes in the peripheral and in the central axon without any knowledge of the gating processes in the active membranes. Comparing the reactions of a human and cat cochlear neuron, we find differences in spiking behavior, e.g. peripherally and centrally evoked spikes arrive with a time difference of about 400 μs in man and 200 μs in cat.

Exploiting inherent relationships in RNN architectures

We provide the relationship between the learning rate and the slope of a nonlinear activation function of a neuron within the framework of nonlinear modular cascaded systems realised through Recurrent Neural Network (RNN) architectures. This leads to reduction in the computational complexity of learning algorithms which continuously adapt the weights of such architectures, because there is a smaller number of independent parameters to optimise. Results are provided for the Gradient Descent (GD) learning algorithm and the Extended Recursive Least Squares (ERLS) algorithm, using a general nonlinear activation function of a neuron. The results obtained degenerate into the corresponding results for single RNNs, when considering only one module in such cascaded systems.

The basic mechanism for the electrical stimulation of the nervous system

Neural signals can be generated or blocked by extracellular electrodes or magnetic coils. New results about artificial excitation are based on a compartmental model of a target neuron and its equivalent electrical network, as well as on the theory of the generalized activating function. The analysis shows that: (i) in most cases, the origin of artificial excitation is within the axon and the soma is much more difficult to excite; (ii) within the central nervous system, positive and negative threshold currents essentially depend on the position and orientation of the neurons relative to the applied electric field; (iii) in several cases, stimulation with positive currents is easier; and (iv) it should be possible to excite synaptic activity without the generation of propagating action potentials. Furthermore, the theory of the generalized activating function gives hints to understanding the blockage of neural activity.

A posteriori error learning in nonlinear adaptive filters

The authors provide relationships between the a priori and a posteriori errors of adaptation algorithms for real-time output-error nonlinear adaptive filters realised as feedforward or recurrent neural networks. The analysis is undertaken for a general nonlinear activation function of a neuron, and for gradient-based learning algorithms, for both a feedforward (FF) and recurrent neural network (RNN). Moreover, the analysis considers both contractive and expansive forms of the nonlinear activation functions within the networks. The relationships so obtained provide the upper and lower error bounds for general gradient based a posteriori learning in neural networks.

A generalized activating function for predicting virtual electrodes in cardiac tissue

To fully understand the mechanisms of defibrillation, it is critical to know how a given electrical stimulus causes membrane polarizations in cardiac tissue. We have extended the concept of the activating function, originally used to describe neuronal stimulation, to derive a new expression that identifies the sources that drive changes in transmembrane potential. Source terms, or virtual electrodes, consist of either second derivatives of extracellular potential weighted by intracellular conductivity or extracellular potential gradients weighted by derivatives of intracellular conductivity. The full response of passive tissue can be considered, in simple cases, to be a convolution of this "generalized activating function" with the impulse response of the tissue. Computer simulations of a two-dimensional sheet of passive myocardium under steady-state conditions demonstrate that this source term is useful for estimating the effects of applied electrical stimuli. The generalized activating function predicts oppositely polarized regions of tissue when unequally anisotropic tissue is point stimulated and a monopolar response when a point stimulus is applied to isotropic tissue. In the bulk of the myocardium, this new expression is helpful for understanding mechanisms by which virtual electrodes can be produced, such as the hypothetical "sawtooth" pattern of polarization, as well as polarization owing to regions of depressed conductivity, missing cells or clefts, changes in fiber diameter, or fiber curvature. In comparing solutions obtained with an assumed extracellular potential distribution to those with fully coupled intra- and extracellular domains, we find that the former provides a reliable estimate of the total solution. Thus the generalized activating function that we have derived provides a useful way of understanding virtual electrode effects in cardiac tissue.

A generalized neuronal activation function derived from ion-channel characteristics

An abstract model neuron, simple enough to be used in analytical studies of neural networks, is defined. It incorporates neuronal adaptation, i.e. the coupling between neuronal activity and excitability, which can control the complexity of neural network dynamics. A generalized neuronal activation function is defined as the output activity, i.e. The firing rate of action potentials, as a function of an input and an excitability variable. For a biological neuron, the intracellular concentration of Ca ions is shown to be a useful excitability variable. The principal features of the generalized activation function are derived by means of a multicompartment model neuron with ionic currents described by Hodgkin-Huxley-type equations. As a relevant neuron type for investigations of cortical associative memory, a neocortical pyramidal cell of the regularly-spiking type is selected. An approximate analytical consideration suggests a simple form of independent variable of the generalized activation function. The complete numerical treatment verifies this form and displays a threshold character of the activation function. The derivation thereby pmvides an interpretation of dynamic threshold models of neuronal adaptation in terms of ionic mechanism. The threshold with respect to the input has well-defined dynamics directly given by the intracellular Ca dynamics. A comparison of the response of the Hodgkin-Huxley-type and reduced-model neumns to a varying input variable indicates that the reduced model is a usefui approximation even for the detailed behaviour of the firing rate. The generalized activation function also accounts for the dependence on neuromodnlation via an adaptability parameter. An absmt model neural network of units thus defined has the capacity to describe complex network dynamics and the simplicity to allow elucidation of mechanisms and applications to large-scale systems.

A generalized neuronal activation function derived from ion-channel characteristics

An abstract model neuron, simple enough to be used in analytical studies of neural networks, is defined. It incorporates neuronal adaptation, i.e. the coupling between neuronal activity and excitability, which can control the complexity of neural network dynamics. A generalized neuronal activation function is defined as the output activity, i.e. The firing rate of action potentials, as a function of an input and an excitability variable. For a biological neuron, the intracellular concentration of Ca ions is shown to be a useful excitability variable. The principal features of the generalized activation function are derived by means of a multicompartment model neuron with ionic currents described by Hodgkin-Huxley-type equations. As a relevant neuron type for investigations of cortical associative memory, a neocortical pyramidal cell of the regularly-spiking type is selected. An approximate analytical consideration suggests a simple form of independent variable of the generalized activation function. The c...

Functionally Equivalent Feedforward Neural Networks

For a feedforward perceptron type architecture with a single hidden layer but with a quite general activation function, we characterize the relation between pairs of weight vectors determining networks with the same input-output function.

Robustness of representations in multilayer feedforward neural networks

Recent research has shown that multilayer feedforward networks with sigmoidal activation functions are universal approximators, and that this holds for more general activations as well. The mathematical underpinning for these results has been various: Kolmogorov's resolution of Hilbert's thirteenth problem; the Stone-Weierstrass theorem; approximation of Fourier and Radon integral representations; and convergence of probability measures. This paper • Rigorously establishes the robustness of feedforward network realizations. • Uses a theorem of Wiener and ideas of translation invariant subspaces to provide conditions for universal approximations toL1 andL2 functions by networks, for quite general activation functions.