Nonmonotonic Transfer Function Research Articles

Statistical mechanics of lossy compression for nonmonotonic multilayer perceptrons

A lossy data compression scheme for uniformly biased Boolean messages is investigated via statistical mechanics techniques. We utilize a treelike committee machine (committee tree) and a treelike parity machine (parity tree) whose transfer functions are nonmonotonic. The scheme performance at the infinite code length limit is analyzed using the replica method. Both committee and parity treelike networks are shown to saturate the Shannon bound. The Almeida-Thouless stability of the replica symmetric solution is analyzed, and the tuning of the nonmonotonic transfer function is also discussed.

Chaotic Behavior for Asymmetrically Diluted Hopfield Neural Network with a Non-Monotonic Transfer Function

We investigate retrieval properties for a synchronous asymmetrically diluted Hopfield neural network with a non-monotonic transfer function by an analytic method. Because of asymmetry of interaction and non-monotonicity of the transfer function, it is difficult to use conventional methods of the equilibrium statistical mechanics in order to investigate the network. We therefore use a generating-function method of path-integral representation, and then obtain an equation for a dynamical order parameter. We find chaotic behavior for the retrieval overlap by choosing a threshold adequately; a Lyapunov exponent is positive in that case. We clarify a bifurcation diagram in a plane specified by a threshold and a load parameter at zero temperature.

非単調ＣＤＭＡニューラルネットを用いた連想記憶イメージセンサの数値的考察

We propose a construction of intelligent image sensors that perform associative memory tasks with nonmonotonic CDMA neural networks. The CDMA approach allows implementing Hopfield's associative neural network on a 2-D rectangular grid (chip), while the storage capacity of the chip is increased by employing a nonmonotonic transfer function. The numerical results of the nonmonotonic CDMA neural networks indicated that i) the CDMA network performed the associative memory task well although the multiple-valued outputs were muxed and demuxed by the CDMA, ii) the number of neurons that could be implemented on a 1 cm×1 cm chip was 340 for a 0.6-μm CMOS process, and iii) the storage capacity (the number of stored patterns per number of neurons) was approximately 0.3, which implies a possible development of associative memory systems on image sensors.

Layered neural networks with non-monotonic transfer functions

We investigate storage capacity and generalization ability for two types of fully connected layered neural networks with non-monotonic transfer functions; random patterns are embedded into the networks by a Hebbian learning rule. One of them is a layered network in which a non-monotonic transfer function of even layers is different from that of odd layers. The other is a layered network with intra-layer connections, in which the non-monotonic transfer function of inter-layer is different from that of intra-layer, and inter-layered neurons and intra-layered neurons are updated alternately. We derive recursion relations for order parameters for those layered networks by the signal-to-noise ratio method. We clarify that the storage capacity and the generalization ability for those layered networks are enhanced in comparison with those with a conventional monotonic transfer function when non-monotonicity of the transfer functions is selected optimally. We also point out that some chaotic behavior appears in the order parameters for the layered networks when non-monotonicity of the transfer functions increases.

Sequence Processing Neural Network with a Non-Monotonic Transfer Function

We investigate storage capacity and retrieval property for a synchronous fully connected neural network with a non-monotonic transfer function which retrieves sequences of patterns, by an analytic method and also by numerical simulations. Because of asymmetry of interactions and non-monotonicity of the transfer function, it is difficult to use conventional methods of the equilibrium statistical mechanics in order to investigate the network. We then use a generating-function method of path-integral representation, and obtain equations for dynamical order parameters in the stationary state. We clarify that the network with the non-monotonic transfer function retrieves more sequences of patterns than that with a monotonic transfer function at zero temperature when non-monotonicity of the transfer function is selected optimally. It is also clarified that some chaotic behavior appears in solutions for the equations of the dynamical order parameters when non-monotonicity of the transfer function increases. The ...

CHAOS IN DILUTED NETWORKS WITH CONTINUOUS NEURONS

The properties of macroscopic chaos displayed by diluted neural networks with continuous neurons and nonmonotonic transfer function is analyzed, with both fixed and dynamic synapses. The responce to external noisy stimuli is also considered: a proper amount of external perturbation removes the chaotic behaviour.

Chaos in neural networks with a nonmonotonic transfer function

Time evolution of diluted neural networks with a nonmonotonic transfer function is analytically described by flow equations for macroscopic variables. The macroscopic dynamics shows a rich variety of behaviors: fixed-point, periodicity, and chaos. We examine in detail the structure of the strange attractor and in particular we study the main features of the stable and unstable manifolds, the hyperbolicity of the attractor, and the existence of homoclinic intersections. We also discuss the problem of the robustness of the chaos and we prove that in the present model chaotic behavior is fragile (chaotic regions are densely intercalated with periodicity windows), according to a recently discussed conjecture. Finally we perform an analysis of the microscopic behavior and in particular we examine the occurrence of damage spreading by studying the time evolution of two almost identical initial configurations. We show that for any choice of the parameters the two initial states remain microscopically distinct.

Long-term properties of time series generated by a perceptron with various transfer functions

We study the effect of various transfer functions on the properties of a time series generated by a continuous-valued feed-forward network in which the next input vector is determined from past output values. The parameter space for monotonic and non-monotonic transfer functions is analyzed in the unstable regions with the following main finding; non-monotonic functions can produce robust chaos whereas monotonic functions generate fragile chaos only. In the case of non-monotonic functions, the number of positive Lyapunov exponents increases as a function of one of the free parameters in the model, hence, high dimensional chaotic attractors can be generated. We extend the analysis to a combination of monotonic and non-monotonic functions.

Associative memory based on parametrically coupled chaotic elements

In this paper, we propose an associative memory system based on parametrically coupled chaotic elements. The proposed system is obtained by adding a new parameter control to our previously proposed system. A chaotic activity in an early association stage makes an efficient association over the memories that are stored by means of autocorrelational learning. When the system successfully recalls the target memory, the system's motion is dominated by a spatially coherent oscillation, while unstable motions remain when the system fails to make the association. In addition, the system has a large memory capacity. A comparison between the proposed system and an approach with a nonmonotonic transfer function is also shown.

Generalization ability of a perceptron with nonmonotonic transfer function

We investigate the generalization ability of a perceptron with nonmonotonic transfer function of a reversed-wedge type in on-line mode. This network is identical to a parity machine, a multilayer network. We consider several learning algorithms. By the perceptron algorithm the generalization error is shown to decrease by the ${\ensuremath{\alpha}}^{\ensuremath{-}1/3}$-law similarly to the case of a simple perceptron in a restricted range of the parameter $a$ characterizing the nonmonotonic transfer function. For other values of $a$, the perceptron algorithm leads to the state where the weight vector of the student is just opposite to that of the teacher. The Hebbian learning algorithm has a similar property; it works only in a limited range of the parameter. The conventional AdaTron algorithm does not give a vanishing generalization error for any values of $a$. We thus introduce a modified AdaTron algorithm that yields a good performance for all values of $a$. We also investigate the effects of optimization of the learning rate as well as of the learning algorithm. Both methods give excellent learning curves proportional to ${\ensuremath{\alpha}}^{\ensuremath{-}1}$. The latter optimization is related to the Bayes statistics and is shown to yield useful hints to extract maximum amount of information necessary to accelerate learning processes.

Learning of non-monotonic rules by simple perceptrons

In this paper, we study the generalization ability of a simple perceptron which learns an unrealizable Boolean function represented by a perceptron with a non-monotonic transfer function of reversed-wedge type. This type of non-monotonic perceptron is considered as a variant of multilayer perceptron and is parametrized by a single `wedge' parameter a. Reflecting the non-monotonic nature of the target function, a discontinuous transition from the poor generalization phase to the good generalization phase is observed in the learning curve for intermediate values of a. We also find that asymptotic learning curves are classified into the following two categories depending on a. For large a, the learning curve obeys a power law with exponent 1. On the other hand, a power law with exponent is obtained for small a. Although these two exponents are obtained from unstable replica symmetric solutions by using the replica method, they are consistent with the results obtainable without using the replica method in a low-dimensional version of this learning problem. This suggests that our results are good approximations even if they are not exact.

On-line learning of non-monotonic rules by simple perceptron

We study the generalization ability of a simple perceptron which learns unlearnable rules. The rules are presented by a teacher perceptron with a non-monotonic transfer function. The student is trained in the on-line mode. The asymptotic behaviour of the generalization error is estimated under various conditions. Several learning strategies are proposed and improved to obtain the theoretical lower bound of the generalization error.

Associative Memory of Analog Networks without the Energy Concept Using Biased Patterns and Positive-Valued Transfer Functions

Associative memory of analog neural networks for which the concept of energy functions is lost is studied by means of the SCSNA (Self-Consistent Signal-to-Noise Analysis). Asymmetric synaptic couplings with biased random patterns are assumed together with positive-valued transfer functions which allow nonmonotonicity resulting from an appropriate cut off of output activity. Phase diagrams are given in terms of several parameters of the networks, showing the occurrence of enhancement of the storage capacity due to nonmonotonic transfer functions. Super retrieval states ensuring errorless memory retrieval under the loading of extensively many patterns are allowed to remain in existence in the presence of asymmetric synaptic couplings with biased patterns. A sample-dependent component in the local fields of neurons, which arises from the assumed asymmetric couplings, is discussed and shown to become of no effect in the super retrieval phase.

On-line AdaTron learning of unlearnable rules

We study the on-line AdaTron learning of linearly non-separable rules by a simple perceptron. Training examples are provided by a perceptron with a non-monotonic transfer function which reduces to the usual monotonic relation in a certain limit. We find that, although the on-line AdaTron learning is a powerful algorithm for the learnable rule, it does not give the best possible generalization error for unlearnable problems. Optimization of the learning rate is shown to greatly improve the performance of the AdaTron algorithm, leading to the best possible generalization error for a wide range of the parameter which controls the shape of the transfer function.)

Retrieval phase diagrams of non-monotonic Hopfield networks

We investigate the retrieval phase diagrams of an asynchronous fully-connected attractor network with non-monotonic transfer function by means of a mean-field approximation. We find for the noiseless zero-temperature case that this non-monotonic Hopfield network can store more patterns than a network with monotonic transfer function investigated by Amit et al. Properties of retrieval phase diagrams of non-monotonic networks agree with the results obtained by Nishimori and Opris who treated synchronous networks. We also investigate the optimal storage capacity of the non-monotonic Hopfield model with state-dependent synaptic couplings introduced by Zertuche et el. We show that the non-monotonic Hopfield model with state-dependent synapses stores more patterns than the conventional Hopfield model. Our formulation can be easily extended to a general transfer function.

Sinusoidal and monotonic transfer functions: Implications for VC dimension

It is sometimes stated that neural networks that employ units with nonmonotonic transfer functions are more difficult to train than networks that use monotonic transfer functions, because the former can be expected to have more local minima. That this is often true arises from the fact that networks using monotonic transfer functions tend to have a smaller VC (Vapnik-Chervonenkis) dimension than networks using nonmonotonic transfer junctions. But the VC dimension of a network is not solely influenced by the nature of the transfer function. We give an example of a network with an infinite VC dimension and demonstrate that it is equivalent to a network which contains only monotonic transfer functions. Thus we show that monotonicity alone is not a sufficient criterion to avoid large VC dimension.

Retrieval properties of analog neural networks and the nonmonotonicity of transfer functions

Characteristic properties of associative memory networks with continuous-time dynamics are extensively studied for a certain class of nonmonotonic transfer functions by means of the self-consistent signal-to-noise analysis (SCSNA) and numerical simulations. The conventional Hebb-type symmetric synaptic connections with unbiased random patterns are assumed. Although the occurrence of instability, including an oscillatory one, makes the storage capacity fall below the upper bound for storage ratio obtained by the SCSNA, the storage capacity remains as large as 0.4 in the optimal cases. It is also noted that noise in the local fields (i.e., the inputs to neurons) can vanish for certain cases of nonmonotonic transfer functions even with an extensive number of stored patterns. Implication of the present results is the possibility of improving the network performances by the achievement of errorless retrieval and enhancement of storage capacity with the use of nonmonotonic transfer functions.

Onset of 'super retrieval phase' and enhancement of the storage capacity in neural networks of nonmonotonic neurons

Analogue neural networks of associative memory with continuous time dynamics are studied for nonmonotonic transfer functions using the method of self-consistent signal-to-noise analysis. The Hebb learning rule with unbiased random patterns is assumed for the synaptic couplings. A novel phenomenon is found to occur as a result of a phase transition concerning the property of the local field distribution. In retrieval states of the newly found phase which the authors refer to as the super retrieval phase, noise in the local field vanishes and the memory retrieval without errors ensures even for an extensive number of memory patterns stored under the local learning rule. The storage capacity is obtained as a function of the parameter representing the degree of nonmonotonicity of the transfer functions, with the result that an enhancement of the storage capacity can also occur.

Radial basis function networks for classifying process faults

It is shown that under certain conditions the backpropagation network classifier can produce nonintuitive, nonrobust decision surfaces. These result from the inherent nature of the sigmoid transfer function, the definition of the training set, and the error function used for training. The backpropagation network has no mechanism in the standard training scheme for identifying regions not in any known classes. The radial basis function network overcomes these difficulties by using a nonmonotonic transfer function based on the Gaussian density function. While producing robust decision surfaces, the radial basis function also provides an estimate of how close a test case is to the original training data, allowing the classifier to signal that a test case potentially represents a novel class while still presenting the most plausible classification. For applications where this type of behavior is important, such as fault diagnosis, the radial basis function network is shown to offer clear advantages over the backpropagation network. The radial basis function is also faster to train because the training of the two layers is decoupled.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>