Piecewise Linear Activation Functions Research Articles

A new chaotic Hopfield network with piecewise linear activation function

This paper presents a new chaotic Hopfield network with a piecewise linear activation function. The dynamic of the network is studied by virtue of the bifurcation diagram, Lyapunov exponents spectrum and power spectrum. Numerical simulations show that the network displays chaotic behaviours for some well selected parameters.

Coexistence and local stability of multiple equilibria in neural networks with piecewise linear nondecreasing activation functions

In this paper, we investigate the neural networks with a class of nondecreasing piecewise linear activation functions with 2r corner points. It is proposed that the n-neuron dynamical systems can have and only have (2r+1)(n) equilibria under some conditions, of which (r+1)(n) are locally exponentially stable and others are unstable. Furthermore, the attraction basins of these stationary equilibria are estimated. In the case of n=2, the precise attraction basin of each stable equilibrium point can be figured out, and their boundaries are composed of the stable manifolds of unstable equilibrium points. Simulations are also provided to illustrate the effectiveness of our results.

Multistability of discrete-time recurrent neural networks with unsaturating piecewise linear activation functions.

This paper studies the multistability of a class of discrete-time recurrent neural networks with unsaturating piecewise linear activation functions. It addresses the nondivergence, global attractivity, and complete stability of the networks. Using the local inhibition, conditions for nondivergence are derived, which not only guarantee nondivergence, but also allow for the existence of multiequilibrium points. Under these nondivergence conditions, global attractive compact sets are obtained. Complete stability is studied via constructing novel energy functions and using the well-known Cauchy Convergence Principle. Examples and simulation results are used to illustrate the theory.

A digital hardware pulse-mode neuron with piecewise linear activation function

This paper proposes a new type of digital pulse-mode neuron that employs piecewise-linear function as its activation function. The neuron is implemented on field programmable gate array (FPGA) and tested by experiments. As well as theoretical analysis, the experimental results show that the piecewise-linear function of the proposed neuron is programmable and robust against the change in the number of input signals. To demonstrate the effect of piecewise-linear activation function, pulse-mode multilayer neural network with on-chip learning is implemented on FPGA with the proposed neuron, and its learning performance is verified by experiments. By approximating the sigmoid function by the piecewise-linear function, the convergence rate of the learning and generalization capability are improved.

The Hi- noon neural simulator and its applications

This paper describes the Hi- noon (hierarchical network of object-oriented neurons) neural simulator, originally conceived as a general-purpose, computationally efficient, object-oriented software system for the simulation of small systems of biological neurons, as an aid to the study of links between neurophysiology and behaviour in lower animals. As such, the artificial neurons employed were spiking in nature; to effect an appropriate compromise between computational complexity and biological realism, modelling was at the transmembrane potential level of abstraction. Further, since real neural systems incorporate different types of neurons specialised to somewhat different functions, the software was written to accommodate a non-homogeneous population of neurons. The computational efficiency of Hi- noon makes it eminently suitable for situated system studies (biological robotics, animats) where real-time operation is a pre-requisite. The flexibility which was a central design goal of Hi- noon means that the system is also capable of modelling interconnections of non-spiking artificial neurons with continuous or piecewise linear activation functions. The efficacy of the simulator is illustrated with respect to some recent applications to situated systems studies. We also consider prospects for integrating Hi- noon with a conventional circuit simulator in the future.

A learning algorithm for oscillatory cellular neural networks

We present a cellular type oscillatory neural network for temporal segregation of stationary input patterns. The model comprises an array of locally connected neural oscillators with connections limited to a 4-connected neighborhood. The architecture is reminiscent of the well-known cellular neural network that consists of local connection for feature extraction. By means of a novel learning rule and an initialization scheme, global synchronization can be accomplished without incurring any erroneous synchrony among uncorrelated objects. Each oscillator comprises two mutually coupled neurons, and neurons share a piecewise-linear activation function characteristic. The dynamics of traditional oscillatory models is simplified by using only one plastic synapse, and the overall complexity for hardware implementation is reduced. Based on the connectedness of image segments, it is shown that global synchronization and desynchronization can be achieved by means of locally connected synapses, and this opens up a tremendous application potential for the proposed architecture. Furthermore, by using special grouping synapses it is demonstrated that temporal segregation of overlapping gray-level and color segments can also be achieved. Finally, simulation results show that the learning rule proposed circumvents the problem of component mismatches, and hence facilitates a large-scale integration.

Nine switch-affine neurons suffice for Turing universality

In a previous work Pollack showed that a particular type of heterogeneous processor network is Turing universal. Siegelmann and Sontag (1991) showed the universality of homogeneous networks of first-order neurons having piecewise-linear activation functions. Their result was generalized by Kilian and Siegelmann (1996) to include various sigmoidal activation functions. Here we focus on a type of high-order neurons called switch-affine neurons, with piecewise-linear activation functions, and prove that nine such neurons suffice for simulating universal Turing machines.

Chaotic Dynamics in Iterated Map Neural Networks with Piecewise Linear Activation Function

The paper examines the discrete-time dynamics of neuron models (of excitatory and inhibitory types) with piecewise linear activation functions, which are connected in a network. The properties of a pair of neurons (one excitatory and the other inhibitory) connected with each other, is studied in detail. Even such a simple system shows a rich variety of behavior, including high-period oscillations and chaos. Border-collision bifurcations and multifractal fragmentation of the phase space is also observed for a range of parameter values. Extension of the model to a larger number of neurons is suggested under certain restrictive assumptions, which makes the resultant network dynamics effectively one-dimensional. Possible applications of the network for information processing are outlined. These include using the network for auto-association, pattern classification, nonlinear function approximation and periodic sequence generation.

Learning with piece-wise linear networks.

A new feedforward architecture is presented for empirical model building and regression. The network consists of two hidden layers of units, where each unit utilizes a piece-wise linear activation function. A procedure for determining both the number of units and their connectivity is developed. The most notable feature of the network is its associated learning algorithm which allows for recursive updating of the parameters. A smoothness constraint is employed to limit the range of solutions, so that practical models may be built with small amounts of data. The network is applied to some function estimation tasks, as well as to a forecasting problem using data from the Santa Fe Institute time-series competition.

Solution of nonlinear ordinary differential equations by feedforward neural networks

It is demonstrated, through theory and numerical examples, how it is possible to directly construct a feedforward neural network to approximate nonlinear ordinary differential equations without the need for training. The method, utilizing a piecewise linear map as the activation function, is linear in storage, and the L 2 norm of the network approximation error decreases monotonically with the increasing number of hidden layer neurons. The construction requires imposing certain constraints on the values of the input, bias, and output weights, and the attribution of certain roles to each of these parameters. All results presented used the piecewise linear activation function. However, the presented approach should also be applicable to the use of hyperbolic tangents, sigmoids, and radial basis functions.

Efficient implementation of piecewise linear activation function for digital VLSI neural networks

A piecewise linear approximation to the sigmoidal neuron activation function is proposed, which maps compactly into a digital integrated circuit realisation.