Linear Threshold Neurons Research Articles

Global Mittag-Leffler Boundedness of Fractional-Order Fuzzy Quaternion-Valued Neural Networks With Linear Threshold Neurons

This article focuses on the global Mittag-Leffler boundedness for fractional-order fuzzy quaternion-valued neural networks (QVNNs) with linear threshold neurons. In order to avert the nonexchangeability for quaternion multiplication, the considered QVNN is separated into four real-valued or two complex-valued systems. By employing Lyapunov function method and fractional-order differential inequalities, several effective conditions according to algebraic inequality and complex-valued linear matrix inequalities are deduced to guarantee the global Mittag-Leffler boundedness of the addressed network. And, the frame of the global Mittag-Leffler attracting sets is presented as well. Here, the numbers of the equilibrium points need not to be concerned. Finally, a numerical example is provided to demonstrate the correctness of the proposed results.

Exponential stability analysis of quaternion-valued neural networks with proportional delays and linear threshold neurons: Continuous-time and discrete-time cases

A class of quaternion-valued neural networks (QVNNs) with proportional delays and linear threshold neurons is proposed in this paper. First, by employing Halanay inequality technique and matrix measure method, the global exponential stability of continuous-time QVNNs with proportional delays and linear threshold neurons is studied, and some sufficient conditions are derived to guarantee global exponential stability of the studied continuous-time systems. Then, the discrete-time analogues of the continuous-time QVNNs with proportional delays and linear threshold neurons are formulated and investigated by using the semi-discretization method. The discrete-time analogues are equivalent to the considered continuous-time neural networks, and possess the convergence behaviors of the considered continuous-time systems without any limitation applied to the discretization step size. Finally, some numerical examples are presented to ensure the effectiveness and correctness of the theoretical results obtained.

The capacity of feedforward neural networks

A long standing open problem in the theory of neural networks is the development of quantitative methods to estimate and compare the capabilities of different architectures. Here we define the capacity of an architecture by the binary logarithm of the number of functions it can compute, as the synaptic weights are varied. The capacity provides an upperbound on the number of bits that can be extracted from the training data and stored in the architecture during learning. We study the capacity of layered, fully-connected, architectures of linear threshold neurons with L layers of size n1,n2,…,nL and show that in essence the capacity is given by a cubic polynomial in the layer sizes: C(n1,…,nL)=∑k=1L−1min(n1,…,nk)nknk+1, where layers that are smaller than all previous layers act as bottlenecks. In proving the main result, we also develop new techniques (multiplexing, enrichment, and stacking) as well as new bounds on the capacity of finite sets. We use the main result to identify architectures with maximal or minimal capacity under a number of natural constraints. This leads to the notion of structural regularization for deep architectures. While in general, everything else being equal, shallow networks compute more functions than deep networks, the functions computed by deep networks are more regular and “interesting”.

Global Mittag-Leffler stability and synchronization analysis of fractional-order quaternion-valued neural networks with linear threshold neurons

This paper talks about the stability and synchronization problems of fractional-order quaternion-valued neural networks (FQVNNs) with linear threshold neurons. On account of the non-commutativity of quaternion multiplication resulting from Hamilton rules, the FQVNN models are separated into four real-valued neural network (RVNN) models. Consequently, the dynamic analysis of FQVNNs can be realized by investigating the real-valued ones. Based on the method of M-matrix, the existence and uniqueness of the equilibrium point of the FQVNNs are obtained without detailed proof. Afterwards, several sufficient criteria ensuring the global Mittag-Leffler stability for the unique equilibrium point of the FQVNNs are derived by applying the Lyapunov direct method, the theory of fractional differential equation, the theory of matrix eigenvalue, and some inequality techniques. In the meanwhile, global Mittag-Leffler synchronization for the drive–response models of the addressed FQVNNs are investigated explicitly. Finally, simulation examples are designed to verify the feasibility and availability of the theoretical results.

Solving Constraint-Satisfaction Problems with Distributed Neocortical-Like Neuronal Networks.

Finding actions that satisfy the constraints imposed by both external inputs and internal representations is central to decision making. We demonstrate that some important classes of constraint satisfaction problems (CSPs) can be solved by networks composed of homogeneous cooperative-competitive modules that have connectivity similar to motifs observed in the superficial layers of neocortex. The winner-take-all modules are sparsely coupled by programming neurons that embed the constraints onto the otherwise homogeneous modular computational substrate. We show rules that embed any instance of the CSP's planar four-color graph coloring, maximum independent set, and sudoku on this substrate and provide mathematical proofs that guarantee these graph coloring problems will convergence to a solution. The network is composed of nonsaturating linear threshold neurons. Their lack of right saturation allows the overall network to explore the problem space driven through the unstable dynamics generated by recurrent excitation. The direction of exploration is steered by the constraint neurons. While many problems can be solved using only linear inhibitory constraints, network performance on hard problems benefits significantly when these negative constraints are implemented by nonlinear multiplicative inhibition. Overall, our results demonstrate the importance of instability rather than stability in network computation and offer insight into the computational role of dual inhibitory mechanisms in neural circuits.

Stability Analysis of Continuous-Time and Discrete-Time Quaternion-Valued Neural Networks With Linear Threshold Neurons.

This paper addresses the problem of stability for continuous-time and discrete-time quaternion-valued neural networks (QVNNs) with linear threshold neurons. Applying the semidiscretization technique to the continuous-time QVNNs, the discrete-time analogs are obtained, which preserve the dynamical characteristics of their continuous-time counterparts. Via the plural decomposition method of quaternion, homeomorphic mapping theorem, as well as Lyapunov theorem, some sufficient conditions on the existence, uniqueness, and global asymptotical stability of the equilibrium point are derived for the continuous-time QVNNs and their discrete-time analogs, respectively. Furthermore, a uniform sufficient condition on the existence, uniqueness, and global asymptotical stability of the equilibrium point is obtained for both continuous-time QVNNs and their discrete-time version. Finally, two numerical examples are provided to substantiate the effectiveness of the proposed results.

Computation in dynamically bounded asymmetric systems.

Previous explanations of computations performed by recurrent networks have focused on symmetrically connected saturating neurons and their convergence toward attractors. Here we analyze the behavior of asymmetrical connected networks of linear threshold neurons, whose positive response is unbounded. We show that, for a wide range of parameters, this asymmetry brings interesting and computationally useful dynamical properties. When driven by input, the network explores potential solutions through highly unstable ‘expansion’ dynamics. This expansion is steered and constrained by negative divergence of the dynamics, which ensures that the dimensionality of the solution space continues to reduce until an acceptable solution manifold is reached. Then the system contracts stably on this manifold towards its final solution trajectory. The unstable positive feedback and cross inhibition that underlie expansion and divergence are common motifs in molecular and neuronal networks. Therefore we propose that very simple organizational constraints that combine these motifs can lead to spontaneous computation and so to the spontaneous modification of entropy that is characteristic of living systems.

Competitive Layer Model of Discrete-Time Recurrent Neural Networks with LT Neurons

This letter discusses the competitive layer model (CLM) for a class of discrete-time recurrent neural networks with linear threshold (LT) neurons. It first addresses the boundedness, global attractivity, and complete stability of the networks. Two theorems are then presented for the networks to have CLM property. We also present the analysis for network dynamics, which performs a column winner-take-all behavior and grouping selection among different layers. Furthermore, we propose a novel synchronous CLM iteration method, which has similar performance and storage allocation but faster convergence compared with the previous asynchronous CLM iteration method (Wersing, Steil, & Ritter, 2001 ). Examples and simulation results are used to illustrate the developed theory, the comparison between two CLM iteration methods, and the application in image segmentation.

Boundedness and Stability for Discrete‐Time Delayed Neural Network with Complex‐Valued Linear Threshold Neurons

The discrete‐time delayed neural network with complex‐valued linear threshold neurons is considered. By constructing appropriate Lyapunov‐Krasovskii functionals and employing linear matrix inequality technique and analysis method, several new delay‐dependent criteria for checking the boundedness and global exponential stability are established. Illustrated examples are also given to show the effectiveness and less conservatism of the proposed criteria.

Discrete-Time Recurrent Neural Networks With Complex-Valued Linear Threshold Neurons

This brief discusses a class of discrete-time recurrent neural networks with complex-valued linear threshold neurons. It addresses the boundedness, global attractivity, and complete stability of such networks. Some conditions for those properties are also derived. Examples and simulation results are used to illustrate the theory.

A Winner-Take-All Neural Networks of N Linear Threshold Neurons without Self-Excitatory Connections

Multistable neural networks have attracted much interests in recent years, since the monostable networks are computationally restricted. This paper studies a N linear threshold neurons recurrent networks without Self-Excitatory connections. Our studies show that this network performs a Winner-Take-All (WTA) behavior, which has been recognized as a basic computational model done in brain. The contributions of this paper are: (1) It proves by mathematics that the proposed model is Non-Divergent. (2) An important implication (Winner-Take-All) of the proposed network model is studied. (3) Digital computer simulations are carried out to validate the performance of the theory findings.

An asynchronous recurrent linear threshold network approach to solving the traveling salesman problem

In this paper, an approach to solving the classical Traveling Salesman Problem (TSP) using a recurrent network of linear threshold (LT) neurons is proposed. It maps the classical TSP onto a single-layered recurrent neural network by embedding the constraints of the problem directly into the dynamics of the network. The proposed method differs from the classical Hopfield network in the update of state dynamics as well as the use of network activation function. Furthermore, parameter settings for the proposed network are obtained using a genetic algorithm, which ensure a stable convergence of the network for different problems. Simulation results illustrate that the proposed network performs better than the classical Hopfield network for optimization.

Recurrent neuronal circuits in the neocortex

We thank our colleagues John Anderson and Tom Binzegger for their collaboration; and EU grant DAISY (FP6-2005-015803) for financial support.

Dynamics Analysis and Analog Associative Memory of Networks With LT Neurons

The additive recurrent network structure of linear threshold neurons represents a class of biologically-motivated models, where nonsaturating transfer functions are necessary for representing neuronal activities, such as that of cortical neurons. This paper extends the existing results of dynamics analysis of such linear threshold networks by establishing new and milder conditions for boundedness and asymptotical stability, while allowing for multistability. As a condition for asymptotical stability, it is found that boundedness does not require a deterministic matrix to be symmetric or possess positive off-diagonal entries. The conditions put forward an explicit way to design and analyze such networks. Based on the established theory, an alternate approach to study such networks is through permitted and forbidden sets. An application of the linear threshold (LT) network is analog associative memory, for which a simple design method describing the associative memory is suggested in this paper. The proposed design method is similar to a generalized Hebbian approach, but with distinctions of additional network parameters for normalization, excitation and inhibition, both on a global and local scale. The computational abilities of the network are dependent on its nonlinear dynamics, which in turn is reliant upon the sparsity of the memory vectors.

Analysis of Cyclic Dynamics for Networks of Linear Threshold Neurons

The network of neurons with linear threshold (LT) transfer functions is a prominent model to emulate the behavior of cortical neurons. The analysis of dynamic properties for LT networks has attracted growing interest, such as multistability and boundedness. However, not much is known about how the connection strength and external inputs are related to oscillatory behaviors. Periodic oscillation is an important characteristic that relates to nondivergence, which shows that the network is still bounded although unstable modes exist. By concentrating on a general parameterized two-cell network, theoretical results for geometrical properties and existence of periodic orbits are presented. Although it is restricted to two-dimensional systems, the analysis can provide a useful contribution to analyze cyclic dynamics of some specific LT networks of high dimension. As an application, it is extended to an important class of biologically motivated networks of large scale: the winner-take-all model using local excitation and global inhibition.

On the power of Boolean computations in generalized RBF neural networks

Generalized radial basis function (RBF) neurons are extensions of the RBF neuron model where the Euclidean norm is replaced by a weighted norm. We study binary-valued variants of generalized RBF neurons and compare their computational power in the Boolean domain with linear threshold neurons. As one of the main results, we show that generalized binary RBF neurons with any weighted norm can compute every Boolean function that is computed by a linear threshold neuron. While this inclusion turns into an equality if the RBF neuron uses the Euclidean norm, we exhibit a weighted norm where the inclusion is proper. Applications of the results yield bounds on the Vapnik–Chervonenkis (VC) dimension of RBF neural networks with binary inputs.

Algorithm for building a neural network for function approximation

A technique for constructing a multilayer tree-structured neural network, which provides a continuous, piece-wise linear function approximation, is presented. The method uses a growth network with linear threshold neurons. Neurons are added to a binary tree until the approximation error at all sampling points is brought down to within a specified ±Δ. The number of neurons in the constructed network depends on the samples provided as well as the specified tolerance Δ, thus enabling a trade-off between accuracy and network size. In comparison to approaches such as backpropagation, the proposed technique requires no assumptions regarding the number of neurons, learning rate, momentum term, or initial weight values. It also does not suffer from problems of local minima. Examples are presented to illustrate the effectiveness of the technique.

A competitive-layer model for feature binding and sensory segmentation.

We present a recurrent neural network for feature binding and sensory segmentation: the competitive-layer model (CLM). The CLM uses topographically structured competitive and cooperative interactions in a layered network to partition a set of input features into salient groups. The dynamics is formulated within a standard additive recurrent network with linear threshold neurons. Contextual relations among features are coded by pairwise compatibilities, which define an energy function to be minimized by the neural dynamics. Due to the usage of dynamical winner-take-all circuits, the model gains more flexible response properties than spin models of segmentation by exploiting amplitude information in the grouping process. We prove analytic results on the convergence and stable attractors of the CLM, which generalize earlier results on winner-take-all networks, and incorporate deterministic annealing for robustness against local minima. The piecewise linear dynamics of the CLM allows a linear eigensubspace analysis, which we use to analyze the dynamics of binding in conjunction with annealing. For the example of contour detection, we show how the CLM can integrate figure-ground segmentation and grouping into a unified model.

On the piecewise analysis of networks of linear threshold neurons

The computational abilities of recurrent networks of neurons with a linear activation function above threshold are analyzed. These networks selectively realise a linear mapping of their input. Using this property, the dynamics as well as the number and the stability of stationary states can be investigated. The important property of the boundedness of neural activities can be guaranteed by global inhibition. If used together with self-excitation, the global inhibition gives rise to a multi stable winner-take-all (WTA) mechanism. A condition for a neuron to be a potential winner of the competing dynamics is derived. The network becomes a largest input selector when the self-excitation is marginal. Slowing down the global inhibition produces oscillations. The study of oscillations of random networks suggests that all cyclic trajectories of linear threshold networks are a result of the existence of partitions with undamped linear oscillations. Chaotic dynamics were never encountered in computer simulations and perhaps do not exist at all in small networks.