Linear Threshold Networks Research Articles

Efficient Reconstruction of Neural Mass Dynamics Modeled by Linear-Threshold Networks

Efficient Reconstruction of Neural Mass Dynamics Modeled by Linear-Threshold Networks

Influence minimization in linear threshold networks

The propagation of information and innovation in social networks has been widely studied in recent years. Most of the previous works focus on solving the problem of influence maximization, which aims to identify a small subset of early adopters in a social network to maximize the influence propagation under a given diffusion model. On the contrary in this paper, motivated by real-world scenarios, we propose two different influence minimization problems. We consider a linear threshold diffusion model and provide a general solution to the first problem by solving an integer linear programming problem. For the second problem, we provide a technique to search for an optimal solution that works only in particular cases and discuss a simple heuristic to find a solution in the general case. Several simulations on real datasets are also presented.

Detecting controlling nodes of boolean regulatory networks

Boolean models of regulatory networks are assumed to be tolerant to perturbations. That qualitatively implies that each function can only depend on a few nodes. Biologically motivated constraints further show that functions found in Boolean regulatory networks belong to certain classes of functions, for example, the unate functions. It turns out that these classes have specific properties in the Fourier domain. That motivates us to study the problem of detecting controlling nodes in classes of Boolean networks using spectral techniques. We consider networks with unbalanced functions and functions of an average sensitivity less than , where k is the number of controlling variables for a function. Further, we consider the class of 1-low networks which include unate networks, linear threshold networks, and networks with nested canalyzing functions. We show that the application of spectral learning algorithms leads to both better time and sample complexity for the detection of controlling nodes compared with algorithms based on exhaustive search. For a particular algorithm, we state analytical upper bounds on the number of samples needed to find the controlling nodes of the Boolean functions. Further, improved algorithms for detecting controlling nodes in large-scale unate networks are given and numerically studied.

An asynchronous recurrent linear threshold network approach to solving the traveling salesman problem

In this paper, an approach to solving the classical Traveling Salesman Problem (TSP) using a recurrent network of linear threshold (LT) neurons is proposed. It maps the classical TSP onto a single-layered recurrent neural network by embedding the constraints of the problem directly into the dynamics of the network. The proposed method differs from the classical Hopfield network in the update of state dynamics as well as the use of network activation function. Furthermore, parameter settings for the proposed network are obtained using a genetic algorithm, which ensure a stable convergence of the network for different problems. Simulation results illustrate that the proposed network performs better than the classical Hopfield network for optimization.

Dynamics Analysis and Analog Associative Memory of Networks With LT Neurons

The additive recurrent network structure of linear threshold neurons represents a class of biologically-motivated models, where nonsaturating transfer functions are necessary for representing neuronal activities, such as that of cortical neurons. This paper extends the existing results of dynamics analysis of such linear threshold networks by establishing new and milder conditions for boundedness and asymptotical stability, while allowing for multistability. As a condition for asymptotical stability, it is found that boundedness does not require a deterministic matrix to be symmetric or possess positive off-diagonal entries. The conditions put forward an explicit way to design and analyze such networks. Based on the established theory, an alternate approach to study such networks is through permitted and forbidden sets. An application of the linear threshold (LT) network is analog associative memory, for which a simple design method describing the associative memory is suggested in this paper. The proposed design method is similar to a generalized Hebbian approach, but with distinctions of additional network parameters for normalization, excitation and inhibition, both on a global and local scale. The computational abilities of the network are dependent on its nonlinear dynamics, which in turn is reliant upon the sparsity of the memory vectors.

Selectively grouping neurons in recurrent networks of lateral inhibition.

Winner-take-all networks have been proposed to underlie many of the brain's fundamental computational abilities. However, not much is known about how to extend the grouping of potential winners in these networks beyond single neuron or uniformly arranged groups of neurons. We show that competition between arbitrary groups of neurons can be realized by organizing lateral inhibition in linear threshold networks. Given a collection of potentially overlapping groups (with the exception of some degenerate cases), the lateral inhibition results in network dynamics such that any permitted set of neurons that can be coactivated by some input at a stable steady state is contained in one of the groups. The information about the input is preserved in this operation. The activity level of a neuron in a permitted set corresponds to its stimulus strength, amplified by some constant. Sets of neurons that are not part of a group cannot be coactivated by any input at a stable steady state. We analyze the storage capacity of such a network for random groups--the number of random groups the network can store as permitted sets without creating too many spurious ones. In this framework, we calculate the optimal sparsity of the groups (maximizing group entropy). We find that for dense inputs, the optimal sparsity is unphysiologically small. However, when the inputs and the groups are equally sparse, we derive a more plausible optimal sparsity. We believe our results are the first steps toward attractor theories in hybrid analog-digital networks.

Hardness results for neural network approximation problems

We consider the problem of efficiently learning in two-layer neural networks. We investigate the computational complexity of agnostically learning with simple families of neural networks as the hypothesis classes. We show that it is NP-hard to find a linear threshold network of a fixed size that approximately minimizes the proportion of misclassified examples in a training set, even if there is a network that correctly classifies all of the training examples. In particular, for a training set that is correctly classified by some two-layer linear threshold network with k hidden units, it is NP-hard to find such a network that makes mistakes on a proportion smaller than c/ k 2 of the examples, for some constant c. We prove a similar result for the problem of approximately minimizing the quadratic loss of a two-layer network with a sigmoid output unit.

Signed digit addition and related operations with threshold logic

Assuming signed digit number representations, we investigate the implementation of some addition related operations assuming linear threshold networks. We measure the depth and size of the networks in terms of linear threshold gates. We show first that a depth-2 network with O(n) size, weight, and fan-in complexities can perform signed digit symmetric functions. Consequently, assuming radix-2 signed digit representation, we show that the two operand addition can be performed by a threshold network of depth-2 having O(n) size complexity and O(1) weight and fan-in complexities. Furthermore, we show that, assuming radix-(2n-1) signed digit representations, the multioperand addition can be computed by a depth-2 network with O(n/sup 3/) size with the weight and fan-in complexities being polynomially bounded. Finally, we show that multiplication can be performed by a linear threshold network of depth-3 with the size of O(n/sup 3/) requiring O(n/sup 3/) weights and O(n/sup 2/ log n) fan-in.

On the piecewise analysis of networks of linear threshold neurons

The computational abilities of recurrent networks of neurons with a linear activation function above threshold are analyzed. These networks selectively realise a linear mapping of their input. Using this property, the dynamics as well as the number and the stability of stationary states can be investigated. The important property of the boundedness of neural activities can be guaranteed by global inhibition. If used together with self-excitation, the global inhibition gives rise to a multi stable winner-take-all (WTA) mechanism. A condition for a neuron to be a potential winner of the competing dynamics is derived. The network becomes a largest input selector when the self-excitation is marginal. Slowing down the global inhibition produces oscillations. The study of oscillations of random networks suggests that all cyclic trajectories of linear threshold networks are a result of the existence of partitions with undamped linear oscillations. Chaotic dynamics were never encountered in computer simulations and perhaps do not exist at all in small networks.

Capacity of multilevel threshold devices

The principal purpose of this research is to discover the underlying properties of linear multilevel threshold devices and networks of linear multilevel threshold devices. The main theoretical developments of this investigation entail generalizing the function-counting theorem associated with linear bi-level threshold devices to that of linear multilevel threshold devices. The investigation reveals surprising connections that underlie linear inequalities, linear separability, linear ordering, and region counting. The results have implications in the field of geometric probability. Computations based on theoretical analysis support Brown's (1964) conjecture and suggest that k/(k-1) is a natural definition for the information-storage capacity of a k-level threshold device. A lower bound on the number of weights required to implement a universal network of k-level threshold devices is derived. Finally, it is shown that the Vapnik-Chervonenki (1971)s dimension (VC-dimension) for the class of multilevel threshold function is d+1 for pattern vectors in /spl Rscr//sup d/. This VC-dimension is also linked to an error-rate bound for multilevel threshold functions within the framework of uniform learnability.

Aspects of systems and circuits for nanoelectronics

A large number of devices, a limitation of wiring, and very low power dissipation density are design constraints of future nanoelectronic circuits composed of quantum-effect devices. Furthermore, functional integration, which is the possibility of exploiting quantum effects to obtain a function specific behavior, becomes a core design principle. This paper analyzes the effect of this technological progress on the design of nanoelectronic circuits and describes computational paradigms revealing novel features such as distributed storage, fault tolerance, self-organization, and local processing. In particular, linear threshold networks, the associative matrix, self-organizing feature maps, and cellular arrays are investigated from the viewpoint of their potential significance for nanoelectronics. Although these concepts have already been implemented using present technologies, the intention of this paper is to give an impression of their usefulness to system implementations with quantum-effect devices.

δ-Bit serial binary addition with linear threshold networks

In this paper we investigate δ-bit serial addition in the context of feed-forward linear threshold gate based networks. We show that twon-bit operands can be added in\(2\left\lceil {\sqrt n } \right\rceil \) overall delay with a feed-forward network constructed with\(\left\lceil {\sqrt n } \right\rceil + 1\) linear threshold gates and\(\frac{1}{2}\left( {5\left\lceil {\sqrt n } \right\rceil ^2 + 9\left\lceil {\sqrt n } \right\rceil } \right) + 2\) latches. The maximum weight value is\(2^{\left\lceil {\sqrt n } \right\rceil } \) and the maximum fan-in is\(3\left\lceil {\sqrt n } \right\rceil + 1\). We also investigate the implications our scheme have to the performance and the cost under small weights and small fan-in requirements. We deduce that if the weight values are to be limited by a constantW, twon-bit operands can be added in\(\left[ {\log W} \right] + \tfrac{n}{{\left[ {\log W} \right]}}\) overall delay with a feed-forward network that has the implementation cost [logW]+1, in terms of linear threshold gates,\(\tfrac{1}{2}(5[\log W]^2 + 9[\log W]) + 2\) in terms of latches and a maximum fan-in of 3[logW]+1. We also prove that, if the fan-in values are to be limited by a constantF+1, twon-bit operands can be added in\([\tfrac{F}{3}] + \tfrac{n}{{[\tfrac{F}{3}]}}\) overall delay with a feed-forward network that has the implementation cost\([\tfrac{F}{3}] + 1\), in terms of linear threshold gates,\(\tfrac{1}{2}(5[\tfrac{F}{3}]^2 + 9[\tfrac{F}{3}]) + 2\) in terms of latches, and a maximum weight value of\(2^{[\tfrac{F}{3}]} \). An asymptotic bound of\(O(\tfrac{n}{{\log n}})\) is derived for the addition overall delay in the case that the weight values have to be linearly bounded, i.e., in the order ofO(n). The implementation cost in this case is in the order ofO(logn), in terms of linear threshold gates, and in the order ofO(log2n), in terms of latches. The maximum fan-in is in the order ofO(logn). Finally, a partition technique, that substantially reduces the overall cost of the implementation for all the schemes in terms of delay, latches, weights, and fan-in with some few additional threshold gates, is also presented.

2-1 addition and related arithmetic operations with threshold logic

In this paper we investigate the reduction of the size for small depth feed-forward linear threshold networks performing binary addition and related functions. For n bit operands we propose a depth-3 O(n <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> /log n) asymptotic size network for the binary addition with O polynomially bounded weights. We propose also a depth-3 addition of optimal O(n) asymptotic sits network and a depth-2 comparison of O(√n) asymptotic size network, both with O(2 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">√n</sup> ) asymptotic size of weight values. For existing architectural formats we show that our schemes, with equal or smaller depth networks, substantially outperform existing schemes in terms of size and fan-in requirements and on occasions in weight requirements.

Generalisation in higher order neural networks

The theory of pac-learning has provided statements about the generalisation capability of linear threshold networks. The Letter presents an extension of this work to higher order threshold neural networks and shows that the generalisation ability of higher order threshold networks can be characterised in the same way as the linear threshold case.

Classes of feedforward neural networks and their circuit complexity

This paper aims to place neural networks in the context of boolean circuit complexity. We define appropriate classes of feedforward neural networks with specified fan-in, accuracy of computation and depth and using techniques of communication complexity proceed to show that the classes fit into a well-studied hierarchy of boolean circuits. Results cover both classes of sigmoid activation function networks and linear threshold networks. This provides a much needed theoretical basis for the study of the computational power of feedforward neural networks.

Linearly Separable Codes for Adaptive Threshold Networks

The significance of the coding assignment in linear threshold networks used for pattern recognition has been discussed. A heuristic procedure for generating linearly separable codes for a sample set of patterns has been outlined. Experimental results showing the recognition performance of codes so chosen have been shown. The performance of these codes is comparable to the 1-out-of-n code commonly used, but more optimum codes may exist. More work on the coding assignment problem is in order and should allow more efficient linear threshold networks in the pattern recognition application.