Thresholds Of Network Research Articles

Generalization Error of Combined Classifiers

We derive an upper bound on the generalization error of classifiers which can be represented as thresholded convex combinations of thresholded convex combinations of functions. Such classifiers include single hidden-layer threshold networks and voted combinations of decision trees (such as those produced by boosting algorithms). The derived bound depends on the proportion of training examples with margin less than some threshold and the average complexity of the combined functions (where the average is over the weights assigned to each function in the convex combination). The complexity of the individual functions in the combination depends on their closeness to threshold. By representing a decision tree as a thresholded convex combination of weighted leaf functions, we apply this result to bound the generalization error of combinations of decision trees. Previous bounds depend on the margin of the combined classifier and the average complexity of the decision trees in the combination, where the complexity of each decision tree depends on the total number of leaves. Our bound also depends on the margin of the combined classifier and the average complexity of the decision trees, but our measure of complexity for an individual decision tree is based on the distribution of training examples over leaves and can be significantly smaller than the total number of leaves.

Hardness results for neural network approximation problems

We consider the problem of efficiently learning in two-layer neural networks. We investigate the computational complexity of agnostically learning with simple families of neural networks as the hypothesis classes. We show that it is NP-hard to find a linear threshold network of a fixed size that approximately minimizes the proportion of misclassified examples in a training set, even if there is a network that correctly classifies all of the training examples. In particular, for a training set that is correctly classified by some two-layer linear threshold network with k hidden units, it is NP-hard to find such a network that makes mistakes on a proportion smaller than c/ k 2 of the examples, for some constant c. We prove a similar result for the problem of approximately minimizing the quadratic loss of a two-layer network with a sigmoid output unit.

Theory and scope of exact representation extraction from feed-forward networks

An algorithm to extract representations from feed-forward threshold networks is outlined. The representation is based on polytopic decision regions in the input space – and is exact not an approximation. Using this exact representation we explore scope questions, such as when and where do networks form artifacts, or what can we tell about network generalization from its representation. The exact nature of the algorithm also lends itself to theoretical questions about representation extraction in general, such as what is the relationship between factors such as input dimensionality, number of hidden units, number of hidden layers, and how the network output is interpreted to the potential complexity of the network’s function.

Criticality in random threshold networks: annealed approximation and beyond

Random Threshold Networks with sparse, asymmetric connections show complex dynamical behavior similar to Random Boolean Networks, with a transition from ordered to chaotic dynamics at a critical average connectivity K c . In this type of model—contrary to Boolean Networks—propagation of local perturbations (damage) depends on the in-degree of the sites. K c is determined analytically, using an annealed approximation, and the results are confirmed by numerical simulations. It is shown that the statistical distributions of damage spreading near the percolation transition obey power-laws, and dynamical correlations between active network clusters become maximal. We investigate the effect of local damage suppression at highly connected nodes for networks with scale-free in-degree distributions. Possible relations of our findings to properties of real-world networks, like robustness and non-trivial degree-distributions, are discussed.

Threshold counters with increments and decrements

A threshold counter is a shared data structure that assumes integer values. It provides two operations: Increment changes the current counter value from v to v+1, while Read returns the value ⌊v/w⌋, where v is the current counter value and w is a fixed constant. Thus, the Read operation returns the “approximate” value of the counter to within the constant w. Threshold counters have many potential uses, including software barrier synchronization. Threshold networks are a class of distributed data structures that can be used to construct highly-concurrent, low-contention implementations of shared threshold counters. In this paper, we give the first proof that any threshold network construction of a threshold counter can be extended to support a Decrement operation that changes the counter value from v to v−1.

Topological evolution of dynamical networks: global criticality from local dynamics.

We evolve network topology of an asymmetrically connected threshold network by a simple local rewiring rule: quiet nodes grow links, active nodes lose links. This leads to convergence of the average connectivity of the network towards the critical value K(c) = 2 in the limit of large system size N. How this principle could generate self-organization in natural complex systems is discussed for two examples: neural networks and regulatory networks in the genome.

Convergence of threshold networks using their dissipative system model

A new dynamical energy system model representation is given for threshold networks. Inspired by the relation between stability and dissipativeness of dynamical systems, the convergence property of threshold networks is investigated. Using the energy function inherent within the given model a condition, namely the dissipativeness of the dynamical system, necessary and sufficient condition for the convergence of the threshold network to a fixed point, is given. Also, an easy to check inequality is stated to test the convergence of the threshold network. Copyright © 2000 John Wiley & Sons, Ltd.

Signed digit addition and related operations with threshold logic

Assuming signed digit number representations, we investigate the implementation of some addition related operations assuming linear threshold networks. We measure the depth and size of the networks in terms of linear threshold gates. We show first that a depth-2 network with O(n) size, weight, and fan-in complexities can perform signed digit symmetric functions. Consequently, assuming radix-2 signed digit representation, we show that the two operand addition can be performed by a threshold network of depth-2 having O(n) size complexity and O(1) weight and fan-in complexities. Furthermore, we show that, assuming radix-(2n-1) signed digit representations, the multioperand addition can be computed by a depth-2 network with O(n/sup 3/) size with the weight and fan-in complexities being polynomially bounded. Finally, we show that multiplication can be performed by a linear threshold network of depth-3 with the size of O(n/sup 3/) requiring O(n/sup 3/) weights and O(n/sup 2/ log n) fan-in.

Convergence of threshold networks using their dissipative system model

Convergence of threshold networks using their dissipative system model

On the Sample Complexity for Nonoverlapping Neural Networks

A neural network is said to be nonoverlapping if there is at most one edge outgoing from each node. We investigate the number of examples that a learning algorithm needs when using nonoverlapping neural networks as hypotheses. We derive bounds for this sample complexity in terms of the Vapnik-Chervonenkis dimension. In particular, we consider networks consisting of threshold, sigmoidal and linear gates. We show that the class of nonoverlapping threshold networks and the class of nonoverlapping sigmoidal networks on n inputs both have Vapnik-Chervonenkis dimension Ω(nlog n). This bound is asymptotically tight for the class of nonoverlapping threshold networks. We also present an upper bound for this class where the constants involved are considerably smaller than in a previous calculation. Finally, we argue that the Vapnik-Chervonenkis dimension of nonoverlapping threshold or sigmoidal networks cannot become larger by allowing the nodes to compute linear functions. This sheds some light on a recent result that exhibited neural networks with quadratic Vapnik-Chervonenkis dimension.

On the piecewise analysis of networks of linear threshold neurons

The computational abilities of recurrent networks of neurons with a linear activation function above threshold are analyzed. These networks selectively realise a linear mapping of their input. Using this property, the dynamics as well as the number and the stability of stationary states can be investigated. The important property of the boundedness of neural activities can be guaranteed by global inhibition. If used together with self-excitation, the global inhibition gives rise to a multi stable winner-take-all (WTA) mechanism. A condition for a neuron to be a potential winner of the competing dynamics is derived. The network becomes a largest input selector when the self-excitation is marginal. Slowing down the global inhibition produces oscillations. The study of oscillations of random networks suggests that all cyclic trajectories of linear threshold networks are a result of the existence of partitions with undamped linear oscillations. Chaotic dynamics were never encountered in computer simulations and perhaps do not exist at all in small networks.

Capacity of multilevel threshold devices

The principal purpose of this research is to discover the underlying properties of linear multilevel threshold devices and networks of linear multilevel threshold devices. The main theoretical developments of this investigation entail generalizing the function-counting theorem associated with linear bi-level threshold devices to that of linear multilevel threshold devices. The investigation reveals surprising connections that underlie linear inequalities, linear separability, linear ordering, and region counting. The results have implications in the field of geometric probability. Computations based on theoretical analysis support Brown's (1964) conjecture and suggest that k/(k-1) is a natural definition for the information-storage capacity of a k-level threshold device. A lower bound on the number of weights required to implement a universal network of k-level threshold devices is derived. Finally, it is shown that the Vapnik-Chervonenki (1971)s dimension (VC-dimension) for the class of multilevel threshold function is d+1 for pattern vectors in /spl Rscr//sup d/. This VC-dimension is also linked to an error-rate bound for multilevel threshold functions within the framework of uniform learnability.

Impossibility results for weak threshold networks

It is shown that a weak threshold network (in particular, threshold network) of width w and depth d cannot be constructed from balancers of width p 0, p 1,…, p m − 1 , if w does not divide P d , where P is the least common multiple of p 0, p 1,…, p m − 1 . This holds regardless of the size of the network, as long as it is finite, and it implies a lower bound of log p w on its depth. More strongly, a lower bound of log p max w is shown on the length of every path from an input wire to any output wire that exhibits the threshold property, where p max is the maximum among p 0, p 1,…, p m − 1 .

Performance Bounds for Single Layer Threshold Networks when Tracking a Drifting Adversary

This paper finds upper bounds for the generalization error of three tracking algorithms when confronted with a worst case adversary. A system identification model is used where both the target and tracking network are single layer threshold networks, with the target weights changing slowly (the drift problem). Previous work considered random unbiased drifting adversaries. This paper focuses on the analysis of a worst case drifting adversary. For a small drift rate of γ, we find that upper bounds for the optimal conservative tracker, the perceptron tracker, and the least mean square (LMS) tracker are respectively 2γ/cos(γπ),2γn/πcos(γπ), and γ(2n + 2.5) where n is the number of inputs. Simulation results validate the analysis and also show that the bounds are tight when γ is small for the perceptron and LMS tracker. The effects of additive noise, correlated inputs and non-Gaussian inputs are also discussed. © 1997 Elsevier Science Ltd.

Aspects of systems and circuits for nanoelectronics

A large number of devices, a limitation of wiring, and very low power dissipation density are design constraints of future nanoelectronic circuits composed of quantum-effect devices. Furthermore, functional integration, which is the possibility of exploiting quantum effects to obtain a function specific behavior, becomes a core design principle. This paper analyzes the effect of this technological progress on the design of nanoelectronic circuits and describes computational paradigms revealing novel features such as distributed storage, fault tolerance, self-organization, and local processing. In particular, linear threshold networks, the associative matrix, self-organizing feature maps, and cellular arrays are investigated from the viewpoint of their potential significance for nanoelectronics. Although these concepts have already been implemented using present technologies, the intention of this paper is to give an impression of their usefulness to system implementations with quantum-effect devices.

Comparison of tracking algorithms for single layer threshold networks in the presence of random drift

This paper analyzes the behavior of a variety of tracking algorithms for single-layer threshold networks in the presence of random drift. We use a system identification model to model a target network where weights slowly change and a tracking network. Tracking algorithms are divided into conservative and nonconservative algorithms. For a random drift rate of /spl gamma/, we find upper bounds for the generalization error of conservative algorithms that are /spl Oscr/(/spl gamma//sup 2/3/) and for nonconservative algorithms that are /spl Oscr/(/spl gamma/). Bounds are found for the perceptron tracker and the least mean square (LMS) tracker. Simulations show the validity of these bounds and show that the bounds are tight when /spl gamma/ is small and the number of inputs n is large. These results show that the perceptron tracker and the LMS tracker can work well in slowly changing nonstationary environments.

Learning Temporally Encoded Patterns in Networks of SpikingNeurons

Networks of spiking neurons are very powerful and versatile models for biological and artificial information processing systems. Especially for modelling pattern analysis tasks in a biologically plausible way that require short response times with high precision they seem to be more appropriate than networks of threshold gates or models that encode analog values in average firing rates. We investigate the question how neurons can learn on the basis of time differences between firing times. In particular, we provide learning rules of the Hebbian type in terms of single spiking events of the pre- and postsynaptic neuron and show that the weights approach some value given by the difference between pre- and postsynaptic firing times with arbitrary high precision.

δ-Bit serial binary addition with linear threshold networks

In this paper we investigate δ-bit serial addition in the context of feed-forward linear threshold gate based networks. We show that twon-bit operands can be added in\(2\left\lceil {\sqrt n } \right\rceil \) overall delay with a feed-forward network constructed with\(\left\lceil {\sqrt n } \right\rceil + 1\) linear threshold gates and\(\frac{1}{2}\left( {5\left\lceil {\sqrt n } \right\rceil ^2 + 9\left\lceil {\sqrt n } \right\rceil } \right) + 2\) latches. The maximum weight value is\(2^{\left\lceil {\sqrt n } \right\rceil } \) and the maximum fan-in is\(3\left\lceil {\sqrt n } \right\rceil + 1\). We also investigate the implications our scheme have to the performance and the cost under small weights and small fan-in requirements. We deduce that if the weight values are to be limited by a constantW, twon-bit operands can be added in\(\left[ {\log W} \right] + \tfrac{n}{{\left[ {\log W} \right]}}\) overall delay with a feed-forward network that has the implementation cost [logW]+1, in terms of linear threshold gates,\(\tfrac{1}{2}(5[\log W]^2 + 9[\log W]) + 2\) in terms of latches and a maximum fan-in of 3[logW]+1. We also prove that, if the fan-in values are to be limited by a constantF+1, twon-bit operands can be added in\([\tfrac{F}{3}] + \tfrac{n}{{[\tfrac{F}{3}]}}\) overall delay with a feed-forward network that has the implementation cost\([\tfrac{F}{3}] + 1\), in terms of linear threshold gates,\(\tfrac{1}{2}(5[\tfrac{F}{3}]^2 + 9[\tfrac{F}{3}]) + 2\) in terms of latches, and a maximum weight value of\(2^{[\tfrac{F}{3}]} \). An asymptotic bound of\(O(\tfrac{n}{{\log n}})\) is derived for the addition overall delay in the case that the weight values have to be linearly bounded, i.e., in the order ofO(n). The implementation cost in this case is in the order ofO(logn), in terms of linear threshold gates, and in the order ofO(log2n), in terms of latches. The maximum fan-in is in the order ofO(logn). Finally, a partition technique, that substantially reduces the overall cost of the implementation for all the schemes in terms of delay, latches, weights, and fan-in with some few additional threshold gates, is also presented.

2-1 addition and related arithmetic operations with threshold logic

In this paper we investigate the reduction of the size for small depth feed-forward linear threshold networks performing binary addition and related functions. For n bit operands we propose a depth-3 O(n <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> /log n) asymptotic size network for the binary addition with O polynomially bounded weights. We propose also a depth-3 addition of optimal O(n) asymptotic sits network and a depth-2 comparison of O(√n) asymptotic size network, both with O(2 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">√n</sup> ) asymptotic size of weight values. For existing architectural formats we show that our schemes, with equal or smaller depth networks, substantially outperform existing schemes in terms of size and fan-in requirements and on occasions in weight requirements.

A global approach to the design of discrete-time cellular neural networks for associative memories

In this paper a global design method for associative memories using discrete-time cellular neural networks (DTCNNs) is presented. The proposed synthesis technique enables to realize associative memories with several advantageous features. First of all, grey-level as well as bipolar images can be stored. Moreover, the proposed approach generates networks with learning and forgetting capabilities. Finally, it is possible to design networks with any kind of predetermined interconnection structure. In particular, neighbourhoods without line crossings can be chosen, greatly simplifying the VLSI implementation of the designed DTCNNs. In the first part of this work a model of a multilevel threshold network is presented and a stability analysis is carried out using basic notions deriving from non-linear dynamical system theory. The synthesis procedure is then developed by means of a pseudoinversion technique, assuring learning and forgetting capabilities of the designed DTCNN. The use of a neighbourhood without line crossings is also discussed. Simulation results are reported to show the capability of the proposed approach.