Local Learning Algorithm Research Articles

Global and local neural network ensembles

Surprisingly simple local learning algorithms are known to outperform many other global non-linear machines. Unfortunately, these algorithms are computationally costly. A means of combining both learning approaches is proposed in and shown to enhance performance.

Learning algorithms based on linearization

The aim of this article is to investigate a mechanical description of learning. A framework for local and simple learning algorithms based on interpreting a neural network as a set of configuration constraints is proposed. For any architectural design and learning task, unsupervised and supervised algorithms can be derived, optionally using unconstrained and hidden neurons. Unlike algorithms based on the gradient in weight space, the proposed tangential correlation (TC) algorithms move along the gradient in state space. This results in optimal scaling properties and simple expressions for the weight updates. The number of synapses is much larger than the number of neurons. A constraint for neural states does not impose a unique constraint for synaptic weights. Which weights to assign credit to can be selected from a parametrization of all weight changes equivalently satisfying the state constraints. At the heart of the parametrization are minimal weight changes. Two supervised algorithms (differing by their parametrizations) operating on a three-layer perceptron are compared with standard backpropagation. The successful training of fixed points of recurrent networks is demonstrated. The unsupervised learning of oscillations with variable frequencies is performed on standard and more sophisticated recurrent networks. The results presented here can be useful both for the analysis and for the synthesis of learning algorithms.

Optical Learning Neural Network Using Photorefractive Waveguides

A model of an optical neural network with learning ability is proposed. We numerically evaluate the learning ability of the proposed network by using parameters determined by experiments. Adaptive connections between artificial neurons are implemented using photorefractive (PR) waveguides that can be optically modified by guided beams. The network consists of three layers and has bipolar weights within the limited range. The bipolar weight is encoded as the difference between optical power transmittances of signal beams in two channels of the PR waveguides. The adaptivity of the transmittance of PR waveguide is experimentally evaluated and is incorporated into the proposed network simulated in a computer. The proposed network is trained by a simplified local learning algorithm. Numerical results showed that the proposed three-layered network with six hidden neurons can solve the exclusive-or problem.

Local Algorithms for Pattern Recognition and Dependencies Estimation

In previous publications (Bottou and Vapnik 1992; Vapnik 1992) we described local learning algorithms, which result in performance improvements for real problems. We present here the theoretical framework on which these algorithms are based. First, we present a new statement of certain learning problems, namely the local risk minimization. We review the basic results of the uniform convergence theory of learning, and extend these results to local risk minimization. We also extend the structural risk minimization principle for both pattern recognition problems and regression problems. This extended induction principle is the basis for a new class of algorithms.

Directed drift: A new linear threshold algorithm for learning binary weights on-line

Learning real weights for a McCulloch-Pitts neuron is equivalent to linear programming and can hence be done in polynomial time. Efficient local learning algorithms such as Perceptron Learning further guarantee convergence in finite time. The problem becomes considerably harder, however, when it is sought to learn binary weights; this is equivalent to integer programming which is known to be NP-complete. A family of probabilistic algorithms which learn binary weights for a McCulloch-Pitts neuron with inputs constrained to be binary is proposed here, the target functions being majority functions of a set literals. These algorithms have low computational demands and are essentially local in character. Rapid (average-case) quadratic rates of convergence for the algorithm are predicted analytically and confirmed through computer simulations when the number of examples is within capacity. It is also shown that, for the functions under consideration, Preceptron Learning converges rapidly (but to an, in general, non-binary solution weight vector).

Associative memories with short-range, higher order couplings

A study of recurrent associative memories with exclusively short-range connections is presented. To increase the capacity, higher order couplings are used. We study capacity and pattern completion ability of networks consisting of units with binary (±1) output. Results show that perfect learning of random patterns is difficult for very short coupling ranges, and that the average expected capacities (allowing small errors) in these cases are much smaller than the theoretical maximum, 2 bits per coupling. However, it is also shown that by choosing ranges longer than certain limit sizes, depending on network size and order, we can come close to the theoretical capacity limit. We indicate that these limit sizes increase very slowly with net size. Thus, couplings to at least 28 and 36 neighbors suffice for second order networks with 400 and 90,000 units, respectively. From simulations it is found that even networks with coupling ranges below the limit size are able to complete input patterns with more than 10% errors. Especially remarkable is the ability to correct inputs with large local errors (part of the pattern is masked). We present a local learning algorithm for heteroassociation in recurrent networks without hidden units. The algorithm is used in a multinet system to improve pattern completion ability on correlated patterns.

Local Learning Algorithms

Very rarely are training data evenly distributed in the input space. Local learning algorithms attempt to locally adjust the capacity of the training system to the properties of the training set in each area of the input space. The family of local learning algorithms contains known methods, like the k-nearest neighbors method (kNN) or the radial basis function networks (RBF), as well as new algorithms. A single analysis models some aspects of these algorithms. In particular, it suggests that neither kNN or RBF, nor nonlocal classifiers, achieve the best compromise between locality and capacity. A careful control of these parameters in a simple local learning algorithm has provided a performance breakthrough for an optical character recognition problem. Both the error rate and the rejection performance have been significantly improved.

Local learning algorithm for optical neural networks

An anti-Hebbian local learning algorithm for two-layer optical neural networks is introduced. With this learning rule, the weight update for a certain connection depends only on the input and output of that connection and a global, scalar error signal. Therefore the backpropagation of error signals through the network, as required by the commonly used back error propagation algorithm, is avoided. It still guarantees, however, that the synaptic weights are updated in the error descent direction. With the apparent advantage of simpler optical implementation this learning rule is also shown by simulations to be computationally effective.

Finite-state neural networks. A step toward the simulation of very large systems

Neural networks composed of neurons withQ N states and synapses withQ ℐ states are studied analytically and numerically. Analytically it is shown that these finite-state networks are much more efficient at information storage than networks with continuous synapses. In order to take the utmost advantage of networks with finite-state elements, a multineuron and multisynapse coding scheme is introduced which allows the simulation of networks having 1.0×109 couplings at a speed of 7.1×109 coupling evaluations per second on asingle processor of the Cray-YMP. A local learning algorithm is also introduced which allows for the efficient training of large networks with finite-state elements.

A THREE-LAYER ADAPTIVE NETWORK FOR PATTERN DENSITY ESTIMATION AND CLASSIFICATION

Based on the assumption that most probability densities in real life can be approximated by a mixture of Gaussian densities, we propose here a three-layer adaptive network with each neuron in the lower hidden layer representing a Gaussian basis function (covariance matrix equal to [Formula: see text] where I is a unit matrix) to estimate various probability densities and serve as a Bayes classifier. The width of the basis function [Formula: see text] may be the same for all neurons in this layer or it may vary from one neuron to another. This paper investigates the effectiveness of the network for both cases and presents a localized learning algorithm to adjust the network parameters. The network was trained with artificial data derived from known mixtures of memoryless Gaussian sources as well as exponential and Gamma densities. The performance of the network as a pattern density estimator was measured in terms of the relative difference between the target probability density function (p.d.f.) which generates the training and testing data and the network output representing the estimation. Samples from two mixtures corresponding to two classes were used to test the network capability as a classifier by comparing its error rate against that of a Bayes classifier. Both one- and two-dimensional cases were explored. The successfulness of the network depended on how well the target p.d.f.’s were represented by the training samples, the number of hidden neurons employed in the network and how thoroughly the network was trained. It was also found that allowing each basis function to have an independent width had a predominant effect on the network performance.

Convergence of an iterative neural network learning algorithm for linearly dependent patterns

The authors show that a local iterative learning algorithm of Diederich and Opper (1987) converges for any set of patterns, including linearly dependent sets, and that the resulting coupling constants for a given set of patterns are the same as those given by the pseudoinverse rule for any maximal linearly independent subset of the given set. Thus, the matrix of couplings produced by the algorithm is the desired projection matrix onto the subspace spanned by the training set.

Training with noise and the storage of correlated patterns in a neural network model

Local iterative learning algorithms for the interactions between Ising spins in neural network models are discussed. They converge to solutions with basins of attraction whose shape is determined by the noise in the training data, provided such solutions exist. The training is applied both to the storage of random patterns and to a model for the storage of correlated words. The existence of correlations increases the storage capacity of a given network beyond that for random patterns. The model can be modified to store cycles of patterns and in particular is applied to the storage of continuous items of English text.

A Local Learning Algorithm for Dynamic Feedforward and Recurrent Networks

Abstract Most known learning algorithms for dynamic neural networks in non-stationary environments need global computations to perform credit assignment. These algorithms either are not local in time or not local in space. Those algorithms which are local in both time and space usually cannot deal sensibly with ‘hidden units’. In contrast, as far as we can judge, learning rules in biological systems with many ‘hidden units’ are local in both space and time. In this paper we propose a parallel on-line learning algorithms which performs local computations only, yet still is designed to deal with hidden units and with units whose past activations are ‘hidden in time’. The approach is inspired by Holland's idea of the bucket brigade for classifier systems, which is transformed to run on a neural network with fixed topology. The result is a feedforward or recurrent ‘neural’ dissipative system which is consuming ‘weight-substance’ and permanently trying to distribute this substance onto its connections in an appropriate way. Simple experiments demonstrating the feasibility of the algorithm are reported.

Content-addressability and learning in neural networks

The content-addressability of patterns stored in Ising-spin neural network models with symmetric interactions is studied. Numerical results from simulations on the ICL distributed array processor (DAP) involving systems with up to 2048 neurons are presented. Behaviour consistent with finite-size scaling, characteristic of a first-order phase transition, is shown to be exhibited by the basins of attraction of the stored patterns both in the case of the Hopfield model and for systems using a local iterative learning algorithm designed to optimise the basins of attraction. Estimates are obtained for the critical minimum overlaps which an input pattern must have with a stored pattern in order to successfully retrieve it.

The space of interactions in neural network models

The typical fraction of the space of interactions between each pair of N Ising spins which solve the problem of storing a given set of p random patterns as N-bit spin configurations is considered. The volume is calculated explicitly as a function of the storage ratio, alpha =p/N, of the value kappa (>0) of the product of the spin and the magnetic field at each site and of the magnetisation, m. Here m may vary between 0 (no correlation) and 1 (completely correlated). The capacity increases with the correlation between patterns from alpha =2 for correlated patterns with kappa =0 and tends to infinity as m tends to 1. The calculations use a saddle-point method and the order parameters at the saddle point are assumed to be replica symmetric. This solution is shown to be locally stable. A local iterative learning algorithm for updating the interactions is given which will converge to a solution of given kappa provided such solutions exist.