Simple Perceptron Research Articles

Training a neuron in sequential learning

Among binary unit-based constructive algorithms, the Sequential Learning is particularly interesting for many reasons, the most significant one being its ability to treat real valued inputs without preprocessing. However, due to the construction process, the classical algorithms derived from the Perceptron cannot be used for learning each unit of the hidden layer. The BCP Max, recently proposed, appears as the first efficient heuristic algorithm to perform the particular neuron training in the Sequential Learning. But the BCP Max principles can be easily extended to the classical Perceptron derivatives usually used in constructive algorithms. In this paper, we show how to extend the Thermal Perceptron, the Pocket algorithm, the Ratchet and the simple Perceptron to train a neuron in the Sequential Learning. Finally all solutions are compared.

Replica Symmetry Breaking and the Kuhn-Tucker Cavity Method in Simple and Multilayer Perceptrons

Within a Kuhn-Tucker cavity method introduced in a former paper, we study optimal stability learning for situations, where in the replica formalism the replica symmetry may be broken, namely (i) the case of a simple perceptron above the critical loading, and (ii) the case of two-layer AND-perceptrons, if one learns with maximal stability. We find that the deviation of our cavity solution from the replica symmetric one in these cases is a clear indication of the necessity of replica symmetry breaking. In any case the cavity solution tends to underestimate the storage capabilities of the networks.

Nonlinear spectrum estimation using a new neural network with a level estimator

This paper proposes a new nonlinear signal processing by using a three-layered network which is trained with self-organized clustering and supervised learning. The network consists of three layers, i.e., a self-organized layer, an evaluation layer, and an output layer. Since the evaluation layer is designed as a simple perceptron network and the output layer is designed as the fixed weight linear nodes, the training complexity is the same as the self-organized clustering and a simple perceptron network. In other words, quite high speed training can be realized. Generally speaking, since the data range usually used in signal processing is arbitrarily large, the network output should also cover this range. However, it may be difficult for only one node in the network to output these data. Instead of this technique, if this dynamic range is covered by using several nodes, the complexity of each node is reduced and the associated range is also quite limited. This results in a higher performance of this network than the conventional ones. As one of the objectives, this paper introduces the spectrum envelope estimation of speech waveforms. It is shown that accurate spectrum envelopes can be obtained.

Dynamic network for identification of character locations using self‐organized clustering

AbstractThis paper proposes a new network that evaluates the locations of targets, i.e., target characters, and recognizes them in an image mixed with other objects, i.e., noncharacters. This network consists of a self‐organized clustering layer, a simple perceptron layer, and a controller. The purpose of the self‐organized clustering is to evaluate the stochastic characteristics of an input data set; the self‐organization is executed by calculating the distance between stochastic characteristics in this layer and the stochastic characteristics of an input vector. The simple perceptron layer determines the location of a target character and simultaneously recognizes its character by linearly dividing all outputs given from the self‐organized clustering layer. In the proposed system, the location identification and its recognition of a target character can be adaptively performed by using two small different compensative networks. The location identification and its recognition of real target characters, which are obtained in printed boards, are also shown in this paper.

Equivalence between learning in noisy perceptrons and tree committee machines.

We study learning from single presentation of examples (on-line learning) in single-layer perceptrons and tree committee machines (TCMs). Lower bounds for the perceptron generalization error as a function of the noise level \ensuremath{\epsilon} in the teacher output are calculated. We find that local learning in a TCM with K hidden units is simply related to learning in a simple perceptron with a corresponding noise level \ensuremath{\epsilon}(K). For a large number of examples and finite K the generalization error decays as ${\mathrm{\ensuremath{\alpha}}}_{\mathrm{CM}}^{\mathrm{\ensuremath{-}}1}$, where ${\mathrm{\ensuremath{\alpha}}}_{\mathrm{CM}}$ is the number of examples per adjustable weight in the TCM. We also show that on-line learning is possible even in the K\ensuremath{\rightarrow}\ensuremath{\infty} limit, but with the generalization error decaying as ${\mathrm{\ensuremath{\alpha}}}_{\mathrm{CM}}^{\mathrm{\ensuremath{-}}1/2}$. The simple Hebb rule can also be applied to the TCM, but now the error decays as ${\mathrm{\ensuremath{\alpha}}}_{\mathrm{CM}}^{\mathrm{\ensuremath{-}}1/2}$ for finite K and ${\mathrm{\ensuremath{\alpha}}}_{\mathrm{CM}}^{\mathrm{\ensuremath{-}}1/4}$ for K\ensuremath{\rightarrow}\ensuremath{\infty}. Exponential decay of the generalization error in both the noisy perceptron learning and in the TCM is obtained by using the learning by queries strategy. \textcopyright{} 1996 The American Physical Society.

Mean field approach to Bayes learning in feed-forward neural networks.

We propose an algorithm to realize Bayes optimal predictions for feed-forward networks which is based on the Thouless-Anderson-Palmer mean field method developed for the statistical mechanics of disordered systems. We conjecture that our approach will be exact in the thermodynamic limit. The algorithm results in a simple built-in leave-one-out cross validation of the predictions. Simulations for the case of the simple perceptron and the committee machine are in excellent agreement with the results of replica theory.

Learning dynamics of simple perceptrons with non-extensive cost functions.

A Tsallis-statistics-based generalization of the gradient descent dynamics (using non- extensive cost functions), recently introduced by one of us, is proposed as a learning rule in a simple perceptron. The resulting Langevin equations are solved numerically for different values of an index q (q = 1 and q ≠ 1 respectively correspond to the extensive and non-extensive cases) and for different cost functions. The results are compared with the learning curve (mean error versus time) obtained from a learning experiment carried out with human beings, showing an excellent agreement for values of q slightly above unity. This fact illustrates the possible importance of including some degree of non-locality (non-extensivity) in computational learning procedures, whenever one wants to mimic human behaviour.

A Perceptron Reveals the Face of Sex

Recognizing the sex of conspecifics is important. Humans rely primarily on visual pattern recognition for this task. A wide variety of linear and nonlinear models have been developed to understand this task of sex recognition from human faces.' These models have used both pixelbased and feature-based representations of the face for input. Fleming and Cottrell (1990) and Golomb et a!. (1991) utilized first an autoencoder compression network on a pixel-based representation, and then a classification network. Brunelli and Poggio (1993) used a type of radial basis function network with geometrical face measurements as input. O'Toole and colleagues (1991, 1993) represented faces as principal components. When the hidden units of an autoencoder have a linear output function, then the N hidden units in the network span the first N principal components of the input (Baldi and Hornik 1989). Bruce et al. (1993) constructed a discriminant function for sex with 2-D and 3-D facial measures. In this note we compare the performance of a simple perceptron and a standard multilayer perceptron (MLP) on the sex classification task. We used a range of spatial resolutions of the face to determine how the reliability of sex discrimination is related to resolution. A normalized pixel-based representation was used for the faces because it explicitly retained texture and shape information while also maintaining geometric relationships. We found that the linear perceptron model can classify sex from facial images with 81% accuracy, compared to 92% accuracy with compression coding on the same data set (Golomb et al. 1991). The advantage of using a simple linear perceptron with normalized pixelbased inputs is that it allows us to see explicitly those regions of the face

Neural Speed Control for Autonomous Road Vehicles

In this paper a neural network automotive speed controller is presented which works at low and at high speed-levels using throttle and brake control input. It can be used for autonomous intelligent cruise control including stop & go traffic situations. The network itself consists of a simple multilayer feedforward perceptron network. A special training method is used where the neural network is trained on a detailed nonlinear dynamical vehicle model. Practical roadtests with the Daimler Benz experimental vehicle OSCAR (MB 300 TE station wagon) show good results for both high and low speed control.

Learning and generalization in multineuron interacting feed-forward neural networks

We consider learning and generalization in the multi-interacting feed-forward network model recently proposed by H O Carmesin (1994). With an a priori definition of the net architecture, based on symmetries presented by the function to be learnt, we define a generalized Hebb rule, extend the maximum stability learning algorithm to multi-interactions, and obtain training and generalization curves. For rules where different orders of synapses are not correlated the results obtained for the simple perceptron concerning the Hebb rule and through replica calculations in the space of couplings may be straightforwardly adapted to multi-interactions through a simple renormalization of the total number of independent couplings. Analytical and numerical simulation results are compared and show excellent agreement.

On-line learning in the committee machine

The dynamics of learning from examples in the K=3 non-overlapping committee machine with single presentation of examples is studied. The optimal algorithm, in the sense of mean generalization, is obtained from a variational analysis of the differential equations which describe the dynamics. The agreement of the theoretical predictions and the results of numerical simulations is excellent. The optimized dynamics has the extra advantage with respect to the non-optimized cases in that it uncouples the differential equations which describe the evolution of the relevant parameters, i.e. the student-teacher overlap and the norm of the student synaptic vector. This, in turn, translates into the possibility of constructing useful practical optimized on-line algorithms. For the optimal algorithm the generalization error decays as approximately 0.88 alpha -1, the same nominal error as for the simple perceptron with optimized dynamics.

Learning to classify in large committee machines.

The ability of a two-layer neural network to learn a specific non-linearly-separable classification task, the proximity problem, is investigated using a statistical mechanics approach. Both the tree and fully connected architectures are investigated in the limit where the number K of hidden units is large, but still much smaller than the number N of inputs. Both have continuous weights. Within the replica symmetric ansatz, we find that for zero temperature training, the tree architecture exhibits a strong overtraining effect. For nonzero temperature the asymptotic error is lowered, but it is still higher than the corresponding value for the simple perceptron. The fully connected architecture is considered for two regimes. First, for a finite number of examples we find a symmetry among the hidden units as each performs equally well. The asymptotic generalization error is finite, and minimal for T\ensuremath{\rightarrow}\ensuremath{\infty} where it goes to the same value as for the simple perceptron. For a large number of examples we find a continuous transition to a phase with broken hidden-unit symmetry, which has an asymptotic generalization error equal to zero.

Fault tolerance in simple perceptrons

The tolerance of a simple perceptron to faults in the neurons in the input layer is studied in terms of the dependence of the maximum storage capacity of the perceptron on the fraction of faulty neurons. Evaluation of the capacity using the space of interactions method in the limit when the number of input neurons is very large shows that the fault tolerance of simple perceptrons can be substantial.

Storage capacity of a two-layer perceptron with fixed preprocessing in the first layer

It is well known that one can store up to 2N uncorrelated patterns in a simple perceptron with N input neurons. Changing the architecture of the net by adding a hidden layer may enhance the storage capacity. Learning in multilayered networks, however, is difficult and slow compared to perceptron learning. In this work a different approach is taken. A large hidden layer with N' neurons is used onto which the patterns are mapped according to a one-to-one code which is fixed beforehand. Only the connections from the hidden layer to the output unit are modified by learning. Here we show how to treat the correlations which are introduced large. Moreover, our results shed new light on the optimal capacity problem for single-layer perceptrons. We find the optimal capacity to be determined by the dimension of the space spanned by the input patterns, rather than by the size of the input layer.

An Off-Line Signature Recognition Using a Fuzzy Net.

This paper presents a method of off-line signature recognition using feature strokes and a fuzzy net. For getting feature strokes a pre-processing of signature data is studied. The pre-processing consists of a thinning method and a method of extracting feature strokes from an original signature and corresponding strokes from new signatures. The structure of fuzzy net is a simple perceptron and each unit in the input layer has a fuzzy membership function. From the connection weights of the trained network, it is easy to know which features of input data are important for signature recognition, i.e. the personal characteristics of the signatures are known from the fuzzy net. An experiment is done to show the feasibility of the new method.

The number of metastable states of a simple perceptron with gradient descent learning algorithm

The number of metastable states of a simple perceptron with gradient descent learning algorithm has been calculated as a function of the storage capacity alpha and the gain parameter beta . For alpha not=0 and beta >0 exponential large number of local minima, similar to the spin glasses and analogue attractor networks, has been found.

Storage of sets of correlated data in neural network memories

In most applications of neural network models one has to consider sets of correlated data. The authors study the problem of the storage of associated patterns in neural network memories. They consider two basic types of correlation: 'semantic' ones-among the different patterns-and 'spatial' ones-among the different sites of the network. They apply Gardner's program to evaluate optimal storage conditions for both kinds of correlation. The 'spatial' ones worsen, generally speaking, the storage properties of a simple perceptron, while they improve them for the Hopfield network. In the case of 'semantic' correlations they obtain bounds of the critical capacity for both kinds of networks; the storage ratio of the perceptron may be significantly increased in this case.

A neural network classifier in experimental particle physics.

A classification problem in high energy physics has been solved on simulated data using a simple multilayer perceptron comprising binary units which was trained with the CHIR algorithm. The unstable training of such a network on a nonseparable set has been overcome by selecting those weight vectors with good performance while providing a flexible choice of the two types of classification errors. Specific features of the problem have been exploited in order to simplify and optimize the solution which has been compared to the popular backpropagation algorithm and found to perform on a similar level. Additional aspects of this work are the use of the CHIR algorithm on continuous input and incorporating the classic idea of a phi-machine in a multilayer perceptron.

Generalization in a two-layer neural network.

Statistical mechanics is applied to study the generalization properties of a two-layer neural network trained to implement a linearly separable problem. For a stochastic learning algorithm the generalization error as a function of the training set size is calculated exactly. The network with three hidden units experiences two first-order phase transitions due to an asymmetric freezing of the hidden units. Compared to a simple perceptron the committee machine is found to generalize worse

Information capacity of a perceptron

The authors study the information storage capacity of a simple perceptron in the error regime. For random unbiased patterns the geometrical analysis gives a logarithmic dependence for the information content in the asymptotic limit. In this case, the statistical physics approach, when used at the simplest level of replica theory, does not give satisfactory results. However for perceptrons with finite stability, the information content can be simply calculated with statistical physics methods in a region above the critical storage level, for biased as well as for unbiased patterns.