AdaTron Learning Research Articles

Non-monotonic convergence of online learning algorithms for perceptrons with noisy teacher

Learning curves of simple perceptron were derived here. The learning curve of the perceptron learning with noisy teacher was shown to be non-monotonic, which has never appeared even though the learning curves have been analyzed for half a century. In this paper, we showed how this phenomenon occurs by analyzing the asymptotic property of the perceptron learning using a method in systems science, that is, calculating the eigenvalues of the system matrix and the corresponding eigenvectors. We also analyzed the AdaTron learning and the Hebbian learning in the same way and found that the learning curve of the AdaTron learning is non-monotonic whereas that of the Hebbian learning is monotonic.

Thermodynamic efficiency of learning a rule in neural networks

Biological systems have to build models from their sensory input data that allow them to efficiently process previously unseen inputs. Here, we study a neural network learning a binary classification rule for these inputs from examples provided by a teacher. We analyse the ability of the network to apply the rule to new inputs, that is to generalise from past experience. Using stochastic thermodynamics, we show that the thermodynamic costs of the learning process provide an upper bound on the amount of information that the network is able to learn from its teacher for both batch and online learning. This allows us to introduce a thermodynamic efficiency of learning. We analytically compute the dynamics and the efficiency of a noisy neural network performing online learning in the thermodynamic limit. In particular, we analyse three popular learning algorithms, namely Hebbian, Perceptron and AdaTron learning. Our work extends the methods of stochastic thermodynamics to a new type of learning problem and might form a suitable basis for investigating the thermodynamics of decision-making.

Statistical-Mechanical Analysis of Semi-Supervised Learning and Its Optimal Scheduling

Semi-supervised learning is a paradigm that uses a large number of unlabeled data and a small number of labeled data. We analyze the dynamical behaviors of semi-supervised learning in the framework of on-line learning by the statistical-mechanical method. A student uses several correlated input vectors in each update. The student is given a desired output for only one input vector out of these correlated input vectors. In this model, we derive simultaneous differential equations with deterministic forms that describe the dynamical behaviors of order parameters using the self-averaging property in the thermodynamic limit. We treat three well-known learning rules, that is, the Hebbian, Perceptron, and AdaTron learning rules. As a result, it is shown that using unlabeled data is effective in the early stages for all three learning rules. In addition, we show that the three learning rules have qualitatively different dynamical behaviors. Furthermore, we propose a new algorithm that improves the generalization...

Statistical Mechanics of Adaptive Weight Perturbation Learning

Weight perturbation learning was proposed as a learning rule in which perturbation is added to the variable parameters of learning machines. The generalization performance of weight perturbation learning was analyzed by statistical mechanical methods and was found to have the same asymptotic generalization property as perceptron learning. In this paper we consider the difference between perceptron learning and AdaTron learning, both of which are well-known learning rules. By applying this difference to weight perturbation learning, we propose adaptive weight perturbation learning. The generalization performance of the proposed rule is analyzed by statistical mechanical methods, and it is shown that the proposed learning rule has an outstanding asymptotic property equivalent to that of AdaTron learning.

Statistical Mechanics of On-Line Learning Using Correlated Examples

We consider a model composed of nonlinear perceptrons and analytically investigate its generalization performance using correlated examples in the framework of on-line learning by a statistical mechanical method. In Hebbian and AdaTron learning, the larger the number of examples used in an update, the slower the learning. In contrast, Perceptron learning does not exhibit such behaviors, and the learning becomes fast in some time region.

A qualitative comparison of Artificial Neural Networks and Support Vector Machines in ECG arrhythmias classification

In this paper, a novel use of Kernel–Adatron (K–A) learning algorithm to aid SVM (Support Vector Machine) for ECG arrhythmias classification is proposed. The proposed pattern classifier is compared with MLP (multi-layered perceptron) using back propagation (BP) learning algorithm. The ECG signals taken from MIT-BIH arrhythmia database are used in training to classify 6 different arrhythmia, plus normal ECG. The MLP and SVM training and testing stages were carried out twice. They were first trained only with one ECG lead signal and then a second ECG lead signal was added to the training and testing datasets. The aim was to investigate its influence on training and testing performance (generalization ability) plus time of training for both classifiers. Implementation of these three criteria for evaluation of ECG signals classification will ease the problem of structural comparisons, which has not been given attention in previous research works. The results indicate that SVM in comparison to MLP is much faster in training stage and nearly seven times higher in performance, but MLP generalization ability in terms of mean square error is more than three times less. The proposed SVM method shows considerable improvement in comparison to recently reported results obtained by Osowski et al. (2008).

Analysis of ensemble learning using simple perceptrons based on online learning theory

Ensemble learning of K nonlinear perceptrons, which determine their outputs by sign functions, is discussed within the framework of online learning and statistical mechanics. One purpose of statistical learning theory is to theoretically obtain the generalization error. This paper shows that ensemble generalization error can be calculated by using two order parameters, that is, the similarity between a teacher and a student, and the similarity among students. The differential equations that describe the dynamical behaviors of these order parameters are derived in the case of general learning rules. The concrete forms of these differential equations are derived analytically in the cases of three well-known rules: Hebbian learning, perceptron learning, and AdaTron (adaptive perceptron) learning. Ensemble generalization errors of these three rules are calculated by using the results determined by solving their differential equations. As a result, these three rules show different characteristics in their affinity for ensemble learning, that is "maintaining variety among students." Results show that AdaTron learning is superior to the other two rules with respect to that affinity.

Analysis of Ensemble Learning Using Simple Perceptrons Based on Online Learning Theory

Ensemble learning of K simple perceptrons, which determine their outputs by sign functions, is discussed within the framework of online learning and statistical mechanics. Hebbian, perceptron and AdaTron learning show different characteristics in their affinity for ensemble learning, that is maintaining variety among students. Results show that AdaTron learning is superior to the other two rules.

On-line AdaTron learning of unlearnable rules

We study the on-line AdaTron learning of linearly non-separable rules by a simple perceptron. Training examples are provided by a perceptron with a non-monotonic transfer function which reduces to the usual monotonic relation in a certain limit. We find that, although the on-line AdaTron learning is a powerful algorithm for the learnable rule, it does not give the best possible generalization error for unlearnable problems. Optimization of the learning rate is shown to greatly improve the performance of the AdaTron algorithm, leading to the best possible generalization error for a wide range of the parameter which controls the shape of the transfer function.)

Generalization ability and information gain of clock-model perceptrons

We study the generalization abilityg Q ofQ-state Clock-model perceptrons for (i) Hebbian and for certain Non-Hebbian learning procedures, namely (ii) learning with maximal stability, (iii) zero stability and (iv) optimal generalization, for the case of random training sets. Among other results we find thatg Q behaves quite different in the Hebbian and in the Non-Hebbian cases in the limitQ→∞. E.g. in the Hebbian case for finite α,g Q vanishes always ∝1/Q, whereas in the Non-Hebbian cases considered,g Q converges forQ→∞ to a non-trivial continuous functiong ∞(α), which vanishes for α 2. This means that for (ii), (iii) and (iv), as a function of α atQ=∞, there is a 2nd-order phase transition from a non-generalizing phase for α≤2 to a generalizing phase for α>2. Different behaviour of the Hebbian and Non-Hebbian cases, respectively, is also observed for the information gain obtained through learning. For the particular case of AdaTron Learning, which is identical to case (ii), we find a geometrical formulation forg Q (α), which is applicable to more general models.