Learning Rule For Neural Networks Research Articles

MLMVN With Soft Margins Learning

In this paper, we consider a modified error-correction learning rule for the multilayer neural network with multivalued neurons (MLMVN). This modification is based on the soft margins technique, which leads to the minimization of the distance between a cluster center and the learning samples belonging to this cluster. MLMVN has a derivative-free learning algorithm based on the error-correction learning rule and demonstrate a higher functionality and better generalization capability than a number of other machine learning techniques. The discrete k-valued multivalued neuron activation function divides a complex plane into k equal sectors. For more efficient and reliable solving of classification problems it is possible to modify the MLMVN learning algorithm in such a way that learning samples belonging to different classes (clusters) will be located as close as possible to the bisector of a desired sector (the cluster center) and as far as possible from each other, respectively. Such a modification based on the soft margins learning technique is considered in this paper. This modified learning algorithm improves the generalization capability of MLMVN when solving classification problems.

Nonlinear behavior of memristive devices during tuning process and its impact on STDP learning rule in memristive neural networks

It is now widely accepted that memristive devices are promising candidates for the emulation of the behavior of biological synapses in neuromorphic systems. This is mainly because of the fact that like the strength of synapse, memristance of the memristive device can be tuned actively for example by the application of voltage or current. In addition, it is also possible to fabricate high density of memristive devices through the nano-crossbar structures. In this paper, we will show that there are some problems associated with memristive devices, which are playing the role of biological synapses. For example, we show that the variation rate of the memristance depends completely on the initial state of the device, and therefore, it can change significantly with time during the learning phase. This phenomenon can degrade the performance of learning methods like spike timing-dependent plasticity and cause the corresponding neuromorphic systems to become unstable. We also illustrate that using two serially connected memristive devices with different polarities as a synapse can somewhat fix the aforementioned problem.

Spine head calcium as a measure of summed postsynaptic activity for driving synaptic plasticity.

We use a computational model of a hippocampal CA1 pyramidal cell to demonstrate that spine head calcium provides an instantaneous readout at each synapse of the postsynaptic weighted sum of all presynaptic activity impinging on the cell. The form of the readout is equivalent to the functions of weighted, summed inputs used in neural network learning rules. Within a dendritic layer, peak spine head calcium levels are either a linear or sigmoidal function of the number of coactive synapses, with nonlinearity depending on the ability of voltage spread in the dendrites to reach calcium spike threshold. This is strongly controlled by the potassium A-type current, with calcium spikes and the consequent sigmoidal increase in peak spine head calcium present only when the A-channel density is low. Other membrane characteristics influence the gain of the relationship between peak calcium and the number of active synapses. In particular, increasing spine neck resistance increases the gain due to increased voltage responses to synaptic input in spine heads. Colocation of stimulated synapses on a single dendritic branch also increases the gain of the response. Input pathways cooperate: CA3 inputs to the proximal apical dendrites can strongly amplify peak calcium levels due to weak EC input to the distal dendrites, but not so strongly vice versa. CA3 inputs to the basal dendrites can boost calcium levels in the proximal apical dendrites, but the relative electrical compactness of the basal dendrites results in the reverse effect being less significant. These results give pointers as to how to better describe the contributions of pre- and postsynaptic activity in the learning "rules" that apply in these cells. The calcium signal is closer in form to the activity measures used in traditional neural network learning rules than to the spike times used in spike-timing-dependent plasticity.

Performance Tuning in Database Management System based on Analysis of Combination of Time and Cost Parameter through Neural Network Learning

Performance tuning in database management system means enhancing the performance of database, i.e. minimizing the response time at a very optimum cost. Query optimization is one of the important aspects of performance tuning. Lots of research work has been done in this field but it is still ongoing process. To achieve high performance at a very low cost identification of KPIs (Key performance indicators) is necessary, so that by altering these parameters dynamically minimum response time with optimum value can be achieved. This paper proposes how to filter cost and time parameters, to prioritize these parameters to get minimum response time. The approach proposes in this paper will be implemented by using neural network learning rules.

Application of Independent Component Analysis in Machine Fault Diagnosis

A novel fast algorithm for lndependent Component Analysis is introduced, which can be used for blind source separation and machine fault diagnosis feature extraction. It is shown how a neural network learning rule can be transformed into a fixed-point iteration, which provides an algorithm that is very simple, does not depend on any user-defined parameters, and is fast to converge to the most accurate solution allowed by the data. The purpose of this paper is to review the application of blind source separation in the machine fault diagnosis,including the following aspects: noise elimination and extraction of the weak signals,the separation of multi-fault sources,redundancy reduction,feature extraction and pattern classification based on independent component analysis. And its application in machine fault diagnosis is illustrated by the examples. In addition, some prospects about using blind source separation for machine fault diagnosis are discussed.

An application of Hebbian learning in the design process decision-making

The design of a product involves a process in which several different aspects are combined in order to obtain a final, suitable and optimum product. Designers must interact with different stakeholder groups, make decisions and complete the design process. In order to achieve this, different evaluation techniques are used. Depending on the chosen technique and on the field and environment in which each member of the design team was trained, each one of the members will consider one or several aspects of the design project but from a point of view or perspective in line with his/her particular professional background. As a result, all decisions which will affect the design process of the product are focused on these aspects and individual viewpoints. In this paper, an evaluation technique is proposed which allows one to take suitable decisions, taking into account all the factors and perspectives which affect the design process in the best way, searching for a balance among them in relation to the aims and interests of a specific design project. The development of this evaluation technique was inspired by the way in which neurons interact with one another in the brain and it has been based on the Hebbian learning rule for neural networks. Lastly, a real application of the proposed technique is presented to demonstrate its applicability in evaluating industrial designs.

Sensitivity-based adaptive learning rules for binary feedforward neural networks.

This paper proposes a set of adaptive learning rules for binary feedforward neural networks (BFNNs) by means of the sensitivity measure that is established to investigate the effect of a BFNN's weight variation on its output. The rules are based on three basic adaptive learning principles: the benefit principle, the minimal disturbance principle, and the burden-sharing principle. In order to follow the benefit principle and the minimal disturbance principle, a neuron selection rule and a weight adaptation rule are developed. Besides, a learning control rule is developed to follow the burden-sharing principle. The advantage of the rules is that they can effectively guide the BFNN's learning to conduct constructive adaptations and avoid destructive ones. With these rules, a sensitivity-based adaptive learning (SBALR) algorithm for BFNNs is presented. Experimental results on a number of benchmark data demonstrate that the SBALR algorithm has better learning performance than the Madaline rule II and backpropagation algorithms.

Contrast normalization contributes to a biologically-plausible model of receptive-field development in primary visual cortex (V1)

Neuronal populations in the primary visual cortex (V1) of mammals exhibit contrast normalization. Neurons that respond strongly to simple visual stimuli – such as sinusoidal gratings – respond less well to the same stimuli when they are presented as part of a more complex stimulus which also excites other, neighboring neurons. This phenomenon is generally attributed to generalized patterns of inhibitory connections between nearby V1 neurons. The Bienenstock, Cooper and Munro (BCM) rule is a neural network learning rule that, when trained on natural images, produces model neurons which, individually, have many tuning properties in common with real V1 neurons. However, when viewed as a population, a BCM network is very different from V1 – each member of the BCM population tends to respond to the same dominant features of visual input, producing an incomplete, highly redundant code for visual information. Here, we demonstrate that, by adding contrast normalization into the BCM rule, we arrive at a neurally-plausible Hebbian learning rule that can learn an efficient sparse, overcomplete representation that is a better model for stimulus selectivity in V1. This suggests that one role of contrast normalization in V1 is to guide the neonatal development of receptive fields, so that neurons respond to different features of visual input.

Stability Analysis and Effect of CES on ANN Based AGC for Frequency Excursion

This paper presents an application of layered Artificial Neural Network controller to study load frequency control problem in power system. The objective of control scheme guarantees that steady state error of frequencies and inadvertent interchange of tie-lines are maintained in a given tolerance limitation. The proposed controller has been designed for a two-area interconnected power system. Only one artificial neural network controller (ANN), which controls the inputs of each area in the power system together, is considered. In this study, back propagation-through time algorithm is used as neural network learning rule. The performance of the power system is simulated by using conventional integral controller and ANN controller, separately. For the first time comparative study has been carried out between SMES and CES unit, all of the areas are included with SMES and CES unit separately. By comparing the results for both cases, the performance of ANN controller with CES unit is found to be better than conventional controllers with SMES, CES and ANN with SMES.

Application of Partial Least Squares Neural Network in Thermal Error Modeling for CNC Machine Tool

This paper presents a partial least squares neural network modeling method for CNC machine tool thermal errors. This method uses the neural network learning rule to obtain the PLS parameters instead of the traditional linear method in partial least squares regression so as to overcome the multicollinearity and nonlinearity problem in thermal error modeling. The basic principle and architecture of PLSNN is described and the new method is applied on the thermal error modeling for a CNC turning center. After model validation with two groups of new testing data and performance comparison with other five different modeling methods, PLSNN performs better than the others with better robustness.

Comparative study among different neural net learning algorithms applied to rainfall time series

AbstractThe present article reports studies to identify a non‐linear methodology to forecast the time series of average summer‐monsoon rainfall over India. Three advanced backpropagation neural network learning rules namely, momentum learning, conjugate gradient descent (CGD) learning, and Levenberg–Marquardt (LM) learning, and a statistical methodology in the form of asymptotic regression are implemented for this purpose. Monsoon rainfall data pertaining to the years from 1871 to 1999 are explored. After a thorough skill comparison using statistical procedures the study reports the potential of CGD as a learning algorithm for the backpropagation neural network to predict the said time series. Copyright © 2008 Royal Meteorological Society

A HETEROSYNAPTIC LEARNING RULE FOR NEURAL NETWORKS

In this article we introduce a novel stochastic Hebb-like learning rule for neural networks that is neurobiologically motivated. This learning rule combines features of unsupervised (Hebbian) and supervised (reinforcement) learning and is stochastic with respect to the selection of the time points when a synapse is modified. Moreover, the learning rule does not only affect the synapse between pre- and postsynaptic neuron, which is called homosynaptic plasticity, but effects also further remote synapses of the pre- and postsynaptic neuron. This more complex form of synaptic plasticity has recently come under investigations in neurobiology and is called heterosynaptic plasticity. We demonstrate that this learning rule is useful in training neural networks by learning parity functions including the exclusive-or (XOR) mapping in a multilayer feed-forward network. We find, that our stochastic learning rule works well, even in the presence of noise. Importantly, the mean learning time increases with the number of patterns to be learned polynomially, indicating efficient learning.

MATHEMATICAL CONSTRAINTS FOR BUILDING LEARNING RULES IN THE PURKINJE CELL SYSTEM

We first present a method to mathematically build a learning rule for closed-loop neural networks. This rule is then applied to climbing fibers in the cerebellar cortex. Our analytical study is based on previous experimental non-analytical studies, which suggests that climbing fibers carry out an error signal to the brain. Thus, our goal is to find the class of functions for the activity propagated by climbing fibers, allowing the output of the Purkinje cell to converge towards a desired output. These functions must tend towards zero when the objective is reached. Our techniques are generalized to other network models.

Simultaneous Perturbation Learning Rule for Recurrent Neural Networks and Its FPGA Implementation

Recurrent neural networks have interesting properties and can handle dynamic information processing unlike ordinary feedforward neural networks. However, they are generally difficult to use because there is no convenient learning scheme. In this paper, a recursive learning scheme for recurrent neural networks using the simultaneous perturbation method is described. The detailed procedure of the scheme for recurrent neural networks is explained. Unlike ordinary correlation learning, this method is applicable to analog learning and the learning of oscillatory solutions of recurrent neural networks. Moreover, as a typical example of recurrent neural networks, we consider the hardware implementation of Hopfield neural networks using a field-programmable gate array (FPGA). The details of the implementation are described. Two examples of a Hopfield neural network system for analog and oscillatory targets are shown. These results show that the learning scheme proposed here is feasible.

A model of STDP based on spatially and temporally local information: Derivation and combination with gated decay

Temporal relationships between neuronal firing and plasticity have received significant attention in recent decades. Neurophysiological studies have shown the phenomenon of spike-timing-dependent plasticity (STDP). Various models were suggested to implement an STDP-like learning rule in artificial networks based on spiking neuronal representations. The rule presented here was developed under three constraints. First, it only depends on the information that is available at the synapse at the time of synaptic modification. Second, it naturally follows from neurophysiological and psychological research starting with Hebb's postulate [D. Hebb. (1949). The organization of behavior. Wiley, New York]. Third, it is simple, computationally cheap and its parameters are straightforward to determine. This rule is further extended by addition of four different types of gating derived from conventionally used types of gated decay in learning rules for continuous firing rate neural networks. The results show that the advantages of using these gatings are transferred to the new rule without sacrificing its dependency on spike-timing.

The application of NN technique to automatic generation control for the power system with three areas including smes units

AbstractThe study includes an application of layered neural network controller to study automatic generation control (AGC) problem of the power system, which contains superconducting magnetic energy storage (SMES) units. The effectiveness of SMES unit over frequency oscillations improvement against load perturbations in power system is well known. In addition, the proposed control scheme provides the steady state error of frequency and inadvertent interchange of tie‐lines to be maintained in steady state values. The power system considered has three areas two of which including steam turbines while the other containing a hydro turbine, and all of them contain SMES units, in addition. In the power system each area with a steam turbine contains the non‐linearity due to reheat effect of the steam turbine and all of the areas contain upper and lower constraints for generation rate. Only one neural network (NN) controller, which controls all the inputs of each area in the power system, is considered. In the NN controller, back propagation‐through‐time algorithm is used as neural network learning rule. The performance of the power system is simulated by using conventional integral controller and NN controller for the cases with or without SMES units in all areas, separately. By comparing the results for both cases, it can be seen that the performance of NN controller is better than conventional controllers.

Automatic Recognition of Woven Fabric Patterns by an Artificial Neural Network

A neural network and image processing technology are introduced for classifying woven fabric patterns. An autocorrelation function is used to determine one weave repeat of the fabric. The reflected fabric image is captured and digitized by the computer system. The learning vector quantization algorithm as a learning rule of the artificial neural network enables recognition of woven fabric types more effectively. The results demon strate that three fundamental weave types can be classified accurately, and structural parameters such as yarn spacing, its variance, and the ratio of warp spacing to weft spacing can also be obtained.

The ups and downs of Hebb synapses.

Abstract Modelers have come up with many different learning rules for neural networks. When a teacher specifies the correct output, rules work better than pure Hebb rules in which the changes in synapse strength depend on the correlation between pre and postsynaptic activities. But for unsupervised learning, Hebb rules can be very effective if they are combined with suitable normalization or unlearning terms to prevent the synapses growing without bound. Hebb rules that use rates of change of activity instead of activity itself are useful for discovering perceptual invariants and may also provide a way of implementing learning. It would be truly wonderful if randomly connected neural networks could turn themselves into useful computing devices by using some simple rule to modify the strengths of synapses. This was the hope that lay behind the original Hebb learning rule and it is the vision that has driven neural network modelers for half a century. Initially, researchers tried simulating various rules to see what would happen. After a decade or two of messing around, researchers realized that there was a much better way to explore the space of possible learning rules: First write down an objective function (a quantitative definition of how well the network is performing) and then use elementary calculus to derive a learning rule that will improve the objective function. For the last few decades, the big theoretical advances in learning rules for neural networks have been associated with new optimization methods and new ideas about what objective function should be optimized. If we think of a neural network as a device for converting input vectors into output vectors, it is obvious that one sensible objective is to minimize some measure of the difference between the output the network actually produces and the output it ought to produce. This approach led to effective error-driven learning rules such as the Widrow-Hoff rule (Widrow & Hoff, 1960) and the perceptron convergence procedure (Rosenblatt, 1961) and it was later generalized to multilayer networks by using backpropagation of the errors to get training signals for intermediate hidden layers (Rumelhart, Hinton, & Williams, 1986). Within the neural network community, the approach of using the product of pre and postsynaptic activities to drive learning was seen as inferior to error-- driven methods that use the product of the presynaptic activity and the postsynaptic activity derivative - the rate at which the objective function changes as the postsynaptic activity is changed. Even when the task was merely to associate random input vectors with random output vectors, it was shown that an rule worked much better than a Hebbian rule. Unfortunately, learning has some serious drawbacks. It requires a teacher to specify the right answer and it is hard to see how neurons could implement the backpropagation required by multilayer versions. It is possible to get the teaching signal from the data itself by trying to predict the next term in a temporal sequence (Elman, 1991) or by trying to reconstruct the input data at the output (Hinton, 1989) but it is also possible to use quite different objective functions for learning. Some of these alternative objective functions lead to learning rules that are far more Hebbian in flavour. A common objective in processing high-dimensional data is to reduce the dimensionality without losing the ability to reconstruct the raw data from the reduced representation. If we measure the accuracy of the reconstruction by the squared error, the optimal strategy is to extract the principal components - the dominant directions of variation in the data. Oja (1982) showed how to extract the first principal component using Hebbian learning to maximize the squared output of a neuron combined with normalization of the synapse strengths to prevent them growing without bound. …

Canadian psychology is basic, applied, and impacts the world: An editorial about the next four years.

Like any new editor, I have set some goals for Canadian Psychology/Psychologie canadienne (CP/PC). My first goal is to increase the international profile of our journal while maintaining its Canadian character. My second goal is to increase the profile of basic science within the journal without diminishing the focus on areas. The first step in achieving these goals involved making some changes to the structure of the editorial board. You will notice that on the masthead, in addition to the many distinguished Canadian researchers who make up our board, we have added a complement of international board members who are authorities in their discipline and who have already started to provide valuable input on important scientific matters. Their presence will increase the visibility of the journal around the world. Our editorial board may be more diverse than ever, as it includes distinguished academics representing both and basic areas. Their expertise span from biopsychology to psychology, from cognitive psychology to professional ethics and standards, and from perception and psychophysics to industrial psychology. Many of my colleagues have told me that they always considered CP/PC to be a clinical or applied journal. This is not so. We welcome important papers from any area of psychology, including, but not limited to, biopsychology, clinical, cognitive, educational, history of psychology, quantitative, social, and so on. We aim to publish papers that would be of interest to large cross sections of psychologists spanning across more than one area of specialty. Topics such as neuronal regeneration, ethics in psychology, and history of our field all fit this category. In an effort to demonstrate the broad mandate of Canadian Psychology/Psychologie canadienne, I have decided to publish a special section on D. 0. Hebb (this issue). I distinctly remember when I first came across Hebb's name. I was a high school senior in Thessaloniki, Greece; I was sitting on a park bench when I read in a newspaper that a psychologist from McGill University in Montreal had changed the way we think about the brain and behaviour. While attending McGill University, I was fortunate to study Introductory Psychology under Ronald Melzack, who offered a course that was largely focused on Hebb's ideas. Hebb, a basic scientist, had a great impact around the world and we are all proud that he has been one of the great minds of Canadian psychology. In this issue, world-class commentators re-evaluate Hebb's neural network learning rule in relation to all relevant new knowledge that has accumulated over the last several decades. We have also included a review of Hebb's classic text, The Organization of Behavior. Although the book was originally published in 1949, it was re-released in 2002 by Lawrence Erlbaum Associates. Richard Tees reviews the book from a contemporary perspective. Coincidentally, one of the papers that was accepted by the previous editor, Vic Catano, was related to Hebb's neural network learning rule. This paper by Lord, Hanges, and Godfrey complements the section on Hebb and, as such, I decided to include it in this issue. …

On a perceptron-type learning rule for higher order Hopfield neural networks including dilation and translation

In the following, we will introduce a new Perceptron-like learning rule to enhance the recall performance of higher order Hopfield neural networks without significant increase of their complexity. In detail, our approach will lead to a generalized Perceptron learning rule which generates higher order Hopfield neural networks with dilation and translation that perform perfectly on the training set in case that the latter fulfills the so-called conditionally strong Γ-separability condition. In this sense, our learning scheme satisfies a kind of optimality criterion which means that it finds appropriate network parameters in a finite number of learning cycles in case that a solution exists.