Simple Learning Rule Research Articles

A Reward-Modulated Hebbian Learning Rule Can Explain Experimentally Observed Network Reorganization in a Brain Control Task

It has recently been shown in a brain-computer interface experiment that motor cortical neurons change their tuning properties selectively to compensate for errors induced by displaced decoding parameters. In particular, it was shown that the three-dimensional tuning curves of neurons whose decoding parameters were reassigned changed more than those of neurons whose decoding parameters had not been reassigned. In this article, we propose a simple learning rule that can reproduce this effect. Our learning rule uses Hebbian weight updates driven by a global reward signal and neuronal noise. In contrast to most previously proposed learning rules, this approach does not require extrinsic information to separate noise from signal. The learning rule is able to optimize the performance of a model system within biologically realistic periods of time under high noise levels. Furthermore, when the model parameters are matched to data recorded during the brain-computer interface learning experiments described above, the model produces learning effects strikingly similar to those found in the experiments.

Learning to represent visual input

One of the central problems in computational neuroscience is to understand how the object-recognition pathway of the cortex learns a deep hierarchy of nonlinear feature detectors. Recent progress in machine learning shows that it is possible to learn deep hierarchies without requiring any labelled data. The feature detectors are learned one layer at a time and the goal of the learning procedure is to form a good generative model of images, not to predict the class of each image. The learning procedure only requires the pairwise correlations between the activations of neuron-like processing units in adjacent layers. The original version of the learning procedure is derived from a quadratic ‘energy’ function but it can be extended to allow third-order, multiplicative interactions in which neurons gate the pairwise interactions between other neurons. A technique for factoring the third-order interactions leads to a learning module that again has a simple learning rule based on pairwise correlations. This module looks remarkably like modules that have been proposed by both biologists trying to explain the responses of neurons and engineers trying to create systems that can recognize objects.

A physiologically-inspired model of numerical classification based on graded stimulus coding

In most natural decision contexts, the process of selecting among competing actions takes place in the presence of informative, but potentially ambiguous, stimuli. Decisions about magnitudes – quantities like time, length, and brightness that are linearly ordered – constitute an important subclass of such decisions. It has long been known that perceptual judgments about such quantities obey Weber's Law, wherein the just-noticeable difference in a magnitude is proportional to the magnitude itself. Current physiologically inspired models of numerical classification assume discriminations are made via a labeled line code of neurons selectively tuned for numerosity, a pattern observed in the firing rates of neurons in the ventral intraparietal area (VIP) of the macaque. By contrast, neurons in the contiguous lateral intraparietal area (LIP) signal numerosity in a graded fashion, suggesting the possibility that numerical classification could be achieved in the absence of neurons tuned for number. Here, we consider the performance of a decision model based on this analog coding scheme in a paradigmatic discrimination task – numerosity bisection. We demonstrate that a basic two-neuron classifier model, derived from experimentally measured monotonic responses of LIP neurons, is sufficient to reproduce the numerosity bisection behavior of monkeys, and that the threshold of the classifier can be set by reward maximization via a simple learning rule. In addition, our model predicts deviations from Weber Law scaling of choice behavior at high numerosity. Together, these results suggest both a generic neuronal framework for magnitude-based decisions and a role for reward contingency in the classification of such stimuli.

Towards human skill adaptive manual control

In processes with direct human control, the operator's machine controlling skills have a tremendous impact on the overall performance. This paper proposes a framework for human skill adaptive control (SAC) systems aiming at increasing the efficiency of human operated task execution in human-machine systems. In the framework the human operator's machine controlling skill level is evaluated continuously. Based on the skill level, the control interface is tuned to be optimal for the operator's current skill level. As the operator is learning, the control interface changes along. In particular, an algorithm for realising the SAC system is proposed. The first step in the algorithm is modelling the operator's dynamics by using the modified optimal control model (MOCM). A method using particle swarm optimisation (PSO) for obtaining the MOCM parameters from trial data is proposed. Based on the identified operator model, the optimal values for the control interface parameters are searched with optimisation. The current values of the parameters are then changed according to a simple learning rule. The usability of the method is tested in a preliminary experiment by using a simulator of a manually controlled trolley crane system.

Dynamics of market structure driven by the degree of consumer’s rationality

We study a simple model of market share dynamics with boundedly rational consumers and firms interacting with each other. As the number of consumers is large, we employ a statistical description to represent firms’ distribution of consumer share, which is characterized by a single parameter representing how rationally the mass of consumers pursue higher utility. As the boundedly rational firm does not know the shape of demand function it faces, it revises production and price so as to raise its profit with the aid of a simple reinforcement learning rule. Simulation results show that (1) three phases of market structure, i.e. the uniform share phase, the oligopolistic phase, and the monopolistic phase, appear depending upon how rational consumers are, and (2) in an oligopolistic phase, the market share distribution of firms follows Zipf’s law and the growth-rate distribution of firms follows Gibrat’s law, and (3) an oligopolistic phase is the best state of market in terms of consumers’ utility but brings the minimum profit to the firms because of severe competition based on the moderate rationality of consumers.

Learning to Throw Out the Rascals: Electoral Agency in the Lab

Models of electoral agency address the amount of discipline and selection that can be achieved by voters in elections. The models are demanding in terms of individual belief-formation and consistency of behavior. We investigate a baseline model of electoral agency in a controlled laboratory environment. This baseline model, although simple, form the core of more complex electoral agency models. Our design seeks to limit the behavioral impact of social preferences. We find little support for the baseline model in our data. However, simple (non-rational) learning rules explain behavioral patterns well. Simulations indicate that non-rational learning pushes behavior most forcefully towards equilibrium in situations that are favorable to bayesian updating.

Learning in Bayesian Games with Binary Actions

This paper considers a simple adaptive learning rule in Bayesian games with binary actions where players employ threshold strategies. Global convergence results are given for supermodular games and potential games. If there is a unique equilibrium, players' strategies converge almost surely to it. Even if there is not, in potential games and in the two-player case in supermodular games, any limit point of the learning process must be an equilibrium. In particular, if equilibria are isolated, the learning process converges to one of them almost surely.

Learning of Spatiotemporal Patterns in Ising-Spin Neural Networks: Analysis of Storage Capacity by Path Integral Methods

We encode periodic spatiotemporal patterns in Ising-spin neural networks, using the simple learning rule inspired by the spike-timing-dependent synaptic plasticity. It is then found that periodically oscillating spin neurons successfully reproduce phase differences of the encoded periodic patterns. The storage capacity of this associative memory neural network is enhanced with an adequate level of asymmetry in synapse connections. To understand the properties of these nonequilibrium retrieval states of the neural network, we carry out an analysis based on a path integral method. The relation of a dynamic crosstalk term to time-persistent oscillation of a correlation function well explains the enhancement of the storage capacity in spite of our approximation on nonpersistent terms. We investigate the accuracy of this approximation further by detailed comparison with numerical simulations.

Stimulus sampling as an exploration mechanism for fast reinforcement learning

Reinforcement learning in neural networks requires a mechanism for exploring new network states in response to a single, nonspecific reward signal. Existing models have introduced synaptic or neuronal noise to drive this exploration. However, those types of noise tend to almost average out-precluding or significantly hindering learning -when coding in neuronal populations or by mean firing rates is considered. Furthermore, careful tuning is required to find the elusive balance between the often conflicting demands of speed and reliability of learning. Here we show that there is in fact no need to rely on intrinsic noise. Instead, ongoing synaptic plasticity triggered by the naturally occurring online sampling of a stimulus out of an entire stimulus set produces enough fluctuations in the synaptic efficacies for successful learning. By combining stimulus sampling with reward attenuation, we demonstrate that a simple Hebbian-like learning rule yields the performance that is very close to that of primates on visuomotor association tasks. In contrast, learning rules based on intrinsic noise (node and weight perturbation) are markedly slower. Furthermore, the performance advantage of our approach persists for more complex tasks and network architectures. We suggest that stimulus sampling and reward attenuation are two key components of a framework by which any single-cell supervised learning rule can be converted into a reinforcement learning rule for networks without requiring any intrinsic noise source.

Correlated Connectivity and the Distribution of Firing Rates in the Neocortex

Two recent experimental observations pose a challenge to many cortical models. First, the activity in the auditory cortex is sparse, and firing rates can be described by a lognormal distribution. Second, the distribution of nonzero synaptic strengths between nearby cortical neurons can also be described by a lognormal distribution. Here we use a simple model of cortical activity to reconcile these observations. The model makes the experimentally testable prediction that synaptic efficacies onto a given cortical neuron are statistically correlated, i.e., it predicts that some neurons receive stronger synapses than other neurons. We propose a simple Hebb-like learning rule that gives rise to such correlations and yields both lognormal firing rates and synaptic efficacies. Our results represent a first step toward reconciling sparse activity and sparse connectivity in cortical networks.

LTD windows of the STDP learning rule and synaptic connections having a large transmission delay enable robust sequence learning amid background noise

Spike-timing-dependent synaptic plasticity (STDP) is a simple and effective learning rule for sequence learning. However, synapses being subject to STDP rules are readily influenced in noisy circumstances because synaptic conductances are modified by pre- and postsynaptic spikes elicited within a few tens of milliseconds, regardless of whether those spikes convey information or not. Noisy firing existing everywhere in the brain may induce irrelevant enhancement of synaptic connections through STDP rules and would result in uncertain memory encoding and obscure memory patterns. We will here show that the LTD windows of the STDP rules enable robust sequence learning amid background noise in cooperation with a large signal transmission delay between neurons and a theta rhythm, using a network model of the entorhinal cortex layer II with entorhinal-hippocampal loop connections. The important element of the present model for robust sequence learning amid background noise is the symmetric STDP rule having LTD windows on both sides of the LTP window, in addition to the loop connections having a large signal transmission delay and the theta rhythm pacing activities of stellate cells. Above all, the LTD window in the range of positive spike-timing is important to prevent influences of noise with the progress of sequence learning.

Encoding and Decoding Spikes for Dynamic Stimuli

Naturally occurring sensory stimuli are dynamic. In this letter, we consider how spiking neural populations might transmit information about continuous dynamic stimulus variables. The combination of simple encoders and temporal stimulus correlations leads to a code in which information is not readily available to downstream neurons. Here, we explore a complex encoder that is paired with a simple decoder that allows representation and manipulation of the dynamic information in neural systems. The encoder we present takes the form of a biologically plausible recurrent spiking neural network where the output population recodes its inputs to produce spikes that are independently decodeable. We show that this network can be learned in a supervised manner by a simple local learning rule.

Emergence of Information Transfer by Inductive Learning

We study a simple game theoretic model of information transfer which we consider to be a baseline model for capturing strategic aspects of epistemological questions. In particular, we focus on the question whether simple learning rules lead to an efficient transfer of information. We find that reinforcement learning, which is based exclusively on payoff experiences, is inadequate to generate efficient networks of information transfer. Fictitious play, the game theoretic counterpart to Carnapian inductive logic and a more sophisticated kind of learning, suffices to produce efficiency in information transfer.

The linearity of emergent spectro-temporal receptive fields in a model of auditory cortex

The responses of cortical neurons are often characterized by measuring their spectro-temporal receptive fields (STRFs). The STRF of a cell can be thought of as a representation of its stimulus ‘preference’ but it is also a filter or ‘kernel’ that represents the best linear prediction of the response of that cell to any stimulus. A range of in vivo STRFs with varying properties have been reported in various species, although none in humans. Using a computational model it has been shown that responses of ensembles of artificial STRFs, derived from limited sets of formative stimuli, preserve information about utterance class and prosody as well as the identity and sex of the speaker in a model speech classification system. In this work we help to put this idea on a biologically plausible footing by developing a simple model thalamo-cortical system built of conductance based neurons and synapses some of which exhibit spike-time-dependent plasticity. We show that the neurons in such a model when exposed to formative stimuli develop STRFs with varying temporal properties exhibiting a range of heterotopic integration. These model neurons also, in common with neurons measured in vivo, exhibit a wide range of non-linearities; this deviation from linearity can be exposed by characterizing the difference between the measured response of each neuron to a stimulus, and the response predicted by the STRF estimated for that neuron. The proposed model, with its simple architecture, learning rule, and modest number of neurons ( < 1000 ), is suitable for implementation in neuromorphic analogue VLSI hardware and hence could form the basis of a developmental, real time, neuromorphic sound classification system.

Dynamics of a market with heterogeneous learning agents

We study a simple market model of heterogeneous interacting agents. There are two types of agents, consumers and firms, who are boundedly rational in the sense that they only have partial information. The consumers have a given amount of income at each time period. They determine from which firm to purchase goods, and then spend all the income to purchase as much as they can. The amount of goods they can obtain depends not only on the price offered by the firm but also the amount produced by the firm and the number of customers of the firm. We employ a statistical description of consumers’ behavior. The Bolzmann distribution is used to represent firms’ share distribution of consumers, which is characterized by a parameter (called “temperature” in physical systems) describing how greedily the consumers pursue higher utility. The firm does not know the shape of demand function it faces, so it revises production and price so as to raise its profit with the aid of a simple reinforcement learning rule which is applied to the “one-armed bandit” problem. Numerical simulations show that there is an optimal greediness, which maximize the time average of consumers’ utility. In the vicinity of the optimal greediness, oligopoly emerges although its membership changes frequently. In an oligopolistic market, the market share distribution of firms follows Zipf’s law.

Simple learning rules to cope with changing environments

We consider an agent that must choose repeatedly among several actions. Each action has a certain probability of giving the agent an energy reward, and costs may be associated with switching between actions. The agent does not know which action has the highest reward probability, and the probabilities change randomly over time. We study two learning rules that have been widely used to model decision-making processes in animals-one deterministic and the other stochastic. In particular, we examine the influence of the rules' 'learning rate' on the agent's energy gain. We compare the performance of each rule with the best performance attainable when the agent has either full knowledge or no knowledge of the environment. Over relatively short periods of time, both rules are successful in enabling agents to exploit their environment. Moreover, under a range of effective learning rates, both rules are equivalent, and can be expressed by a third rule that requires the agent to select the action for which the current run of unsuccessful trials is shortest. However, the performance of both rules is relatively poor over longer periods of time, and under most circumstances no better than the performance an agent could achieve without knowledge of the environment. We propose a simple extension to the original rules that enables agents to learn about and effectively exploit a changing environment for an unlimited period of time.

Evolutionary models of color categorization based on discrimination

Specifying the factors that contribute to the universality of color categorization across individuals and cultures is a longstanding and still controversial issue in psychology, linguistics, and anthropology. This article approaches this issue through the simulated evolution of color lexicons. It is shown that the combination of a minimal perceptual psychology of discrimination, simple pragmatic constraints involving communication, and simple learning rules is enough to evolve color-naming systems. Implications of this result for psychological theories of color categorization and the evolution of color-naming systems in human societies are discussed.

Reinforcement Learning Through Modulation of Spike-Timing-Dependent Synaptic Plasticity

The persistent modification of synaptic efficacy as a function of the relative timing of pre- and postsynaptic spikes is a phenomenon known as spike-timing-dependent plasticity (STDP). Here we show that the modulation of STDP by a global reward signal leads to reinforcement learning. We first derive analytically learning rules involving reward-modulated spike-timing-dependent synaptic and intrinsic plasticity, by applying a reinforcement learning algorithm to the stochastic spike response model of spiking neurons. These rules have several features common to plasticity mechanisms experimentally found in the brain. We then demonstrate in simulations of networks of integrate-and-fire neurons the efficacy of two simple learning rules involving modulated STDP. One rule is a direct extension of the standard STDP model (modulated STDP), and the other one involves an eligibility trace stored at each synapse that keeps a decaying memory of the relationships between the recent pairs of pre- and postsynaptic spike pairs (modulated STDP with eligibility trace). This latter rule permits learning even if the reward signal is delayed. The proposed rules are able to solve the XOR problem with both rate coded and temporally coded input and to learn a target output firing-rate pattern. These learning rules are biologically plausible, may be used for training generic artificial spiking neural networks, regardless of the neural model used, and suggest the experimental investigation in animals of the existence of reward-modulated STDP.

Exploration or exploitation: life expectancy changes the value of learning in foraging strategies

The acquisition of information is a fundamental part of individual foraging behaviour in heterogeneous and changing environments. We examine how foragers may benefit from utilizing a simple learning rule to update estimates of temporal changes in resource levels. In the model, initial expectation of resource conditions and rate of replacing past information by new experiences are genetically inherited traits. Patch‐time allocation differs between learners and foragers that use a fixed patch‐leaving threshold throughout the foraging season. It also deviates from foragers that obtain information about the environment at no cost. At the start of a foraging season, learners sample the environment by frequent movements between patches, sacrificing current resource intake for information acquisition. This is done to obtain more precise and accurate estimates of resource levels, resulting in increased intake rates later in season. Risk of mortality may alter the trade‐off between exploration and exploitation and thus change patch sampling effort. As lifetime expectancy decreases, learners invest less in information acquisition and show lower foraging performance when resource level changes through time.

Global Nash convergence of Foster and Young's regret testing

We construct an uncoupled randomized strategy of repeated play such that, if every player plays according to it, mixed action profiles converge almost surely to a Nash equilibrium of the stage game. The strategy requires very little in terms of information about the game, as players' actions are based only on their own past payoffs. Moreover, in a variant of the procedure, players need not know that there are other players in the game and that payoffs are determined through other players' actions. The procedure works for finite generic N-player games and is based on appropriate modifications of a simple stochastic learning rule introduced by Foster and Young [Foster, D.P., Young, H.P., 2006. Regret testing: Learning to play Nash equilibrium without knowing you have an opponent. Theoretical Economics 1, 341–367].