Herrnstein's Matching Law Research Articles

Statistical Mechanics of Reward-Modulated Learning in Decision-Making Networks

The neural substrates of decision making have been intensively studied using experimental and computational approaches. Alternative-choice tasks accompanying reinforcement have often been employed in investigations into decision making. Choice behavior has been empirically found in many experiments to follow Herrnstein's matching law. A number of theoretical studies have been done on explaining the mechanisms responsible for matching behavior. Various learning rules have been proved in these studies to achieve matching behavior as a steady state of learning processes. The models in the studies have consisted of a few parameters. However, a large number of neurons and synapses are expected to participate in decision making in the brain. We investigated learning behavior in simple but large-scale decision-making networks. We considered the covariance learning rule, which has been demonstrated to achieve matching behavior as a steady state (Loewenstein & Seung, 2006 ). We analyzed model behavior in a thermodynamic limit where the number of plastic synapses went to infinity. By means of techniques of the statistical mechanics, we can derive deterministic differential equations in this limit for the order parameters, which allow an exact calculation of the evolution of choice behavior. As a result, we found that matching behavior cannot be a steady state of learning when the fluctuations in input from individual sensory neurons are so large that they affect the net input to value-encoding neurons. This situation naturally arises when the synaptic strength is sufficiently strong and the excitatory input and the inhibitory input to the value-encoding neurons are balanced. The deviation from matching behavior is caused by increasing variance in the input potential due to the diffusion of synaptic efficacies. This effect causes an undermatching phenomenon, which has been often observed in behavioral experiments.

Operant Matching as a Nash Equilibrium of an Intertemporal Game

Over the past several decades, economists, psychologists, and neuroscientists have conducted experiments in which a subject, human or animal, repeatedly chooses between alternative actions and is rewarded based on choice history. While individual choices are unpredictable, aggregate behavior typically follows Herrnstein's matching law: the average reward per choice is equal for all chosen alternatives. In general, matching behavior does not maximize the overall reward delivered to the subject, and therefore matching appears inconsistent with the principle of utility maximization. Here we show that matching can be made consistent with maximization by regarding the choices of a single subject as being made by a sequence of multiple selves-one for each instant of time. If each self is blind to the state of the world and discounts future rewards completely, then the resulting game has at least one Nash equilibrium that satisfies both Herrnstein's matching law and the unpredictability of individual choices. This equilibrium is, in general, Pareto suboptimal, and can be understood as a mutual defection of the multiple selves in an intertemporal prisoner's dilemma. The mathematical assumptions about the multiple selves should not be interpreted literally as psychological assumptions. Human and animals do remember past choices and care about future rewards. However, they may be unable to comprehend or take into account the relationship between past and future. This can be made more explicit when a mechanism that converges on the equilibrium, such as reinforcement learning, is considered. Using specific examples, we show that there exist behaviors that satisfy the matching law but are not Nash equilibria. We expect that these behaviors will not be observed experimentally in animals and humans. If this is the case, the Nash equilibrium formulation can be regarded as a refinement of Herrnstein's matching law.

Robustness of Learning That Is Based on Covariance-Driven Synaptic Plasticity

It is widely believed that learning is due, at least in part, to long-lasting modifications of the strengths of synapses in the brain. Theoretical studies have shown that a family of synaptic plasticity rules, in which synaptic changes are driven by covariance, is particularly useful for many forms of learning, including associative memory, gradient estimation, and operant conditioning. Covariance-based plasticity is inherently sensitive. Even a slight mistuning of the parameters of a covariance-based plasticity rule is likely to result in substantial changes in synaptic efficacies. Therefore, the biological relevance of covariance-based plasticity models is questionable. Here, we study the effects of mistuning parameters of the plasticity rule in a decision making model in which synaptic plasticity is driven by the covariance of reward and neural activity. An exact covariance plasticity rule yields Herrnstein's matching law. We show that although the effect of slight mistuning of the plasticity rule on the synaptic efficacies is large, the behavioral effect is small. Thus, matching behavior is robust to mistuning of the parameters of the covariance-based plasticity rule. Furthermore, the mistuned covariance rule results in undermatching, which is consistent with experimentally observed behavior. These results substantiate the hypothesis that approximate covariance-based synaptic plasticity underlies operant conditioning. However, we show that the mistuning of the mean subtraction makes behavior sensitive to the mistuning of the properties of the decision making network. Thus, there is a tradeoff between the robustness of matching behavior to changes in the plasticity rule and its robustness to changes in the properties of the decision making network.

Operant matching is a generic outcome of synaptic plasticity based on the covariance between reward and neural activity

The probability of choosing an alternative in a long sequence of repeated choices is proportional to the total reward derived from that alternative, a phenomenon known as Herrnstein's matching law. This behavior is remarkably conserved across species and experimental conditions, but its underlying neural mechanisms still are unknown. Here, we propose a neural explanation of this empirical law of behavior. We hypothesize that there are forms of synaptic plasticity driven by the covariance between reward and neural activity and prove mathematically that matching is a generic outcome of such plasticity. Two hypothetical types of synaptic plasticity, embedded in decision-making neural network models, are shown to yield matching behavior in numerical simulations, in accord with our general theorem. We show how this class of models can be tested experimentally by making reward not only contingent on the choices of the subject but also directly contingent on fluctuations in neural activity. Maximization is shown to be a generic outcome of synaptic plasticity driven by the sum of the covariances between reward and all past neural activities.

The living legacy of the Harvard Pigeon Lab: quantitative analysis in the wide world.

From the Harvard Pigeon Lab of the 1960s arose a behavior-analytic approach that was quantitative and rigorous, rooted in Herrnstein's matching law. Researchers modified the matching law to describe choice behavior in a variety of different settings and examined its relations with other quantitative models. Beginning in the early 1970s, researchers began using the Harvard Pigeon Lab's quantitative framework to study in the laboratory specific aspects of the world outside the laboratory. Much of this work concerned investigations of self-control-choice of a larger, more delayed reinforcer over a smaller, less delayed reinforcer. Experiments using a quantitative framework derived from the matching law have also been conducted outside the laboratory; however, these have been far less frequent. Current and future researchers will benefit the field by devising new, creative ways to investigate the matching law and related quantitative models outside the laboratory. Such research can help to demonstrate the validity of these models as basic principles of behavior, can enhance public opinion of and rewards for such research, and can stimulate further development of the Harvard Pigeon Lab's quantitative approach by using that approach with new variables.

One Hundred Years of Laws in Psychology

Mainstream psychology in the 20th century has been conceived as a nomothetic science, but few psychological "laws" have been proposed. A PsycLit search of journal abstracts from 1900 to 1999 yielded a total of 3,093 "law" citations, or 22 per 10,000 entries, with two psychophysical laws (Weber's law and Stevens's power law) and two learning laws (Herrnstein's matching law and Thorndike's law of effect) as the most frequently cited. The number of law citations has been decreasing throughout the century, to 10 per 10,000 entries in the last decade, with few references to laws of recent origin. This could be the result of increasing doubts about the lawfulness of psychological processes coupled with a general preference for less ambitious terms (such as effects, principles, models, or functions).

Coregulation of Balance Between Children's Prosocial Approaches and Acts of Compliance: A Pathway to Mother-Child Cooperation?

Examined the free-field interaction of 32 mother-child dyads who volunteered to participate in a 1-hr home observation. Observers coded mother instructions, child compliance, and child prosocial approaches plus mothers' social attention as potential reinforcers for the children's compliance and social approaches. Herrnstein's matching law was used to analyze covariations between mothers' attention and the children's 2 responses. This analysis was followed by correlational and sequential probability analyses to determine linkages between these 2 child responses and the children's willingness to obey their mothers' instructions. Results showed consistent matching between mothers' social attention and the children's production of prosocial approaches and acts of compliance. An index of the proportions of these 2 responses also covaried with the children's compliance probabilities, and the prosocial approach component was the direct covariate. These findings are discussed within an interactional synchrony framework in which children's willingness to obey their mothers is influenced by opportunities for the dyad to engage each other in specific forms of social interaction.

Multiple variable interval schedule behaviour in humans: effects of ethanol, mood, and reinforcer size on responding maintained by monetary reinforcement.

Ethanol is an effective reinforcer but, in common with other drugs of abuse, it may derive some of its reinforcing properties from the effects it has on behaviour maintained by other reinforcers. However, any assessment of ethanol's hypothesized effect on behaviour maintained by other reinforcers must take into account the fact that ethanol may have multiple mechanisms of action. In order to address this problem the experiments reported herein used a procedure based upon Herrnstein's Matching Law which allowed joint assessment of subjects' motor capacity and reinforcer sensitivity. The effect of ethanol (0, 0.3, and 0.6 g/kg) on motor capacity and reinforcer sensitivity was assessed by studying behaviour maintained by monetary reinforcement. In the first experiment the procedure was validated by showing that the behaviour of subjects was sensitive to changes in reinforcer value and in the second experiment 0.6 g/kg ethanol reduced motor capacity but did not affect reinforcer sensitivity. As a secondary hypothesis we also studied the effect of mood on reinforcer sensitivity and motor capacity. It was found that lower mood scores (lower hedonic tone) were associated with reduced reinforcer sensitivity and that male subjects showed higher motor capacity than females. However, there was also a mood by sex interaction, which indicated that higher motor capacity in males was only found in the presence of lower mood scores. The results are discussed in relation to the mechanisms of ethanol's dopaminergic effects, interactions between ethanol and other drugs of abuse, and the changes in reinforcer sensitivity which are thought to occur in depression.

Predicting teenage girls' sexual activity and contraception use: An application of Matching Law

The high incidence of teenage sexual activity and poor contraceptive practices is a significant social problem. Previous attempts to explicate the variables associated with high-risk sexual behavior in teens have uncovered a multitude of associated factors. However, this research has not led to a comprehensive understanding of the phenomenon, nor has it led to the development of effective interventions to remediate this problem. The application of Herrnstein's Matching Law was proposed as an effective means of explaining, in simple terms, the reasons that some youths demonstrate a greater predilection for high-risk sexual behavior. Ethnically diverse high school students from low and middle socioeconomic classes were sampled. Results of this study indicated that Matching Law could be used to predict high-risk sexual behavior with a good deal of accuracy across different age, ethnic, and socioeconomic groups. © 1998 John Wiley & Sons, Inc.

Applying Herrnstein's Matching Law to Influence Students' Choice to Complete Difficult Academic Tasks

Abstract In an extension of choice research and Herrnstein's matching law to a more ecologically valid setting, college students were asked to work on a control mathematics assignment containing 16 three-digit by two-digit multiplication (3 × 2) problems and an experimental assignment that contained 16 equivalent 3 × 2 problems plus six interspersed 1 × 1 problems. Afterwards, they rated the experimental assignment as less difficult and time-consuming and requiring less effort than the control assignment. Performance on 3 × 2 problems was equivalent across the control and experimental assignments. Furthermore, when given the opportunity to choose a new control-format or experimental-format assignment (each containing similar, but different, problems as the initial assignments), significantly more students chose the latter format Because these results could be explained by the novelty of the problems interspersed in the experimental assignment, another experiment that interspersed division problems was con...

Response-reinforcement relationships in chronic pain syndrome: Applicability of Herrnstein's law

The quantitative relationship between chronic pain behavior and reinforcement has been generally found to be positive. Herrnstein's statement of the law of effect adds a qualification: it predicts that response rate varies hyperbolically with contingent reinforcement, and that response rate is also negatively related to extraneous reinforcement. The present study is an original investigation of the first of these assertions with regard to the naturally occurring behaviors in chronic pain syndrome. It was found that the stereotypic pain behaviors of 12 chronic pain patients were hyperbolically rather than proportionally related to contingent reinforcement from their significant others. Likewise, healthy behaviors in the same group were a hyperbolic function of contingent reinforcement. Estimates of extraneous reinforcement were also obtained for each category of behaviors. As a group, the subjects showed an inverse relationship between pain behaviors and healthy behaviors, consistent with previous research in the pain area. The present findings add generality to Herrnstein's matching law which serves as a descriptive model of target behaviors in chronic pain patients. The findings also have prescriptive implications for the modification of behavior among chronic pain sufferers.

Time-, schedule-, and reinforcer-dependent effects of pimozide and amphetamine

Rats performed on two multiple random-interval schedules, in which sequences of ascending or descending reinforcement densities were balanced between the schedules and between the two halves of the session. Using a standard reinforcer (10% sucrose pellets), pimozide decreased response rates, while amphetamine increased responding. The effects of both drugs were schedule dependent: larger changes were evident in low response rate, reinforcement-lean components than in high response rate, reinforcement-rich components. Both effects were also time dependent, increasing over the course of the session; this casts serious doubt on the applicability of Herrnstein's matching law for studying agents acting on brain dopamine. Increasing the period of food deprivation increased response rates, while withdrawing food deprivation decreased responding. These effects were also schedule dependent, but were time dependent. Substituting 95% sucrose pellets for standard 10% sucrose pellets caused an immediate and sustained decrease in responding, and up to 10% of earned reinforcement was not consumed. Pimozide increased response rates within reinforcement-lean components and reinstated the complete consumption of earned reward typical of standard reinforcement. These apparently paradoxical effects may be consistent with a decrease in the rewarding properties of sucrose pellets. Despite low response rates, amphetamine did not affect responding maintained by 95% sucrose pellets but did further reduce the consumption of earned reward. These results call into question the generality of the rate-dependency principle in the action of psychomotor stimulants.

Research on self-control: An integrating framework

AbstractThe tendency to choose a larger, more delayed reinforcer over a smaller, less delayed one has frequently been termed “selfcontrol.” Three very different research traditions – two models emphasizing the control of local contingencies of reinforcement (Mischel's social learning theory and Herrnstein's matching law) and molar maximization models (specifically optimal foraging theory) – have all investigated behavior within the self-control paradigm. A framework is proposed to integrate research from all three research areas. This framework consists of three parts: a procedural analysis, a causal analysis, and a theoretical analysis. The procedural analysis provides a common procedural terminology for all three areas. The causal analysis establishes that, in all three research traditions, self-control varies directly with the current physical values of the reinforcers; that is, choices increase with reinforcer amount and decrease with reinforcer delay. But self-control also varies according to past events to which a subject has been exposed, and according to current factors other than the reinforcers. Each of the three models has therefore incorporated these indirect effects on self-control by postulating unobservable mechanisms. In all three cases, these mechanisms represent a subject's behavior as a function of a perceived environment. The theoretical analysis demonstrates that evolutionary theory can encompass the research from all three areas by considering differences in the adaptiveness of self-control in different situations. This integration provides a better and more predictive description of self-control.

Effects of amphetamine and pimozide on reinforcement and motor parameters in variable-interval performance

The effects of amphetamine and pimozide were studied in rats performing on variable-interval (VI) schedules of reinforcement. In experiment 1, a five-component multiple VI schedule was used; in experiments 2 and 3, two VI schedules were presented on alternate days; a fourth experiment was carried out to validate the two VI methods. The Herrnstein matching law was used to distinguish between changes in reinforcer efficacy and changes in motor capacity. In both procedures, amphetamine, at 0.5 mg/kg, caused changes compatible with an increase in reinforcer efficacy with no change in motor capacity; higher doses (only tested in the multiple schedule) appeared further to increase reinforcer efficacy and also to decrease motor capacity. In the multiple schedule, pimozide caused changes compatible with either a decrease in reinforcer efficacy or a decrease in motor capacity, depending on the order of presentation of the schedule components; in the alternating schedule, pimozide had both effects. These discrepancies appeared to be due to changes in the effect of pimozide over time, which could not be explained either by satiation or by a gradual onset of drug action.

Matching in Multiple Schedules Using Long Component Durations

The performance of 3 rats in multiple variable-interval variable-interval schedules, employing 10-min. component durations, conformed to Herrnstein's Matching Law.

THE MATCHING LAW IN AND WITHIN GROUPS OF RATS1

In each of the two experiments, a group of five rats lived in a complex maze containing four small single-lever operant chambers. In two of these chambers, food was available on variable-interval schedules of reinforcement. In Experiment I, nine combinations of variable intervals were used, and the aggregate lever-pressing rates (by the five rats together) were studied. The log ratio of the rates in the two chambers was linearly related to the log ratio of the reinforcement rates in them; this is an instance of Herrnstein's matching law, as generalized by Baum. Summing over the two food chambers, food consumption decreased, and response output increased, as the time required to earn each pellet increased. In Experiment II, the behavior of individual rats was observed by time-sampling on selected days, while different variable-interval schedules were arranged in the two chambers where food was available. Individual lever-pressing rates for the rats were obtained, and their median bore the same "matching" relationship to the reinforcement rates as the group aggregate in Experiment I. There were differences between the rats in their distribution of time and responses between the two food chambers; these differences were correlated with differences in the proportions of reinforcements the rats obtained from each chamber.