Relative Reinforcement Rate Research Articles

Knockout of NMDARs in CA1 and dentate gyrus fails to impair temporal control of conditioned behavior in mice.

The hippocampus has been implicated in temporal learning. Plasticity within the hippocampus requires NMDA receptor-dependent glutamatergic neurotransmission. We tested the prediction that hippocampal NMDA receptors are required for learning about time by testing mice that lack postembryonal NMDARs in the CA1 and dentate gyrus (DG) hippocampal subfields on three different appetitive temporal learning procedures. The conditional knockout mice (Grin1ΔDCA1 ) showed normal sensitivity to cue duration, responding at a higher level to a short duration cue than compared to a long duration cue. Knockout mice also showed normal precision and accuracy of response timing in the peak procedure in which reinforcement occurred after 10 s delay within a 30 s cue presentation. Mice were tested on the matching of response rates to reinforcement rates on instrumental conditioning with two levers reinforced on a concurrent variable interval schedule. Pressing on one lever was reinforced at a higher rate than the other lever. Grin1ΔDGCA1 mice showed normal sensitivity to the relative reinforcement rates of the levers. In contrast to the lack of effect of hippocampal NMDAR deletion on measures of temporal sensitivity, Grin1ΔDGCA1 mice showed increased baseline measures of magazine activity and lever pressing. Furthermore, reversal learning was enhanced when the reward contingencies were switched in the lever pressing task, but this was true only for mice trained with a large difference between relative reinforcement rates between the levers. The results failed to demonstrate a role for NMDARs in excitatory CA1 and DG neurons in learning about temporal information.

Zebrafish choice behavior is sensitive to reinforcer rate, immediacy, and magnitude ratios.

Behavioral flexibility has, in part, been defined by choice behavior changing as a function of changes in reinforcer payoffs. We examined whether the generalized matching law quantitatively described changes in choice behavior in zebrafish when relative reinforcer rates, delays/immediacy, and magnitudes changed between two alternatives across conditions. Choice was sensitive to each of the three reinforcer properties. Sensitivity estimates to changes in relative reinforcer rates were greater when 2 variable-interval schedules were arranged independently between alternatives (Experiment 1a) than when a single schedule pseudorandomly arranged reinforcers between alternatives (Experiment 1b). Sensitivity estimates for changes in relative reinforcer immediacy (Experiment 2) and magnitude (Experiment 3) were similar but lower than estimates for reinforcer rates. These differences in sensitivity estimates are consistent with studies examining other species, suggesting flexibility in zebrafish choice behavior in the face of changes in payoff as described by the generalized matching law.

Numerical analysis and formulae for SCF reduction coefficients of CFRP-strengthened CHS gap K-joints

The focus of this paper is on estimating stress concentration factor (SCF) reduction coefficients (ψ) of circular hollow section (CHS) gap K-joints strengthened with carbon fibre reinforced polymer (CFRP) sheets (CFRP-CHS K-joints), so that the SCFs can be calculated on the basis of the existing formulae for unstrengthened joints. Three-dimensional finite element models are developed for both CFRP-strengthened and unstrengthened CHS gap K-joints to calculate the SCFs, which are verified using previous test data. A parametric study is then carried out to explore the effects of various parameters (11 independent CFRP strengthening parameters and 5 basic geometric parameters describing the steel K-joints) on the efficiency of CFRP strengthening in terms of SCF reduction coefficient ψ. It is found that ψ is primarily affected by the CFRP relative reinforcement rates, the CFRP-to-steel tensile modulus ratios, the adhesive modulus, and non-dimensional geometric parameters. Through regression analyses, parametric formulae are finally developed for estimating ψ at six critical hot spots in CFRP-CHS K-joints under balanced axial loading. The proposed parametric formulae agree well with the experimental data.

An exploratory analysis of Relational Density Theory: Relational resistance and gravity

We conducted an initial exploration of several model assumptions in Relational Density Theory (Belisle & Dixon, in press) across two experiments with 18 college student participants. The first experiment showed that relational density (response accuracy/response time) may be influenced by relative rates of reinforcement or training exposure, and that denser classes were more resistant to counterconditioning (i.e., relational resistance). In the second experiment, two classes were established that differed in size, and for most initial participants, greater density was obtained for the smaller class. When a third class was established, and a class member from the third class was presented as a sample along with competing comparison stimuli from the initial two classes, most participants matched the new class member with the denser class (i.e., relational gravity). Additional participants were added, and a familiar stimulus was substituted into the larger class. For all additional participants, greater density was observed for the larger class containing the familiar stimulus, and participants matched the new class member with the larger-denser class. Results support several model assumptions, but several model inconsistent findings are presented as avenues for future research.

Relative reinforcer rates determine pigeons' attention allocation when separately trained stimuli are presented together.

Previous research suggests that organisms allocate more attention to stimuli associated with higher reinforcer rates. This finding has been replicated several times when stimuli are trained together as compounds, but not in other procedures. Thus, the generality of the relation between relative reinforcer rates and divided attention is not well established. Therefore, we investigated whether relative reinforcer rates determine attention allocation when stimuli are trained separately and then encountered together. Pigeons learned to associate two colors and two frequencies of key light on/off alternation with a left or right comparison key in a symbolic 0-s delayed matching-to-sample task. Across conditions, we varied the probability of reinforcement associated with each stimulus dimension during training. After training, we introduced test trials in which a color and flash-frequency stimulus were presented simultaneously. During sample-stimulus presentation in test trials, all pigeons preferred the stimulus associated with the higher reinforcer rate, suggesting that more attention was allocated to that stimulus. Interestingly, such attention allocation did not result in preference for the comparison that matched that stimulus. Instead, all pigeons preferred the comparison that was physically closer to the stimulus associated with the higher reinforcer rate, suggesting that comparison choice was controlled by the location of that stimulus. Nevertheless, overall, our results provide the first evidence that relative reinforcer rates determine divided attention between separately trained stimuli and thus demonstrate the generality of the relation between relative reinforcement and attention allocation. We suggest several avenues for future research to establish further the generality of this relation.

Timing or counting? Control by contingency reversals at fixed times or numbers of responses.

Behavior may come under the control of time or number when such cues predict food availability, but time tends to exert stronger control than does number. We asked how the control of behavior is divided between time and number in a procedure where the likely location of a reinforcer reversed systematically at a particular point in a trial. In Phase 1, we asked whether pigeons would choose to time or count when both time and number of responses could be used to discriminate the occurrence of the reversal. Pigeons preferred to time, even when numbers of events were more reliable predictors of reinforcer availability. In the Phase 2, we asked whether control could be shifted from time to number when number was highly discriminable and time was a highly unreliable predictor of the reversal. In this phase, pigeons continued to favor time over number. These findings suggest divided stimulus control between time and numbers depends both on recent experience with relative reinforcer rates and on factors relating to the animal's longer term history, such as training. This research adds to a growing body of work demonstrating strong control by elapsed time when elapsed time is not the most reliable predictor of reinforcer availability. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Reforzamiento Concurrente de Secuencias de Respuestas

Decidir si una secuencia de respuestas es una unidad conductual diferente a los elementos que la constituyen y que obedece a las mismas reglas de reforzamiento que respuestas discretas ha representado un problema difícil de resolver. Una forma de mostrar que una secuencia es una unidad de conducta, es observar si dicha secuencia cumple la relación propuesta por la ley de igualación. Con este propósito, se entrenó a cuatro ratas privadas de alimento y experimentalmente ingenuas a apretar en secuencia dos operandos para recibir una gota de leche. Se usaron diferentes programas concurrentes de intervalo variable (VI) para reforzar cada una de cuatro posibles secuencias. Cuando las ratas produjeron una secuencia no reforzada, las luces de la cámara experimental se apagaron durante 2 s. En tres diferentes fases se varió la tasa de reforzamiento para cada secuencia. Los resultados mostraron que las ratas fueron altamente sensibles a las tasas relativas de reforzamiento. Sin embargo, las pendientes de las curvas de ajuste fueron menores a uno, mostrando sub-igualación. Se concluyó que una secuencia de respuestas puede comportarse como un patrón integrado pero no representa una unidad conductual estrictamente equivalente a las operantes discretas.

Choice among two and three alternatives.

Although choice between two alternatives has been widely researched, fewer studies have examined choice across multiple (more than two) alternatives. Past models of choice behavior predict that the number of alternatives should not affect relative response allocation, but more recent research has found violations of this principle. Five pigeons were presented with three concurrently scheduled alternatives. Relative reinforcement rates across these alternatives were assigned 9:3:1. In some conditions three keys were available; in others, only two keys were available. The number of available alternatives did not affect relative response rates for pairs of alternatives; there were no significant differences in behavior between the two and three key conditions. For two birds in the three-alternative conditions and three birds in the two-alternative conditions, preference was more extreme for the pair of alternatives with the lower overall pairwise reinforcer rate (3:1) than the pair with higher overall reinforcer rate (9:3). However, when responding during the changeover was removed three birds showed the opposite pattern in the three-alternative conditions; preference was more extreme for the pair of alternatives with the higher overall reinforcer rate. These findings differ from past research and do not support established theories of choice behavior.

Visual reinforcement shapes eye movements in visual search.

We use eye movements to gain information about our visual environment; this information can indirectly be used to affect the environment. Whereas eye movements are affected by explicit rewards such as points or money, it is not clear whether the information gained by finding a hidden target has a similar reward value. Here we tested whether finding a visual target can reinforce eye movements in visual search performed in a noise background, which conforms to natural scene statistics and contains a large number of possible target locations. First we tested whether presenting the target more often in one specific quadrant would modify eye movement search behavior. Surprisingly, participants did not learn to search for the target more often in high probability areas. Presumably, participants could not learn the reward structure of the environment. In two subsequent experiments we used a gaze-contingent display to gain full control over the reinforcement schedule. The target was presented more often after saccades into a specific quadrant or a specific direction. The proportions of saccades meeting the reinforcement criteria increased considerably, and participants matched their search behavior to the relative reinforcement rates of targets. Reinforcement learning seems to serve as the mechanism to optimize search behavior with respect to the statistics of the task.

Choice in transition: Replication and extension to preschool children in a naturalistic setting.

This study replicated previous basic research into the dynamics of choice and extended this analysis to children's behavior in a naturalistic setting. Two preschoolers with disabilities were observed interacting with their teachers at baseline and during an experimental analysis involving four pairs of concurrent variable-interval schedules of adult attention implemented by an experimenter. Each child was exposed to four experimental phases in which the relative reinforcer rates for on- and off-task behavior were 10:1, 1:1, 1:10, and reversed back to 10:1. The 10:1 phase was designed to mimic the same schedules and types of adult attention observed at baseline. We used the generalized matching equation to model steady-state behavior at the end of the transition phases and to evaluate changes in sensitivity at various points throughout the phases. Choice in transition was evaluated by plotting log behavior ratios by session, cumulated time on- and off-task and cumulated attention for on- and off-task behavior by session, and interreinforcer behavior ratios following different sequences of the first four reinforcer deliveries. The generalized matching equation accounted for a large proportion of variance in steady-state responding, sensitivity values increased steadily throughout the phases, patterns of choice in transition were similar to those reported in basic research, and interreinforcer preference generally shifted toward the just-reinforced alternative. These findings are consistent with previous basic research and support the generality of the dynamics of choice to children's on- and off-task behavior reinforced by adult attention.

The distribution of response bout lengths and its sensitivity to differential reinforcement.

Response bouts are clusters of responses that occur in rapid succession and are punctuated by pauses during which the response does not occur. Under variable interval schedules of reinforcement, the number of responses in each bout (the bout length) varies among bouts. This experiment was aimed at determining whether the relative rate of reinforcement influenced the relative frequency of bouts of different lengths. Lever pressing in rats was reinforced under a tandem variable time (VT) 150-s fixed ratio (FR) X, where X could be 1 or 5 and varied randomly after each reinforcer. Two conditions were included: majority FR1 (mFR1) and majority FR5 (mFR5). In mFR1, 75% of reinforcers had a tandem FR requirement of 1 and 25% had a tandem FR requirement of 5; this distribution was reversed in mFR5. The dynamic bi-exponential refractory model of response bouts was fitted to the interresponse times (IRTs) in each condition. Model parameter estimates and IRTs were then used to simulate probable distributions of bout lengths. These distributions comprised a mixture of short geometrically-distributed bout lengths and long negative-binomially-distributed bout lengths. Long bouts were significantly longer in the mFR5 condition than in the mFR1 condition. In conjunction with previous data, the present study suggests that the prevalence of long bouts increases with the proportion of reinforcers with FR5 requirement. These results suggest that bouts of different lengths are sensitive to the rate at which they are reinforced.

Dynamics of Neuroeconomics Decision-Making

Quantification of choice has been a major area of research for neuro scientists for several decades. This is, in part, due to the discovery of the ‘Matching Law’ that stipulates that relative response rate on concurrently available alternatives ‘match’ the available relative reinforcement rates. This theoretical construct has been developed to describe response allocation in more complex situations. People often fail to design ‘rational’ decisions. Economics agents are subject to multiple biases that affect the way they perceive events, act upon them and learn from experience. These behaviours cannot be ignored since they have disastrous consequences for organisations. When faced with complex decision, individuals engage in simplifying strategies. Adaptive decision making in real-world contexts relies on strategic simplifications of decision problems. Yet, neural mechanisms that shape these strategies and their implementation remain largely unknown. Although we now know much about how brain encodes specific decision factors, much less is known about how brain selects among multiple strategies for managing computational demands of complex decision-making task. Expansion of neuroeconomics parallels development of cognitive science. Neuroeconomics has bridged the contrasting fields of economics and psychology. Economics, psychology, and neuroscience are converging today into a single, unified discipline with the ultimate aim of providing a single, general theory of human behaviour. This is the emerging field of Neuroeconomics in which consilience, accordance of two or more inductions drawn from different groups of phenomena, seems to be operating. Economists and psychologists are providing rich conceptual tools for understanding and modeling behaviour, while neurobiologists provide tools for the study of mechanism. The goal of this discipline is thus to understand the processes that connect sensation and action by revealing the neurobiological mechanisms by which decisions are made. Such union is almost exclusively attributable to changes within economics. Neuroeconomics has inspired change because the important findings have posed more of a challenge to standard economics perspective. The important source of inspiration for neuro economist has been neuro judgment research, which can, in turn, be seen as an amalgamation of ideas from cognitive science and economics. Neuroeconomics has primarily challenged customary economics postulation that decision-making is a unitary process a simple matter of integrated and coherent utility maximization suggesting instead that it is driven by interaction between automatic and controlled processes. Despite substantial advances, question of how we design and how we ought to craft judgments and decisions has engaged researchers for decades, with different disciplines approaching the problem through characteristically different techniques. However, neuroeconomics decision making has recently emerged as an inter-disciplinary effort to bridge this gap. It has sought to integrate ideas from fields of organisational psychology, neuroscience and neuroeconomics in an effort to specify accurate models of choice and decision. Research investigates neural bases of decision predictability and value, central parameters in economics model of expected utility. Neuro-multiple-systems approach to decision-making, in turn, influences economics, a perspective strongly rooted in organisational psychology and neuroscience. Integration of these approaches and methodologies offers exciting potential for construction of near-accurate models of decision-making.

Lesions of the nucleus accumbens core modulate development of matching behavior.

BackgroundThe development of choice is a crucial determinant in the performance of appetitive responses. Given two options with different reinforcement rates, animals match their relative rate of responding to the relative rates of reinforcement (i.e., matching behavior). A previous study has shown that the nucleus accumbens core (AcbC) is involved in the performance of matching behavior in trained animals. However, the role of the AcbC in the acquisition of matching behavior has not been addressed.ResultsWe conducted a series of experimental sessions to examine the role of the AcbC on the development of matching behavior. Instrumental responding was measured in rats with excitotoxic lesions of the AcbC. Rats were given two options that differed in the relative rate of reinforcement under concurrent variable-interval schedules. The locations of the more frequently reinforced option and the alternative option were randomly switched between sessions. Lesions of the AcbC accelerated the development of matching behavior compared to the sham-operated group. The AcbC-lesioned rats exhibited closer conformity to the matching law than shams when the options were in the same positions as in the previous session (the same condition), but not when the option locations had been switched (the different condition). The AcbC rats showed smaller probabilities of switching behavior between alternatives than shams. Post-reinforcement pausing was not affected by the AcbC lesion. Neither numbers of rewards obtained nor number of lever presses were different between the AcbC-lesioned rats and shams over session blocks.ConclusionsOur results suggest that the AcbC plays a regulatory role in the development of matching behavior through switching probabilities rather than perception of reward magnitude. The differential effect of AcbC lesions on the matching behavior between the same and different conditions suggests influence of the spontaneous recovery, that is, reversion to a previously reinforced choice at the beginning of the next session, on the development of matching behavior in the AcbC-lesioned rats.

On applying the matching law to between-subject data

The matching law is a descriptive mathematical model thatconceptualizes an organism’s response rates as a function of rela-tive rates of reinforcement (Herrnstein 1961). This relation isexpressed by equation (1):B1=B2 ¼ R1=R2 (1)where Bs represent rates of behaviour and Rs represent reinforcerrates. However, extensive research has confirmed that organisms’behaviour systematically deviates from predictions of equation (1)(Baum 1974, 1979 , 1983; Wearden & Burgess 1982; Davison M McDowell 2012). The power function version,known as the generalized matching law (Baum 1974), providesbetter descriptions of animals’ and humans’ choices in concurrentschedules and is represented logarithmically by equation (2):log ðB1=B2Þ¼a log ðR1=R2Þþlog c (2)where Bs and Rs are the same as in equation (1), a is referred to assensitivity and represents the organism’s adjustment to the con-tingencies, and c is called the bias parameter and represents theorganism’s preference when choice is asymmetrical. Note thatequation (2) reduces to equation (1) when a ¼ c ¼ 1, and is usuallycalled strict matching. Furthermore, most studies have found thatthe sensitivity value varies around 0.8, which is referred to asundermatching.Herrnstein (1961) developed the matching law to describe in-dividual choices. A misunderstanding of this fundamentalassumption has led several authors (i.e. Schroeder & Holland 1969;Conger & Killeen 1974; Schmitt 1974; Vollmer & Bourret 2000;Borrero & Vollmer 2002; Alferink et al. 2009; McDowell & Caron2010a, b) to conduct their analyses incorrectly. For instance, theyused a small number of levels of the independent variable andpooled their subjects’ data, which combined idiosyncraticdependent variables into a single analysis and did not account forwithin-subject and within-level variances. At best, between-subject data will estimate an inaccurate model with erroneousparameter values and, at worst, will violate several statistical as-sumptions and lead to uninterpretable results. The purpose of thecurrent study is to prevent future misunderstanding by clarifyingthese errors.Pooling Different VariablesStatistically, the dependent variable must be the same throughall levels of the independent variable. Applied to the matching law,this means that the response ratio must remain the same at alllevels of the reinforcer ratios. Therefore, between-subject analysesmust rely on the same responses for every subject. However, if thesubjects had different target behaviours, then their log response

Resistance to change and preference for variable versus fixed response sequences.

In Experiment 1, 4 pigeons were trained on a multiple chain schedule in which the initial link was a variable-interval (VI) 20-s schedule signalled by a red or green center key, and terminal links required four responses made to the left (L) and/or right (R) keys. In the REPEAT component, signalled by red keylights, only LRLR terminal-link response sequences were reinforced, while in the VARY component, signalled by green keylights, terminal-link response sequences were reinforced if they satisfied a variability criterion. The reinforcer rate for both components was equated by adjusting the reinforcer probability for correct REPEAT sequences across sessions. Results showed that initial- and terminal-link responding in the VARY component was generally more resistant to prefeeding, extinction, and response-independent food than responding in the REPEAT component. In Experiment 2, the REPEAT and VARY contingencies were arranged as terminal links of a concurrent chain and the relative reinforcer rate was manipulated across conditions. For all pigeons, initial-link response allocation was biased toward the alternative associated with the VARY terminal link. These results replicate previous reports that operant variation is more resistant to change than operant repetition (Doughty & Lattal, 2001), and show that variation is preferred to repetition with reinforcer-related variables controlled. Behavioral momentum theory (Nevin & Grace, 2000) predicts the covariation of preference and resistance to change in Experiments 1 and 2, but does not explain why these aspects of behavior should depend on contingencies that require repetition or variation.

Testing the linearity and independence assumptions of the generalized matching law for reinforcer magnitude: A residual meta-analysis

We conducted a residual meta-analysis to test the assumptions of the generalized matching law that effects of relative reinforcer magnitude on response allocation in concurrent schedules can be described by a power function and are independent from the effects of relative reinforcer rate. We identified five studies which varied magnitude ratios over at least four levels and six studies in which reinforcer rate and magnitude ratios were varied factorially. The generalized matching law provided a reasonably good description of the data, accounting for 77.1% and 90.1% of the variance in the two sets of studies. Results of polynomial regressions showed that there were no systematic patterns in pooled residuals as a function of predicted log response ratios for data sets in which relative magnitude was varied. For data sets in which relative rate and magnitude were varied factorially, there was a significant negative cubic pattern in the pooled residuals, suggesting that obtained response allocation was less extreme than predicted for conditions with extreme predicted values. However, subsequent analyses showed that this result was associated with results from conditions in one study in which the product of the rate and magnitude ratios was 63:1, and in which response allocation may have been attenuated by a ceiling effect. When data from these conditions were omitted, there were no significant components in the residuals. Although the number of available studies was small, results provide tentative support for the assumptions of the generalized matching law that effects of reinforcer magnitude ratios on choice can be described by a power function and are independent from reinforcer rate ratios.

Concurrent VR VI schedules: Primacy of molar control of preference and molecular control of response rates

In the first condition in Experiment 1, 6 rats were exposed to concurrent variable ratio (VR) 30, variable interval (VI) 30-sec schedules. In the next two conditions, the subjects were exposed to concurrent VI VI schedules and concurrent tandem VI-differential-reinforcement-of-high-rate VI schedules. For the latter conditions, the overall and relative reinforcer rates equaled those in the first condition. Only minor differences appeared in time allocation (a molar measure) across conditions. However, local response rate differences (a molecular measure) appeared between schedule types consistently with the interresponse times these schedules reinforced. In Experiment 2, these findings reappeared when the prior experiment was replicated with 5 subjects, except that the VR schedule was replaced by a VI plus linear feedback schedule. These results suggest that within the context tested, the molar factor of relative reinforcement rate controls preference, whereas the molecular factor of the relation between interresponse times and reinforcer probability controls the local response rate.

A further application of the active time model to multiple concurrent variable-interval schedules

In this experiment we show that the active time model (ATM) accurately predicts probe data from multiple concurrent VI VI schedules. Subjects were trained under a concurrent VI 30-s VI 60-s and a concurrent VI 60-s VI 120-s schedule. Two types of unreinforced probes were then conducted. The first paired the two VI 60-s stimuli. These stimuli, while equivalent in their associated absolute rates of reinforcement, differed in their relative rates of reinforcement. The second probe paired the VI 30-s stimulus with the relatively rich VI 60-s stimulus. In contrast with the first probe, these stimuli differed in their absolute rates of reinforcement, while being similar in their relative rates. During the first set of probes, birds preferred the VI 60-s stimulus trained with the VI 120-s schedule. During the second set of probes, birds were indifferent to the two stimuli. These results are less extreme than others reported in the literature. Nonetheless, we found that ATM accurately fit individual subject data in both sets of probes. In contrast a variant of scalar expectancy theory did not fit the data at either the individual or group level.

Concurrent reinforcement schedules for problem behavior and appropriate behavior: experimental applications of the matching law.

This study evaluated how children who exhibited functionally equivalent problem and appropriate behavior allocate responding to experimentally arranged reinforcer rates. Relative reinforcer rates were arranged on concurrent variable-interval schedules and effects on relative response rates were interpreted using the generalized matching equation. Results showed that relative rates of responding approximated relative rates of reinforcement. Finally, interventions for problem behavior were evaluated and differential reinforcement of alternative behavior and extinction procedures were implemented to increase appropriate behavior and decrease problem behavior. Practical considerations for the application of the generalized matching equation specific to severe problem behavior are discussed, including difficulties associated with defining a reinforced response, and obtaining steady state responding in clinical settings.

Four-alternative choice violates the constant-ratio rule

Six pigeons responded on a four-key concurrent variable-interval schedule in which a 27:9:3:1 distribution of reinforcers between the keys changed every 10 reinforcers. Their behaviour quickly came under the control of this changing four-way reinforcer ratio. However, preference between a pair of keys depended not only on the relative reinforcer rates on those keys, but also on the absolute levels of those rates. This contradicts the constant-ratio rule that underpins the matching approach to choice, but is predicted by a contingency-discriminability model that assumes that organisms may occasionally misattribute reinforcers to a response that did not produce them. Reinforcers produced strong preference pulses, or transient increases in responding on the just-reinforced key. Despite accurate tracking of the reinforcer ratio, reinforcers obtained late in components and from leaner keys still produced strong pulses, suggesting both extended and local control of behaviour. Patterns of switching between keys were graded and similarly controlled by the reinforcer rates on each key. Whether considered in terms of switching, local preference pulses, or extended preference, behaviour was controlled by a rapidly changing four-way reinforcer ratio in a graduated, continuous manner that is unlikely to be explained by a simple heuristic such as fix-and-sample.