Concurrent Variable Interval Research Articles

Toward a Procedure to Study Rule-Governed Choice: Preliminary Data

AbstractThe aim of this study was to model a situation that induced choice between following two incompatible rules, each associated with a different rate of reinforcement. In Experiment 1, eight undergraduate students were exposed to a two-component multiple schedule (training). In each component, there was a concurrent variable interval (VI)–extinction (EXT) schedule. Participants were given two rules that instructed them to respond to the VI alternative in the presence of different discriminative stimuli. The side of the VI schedule changed in each component and offered a different reinforcer rate according to the discriminative stimuli in the operation. When both discriminative stimuli were concurrently presented (test), participants favored the alternative previously instructed by the rule, which was associated with the greatest reinforcer rate, whereas indifference was observed in the absence of discriminative stimuli. Experiment 2 tested the effects of reinforcement rate using the same procedure without providing rules. During training, participants gradually developed a preference for the VI alternatives. In the choice test phase, participants favored the alternative associated with the stimuli with the highest reinforcer rate when both discriminative stimuli were present. Unsystematic preference was observed in the absence of discriminative stimuli. Two alternative explanations were provided for the findings.

The active time model of concurrent choice.

The following paper describes a steady-state model of concurrent choice, termed the active time model (ATM). ATM is derived from maximization principles and is characterized by a semi-Markov process. The model proposes that the controlling stimulus in concurrent variable-interval (VI) VI schedules of reinforcement is the time interval since the most recent response, termed here "the active interresponse time" or simply "active time." In the model after a response is generated, it is categorized by a function that relates active times to switch/stay probabilities. In the paper the output of ATM is compared with predictions made by three other models of operant conditioning: melioration, a version of scalar expectancy theory (SET), and momentary maximization. Data sets considered include preferences in multiple-concurrent VI VI schedules, molecular choice patterns, correlations between switching and perseveration, and molar choice proportions. It is shown that ATM can account for all of these data sets, while the other models produce more limited fits. However, rather than argue that ATM is the singular model for concurrent VI VI choice, a consideration of its concept space leads to the conclusion that operant choice is multiply-determined, and that an adaptive viewpoint-one that considers experimental procedures both as selecting mechanisms for animal choice as well as tests of the controlling variables of that choice-is warranted.

How do animals weigh conflicting information about reward sources over time? Comparing dynamic averaging models

Optimal foraging theory suggests that animals make decisions which maximize their food intake per unit time when foraging, but the mechanisms animals use to track the value of behavioral alternatives and choose between them remain unclear. Several models for how animals integrate past experience have been suggested. However, these models make differential predictions for the occurrence of spontaneous recovery of choice: a behavioral phenomenon in which a hiatus from the experimental environment results in animals reverting to a behavioral allocation consistent with a reward distribution from the more distant past, rather than one consistent with their most recently experienced distribution. To explore this phenomenon and compare these models, three free-operant experiments with rats were conducted using a serial reversal design. In Phase 1, two responses (A and B) were baited with pellets on concurrent variable interval schedules, favoring option A. In Phase 2, lever baiting was reversed to favor option B. Rats then entered a delay period, where they were maintained at weight in their home cages and no experimental sessions took place. Following this delay, preference was assessed using initial responding in test sessions where levers were presented, but not baited. Models were compared in performance, including an exponentially weighted moving average, the Temporal Weighting Rule, and variants of these models. While the data provided strong evidence of spontaneous recovery of choice, the form and extent of recovery was inconsistent with the models under investigation. Potential interpretations are discussed in relation to both the decision rule and valuation functions employed.

Knockout of NMDARs in CA1 and dentate gyrus fails to impair temporal control of conditioned behavior in mice.

The hippocampus has been implicated in temporal learning. Plasticity within the hippocampus requires NMDA receptor-dependent glutamatergic neurotransmission. We tested the prediction that hippocampal NMDA receptors are required for learning about time by testing mice that lack postembryonal NMDARs in the CA1 and dentate gyrus (DG) hippocampal subfields on three different appetitive temporal learning procedures. The conditional knockout mice (Grin1ΔDCA1 ) showed normal sensitivity to cue duration, responding at a higher level to a short duration cue than compared to a long duration cue. Knockout mice also showed normal precision and accuracy of response timing in the peak procedure in which reinforcement occurred after 10 s delay within a 30 s cue presentation. Mice were tested on the matching of response rates to reinforcement rates on instrumental conditioning with two levers reinforced on a concurrent variable interval schedule. Pressing on one lever was reinforced at a higher rate than the other lever. Grin1ΔDGCA1 mice showed normal sensitivity to the relative reinforcement rates of the levers. In contrast to the lack of effect of hippocampal NMDAR deletion on measures of temporal sensitivity, Grin1ΔDGCA1 mice showed increased baseline measures of magazine activity and lever pressing. Furthermore, reversal learning was enhanced when the reward contingencies were switched in the lever pressing task, but this was true only for mice trained with a large difference between relative reinforcement rates between the levers. The results failed to demonstrate a role for NMDARs in excitatory CA1 and DG neurons in learning about temporal information.

Is increased sensitivity to punishment a common characteristic of attention deficit/hyperactivity disorder? An experimental study of response allocation in Japanese children

Research on motivational processes in attention deficit/hyperactivity disorder (ADHD) focuses on reward. Studies with punishment are limited and findings mixed. This study evaluated the effects of punishment on response allocation in Japanese children with and without ADHD. Thirty-four children meeting DSM-IV criteria for ADHD and 59 typically developing control-group children completed an operant task in which they choose between playing two simultaneously available games. Reward was arranged symmetrically across the games under concurrent variable interval schedules. Asymmetric punishment schedules were superimposed with responses on one game punished four times as often as responses on the other. Children with ADHD showed greater behavioral sensitivity to punishment than controls. They allocated significantly more responses to the less frequently punished alternative and were more likely to play this game on consecutive trials and responded more slowly to the more punished game. Control group children allocated their responses evenly across games. Punishment exerted greater control over the behavior of Japanese children with ADHD than controls, similar to findings with children from Western countries, suggesting this is a common characteristic of the disorder. The behavior of typically developing Japanese children, while demonstrating awareness of punishment, was not controlled by the frequency of its occurrence.

Reforzamiento Concurrente de Secuencias de Respuestas

Decidir si una secuencia de respuestas es una unidad conductual diferente a los elementos que la constituyen y que obedece a las mismas reglas de reforzamiento que respuestas discretas ha representado un problema difícil de resolver. Una forma de mostrar que una secuencia es una unidad de conducta, es observar si dicha secuencia cumple la relación propuesta por la ley de igualación. Con este propósito, se entrenó a cuatro ratas privadas de alimento y experimentalmente ingenuas a apretar en secuencia dos operandos para recibir una gota de leche. Se usaron diferentes programas concurrentes de intervalo variable (VI) para reforzar cada una de cuatro posibles secuencias. Cuando las ratas produjeron una secuencia no reforzada, las luces de la cámara experimental se apagaron durante 2 s. En tres diferentes fases se varió la tasa de reforzamiento para cada secuencia. Los resultados mostraron que las ratas fueron altamente sensibles a las tasas relativas de reforzamiento. Sin embargo, las pendientes de las curvas de ajuste fueron menores a uno, mostrando sub-igualación. Se concluyó que una secuencia de respuestas puede comportarse como un patrón integrado pero no representa una unidad conductual estrictamente equivalente a las operantes discretas.

The effect of scopolamine on matching behavior and the estimation of relative reward magnitude.

We investigated the behavioral effects of scopolamine on rats that bar pressed for trains of electrically stimulating pulses under concurrent variable interval schedules of reward. For the first half of the session (30 min) a 1:4 ratio in the programmed number of stimulation trains delivered at each option was in effect. At the start of the second half of the session, an unsignaled reversal in the relative train number (4:1) occurred. We tracked the relative magnitude of reward estimated for each contiguous pair of reinforced visits to competing options. Scopolamine hydrobromide led to a reduction in the relative magnitude of reward. A similar result was obtained in a follow-up test in which relative magnitude was manipulated by varying the pulse frequency of stimulation, while equating the train number at each option. The effect of scopolamine hydrobromide could not be attributed to undermatching, side bias, nor to an effect of scopolamine on the reward integration process. When the same rats were treated with scopolamine methylbromide, no effects on matching behavior were observed. Our results suggest a cholinergic basis for the computation of choice variables related to matching behavior. (PsycINFO Database Record

Choice predicts the feedback negativity.

Choosing the appropriate response given the circumstance is integral to all aspects of human behavior. One way of elucidating the mechanisms of choice is to relate behavior to neural correlates. Electrophysiological evidence implicates the ERP feedback-negativity (FN) and the P300 as promising neural correlates of reward processing, an integral component of learning. However, prior research has not adequately addressed how the development of a preference to select one option over another (choice preference) relates to the FN and the P300. We assessed whether variation in choice preference predicted the FN and P300 amplitude within subjects. We used a discrete-trials two-alternative choice procedure, where the reinforcer rate for each option was dependently scheduled by a concurrent variable interval. The reinforcer ratio for selecting each option was varied between sessions. Choice was quantified using both the generalized matching law sensitivity and the log odds of staying on the same versus switching to the other alternative (stay preference). The relationship between stay preference, FN, and P300 amplitudes was assessed using the innovative application of hierarchical Bayesian linear regression. The results demonstrate that stay preference was controlled by the reinforcer ratios and credibly predicted the FN amplitude but not P300 amplitude. The findings are consistent with the view that reinforcers may guide behavior by what they signal about future reinforcement, with the FN related to such a process.

Evidence for increased behavioral control by punishment in children with attention-deficit hyperactivity disorder.

The behavioral sensitivity of children with ADHD to punishment has received limited theoretical and experimental attention. This study evaluated the effects of punishment on the response allocation of children with ADHD and typically developing children. Two hundred and ten children, 145 diagnosed with ADHD, completed an operant task in which they chose between playing two simultaneously available games. Reward was arranged symmetrically across the games under concurrent variable interval schedules. Asymmetric punishment schedules were superimposed; responses on one game were punished four times as often as responses on the other. Both groups allocated more of their responses to the less frequently punished alternative. Response bias increased significantly in the ADHD group during later trials, resulting in missed reward trials and reduced earnings. Punishment exerted greater control over the response allocation of children with ADHD with increased time on task. Children with ADHD appear more sensitive to the cumulative effects of punishment than typically developing children.

The effect of MDMA on sensitivity to reinforcement rate.

Administration of (±)3,4-methylenedioxymethamphetamine (MDMA) causes memory errors by increasing proactive interference. This might occur because MDMA alters sensitivity to reinforcement. The current 2 experiments investigated this directly by assessing the acute (Experiment 1) and chronic (Experiment 2) effects of MDMA on sensitivity to reinforcement. We presented 5 pairs of concurrent variable interval schedules within each session and calculated sensitivity to reinforcement for 3 acute doses of MDMA. In contrast to the related drug, d-amphetamine, and in spite of producing reductions in response rate, MDMA did not reduce sensitivity to reinforcement rate. Chronic administration of a fixed dose of MDMA following each session reduced response rate but did not affect sensitivity to reinforcement rate. In combination with previous research, these results indicate that related drugs may have different effects on sensitivity to reinforcement and that these effects should be considered when interpreting disruptions to operant task performance caused by drug administration. (PsycINFO Database Record

Saccadic reaction time distributions follow the matching law in a concurrent variable interval reinforcement schedule

Saccadic reaction time distributions follow the matching law in a concurrent variable interval reinforcement schedule

Pausing as an operant: Choice and discriminated responding

The effects of intermittent schedules of reinforcement for pausing were evaluated in two experiments. In Experiment 1, across a series of conditions, a variable-interval (VI) baseline schedule, in which pigeons' key pecks produced food, alternated with conditions in which food was delivered according to a concurrent VI (for key pecking) tandem variable-time differential-reinforcement-of-other-behavior (DRO) 5-s schedule. Time spent pausing within a session was proportional to the reinforcement rates associated with the tandem schedule. To examine the control of pausing by antecedent events, Experiment 2 arranged a multiple schedule in which pecking and pausing in either component were maintained according to concurrent schedules like those used in the first experiment. The availability of reinforcement for pausing was signaled in one component while signals uncorrelated with reinforcement were presented in the other. Signaled reinforcement for pausing, relative to the presentation of uncorrelated signals, decreased time spent pausing, a finding consistent with existing research on the effects of signaled VI reinforcement for key pecking in pigeons. The results of the two experiments show that pausing functions as an operant in much the same way that discrete responses, like key pecks, do, and that pausing and other operants are similarly affected by both antecedent and consequent events.

Choice with frequently changing food rates and food ratios

In studies of operant choice, when one schedule of a concurrent pair is varied while the other is held constant, the constancy of the constant schedule may exert discriminative control over performance. In our earlier experiments, schedules varied reciprocally across components within sessions, so that while food ratio varied food rate remained constant. In the present experiment, we held one variable-interval (VI) schedule constant while varying the concurrent VI schedule within sessions. We studied five conditions, each with a different constant left VI schedule. On the right key, seven different VI schedules were presented in seven different unsignaled components. We analyzed performances at several different time scales. At the longest time scale, across conditions, behavior ratios varied with food ratios as would be expected from the generalized matching law. At shorter time scales, effects due to holding the left VI constant became more and more apparent, the shorter the time scale. In choice relations across components, preference for the left key leveled off as the right key became leaner. Interfood choice approximated strict matching for the varied right key, whereas interfood choice hardly varied at all for the constant left key. At the shortest time scale, visit patterns differed for the left and right keys. Much evidence indicated the development of a fix-and-sample pattern. In sum, the procedural difference made a large difference to performance, except for choice at the longest time scale and the fix-and-sample pattern at the shortest time scale.

Dynamical Regimes in Neural Network Models of Matching Behavior

The matching law constitutes a quantitative description of choice behavior that is often observed in foraging tasks. According to the matching law, organisms distribute their behavior across available response alternatives in the same proportion that reinforcers are distributed across those alternatives. Recently a few biophysically plausible neural network models have been proposed to explain the matching behavior observed in the experiments. Here we study systematically the learning dynamics of these networks while performing a matching task on the concurrent variable interval (VI) schedule. We found that the model neural network can operate in one of three qualitatively different regimes depending on the parameters that characterize the synaptic dynamics and the reward schedule: (1) a matching behavior regime, in which the probability of choosing an option is roughly proportional to the baiting fractional probability of that option; (2) a perseverative regime, in which the network tends to make always the same decision; and (3) a tristable regime, in which the network can either perseverate or choose the two targets randomly approximately with the same probability. Different parameters of the synaptic dynamics lead to different types of deviations from the matching law, some of which have been observed experimentally. We show that the performance of the network depends on the number of stable states of each synapse and that bistable synapses perform close to optimal when the proper learning rate is chosen. Because our model provides a link between synaptic dynamics and qualitatively different behaviors, this work provides us with insight into the effects of neuromodulators on adaptive behaviors and psychiatric disorders.

Covariance-Based Synaptic Plasticity in an Attractor Network Model Accounts for Fast Adaptation in Free Operant Learning

In free operant experiments, subjects alternate at will between targets that yield rewards stochastically. Behavior in these experiments is typically characterized by (1) an exponential distribution of stay durations, (2) matching of the relative time spent at a target to its relative share of the total number of rewards, and (3) adaptation after a change in the reward rates that can be very fast. The neural mechanism underlying these regularities is largely unknown. Moreover, current decision-making neural network models typically aim at explaining behavior in discrete-time experiments in which a single decision is made once in every trial, making these models hard to extend to the more natural case of free operant decisions. Here we show that a model based on attractor dynamics, in which transitions are induced by noise and preference is formed via covariance-based synaptic plasticity, can account for the characteristics of behavior in free operant experiments. We compare a specific instance of such a model, in which two recurrently excited populations of neurons compete for higher activity, to the behavior of rats responding on two levers for rewarding brain stimulation on a concurrent variable interval reward schedule (Gallistel et al., 2001). We show that the model is consistent with the rats' behavior, and in particular, with the observed fast adaptation to matching behavior. Further, we show that the neural model can be reduced to a behavioral model, and we use this model to deduce a novel "conservation law," which is consistent with the behavior of the rats.

Obese and lean Zucker rats demonstrate differential sensitivity to rates of food reinforcement in a choice procedure

The obese Zucker rat carries two recessive fa alleles that result in the expression of an obese phenotype. Obese Zuckers have higher food intake than lean controls in free-feed studies in which rats have ready access to a large amount of one type of food. The present study examined differences in obese and lean Zucker rats using concurrent schedules of reinforcement, which more ecologically models food selection using two food choices that have limited, but generally predictable availability. Lever-pressing of ten lean (Fa/Fa or Fa/fa) and ten obese (fa/fa) Zucker rats was placed under three concurrent variable interval variable interval (conc VI VI) schedules of sucrose and carrot reinforcement, in which the programmed reinforcer ratios for 45-mg food pellets were 5:1, 1:1, and 1:5. Allocation of responses to the two food alternatives was characterized using the generalized matching equation, which allows sensitivity to reinforcer rates (a) and bias toward one alternative (log k) to be quantified. All rats showed a bias toward sucrose, though there were no differences between lean and obese Zucker rats. In addition, obese Zucker rats exhibited higher sensitivity to reinforcement rates than lean rats. This efficient pattern of responding was related to overall higher deliveries of food pellets. Effective matching for food, then, may be another behavioral pattern that contributes to an obese phenotype.

Exploratory studies in sensory reinforcement in male rats: Effects of methamphetamine.

Understanding sensory reinforcement and the effects of stimulant drugs on sensory reinforcers is potentially important for understanding their influence on addiction processes. Experiment 1 explored the reinforcing properties of a visual stimulus and the effects of methamphetamine (METH) on responding maintained by a visual reinforcer (VRF) in male rats. Snout poke responses to the active alternative produced the VRF according to variable interval (VI) schedules of reinforcement, and responses to an inactive alternative had no programmed effect. Experiment 2 explored the effects of METH on choice between the VRF and a water reinforcer (H2ORF) using concurrent VI schedules in male rats. In Experiment 1, response-contingent onset of the VRF produced an increase in both the relative frequency and absolute rate of active responding. The rate of both active and inactive responding declined across the 40-min test sessions. METH did not differentially enhance active responding for the VRF. Instead, METH nondifferentially increased the rate of responding and attenuated the within-session decline of responding. In Experiment 2, METH differentially increased the rate of responding for the VRF relative to the H2ORF. The results of these exploratory experiments indicate that the reinforcing effects of the VRF were weak and transient. In addition, METH treatment increased responding, and the specificity of the enhancement of METH was dependent upon the testing conditions. Potential explanations of these differences, such as novelty and reinforcer type, are discussed.

PREFERENCE AS A FUNCTION OF ACTIVE INTERRESPONSE TIMES: A TEST OF THE ACTIVE TIME MODEL

In this article, we describe a test of the active time model for concurrent variable interval (VI) choice. The active time model (ATM) suggests that the time since the most recent response is one of the variables controlling choice in concurrent VI VI schedules of reinforcement. In our experiment, pigeons were trained in a multiple concurrent similar to that employed by Belke (1992), with VI 20-s and VI 40-s schedules in one component, and VI 40-s and VI 80-s schedules in the other component. However, rather than use a free-operant design, we used a discrete-trial procedure that restricted interresponse times to a range of 0.5-9.0 s. After 45 sessions of training, unreinforced probe periods were mixed with reinforced training periods. These probes paired the two stimuli associated with the VI 40-s schedules. Further, the probes were defined such that during their occurrence, interresponse times were either "short" (0.5-3.0 s) or "long" (7.5-9.0 s). All pigeons showed a preference for the stimulus associated with the relatively rich VI 40-s schedule--a result mirroring that of Belke. We also observed, though, that this preference was more extreme during long probes than during short probes--a result predicted by ATM.

Concurrent VR VI schedules: Primacy of molar control of preference and molecular control of response rates

In the first condition in Experiment 1, 6 rats were exposed to concurrent variable ratio (VR) 30, variable interval (VI) 30-sec schedules. In the next two conditions, the subjects were exposed to concurrent VI VI schedules and concurrent tandem VI-differential-reinforcement-of-high-rate VI schedules. For the latter conditions, the overall and relative reinforcer rates equaled those in the first condition. Only minor differences appeared in time allocation (a molar measure) across conditions. However, local response rate differences (a molecular measure) appeared between schedule types consistently with the interresponse times these schedules reinforced. In Experiment 2, these findings reappeared when the prior experiment was replicated with 5 subjects, except that the VR schedule was replaced by a VI plus linear feedback schedule. These results suggest that within the context tested, the molar factor of relative reinforcement rate controls preference, whereas the molecular factor of the relation between interresponse times and reinforcer probability controls the local response rate.

Coyotes (Canis latrans) and the matching law

Environmental change is accelerating due to anthropogenic influence. Species that have greater behavioral flexibility may be better adapted to exploit new or constantly changing habitats. There are few mammals and even fewer carnivores that better illustrate widespread adaptability and behavioral flexibility in the wake of human disturbance than coyotes (Canis latrans). Yet how such predators successfully track resources, enabling them to survive and extend their range in stochastic environments remains unknown. We tested eight wild-born, captive coyotes individually on an operant two-choice test using concurrent variable interval (VI) schedules. We held the overall rate of reinforcement constant but manipulated the ratio of reinforcement available from the two choices. We analyzed sensitivity of coyotes' tracking of resource change by fitting the generalized matching equation to the data. Results showed all coyotes efficiently tracked changes in reinforcement ratios within the first few sessions of each new condition and matched their relative rate of foraging time to relative rate of resources. We suggest the matching paradigm provides a methodology to explore coyote foraging strategies, and a potential framework to compare behavioral flexibility across species, by measuring the ability to track resource change under variable resource conditions.