Variable-interval Schedules Research Articles

Relationship between reinforcement rate and response rate in Pavlovian and operant conditioning with compound stimuli in rats

Harris, Andrew, and Livesey (2012) proposed that response rate (R) to a stimulus (e.g., A) of a compound (e.g., AB) would be directly related to the difference between reinforcement rate (r) during AB and the other stimulus (e.g., B). To explore the extent of this proposal to operant conditioning, three experiments were carried out. Experiment 1 replicated Harris et al. (2012 – Experiment 1) using sucrose as US in Pavlovian conditioning. In Experiment 2 rats pressed a lever under a variable interval schedule in two phases. In Training, responses during a compound stimulus AB100% were reinforced 100%, responses during B50% and C50% 50% and responses during D25%, 25% of the trials. Then, nonreinforced A trials were introduced (Probe). Results from Experiment 2 showed that r controlled the R to AB100%, B50%, C50% and D25% during training and probe, however, were inconclusive about r and R relationship during A. In Experiment 3, responses during A100% were reinforced 100% and during B50% and C50% 50% of the trials (Training). Next, nonreinforced BC trials were presented (Probe). Results from Experiment 3 showed that response rate during BC were similar to A100%, but higher than B50% and C50%, suggesting summation of B50% and C50% reinforcement rates. Overall, the results showed that Harris et al. (2012) proposal extends to discriminative operant relations.

Zebrafish choice behavior is sensitive to reinforcer rate, immediacy, and magnitude ratios.

Behavioral flexibility has, in part, been defined by choice behavior changing as a function of changes in reinforcer payoffs. We examined whether the generalized matching law quantitatively described changes in choice behavior in zebrafish when relative reinforcer rates, delays/immediacy, and magnitudes changed between two alternatives across conditions. Choice was sensitive to each of the three reinforcer properties. Sensitivity estimates to changes in relative reinforcer rates were greater when 2 variable-interval schedules were arranged independently between alternatives (Experiment 1a) than when a single schedule pseudorandomly arranged reinforcers between alternatives (Experiment 1b). Sensitivity estimates for changes in relative reinforcer immediacy (Experiment 2) and magnitude (Experiment 3) were similar but lower than estimates for reinforcer rates. These differences in sensitivity estimates are consistent with studies examining other species, suggesting flexibility in zebrafish choice behavior in the face of changes in payoff as described by the generalized matching law.

Behavioral effects of changeover delays in a concurrent-chains schedule with different delays and magnitudes of reinforcement

The present study examined the effects of a changeover delay (COD) on the choice between smaller, shorter delayed reinforcement and larger, longer delayed reinforcement in a complex concurrent-chains schedule. Four pigeons were exposed to three conditions in an ABA or BAB reversal design. A COD of 2 seconds was included in the initial links during condition B. The concurrent-chains schedule had identical variable-interval schedules in the initial links and different delays and magnitudes of reinforcement that were programmed according to a fixed-ratio 10 (FR10) schedule in the terminal links. Each experimental condition lasted for two long sessions and five short sessions. Changeover rates were lower, and preference for the larger, longer delayed reinforcer was more pronounced during conditions with the programmed 2-second COD. The analysis of initial pauses and running rates during the FR10 schedules of the terminal links revealed no systematic effects of the COD on behavior in the terminal links. The COD had similar effects on choice performance in this modified concurrent-chain procedure as on simple concurrent performance.Keywords: Choice behavior, concurrent-chains, COD, self-control paradigm, pigeons.

Stimulus generalization of behavioral history: Interspecies generality and persistence of generalization gradients.

Four pigeons were exposed to a tandem variable-interval (VI) fixed-ratio (FR) schedule in the presence of a 50-pixel (about 15 mm) square or an 80-pixel (about 24 mm) square and to a tandem VI differential-reinforcement-of-low-rate (DRL) schedule when a second 80-pixel or 50-pixel square was present. The values of the VI and FR schedules were adjusted to equate reinforcement rates in the two tandem schedules. Following this, a square-size continuum generalization test was administered under a fixed-interval (FI) schedule or extinction. In the first testing session, response frequency was a graded function of the similarity of the test stimuli to the training stimuli for all pigeons. These systematic generalization gradients persisted longer under the FI schedule than under extinction.

Matching, induction, and covariance with mixed response-contingent food and noncontingent food.

The multiscale molar view of behavior is based on three basic laws of behavior: the Law of Allocation, the Law of Induction, and the Law of Covariance. Experiments that mix response-contingent food with noncontingent food shed light on these three laws. Food, like other phylogenetically important events, induces various activities that compete in allocation. Quantitative accounts represent induction with power functions. These power functions define activities' competitive weights, and relative time allocation among activities matches relative competitive weight. Behavior-food covariance determines which activities are induced. Phylogenetic (behavior-fitness) covariance determines which adjunctive activities are induced. Ontogenetic covariance may be represented in feedback functions. Feedback functions for variable-interval schedules may be observed even when overlaid by noncontingent food deliveries. Equations derived from the three laws describe responding in experiments with mixed response-contingent and noncontingent food. Equations derived here accounted for responding in three data sets: (a) from Rachlin and Baum(1972); (b) a new data set in which overall food rate was fixed while the proportion of response-contingent and noncontingent food varied; and (c) a new data set in which food occurred according to various variable-interval schedules. The same pigeons served throughout. All results were accommodated by the derived equations.

Longer operant lever-press duration requirements induce fewer but longer response bouts in rats.

Operant behavior is organized in bouts that are particularly visible under variable-interval (VI) schedules of reinforcement. Previous research showed that increasing the work required to produce a response decreases the rate at which bouts are emitted and increases the minimum interresponse time (IRT). In the current study, the minimum effective IRT was directly manipulated by changing the minimum duration of effective lever presses reinforced on a VI 40-s schedule. Contrary to assumptions of previous models, response durations were variable. Response durations were typically 0.5 s greater than the minimum duration threshold; durations that exceeded this threshold were approximately log-normally distributed. As the required duration threshold increased, rats emitted fewer but longer bouts. This effect may reflect an effort-induced reduction in motivation and a duration-induced facilitation of a response-outcome association.

COMPARING COORDINATED RESPONDING IN PAIRS OF RATS WHEN BOTH OPERANTS AND AGGREGATE PRODUCTS ARE SELECTED BY INTERMITTENT TEMPORAL CONSEQUENCES

Three pairs of rats worked in two adjacent operant chambers separated by a Plexiglas wall. In some experimental conditions, lever-presses were independent operants maintained by either a fixed-interval (FI) or variable interval (VI) schedule. In other conditions, each rat lever- press was part of two interlocked behavior contingencies as part of a metacontingency: its aggregate product, a brief flash of light, was maintained either by a FI or a VI schedule of mutual water presentation. The results clearly show that the interlocking of behavioral contingencies in the design of an experimental metacontingency established a pattern of cooperation between the rats.

Resurgence without overall worsening of alternative reinforcement

Three experiments were conducted with pigeons to assess discriminated periods of nonreinforcement as precipitators of resurgence. Each experiment occurred in three phases. In the Training phase, key-pecking was reinforced according to variable-interval schedules that alternated between two response keys (Experiment 1) or were concurrently available on two response keys (Experiments 2a & 2b). In the Alternative-Reinforcement phase, responding to one key was extinguished, while that to the other was reinforced according to tandem schedules. These then were replaced by chained schedules with the same programmed reinforcement rate in the Resurgence-Test phase. Resurgence occurred both when the signaled period of nonreinforcement was a darkened keylight in the terminal link of the chain schedule (Experiment 1) and a darkened keylight (Experiment 2a) or keylight color change (Experiment 2b) in the initial link of the chain schedule. Thus, signaled periods of extinction, without accompanying reductions in reinforcement rate, precipitated resurgence, suggesting that resurgence is not the result of worsening of overall reinforcement conditions, but also occurs when local conditions of reinforcement are worsened.

Early extinction effects following intermittent reinforcement: Little evidence of extinction bursts.

The occurrence of extinction bursts-transient increases in response rate in excess of those observed in baseline during the period immediately following discontinuation of reinforcement of a response-was examined. In Experiment 1, key pecking of pigeons was reinforced according to a multiple schedule in which a variable-ratio schedule alternated with an interval schedule in which the reinforcers were yoked to the preceding variable-ratio component. In Experiments 2 and 3, rats were screened such that the lever-press response rates of different rats maintained by variable-interval schedules were either relatively high or relatively low. Following these baseline conditions, in Experiments 1 and 2 responding was extinguished by eliminating the food reinforcer and in Experiment 3 by removing the response-reinforcer dependency. Responses immediately following extinction implementation were examined. Response increases relative to baseline during the first 20 min of a 324.75-min extinction session (Experiment 1) or during the first 30-min extinction session (Experiments 2 and 3) were rare and unsystematic. The results (a) reinforce earlier meta-analyses concluding that extinction bursts may be a less ubiquitous early effect of extinction than has been suggested and (b) invite further experimentation to establish their generality as a function of preceding reinforcement conditions.

Assessing the motivation to learn in cattle

Cognitive challenges may provide a form of enrichment to improve the welfare of captive animals. Primates, dolphins, and goats will voluntarily participate in learning tasks suggesting that these are rewarding, but little work has been conducted on livestock species. We investigated the motivation of 10 pairs of Holstein heifers to experience learning opportunities using a yoked design. All heifers were trained to perform an operant response (nose touch) on a variable interval schedule. Learning heifers then performed this response to access a discrimination learning task in which colour and texture of feed-bin lids signified a preferred reward (grain) vs. a non-preferred reward (straw). Control heifers received the same feed without a choice of bins or association of feed with lid type. Learning heifers approached the target to begin sessions faster (p = 0.024) and tended to perform more operant responses (p = 0.08), indicating stronger motivation. Treatments did not differ in the frequency with which heifers participated in voluntary training sessions. We conclude that heifers are motivated to participate in learning tasks, but that aspects of the experience other than discrimination learning were also rewarding. Cognitive challenges and other opportunities to exert control over the environment may improve animal welfare.

Repeated resurgence with and without a context change

Resurgence is the reoccurrence of a target response when reinforcement for a more recently reinforced alternative response is eliminated or reduced. The present study arranged two successive three-phase procedures to assess whether resurgence decreases with repeated assessments. Moreover, we arranged a contextual change from the first to second assessment for some groups. Phase 1 reinforced a target response on a touchscreen computer with typically developing adults as participants according to either variable-ratio or variable-interval schedules of reinforcement. Phase 2 extinguished target responding and reinforced alternative responding. Phase 3 tested for resurgence by extinguishing alternative responding. Resurgence reliably occurred in all tests and decreased from the first to second exposure to the procedures but there were no effects of context change. Therefore, repeated exposures to resurgence tests reduced those effects but contextual changes had no effect.

Matching theory and induction explain operant performance.

Matching theory is a general framework for understanding allocation of behavior among activities. It applies to choice in concurrent schedules and was extended to single schedules by assuming that other unrecorded behavior competes with operant behavior. Baum and Davison (2014) found that the competing activities apparently are induced by the "reinforcers" (phylogenetically important events, e.g., food) according to power functions. Combined with power-function induction, matching theory provides new equations with greater explanatory power. Four pigeons were exposed to conditions in which 7 different schedules of food delivery were presented within each experimental session. We replicated earlier results with variable-interval schedules: (a) a negatively accelerated increase of peck rate as food rate increased in the low range of food rates; (b) an upturn in pecking at higher rates; and (c) a downturn in pecking at extremely high food rates. When the contingency between pecking and food was removed, the food continued to induce pecking, even after 20 sessions with no contingency. A ratio schedule inserted in place of 1 variable-interval schedule maintained peck rates comparable to peck rates maintained by short interval schedules. We explained the results by fitting equations that combined matching theory, competition, and induction.

The effects of ratio and interval schedules on the location variability of pecking responses in pigeons: Application of Bayesian statistical model

Many studies have investigated how variability in animal behavior is induced by different reinforcement schedules. However, the animal species and experimental settings have varied between these studies. The present study investigated the variability of pigeons’ pecking location to operandum under fixed-ratio, fixed-interval, variable-ratio, and variable-interval schedules. A circular response area that was 22.4 cm in diameter was used so that the pecking responses would be effective over a wide range. Pigeons were exposed to a multiple-ratio yoked-interval schedule; the pairs of schedules were fixed-ratio and fixed-interval or variable-ratio and variable-interval. We used a Bayesian statistical approach to probabilistically evaluate the effects of reinforcement schedules on the variability. Bayesian statistical analysis showed the following: (1) interval schedules provided greater variability than did the ratio schedules under the same interreinforcement intervals, (2) fixed schedules provided greater variability than did variable schedules for both the ratio and interval schedule requirements, (3) the larger schedule requirement generated greater variability under the fixed schedules but this effect was not observed on the variable schedules, and (4) the variability did not change as time elapsed in each trial. These results suggested that each reinforcement schedule has a specific effect on the variability of the response location.

Lesion of striatal patches disrupts habitual behaviors and increases behavioral variability.

Habits are automated behaviors that are insensitive to changes in behavioral outcomes. Habitual responding is thought to be mediated by the striatum, with medial striatum guiding goal-directed action and lateral striatum promoting habits. However, interspersed throughout the striatum are neurochemically differing subcompartments known as patches, which are characterized by distinct molecular profiles relative to the surrounding matrix tissue. These structures have been thoroughly characterized neurochemically and anatomically, but little is known regarding their function. Patches have been shown to be selectively activated during inflexible motor stereotypies elicited by stimulants, suggesting that patches may subserve habitual behaviors. To explore this possibility, we utilized transgenic mice (Sepw1 NP67) preferentially expressing Cre recombinase in striatal patch neurons to target these neurons for ablation with a virus driving Cre-dependent expression of caspase 3. Mice were then trained to press a lever for sucrose rewards on a variable interval schedule to elicit habitual responding. Mice were not impaired on the acquisition of this task, but lesioning striatal patches disrupted behavioral stability across training, and lesioned mice utilized a more goal-directed behavioral strategy during training. Similarly, when mice were forced to omit responses to receive sucrose rewards, habitual responding was impaired in lesioned mice. To rule out effects of lesion on motor behaviors, mice were then tested for impairments in motor learning on a rotarod and locomotion in an open field. We found that patch lesions partially impaired initial performance on the rotarod without modifying locomotor behaviors in open field. This work indicates that patches promote behavioral stability and habitual responding, adding to a growing literature implicating striatal patches in stimulus-response behaviors.

EFFECTS OF RESPONSE-REINFORCER RELATIONS ON THE MAINTENANCE OF COORDINATED RESPONDING UNDER SCHEDULES OF SIMULTANEOUS REINFORCEMENT

The effects of response-reinforcer relations on coordinated responding were investigated. Coordinated responding was defined as two lever presses, one by each rat that occurred within 500 ms of one another. Four conditions were arranged in an ABCB design. Coordinated responding was reinforced according to a fixed-ratio 6 (FR 6) schedule in Condition A. In Condition B, a response by each rat was required, independent of their temporal proximity, to produce water delivery under a variable-interval schedule. Condition C was a replication of Condition B, except that coordinated responding was required for reinforcer deliveries. All conditions involved simultaneous reinforcement, that is the rats received access to reinforcers at the same time. The results extended previous findings by demonstrating the requirement of coordinated responses to produce reinforcement affected both coordinated response rates and the proportion of such responses relative to the total responses in a session, in that both measures were higher in Conditions A and C than in Condition B. There also was control of the temporal distribution of coordinated responding by the type of schedule (FR or VI): A “break-and-run” pattern was observed under the FR schedule, and a constant response rate was observed under the VI schedule.

Contingency, contiguity, and causality in conditioning: Applying information theory and Weber's Law to the assignment of credit problem.

Contingency is a critical concept for theories of associative learning and the assignment of credit problem in reinforcement learning. Measuring and manipulating it has, however, been problematic. The information-theoretic definition of contingency-normalized mutual information-makes it a readily computed property of the relation between reinforcing events, the stimuli that predict them and the responses that produce them. When necessary, the dynamic range of the required temporal representation divided by the Weber fraction gives a psychologically realistic plug-in estimates of the entropies. There is no measurable prospective contingency between a peck and reinforcement when pigeons peck on a variable interval schedule of reinforcement. There is, however, a perfect retrospective contingency between reinforcement and the immediately preceding peck. Degrading the retrospective contingency by gratis reinforcement reveals a critical value (.25), below which performance declines rapidly. Contingency is time scale invariant, whereas the perception of proximate causality depends-we assume-on there being a short, fixed psychologically negligible critical interval between cause and effect. Increasing the interval between a response and reinforcement that it triggers degrades the retrograde contingency, leading to a decline in performance that restores it to at or above its critical value. Thus, there is no critical interval in the retrospective effect of reinforcement. We conclude with a short review of the broad explanatory scope of information-theoretic contingencies when regarded as causal variables in conditioning. We suggest that the computation of contingencies may supplant the computation of the sum of all future rewards in models of reinforcement learning. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Is increased sensitivity to punishment a common characteristic of attention deficit/hyperactivity disorder? An experimental study of response allocation in Japanese children

Research on motivational processes in attention deficit/hyperactivity disorder (ADHD) focuses on reward. Studies with punishment are limited and findings mixed. This study evaluated the effects of punishment on response allocation in Japanese children with and without ADHD. Thirty-four children meeting DSM-IV criteria for ADHD and 59 typically developing control-group children completed an operant task in which they choose between playing two simultaneously available games. Reward was arranged symmetrically across the games under concurrent variable interval schedules. Asymmetric punishment schedules were superimposed with responses on one game punished four times as often as responses on the other. Children with ADHD showed greater behavioral sensitivity to punishment than controls. They allocated significantly more responses to the less frequently punished alternative and were more likely to play this game on consecutive trials and responded more slowly to the more punished game. Control group children allocated their responses evenly across games. Punishment exerted greater control over the behavior of Japanese children with ADHD than controls, similar to findings with children from Western countries, suggesting this is a common characteristic of the disorder. The behavior of typically developing Japanese children, while demonstrating awareness of punishment, was not controlled by the frequency of its occurrence.

Reversing Breathing Paralysis through Optimization of Intermittent Hypoxia Treatment after Cervical Spinal Cord Injury

Spinal cord injury (SCI) most commonly occurs at the cervical level and can interrupt descending neural pathways, causing paralysis of the diaphragm, as well as profound breathing motor difficulties which threaten survival and greatly decrease quality of life. Intermittent hypoxia (IH) treatment is often utilized in preclinical models to attenuate breathing motor deficits resulting from cervical SCI by inducing a prolonged increase in respiratory motor output known as long term facilitation (LTF), a form of breathing motor plasticity. IH typically consists of the repeated, alternating exposure of a subject to consistent and equal 5‐minute periods of hypoxia and normoxia and thus can be fittingly termed fixed interval intermittent hypoxia (FIH). FIH exhibits similarity to the psychological construct of operant conditioning in which the increased incidence and persistence of a desired, spontaneous behavior is trained through reinforcement. As such, each interval of hypoxia can be construed as the period during which the subject responds with heightened respiratory drive and is subsequently reinforced by an interval of normoxia. Provided that IH is a form of operant conditioning, it can be optimized through application of seminal psychological findings which established variable interval schedules of reinforcement as more effective than fixed interval schedules for learning. Therefore, using the duration of the hypoxic interval as our independent variable, we hypothesize that varied interval intermittent hypoxia (VIH) treatment will induce a greater, more prolonged increase in respiratory motor output than FIH after injury. We utilized the C2 hemisection injury model in rats and treated with VIH or FIH for 5 days at 1‐week post‐injury. Following treatment, we conducted diaphragm electromyograph recordings to assess breathing motor recovery within each animal by comparing baseline activity to maximal output induced by nasal occlusion, occurring immediately after cessation of IH treatment. Preliminary results show that 1‐week post‐injury VIH treated animals exhibited recovery on average equaling 33.87±7.89% as compared to the 19.28±6.02% recovery in FIH treated animals (differences nonsignificant by unpaired t‐test, p>0.05 with uncertainty shown as standard error of the mean). These data suggest that VIH may induce more increased and prolonged recovery than FIH in post‐injury models. Ongoing work includes evaluation of VIH treatment in animals at 8 weeks after injury and will expand outcome measurement timepoints to include 1 week after cessation of treatment to interrogate the persistence of induced recovery. Additionally, further exploration will focus on the molecular markers present within the phrenic motor nucleus of the cervical spinal cord and on the development of IH paradigms based on breathing frequency.Support or Funding InformationNINDS R01NS101105 (Warren J. Alilain)This abstract is from the Experimental Biology 2019 Meeting. There is no full text article associated with this abstract published in The FASEB Journal.

The nanoeconomics of concurrent choice behavior.

We introduce to behavior analysis a way of analyzing choice behavior that exploits recent developments in nanoeconomics, financial economics, and econometrics. A response return, modeled on an economic return, is the log differenced count of responses allocated to each of two alternatives during a short time window, compared with that in the immediately preceding window. The response return is a new dependent variable which offers a novel and useful way of looking at operant behavior, especially at the molecular level of analysis. The response-return series is a near-instantaneous measure of an organism's dynamic preferences for each of two alternatives. Analyzing such a series requires time-analytic techniques, including Auto-Regressive Conditional Heteroskedastic (ARCH) models. We illustrate these techniques by analyzing choices between combinations of arithmetic and exponential variable-interval schedules with pigeon subjects. All response-return series were well-fitted by one of three ARCH-family models. The fitted models were differentially sensitive to transition versus steady-state data samples. The particular insights that the ARCH analyses offer are improved understanding of the effects of the instantaneous effects of reinforcers and their absence, of how the distribution of reinforcers in time affects choice, and of the differences between choice in transition and at steady state.

Differential effects of ADHD medications on impulsive action in the mouse 5-choice serial reaction time task

Aberrant impulsivity is found in a number of psychiatric disorders including attention deficit hyperactivity disorder (ADHD). The 5-choice serial reaction time task (5-CSRTT) is a paradigm commonly used to assess impulsive control. We recently developed a protocol to habituate mice to a variable intertrial interval (vITI) schedule before assessing pharmacological effects on “waiting” impulsivity. This study aimed to develop on that initial investigation by testing the effects of three conventional ADHD medications. Consistent premature response rates were achieved in male C57BL/6 J mice in the first week out of 15 vITI (5-, 10- or 15-s) days (four training days followed by one drug treatment day per week for three weeks) before each drug study commenced. The effects of atomoxetine (1, 3 mg/kg), methylphenidate (1, 2 mg/kg) and guanfacine (0.03, 0.1 mg/kg) were investigated using a Latin-square design. High- and low-impulsive subgroups were determined based on initial training day data before the drug studies initiated. Both 1 and 3 mg/kg atomoxetine reduced premature responding at the 10- (P < 0.001, P < 0.05) and 15-s (P < 0.001) lengths. 2 mg/kg methylphenidate increased impulsive action at the longest 15-s ITI (P < 0.05). Guanfacine exerted no effects on premature responding rates at any dose or ITI. Impulsive subgrouping did not reveal any specific drug by subgroup effects. This study indicates that these current ADHD medications have differential effects on impulsive action. In summary, this protocol is a useful preclinical model for testing potential treatments for disorders with dysfunctional impulsive control.