Interreinforcement Interval Research Articles

A reinforcement schedule with two types of temporal information

Decades of research on reinforcement schedules have demonstrated that temporal information regarding the arrival or nonarrival of biologically significant events controls animal behavior. The fixed interval (FI) schedule, which is a time-based reinforcement schedule, suggests that responses are regulated by the time elapsed since the last reinforcement. This raises the question of how behavior is controlled when two distinct temporal cues regarding the availability of reinforcers are simultaneously presented. We modified the random interval (RI) schedule to create a reinforcement schedule incorporating two types of temporal information: the time elapsed since the last reinforcement, and the inter-reinforcement interval (IRI). Through long-term operant conditioning in rats, we examined how temporal information controls responses. When two temporal cues were available, we found that behavior was influenced by each cue. Moreover, we discovered that the effects of these cues could be analytically separated within the dynamics of response rates. The findings revealed that in controlling behavior, living organisms can utilize two temporal cues rather than relying on a single, clear cue.•We modified the random interval (RI) schedule to create a reinforcement schedule incorporating two types of temporal information.•When two temporal cues were available, we found that behavior was influenced by each cue.•Rats can utilize two temporal cues rather than relying on a single, clear cue.

Coordinated and individual responding in pairs of rats under variable interval schedules

The coordination of responses in pairs of rats was investigated by parametrically manipulating the duration of variable-interval (VI) schedule mean inter-reinforcer intervals (IRIs). Two conditions required coordination for a group of three pairs of rats (VI 5 s, 10 s, 20 s, 40 s, 80 s, 120 s, 160 s, 60 s, and 243 s) and individual responding for another group of three pairs of rats (VI 5, 10 s, 20 s, 40 s and 80 s). An inverted U-shaped function related responding measures and reinforcement rates: Overall response rates, coordinated episode rates and percentage of coordination increased with increasing reinforcement rates, and decreased at the highest rates of reinforcement in both groups. Rats exposed to VI schedules that required coordination generally showed higher rates and higher percentages of coordinated responses when compared to rats that responded under the individual schedules. Local rates of coordinated episodes showed that coordination occurred at a steady rate across the VI IRIs for the Coordination Group. The results extend previous research on the patterns of coordinated responses maintained by intermittent schedules of reinforcement.

Influence of Recent Trial History on Interval Timing.

Interval timing is involved in a variety of cognitive behaviors such as associative learning and decision-making. While it has been shown that time estimation is adaptive to the temporal context, it remains unclear how interval timing behavior is influenced by recent trial history. Here we found that, in mice trained to perform a licking-based interval timing task, a decrease of inter-reinforcement interval in the previous trial rapidly shifted the time of anticipatory licking earlier. Optogenetic inactivation of the anterior lateral motor cortex (ALM), but not the medial prefrontal cortex, for a short time before reward delivery caused a decrease in the peak time of anticipatory licking in the next trial. Electrophysiological recordings from the ALM showed that the response profiles preceded by short and long inter-reinforcement intervals exhibited task-engagement-dependent temporal scaling. Thus, interval timing is adaptive to recent experience of the temporal interval, and ALM activity during time estimation reflects recent experience of interval.

Cooperative responding in rats: II. Performance on fixed-ratio schedules of mutual reinforcement.

Coordinated responses of 5 dyads of rats were investigated under fixed-ratio (FR) schedules of mutual water reinforcement. Coordinated responding was defined as 2 consecutive lever-presses, 1 from each of 2 rats, occurring <.5 s apart. In the FR schedules, each coordinated episode was defined as 1 response in the FR sequence. The size of FR schedules was parametrically manipulated assuming the values of FR 1, 6, 12, 18, 24, 30, 50, and 9, in this order. Each FR remained in effect until responding reached stability. Under all conditions, pairs of rats received access to water simultaneously (mutual reinforcement). Rates and proportions of coordinated responding showed a bitonic inverted U-shaped function of ratio size. Postreinforcement pauses increased systematically as the interreinforcement interval increased. Local rates and proportions increased as a function of response location within ratios. Results of a control condition with relaxed temporal constraints for mutual reinforcement showed decreases in rates and proportion of coordinated responses, suggesting that the coordinated responses were controlled by the mutual reinforcement contingencies. The present experiment showed that coordinated responding is quantitatively affected by 3 properties of FR schedules: response requirement, reinforcement rates, and proximity to reinforcement.

Does DRO type matter?: Cycle versus resetting contingencies in eliminating responding

Following lever-press training on variable-interval 30-s schedules, rats were exposed to three types of schedules designed to eliminate lever pressing. The first two were variations on what is called a differential-reinforcement-of-other-behavior (DRO, “zero rate”, or [target response] omission schedule) schedule. Under both variations, reinforcers were scheduled to occur in different conditions after either fixed or variable inter-reinforcer intervals (IRIs). Under one variation each lever press reset the time interval (i.e., “resetting DRO”) and under the other a reinforcer delivery scheduled at the end of an IRI was cancelled by the first response during the IRI (i.e., “cycle DRO”). In another condition reinforcers were delivered independently of responding after fixed or variable time periods. Each of the DRO procedures reduced response rates quickly and to near zero across ten sessions. The time schedules also reduced responding, albeit at a slower rate. The results extend the analogy of omission training to freeoperant avoidance to shock-deletion avoidance schedules.

The effects of ratio and interval schedules on the location variability of pecking responses in pigeons: Application of Bayesian statistical model

Many studies have investigated how variability in animal behavior is induced by different reinforcement schedules. However, the animal species and experimental settings have varied between these studies. The present study investigated the variability of pigeons’ pecking location to operandum under fixed-ratio, fixed-interval, variable-ratio, and variable-interval schedules. A circular response area that was 22.4 cm in diameter was used so that the pecking responses would be effective over a wide range. Pigeons were exposed to a multiple-ratio yoked-interval schedule; the pairs of schedules were fixed-ratio and fixed-interval or variable-ratio and variable-interval. We used a Bayesian statistical approach to probabilistically evaluate the effects of reinforcement schedules on the variability. Bayesian statistical analysis showed the following: (1) interval schedules provided greater variability than did the ratio schedules under the same interreinforcement intervals, (2) fixed schedules provided greater variability than did variable schedules for both the ratio and interval schedule requirements, (3) the larger schedule requirement generated greater variability under the fixed schedules but this effect was not observed on the variable schedules, and (4) the variability did not change as time elapsed in each trial. These results suggested that each reinforcement schedule has a specific effect on the variability of the response location.

TRANSITIONS FROM RICH-TO-LEAN SCHEDULES INCREASE ATTACK IN A LABORATORY MODEL OF SOCIAL AGGRESSION IN PIGEONS: II. FIXED-INTERVAL SCHEDULES.

Pigeons key pecked under two-component multiple fixed-interval (FI) schedules. Each component provided a different reinforcer magnitude (small or large), signaled by the color of the key light. Attacks toward a live, protected target pigeon were measured. Large- (rich) and small- (lean) reinforcer components alternated irregularly such that four different interval types (transitions) between the size of the immediately preceding reinforcer and the size of the upcoming reinforcer occurred within each session: lean-to-lean, lean-to-rich, rich-to-lean, and rich-torich transitions. The FI for each component was the same within each phase, but was manipulated across phases. For all pigeons, more attack occurred following the presentations of the larger reinforcer (i.e., during rich-to-lean and rich-to-rich transitions). For 2 of the 3 pigeons, this effect was modulated by the size of the upcoming reinforcer; attack following larger reinforcers was elevated when the upcoming reinforcer was small (i.e., during rich-to-lean transitions). This rich-to-lean effect on attack diminished or disappeared as the length of the FI schedule was increased (i.e., control over attack by the upcoming reinforcer size diminished with increases in the inter-reinforcement interval). For all pigeons and at all FIs, however, postreinforcement pauses were longest during the rich-to-lean transitions. These data (1) are consistent with the notion that postreinforcement periods during intermittent schedules function aversively and, thus, can precipitate aggressive behavior, and (2) suggest that rich-to-lean conditions may be especially aversive. They also indicate, however, that aversive effects of rich-to-lean transitions may differ across fixed-ratio (FR) and FI schedules, and that variables controlling attacking and pausing may not be isomorphic between these different schedule types.

Modeling bout–pause response patterns in variable-ratio and variable-interval schedules using hierarchical Bayesian methodology

Streams of operant responses are arranged in bouts separated by pauses and differences in performance in reinforcement schedules with identical inter-reinforcement intervals (IRIs) are primarily due to differences in within–bout response rate, not in bout–initiation rate. The present study used hierarchical Bayesian modeling as a new method to quantify the properties of the response bout. A Bernoulli distribution was utilized to express the probability to stay in bout/pause, while a Poisson distribution was utilized to quantify the within–bout response rates. We compared bout/pause patterns between variable–ratio (VR) and variable–interval (VI) schedules across IRIs. The model estimation revealed no difference in within-bout staying probability between schedules. However, response rates of within–bout responses were higher in VR than VI across IRIs. These results are consistent with previous analyses using a log–survivor plot to describe within–bout responses and bouts–initiation responses. In addition, a simulation study was performed to examine how sensitively the model estimate the parameters according to different bout initiation rates. These result showed that the within–bout staying probability was affected by changes in between–bout while within–bout response rate parameters were not. This suggests model estimation robustness of the model estimation to dissociate within–bout and between–bout parameters during different reinforcement schedules.

Response-reinforcer dependency and resistance to change.

The effects of the response-reinforcer dependency on resistance to change were studied in three experiments with rats. In Experiment 1, lever pressing produced reinforcers at similar rates after variable interreinforcer intervals in each component of a two-component multiple schedule. Across conditions, in the fixed component, all reinforcers were response-dependent; in the alternative component, the percentage of response-dependent reinforcers was 100, 50 (i.e., 50% response-dependent and 50% response-independent) or 10% (i.e., 10% response-dependent and 90% response-independent). Resistance to extinction was greater in the alternative than in the fixed component when the dependency in the former was 10%, but was similar between components when this dependency was 100 or 50%. In Experiment 2, a three-component multiple schedule was used. The dependency was 100% in one component and 10% in the other two. The 10% components differed on how reinforcers were programmed. In one component, as in Experiment 1, a reinforcer had to be collected before the scheduling of other response-dependent or independent reinforcers. In the other component, response-dependent and -independent reinforcers were programmed by superimposing a variable-time schedule on an independent variable-interval schedule. Regardless of the procedure used to program the dependency, resistance to extinction was greater in the 10% components than in the 100% component. These results were replicated in Experiment 3 in which, instead of extinction, VT schedules replaced the baseline schedules in each multiple-schedule component during the test. We argue that the relative change in dependency from Baseline to Test, which is greater when baseline dependencies are high rather than low, could account for the differential resistance to change in the present experiments. The inconsistencies in results across the present and previous experiments suggest that the effects of dependency on resistance to change are not well understood. Additional systematic analyses are important to further understand the effects of the response-reinforcer relation on resistance to change and to the development of a more comprehensive theory of behavioral persistence.

Pre-asymptotic response rates as a function of the delay-of-reinforcement gradient summation for Catania’s Operant Reserve: A reply to Berg & McDowell (2011)

Catania’s Operant Reserve (COR; Catania, 2005) is a computational model of operant behaviour. In COR, responding depletes the reserve while reinforcement replenishes the reserve. The replenishment to the reserve depends on the location of responses within the most recent inter-reinforcement interval. The rule that maps replenishment to the responses within an inter-reinforcement interval is given by a delay-of-reinforcement gradient (DOR). Previous research (Berg and McDowell, 2011) found that non-linear DORs produce sigmoidal response rates on single variable-interval schedules while a linear DOR produces hyperbolic response rates. Berg and McDowell took these sigmoidal response rates as evidence against the viability of COR for modelling variable-interval performance. However, Berg and McDowell did not consider the effect of the slope of each DOR. The present conjecture is that the response rates from COR can be made hyperbolic by manipulating the area under the DOR. Our results show that the manipulation of the area under the DOR allows COR to produce hyperbolic response rates regardless of the parametric form of the DOR. Hence, if COR is to be used to model single variable-interval performance, the reinforcement effect over time should be taken into consideration through the manipulation of the area under the DOR.

Response-bout analysis of interresponse times in variable-ratio and variable-interval schedules

The purpose of this study was to analyze the response pattern difference between variable-ratio (VR) and variable-interval (VI) schedules of reinforcement by modeling interresponse time distributions of rats’ lever presses. All eight rats showed higher response rates under VR 30 than under inter-reinforcement intervals yoked VI. The 30 models consisting of single Exponential (with and without the lower limit on interresponse times), Weibull, Normal, Log-Normal or Gamma distributions, all possible two component combinations of those, and 3 and 4 component models consisting of Weibull, Normal, Log-Normal, or Gamma distribution combinations were compared. The 4 component Log-Normal model was the best in terms of the Akaike information criterion and visual inspection of fitting outcome. Parameter estimates for the L4 model showed that the VR-VI response rate difference is due to a difference in short interresponse times or within bout responses. This results suggests that the VR-VI response rate difference is not an indication of a difference in the overall tendency to respond but it is rather a difference in terms of what types of response patterns are engendered between the two schedules.

Role of the discriminative properties of the reinforcer in resurgence.

In three experiments with rat subjects, we examined the effects of the discriminative effects of reinforcers that were presented during or after operant extinction. Experiments 1 and 2 examined resurgence, in which an extinguished operant response (R1) recovers when a second behavior (R2) that has been reinforced to replace it is also placed in extinction. The results of Experiment 1 suggest that the amount of R1's resurgence is a decreasing linear function of the interreinforcement interval used during the reinforcement of R2. In Experiment 2, R1 was reinforced with one outcome (O1), and R2 was then reinforced with a second outcome (O2) while R1 was extinguished. In resurgence tests, response-independent (noncontingent) presentations of O2 prevented resurgence of R1, which otherwise occurred when testing was conducted with either no reinforcers or noncontingent presentations of O1. In Experiment 3, we then examined the effects of noncontingent O1 and O2 presentations after simple extinction in either the presence or the absence of noncontingent presentations of O2. Overall, the results are consistent with a role for the discriminative properties of the reinforcer in controlling operant behavior. In resurgence, the reinforcer used during response elimination provides a distinct context that controls the inhibition of R1. The results are less consistent with an alternative view emphasizing the disrupting effects of alternative reinforcement.

Phasic excitation of ventral tegmental dopamine neurons potentiates the initiation of conditioned approach behavior: parametric and reinforcement-schedule analyses.

Midbrain dopamine neurons are implicated in motivation and learning. However, it is unclear how phasic excitation of dopamine neurons, which is implicated in learning, is involved in motivation. Here we used a self-stimulation procedure to examine how mice seek for optogenetically-induced phasic excitation of dopamine neurons, with an emphasis on the temporal dimension. TH-Cre transgenic mice received adeno-associated viral vectors encoding channelrhodopsin-2 into the ventral tegmental area, resulting in selective expression of the opsin in dopamine neurons. These mice were trained to press on a lever for photo-pulse trains that phasically excited dopamine neurons. They learned to self-stimulate in a fast, constant manner, and rapidly reduced pressing during extinction. We first determined effective parameters of photo-pulse trains in self-stimulation. Lever-press rates changed as a function of the manipulation of pulse number, duration, intensity, and frequency. We then examined effects of interval and ratio schedules of reinforcement on photo-pulse train reinforcement, which was contrasted with food reinforcement. Reinforcement with food inhibited lever pressing for a few seconds, after which pressing was robustly regulated in a goal-directed manner. In contrast, phasic excitation of dopamine neurons robustly potentiated the initiation of lever pressing; however, this effect did not last more than 1 s and quickly diminished. Indeed, response rates markedly decreased when lever pressing was reinforced with inter-reinforcement interval schedules of 3 or 10 s or ratio schedules requiring multiple responses per reinforcement. Thus, phasic excitation of dopamine neurons briefly potentiates the initiation of approach behavior with apparent lack of long-term motivational regulation.

Delayed reinforcement and fixed-ratio performance

Effects of delayed reinforcement on fixed-ratio (FR) maintained responding of pigeons were investigated. In Experiments 1-3, the delay of reinforcement was increased across successive sessions until pigeons paused for 300 s. Both signaled and unsignaled delays were studied across different conditions. Overall response rates and run rates (timed from the first to last response of a ratio) decreased and postreinforcement pauses increased as delays increased in each experiment. As delays increased, the likelihood of pausing during a ratio run also increased. When these measures were plotted as a function of obtained delays, signaled delays had less of an effect on the above measures than did unsignaled ones. In Experiment 2, delays had a greater effect on the above measures than did a control condition arranging equivalent interreinforcer intervals to those accompanying the delays. Experiments 3 and 4 examined the generality of the effects obtained in the first two experiments. In Experiment 3, delays imposed on FR or yoked-interval schedules had similar behavioral effects. In Experiment 4, effects similar to those found in Experiments 1-3 for 1, 10, and 20-s delays imposed on FR 50 schedules were found when the FR requirement increased across sessions. Despite the different contingencies relating response rate and reinforcement rates on interval and ratio schedules, delays of reinforcement generally affect performance on these schedules similarly.

Effects of SKF-83566 and haloperidol on performance on progressive ratio schedules maintained by sucrose and corn oil reinforcement: quantitative analysis using a new model derived from the Mathematical Principles of Reinforcement (MPR)

RationaleMathematical models can assist the interpretation of the effects of interventions on schedule-controlled behaviour and help to differentiate between processes that may be confounded in traditional performance measures such as response rate and the breakpoint in progressive ratio (PR) schedules.ObjectiveThe effects of a D1-like dopamine receptor antagonist, 8-bromo-2,3,4,5-tetrahydro-3-methyl-5-phenyl-1H-3-benzazepin-7-ol hydrobromide (SKF-83566), and a D2-like receptor antagonist, haloperidol, on rats’ performance on PR schedules maintained by sucrose and corn oil reinforcers were assessed using a new model derived from Killeen’s (Behav Brain Sci 17:105–172, 1994) Mathematical Principles of Reinforcement.MethodSeparate groups of rats were trained under a PR schedule using sucrose or corn oil reinforcers. SKF-83566 (0.015 and 0.03 mg kg−1) and haloperidol (0.05 and 0.1 mg kg−1) were administered intraperitoneally (five administrations of each treatment). Running and overall response rates in successive ratios were analysed using the new model, and estimates of the model’s parameters were compared between treatments.ResultsHaloperidol reduced a (the parameter expressing incentive value) in the case of both reinforcers, but did not affect the parameters related to response time and post-reinforcement pausing. SKF-83566 reduced a and k (the parameter expressing sensitivity of post-reinforcement pausing to the prior inter-reinforcement interval) in the case of sucrose, but did not affect any of the parameters in the case of corn oil.ConclusionsThe results are consistent with the hypothesis that blockade of both D1-like and D2-like receptors reduces the incentive value of sucrose, whereas the incentive value of corn oil is more sensitive to blockade of D2-like than D1-like receptors.

Temporal control in chained fixed-ratio, fixed-interval schedules.

Four rats were subjected to chained fixed-ratio (FR), fixed-interval (FI) schedules of reinforcement (chain FR 5 FI). A FR schedule at one lever produced a discriminative stimulus (i.e., light) associated with an FI schedule of primary reinforcement (water) at the second response lever. The FR schedule was kept constant, whereas the FI length was changed from 10 to 60 s under five different experimental conditions. Increases in the FI length resulted in increases in pre-ratio pauses, but pauses in the FI tended to be a constant percentage of FI length. Data from this experiment indicate that pre-ratio pauses are also a function of the interreinforcement interval (IRI). Data from three experiments with chained FR 5 FI 60-s schedules indicate that pausing in the FI component of chained FR FI schedules with the FI as the second component of the chain may tend to disappear as the IRI duration increases.

Choices between segmented and unsegmented schedules and the self-control paradigm

A concurrent-chains procedure was used to examine choice between a segmented (two- or three-terminal-link segments schedules) and an unsegmented schedule (simple schedule) in terminal links with equal interreinforcement intervals. In most such experiments, preference for the unsegmented schedule has been found, but in a recent study with humans (Alessandri et al., 2010) a reversal in preference was found when, in the segmented schedule, the terminal link segmenting stimulus was presented briefly and closer to food delivery such that the early terminal link stimulus was temporally closer to the food delivery. In Experiment 1, an attempt to replicate this latter effect with pigeons was unsuccessful but this outcome was consistent with an account in terms of a self-control contingency involving conditioned reinforcers. According to this account, the unsegmented alternative consisted of an immediate, smaller presentation of a conditioned reinforcer (i.e., the impulsive, and thus usually the preferred, option in several experiments) and the segmented schedule led to a delayed, larger conditioned reinforcer (i.e., the self-control option). In Experiment 2, a reversal of preference toward the segmented schedule was found when a delay was added to both terminal links between the reinforced initial-link response and the onset of the corresponding terminal link stimulus. This result is consistent with a similar effect found with primary reinforcers in the self-control literature suggesting the utility of self-control as an account of preferences for unsegmented terminal links of concurrent chains schedules.

A theory of behaviour on progressive ratio schedules, with applications in behavioural pharmacology

Mathematical principles of reinforcement (MPR) provide the theoretical basis for a family of models of schedule-controlled behaviour. A model of fixed-ratio schedule performance that was applied to behaviour on progressive ratio (PR) schedules showed systematic departures from the data. This study aims to derive a new model from MPR that will account for overall and running response rates in the component ratios of PR schedules, and their decline toward 0, the breakpoint. The role of pausing is represented in a real-time model containing four parameters: T (0) and k are the intercept and slope of the linear relation between post-reinforcement pause duration and the prior inter-reinforcer interval; a (specific activation) measures the incentive value of the reinforcer; δ (response time) sets biomechanical limits on response rate. Running rate is predicted to decrease with negative acceleration as ratio size increments, overall rate to increase and then decrease. Differences due to type of progression are explained as hysteresis in the control by reinforcement rates. Re-analysis of extant data focuses on the effects of acute treatment with antipsychotic drugs, lesions of the nucleus accumbens core, and destruction of orexinergic neurones of the lateral hypothalamus. The new model resolves some anomalies evident in earlier analyses, and provides new insights to the results of these interventions. Because they can render biologically relevant parameters, mathematical models can provide greater power in interpreting the effects of interventions on the processes underlying schedule-controlled behaviour than is possible for first-order data such as the breakpoint.

Reinforcement: food signals the time and location of future food.

It has long been understood that food deliveries may act as signals of future food location, and not only as strengtheners of prefood responding as the law of effect suggests. Recent research has taken this idea further--the main effect of food deliveries, or other "reinforcers", may be signaling rather than strengthening. The present experiment investigated the ability of food deliveries to signal food contingencies across time after food. In Phase 1, the next food delivery was always equally likely to be arranged for a left- or a right-key response. Conditions were arranged such that the next food delivery was likely to occur either sooner on the left (or right) key, or sooner on the just-productive (or not-just-productive) key. In Phase 2, similar contingencies were arranged, but the last-food location was signaled by a red keylight. Preference, measured in 2-s bins across interfood intervals, was jointly controlled by the likely time and location of the next food delivery. In Phase 1, when any food delivery signaled a likely sooner next food delivery on a particular key, postfood preference was strongly toward that key, and moved toward the other key across the interreinforcer interval. In other conditions in which food delivery on the two keys signaled different subsequent contingencies, postfood preference was less extreme, and quickly moved toward indifference. In Phase 2, in all three conditions, initial preference was strongly toward the likely-sooner food key, and moved to the other key across the interfood interval. In both phases, at a more extended level of analysis, sequences of same-key food deliveries caused a small increase in preference for the just-productive key, suggesting the presence of a "reinforcement effect", albeit one that was very small.

THE EFFECTS OF A LOCAL NEGATIVE FEEDBACK FUNCTION BETWEEN CHOICE AND RELATIVE REINFORCER RATE

Four pigeons were trained on two-key concurrent variable-interval schedules with no changeover delay. In Phase 1, relative reinforcers on the two alternatives were varied over five conditions from .1 to .9. In Phases 2 and 3, we instituted a molar feedback function between relative choice in an interreinforcer interval and the probability of reinforcers on the two keys ending the next interreinforcer interval. The feedback function was linear, and was negatively sloped so that more extreme choice in an interreinforcer interval made it more likely that a reinforcer would be available on the other key at the end of the next interval. The slope of the feedback function was -1 in Phase 2 and -3 in Phase 3. We varied relative reinforcers in each of these phases by changing the intercept of the feedback function. Little effect of the feedback functions was discernible at the local (interreinforcer interval) level, but choice measured at an extended level across sessions was strongly and significantly decreased by increasing the negative slope of the feedback function.