Random-interval Schedules Of Reinforcement Research Articles

Random interval schedule of reinforcement influences punishment resistance for cocaine in rats

In an animal model of compulsive drug use, a subset of rats continues to self-administer cocaine despite footshock consequences and is considered punishment resistant. We recently found that punishment resistance is associated with habits that persist under conditions that typically encourage a transition to goal-directed control. Given that random ratio (RR) and random interval (RI) schedules of reinforcement influence whether responding is goal-directed or habitual, we investigated the influence of these schedules on punishment resistance for cocaine or food. Male and female Sprague Dawley rats were trained to self-administer either intravenous cocaine or food pellets on a seeking-taking chained schedule of reinforcement, with the seeking lever requiring completion of either an RR20 or RI60 schedule. Rats were then given four days of punishment testing with footshock administered at the completion of seeking on a random one-third of trials. For cocaine-trained rats, the RI60 schedule led to greater punishment resistance (i.e., more trials completed) than the RR20 schedule in males and females. For food-trained rats, the RI60 schedule led to greater punishment resistance (i.e., higher reward rates) than the RR20 schedule in female rats, although male rats showed punishment resistance on both RR20 and RI60 schedules. For both cocaine and food, we found that seeking responses were suppressed to a greater degree than reward rate with the RI60 schedule, whereas response rate and reward rate were equally suppressed with the RR20 schedule. This dissociation between punishment effects on reward rate and response rate with the RI60 schedule can be explained by the nonlinear relation between these variables on RI schedules, but it does not account for the enhanced resistance to punishment. Overall, the results show greater punishment resistance with the RI60 schedule as compared to the RR20 schedule, indicating that schedules of reinforcement are an influencing factor on resistance to negative consequences.

A discriminated rapid-acquisition laboratory procedure for human continuous choice.

Previous continuous choice laboratory procedures for human participants are either prohibitively time-intensive or result in inadequate fits of the generalized matching law (GML). We developed a rapid-acquisition laboratory procedure (Procedure for Rapidly Establishing Steady-State Behavior, or PRESS-B) for studying human continuous choice that reduces participant burden and produces data that is well-described by the GML. To test the procedure, 27 human participants were exposed to 9 independent concurrent random-interval random-interval reinforcement schedules over the course of a single, 37-min session. Fits of the GML to the participants' data accounted for large proportions of variance (median R2 : 0.94), with parameter estimates that were similar to those previously found in human continuous choice studies [median a: 0.67; median log(b): -0.02]. In summary, PRESS-B generates human continuous choice behavior in the laboratory that conforms to the GML with limited experimental duration.

Unexpected food outcomes can return a habit to goal-directed action.

Three experiments examined the return of a habitual instrumental response to the status of goal-directed action. In all experiments, rats received extensive training in which lever pressing was reinforced with food pellets on a random-interval schedule of reinforcement. In Experiment 1, the extensively-trained response was not affected by conditioning a taste aversion to the reinforcer, and was therefore considered a habit. However, if the response had earned a new and unexpected food pellet during the final training session, the response was affected by taste aversion conditioning to the (first) reinforcer, and had thus been converted to a goal-directed action. In Experiment 3, 30 min of prefeeding with an irrelevant food pellet immediately before the test also converted a habit back to action, as judged by the taste-aversion devaluation method. That result was consistent with difficulty in finding evidence of habit with the sensory-specific satiety method after extensive instrumental training (Experiment 2). The results suggest that an instrumental behavior's status as a habit is not permanent, and that a habit can be returned to action status by associating it with a surprising reinforcer (Experiment 1) or by giving the animal an unexpected prefeeding immediately prior to the action/habit test (Experiment 3).

Evidence of non-circadian timing in a low response-cost daily Time-Place Learning task with pigeons Columba Livia

Previous research has shown that rats require high response cost in order to display circadian timing in daily Time-Place Learning (TPL) tasks. For many possible reasons, no explicit effort to explore the effects of response cost on the performance of other species in these tasks has been made. Therefore, the present paper explores the effects of response cost on pigeon’s performance on a daily TPL task. Head entry responses were reinforced according to a Random Interval schedule of reinforcement on one feeder during morning sessions and on another feeder during afternoon sessions. Feeders were located 8 cm apart for one group of birds (Group Near) and 56 cm apart for another group (Group Far). After 50 training sessions, testing began. Test sessions consisted of skipping either the morning or the afternoon session. Results show that most birds in the near group respond primarily on the opposite feeder during the first 20 s of the test sessions and then they switch to the correct feeder. On the other hand, most birds in Group Far respond at the same rate on both opposite and correct feeders during 20 s, and then they respond primarily on the correct feeder. The possibility of these data revealing non circadian timing for birds in a low response-cost daily TPL task is discussed along with the implications of such a finding for previous literature that claims that this type of performance could be unique to rats.

Lesions of the Patch Compartment of Dorsolateral Striatum Disrupt Stimulus–Response Learning

The striatum mediates habit formation and reward association. The striatum can be divided into the patch and matrix compartment, which are two distinct regions that sub-serve different aspects of behavior. The patch compartment may mediate reward-related behaviors, while the matrix compartment may mediate adaptive motor functions. Previous studies indicate that enhanced relative activation of the patch versus matrix compartment is associated with inflexible behaviors, such as stereotypy. Habitual behaviors are also inflexible in nature, but whether enhanced activation of the patch compartment contributes to habitual behavior is not known. The goal of the current study was to examine the role of patch compartment in the development of habit formation. We used dermorphin–saporin to ablate neurons of the patch compartment in the dorsolateral striatum prior to training animals to self-administer sucrose on a random interval schedule of reinforcement. Our data showed that patch compartment lesions in the dorsolateral striatum reduced the reinstatement of sucrose self-administration after sucrose devaluation, indicating that destruction of this region prevented the development of habitual behavior. Additionally, in animals with patch compartment lesions in the DLS that did not develop habitual behavior, activation of the dorsolateral striatum and sensorimotor cortex was diminished, while activity in the dorsomedial striatum and prefrontal cortex was increased, suggesting less engagement of regions that mediate habitual behaviors and heightened engagement of regions that mediate goal-directed behaviors occurs with reduced habit formation. These data indicate that the dorsolateral patch compartment may mediate habit formation by altering information flow through basal ganglia circuits.

Bayesian methods applied to the generalized matching law.

We demonstrate the usefulness of Bayesian methods in developing, evaluating, and using psychological models in the experimental analysis of behavior. We do this through a case study, involving new experimental data that measure the response count and time allocation behavior in pigeons under concurrent random-ratio random-interval schedules of reinforcement. To analyze these data, we implement a series of behavioral models, based on the generalized matching law, as graphical models, and use computational methods to perform fully Bayesian inference. We demonstrate how Bayesian methods, implemented in this way, make inferences about parameters representing psychological variables, how they test the descriptive adequacy of models as accounts of behavior, and how they compare multiple competing models. We also demonstrate how the Bayesian graphical modeling approach allows for more complicated modeling structures, including hierarchical, common cause, and latent mixture structures, to formalize more complicated behavioral models. As part of the case study, we demonstrate how the statistical properties of Bayesian methods allow them to provide more direct and intuitive tests of theories and hypotheses, and how they support the creative and exploratory development of new theories and models.

Saccadic latency and choice in a concurrent random interval reinforcement schedule.

Conventional decision models view reaction time as a consequence of the duration of decision-making process. However, some studies have shown that reaction time distributions may be strongly affected by reinforcement contingencies (Madelain et al., 2007). Here, we probe the possibility to voluntarily control saccadic latencies in a choice paradigm. Three subjects (including the two authors) tracked a visual target stepping horizontally by 10 deg between two fixed locations on a screen. Any trial with saccadic latencies greater than 300 ms or shorter than 80 ms was interrupted. Using the first and last quartiles of individual baseline latency distributions, we first defined two classes, i.e. short and long saccadic latencies, respectively. We then concurrently reinforced each type of latencies in random interval reinforcement schedules: short and long latencies were reinforced with three different sets of probabilities such that the relative ratio of reinforcing short latencies was either 9/1, 1/9 or 1/1. After training (20 800 trials), we observed bimodal latency distributions –with a peak for short and another for long latencies– for two subjects and to a lesser extent for the third subject. To further probe the extent of control over saccadic latencies, we then analyzed the data using the generalized matching law (Baum, 1974) –which states that the relative proportion of choices made to an option matches the relative proportion of reinforcers earned from that option. We found an almost perfect match between the relative proportion of short and long latencies and the relative obtained reinforcement for two subjects (sensitivities were equal to 0.95 and 0.87) and typical undermatching parameter for the third one (sensitivity = 0.58). These results indicate that saccades may be allocated in time following the reinforcement contingencies in force, which support the idea of a voluntary control of saccadic reaction time.

Contextual control of instrumental actions and habits.

After a relatively small amount of training, instrumental behavior is thought to be an action under the control of the motivational status of its goal or reinforcer. After more extended training, behavior can become habitual and insensitive to changes in reinforcer value. Recently, instrumental responding has been shown to weaken when tested outside of the training context. The present experiments compared the sensitivity of instrumental responding in rats with a context switch after training procedures that might differentially generate actions or habits. In Experiment 1, lever pressing was decremented in a new context after either short, medium, or long periods of training on either random-ratio or yoked random-interval reinforcement schedules. Experiment 2 found that more minimally trained responding was also sensitive to a context switch. Moreover, Experiment 3 showed that when the goal-directed component of responding was removed by devaluing the reinforcer, the residual responding that remained was still sensitive to the change of context. Goal-directed responding, in contrast, transferred across contexts. Experiment 4 then found that after extensive training, a habit that was insensitive to reinforcer devaluation was still decremented by a context switch. Overall, the results suggest that a context switch primarily influences instrumental habit rather than action. In addition, even a response that has received relatively minimal training may have a habit component that is insensitive to reinforcer devaluation but sensitive to the effects of a context switch.

Undermatching is an emergent property of selection by consequences

A computational theory of selection by consequences [McDowell, J.J, 2004. A computational model of selection by consequences. J. Exp. Anal. Behav. 81, 297-317] was tested by studying the responding of virtual organisms that were animated by the theory on random interval schedules of reinforcement. The theory generated responding by applying principles of selection, reproduction, and mutation to a population of potential behaviors that evolved in response to the selection pressure exerted by reinforcement. The organisms' equilibrium response rates were well described by the modern version of the Herrnstein hyperbola, which includes an exponent on reinforcement rate. Under strong selection pressure this exponent decreased with increasing mutation rate from a value near 1.0 at 1% mutation to an asymptotic value of 0.83 at mutation rates of 10% and greater. This asymptotic value is consistent with values obtained by fitting the equation to data from live organisms responding on single schedules, and with the value of about 0.80 that is expected on the basis of extensive research with live organisms responding on concurrent schedules. These results show that the computational theory is consistent with the modern theory of matching [McDowell, J.J, 2005. On the classic and modern theories of matching. J. Exp. Anal. Behav. 84, 111-127], and that it is a viable candidate for a mathematical dynamics of behavior.

Augmented reinforcer value and accelerated habit formation after repeated amphetamine treatment

Various processes might explain the progression from casual to compulsive drug use underlying the development of drug addiction. Two of these, accelerated stimulus–response (S–R) habit learning and augmented assignment of motivational value to reinforcers, could be mediated via neuroadaptations associated with long-lasting sensitization to psychostimulant drugs, i.e. augmented dopaminergic neurotransmission in the striatum. Here, we tested the hypothesis that both processes, which are often regarded as mutually exclusive alternatives, are present in amphetamine-sensitized rats. Amphetamine-sensitized rats showed increased responding for food under a random ratio schedule of reinforcement, indicating increased incentive motivational value of food. In addition, satiety-specific devaluation experiments under a random interval schedule of reinforcement showed that amphetamine-sensitized animals exhibit accelerated development of S–R habits. These data show that both habit formation and motivational value of reinforcers are augmented in amphetamine-sensitized rats, and suggest that the task demands determine which behavioral alteration is most prominently expressed.

PIGEONS' CHOICES BETWEEN FIXED-INTERVAL AND RANDOM-INTERVAL SCHEDULES: UTILITY OF VARIABILITY?

Pigeons' choosing between fixed-interval and random-interval schedules of reinforcement was investigated in three experiments using a discrete-trial procedure. In all three experiments, the random-interval schedule was generated by sampling a probability distribution at an interval (and in multiples of the interval) equal to that of the fixed-interval schedule. Thus the programmed delays to reinforcement on the random alternative were never shorter and were often longer than the fixed interval. Despite this feature, the fixed schedule was not strongly preferred. Increases in the probability used to generate the random interval resulted in decreased preferences for the fixed schedule. In addition, the number of consecutive choices on the preferred alternative varied directly with preference, whereas the consecutive number of choices on the nonpreferred alternative was fairly constant. The probability of choosing the random alternative was unaffected by the immediately prior interval encountered on that schedule, even when it was very long relative to the average value. The results loosely support conceptions of a "preference for variability" from foraging theory and the "utility of behavioral variability" from human decision-making literatures.

In utero lead exposure in squirrel monkeys: motor effects seen with schedule-controlled behavior.

Timed-pregnant squirrel monkeys were exposed orally to lead during the last 1/2 to 2/3 of gestation such that maternal lead levels ranged from 21 to 70 micrograms/dl in blood. Offspring of these lead-exposed monkeys were compared to gender-matched, untreated controls (blood-lead levels from 4 to 9 micrograms/dl), born at about the same time. When the monkeys were 3 to 7 years old they were trained to pull a T-shaped bar against 1 kg spring through a displacement of 1 cm. This performance was examined during acquisition of different fixed-ratio (1, 5, and 20) and fixed-interval (120", 300", and 600") schedules of reinforcement and during steady state under the fixed-ratio 5 and fixed-interval 600". Monkeys exposed prenatally to lead showed an increased number of responses failing to meet the requirement of pulling against 1 kg spring through a 1 cm displacement when behavior was maintained by a fixed-ratio schedule, which engenders a vigorous, high-rate pattern of responding. This increased number of incomplete responses first appeared in the acquisition of a fixed-ratio 5 and fixed-ratio 20 schedules of reinforcement, remained after the fixed-ratio 5 schedule was allowed to reach steady state, and did not appear under the fixed-interval schedule. Neither body weight not response rate were affected by lead, but it was necessary to control for these variables using multiple regression to isolate lead's effect. The appearance of incomplete responses while the monkeys pulled vigorously against a 1 kg spring suggests that lead exposure during gestation produced subtle motor impairments years after exposure has ended. Deficits in the acquisition of behavior (learning) under Concurrent Random Interval schedules of reinforcement have also been reported with these monkeys. Together, these reports reveal prolonged deficits in learning and motor function resulting from in utero exposure to lead at maternal blood lead levels (21-70 micrograms/dl) that could result from exposure to ambient air in heavily polluted urban environments or in occupational settings meeting current World Health Organization standards.

An eight-alternative concurrent schedule: foraging in a radial maze.

In two experiments conducted in an eight-arm radial maze, food pellets were delivered when a photocell beam was broken at the end of each arm via a nose poke, according to either fixed-interval or random-interval schedules of reinforcement, with each arm providing a different frequency of reinforcement. The behavior of rats exposed to these procedures was well described by the generalized matching law; that is, the relationships between log behavior ratios and log pellet ratios were approximated by linear functions. The slopes of these log-log functions, an index of sensitivity to reinforcement frequency, were greatest for nose pokes, intermediate for time spent in an arm, and least for arm entries. Similar results were obtained with both fixed-interval and random-interval schedules. Addition of a 10-s changeover delay in both experiments eliminated the slope differentials between nose pokes and time spent in an arm by reducing the slopes of the nose-poke functions. These results suggest that different aspects of foraging may be differentially sensitive to reinforcement frequency. With concurrent fixed-interval schedules, the degree of temporal control exerted by individual fixed-interval schedules was directly related to reinforcement frequency.

Prolonged Behavioral Effects of in Utero Exposure to Lead or Methyl Mercury: Reduced Sensitivity to Changes in Reinforcement Contingencies during Behavioral Transitions and in Steady State

Postnatal exposure to lead or methyl mercury results in mental retardation, learning deficits, and other neurobehavioral effects in humans, and adverse consequences of prenatal exposure have been clearly documented with methyl mercury. To examine the developmental neurotoxicity of these metals, especially lead, concurrent schedules of food reinforcement were used to identify learning deficits in squirrel monkeys exposed during gestation to either methyl mercury or lead. Pregnant squirrel monkeys were administered methyl mercury (0.7 to 0.9 ppm in maternal blood) or lead (21 to 79 μg/dl in maternal blood) during the last half to two-thirds of gestation. At about 5-6 years of age, offspring were trained to lever press under concurrent schedules of reinforcement in which separate random interval reinforcement schedules operated independently on two levers. Reinforcement densities were varied such that 20 to 90% of the reinforcers were programmed to derive from the left lever (i.e., one lever was "richer" than the other). At steady state, the behavior of the controls was sensitive to reinforcement density and showed little lever bias, but the behavior of monkeys exposed to more than 40 μg/dl of lead and to methyl mercury was less sensitive to reinforcement rates and heavily biased. When relative reinforcement density on a lever changed, the unexposed animals′ response rates gradually shifted to the newly rich lever. The behavior of monkeys exposed to methyl mercury or more than 40 μg/dl of lead changed slowly, not at all, or in the wrong direction. Steady-state behavior of monkeys exposed to less than 40 μg/dl resembled controls, but acquisition progressed more slowly and required 2-4 times as many reinforcers to complete. These effects suggest a behavioral mechanism-insensitivity to changing reinforcement contingencies-by which learning deficits and behavioral changes associated with these metals might be related to toxicant exposure. Since maternal blood levels corresponded to those that could be experienced in occupational settings, the present data raise the possibility of fetal hazards associated with maternal lead exposures at levels tolerated in humans in occupational settings.

PERFORMANCE OF CHILDREN UNDER A MULTIPLE RANDOM‐RATIO RANDOM‐INTERVAL SCHEDULE OF REINFORCEMENT

Three children, aged 1.5, 2.5, and 4.5 years, pressed telegraph keys under a two-component multiple random-ratio random-interval schedule of reinforcement. In the first condition, responses on the left key were reinforced under a random-interval schedule and responses on the right key were reinforced under a random-ratio schedule. In the second condition, the schedule components were reversed. In the third condition, the original arrangement was reinstated. For all subjects, rates of responding were higher in the random-ratio component despite higher rates of reinforcement in the random-interval component. The average interreinforcement interval of the random-interval component was increased in the fourth condition, resulting in more similar rates of reinforcement for both schedule components, and then returned to its original value in the fifth condition. In both conditions, all subjects continued to exhibit higher rates of responding in the ratio component than in the interval component. Although these observations are consistent with results from studies with pigeons, it is argued that the response-rate differences between the interval and ratio schedule components are sufficient to demonstrate schedule sensitivity.

A matching law analysis of the effects of dopamine receptor antagonists

Herrnstein's matching equation was used to analyze drug effects on performance in random interval reinforcement schedules. Pimozide caused effects compatible with both motor and motivational impairments, in a 5-component multiple schedule, a 3-schedule 3-day cycle (ALT-3), and a 2-schedule 2-day cycle (ALT-2). However, at low doses, both sulpiride and SCH-23390, tested in the ALT-3 and ALT-2 procedures, caused effects compatible with selective motivational impairments. In experiments using the non-multiple schedules, motivational effects increased during the course of the experimental session, under all three drugs. The interpretation of "motor" and "motivational" deficits in the ALT-2 procedure was validated by experiments in which the response-force and deprivation level were systematically varied. The results support the view that dopamine may be involved in the maintenance of rewarded behaviour, but not differentially implicate the D1 or the D2 receptor subtype.

Exposure of Japanese quail embryos to o,p′‐DDT has long‐term effects on reproductive behaviors, hematology, and feather morphology

Japanese quail eggs were injected with 1-(2-chlorophenyl)-1-(4-chlorophenyl)-2,2,2-trichloroethane o,p'-DDT(1-10 mg),1,1-bis(4-chlorophenyl)-2,2,2-trichloroethane p,p'-DDT (1-10 mg), or, in one study, 0.5 mg chlordecone dissolved in 50 microliters of corn oil on day 1 of incubation. Hatchability was not decreased by o,p'-DDT or p,p'-DDT, as compared to corn-oil-injected controls, but was reduced in progeny of parents injected in ovo with either isomer. Tremor was observed for up to 4 days posthatching only in birds injected with 1.75-10 mg p,p'-DDT or chlordecone. Survivability to 5 weeks posthatch was reduced (less than or equal to 50%) in birds injected in ovo with 6.25-7.5 mg, o,p'-DDT or 1.75-5 mg p,p'-DDT as compared to corn oil (96%). Reproductive behaviors were attenuated in birds injected during development with o,p'-DDT, both DDT isomers decreased the total number of ovipositions, and o,p'-DDT increased the total number of eggshell malformations. Neither body weights nor reproductive organ weights at 12 weeks were affected by injection of either isomer. Exposure to DDT did not affect acquisition of a matched-to-sample food-reinforced response or subsequent responding on a random interval schedule of reinforcement. In another experiment, total circulating erythrocyte numbers were reduced in females after injection in ovo with o,p'-DDT but not after injection with p,p'-DDT. A primary humoral immune response was not affected by in ovo exposure to either isomer of DDT. In ovo exposure to o,p'-DDT but not to p,p'-DDT had long-term and estrogen-like effects on behavior and hematology in Japanese quail. Posthatch primary feather morphology was also altered by embryonic exposure to o,p'-DDT, p,p'-DDT, and chlordecone.

Differential effects of pentobarbital and cocaine on punished and nonpunished responding.

Similar rates of punished and nonpunished responding, maintained with equated rates of reinforcement, were established in pairs of rats. One subject of each pair was exposed to a random-ratio schedule of food presentation. The interreinforcement intervals for this subject comprised the intervals of a random-interval schedule of reinforcement for the other (yoked) rat. The random-ratio schedule maintained rates of responding higher than those maintained by the same rate of reinforcement schedule according to the yoked random-interval contingency. A random-ratio schedule of electric foot shock added to the random-ratio schedule of food presentation suppressed rates of responding such that similar rates of responding were observed in rats of both groups. Pentobarbital (3.0 to 17.0 mg/kg) increased punished responding at doses that had little effect on or decreased nonpunished responding, whereas cocaine (5.6 to 30 mg/kg) increased nonpunished responding at doses that decreased or did not alter punished responding. Qualitatively different effects of pharmacological agents on punished and nonpunished responding can be obtained using procedures that generate similar rates and temporal patterns of punished and nonpunished responding. The effects of pentobarbital and cocaine on responding can be determined by factors other than simply the baseline rate of responding.

Time-dependent and schedule-dependent effects of dopamine receptor blockade

Rats were trained on one of two multiple random-interval schedules of reinforcement, which contained both ascending and descending reinforcement density sequences. The design of the schedules made it possible to determine whether performance changes in a particular component depended on the reinforcement density in that component, or on its temporal position within the schedule. Performance impairments following administration of the dopamine D1 antagonist SCH-23390 were both time- and schedule-dependent: SCH-23390 caused a relatively greater suppression of poorly reinforced responding that increased over time. The results, which are compatible with data on the effects of dopamine antagonists in other behavioural paradigms, illustrate the methodological problems of using Herrnstein's matching equation to interpret time-dependent behavioural changes.

Variability of force and interresponse time under random interval reinforcement schedules

Response variability was examined under five random interval schedules with minimum mean interreinforcement intervals of 15, 30, 60, 120, and 240 sec. Distributions of response force-time integrals showed no systematic variability changes with increases in reinforcement intermittency. Corresponding distributions of interresponse times revealed increases in variability accompanying increases in reinforcement intermittency. The relationship of variability to these two measures, and the potential effect of behavior outside the experimentally measured response class, on recorded variability within the class, are considered.