Delay Of Reinforcement Research Articles

Dopamine Receptor D1/D5 Gene Expression in the Medial Prefrontal Cortex Predicts Impulsive Choice in Rats

A neuropsychological hallmark of attention deficit/hyperactivity disorder (ADHD) is the reduced ability to tolerate delay of reinforcement, leading to impulsive choice. Genetic association studies have implicated several genes involved in dopaminergic neurotransmission in ADHD. In this study, we investigated whether differences in the expression level of these dopamine-related genes of rats predict the individual level of impulsive choice. Among all frontostriatal brain regions tested, only in the medial prefrontal cortex (mPFC), we observed significant positive correlations between impulsive choice and transcript levels of the dopamine receptor D(1), the dopamine receptor D(5) and calcyon. Local mPFC infusions of the D(1)/D(5) receptor antagonist SCH 23390 and agonist SKF 38393 resulted in increased impulsive choice, in agreement with the idea that endogenous receptor D(1)/D(5) stimulation in the mPFC promotes the choice of large delayed rewards. Together, these data indicate that this class of dopamine receptors in the mPFC plays a pivotal role in impulsive choice, and aberrancies thereof might contribute to ADHD symptomatology.

Sensitivity to delay of reinforcement in two animal models of attention deficit hyperactivity disorder (ADHD)

An altered response to reinforcement has been proposed as a mechanism underlying many of the symptoms of attention deficit hyperactivity disorder (ADHD). We measured sensitivity to delay of reinforcement in two animal models of ADHD, the spontaneously hypertensive rat (SHR) and a newly proposed model, the genetically hypertensive (GH) rat. A task previously used to measure effects of delay of reinforcement in children with ADHD was adapted for use in the present experiment. The SHR and GH rats were compared to their respective genetic control strains, Wistar-Kyoto (WKY), and Wistar (WI). The experimental task required pressing one of two available levers each trial. One lever delivered an immediate reinforcement, and the other lever a delayed reinforcement. Both the SHR and GH strains allocated significantly more responses to the immediately reinforced lever than their genetic control strains. Individual instances of reinforcement differentially affected response allocation in the GH but not the SHR. These findings support the use of the SHR and GH rat to model altered response to reinforcement, and demonstrate the additional value of the GH strain to model the effects of individual instances of reinforcement in children with ADHD.

Assessing equine prospective memory in a Y-maze apparatus

Eight horses were tested in a Y-maze to see if they would approach the side on which they had seen and heard food being delivered into a food trough. The horses were tested in five experimental contexts: immediate release (IR), delayed release 3 s (3DR), 6 s (6DR), 9 s (9DR) and 12 s (12DR) after food delivery. Individual subjects performed five IR trials, followed by five each of 3DR, 6DR, 9DR and 12DR trials. A re-rest was performed 1 week later. Data were analysed by log-linear analysis of frequencies. The results showed that the horses were capable of achieving the correct choice during IR trials, were successful during the DR trials and improved marginally during the re-test. The findings suggest that horses could have a trainable form of prospective memory and benefit from incremental memory-training exercises. However, there is a risk of overestimating equine cognitive ability and unnecessary reinforcement delays are likely to lead to welfare and training setbacks within equitation.

Effects of lesions of the nucleus accumbens core on inter-temporal choice: Further observations with an adjusting-delay procedure

Previous experiments using progressive-delay schedules showed that destruction of the nucleus accumbens core (AcbC) altered rats’ choice between food reinforcers differing in size and delay. Application of a quantitative model of inter-temporal choice suggested that lesions of the AcbC increase the delay-dependent degradation of reinforcer value (delay discounting) without altering instantaneous reinforcer value. This experiment examined the effect of lesions of the AcbC on inter-temporal choice using an adjusting-delay schedule. Rats received excitotoxin-induced lesions of the AcbC or sham lesions. They were trained to press levers A and B for food-pellet reinforcers in an adjusting-delay schedule in which the delay to the larger reinforcer, d B , varied in accordance with the rats’ choices between the two levers. In two experimental conditions, the reinforcers associated with levers A and B were 1 vs. 4 and 2 vs. 4 pellets. The AcbC-lesioned group showed shorter indifference delays to reinforcer B ( d B(50) ) than the sham-lesioned group under both conditions. In confirmation of a prediction derived from the model of inter-temporal choice, the ratio of the indifference delays from the two conditions did not differ between the groups. Analysis of the cyclical changes in d B by Fourier transform showed that the period of oscillation and power within the dominant frequency band did not differ between the groups, suggesting that the lesion did not disrupt the rats’ ability to detect short-term changes in delay of reinforcement. The results are consistent with previous findings that indicate a role for the AcbC in delay discounting.

ISOLATING BEHAVIORAL MECHANISMS OF INTERTEMPORAL CHOICE: NICOTINE EFFECT ON DELAY DISCOUNTING AND AMOUNT SENSITIVITY

Many drugs of abuse produce changes in impulsive choice, that is, choice for a smaller-sooner reinforcer over a larger-later reinforcer. Because the alternatives differ in both delay and amount, it is not clear whether these drug effects are due to the differences in reinforcer delay or amount. To isolate the effects of delay, we used a titrating delay procedure. In phase 1, 9 rats made discrete choices between variable delays (1 or 19 s, equal probability of each) and a delay to a single food pellet. The computer titrated the delay to a single food pellet until the rats were indifferent between the two options. This indifference delay was used as the starting value for the titrating delay for all future sessions. We next evaluated the acute effects of nicotine (subcutaneous 1.0, 0.3, 0.1, and 0.03 mg/kg) on choice. If nicotine increases delay discounting, it should have increased preference for the variable delay. Instead, nicotine had very little effect on choice. In a second phase, the titrated delay alternative produced three food pellets instead of one, which was again produced by the variable delay (1 s or 19 s) alternative. Under this procedure, nicotine increased preference for the one pellet alternative. Nicotine-induced changes in impulsive choice are therefore likely due to differences in reinforcer amount rather than differences in reinforcer delay. In addition, it may be necessary to include an amount sensitivity parameter in any mathematical model of choice when the alternatives differ in reinforcer amount.

Delay-amount tradeoffs in choices by pigeons and rats: hyperbolic versus exponential discounting.

An adjusting-delay procedure was used to study the choices of pigeons and rats when both delay and amount of reinforcement were varied. In different conditions, the choice alternatives included one versus two reinforcers, one versus three reinforcers, and three versus two reinforcers. The delay to one alternative (the standard alternative) was kept constant in a condition, and the delay to the other (the adjusting alternative) was increased or decreased many times a session so as to estimate an indifference point--a delay at which the two alternatives were chosen about equally often. Indifference functions were constructed by plotting the adjusting delay as a function of the standard delay for each pair of reinforcer amounts. The experiments were designed to test the prediction of a hyperbolic decay equation that the slopes of the indifference functions should increase as the ratio of the two reinforcer amounts increased. Consistent with the hyperbolic equation, the slopes of the indifference functions depended on the ratios of the two reinforcer amounts for both pigeons and rats. These results were not compatible with an exponential decay equation, which predicts slopes of 1 regardless of the reinforcer amounts. Combined with other data, these findings provide further evidence that delay discounting is well described by a hyperbolic equation for both species, but not by an exponential equation. Quantitative differences in the y-intercepts of the indifference functions from the two species suggested that the rate at which reinforcer strength decreases with increasing delay may be four or five times slower for rats than for pigeons.

Effects of d-amphetamine on behavior maintained by signaled and unsignaled delays to reinforcement

Four pigeons responded under a progressive-delay procedure. In a signaled-delay condition, a chained variable interval (VI) 30-s progressive time (PT) 4-s schedule was arranged; in an unsignaled-delay condition, a tandem VI 30-s PT 4-s schedule was arranged. Two pigeons experienced a signaled-unsignaled-signaled sequence; whereas, two pigeons experienced an unsignaled-signaled-unsignaled sequence. Effects of saline and d-amphetamine were determined under each condition. At intermediate doses (1.0 and 1.78m/kg) delay functions were shallower, area under the curve was increased, and, when possible, break points were increased compared to saline; these effects were not systematically related to signaling conditions. These effects on control by delay often were accompanied by decreased response rates at 0s. These results suggest that stimulus conditions associated with the delay may not play a crucial role in effects of d-amphetamine and other stimulants on behavior controlled by reinforcement delay.

Effects of reinforcer magnitude and alternative reinforcer delay on preference for alcohol during a multiple-choice procedure

The Multiple-Choice Procedure [MCP; Griffiths, R.R., Troisi II, J.R., Silverman, K., Mumford, G.K., 1993. Multiple-choice procedure: an efficient approach for investigating drug reinforcement in humans. Behav. Pharmacol. 4, 3–13; Griffiths, R.R., Rush, C.R., Puhala, K.A., 1996. Validation of the multiple-choice procedure for investigating drug reinforcement in humans. Behav. Pharmacol. 4, 3–13] was developed to investigate the relationship between drug preferences and alternative reinforcers. The current study was designed to better characterize the relationships among reinforcer magnitude, delay associated with an alternative reinforcer, and preference for alcohol among a sample of undergraduate drinkers. Participants were 27 male undergraduates, over the age of 21, who reported engaging in at least three occasions of binge drinking over the previous month. All participants took part in a single laboratory session, during which six versions of the MCP questionnaire were administered. Across the versions, participants made a total of 180 discrete choices between three amounts of alcohol (12, 24, or 36 ounces of beer; deception was used to mask the fact that 36 ounces of beer would not actually be provided during the sessions) and escalating amounts of money ($0 to $20) delivered immediately or after a one-week delay. Response on the MCP varied as a function of the amount of alcohol available and the delay associated with the alternative monetary reinforcer, with the effect of the delay diminishing as the dose of alcohol increased. Results of the current study are consistent with previous laboratory studies of choice behavior and with behavioral economic theories of substance use.

Effects of reinforcer delay and variability in a successive-encounters procedure.

Pigeons responded in a successive-encounters procedure that consisted of a search period, a choice period, and a handling period. The search period was either a fixed-interval or a mixed-interval schedule presented on the center key of a three-key chamber. Upon completion of the search period, the center key was turned off and the two side keys were lit. A pigeon could either accept a delay followed by food (by pecking the right key) or reject this option and return to the search period (by pecking the left key). During the choice period, a red right key represented the long alternative (a long handling delay followed by food), and a green right key represented the short alternative (a short handling delay followed by food). The experiment consisted of a series of comparisons for which optimal diet theory predicted no changes in preference for the long alternative (because the overall rates of reinforcement were unchanged), whereas the hyperbolic-decay model predicted changes in preference (because the delays to the next possible reinforcer were varied). In all comparisons, the results supported the predictions of the hyperbolic-decay model, which states that the value of a reinforcer is inversely related to the delay between a choice response and reinforcer delivery.

Using a Time Timer™ to increase appropriate waiting behavior in a child with developmental disabilities

This study aimed to examine the use of a predictive stimulus (Time Timer™) and delayed reinforcement to increase appropriate waiting behavior in a child with developmental disabilities and problem behavior maintained by access to tangible items and activities. The study employed a changing criterion design across settings to gradually increase reinforcement delay from 1 s to 10 min. Firstly a baseline phase was conducted to measure the duration of appropriate waiting behavior to access tangible reinforcers/activities. Phase 2 involved the use of a red cue card and the verbal instruction “wait”. Phase 3 involved the introduction of the Time Timer™ with the cue card attached, and the verbal instruction “wait”. Finally, Phase 4 utilised the Time Timer™ without the cue card. This method was an effective strategy for increasing appropriate waiting behavior with this participant in a school setting. The role of adding a concurrent activity during the reinforcement delay, using cues to predict reinforcement, future generalization, maintenance and the teaching of functionally equivalent skills are discussed.

Different Structural Sequences: A positive Effect of Reinforcement Delay in a Conditional Discrimination Task

Nine pigeons were exposed to a matching-to-sample task in which the reinforcer delay was varied. Specific sample stimulus, comparison stimulus, and reinforcement conditional probabilities were used to establish different structural sequences. Pigeons were assigned to three different groups. Reinforcer delays for the three groups were 3 s and 1 s for group 1, 2 s for group 2, and 0 s for group 3. Pigeons in group 1 performed similarly to those in group 3, while those in group 2 performed most poorly. These data indicate that matching-to-sample performance may be enhanced by different structural sequences even when the element that makes the difference—reinforcer delay—has been previously reported as having a negative effect on matching accuracy.

ON LOSS AVERSION IN CAPUCHIN MONKEYS

Chen, Lakshminarayanan, and Santos (2006) claim to show in three choice experiments that monkeys react rationally to price and wealth shocks, but, when faced with gambles, display hallmark, human-like biases that include loss aversion. We present three experiments with monkeys and humans consistent with a reinterpretation of their data that attributes their results not to loss aversion, but to differences between choice alternatives in delay of reinforcement.

RAPID ACQUISITION OF PREFERENCE IN CONCURRENT CHAINS: EFFECTS OF d‐AMPHETAMINE ON SENSITIVITY TO REINFORCEMENT DELAY

The purpose of this study was to examine effects of d-amphetamine on choice controlled by reinforcement delay. Eight pigeons responded under a concurrent-chains procedure in which one terminal-link schedule was always fixed-interval 8 s, and the other terminal-link schedule changed from session to session between fixed-interval 4 s and fixed-interval 16 s according to a 31-step pseudorandom binary sequence. After sufficient exposure to these contingencies (at least once through the pseudorandom binary sequence), the pigeons acquired a preference for the shorter reinforcement delay within each session. Estimates of the sensitivity to reinforcement immediacy were similar to those obtained in previous studies. For all pigeons, at least one dose of d-amphetamine attenuated preference and, hence, decreased estimates of sensitivity to reinforcement immediacy; in most cases, this effect occurred without a change in overall response rates. In many cases, the reduced sensitivity to reinforcement delay produced by d-amphetamine resulted primarily from a decrease in the asymptotic level of preference achieved within the session; in some cases, d-amphetamine produced complete indifference. These findings suggest that a reduction in the sensitivity to reinforcement delay may be an important behavioral mechanism of the effects of psychomotor stimulants.

Contingência e contigüidade no responder de ratos submetidos a esquemas de razão, intervalo e tempo variáveis

No presente estudo, foram investigados os efeitos produzidos pela introdução de atraso do reforço sobre a taxa e a distribuição de freqüência das respostas de pressão à barra emitidas por oito ratos expostos a esquemas de intervalo, razão e tempo variáveis (VI, VR e VT, respectivamente). A partir de condições de linha de base sob reforçamento imediato, aplicou-se o atraso do reforço não sinalizado e non-resetting pelo tempo fixo de 5 s a cada um dos esquemas. Os resultados mostraram que, com a introdução do atraso do reforço, houve diminuição nas taxas de respostas de todos os sujeitos, em todos os três esquemas, mais acentuadamente sob VT, VI e VR, nesta ordem. A análise molecular dos dados revelou que, para os sujeitos sob VI, os valores efetivamente registrados de atrasos concentraram-se proximamente ao atraso programado de 5 s, enquanto que, para os sujeitos expostos a VR, os intervalos assumiram valores menores. Esses resultados demonstram, portanto, que os efeitos do atraso do reforço sobre o responder operante variam em função do esquema de reforçamento vigente. No presente estudo, o uso de VI e VR, cujos controles pelo tempo e pelo número de respostas caracterizam, respectivamente, cada um desses esquemas, foi acompanhado por efeitos diferenciados tanto nas taxas de resposta quanto na distribuição do responder no tempo.Palavras-chave: contingência; contigüidade; atraso do reforço; esquemas de reforçamento; ratos.

A THEORY OF ATTENDING, REMEMBERING, AND REINFORCEMENT IN DELAYED MATCHING TO SAMPLE

A theory of attending and reinforcement in conditional discriminations is extended to working memory in delayed matching to sample by adding terms for disruption of attending during the retention interval. Like its predecessor, the theory assumes that reinforcers and disruptors affect the independent probabilities of attending to sample and comparison stimuli in the same way as the rate of overt free-operant responding as suggested by Nevin and Grace, and that attending is translated into discriminative performance by the model of Davison and Nevin. The theory accounts for the effects of sample-stimulus discriminability and retention-interval disruption on the levels and slopes of forgetting functions, and for the diverse relations between accuracy and sensitivity to reinforcement reported in the literature. It also accounts for the effects of reinforcer probability in multiple schedules on the levels and resistance to change of forgetting functions; for the effects of reinforcer probabilities signaled within delayed-matching trials; and for the effects of reinforcer delay, sample duration, and intertrial-interval duration. The model accounts for some data that have been problematic for previous theories, and makes testably different predictions of the effects of reinforcer probabilities and disruptors on forgetting functions in multiple schedules and signaled trials.

Remembering as discrimination in delayed matching to sample: Discriminability and bias

Task difficulty in delayed matching-to-sample tasks (DMTS) is increased by increasing the length of a retention interval. When tasks become more difficult, choice behavior becomes more susceptible to bias produced by unequal reinforcer ratios. Delaying reinforcement from choice behavior also increases both task difficulty and the biasing effect of unequal reinforcer probability. Six pigeons completed nine DMTS conditions with retention intervals of 0, 2, 4, 6, and 8 sec, in which reinforcer delays of 0, 2, and 4 sec were combined with ratios of reinforcer probabilities of .5/.5, .2/.8, and .8/.2 for correct red and green responses. Discriminability (log d) decreased with both increasing retention interval duration and increasing reinforcer delay. Sensitivity to reinforcement, the tendency for ratios of choice responses to follow unequal reinforcer probabilities, also increased as a function of both increasing retention interval and increasing reinforcer delay. The result is consistent with the view that remembering in DMTS tasks is a discriminated operant in which increasing task difficulty increases sensitivity to reinforcement.

RESPONSE INDUCTION DURING THE ACQUISITION AND MAINTENANCE OF LEVER PRESSING WITH DELAYED REINFORCEMENT

The acquisition of lever pressing by rats and the occurrence of unreinforced presses at a location different from that of the reinforced response were studied using different delays of reinforcement. An experimental chamber containing seven identical adjoining levers was used. Only presses on the central (operative) lever produced food pellets. Groups of 3 rats were exposed to one of seven different tandem random-interval (RI) fixed-time (FT) schedules. The average RI duration was the complement of the FT duration such that their sum yielded a nominal 32-s interreinforcement interval on average. Response rate on the operative lever decreased as the FT value was lengthened. The spatial distribution of responses on the seven levers converged on the operative lever when the FT was 0 or 2 s and spread across the seven levers as the FT value was lengthened to 16 or 32 s. Presses on the seven levers were infrequent during the FT schedule. Both operative- and inoperative-lever pressing intertwined in repetitive patterns that were consistent within subjects but differed between subjects. These findings suggest that reinforcer delay determined the response-induction gradient.

Maintenance of responding when reinforcement becomes delayed

In four experiments with rats, we examined the persistence of behavior when reinforcement was switched from immediate to delayed. In Experiment 1, lever pressing elicited by instrumental training with immediate reinforcement continued when a 20-sec delay of reinforcement was introduced (easy-to-hard condition), whereas when the delay condition was introduced from the start (hard-to-hard condition), responding remained low throughout. A similar result was obtained in Experiment 2, in which lever pressing was elicited by a classical conditioning (autoshaping) procedure. In Experiment 3, rats initially trained with delayed reinforcement continued to respond at a low rate when switched to immediate reinforcement (hard-to-easy condition). By measuring magazine entry (goal tracking) as well as lever pressing (sign tracking) in Experiment 4, we confirmed that such transfer effects at least partly involve the persistence of whatever type of behavior was initially dominant.

The Effects of a Delay of Noncontingent Reinforcement during a Pairing Procedure in the Development of Stimulus Control of Automatically Reinforced Vocalizations.

Children with autism and related disabilities frequently fail to develop echoic repertoires. Among the ways in which treatment approaches vary is the extent to which automatic reinforcement is utilized. The present experiment was designed to test the efficacy of a procedure that incorporates automatic reinforcement and socially mediated reinforcement in the development of an echoic repertoire. The implementation of this treatment package resulted in an increase in the vocal play and echoic behavior for two participants, each of whom had very limited verbal repertoires. The study was conducted as a multiple probe across sounds design. Vocal play for Participant 1 increased from baseline rates as low as .2 per min to 4.5 per min during treatment, and her percent correct echoic behavior increased from 0% to at least 90% on both sounds. Participant 2 showed rapid gains in echoic control on one sound before requiring dismissal from the study. Interobserver agreement on the occurrence of target vocals equaled 100%. The results of this study have implications for which procedures to use in the establishment of echoic, echoic mand, and mand repertoires.

The surprising influence of delayed primary reinforcement on choice.

Abstract It is well known that the duration of the delay between a response and consequence is inversely related to the impact of that consequence on future responding, and even short delays can greatly undermine the effectiveness of a consequence. However, several studies have shown that delayed primary reinforcement can have a substantial impact on responding in situations in which it was assumed to exert little or no influence. For example, delayed primary reinforcement has produced surprisingly strong effects on responding in procedures with simple concurrent schedules and concurrent chains schedules. This article will highlight two studies (McDevitt & Williams 2001; Ploog, 2001) that demonstrate that delayed primary reinforcement can have direct effects on choice. Keywords: Delay of reinforcement, unsignaled delay, choice, key peck, pigeons ********** As behavior analysts, we are all familiar with the fundamental concept that delaying the delivery of reinforcement comprises its effectiveness. For a primary reinforcer to be directly effective (i.e., without resorting to the mediating influence of conditioned reinforcement), it should be presented immediately following the behavior. It is well established that increasing the delay to primary reinforcers systematically decreases the effectiveness of those reinforcers. We do well to provide immediate consequences for behavior when we want those consequences to influence future behavior, and just as importantly, we should increase the delay to consequences when we want to minimize the influence on future behavior. I sometimes use this latter approach when I find that I have inadvertently reinforced a behavior. For example, several years ago I was embarrassed to realize that I had unintentionally reinforced my cat's behavior of crying (painfully loudly) by feeding her shortly after the behavior. By simply increasing the delay from the behavior (crying) to the primary reinforcer (feeding), I easily eliminated the behavior. (1) My cat now sits quietly by her food dish staring at me when she wants to be fed. Some notable research studies have highlighted the importance of the temporal relationship between responses and consequences. For example, Williams (1976, Experiment 1) interposed a delay between responding and reinforcement after training pigeons to respond to a variable-interval (VI) 2-min schedule of reinforcement. Across conditions, the delay period was varied from 3 s to 15 s, and was unsignaled (the peck that met the VI requirement started the delay timer but was not associated with any stimulus change, and food was delivered at the end of the delay period). Even with the shortest (3 s) delay period, response rates were reduced 70-80% compared to baseline responding. Thus, a small delay had a profound effect on responding. The results are even more striking when one considers that the actual delay between the last response and the reinforcer was likely shorter than 3 s since the pigeons could continue to respond during the delay period. Similarly, other researchers have found considerable attenuation of response rates when short delays are interposed between responses and reinforcers (e.g., Black, Belluzzi, & Stein, 1985; Royalty, Williams, & Fantino, 1987; Sizemore and Lattal, 1977). The observation that short delays can have large negative effects is also evident in the widespread practice of using very short changeover delays (CODs) when assessing preference in a choice procedure, often as short as 1.5 s (e.g., Herrnstein, 1961). In other words, simply delaying the reinforcer by less than 2 s appears sufficient to disrupt adventitious reinforcement of switching behavior. Although the delay to primary reinforcement clearly is important, can we assume that a delayed reinforcer is always ineffective? It has long been presumed that primary reinforcement delivered after a long delay does not directly affect behavior, but acts indirectly due to intervening conditioned reinforcers (e. …