Delay Of Reinforcement Research Articles

Influence of amount and delay of reward on choice and response rate: A free-operant, multiple-schedule analogue of a discrete-trial procedure.

The current study explored a free-operant analogue of discrete-trial procedures to study the effects of amount and delay of reinforcement on choice and response rate. Rats responded on a multiple variable-interval (VI) 45-s, 45-s schedule, with interspersed choice probe trials. Comparison of relative response rates and percentage of choice revealed some discrepancies between the free-operant analogue and discrete-trial procedures. Amount of reward controlled choice behavior when the ratios of delays were similar. When reward delays were more discrepant, delay length controlled choice behavior. Whereas the percentage of choice was larger for the larger magnitude reward, the relative rate of response for the larger magnitude was less than .50. In contrast, when the percentage of choice generally fell to below 50% (with large amount and large delay differences between alternatives), relative response rate indicated a preference for the larger amount alternative. This study shows the feasibility and utility of a free-operant analogue of discrete-choice studies that could be used to develop an analysis of preference.

Pigeons' (Columba livia) intertemporal choice in binary-choice and patch-leaving contexts.

Typical approaches to study self-control present subjects with a simultaneous choice between a larger-later (LL) reinforcer and a smaller-sooner (SS) reinforcer. In contrast, in patch-leaving tasks, subjects choose between staying at a patch for an SS (or LL) reinforcer and leaving for an LL (or SS) reinforcer. Previous studies show that blue jays, monkeys, humans, and rats prefer the SS reinforcer in binary-choice tasks, whereas the same subjects prefer the LL reinforcer in equivalent patch-leaving tasks. The current study systematically replicated this research using pigeons. Six pigeons responded in a binary-choice task and in two patch-leaving tasks in which staying led to an LL (Patch-L) or SS (Patch-S) reinforcer. Across conditions, the SS reinforcer delay varied from 5 to 55 s; the LL reinforcer delay was always 60 s. In binary-choice conditions, subjects preferred the SS reinforcer. In Patch-L and Patch-S conditions, subjects preferred the LL and SS reinforcer, respectively, reflecting a bias to stay at the patch. This bias persisted when the stay response was more effortful and when the delays to both reinforcers were equal. This may reflect a species-specific win-stay bias and the differential consequences of staying (which led to a stimulus signaling food) versus leaving (which led to a stimulus never associated with food). Thus, we propose a conditioned-reinforcement account of intertemporal choice in patch-leaving contexts. We suggest several avenues for further investigations of the mechanisms underlying intertemporal choice in different contexts and question the economic equivalence of the operant and patch-leaving procedures. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

The effect of periodicity of reinforcement on food accumulation

Recent studies on food accumulation have shown that the rate of food-procuring responses increases by delaying the delivery of food (i.e., an increasing delay gradient). However, these studies have only delivered the food periodically, which could lead to the establishment of a behavior sequence between successive food deliveries. Therefore, the increasing delay gradient could be due to the establishment of different behavior sequences, composed of more food-procuring responses as the delay of reinforcement is lengthened. The present study investigated whether the increasing delay gradient occurs when the establishment of a behavior sequence is hindered by eliminating the periodicity of reinforcement from a situation of food accumulation. The periodicity of reinforcement was eliminated by using a variable delay in Experiment 1 and a variable inter-trial interval in Experiment 2. It was found that the increasing delay gradient occurred both with periodic and aperiodic reinforcement, which may imply that it does not depend on the establishment of a behavior sequence. In contrast, the increasing delay gradient was attributed to the temporal distribution of the food-procuring period within the inter-reinforcement time.

Delay of reinforcement switches attention between elements of a compound stimulus in humans, Reynolds (1961) revisited

One experiment in instrumental conditioning studied the effect of delayed reinforcement on attention, considered as the stimulus control produced by each element of a compound sample stimulus (CSS). A two-element (shape and color) CSS indicated the response to be reinforced between three comparison stimuli in an online matching-to-sample task with human participants. Forty-eight university students were assigned to three groups with different reinforcement delays: 0 s, 8 s, and 16 s (D0, D8, and D16). After learning the task, a separate test took place, with the elements of the CSS presented separately. Group D0 responded more to the shape. The D8 group responded to both elements, while the D16 group responded more to the color. Results showed that attention to each element of the CSS changes as reinforcement delay increases. These findings are similar to those of Urcelay and Miller (2009) on Pavlovian conditioning, pointing out similarities between Reynolds’ findings and Pavlovian overshadowing.

Autoshaped impulsivity: Some explorations with a neural network model

This study evaluated the effect of delay and magnitude of reinforcement in Pavlovian contingencies, extending the understanding of the phenomenon of autoshaped impulsivity as described in Alcalá's thesis (2017) and Burgos and García-Leal (2015). The effects of adding a trace interval were analyzed on the maintained responses of impulsive choice, seen as the preference of a small and immediate reinforcer over a larger and delayed one, and the role of the contextual unit, as well as the inhibitory units according to the Diffuse Discrepancy Model. In the Simulation, the model with inhibitory units was used, trained in two signals with different delays and reinforcement magnitudes, and subsequently presented concurrently in choice tasks without reinforcement nor learning, using an ABA within-subject design. In general, the DD model successfully simulated the phenomenon of autoshaped impulsivity, consistent with studies from Alcalá's thesis (2017), Burgos and García-Leal (2015), and Picker and Poling (1982). It also predicted the elimination of this effect (autoshaped impulsivity) after introducing a trace interval. The observed results and their implications are discussed, as well as possible future studies with animals and humans.

Behavioral mechanisms of oxycodone's effects in female and male rats: Reinforcement delay and impulsive choice.

μ-Opioid agonists (e.g., morphine) typically increase impulsive choice, which has been interpreted as an opioid-induced increase in sensitivity to reinforcement delay. Relatively little research has been done with opioids other than morphine (e.g., oxycodone), or on sex differences in opioid effects, on impulsive choice. The present study investigated the effects of acute (0.1-1.0 mg/kg) and chronic (1.0 mg/kg twice/day) administration of oxycodone on choice controlled by reinforcement delay, a primary mechanism implicated in impulsive choice, in female and male rats. Rats responded under a concurrent-chains procedure designed to quantify the effects of reinforcement delay on choice within each session. For both sexes, choice was sensitive to delay under this procedure. Sensitivity to delay under baseline was slightly higher for males than females, suggesting more impulsive choice with males. When given acutely, intermediate and higher doses of oxycodone decreased sensitivity to delay; this effect was larger and more reliable in males than females. When given chronically, sex differences were also observed: tolerance developed to the sensitivity-decreasing effects in females, whereas sensitization developed in males. These data suggest that reinforcement delay may play an important role in sex differences in impulsive choice, as well as in the effects of acute and chronic administration of opioids in impulsive choice. However, drug-induced changes in impulsive choice could be related to at least two potential behavioral mechanisms: reinforcement delay and/or reinforcement magnitude. Effects of oxycodone on sensitivity to reinforcement magnitude remain to be fully characterized. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Evaluating Technology-Based Self-Monitoring of Performance with Differential Reinforcement for Students with Disabilities.

This study evaluated the use of a technology-based self-monitoring of performance (SMP) with differential reinforcement to increase task completion and reduce off-task behavior for three 5th-grade students with disabilities. A concurrent multiple baseline design across participants was used to examine the impact of the intervention on the targeted behaviors when implemented by a general education teacher and its maintenance effects with a delay of reinforcement. The implementation involved training students to use a mobile app for SMP and providing differential reinforcement contingent on task completion and accuracy of self-monitoring during academic periods. The secondary measure of off-task behavior was included to evaluate the relationship between task completion and engagement. The results demonstrated that the technology-based SMP with differential reinforcement increased task completion and reduced off-task behavior for all students. Furthermore, the gradual fading of the reinforcement, with a 45 min delay, was successful for all students. The efficiency and immediacy of the intervention suggest that technology-based SMP with differential reinforcement holds promise as a practical, efficient, and effective school-based intervention.

Previous experience with delays affects delay discounting in animal model of ADHD

BackgroundADHD is a disorder where a common symptom is impulsive behaviour, a broad term associated with making sub-optimal choices. One frequently used method to investigate impulsive behaviour is delay discounting, which involves choosing between a small, immediate reinforcer and a delayed, larger one. Choosing the small immediate reinforcer is by itself, however, not sufficient for terming the choice impulsive, as all organisms eventually switch to choosing the small, immediate reinforcer when the delay to the larger reinforcer becomes long. This switch can be termed impulsive only when it occurs more frequently, or at shorter LL delays, than typically observed in normal controls. A poorly understood aspect is how choice is influenced by previous experience with delays. Using an animal model of Attention-Deficit/Hyperactivity Disorder, the Spontaneously Hypertensive Rat, we manipulated the order of exposure to delays in a delay discounting task. Following a preference test, the Ascending group experienced gradually increasing delays between choice and reinforcer while the Descending group were exposed to these delays in reverse order.ResultsThe results showed that the Descending group chose the small, immediate reinforcer over the larger delayed to a much larger extent than the Ascending group, and continued to do so even when the delay component was ultimately removed. Strain effects were found in the Ascending group, with SHRs switching to the small, immediate reinforcer earlier than controls as the delay to the larger reinforcer increased.ConclusionThe data suggests that delay discounting is affected by history of exposure to delayed consequences. When reinforcement contingencies are incrementally changed from having no response-reinforcer delay to a long delay, discounting of delayed consequences is gradual. However, a sudden change from no delay to a long delay, without intermediate training, results in a rapid switch to the small, immediate reinforcer option, and this behaviour is somewhat resilient to the shortening and eventual removal of the large reinforcer delay. The implication is that attempting to reduce already existing impulsive behaviour in children with ADHD will require gradual habituation and not sudden changes in reinforcement contingencies.

A Rapid Assessment of Sensitivity to Reward Delays and Classwide Token Economy Savings for School-Aged Children.

Delay discounting tasks measure the relation between reinforcer delay and efficacy. The present study established the association between delay discounting and classroom behavior and introduced a brief measure quantifying sensitivity to reward delays for school-aged children. Study 1 reanalyzed data collected by Reed and Martens (J Appl Behav Anal 44(1):1-18, https://doi.org/10.1901/jaba.2011.44-1, 2011) and found that 1-month delay choices predicted student classroom behavior. Study 2 examined the utility of the 1-month delay indifference point in predicting saving and spending behavior of second-grade students using token economies with two different token production schedules. Collectively, results showed (a) the 1-month delay indifference point predicted classroom behavior, (b) children who discounted less and had greater self-regulation, accrued and saved more tokens, and (c) a variable token production schedule better correlated with discounting than a fixed schedule. Implications are discussed regarding utility of a rapid discounting assessment for applied use.

Increment in the consummatory response induced by reward delay: An animal model of binge-like eating episodes

The omission of an expected palatable reinforcer produces an aversive emotional state, called frustration, that could function as an important factor in the etiology and maintenance of the binge-eating disorder. A series of experiments carried out in adult male Wistar rats are presented, in which the consummatory response of sugary drinks was evaluated after events of frustration due to reward delay. The animals were trained in a consummatory behavior acquisition phase, in which they had access to different concentrations of sucrose solutions. In the test phase, a control group received the reinforcer normally, while an experimental group received it after a delay. A significant increase in the consummatory behavior of a 32 % sucrose was found in the group that was exposed to a 2-min delay before re-introducing the reinforcer (Experiment 1). The increment found in the consummatory response was dependent on the aversive state of frustration. This could be concluded because, in a less frustrating condition, the animals did not exhibit the phenomenon (Experiment 2), and a more frustrating one produced an increment of the effect (Experiment 3). This result was also observed using a direct measure of consumption -kilocalorie intake- during several events of reinforcement delay (Experiment 4).

Effects of signaled and unsignaled delays of reinforcement on response maintenance in zebrafish (Danio rerio).

Effects of delays of reinforcement on zebrafish behavior were examined following training with immediate reinforcement. The delay was either signaled by an exteroceptive stimulus present for the entire delay period (fully signaled), signaled briefly only at delay onset (partial signal), or without a stimulus (unsignaled). Unsignaled delays consistently resulted in low response rates. Fully signaled delays resulted in higher response rates than unsignaled delays when the delay was 3 s, but this difference in response rates disappeared at 6-s delays. Partially signaled delays were less effective in maintaining responding than fully signaled delays, but more effective than unsignaled delays, although these results were only suggestive. These results indicate that stimulus changes that occur during delays to reinforcement have similar effects with zebrafish as with other species, but also that responding of zebrafish has a relatively low tolerance to the delay duration. A vast majority of experiments examining zebrafish behavior suggests that the fish have potential to serve as an interface between biological and behavioral science, but this may not be the case in some research areas involving delays, such as delay discounting.

Covariance, feedback, and discounting in ratio schedules.

Action-outcome relations in the everyday world are often looser than the relations programmed in the laboratory. Reinforcement theory addressed this looseness by theorizing about "delay of reinforcement." Response-reinforcer interval affects response rate and choice, but need not be thought of as delay. Experiments on temporal discounting are often said to reveal effects of delay also, but the intervals chosen differ from delays, because an outcome after a week, month, or year occurs after life has continued and many events have intervened. They are better called lag intervals. The present experiment studied the effects of varying lag interval on pigeons' responding on ratio-like schedules of food. In a continuous procedure, a lag interval separated pecking that earned food from delivery of the food earned. In one series of conditions, the relation was fixed equivalent to variable-ratio (VR) 40 while the lag interval varied. In a second series, the equivalent VR varied while the lag interval was fixed. The results accorded with matching theory, but with a factor for temporal discounting or loose feedback. The results suggest that both temporal discounting and probability discounting may be characterized within matching theory, but that temporal discounting requires a separate treatment from probability discounting.

Effects of delay of reinforcement on ABA renewal

ABA renewal was assessed by decreasing response rates by means of extinction or a 12-s delay of reinforcement in one experiment with rats. In a first phase, rats were trained to lever press for water in Context A. Rats were exposed to a second phase in Context B, where lever-pressing decreased for all rats; for one group of rats, lever-pressing decreased by means of extinction, while lever-pressing decreased by means of a 12-s delay for the other group. During a third phase, both groups were once again exposed to Context A with no reinforcer deliveries for lever-pressing. Once rats were exposed to the latter phase, renewal was observed for both groups. Results suggest that delay of reinforcement can be used as an alternative procedure to help reduce behavior and mitigate relapse.

Effects of effort training on effort-based impulsive choice

Because impulsive decision-making is correlated with many maladaptive tendencies, researchers have increasingly studied methods for reducing impulsive choice. These efforts have primarily focused on increasing choices of larger, more-delayed rewards. A second type of impulsive choice is selecting a smaller, less-effortful reward over a larger, more-effortful one. Little nonhuman research has examined experimental methods for reducing effort-based impulsive choice. Within the realm of delay-based impulsive choice, extended exposure to reinforcer delays has proven effective in reducing impulsive choices in rats. The current study took a similar tack by evaluating if reductions in effort-based impulsive choice could be achieved by providing rats with extended exposure to high-effort training, that is, reinforcement contingencies requiring a large number of responses. Male rats were randomly assigned to either a high-effort training (fixed-ratio 50) or low-effort training (fixed-ratio 1) group. Following training, both groups completed a test of effort-based impulsive choice. High-effort training produced a temporary reduction in effort-based impulsive choice. In the same test phase, groups also differed in response run durations (shorter following high-effort training), with larger differences in the initial sessions, which may have contributed to the short-term effect of high-effort training in effort-based impulsive choice.

Baseline-dependency: How genotype and signaled delays influence amphetamine's effects on delay discounting

RationaleDelay discounting, in which an animal chooses between a small, immediate or large, delayed reinforcer, is an experimental model of impulsivity. In previous studies, d-amphetamine has both increased and decreased preference for larger-delayed reinforcers depending on experimental conditions. ObjectiveIdentify genotype X environment interactions responsible for these disparate findings in a single study and assess the hypothesis that baseline-dependence unifies d-amphetamine's effects. MethodsDelay discounting by BALB/c and C57Bl/6 mice was evaluated using a choice procedure in which six delays to a larger reinforcer were presented in a single session. Components were presented both with and without stimuli that uniquely signaled reinforcer delays. d-Amphetamine's (0.1–1.7 mg/kg) effects on delay and magnitude sensitivity were assessed when specific stimuli did or did not uniquely signal the delay to a larger reinforcer. d-Amphetamine's effects were determined using a model-comparison approach. ResultsDuring baseline, magnitude and delay sensitivity were identical across signal conditions for BALB/c mice and generally greater than the C57Bl/6 mice. For C57Bl/6 mice, magnitude and delay sensitivity were higher during the signaled than the unsignaled component. Amphetamine decreased delay sensitivity during both components for BALB/c mice, but this effect was attenuated by delay-specific stimuli. For C57Bl/6 mice, amphetamine decreased their high magnitude and delay sensitivity when delays were signaled and, conversely, increased the low magnitude and delay sensitivity when delays were unsignaled. ConclusionsBALB/c mice showed high delay and magnitude sensitivity regardless of signal conditions. C57Bl/6's magnitude and delay sensitivity depended on signaling. d-Amphetamine usually decreased high baseline delay- and magnitude sensitivity and increased low sensitivities, a baseline-dependence that occurred regardless of whether delay sensitivity was driven by biological (genotype) or environmental (signaling) variables. The C57Bl/6 mouse may be a good model of environmentally-induced impulsivity while BALB/c mice could model impulsivity with a strong genetic contribution.

Reinforcer pathology II: Reward magnitude, reward delay, and demand for alcohol collectively relate to college students' alcohol related problems.

The reinforcer pathologies model of addiction posits that two characteristic patterns of operant behavior characterize addiction. Specifically, individuals suffering from addiction have elevated levels of behavioral economic demand for their substances of abuse and have an elevated tendency to devalue delayed rewards (reflected in high delay discounting rates). Prior research has demonstrated that these behavioral economic markers are significant predictors of many of college students' alcohol-related problems. Delay discounting, however, is a complex behavioral performance likely undergirded by multiple behavioral processes. Emerging analytical approaches have isolated the role of participants' sensitivity to changes in reinforcer magnitude and changes in reinforcer delay. The current study uses these analytic approaches to compare participants' discounting of money versus alcohol, and to build regression models that leverage these new insights to predict a wider range of college students' alcohol related problems. Using these techniques, we were able to 1) demonstrate that individuals differed in their sensitivity to magnitudes of alcohol versus money, but not sensitivity to delays to those commodities and 2) that we could use our behavioral economic measures to predict a range of students' alcohol related problems.

Failure to find a distance effect in pigeon choice: Manipulating amount and delay of reinforcement

The choice behavior of primates, including humans, displays a distance effect: Latency to choose between alternatives appears to increase with smaller differences in value. There is, so far, no demonstration of this effect in birds. Tests of distance effects in birds have been conducted in binary choice situations with a dominant alternative, where one alternative is superior to the other in all aspects that meaningfully contribute to value (e.g., provides access to the same reinforcer, but with a shorter delay). The present study considers the possibility that including dominant alternatives in choice tests precludes distance effects. Four pigeons were presented with binary choices between alternatives that varied in amount and delay. Some choices had a dominant alternative (smaller-sooner or larger-later vs. smaller-later) and some did not (smaller-sooner vs. larger-later). Across phases, only the delay to the smaller-sooner reinforcer varied. Distance effects were expected to be expressed as longer latencies as choice between smaller-sooner and larger-later reinforcers approached indifference. Despite the sensitivity of choice to differences in amount and delay, no distance effect was observed. Alternative explanations for the failure to find a distance effect in pigeon choice, including the Sequential Choice Model (SCM), are discussed.

Enhancing "self-control": The paradoxical effect of delay of reinforcement.

Delay of reinforcement is generally thought to be inversely correlated with speed of acquisition. However, in the case of simultaneous discrimination learning, in which choice results in immediate reinforcement, delay of reinforcement can improve acquisition. For example, in the ephemeral reward task, animals are given a choice between two alternatives, A and B. Choice of A provides reinforcement, and the trial is over. Choice of B provides reinforcement and access to alternative A (thus, two reinforcements). Many animals appear unable to learn to choose B consistently, but inserting a 20-s delay between choice and outcome has been shown to facilitate optimal choice. Similarly, pigeons given a choice between a signal for one pellet and a signal for two pellets (each occurring without a delay) have difficulty learning to choose the two-pellet alternative, unless the reinforcement is delayed. In a version of object permanence, food is placed in one of two containers, and the pigeon must choose the container with the food. Pigeons have difficulty reliably choosing the correct container unless a brief delay is inserted between baiting and choice. Finally, pigeons have been shown to prefer a suboptimal alternative (a 20% chance of getting a cue for reinforcement) over an optimal alternative (a 100% chance of getting a cue for 50% reinforcement). However, if pigeons are forced to wait 20 s following their choice to receive the cues, no preference for the suboptimal alternative is found. Thus, impulsive choice may be reduced by delaying the consequence of that choice.

Delayed rewards facilitate habit formation.

The transition from goal-directed to habitual forms of instrumental behavior is determined by variables such as the amount of training, schedules of reinforcement, the availability of choices, and exposure to drugs of abuse. Less is known about the control of goal-directed behavior when reinforcement is delayed rather than immediate. In these experiments, we investigated in rats the role of response-outcome contiguity on the control of goal-directed action, assessed through satiety-specific outcome devaluation tests. In Experiment 1 using a within-subjects design we observed goal-directed behavior after 6 days of FR1 training when the outcome was presented immediately following the lever press, but not when it was delayed for 20 s, revealing habit formation with delayed outcomes. Experiment 2 revealed that the habitual control observed with 20-s delays of reinforcement can be prevented if, immediately before each instrumental training session, the rats were exposed to the experimental context in the absence of both the lever and reinforcement. In summary, these experiments suggest that response-outcome contiguity plays an important role in the control of goal-directed actions and habits. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Effects of schedule type and terminal‐link duration on choice in concurrent chains

Four pigeons responded in a three-component concurrent chains procedure in which the terminal-link schedules were either both fixed-interval (FI FI), both variable-interval (VI VI), both mixed-interval (MI MI) or variable-interval fixed-interval (VI FI). Across components within sessions, overall terminal-link duration was varied while schedule types were varied across conditions. For the conditions with homogeneous schedules, the strongest preference was obtained with FI FI, intermediate with MI MI, and weakest with VI VI. Preference increased with overall terminal-link duration for all schedules, but the increase was more rapid for VI FI. The hyperbolic value-added model (HVA) and cumulative decision model (CDM) were fitted to the choice data and accounted for 83.9% and 76.1% of the variance, respectively. However, deviations from both models' predictions were systematic. A modification of the CDM, which assumed that reinforcers delivered after variable delays were more effective, improved the fit of the model and substantially reduced the systematic deviations in residuals. Comparable modifications of HVA produced only limited improvement. Results show that preference in concurrent chains with homogeneous terminal links depends on the degree of variability in reinforcer delays, and that the CDM can provide an excellent account of results across different terminal-link schedules.