Variable-ratio Schedules Research Articles

Using Programmed Schedules of Reinforcement to Increase the Variability of Reinforcer Delivery in Classroom Assistant-Implemented Variable-Ratio Schedules

Using Programmed Schedules of Reinforcement to Increase the Variability of Reinforcer Delivery in Classroom Assistant-Implemented Variable-Ratio Schedules

Feedback‐induced action–outcome associations increase consumer impatience

AbstractDo various kinds of feedback influence consumer impatience? Five studies involving hypothetical and real behavioral consequences demonstrate that compared with lump‐sum feedback (i.e., feedback provided in bulk at the end), piecemeal feedback (i.e., feedback provided piece by piece in the process) increases consumer impatience (i.e., preference toward now options). This effect occurs because piecemeal feedback (vs. lump‐sum feedback) establishes a reliable action–outcome association, which activates a general action goal to induce consumers to be more action oriented, making them more eager to complete actions for outcomes in subsequent related or unrelated situations. This effect is robust regardless of whether the valence of feedback is positive or negative, whether the outcome involves gain or loss (Study 1), and whether the form of feedback is monetary or informative (Study 2). Furthermore, we show that piecemeal feedback increases consumer impatience only when it is provided at a fixed ratio rather than at a variable ratio schedule (Study 3), when it is provided directly following behaviors (Study 4), and when it is directed to actions (vs. inactions) (Study 5). These findings contribute to the action goal research and consumer impatience literature.

Choice between food and cocaine reinforcers under fixed and variable schedules in female and male rhesus monkeys.

Illicit drugs like cocaine may be uncertain in terms of the time and effort required to obtain them. Behavior maintained by variable schedules resembles excessive drug-taking compared with fixed schedules. However, no prior research has examined fixed versus variable schedules in drug versus nondrug choice. The present study evaluated cocaine versus food choice under fixed- (FR) and variable-ratio (VR) schedules. The simpler food versus food and cocaine versus cocaine arrangements also were included. Adult female (n = 6) and male (n = 7) rhesus monkeys chose between cocaine (0.01-0.18 mg/kg/injection) and food (4 pellets/delivery), food and food (4 pellets/delivery), or cocaine and cocaine (0.018-0.03 mg/kg/injection) under FR and VR 100 and 200 schedules. In cocaine versus food choice, cocaine's potency to maintain choice was greatest when available under a VR 100 or 200 schedule and food under an FR schedule and was lowest when cocaine was available under an FR 200 schedule and food was available under a VR 200 schedule. In food versus food choice, males chose food associated with a VR schedule more than food associated with an FR schedule. In cocaine versus cocaine choice, females and males chose cocaine associated with a VR schedule more than cocaine associated with an FR schedule, particularly under VR 200. These findings suggest that uncertainty in terms of time and effort required to obtain cocaine, or perhaps the occasional low-cost access that results from VR schedules, results in greater allocation of behavior toward drug reinforcers at the expense of more certain, nondrug alternatives. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Choice dynamics in concurrent ratio schedules of reinforcement.

The generalized matching law predicts performance on concurrent schedules when variable-interval schedules are programmed but is trivially applicable when independent ratio schedules are used. Responding usually is exclusive to the schedule with the lowest response requirement. Determining a method to program concurrent ratio schedules such that matching analyses can be usefully employed would extend the generality of matching research and lead to new avenues of research. In the present experiments, ratio schedules were programmed dependently such that responses to either of the two options progressed the requirement on both schedules. Responding is not exclusive because the probability of reinforcement increases on both schedules as responses are allocated to either schedule. In Experiment 1, performance on concurrent variable-ratio schedules was assessed, and reinforcer ratios were varied across conditions to investigate changes in sensitivity. Additionally, the length of a changeover delay was manipulated. In Experiment 2, performance was compared under concurrently available, dependently programmed variable-ratio and fixed-ratio schedules. Performance was well described by the generalized matching law. Increases in the changeover delay decreased sensitivity, whereas sensitivity was higher when variable-ratio schedules were employed, compared with fixed-ratio schedules. Concurrent ratio schedules can be a viable approach to studying functional differences between ratio and interval schedules.

Rate matching, probability matching, and optimization in concurrent ratio schedules.

Much research has documented rate matching in concurrent variable-interval schedules, but comparatively little research has examined performance in concurrent variable-ratio schedules, except in discrete-trials procedures that sometimes produce probability matching. One should expect that the two types of schedule would result in different performances, because ratio schedules cannot improve with time the way interval schedules do; ratio schedules lack the temporal dynamics of interval schedules. The present experiment exposed rats to concurrent variable-ratio schedules. Seven unsignaled components were presented in random order within each daily session, with probability ratios ranging from 1:8 to 8:1. Three conditions were studied that varied the overall probability of food while leaving probability ratios the same. Choice appeared to conform to probability matching, because sensitivities in the rate-matching relation were close to 0.5, whereas sensitivities to probability ratio were close to 1.0. The sensitivities alone, however, could not confirm probability matching, because undermatching to rate occurs often. Analyses at smaller time scales supported the interpretation of probability matching. In particular, control by food deliveries was highly local in concurrent variable-ratio schedules, in contrast with concurrent variable-interval schedules, in which control is extended. Activity continued to switch between alternatives throughout components, contradicting optimal sampling theory.

Reinforcement History Dependent Effects of Low Dose Ethanol on Reward Motivation in Male and Female Mice.

Alcohol use disorders (AUDs) are more prevalent in men than in women, though AUD diagnoses in women are growing rapidly, making an understanding of sex differences in alcohol-related behaviors increasingly important. The development of AUDs involves the transition from casual, low levels of alcohol drinking to higher, maladaptive levels. The ability of low dose alcohol to drive reward and drug seeking may differ in males and females, and this could underlie differences in susceptibility to AUD. In this study we sought to determine whether a history of chronic, low dose ethanol exposure (0.5 g/kg; i.p.) could drive sucrose reward seeking and motivation, and whether this differed between male and female mice. Adult mice were trained to lever press for a liquid sucrose reward on two reinforcement schedules: a random interval (RI) schedule and a variable ratio (VR) schedule. After training, mice were tested on each of these levers for reward motivation using a progressive ratio test. We found that a history of low dose ethanol exposure increased sucrose reward motivation in male mice, but only on the RI lever and only when exposure occurred proximal to learning. Female mice were more motivated for sucrose on the RI lever than the VR lever regardless of ethanol exposure condition. These findings indicate that training on different reinforcement schedules affects reward motivation. Further, we show that males are more susceptible to the effects of low dose ethanol on sucrose reward motivation than females. These data broaden our understanding of sex differences in reward seeking as a result of ethanol exposure.

The influence of reinforcement schedule on experience-dependent changes in motivation.

The progressive ratio procedure is used across fields to assess motivation for different reinforcers, define the effects of experimental interventions on motivation, and determine experience-dependent changes in motivation. However, less is known about how operant training schedules affect performance on this widely utilized task. Here we designed an experiment to examine the effect of variable ratio versus fixed ratio training schedules of reinforcement on progressive ratio performance while holding other performance variables constant between groups. We found a robust increase in maximum ratio completed between the pretest and posttraining test highlighting a robust training effect on progressive ratio performance. However, it did not matter if the training was under a fixed or variable ratio schedule. Additionally, we show that neither individual rates during training nor extinction responding correlated with maximum ratio achieved during the sessions. Finally, we show that rates during the training sessions do correlate with extinction performance, suggesting that these variables measure a different aspect of performance that does not predict motivation.

Addictive Games: Case Study on Multi-Armed Bandit Game

The attraction of games comes from the player being able to have fun in games. Gambling games that are based on the Variable-Ratio schedule in Skinner’s experiment are the most typical addictive games. It is necessary to clarify the reason why typical gambling games are simple but addictive. Also, the Multiarmed Bandit game is a typical test for Skinner Box design and is most popular in the gambling house, which is a good example to analyze. This article mainly focuses on expanding on the idea of the motion in mind model in the scene of Multiarmed Bandit games, quantifying the player’s psychological inclination by simulation experimental data. By relating with the quantification of player satisfaction and play comfort, the expectation’s feeling is discussed from the energy perspective. Two different energies are proposed: player-side (Er) and game-side energy (Ei). This provides the difference of player-side (Er) and game-side energy (Ei), denoted as Ed to show the player’s psychological gap. Ten settings of mass bandit were simulated. It was found that the setting of the best player confidence (Er) and entry difficulty (Ei) can balance player expectation. The simulation results show that when m=0.3,0.7, the player has the biggest psychological gap, which expresses that player will be motivated by not being reconciled. Moreover, addiction is likely to occur when m∈[0.5,0.7]. Such an approach can also help the developers and educators increase edutainment games’ efficiency and make the game more attractive.

Towards an animal model of consciousness based on the platform theory

The evolution of intellectual capacities has brought forth a continuum of consciousness levels subserved by neuronal networks of varying complexity. Brain pathologies, neurodegenerative, and mental diseases affect conscious cognition and behavior. Although impairments in consciousness are among the most devastating consequences of neurological and mental diseases, valid and reliable animal models of consciousness, that could be used for preclinical research are missing. The platform theory holds that the brain enters a conscious operation mode, whenever mental representations of stimuli, associations, concepts, memories, and experiences are effortfully maintained (in working memory) and actively manipulated. We used the platform theory as a framework and evaluation standard to categorize behavioral paradigms with respect to the level of consciousness involved in task performance. According to the platform theory, a behavioral paradigm involves conscious cognitive operations, when the problem posed is unexpected, novel or requires the maintenance and manipulation of a large amount of information to perform cognitive operations on them. Conscious cognitive operations are associated with a relocation of processing resources and the redirection of attentional focus. A consciousness behavioral test battery is proposed that is composed of tests which are assumed to require higher levels of consciousness as compared to other tasks and paradigms. The consciousness test battery for rodents includes the following tests: Working memory in the radial arm maze, episodic-like memory, prospective memory, detour test, and operant conditioning with concurrent variable-interval variable-ratio schedules. Performance in this test battery can be contrasted with the performance in paradigms and tests that require lower levels of consciousness. Additionally, a second more comprehensive behavioral test battery is proposed to control for behavioral phenotypes not related to consciousness. Our theory could serve as a guidance for the decryption of the neurobiological basis of consciousness.

An optimized procedure for robust volitional cocaine intake in mice.

Substance use disorder (SUD) is a behavioral disorder characterized by volitional drug consumption. Mouse models of SUD allow for the use of molecular, genetic, and circuit-level tools, providing enormous potential for defining the underlying mechanisms of this disorder. However, the relevance of results depends on the validity of the mouse models used. Self-administration models have long been the preferred preclinical model for SUD as they allow for volitional drug consumption, thus providing strong face validity. While previous work has defined the parameters that influence intravenous cocaine self-administration in other species-such as rats and primates-many of these parameters have not been explicitly assessed in mice. In a series of experiments, we showed that commonly used mouse models of self-administration, where behavior is maintained on a fixed-ratio schedule of reinforcement, show similar levels of responding in the presence and absence of drug delivery-demonstrating that it is impossible to determine when drug consumption is and is not volitional. To address these issues, we have developed a novel mouse self-administration procedure where animals do not need to be pretrained on sucrose and behavior is maintained on a variable-ratio schedule of reinforcement. This procedure increases rates of reinforcement behavior, increases levels of drug intake, and results in clearer delineation between drug-reinforced and saline conditions. Together, these data highlight a major issue with fixed-ratio models in mice that complicates subsequent analysis and provide a simple approach to minimize these confounds with variable-ratio schedules of reinforcement. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

An experimental analysis of the extinction-induced response burst.

The occurrence, time course, and repeatability of response rate increases following the onset of extinction, the extinction burst, were studied in three experiments. Nine pigeons were exposed to at least 5 cycles of 5-session blocks of conditioning followed by 8-session blocks of extinction. In different experiments, conditioning sessions either were a fixed-ratio (FR) or variable-ratio (VR) schedule, and transitions from the last conditioning session in each cycle to the first extinction session were conducted between or within sessions. A single response rate increase occurred when overall response rates were considered. Restricting analyses to the first minute of extinction sessions sometimes revealed increases in response rates, although this finding was inconsistent. The frequency and magnitude of these increases differed across exposures to extinction both across and within pigeons. Additionally, how responding during extinction was measured (i.e., the level of analysis) influenced whether increases above baseline levels were observed. These results suggest that the extinction burst may be influenced by the manner of transition to extinction and the way in which early extinction responding is measured. Under the best conditions, the extinction burst does not appear to be a ubiquitous effect of extinction.

Early extinction effects following intermittent reinforcement: Little evidence of extinction bursts.

The occurrence of extinction bursts-transient increases in response rate in excess of those observed in baseline during the period immediately following discontinuation of reinforcement of a response-was examined. In Experiment 1, key pecking of pigeons was reinforced according to a multiple schedule in which a variable-ratio schedule alternated with an interval schedule in which the reinforcers were yoked to the preceding variable-ratio component. In Experiments 2 and 3, rats were screened such that the lever-press response rates of different rats maintained by variable-interval schedules were either relatively high or relatively low. Following these baseline conditions, in Experiments 1 and 2 responding was extinguished by eliminating the food reinforcer and in Experiment 3 by removing the response-reinforcer dependency. Responses immediately following extinction implementation were examined. Response increases relative to baseline during the first 20 min of a 324.75-min extinction session (Experiment 1) or during the first 30-min extinction session (Experiments 2 and 3) were rare and unsystematic. The results (a) reinforce earlier meta-analyses concluding that extinction bursts may be a less ubiquitous early effect of extinction than has been suggested and (b) invite further experimentation to establish their generality as a function of preceding reinforcement conditions.

Adventitious reinforcement during long-duration DRO exposure.

Differential reinforcement of other behavior (DRO) is a procedure often used to decrease problem behavior, but the processes responsible for behavior reduction are not well understood. This study assessed whether adventitious reinforcement of other behavior contributes to DRO effectiveness when, relative to previous research, DRO exposure is prolonged. Two response options were presented on a computer and target responding was reinforced on a variable-ratio schedule. Response rates were then compared during DRO versus yoked variable-time or extinction probes. Across 2 experiments, DRO decreased target responding and increased other responding more than control conditions. However, increases in other responding did not usually maintain despite target responding remaining at low levels. DRO might adventitiously reinforce other responses transiently but the decreases in target behavior could not be entirely explained by adventitious reinforcement of the other response. Instead, reductions in target responding likely depend on the discriminability of the DRO contingency.

The role of adventitious reinforcement during differential reinforcement of other behavior: A systematic replication

Differential reinforcement of other behavior (DRO) is commonly used to decrease problem behavior by presenting reinforcers contingent upon the absence of a target response. Although it is well demonstrated that DROs decrease response rates, the processes producing these decreases are not well understood. The present study systematically replicated previous research assessing whether adventitious reinforcement of alternative behavior contributes to the effectiveness of DRO. We presented university students with two options on a computer and reinforced target responding on a variable-ratio schedule. Next, we compared decreases in target-response rates and any increases in alternative responding during DRO schedules versus yoked variable-time schedules or extinction probes. DRO schedules resulted in the lowest target-response rate and highest alternative-response rate. These findings generally provide some support for the adventitious reinforcement of "other" behavior.

Effects of exposure to chronic uncertainty and a sensitizing regimen of amphetamine injections on locomotion, decision-making, and dopamine receptors in rats.

Gambling disorder (GD) is a behavioral addiction that may be linked to alterations in dopamine (DA) systems. Gambling involves chronic exposure to uncertain reward, which can sensitize the activity of DA systems. Here we explored how combinations of Pavlovian and instrumental uncertainty impact DA sensitization and risky decision-making. Experiment 1: 40 rats underwent 66 uncertainty exposure (UE) sessions during which they responded for saccharin. Animal responding was reinforced according to a fixed or variable (FR/VR) ratio schedule that turned on a conditioned stimulus (CS; light), which predicted saccharin on 50% or 100% of trials. Animals responded under one of the four conditions: FR-CS100% (no uncertainty), VR-CS100%, FR-CS50%, and VR-CS50% (maximal uncertainty). DA sensitization was inferred from an enhanced locomotor response to d-amphetamine (d-AMPH; 0.5 mg/kg) challenge. The rat gambling task (rGT) was used to assess decision-making. Experiment 2: 24 rats received 5 weeks of sensitizing d-AMPH or saline doses, followed by locomotor activity and rGT testing. Experiment 3: Effects of UE and a sensitizing d-AMPH regimen on DA D1, D2, and D3 receptor binding were assessed in 44 rats using autoradiography. Compared to FR-CS100%, VR-CS100% and VR-CS50% rats displayed a greater locomotor response to d-AMPH, and VR-CS50% rats demonstrated riskier decision-making. Chronic d-AMPH-treated rats mirrored the effects of VR-CS50% groups on these two indices. Both VR-CS50% and d-AMPH-treated groups had increased striatal DA D2 receptor binding. These results suggest that chronic uncertainty exposure, similar to exposure to a sensitizing d-AMPH regimen, sensitized the function of DA systems and increased risky decision-making.

Examining stimuli paired with alternative reinforcement to mitigate resurgence in children diagnosed with autism spectrum disorder and pigeons.

In two laboratory experiments, we examined whether stimuli paired with alternative reinforcers could mitigate resurgence of a previously reinforced target response with pigeons (Experiment 1) and children diagnosed with Autism Spectrum Disorder (Experiment 2). In Phase 1, we arranged food reinforcement according to a variable-ratio schedule for engaging in a target response. In Phase 2, we arranged extinction for target responding and differentially reinforced alternative responding according to a fixed-ratio schedule, with every alternative-reinforcer delivery paired with a change in keylight color (Experiment 1) or automated verbal (praise) statement (Experiment 2). In Phase 3, we assessed resurgence during extinction of target and alternative responding in the presence versus absence of continued presentation of the paired stimulus. Despite variation across sessions, resurgence on average was lower when continuing to present the paired stimuli in all pigeons and children while maintenance of alternative responding did not differ between assessments. These findings indicate that stimuli paired with alternative reinforcement can modestly decrease resurgence, but further examination of their efficacy and a better understanding of the underlying processes are necessary before they can be recommended for clinical use in reducing resurgence of clinically relevant problem behavior.

Context affects resurgence of negatively reinforced human behavior

The effects of context on the resurgence of negatively reinforced (escape) responding was studied in an experiment with undergraduate students in which resurgence and renewal procedures were combined. Across conditions, in baseline (BL), key-pressing produced 3-s timeouts from pressing a force cell on a variable-ratio schedule of reinforcement; in the Alternative-Reinforcement and Test phases, a differential-reinforcement-of-other-behavior schedule and extinction were in effect, respectively. Conditions differed according to the context (the computer-screen color) in effect in each phase: ABA vs. ABB (with each letter representing a context; order of exposure to conditions was counterbalanced across participants). For each of six participants, independent of (a) order of exposure to conditions, (b) slight differences in BL reinforcement, and (c) differences in BL key-pressing rates, resurgence of greater magnitude occurred in the ABA than in the ABB condition. These results replicate and extend to contingencies of negative reinforcement previous findings with nonhumans and humans showing that context modulates the magnitude of resurgence.

Reinforcement schedules differentially affect learning in neuronal operant conditioning in rats

Operant conditioning of neuronal activity is a core process for better operation of brain-machine interfaces. However, few studies have investigated the role of reinforcement schedules in neuronal operant conditioning, although they are very effective in behavioral operant conditioning. To test the effect of different reinforcement schedules, the authors trained single-neuron activity in the motor cortex using fixed ratio (FR) and variable ratio (VR) schedules in rats. Neuronal firing rates were enhanced in the FR but not in the VR schedule during conditioning, suggesting that the principles of operant conditioning of neuronal activity are different from those of behavioral responses.

Adquisición y Dinámica de Preferencias en Programas Múltiples de Reforzamiento

Ocho palomas fueron entrenadas en programas múltiples de reforzamiento Razón Variable-Razón Variable (mult RV-RV) con cambios rápidos e imprevistos en las distribuciones de refuerzo en ambos componentes del programa múltiple. El objetivo principal fue evaluar cómo se ajustan las tasas de respuestas a cambios abruptos y no señalados en las condiciones de reforzamiento en distintos períodos y en particular determinar si la dinámica del ajuste del comportamiento es dirigida por la razón o por la diferencia en las probabilidades en las tasas de reforzamiento obtenido en dos componentes de un programa múltiple. Los principales hallazgos fueron que cuando las diferencias entre dos programas de reforzamiento (pobre y rico) son constantes, el desarrollo de la preferencia por una de las respuestas del programa múltiple fue más rápido cuando la razón de las probabilidades de reforzamiento fue mayor (5 a 1), lo cual es congruente con los resultados obtenidos en programas de reforzamiento concurrentes. Sin embargo, cuando la razón se mantuvo constante, la tasa de adquisición no fue más rápida cuando la diferencia entre la probabilidad de reforzamiento fue mayor, hallazgo distinto al reportado en experimentos con programas concurrentes. Los resultados resaltan la importancia de la discriminabilidad en probabilidades de reforzamiento entre la fase de entrenamiento y de transición.

Equidad en Intercambios de Esfuerzo y Ganancias: Efectos de la Información

Este trabajo aborda experimentalmente el estudio de los intercambios equitativos en una tarea en la que doce díadas de estudiantes de licenciatura en psicología asignaban a su compañero uno de tres posibles ―ujegos‖ asociados con requisitos diferenciales de un programa de reforzamiento de razón variable (RV: 5, 10 ó 20). En una segunda condición, los participantes asignaban también la cantidad de puntos que obtendría su pareja (1, 2 ó 4 puntos) con cada requisito de razón. Se programaron dos clases de sesiones: con información o sin información de la asignación de juego que hacían los compañeros en cada ensayo. Para evaluar las estrategias utilizadas se empleó el modelo Aristotélico de la Equidad (Adams, 1965; Anderson, 1976), mismo que describió adecuadamente la razón de esfuerzo y ganancias en los intercambios bajo la condición de información. Se discuten aquellas estrategias diádicas que resultaron en una distribución de mayor cantidad de ganancias con el menor esfuerzo posible, así como aquellas que produjeron distribuciones asimétricas de ganancias y esfuerzo, lo que desencadenó ganancias relativamente reducidas a lo largo de las sesiones. Especial énfasis recibe la discusión del mecanismo de reciprocidad en el intercambio social.