Partial Reinforcement Research Articles

Context Learning in the Rodent Hippocampus

We present a Bayesian statistical theory of context learning in the rodent hippocampus. While context is often defined in an experimental setting in relation to specific background cues or task demands, we advance a single, more general notion of context that suffices for a variety of learning phenomena. Specifically, a context is defined as a statistically stationary distribution of experiences, and context learning is defined as the problem of how to form contexts out of groups of experiences that cluster together in time. The challenge of context learning is solving the model selection problem: How many contexts make up the rodent's world? Solving this problem requires balancing two opposing goals: minimize the variability of the distribution of experiences within a context and minimize the likelihood of transitioning between contexts. The theory provides an understanding of why hippocampal place cell remapping sometimes develops gradually over many days of experience and why even consistent landmark differences may need to be relearned after other environmental changes. The theory provides an explanation for progressive performance improvements in serial reversal learning, based on a clear dissociation between the incremental process of context learning and the relatively abrupt context selection process. The impact of partial reinforcement on reversal learning is also addressed. Finally, the theory explains why alternating sequence learning does not consistently result in unique context-dependent sequence representations in hippocampus.

CS-duration and partial-reinforcement effects counteract overshadowing in select situations

Two experiments used rats in a conditioned lick suppression preparation to investigate how the conditioned stimulus (CS)-duration and partial-reinforcement effects (i.e., weakened responding due to conditioning with a CS of longer duration and presenting nonreinforced CSs intermingled with CS-unconditioned stimulus [US] pairings, respectively) interact with overshadowing. Experiment 1 found that when overshadowing treatment was combined with either extended CS duration or partial reinforcement, the response deficit was weaker than when either of these three treatments was administered alone. In Experiment 2, the generality of the findings in Experiment 1 was investigated by replicating it with various US-US intervals. This time counteraction was observed only when both the absolute duration of total CS exposure and the US-US interval were short. The results support neither the view that the ratio between the total CS exposure and total time in the context determines the CS-duration and the partial-reinforcement effects nor the view that these two effects arise from a loss of effectiveness of the excitatory CS-US association during CS-alone exposures in partial reinforcement or early periods of CS exposure with long CSs.

Spontaneous recovery after reversal and partial reinforcement

Six experiments used magazine approach in rat subjects to explore changes with time in responding for stimuli brought to a common moderate level of performance through acquisition or extinction. They found no evidence for increases with time in behavior during stimuli given simple acquisition. However, stimuli brought to that same level by reversal learning, repeated reversal, or partial reinforcement all showed increases in responding with time. These results suggest that the decremental process established by nonreinforcement endures through subsequent reinforcement and is especially sensitive to the passage of time.

Resistance to extinction and punishment following training with shock and non-reinforcement: Failure to obtain cross-tolerance

Two experiments investigated the phenomenon of cross-tolerance between the partial reinforcement extinction effect (PREE) and the partial punishment effect (PPE). Three groups of rats were trained in acquisition to run in a straight alley. The continuously reinforced (CRF) group received a reward on every trial. The partially reinforced (PRF) group was rewarded on a quasi-random 50% schedule. The partially punished (PP) group received food reward on every trial but, in addition, received foot shocks of gradually increasing intensity in the goal box on a random 50% of the trials. In the test stage, half of the animals in each training condition were tested in extinction, where no reward was given on any of the trials, and the other half were tested in punishment, with both food and shock presented on each trial. Experiment 1 used a 1-trial/day procedure; Experiment 2 used a multi-trial procedure. In both procedures, clear PREE and PPE were obtained. In the 1-trial/day procedure, no cross-tolerance was evid...

Reversed PREE under multiple schedules: Exploration of a modulation hypothesis

When reinforcer rates are manipulated in within-subjects designs, persistence in subsequent extinction trials is sometimes greater to the response alternative associated with the higher reinforcer rate (“reversed” partial reinforcement extinction effect, RPREE). The RPREE is often held to be a contradiction to the conventional PREE. To explore the interaction between these effects, student subjects were exposed to a multiple high-density and low-density reinforcement schedule, then to extinction conditions. Subjects in two control conditions received the high- and low-density schedules only. For all groups, general extinction persistence was elevated, and a within-subjects RPREE occurred under the multiple schedule. The RPREE was caused by the low-density component increasing persistence to the high-density component; the high-density component did not decrease persistence to the low-density component. These results support the view that the RPREE may be a within-subjects effect of the conventional PREE.

Interoceptive fear conditioning as a learning model of panic disorder: An experimental evaluation using 20% CO 2-enriched air in a non-clinical sample

Despite the role afforded interoceptive fear conditioning in etiologic accounts of panic disorder, there are no good experimental demonstrations of such learning in humans. The aim of the present study was to evaluate the interoceptive conditioning account using 20% carbon dioxide (CO 2)-enriched air as an interoceptive conditioned stimulus (CS) (i.e., physiologically inert 5-s exposures) and unconditioned stimulus (US) (i.e., physiologically prepotent 15-s exposures). Healthy participants ( N=42) were randomly assigned to one of three conditions: a CS-only, contingent CS–US pairings, or unpaired/non-contingent CS and US presentations. Electrodermal and self-report (e.g., distress, fear) served as indices of conditioned emotional responding. Results showed greater magnitude electrodermal and evaluative fear conditioning in the paired relative to the CS-only condition. The explicitly unpaired condition showed even greater electrodermal and evaluative responding during acquisition, and marked resistance to extinction. The latter results are consistent with the possibility that the unpaired procedure constituted a partial reinforcement procedure in which CO 2 onset was paired with more extended CO 2 exposure on 50% of the trials. Overall, the findings are consistent with contemporary learning theory accounts of panic.

Using TD learning to simulate working memory performance in a model of the prefrontal cortex and basal ganglia

Delayed-response tasks (DRTs) have been used to assess working memory (WM) processes in human and nonhuman animals. Experiments have shown that the basal ganglia (BG) and dorsolateral prefrontal cortex (DLPFC) subserve DRT performance. Here, we report the results of simulation studies of a systems-level model of DRT performance. The model was trained using the temporal difference (TD) algorithm and uses an actor-critic architecture. The matrisomes of the BG represent the actor and the striosomes represent the critic. Unlike existing models, we hypothesize that the BG subserve the selection of both motor- and cognitive-related information in these tasks. We also assume that the learning of both processes is based on reward presentation. A novel feature of the model is the incorporation of delay-active neurons in the matrisomes, in addition to DLPFC. Another novel feature of the model is the subdivision of the matrisomal neurons into segregated winner-take-all (WTA) networks consisting of delay- versus transiently-active units. Our simulation model proposes a new neural mechanism to account for the occurrence of perseverative responses in WM tasks in striatal-, as well as in prefrontal damaged subjects. Simulation results also show that the model both accounts for the phenomenon of time shifting of dopamine phasic signals and the effects of partial reinforcement and reward magnitude on WM performance at both behavioral and neural levels. Our simulation results also found that the TD algorithm can subserve learning in delayed-reversal tasks.

Hemodynamic responses in human multisensory and auditory association cortex to purely visual stimulation

BackgroundRecent findings of a tight coupling between visual and auditory association cortices during multisensory perception in monkeys and humans raise the question whether consistent paired presentation of simple visual and auditory stimuli prompts conditioned responses in unimodal auditory regions or multimodal association cortex once visual stimuli are presented in isolation in a post-conditioning run. To address this issue fifteen healthy participants partook in a "silent" sparse temporal event-related fMRI study. In the first (visual control) habituation phase they were presented with briefly red flashing visual stimuli. In the second (auditory control) habituation phase they heard brief telephone ringing. In the third (conditioning) phase we coincidently presented the visual stimulus (CS) paired with the auditory stimulus (UCS). In the fourth phase participants either viewed flashes paired with the auditory stimulus (maintenance, CS-) or viewed the visual stimulus in isolation (extinction, CS+) according to a 5:10 partial reinforcement schedule. The participants had no other task than attending to the stimuli and indicating the end of each trial by pressing a button.ResultsDuring unpaired visual presentations (preceding and following the paired presentation) we observed significant brain responses beyond primary visual cortex in the bilateral posterior auditory association cortex (planum temporale, planum parietale) and in the right superior temporal sulcus whereas the primary auditory regions were not involved. By contrast, the activity in auditory core regions was markedly larger when participants were presented with auditory stimuli.ConclusionThese results demonstrate involvement of multisensory and auditory association areas in perception of unimodal visual stimulation which may reflect the instantaneous forming of multisensory associations and cannot be attributed to sensation of an auditory event. More importantly, we are able to show that brain responses in multisensory cortices do not necessarily emerge from associative learning but even occur spontaneously to simple visual stimulation.

Perseveration of the Partial Reinforcement Effect in Extinction with Rats over Two Phases of Extinction and Two Stages of Continuous Reinforcement

One group of 10 male albino rats was given partial reinforcement while the other 10 rats received continuous reinforcement in a straight alley. Subjects then experienced five consecutive stages of Extinction 1, Continuous Reinforcement 1, Extinction 2, Continuous Reinforcement 2, and finally, Extinction 3. Analysis showed the partial reinforcement effect in extinction was sustained over two stages of extinction and two stages of continuous reinforcement, since subjects receiving partial reinforcement ran faster than rats given continuous reinforcement throughout all three of the extinction periods. The results seem to support those of Amsel's (1967) and Cabpaldi's (1967) theoretical formulations of the partial reinforcement effect in extinction.

The Speed of Learning in Noisy Games: Partial Reinforcement and the Sustainability of Cooperation

In an experiment, players' ability to learn to cooperate in the repeated prisoner's dilemma was substantially diminished when the payoffs were noisy, even though players could monitor one another's past actions perfectly. In contrast, in one-time play against a succession of opponents, noisy payoffs increased cooperation, by slowing the rate at which cooperation decays. These observations are consistent with the robust observation from the psychology literature that partial reinforcement (adding randomness to the link between an action and its consequences while holding expected payoffs constant) slows learning. This effect is magnified in the repeated game: when others are slow to learn to cooperate, the benefits of cooperation are reduced, which further hampers cooperation. These results show that a small change in the payoff environment, which changes the speed of individual learning, can have a large effect on collective behavior. And they show that there may be interesting comparative dynamics that can be derived from careful attention to the fact that at least some economic behavior is learned from experience.

The Speed of Learning in Noisy Games: Partial Reinforcement and the Sustainability of Cooperation

In an experiment, players’ ability to learn to cooperate in the repeated prisoner’s dilemma was substantially diminished when the payoffs were noisy, even though players could monitor one another's past actions perfectly. In contrast, in one-time play against a succession of opponents, noisy payoffs increased cooperation, by slowing the rate at which cooperation decays. These observations are consistent with the robust observation from the psychology literature that partial reinforcement (adding randomness to the link between an action and its consequences while holding expected payoffs constant) slows learning. This effect is magnified in the repeated game: When others are slow to learn to cooperate, the benefits of cooperation are reduced, which further hampers cooperation. These results show that a small change in the payoff environment, which changes the speed of individual learning, can have a large effect on collective behavior. And they show that there may be interesting comparative dynamics that can be derived from careful attention to the fact that at least some economic behavior is learned from experience.

Grading odor similarities in a Go/No-Go task

Recent studies show that some features of odor perception are predicted by olfactory receptor biophysics and olfactory bulb physiology. Those studies used a behavioral assay in which rodents dig in a dish of scented cage bedding after pretraining to associate a buried reward with an odorant. The advantage of the digging task is an intensity measure of similarity (number of seconds spent digging). The method has the disadvantages of odorant contamination and low control over concentration and timing, making it difficult to use in electrophysiology. We describe an operant task that avoids these disadvantages and provides a reliable intensity-based similarity measure. Odorants can be delivered with a standard air dilution olfactometer, and rats learn to lever press to one odorant and avoid pressing to another in a Go/No-Go (CS+/CS−) task with a partial reinforcement protocol. Generalization tests substitute a portion of the unrewarded CS+ trials with test odorants. The number of generalization trials on which a subject responds to a test odorant is the measure of odor similarity intensity. We present validation tests using mixture component recognition, which show high repeatability, little variability across subjects and no decrease in responding across sessions. The results match those obtained with the digging task in four of five mixtures tested. This method allows optimal control over stimulus parameters and is compatible with simultaneous electrophysiological recording.

Determinants of instrumental extinction in terrestrial toads ( Bufo arenarum)

Previous research in a water-reinforced instrumental training situation with toads ( Bufo arenarum) has shown that performance in both acquisition and extinction is poorer after partial, rather than continuous reinforcement training. In Experiment 1, the performance of a group receiving 24 trials on a 50% partial reinforcement schedule was poorer in acquisition and extinction than that of continuously reinforced groups matched for trials or reinforcements. However, partially reinforced toads extinguished at the same rapid rate as a continuously reinforced group that received training only on the days in which the partial toads received water reinforcement. In Experiment 2, extinction was faster after 10 reinforced acquisition trials than after 30 trials. This evidence suggests that the deleterious effects of partial reinforcement in toads can be explained by a combination of two factors, namely, the distribution of reinforced trials across days and the total number of reinforcements.

An analysis of sequential variables in Pavlovian conditioning employing extended and limited acquisition training

Sequential theory’s memory model of learning has been successfully applied in response contingent instrumental conditioning experiments ( Capaldi, 1966, 1967, 1994; Capaldi & Miller, 2003). However, it has not been systematically tested in nonresponse contingent Pavlovian conditioning experiments. The present experiments attempted to determine if several sequential variables affect responding in Pavlovian situations as they do in instrumental ones. Of primary concern here were the effects on extinction of number of NR transitions (the number of times a nonreinforced trial is followed by a reinforced trial), N-length (the number of successive nonreinforced trials that precede a reinforced trial), and percentage of reinforcement (50 versus 100%) following either extended acquisition training (Experiment 1, 720 trials) or limited acquisition training (Experiment 3, 24 trials). In agreement with a sequential analysis, N-length increased resistance to extinction more than number of NR transitions following extensive training with the opposite occurring following limited training. In Experiment 1, greater resistance to extinction was associated with 50% than with 100% reinforcement, a partial reinforcement extinction effect (PREE). Experiment 2 examined an anomalous finding obtained in Experiment 1. A major theoretical difference between instrumental and Pavlovian conditioning has been held to be the greater ease of producing a PREE in instrumental than in Pavlovian conditioning ( Kimble, 1961; Mackintosh, 1974). However, the findings obtained here suggest that the probability of obtaining a PREE and other Pavlovian extinction effects, as in instrumental conditioning, increases along with the effectiveness of the sequential variables employed.

Effects of partial reinforcement and time between reinforced trials on terminal response rate in pigeon autoshaping

Partial reinforcement often leads to asymptotically higher rates of responding and number of trials with a response than does continuous reinforcement in pigeon autoshaping. However, comparisons typically involve a partial reinforcement schedule that differs from the continuous reinforcement schedule in both time between reinforced trials and probability of reinforcement. Two experiments examined the relative contributions of these two manipulations to asymptotic response rate. Results suggest that the greater responding previously seen with partial reinforcement is primarily due to differential probability of reinforcement and not differential time between reinforced trials. Further, once established, differences in responding are resistant to a change in stimulus and contingency. Secondary response theories of autoshaped responding (theories that posit additional response-augmenting or response-attenuating mechanisms specific to partial or continuous reinforcement) cannot fully accommodate the current body of data. It is suggested that researchers who study pigeon autoshaping train animals on a common task prior to training them under different conditions.

Partial reinforcement and resistance to extinction in newts

Newts were trained under a massed-trial condition and then extinguished in a straight alleyway to assess the partial reinforcement extinction effect (PREE). There were four groups: one partial reinforcement group trained using a random schedule (RA), two partial groups for which reinforcement ratios were gradually increased (GI) or decreased (GD), and a continuous reinforcement group (CR). They showed significant acquisition and extinction effects. The CR group was the most resistant to extinction, with no significant difference among the other three. The results indicate a reverse PREE in newts, and suggest that their learning is controlled mainly by a simple Strengthening-Weakening mechanism.

The effects of acquisition training schedule on extinction and reinstatement of cocaine self-administration in male rats.

Partial reinforcement is known to increase resistance to extinction (Rn) relative to training with continuous reinforcement. This phenomenon, referred to as the partial reinforcement extinction effect, is one of the most robust in learning and conditioning studies. Experiment 1 investigated manipulations known to affect the partial reinforcement extinction effect and determined their possible relevance for drug use patterns. Male rats received intravenous cocaine self-administration training under partial reinforcement (FR-10) training or continuous reinforcement (FR-1) conditions with either a low (0.25 mg/kg infusion) or a high cocaine dose (1.00 mg/kg infusion). Animals were placed on an extinction (recurrent nonreward) schedule for 10 days (1-hr sessions) prior to being tested for cue-induced reinstatement (single 2-hr session). Experiment 2 involved acquisition of cocaine self-administration under FR-1 conditions of short training (15 days) or extended training (30 days) with a low dose (0.25 mg/kg infusion) or a medium dose (0.50 mg/kg infusion) of cocaine reward prior to extinction or reinstatement. Experiment 1 showed that rats trained with FR-10-high dose outcomes exhibited greater Rn than the remaining groups. Additionally, FR-10-high dose and FR-10-low dose rats were more likely to return to active drug seeking during the reinstatement test. In Experiment 2, rats trained under FR-1-medium dose conditions were more persistent during extinction following short acquisition training than comparable rats experiencing extended acquisition training. The reinstatement test was conducted following extinction, in which it was observed that overtraining under FR-1-medium dose reward schedules resulted in a decrease in the tendency to return to active drug seeking.

When discrimination fails (or at least falters).

Pigeons learned 2 concurrent simultaneous discriminations in which the pecking of left keys versus right keys was reinforced depending on a color that appeared on one of the keys. When the reinforced choice was to peck directly at the color, accuracy was very high initially, but dropped noticeably with continued training. Partial reinforcement of these choices exacerbated the dropoff, sometimes causing accuracies to fall close to zero. By contrast, when the reinforced choice was to peck the alternate-key stimulus, accuracy was very low, initially, but remained high and stable following acquisition. Lowering the reinforcement probability, even to zero, for the latter choices had little effect on their accuracy but yielded increased accuracy on color-choice trials. These results resemble the ambiguous-cue effect and suggest the process of value transfer.

The influence of trait anxiety on autonomic response and cognitive performance during an anticipatory anxiety task

The interaction between emotion and cognition is thought to be intimately involved in the development and maintenance of anxiety disorders. In a set of studies, we investigated whether trait anxiety modulates cognitive performance and autonomic activity during an anticipatory anxiety task. Participants completed a letter-size decision-making task with two alternating 28-32 s background screen color-blocks. One of the colors was associated with the presentation of an aversive noise [unconditioned stimulus (UCS)]. Participants were aware of the background color that would (CTX+) and would not (CTX-) be paired with the UCS but did not know when or how often the UCS would be presented. Two experiments were conducted. In Experiment 1, the UCS was presented during the decision-making task in the CTX+ color-blocks using a partial reinforcement schedule. Different noises were presented each time to increase unpredictability and prevent habituation. In Experiment 2, the UCS was never presented during the decision-making task. Results suggested that only the paradigm used in Experiment 1 was successful in eliciting anticipatory anxiety. In Experiment 1, continuously measured skin conductance response (SCR) data suggested that anxiety was significantly greater during CTX+ compared to CTX- trials; no SCR differences were found between high and low trait-anxious participants. Results further indicated that high trait-anxious participants responded significantly faster on the decision-making task during CTX+ compared to CTX- trials, whereas low trait-anxious participants displayed the opposite pattern. Our results reveal an interesting dissociation between the effects of individual differences in trait anxiety on autonomic activity and cognitive performance during an anticipatory anxiety task.

Trial order and retention interval in human predictive judgment

The influences of order of trial type and retention interval on human predictive judgments were assessed for a cue that was reinforced on half of its training presentations. Subjects observed 10 cue-outcome presentations (i.e., reinforced trials) and 10 cue-alone presentations (i.e., nonreinforced trials) in one of three different orders: all nonreinforced trials followed by all reinforced trials(latent inhibition), reinforced and nonreinforced trials interspersed (partial reinforcement), or al lreinforced trials followed by all nonreinforced trials (extinction). Ratings were based mainly on the most recent event type (i.e., a recency effect) when the test occurred immediately after training but were based mainly on initial event types (i.e., a primacy effect) when the test occurred after a 48-h delay. The subjects tested both immediately and with a long retention interval did not exhibit this shift to primacy (i.e., the recency effect persisted). These results demonstrate noncatastrophic forgetting and the flexible use of trial order information in predictive judgments.