Midsession Reversal Task Research Articles

Modeling within-session dynamics of categorical and item-memory mechanisms in pigeons.

Past studies have shown that pigeons can learn complex categories and can also remember large numbers of individual objects. In recent work, Cook et al. Psychonomic Bulletin & Review, 28, 548-555, (2021) provided evidence that pigeons may use a dynamic combination of both category-based information and item-specific memorization to solve a categorical variation of the mid-session reversal (MSR) task, which is an influential task for exploring the nature of temporally organized behaviors in animals. To provide greater insight into these pigeons' behaviors, in this article we developed and investigated different computational models and their variations to account for these data. Of these, two models emerged as good candidates. One was a multinomial-processing-tree categorization/memory model, formalizing the two-process mechanism initially proposed by Cook et al. Psychonomic Bulletin & Review, 28, 548-555, (2021). The second was a new object/time-coding model, which posits the storage of object-specific memories with an additional within-session time code and assumes that a basic stimulus generalization process underlies the pigeons' choice behavior. Both provided high-quality fits to the published sets of training and transfer data collected in the categorical MSR task. These computational efforts give deeper insights into the theoretical mechanisms underlying the temporal and sequential structure of behavior in animals and stimulate future empirical research further revealing the organization of the pigeons' cognitive processes.

Giving time a chance in the midsession reversal task.

The midsession reversal task involves a simultaneous discrimination between stimuli S1 and S2. Choice of S1 but not S2 is reinforced during the first 40 trials, and choice of S2 but not S1 is reinforced during the last 40 trials. Trials are separated by a constant intertrial interval (ITI). Pigeons learn the task seemingly by timing the moment of the reversal trial. Hence, most of their errors occur around trial 40 (S2 choices before trial 41 and S1 choices after trial 40). It has been found that when the ITI is doubled on a test session, the reversal trial is halved, a result consistent with timing. However, inconsistent with timing, halving the ITI on a test session did not double the reversal trial. The asymmetry of ITI effects could be due to the intrusion of novel cues during testing, cues that preempt the timing cue. To test this hypothesis, we ran two types of tests after the regular training in the midsession reversal task, one with S1 and S2 choices always reinforced, and another with S1 always reinforced but S2 reinforced only after 20 trials when the ITI doubled or 40 trials when the ITI halved. For most pigeons, performance was consistent with timing both when the ITI doubled and when it was halved, but some pigeons appeared to follow strategies based on counting or on reinforcement contingencies.

Midsession reversal task with starlings: A quantitative test of the timing hypothesis

In the Mid-Session Reversal task (MSR), an animal chooses between two options, S1 and S2. Rewards follow S1 but not S2 from trials 1–40, and S2 but not S1 from trials 41–80. With pigeons, the psychometric function relating S1 choice proportion to trial number starts close to 1 and ends close to 0, with indifference (PSE) close to trial 40. Surprisingly, pigeons make anticipatory errors, choosing S2 before trial 41, and perseverative errors, choosing S1 after trial 40. These errors suggest that they use time into the session as the preference reversal cue. We tested this timing hypothesis with 10 Spotless starlings. After learning the MSR task with a T-s Inter-Trial Interval (ITI), they were exposed to either 2 T or T/2 ITIs during testing. Doubling the ITI should shift the psychometric function to the left and halve its PSE, whereas halving the ITI should shift the function to the right and double its PSE. When the starlings received one pellet per reward, the ITI manipulation was effective: The psychometric functions shifted in the direction and by the amount predicted by the timing hypothesis. However, non-temporal cues also influenced choice.

Constantly timing, but not always controlled by time: Evidence from the midsession reversal task.

We used a midsession reversal task to investigate how temporal and situational cues may combine to determine choice in frequently changing environments. Pigeons learned a simultaneous discrimination with 2 stimuli: S1 and S2. Choices of S1 were reinforced only during the first trials, and choices of S2 were reinforced only during the last trials of the session, that is, the reinforcement contingencies reversed once during the session. To weaken the temporal cue (time into the session) that signaled the reversal trial, we varied the location of reversal trial randomly across sessions; to weaken the situational cue (the outcome of the previous trials that might support a win-stay/loose shift strategy), we varied the payoff probabilities associated with S1 and S2. Performance was consistent neither with the exclusive use of a timing strategy, nor with the exclusive use of a situational, win-stay/lose-shift strategy. Instead, choice seemed to be under joint control of both cues. The relative influence of these cues was dynamic: When payoff was higher for S1 than S2, behavior was less time-controlled than when the payoff was higher for S2 that S1, or when they were equal. We advance a descriptive mixture model of joint control for the midsession reversal task. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Step changes in the intertrial interval in the midsession reversal task: Predicting pigeons' performance with the learning-to-time model.

Our goal was to assess the role of timing in pigeons' performance in the midsession reversal task. In discrete-trial sessions, pigeons learned to discriminate between 2 stimuli, S1 and S2. Choices of S1 were reinforced only in the first half of the session and choices of S2 were reinforced only in the second half. Typically, pigeons choose S2 before the contingency reverses (anticipatory errors) and S1 after (perseverative errors), suggesting that they time the interval from the beginning of the session to the contingency reversal. To test this hypothesis, we exposed pigeons to a midsession reversal task and, depending on the group, either increased or decreased the ITI duration. We then contrasted the pigeons' performance with the predictions of the Learning-to-Time (LeT) model: In both conditions, preference was expected to reverse at the same time as in the previous sessions. When the ITI was doubled, pigeons' preference reversal occurred at half the trial number but at the same time as in the previous sessions. When the ITI was halved, pigeons' preference reversal occurred at a later trial but at an earlier time than in the previous sessions. Hence, pigeons' performance was only partially consistent with the predictions of LeT, suggesting that besides timing, other sources of control, such as the outcome of previous trials, seem to influence choice.

Pigeons' midsession reversal: Greater magnitude of reinforcement on the first half of the session leads to improved accuracy.

In the midsession reversal task, pigeons are trained on a simultaneous two-alternative discrimination in which S1 is correct for the first half of the session and S2 is correct for the second half of the session. Optimally, pigeons should choose S1 until it stops being correct and choose S2 afterward. Instead, pigeons anticipate S2 too early and continue choosing S1 even after the reversal. Research suggests that they attempt to time the reversal rather than use the feedback from the preceding response(s). Recently, there is evidence that performance is almost optimized by generating an asymmetry between S1 and S2. For example, pigeons' accuracy improves if correct S1 responses are reinforced 100% of the time, but correct S2 responses are reinforced only 20% of the time. Similarly, accuracy improves if S1 requires one peck but S2 requires 10 pecks. Accuracy does not improve, however, if the value of S1 is less than the value of S2. In the current experiment, we manipulated the magnitude of reinforcement. For the experimental group, correct responses to S1 were reinforced with five pellets of food and correct responses to S2 were reinforced with one pellet. For the control group, all correct responses were reinforced with three pellets. Consistent with the earlier findings, results indicated that there was a significant reduction in anticipatory errors in the experimental group compared with the control, and there was no significant increase in perseverative errors.

The Midsession Reversal Task with Pigeons Does a Brief Delay Between Choice and Reinforcement Facilitate Reversal Learning?

In a midsession reversal task, the session begins with a simple simultaneous discrimination in which one stimulus (S1) is correct and the other stimulus (S2) is incorrect (S1+/S2-). At the midpoint of the session, the discrimination reverses and S2 becomes the correct choice (S2+/S1-). When choosing optimally, a pigeon should choose S1 until the first trial in which its choice is not reinforced and then it should shift to S2 (win-stay/lose-shift). With this task, pigeons have been shown to respond suboptimally by anticipating the reversal (making anticipatory errors) and continuing to choose S1 after the reversal (making perseverative errors). This suboptimal behavior may result from a pigeon’s relative impulsivity due to the immediacy of reinforcement following choice. In other choice tasks, there is evidence that the introduction of a short delay between choice and reinforcement may decrease pigeons’ impulsivity. In the present experiment, a delay was introduced between stimulus selection and reinforcement to assess whether it results in a decrease in anticipatory and perseverative errors. Pigeons that had a delay between choice and reinforcement were a bit slower in acquiring the midsession reversal task compared to those without a delay, but showed no decrease in either anticipatory or perseverative errors. It is likely that the pigeons’ natural tendency to use time from the start of the session to the reversal as a cue to reverse prevented the delay from increasing accuracy on this task.

The midsession reversal task: A theoretical analysis.

In the midsession reversal task, choice of one stimulus (S1) is correct for the first half of each session and choice of the other stimulus (S2) is correct for the last half of each session. Although humans and rats develop very close to what has been called a win-stay/lose-shift response strategy, pigeons do not. Pigeons start choosing S2 before the reversal, making anticipatory errors, and they keep choosing S1 after the reversal, making perseverative errors. Research suggests that the pigeons are timing the reversal from the start of the session. However, making the reversal unpredictable does not discourage the pigeons from timing. Curiously, pigeons' accuracy improves if one decreases the value of the S2 stimulus relative to the S1 stimulus. Another form of asymmetry between S1 and S2 can be found by varying, over trials, the number of S1 or S2 stimuli. Counterintuitively, if the number of S2 stimuli varies, it results in a large increase in anticipatory errors but little increase in perseverative errors. However, if the number of S1 stimuli varies over trials, it results in a large increase in perseverative errors but no increase in anticipatory errors. These last two effects suggest that in the original midsession reversal task, the pigeons are learning to reject S2 during the first half of each session and learning to reject S1 during the last half of each session. These results suggest that reject learning may also play an important role in the learning of simple simultaneous discriminations.

Midsession reversal learning: Pigeons learn what stimulus to avoid.

The midsession reversal task involves a simple simultaneous discrimination in which, each session, choice of 1 stimulus (S1) is correct for the first 40 trials of each session, and choice of the other stimulus (S2) is correct for the remaining 40 trials. After considerable training with this task, pigeons typically continue to choose S2 too early (making anticipatory errors) and continue choosing S1 for following the reversal (making perseverative errors). Errors can be reduced, however, by decreasing the probability of reinforcement for correct S2 choices or by increasing the response requirement for S2 choices. Increasing the number of S2 stimuli (over trials, 1 S2 stimulus on each trial), however, does not reduce errors. Instead, it results in an increase in anticipatory errors but no change in perseverative errors. In the present experiment, we increased the number of S1 stimuli (over trials, 1 S1 stimulus on each trial) and found an increase in the number of perseverative errors but no change in anticipatory errors. The results suggest that the pigeons acquire this task by learning which stimuli to avoid, rather than which stimuli to choose, although it is also possible that these effects result from attention drawn to the variable stimuli when they are incorrect. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Past outcomes and time flexibly exert joint control over midsession reversal performance in the rat

In a midsession reversal task, subjects choose between two stimuli on every trial; only responses to one stimulus are reinforced. Halfway throughout the session, contingencies are reversed: previously reinforced responses are now extinguished and vice versa. Both, the outcome of the previous trial and the time elapsed since the beginning of the session, may predict the availability of reinforcement and determine choice. Thus, this task has typically been used to study cognitive flexibility and the temporal organization of behavior. This study assessed how past outcomes and time interact for behavioral control when each cue predicts the availability of reinforcement to a different extent. Eight rats were trained in four variations of the midsession reversal task differing in the reliability of outcomes and time as predictors of the reinforced response. We manipulated the reliability of the outcomes by providing either continuous or partial reinforcement, and the reliability of time by fixing the moment of reversal (middle of the session) or making the reversal unpredictable (semi-random trial). Results suggest that behavioral control alternates between outcomes and time according to the relative reliability of each cue. Model simulations show that outcomes and time may jointly determine behavior, and that momentary reinforcement rate may determine their relative influence.

Less information results in better midsession reversal accuracy by pigeons.

The midsession reversal task involves a simultaneous discrimination between Stimulus 1 (S1) and Stimulus 2 (S2) in which, for the first half of each session, choice of S1 is reinforced and S2 is not, and for the last half of each session, choice of S2 is reinforced and S1 is not. With this task, even after considerable training, pigeons tend to make anticipatory errors as they approach the reversal and they continue to make perseverative errors following the reversal. In the present research, we tested the hypothesis that reversal accuracy would improve by devaluing choice of S2 relative to S1. In Experiment 1, correct choice of S1 was reinforced 100% of the time, whereas correct choice of S2 was reinforced only 20% of the time. This manipulation reduced anticipatory errors but did not increase perseverative errors. In Experiment 2, choice of S1 required a single peck, whereas choice of S2 was devalued by requiring 10 pecks. A similar result was found. In Experiment 3 we devalued S1 by requiring 10 pecks and found decreased accuracy in the form of increased anticipatory errors. Paradoxically, in Experiments 1 and 2, by encouraging the pigeons to avoid using the feedback from choice of S2, and rely solely on feedback from choice of S1, discrimination reversal errors were reduced. The results have implications for attentional theories of learning and theories of behavior change. They also have implications for the conditions responsible for pigeons' tendency to time the occurrence of the change in reinforcement contingencies. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Midsession reversal learning by pigeons: Effect on accuracy of increasing the number of stimuli associated with one of the alternatives.

The midsession reversal task involves a simultaneous discrimination in which choice of one stimulus (S1) is correct for the first 40 trials and choice of the other stimulus (S2) is correct for the last 40 trials of each 80-trial session. When pigeons are trained on the midsession reversal task, they appear to use the passage of time from the start of the session as a cue to reverse. As the reversal approaches, they begin to make anticipatory errors, choosing S2 early, and following the reversal they make perseverative errors, continuing to choose S1. Recent research suggests that anticipatory errors can be reduced (while not increasing perseverative errors) by reducing the probability of reinforcement for correct S2 choices from 100% to 20%. A similar effect can be found by increasing the response requirement for choice of S2 from one peck to ten pecks. In the present experiments, we asked if a similar effect could be attained by increasing the number of stimuli that, over trials, could serve as S2. Instead, in both experiments, we found that increasing the number of S2 stimuli actually increased the number of anticipatory errors. Several interpretations of this result are provided, including the possibility that attention to the variable S2 stimuli may have interfered with attention to the S1 stimulus.

EFFECTS OF PERCENT REINFORCEMENT ON PIGEON'S ACCURACY IN MIDSESSION REVERSAL TASKS [University of Kentucky

This experiment tests the effects of a reducing the value of one stimulus compared to another. We hypothesize that in a midsession reversal task, accuracy would improve by devaluing choice of S2 relative to S1. A midsession reversal task requires a simultaneous discrimination between 2 stimuli (S1, S2), during which, halfway through the session the reinforcing stimulus is reversed. A current theory, win-stay/lose-shift, is a strategy for optimal choices in this task. In this strategy, an animal would continue to repeat choice of a correct stimulus, but once the stimulus becomes incorrect, the animal should switch to the other stimulus. In this experiment, ten unsexed White Carneau pigeons were used as subjects. The experiment took place in an operant test chamber. Inside, three horizontally aligned circular response keys are attached to a response panel. The keys are lit by mounted projectors showing red and green hues. Reinforcement consisted of 1.5 seconds of access to mixed-grain from a feeder. In the experimental group, all correct responses to S1 were reinforced but only a random 20% of the correct responses to S2 were reinforced. Our findings suggest that paradoxically, the reduction in the probability of reinforcement for correct S2 responses from 100% to 20% had a net positive effect on correct responses. There was a significant reduction in errors prior to the feedback from the reversal as well as no increase in errors following the feedback from the reversal. Analysis of the trials on either side of the reversal made it clear that there was a net benefit in percentage correct due to the decrease in the probability of reinforcement associated with correct responses to S2. This study suggests the negative impacts of too much reinforcement on subjects performance in a task.

The effect of reinforcement probability on time discrimination in the midsession reversal task.

We examined how biasing time perception affects choice in a midsession reversal task. Given a simultaneous discrimination between stimuli S1 and S2, with choices of S1 reinforced during the first, but not the second half of the trials, and choices of S2 reinforced during the second, but not the first half of the trials, pigeons show anticipation errors (premature choices of S2) and perseveration errors (belated choices of S1). This suggests that choice depends on timing processes, on predicting when the contingency reverses based on session duration. We exposed 7 pigeons to a midsession reversal task and manipulated the reinforcement rate on each half of the session. Compared to equal reinforcement rates on both halves of the session, when the reinforcement rate on the first half was lower than on the second half, performance showed more anticipation and less perseveration errors, and when the reinforcement rate on the first half was higher than on the second half, performance showed a remarkable reduction of both types of errors. These results suggest that choice depends on both time into the session and the outcome of previous trials. They also challenge current models of timing to integrate local effects.

Anticipation of a midsession reversal in humans

In a two-stimulus visual discrimination choice task with a reversal in reward contingencies midway through each session, pigeons produce a surprising number of anticipatory errors (i.e., responding to the second-correct stimulus before the reversal) based on failure to inhibit timing-based intrusion errors; limited prior research has suggested humans’ performance is qualitatively different. Here we illustrate a partial replication of previous findings in humans, but suggest based on our results that humans process these tasks in a manner similar to pigeons. Humans made relatively few but consistent errors across both simultaneous- and successive-choice experiments. Anticipation errors were limited when the identity of the first-correct stimulus alternated between sessions, consistent with the behaviour of pigeons. Subsequent experiments found evidence for anticipation on a purely temporal simultaneous choice task, and fewer errors with symmetrical reinforcement and punishment of responses on a sequential choice task. Interval timing causes conflicts with decision-making processes on the midsession reversal task that are consistent, but differ in magnitude, across species.

Midsession reversal task with pigeons: Parallel processing of alternatives explains choices.

Most models of choice assume a "tug of war" (ToW) between options present at the time of the choice, arguing that preferences are built on this process, and implying that adding options increases delay to act. In contrast, the sequential choice model (SCM) proposes that choices are driven by parallel expression of the mechanisms that control action in sequential encounters, without comparative deliberation at choice time. Only the SCM predicts choice preferences based on latencies to respond in single-option encounters. SCM further predicts that latencies to choose should either be the same or shorter than those in sequential encounters. We contrasted these models using a midsession reversal task with pigeons. Responses to one alternative (S1) were rewarded in the first half of each session and those to the other (S2) in the second half. Single-option (sequential) and two-option (choice) trials were intermingled. In choice trials subjects strongly preferred S1 early in the session, showed intermediate preferences toward the midsession, and preferred S2 late. These preferences were all predicted by changes in latency toward the presently negative alternative (S2 early and S1 late) in single-option trials. Latency toward presently positive stimuli were minimal throughout, in both single and two option trials, with no evidence of an evaluation time cost of choice. The ability to predict choice preference from latencies in sequential encounters and the absence of a choice delay support the SCM against ToW models, consistently with results from other protocols and species. (PsycINFO Database Record

Cognitive flexibility and dual processing in pigeons: Temporal and contextual control of midsession reversal.

Evidence is reported showing that pigeons flexibly use temporal and contextual cues to maximize reward obtained in a midsession reversal task. Pigeons were trained to choose between red and green sidekeys for 60 trials in a session, with choice of one color correct on Trials 1-30 and choice of the other color correct on Trials 31-60 (midsession reversal). Pigeons showed anticipatory errors before reversal and perseverative errors after reversal, and manipulations of the length of the intertrial interval and the point of reversal suggested that pigeons used an internal timer to track the point of reversal. When houselight context cues that signaled the correct choice were presented throughout trials in Experiment 1, choice behavior rapidly came under context control, leading pigeons to rarely make errors and to show no effect of intertrial interval or point of reversal. In Experiments 2 and 3, switches between context and no-context cues occurred among trials. These manipulations revealed that pigeons can readily switch between context control and temporal control of behavior. The internal timer continued to run throughout context trials and could readily be accessed to control choice behavior when a context cue was removed. (PsycINFO Database Record

Spatial midsession reversal learning in rats: Effects of egocentric Cue use and memory

The midsession reversal task has been used to investigate behavioral flexibility and cue use in non-human animals, with results indicating differences in the degree of control by environmental cues across species. For example, time-based control has been found in rats only when tested in a T-maze apparatus and under specific conditions in which position and orientation (i.e., egocentric) cues during the intertrial interval could not be used to aid performance. Other research in an operant setting has shown that rats often produce minimal errors around the reversal location, demonstrating response patterns similar to patterns exhibited by humans and primates in this task. The current study aimed to reduce, but not eliminate, the ability for rats to utilize egocentric cues by placing the response levers on the opposite wall of the chamber in relation to the pellet dispenser. Results showed that rats made minimal errors prior to the reversal, suggesting time-based cues were not controlling responses, and that they switched to the second correct stimulus within a few trials after the reversal event. Video recordings also revealed highly structured patterns of behavior by the majority of rats, which often differed depending on which response was reinforced. We interpret these findings as evidence that rats are adept at utilizing their own egocentric cues and that these cues, along with memory for the recent response-reinforcement contingencies, aid in maximizing reinforcement over the session.

Supplemental Material for Midsession Reversal Task With Pigeons: Parallel Processing of Alternatives Explains Choices

Supplemental Material for Midsession Reversal Task With Pigeons: Parallel Processing of Alternatives Explains Choices

Within-session reversal learning in rhesus macaques (Macaca mulatta).

In a midsession reversal (MSR) task, animals are typically presented with a simple, simultaneous discrimination (S1+, S2-) where contingencies are reversed (S1-, S2+) half-way through each session. This paradigm creates multiple, relevant cues that can aid in maximizing overall reinforcement. Recent research has shown that pigeons show systematic anticipatory and perseverative errors across the session, which increase as a function of proximity to the reversal trial. This behavior has been theorized to indicate primary control by temporal cues across the session, instead of the cues provided by recent reinforcement history that appear to control behavior shown by humans. Rats, however, appear to be guided by recent reinforcement history when tested in an operant context, thereby demonstrating behavior that parallels that seen in humans, but they appear to be guided by temporal cues when tested in an open-field apparatus, showing behavior more akin to that seen in pigeons. We tested rhesus macaques (Macaca mulatta) on the MSR with a computerized simultaneous visual discrimination to assess whether they would show errors indicative of control by time or by recent reinforcement history. When a single reversal point occurred midsession, rhesus macaques showed no anticipation of the reversal and a similar level of perseveration to rats tested in an operant setting. Nearly identical results also were observed when the monkeys were trained with a single, variable reversal point or with multiple, variable reversal points within a session. These results indicate that temporal cues are not guiding response flexibility in rhesus macaque visual discrimination.