Overall Probability Of Reinforcement Research Articles

The role of contingency discriminability in suboptimal choice

Suboptimal choice consists of a preference for an alternative with a lower probability of reinforcement (suboptimal alternative) over another with a higher probability of reinforcement (optimal alternative) when the former has discriminative stimuli that signal in which trials a reinforcer will be delivered and in which trials it will not. Discriminating the contingencies of reinforcement associated with the stimuli of the suboptimal alternative is necessary to produce suboptimal choice, but the impact of different degrees of discriminability has not been systematically studied. The discriminability of the contingencies of reinforcement depends on the difference in the probability of reinforcement of the two stimuli; higher differences yield higher discriminability. Pigeons were exposed to a procedure that presented a choice between two alternatives, each associated with two stimuli. The contingency discriminability of the suboptimal alternative was manipulated across conditions, while the contingency discriminability of the optimal alternative was absent in all conditions. The overall probability of reinforcement of each alternative remained the same throughout the experiment (p = .2 and p = .5 for the suboptimal and optimal alternatives, respectively). The preference for the suboptimal alternative increased as its discriminability increased. There was a positive correlation between discrimination index and preference for the suboptimal alternative. These results highlight the importance of contingency discriminability to generate suboptimal choice.

Testing the Δ-∑ hypothesis in the suboptimal choice task: Same delta with different probabilities of reinforcement.

In a concurrent-chain procedure, pigeons choose between 2 initial-link stimuli; one is followed by terminal link stimuli that signal reliably whether food will be delivered after a delay; the other is followed by terminal link stimuli that do not signal whether food will be delivered after the delay. Pigeons prefer the former alternative even when it yields a lower overall probability of food. Recently, we proposed the Delta-Sigma (∆-∑) hypothesis to explain the effect: Preference depends on the difference (∆) between the reinforcement probabilities associated with the terminal link stimuli, and the overall probability of reinforcement (∑) associated with the alternative. The hypothesis predicts that, for constant ∑, animals should prefer alternatives with greater ∆ values regardless of the specific probabilities of reinforcement that determine ∆. In 2 experiments, we tested this prediction by comparing a ∆ = .5 against a ∆ = 0 alternative, with the former obtained with different pairs of reinforcement probabilities across conditions. The results supported the hypothesis when the 2 probabilities defining ∆ were significantly greater than 0, but not when one of them was close to 0. The results challenge our theoretical accounts of suboptimal choice and the variables considered to determine pigeons' preference.

An Animal Model of Human Gambling

Human gambling generally involves taking a risk on a low probability high outcome alternative over the more economically optimal high probability low outcome alternative (not gambling). Surprisingly, although optimal foraging theory suggests that animals should be sensitive to the overall probability of reinforcement, the results of many experiments suggest otherwise. For example, they do not prefer an alternative that 100% of the time provides them with a stimulus that always predicts reinforcement over an alternative that provides them with a stimulus that predicts reinforcement 50% of the time. This line of research leads to the conclusion that preference depends on the predictive value of the stimulus that follows and surprisingly, not on its frequency. A similar mechanism likely accounts for the suboptimal choice that humans have to engage in commercial gambling.

Rats are optimal in a choice task in which pigeons are not

In an extensive list of studies, it has been found that pigeons prefer an alternative associated with discriminative stimuli over another associated with non-discriminative stimuli, even when the probability of reinforcement is higher in the latter. This behavior has been named “suboptimal choice”. In the present experiment, we evaluated whether rats, another widely studied species within the Experimental Analysis of Behavior, also shows this behavior. We systematically replicated the procedure employed with pigeons, and found that rats are not suboptimal, i.e., they prefer the non-discriminative alternative associated with .5 probability of reinforcement, over the discriminative alternative associated with .2 probability of reinforcement. This effect occurred even though rats discriminated the contingencies of reinforcement associated with each stimulus, suggesting that rats' optimal choice was driven by the overall probability of reinforcement of each alternative. Different procedural details are offered as possibilities for explaining this apparent inter-species difference.

Further investigation of the Monty Hall Dilemma in pigeons and rats

In the Monty Hall Dilemma (MHD), three doors are presented with a prize behind one and participants are instructed to choose a door. One of the unchosen doors not containing the prize is revealed, following which the participant can choose to stay with their chosen door or switch to the other one. The optimal strategy is to switch. Herbranson and Schroeder (2010) found that humans performed poorly on this task, whereas pigeons learned to switch readily. We found that pigeons performed only slightly better than humans and that pigeons stayed nearly exclusively when staying and switching were reinforced equally and when staying was the optimal strategy (Stagner et al., 2013b). In Experiment 1 of the present research, rats were trained under these same conditions to observe if possible differences in foraging strategy would influence performance on this task. In Experiment 2, pigeons were trained in an analogous procedure to better compare the two species. We found that both species were sensitive to the overall probability of reinforcement, as both switched significantly more often than subjects that were reinforced equally for staying and switching or reinforced more often for staying. Overall, the two species performed very similarly within the parameters of the current procedure.“This article is part of a Special Issue entitled: Tribute to Tom Zentall.”

"Pigeons prefer discriminative stimuli independently of the overall probability of reinforcement and of the number of presentations of the conditioned reinforcer": Correction to Stagner et al. (2012).

"Pigeons prefer discriminative stimuli independently of the overall probability of reinforcement and of the number of presentations of the conditioned reinforcer": Correction to Stagner et al. (2012).

Pigeons prefer discriminative stimuli independently of the overall probability of reinforcement and of the number of presentations of the conditioned reinforcer.

When pigeons are given a choice between two alternatives, one leading to a stimulus 20% of the time that always signals reinforcement (S+) or another stimulus 80% of the time that signals the absence of reinforcement (S-) and the other alternative leading to one of two stimuli each signaling reinforcement 50% of the time, the 20% reinforcement alternative is preferred although it provides only 40% as much reinforcement. In Phase 1 of the present experiment, we tested the hypothesis that pigeons compare the S+ associated with each alternative and ignore the S- by giving them a choice between two pairs of discriminative stimuli (20% S+, 80% S- and 50% S+, 50% S-). Reinforcement theory suggests that the alternative associated with more reinforcement should be preferred but the pigeons showed indifference. In Phase 2, the pigeons were divided into two groups. For one group, the discriminative function was removed from the 50% reinforcement alternative and a strong preference for the 20% reinforcement alternative was found. For the other group, the discriminative function was removed from both alternatives and a strong preference was found for the 50% reinforcement alternative. Thus, the indifference found in Phase 1 was not due to the absence of discriminability of the differential reinforcement associated with the two alternatives (20% vs. 50% reinforcement); rather, the indifference can be attributed to the pigeons' insensitivity to the differential frequency of the two S+ and two S- stimuli. The relevance to human gambling behavior is discussed.

Hungry pigeons make suboptimal choices, less hungry pigeons do not

Hungry animals will often choose suboptimally by being attracted to reliable signals for food that occur infrequently (they gamble) over less reliable signals for food that occur more often. That is, pigeons prefer an option that 50 % of the time provides them with a reliable signal for the appearance of food but 50 % of the time provides them with a reliable signal for the absence of food (overall 50 % reinforcement) over an alternative that always provides them with a signal for the appearance of food 75 % of the time (overall 75 % reinforcement). The pigeons appear to choose impulsively for the possibility of obtaining the reliable signal for reinforcement. There is evidence that greater hunger is associated with greater impulsivity. We tested the hypothesis that if the pigeons were less hungry, they would be less impulsive and, thus, would choose more optimally (i.e., on the basis of the overall probability of reinforcement). We found that hungry pigeons choose the 50 % reinforcement alternative suboptimally but less hungry pigeons prefer the more optimal 75 % reinforcement. Paradoxically, pigeons that needed the food more received less of it. These findings have implications for how level of motivation may also affect human suboptimal choice (e.g., purchase of lottery tickets and playing slot machines).

Matching-to-sample in pigeons: In the absence of sample memory, sample frequency is a better predictor of comparison choice than the probability of reinforcement for comparison choice

In matching-to-sample, comparison choice should be controlled by the identity of the sample and, when the sample is not available, by the overall probability of reinforcement associated with each of the comparisons. In the present research, pigeons were trained to match a frequent sample (appearing on 80% of the trials) to one comparison (C(fr)) and an infrequent sample (appearing on 20% of the trials) to the other (C(inf)), with the number of reinforcements associated with each sample equated. In Experiment 1, the task was identity matching; in Experiments 2 and 3, it was symbolic matching. We asked whether, when control of comparison choice by the sample was reduced (by inserting a delay between the sample and the comparisons), pigeons would choose comparisons on the basis of (1) the number of reinforcements per comparison (and thus show no comparison bias), (2) the comparison associated with the more frequent sample during training (and show a preference for C(fr)), or (3) the probability of reinforcement given a correct response (and show a preference for C(inf)), or (4) inhibition produced by nonreinforced choice of the more frequently correct comparison (and show a preference for C(inf)). Pigeons showed a significant tendency to choose C(fr). In Experiment 3, we showed that this bias did not result from the effects of intertrial facilitation or interference. Thus, it appears that when control of comparison choice by the sample is reduced, pigeons' choice is controlled not merely by the probability of reinforcement but also by overall sample frequency.

Differential Conditioning Based on a Difference in the Predictability of Reinforcement

A Pavlovian procedure containing a set of several alternative reliable trial stimuli and a set of several alternative unreliable trial stimuli was used to determine if reliability of outcome, per se, would differentially condition additional trial stimuli that either preceded or followed those original trial stimuli in a serial conditioning paradigm. The reinforcing effectiveness of the antecedent stimuli themselves was later tested in a higher-order conditioning design. It was found that different rates could be established to the different antecedent stimuli in the absence of a difference in the probability of reinforcement. Further, stimuli paired with those antecedent stimuli in the higher-order conditioning test also exhibited a difference in rate. It was concluded that a combination of a stimulus that is always followed by food and another that is never followed by food are, in net, a more effective reinforcer than a common stimulus that has the same overall probability of reinforcement but which is followed by both outcomes.

Observing Behavior in Pigeons: The Effect of Reinforcement Probability and Response Cost Using a Symmetrical Choice Procedure

Observing behavior allows an organism to observe discriminative stimuli without affecting the overall probability of reinforcement. Several explanations have been proposed for what underlies this behavior, including uncertainty reduction and various forms of conditioned reinforcement. In the present research, we attempted to assess the effect of probability of reinforcement using a two-alternative choice procedure. In Experiment 1, the response cost for observing the discriminative stimuli was the same as that for observing the nondiscriminative stimuli. The preference for discriminative stimuli was high at all levels of overall reinforcement probability from 12.5 to 87.5%, but it was highest when the overall probability of reinforcement was low. In Experiment 2, the strength of the preference for the discriminative stimuli was assessed by progressively increasing the response cost to obtain them (while the response cost for obtaining the nondiscriminative stimuli remained low) until the preference was reversed. The lower the reinforcement probability, the greater the response cost to obtain the discriminative stimuli that was required to reverse the preference. The present results offer support for certain conditioned reinforcement accounts of observing behavior. Furthermore, they suggest that the choice procedure, together with the titration of effort to obtain the discriminative stimuli, offers a sensitive test of the preference for discriminative stimuli.

Operant conditioning of behavioral variability using a percentile reinforcement schedule.

The present investigation developed and tested a new percentile reinforcement schedule suited to study pattern variability, whose main feature was the relative dissociation it provided between the variability requirement defining criterional responses and overall probability of reinforcement. In a discrete-trials procedure, pigeons produced patterns of four pecks on two response keys. If the pattern emitted on the current trial differed from the N preceding patterns, reinforcement was delivered with probability mu. The schedule continuously adjusted the criterion N such that the probability of a criterional response, estimated from the subject's recent behavior, was always constant. In these circumstances, the criterion corresponded to an invariant percentile in the distribution of recent responses. Using a between-subjects design, Experiment 1 manipulated the variability requirement--the percentile--while keeping overall reinforcement probability constant. The degree of variability varied directly with the requirement. In addition, an inverse relationship existed between the requirement and within-group variance. Experiment 2 manipulated probability of reinforcement while maintaining the variability requirement constant. No consistent relationship was found between variability and reinforcement probability. A tentative hypothesis was advanced ascribing the operant conditioning of behavioral variability to a process of probability-dependent selection.

ELICITED RESPONDING TO SIGNALS FOR REINFORCEMENT: THE EFFECTS OF OVERALL VERSUS LOCAL CHANGES IN REINFORCEMENT PROBABILITY1

Pigeons were studied on a three-component multiple schedule where all reinforcement was independent of responding. Two components were cued by different keylights and were associated with different rates of reinforcement. The third was always a no-key period associated with extinction. After a few sessions, pecking was elicited by the keylights signalling the reinforcement and continued to be maintained indefinitely. The duration and sequence of the three components were varied to determine if the primary controlling variable was differences in the overall probability of reinforcement, or if it was the immediate change in reinforcement signalled by the onset and/or offset of the stimulus. Both variables were found to control behavior. When 30-sec components were used, the primary controlling variable was the overall probability of reinforcement, but when 3-min components were used, overall probability had little effect. Control by local changes in reinforcement also occurred, although the type of local control varied both across subjects and experimental conditions. Some behaviors were controlled more by the change in reinforcement signalled by the onset of the stimulus, while others were controlled more by the change signalled by the offset of the stimulus.

EFFECTS OF TWO PROCEDURES FOR VARYING INFORMATION TRANSMISSION ON OBSERVING RESPONSES1

Two experiments were conducted with pigeons to examine the effects of procedures that varied information transmission on observing responses. The basic procedure for Experiment I was one in which a trial terminated in either non-contingent reinforcement or timeout. Pecking during a trial produced either green (positive) or red (negative) keylights. If no pecking occurred no differential stimuli appeared. The probability of positive trials was either 0.25, 0.50, or 0.75. Observing response rates and relative frequencies of occurrence were highest when the probability of positive trials was 0.25 and lowest at 0.75. In Experiment II, a modified chain procedure was used in which responding produced either red or green lights. Reinforcement or timeout followed light onset by 15 sec. The correlation between the stimuli and the event at the end of the trial (reinforcement or timeout) was varied. Reinforcement followed green 100%, 90%, 70%, or 50% of the time that green occurred. Since the overall probability of reinforcement remained at 0.50, reinforcement followed red in either 0%, 10%, 30%, or 50% of the time that it occurred. The rate of responses that produced these stimuli varied as a function of the correlation. The greater the probability of reinforcement after green, the higher the response rate.