Relative Frequency Of Reinforcement Research Articles

Divided stimulus control: Which key did you peck, or what color was it?

Responding on concurrent schedules produced a conditional discrimination (Phases 1 and 2), asking either which peck produced the event, or which color the keys were when the event was produced. In Phases 3 and 4, reinforcer delivery or a delay in blackout was interpolated between responding and the conditional discrimination. In Phase 1, location versus color discrimination accuracy was controlled by the relative reinforcer frequency for correct responses to these questions (divided stimulus control). In Phases 2 to 4, relative reinforcer frequency for correct responses to these questions was .5, and the relative frequency with which concurrent-schedule responses produced the questions was varied. This variation had no clear effect on the accuracy of reporting Location or Color. These results are consistent with the model of divided control suggested by Davison and Elliffe (2010). Arranging a 3-s reinforcer between responding and choice decreased both color and location accuracy, but a 3-s delay only decreased location accuracy. Thus, in concurrent-schedule performance, both ambient stimuli prior to a reinforcer and the location of the just-reinforced response are available as discriminative stimuli following the reinforcer. Control of postreinforcer responding is divided between these according to their association with the relative frequency of subsequent reinforcers.

Inter-response-time reinforcement and relative reinforcer frequency control choice.

In Conditions 1 and 3 of our Experiment 1, rats pressed levers for food in a two-component multiple schedule. The first component was concurrent variable-ratio (VR) 20 variable-interval (VI) 90 s, and the second was concurrent yoked VI (its reinforcement rate equaled that of the prior component's VR) VI 90 s. In Condition 2, the VR was changed to tandem VR 20, differential reinforcement of low rates (DRL) 0.8 s. Local response rates were higher in the VR than in the yoked VI schedule, and this difference disappeared between tandem VR DRL and yoked VI. The relative time allocations to VR and yoked VI, as well as to tandem VR DRL and yoked VI, were approximately the same across conditions. In Experiment 2, rats chose in a single session between five different VI pairs, each lasting for 12 reinforcer presentations (variable-environment procedure). The across-schedule hourly reinforcement rates were 120 and 40, respectively, in Conditions 1-3 and 4-6. During Conditions 2 and 5, one lever's VI was changed to tandem VI, DRL 2 s. High covariation between relative time allocations and relative reinforcer frequencies, as well as invariance in local response rates to the schedules, was evident in all conditions. In addition, the relative local response rates were biased toward the unchanged VI in Conditions 2 and 5. These results demonstrate two-process control of choice: Inter-response-time reinforcement controls the local response rate, and relative reinforcer frequency controls relative time allocations.

RAPID ACQUISITION OF BIAS IN SIGNAL DETECTION: DYNAMICS OF EFFECTIVE REINFORCEMENT ALLOCATION

We investigated changes in bias (preference for one response alternative) in signal detection when relative reinforcer frequency for correct responses varied across sessions. In Experiment 1, 4 rats responded in a two-stimulus, two-response identification procedure employing temporal stimuli (short vs. long houselight presentations). Relative reinforcer frequency varied according to a 31-step pseudorandom binary sequence and stimulus duration difference varied over two values across conditions. In Experiment 2, 3 rats responded in a five-stimulus, two-response classification procedure employing temporal stimuli. Relative reinforcer frequency was varied according to a 36-step pseudorandom ternary sequence. Results of both experiments were analyzed according to a behavioral model of detection. The model was extended to incorporate the effects of current and previous session reinforcer frequency ratios on current-session performance. Similar to findings with concurrent schedules, effects on bias of relative reinforcer frequency were highest for the current session. However, carryover from reinforcer ratios of previous sessions was evident. Generally, the results indicate that bias can come under control of frequent changes in relative reinforcer frequency in both identification and classification procedures.

DIVIDED STIMULUS CONTROL: A REPLICATION AND A QUANTITATIVE MODEL

Four pigeons were trained on a conditional discrimination. The conditional stimuli were compounds of pairs of stimuli from two different dimensions, fast versus slow cycles of red or green stimuli, and short- versus long-duration presentations of these cycles. Across conditions, the probability of reinforcers for correctly responding to each dimension was varied from 0 to 1. Discriminability, measured by log d, for stimuli on a dimension increased as the relative frequency of reinforcers for that dimension increased, replicating the results of Shahan and Podlesnik (2006). Two further conditions showed that discriminability between stimuli on each dimension was unaffected by whether the stimuli on the other dimension varied or were constant. Finally, maximal discriminability was unchanged in a redundant-relevant cues condition in which either of the stimuli comprising a compound signaled the same correct response. Davison and Nevin's (1999) model provided an excellent quantitative account of the effect of relative reinforcer frequency on discriminability, and thus of the way in which divided stimulus control is itself controlled by relative reinforcement.

EFFECTS OF POINT‐LOSS PUNISHERS ON HUMAN SIGNAL‐DETECTION PERFORMANCE

Three experiments using human participants varied the distribution of point-gain reinforcers or point-loss punishers in two-alternative signal-detection procedures. Experiment 1 varied the distribution of point-gain reinforcers for correct responses (Group A) and point-loss punishers for errors (Group B) across conditions. Response bias varied systematically as a function of the relative reinforcer or punisher frequencies. Experiment 2 arranged two conditions - one where an unequal ratio of reinforcement (5ratio1 or 1ratio5) was presented without punishment (R-only), and another where the same reinforcer ratio was presented with an equal distribution of point-loss punishers (R+P). Response bias was significantly greater in the R-only condition than the R+P condition, supporting a subtractive model of punishment. Experiment 3 varied the distribution of point-gain reinforcers for correct responses across four unequal reinforcer ratios (5ratio1, 2ratio1, 1ratio2, 1ratio5) both without (R-only) and with (R+P) an equal distribution of point-loss punishers for errors. Response bias varied systematically with changes in relative reinforcer frequency for both R-only and R+P conditions, with 5 out of 8 participants showing increases in sensitivity estimates from R-only to R+P conditions. Overall, the results indicated that punishers have similar but opposite effects to reinforcers in detection procedures and that combined reinforcer and punisher effects might be better modeled by a subtractive punishment model than an additive punishment model, consistent with research using concurrent-schedule choice procedures.

MATCHING: ITS ACQUISITION AND GENERALIZATION

Choice typically is studied by exposing organisms to concurrent variable-interval schedules in which not only responses controlled by stimuli on the key are acquired but also switching responses and likely other operants as well. In the present research, discriminated key-pecking responses in pigeons were first acquired using a multiple schedule that minimized the reinforcement of switching operants. Then, choice was assessed during concurrent-probe periods in which pairs of discriminative stimuli were presented concurrently. Upon initial exposure to concurrently presented stimuli, choice approximated exclusive preference for the alternative associated with the higher reinforcement frequency. Concurrent schedules were then implemented that gave increasingly greater opportunities for switching operants to be conditioned. As these operants were acquired, the relation of relative response frequency to relative reinforcement frequency converged toward a matching relation. An account of matching with concurrent schedules is proposed in which responding exclusively to the discriminative stimulus associated with the higher reinforcement frequency declines as the concurrent stimuli become more similar and other operants-notably switching-are acquired and generalize to stimuli from both alternatives. The concerted effect of these processes fosters an approximate matching relation in commonly used concurrent procedures.

Molar function depends on molecular structure of behavior.

Pigeons chose between green side keys, then waited a shorter or longer time before pecking a center key, and finally chose between red side keys. Two successive left choices (to green and then red) with a shorter wait intervening, or 2 successive right choices with a longer wait intervening, were intermittently reinforced with food. The 2 required waiting times and the relative frequency of reinforcement for the shorter reinforced pattern were varied. Molar preference, defined in terms of coherent responses that satisfied the molecular reinforcement contingency, conformed to the highly adaptive matching level, but molar preference, defined in terms of incoherent responses, did not. The molar matching result therefore generalizes to responses with complex molecular structures provided that analyses distinguish between coherent and incoherent responses. The results are compatible with the idea that awareness can facilitate adaptation.

Computational behavior dynamics: an alternative description of Nevin (1969).

A computational processing behavior-dynamic model was instantiated in the form of a computer program that "behaved" on the task developed by Nevin (1969). In this classic discrete-trials experiment, the relative frequency of choosing a response alternative matched the relative frequency of reinforcement for that alternative, the local structure of responding was opposite that predicted by momentary maximizing (i.e., the probability of a changeover decreased with run length), and absolute and relative response rates varied independently. The behavior-dynamic model developed here qualitatively reproduced these three results (but not in quantitative and specific detail) and also generated some interesting, as-yet-untested predictions about performance in Nevin's task. The model was discussed as an example of a stochastic behavior-dynamic alternative to algebraic behavior theory.

BEHAVIOR‐DEPENDENT REINFORCER‐RATE CHANGES IN CONCURRENT SCHEDULES: A FURTHER ANALYSIS

Six pigeons were trained on concurrent variable-interval schedules in three different procedures. The first procedure was a standard concurrent schedule, and the relative reinforcer frequency for responding was varied. The second was a schedule in which a relative left-key response rate (over a fixed period of time) exceeding .75 produced, in the next identical time period a higher reinforcer rate on the right key. If this criterion was not exceeded, equal reinforcer rates were arranged on the two keys in this period. This was the dependent procedure. In the third (independent) procedure, the periods of higher right-key reinforcer rates occurred with the same probability as in the second procedure, but occurred independently of behavior. In the second and third procedures, the fixed-time period (window) was varied from 5 s to 60 s, and to 240 s in the second procedure only. Performance on the two keys was similar in the concurrent and independent procedures. The procedure used in the dependent conditions generally affected performance when the windows were shorter than about 30 s. Models of performance that assume that subjects do not discriminate changes in local relative reinforcer rates cannot account for the data. Moreover, existing models are inherently unable to account for the effects of contingencies of reinforcement between responding on one alternative and gaining reinforcers on another that are arranged or that emerge as a result of time allocated to alternative schedules. Undermatching on concurrent variable-interval schedules may result from such emergent contingencies.

CHOICE, CHANGEOVER, AND TRAVEL: A QUANTITATIVE MODEL

Six pigeons were trained on concurrent variable-interval schedules in which responding on fixed-interval schedules was required to give access to the alternate schedule. Responding on the concurrent schedules was not allowed, after changing over had commenced, until the changeover schedule had been completed. In Parts 1 to 3 of the experiment, the changeover fixed-interval schedules were equal and were 0 s, 10 s, and 20 s, respectively. In each part, the relative frequency of reinforcement obtained on the concurrent schedules was varied over at least five conditions. In Part 4, the concurrent schedules were equal, and one changeover fixed-interval schedule was twice the other. Under these conditions, the absolute sizes of the changeover schedules were varied. Increasing the changeover requirement from 0 s to 10 s (Parts 1 and 2) resulted in increases in the sensitivity of behavior allocation to reinforcers obtained, but no further increase was obtained when the changeover schedules were increased to 20 s (Part 3). In Part 4, performance was biased towards the concurrent schedule that took less time to enter. These results are consistent with a subtractive punishment model of travel in which the degree of punishment is measured by the number of reinforcers apparently lost from a schedule when the subject changes to that schedule. Absolute times spent on the main keys could be accurately described by a previous model of changeover performance.

Intuitive statistical inference: How pigeons categorize binomial samples

Pigeons categorized binomial samples produced by two complementary random processes. Samples were 1, 2, 4, or 8 successively presented outcomes (vertical or horizontal lines) of, in effect, tossing one of two equally likely coins. One coin (A) was biased in favor of vertical, and the other coin (B) was similarly biased in favor of horizontal. Choosing red or green stimuli presented after each sample was reinforced with food if coin A or coin B had been tossed, respectively. In that sense, choosing red or green was logically analogous to inferring that the statistical evidence reflected tossing of coin A or coin B, respectively. The statistical diagnosticity of a sample, the relative likelihood of its having been produced by a particular coin, equaled, except for sampling fluctuation, the relative frequency of reinforcement of a particular color given that sample, and was experimentally varied by the bias on the coins and by sample size. All the variables that affect optimal, formal inference about binomial samples also affected intuitive inference. But inferences were very suboptimal: “undermatching” was obtained in part due to control of categorization by the sequential structure of binomial samples. These results reveal limitations of optimality theories for animal decision making in the face of uncertainty when observations in samples are presented successively. On the other hand, they are generally compatible with molecular analyses of instrumental learning which assign an important role to the local temporal organization of events preceding reinforcement. Most generally, they show that maladaptive control over intuitive statistical inference by a variable upon which optimal performance does not depend is neither a uniquely human phenomenon nor dependent upon linguistic strategies.

Local patterns in human operant behavior and a behaving model to interrelate animal and human performances.

College students pressed buttons for points delivered according to molecular concurrent reinforcement contingencies similar to those used in previous experiments with animal subjects. Relative frequency of reinforcement and the relative and absolute durations of two reinforced patterns of button pressing were experimentally varied. Qualitative effects of all three variables resembled those obtained previously with animals. Computer simulation of a cognitive processing model described these qualitative effects. A similar model was used previously to describe corresponding behavior in pigeons. Therefore, similar cognitive processing may underlie the local temporal patterning of animal and human operant behavior maintained by this concurrent reinforcement contingency.

EFFECTS OF RELATIVE REINFORCER FREQUENCY ON COMPLEX COLOR DETECTION

Pigeons were trained under a discrete-trials detection procedure in which one of a set of color stimuli was presented on the center key and a single response turned off the stimulus and illuminated two side keys. Single responses to one or the other side key produced occasional reinforcers depending on the value of the color stimulus. In Experiment 1, one color-stimulus set comprised 559, 564, 569, and 574 nm, and right-key pecks were occasionally reinforced following presentations of members of this set. The other stimulus set comprised 579, 584, 589, and 594 nm, and left-key pecks were occasionally reinforced following presentations of members of this set. Across seven experimental conditions, the left/(left + right) relative reinforcer frequency was varied from .1 to .9. In Experiment 2, one stimulus set contained only one member, 574 nm, and right-key responses were occasionally reinforced following its presentation. Over 12 experimental conditions, two manipulations were carried out. First, the number of stimuli comprising the other stimulus set was increased from one (579 nm) to two (579 and 584 nm) to three (579, 584, and 589 nm) and to four (579, 584, 589, and 594 nm), and left-key responses were reinforced occasionally following center-key presentations of members of this set. Second, for each stimulus combination, the left/(left + right) relative reinforcer frequency was varied from .1 to .5 to .9 across three experimental conditions. The principal finding of Experiments 1 and 2 was that reinforcers and stimuli interacted in their effects on behavior. In Experiment 3, pairs of adjacent stimuli (5 nm apart) in the range 559 to 594 nm were presented in each experimental condition, and the left/(left + right) relative reinforcer frequency was held constant at .5. The data from all three experiments were analyzed according to a detection model describing performance in multiple-stimulus two-response procedures. This model provided independent measures of stimulus discriminability, contingency discriminability, and bias. The analysis showed that (a) consistent with the color-naming function, pigeons were better able to discriminate between higher nanometer values than lower nanometer values; (b) their ability to discriminate between the stimuli was independent of the number of wavelengths comprising each stimulus set; (c) they allocated delivered reinforcers very accurately to the previously emitted response; and (d) no consistent biases emerged.

CONCURRENT VARIABLE‐RATIO SCHEDULES: IMPLICATIONS FOR THE GENERALIZED MATCHING LAW

Rats' responses were reinforced on concurrent variable-ratio variable-ratio schedules in which responses on one lever incremented the ratio counter and responses on a second lever changed the schedule and correlated stimulus. The relative frequency of reinforcement was varied from .10 to .99. In one set of conditions, responding on the main lever incremented both ratio counters, but reinforcement required a response in the presence of the stimulus correlated with the ratio that had been completed. In a second set of conditions, responses on the main lever incremented only the ratio correlated with the stimulus that was currently present. When main-lever responses incremented both ratio counters, subjects distributed responding and time in a manner consistent with the generalized matching law. When responses on the main lever incremented only the schedule currently in effect, the rats responded almost exclusively on the schedule producing the higher frequency of reinforcement. These results extend the applicability of the generalized matching law to dependent ratio schedules.

Reinforcer frequency and reinforcer duration as biasers of signal detection

Five pigeons were trained to discriminate a constant luminance difference when both reinforcer frequency and reinforcer magnitude were varied in an animal analogue of the human discrete-trial yes- no detection task. Variation of relative reinforcer frequency was more effective in biasing detection performance than variation of relative reinforcer duration. Stimulus discriminability was independent of both reinforcer frequency and reinforcer duration.

Delayed signal detection, differential reinforcement, and short-term memory in the pigeon.

In two discrete-trial delayed-detection experiments, six pigeons were trained on dependent concurrent variable-interval schedules. Pecking a red side key was reinforced when the brighter of two white lights (S1) had been presented on the center key, and pecking a green side key was reinforced when the duller of two white lights (S2) had been presented on the center key. Incorrect responses were red side-key pecks following S2 presentations and green side-key pecks following S1 presentations; these resulted in three-second blackouts. In Experiment 1, the time between presentation of S1 or S2 on the center key and the onset of the red and green side keys was varied nonsystematically from 0.06 seconds to 19.69 seconds across experimental conditions. Stimulus discriminability decreased as the stimulus-choice delay increased. A rectangular-hyperbolic function better described this decrease in discriminability over time than did a negative-exponential function. In Experiment 2, at each of three stimulus-choice delays (0.06, 3.85, and 10.36 seconds), relative reinforcer frequency for correct responses to the red and green side keys was varied by changing the values of the dependent concurrent variable-interval schedules. The sensitivity of choice to relative reinforcer frequency was independent of the decrease in stimulus discriminability with increasing stimulus-choice delay.

Isobias and alloiobias functions in animal psychophysics.

An analogue of the human yes-no detection task was used to train six pigeons to discriminate luminance differences under two different reinforcer-scheduling procedures. When a controlled reinforce-ratio procedure was used, relative stimulus frequency was constant at .5, and relative reinforcer frequency for correct detections was held constant at three different values for each of five luminance differences. When an uncontrolled reinforcer-ratio procedure (the typical detection paradigm) was used, relative reinforcer frequency for correct detections was allowed to covary with changes in relative stimulus frequency for each of five luminance differences. Two measures of bias, response bias (Davison & Tustin, 1978) and the detection-theory likelihood-ratio measure (beta obt), were compared. The controlled reinforcer-ratio procedure generated equal- or iso-response-bias functions, and the uncontrolled reinforcer-ratio procedure gave changing or alloio-response-bias functions. The Davison-Tustin model accounted for 88% and 93% of the data variance in the controlled and uncontrolled reinforcer-ratio procedures, respectively. The best-fitting equal-beta functions accounted for an average of 53% and 69%, respectively, in the two procedures. In addition, neither procedure gave constant measures of beta obt for constant bias manipulations across different discriminability measures.

CHOICE BETWEEN RESPONSE UNITS: THE RATE CONSTANCY MODEL

In a conjoint schedule, reinforcement is available simultaneously on two or more schedules for the same response. The present experiments provided food for key pecking on both a random-interval and a differential-reinforcement-of-low-rate (DRL) schedule. Experiment 1 involved ordinary DRL schedules; Experiment 2 added an external stimulus to indicate when the required interresponse time had elapsed. In both experiments, the potential reinforcer frequency from each component was varied by means of a second-order fixed-ratio schedule, and the DRL time parameter was changed as well. Response rates were described by a model stating that time allocation to each component matches the relative frequency of reinforcement for that component. When spending time in a given component, the subject is assumed to respond at the rate characteristic of baseline performance. This model appeared preferable to the absolute-rate version of the matching law. The model was shown to be applicable to multiple-response concurrent schedules as well as to conjoint schedules, and it described some of the necessary conditions for response matching, undermatching, and bias. In addition, the pigeons did not optimize reinforcer frequency.

Signal detection and matching: analyzing choice on concurrent variable-interval schedules.

Pigeons' pecks on a red key and a green key were followed by access to grain according to pairs of concurrent independent variable-interval schedules in a combined signal detection/matching law paradigm. Pecks on the red key were reinforced by the richer variable-interval schedule if a short-duration tone had been presented; pecks on the green key were reinforced by the richer variable-interval schedule if a long-duration tone had been presented. Pecks on the green key given a short-duration tone, or on the red key given a long-duration tone, were reinforced by the leaner variable-interval schedule. The data were analyzed according to both signal detection's and the matching law's separate measures of, first, the discrimination of the choices and, second, the bias to make one response or another. Increasing the difficulty of the tone-duration discrimination decreased both methods' measures of the discrimination of the choices and did not change both methods' measures of the bias to make one response or another. Changing the leaner variable-interval schedule so that it approached the richer variable-interval schedule decreased signal detection's measure of discrimination but left its measure of response bias and the matching law measures unchanged. Data collected only until a subject's first changeover response following presentation of a long or a short tone showed higher values for both methods' measures of discrimination, no change in signal detection's measure of response bias, and lower values for the matching law's measure of response bias. Relationships between the matching law's and signal detection's methods of analyzing choice are discussed. It is concluded that a signal detection analysis is more efficient for examining changes in the difficulty of a discrimination, whereas a matching law analysis is more effective for examining the effects of changes in relative reinforcer frequency.

CHOICE AND BEHAVIORAL PATTERNING

Ten pigeons pecked left and right keys in a discrete-trials experiment in which access to food was contingent upon changeovers to the right key after particular runs of left-key pecks. In each of three sets of conditions, two run lengths were reinforced according to a concurrent variable-interval schedule: reinforcement followed runs of either 1 or 2, 1 or 4, or 2 or 4 left-key pecks preceding changeovers. The intertrial interval separating successive pecks was varied from .5 to 10.0 sec, and the relative frequency of reinforcement for the shorter of the two reinforced runs was varied from 0 to .75. The contingencies established local behavioral patterning that roughly approximated that required for reinforcement. For a fixed pair of reinforced run lengths, preference for the shorter of the two frequently increased as the intertrial interval increased and therefore as the minimum temporal durations of both reinforced runs increased. Preference for the shorter of the two also increased as its corresponding relative frequency of reinforcement increased. Both of these effects on preference were qualitatively similar to corresponding effects in previous research with two different kinds of reinforced behavioral patterns, interresponse times and interchangeover times. In all these experiments, analytical units were found in the temporal patterns of behavior, not in the behavior immediately contiguous with a reinforcer. It is suggested that a particular local temporal pattern of behavior is established to the extent to which it is repeatedly remembered when reinforcers are delivered, regardless of whether the delivery of a reinforcer is explicitly contingent upon that pattern.