Reward Distributions Research Articles

The twin threshold model: risk-intermediate foraging by rufous hummingbirds, Selasphorus rufus

I developed two versions of the twin threshold model (TTM) to assess risk-sensitive foraging decisions by rufous hummingbirds. The model incorporates energy thresholds for both starvation and reproduction and assesses how three reward distributions with a common mean but different levels of variance interact with these critical thresholds to determine fitness. Fitness, a combination of survival and reproduction, is influenced by both the amount of variance in the distributions and the relative position of the common mean between the thresholds. The model predicts that risk-intermediate foraging is often the optimal policy, and that risk aversion is favoured as the common mean of the distributions approaches the starvation threshold, whereas risk preference is favoured as the common mean approaches the reproduction threshold. Tests with free-living hummingbirds supported these predictions. Hummingbirds were presented with three distributions of nectar rewards that had a common mean but Nil, Moderate or High levels of variance. Birds preferred intermediate levels of variance (Moderate) when presented with all three rewards simultaneously, and became more risk-averse as the mean of the distributions was decreased but more risk-prone as the mean was increased. Birds preferred Nil when it was paired with Moderate or with High, but preferred Moderate in the presence of Nil and High together. This reversal of preference is a violation of regularity, conventionally interpreted as irrational choice behaviour. I provide an alternative version of the TTM demonstrating that violations of regularity can occur when relative instead of absolute evaluation mechanisms are used.

Modeling learning and adaptation processes in activity-travel choice A framework and numerical experiment

This paper develops a framework for modeling dynamic choice based on a theory of reinforcement learning and adaptation. According to this theory, individuals develop and continuously adapt choice rules while interacting with their environment. The proposed model framework specifies required components of learning systems including a reward function, incremental action value functions, and action selection methods. Furthermore, the system incorporates an incremental induction method that identifies relevant states based on reward distributions received in the past. The system assumes multi-stage decision making in potentially very large condition spaces and can deal with stochastic, non-stationary, and discontinuous reward functions. A hypothetical case is considered that combines route, destination, and mode choice for an activity under time-varying conditions of the activity schedule and road congestion probabilities. As it turns out, the system is quite robust for parameter settings and has good face validity. We therefore argue that it provides a useful and comprehensive framework for modeling learning and adaptation in the area of activity-travel choice.

Reward value invariant place responses and reward site associated activity in hippocampal neurons of behaving rats.

To investigate the involvement of the hippocampal-accumbens system in goal-oriented displacement behaviors, hippocampal neuronal activity was recorded in rats learning and recalling new distributions of different volumes of liquid reward among the arms of a plus maze. Each arm had a reward box containing a water trough and identical visual cues that could be illuminated independently. As the water-restricted rat successively visited the respective boxes, it received 7, 5, and 3 drops of water, and then 1 drop, provided at 1-s intervals. (Reward distributions were reassigned daily and mid-session.) In the training phase, reward boxes were lit individually. In the recall phase, the lamps on all arms were lit and then turned off as the rat visited the boxes in order of descending value. Neuronal firing rates were analyzed for changes related to reward value or to shifts between learning and recall phases. The principal finding is that place responses remained unchanged after these manipulations and that these neurons showed no evidence of explicit coding of reward value. In addition, two other types of responses appeared while the rat was stationary at the reward boxes awaiting multiple rewards. These were observed primarily in neurons within the dentate gyrus, but also in CA1. Position-selective reward site responses were regular at 20-60 impulses per second, while position-independent discharges bursted irregularly at about 5 impulses per second. Such responses could explain controversial reports of reward dependence in hippocampal neurons. The higher incidence of the latter responses in the temporal ("ventral") hippocampus is consistent with the distinctive anatomical and functional properties of this subregion.

Finite-time Analysis of the Multiarmed Bandit Problem

Reinforcement learning policies face the exploration versus exploitation dilemma, i.e. the search for a balance between exploring the environment to find profitable actions while taking the empirically best action as often as possible. A popular measure of a policy's success in addressing this dilemma is the regret, that is the loss due to the fact that the globally optimal policy is not followed all the times. One of the simplest examples of the exploration/exploitation dilemma is the multi-armed bandit problem. Lai and Robbins were the first ones to show that the regret for this problem has to grow at least logarithmically in the number of plays. Since then, policies which asymptotically achieve this regret have been devised by Lai and Robbins and many others. In this work we show that the optimal logarithmic regret is also achievable uniformly over time, with simple and efficient policies, and for all reward distributions with bounded support.

Risk-sensitive foraging: choice behaviour of honeybees in response to variability in volume of reward

We tested risk sensitivity towards variability in volume of reward with harnessed honeybees,Apis mellifera, in a proboscis-extension conditioning paradigm. We conditioned each subject to turn its head and extend its proboscis towards one of two presented odours; one odour was associated with a constant reward volume and the other with a variable reward volume that was either low or high, with probabilitiesP=0.75 and (1−P)=0.25, respectively. The volumes of rewards were varied among three experimental conditions. In conditions I and II, the variable reward option included a low reward of zero (i.e. reinforcement was withheld in the low reward value); in condition I, the mean of the variable and of the constant reward options were the same, and in condition II, the variable reward option had a higher mean reward than the constant reward option. The behaviour of subjects did not differ between treatments and the majority of individuals were risk averse. In condition III, the variable reward option did not include a zero reward and the mean reward did not differ between options. Very few of the individuals assigned to condition III developed a preference for either reward option. Thus, honeybees are risk sensitive to variability in volume of reward in some conditions and the degree of risk sensitivity depends on characteristics of the reward distributions. The most salient characteristic may be a relative measure of variability, such as the value of the coefficient of variation of reward. The experimental paradigm that we developed is a powerful tool for studying the mechanism of risk sensitivity in bees, as well as other aspects of learning, decision making, perception and memory.

EFFECTS OF REWARD DISTRIBUTION AND PERFORMANCE FEEDBACK ON COMPETITIVE RESPONDING

To implement competitive contingencies, one must select a distribution of unequal rewards and a schedule of feedback for competitors regarding one another's performance. This study investigated three bases for distributing rewards and two performance feedback conditions. Pairs of college students competed over a series of 2‐min contests in which the competitive response was a knob pull. A sum of money was divided using a proportional distribution or one of two fixed reward distributions. In the proportional distribution, a subject's proportion of the sum was his or her proportion of the total number of responses. The two fixed distributions were divisions of 100%/0% or 67%/33%. Also, in every contest either subject could make a response that would end the contest prematurely and give both subjects the same amount—a sum equal to 33% of the competitive total. In the two feedback conditions, cumulative responses by each subject were either shown to both subjects during the contest or were not shown. The proportional distribution was clearly superior to either of the fixed distributions in number of responses produced across contests. The proportional distribution with feedback produced the largest number of competitive responses, and the 100%/0% distribution without feedback produced the smallest number. Differences among distributions typically emerged only during later blocks of contests. Fixed distributions of rewards often produced decelerating rates of responding, with losing competitors ending the contests before they were completed. Response‐rate decreases were greatest for pairs in which the 2 subjects differed most in their response rates and proportion of wins. The presence of feedback had a small effect, increasing responding for some pairs in the 100%/0% distribution. Performance patterns were interpreted in terms of the consequences arranged for the individual participants by the reward distributions and differences in performance between competitors.

A class of bandit problems yielding myopic optimal strategies

We consider the class of bandit problems in which each of the n ≧ 2 independent arms generates rewards according to one of the same two reward distributions, and discounting is geometric over an infinite horizon. We show that the dynamic allocation index of Gittins and Jones (1974) in this context is strictly increasing in the probability that an arm is the better of the two distributions. It follows as an immediate consequence that myopic strategies are the uniquely optimal strategies in this class of bandit problems, regardless of the value of the discount parameter or the shape of the reward distributions. Some implications of this result for bandits with Bernoulli reward distributions are given.

A class of bandit problems yielding myopic optimal strategies

We consider the class of bandit problems in which each of the n ≧ 2 independent arms generates rewards according to one of the same two reward distributions, and discounting is geometric over an infinite horizon. We show that the dynamic allocation index of Gittins and Jones (1974) in this context is strictly increasing in the probability that an arm is the better of the two distributions. It follows as an immediate consequence that myopic strategies are the uniquely optimal strategies in this class of bandit problems, regardless of the value of the discount parameter or the shape of the reward distributions. Some implications of this result for bandits with Bernoulli reward distributions are given.

Process and Outcome: Perspectives on the Distribution of Rewards in Organizations

James N. Baron Stanford University Karen S. Cook University of Washington Gaining a better understanding of reward systems in organizations has become a high priority, not only for scholars in diverse disciplines but also, increasingly, for managers, human resource specialists, and those who craft public policy. Organizations are claimed to be undergoing a period of unprecedented experimentation in appraising and rewarding employees. In the current presidential campaign, candidates are staking out positions on levels of U.S. executive compensation. Journalistic accounts contrasting American greed and individualism with the strong group ethic and compressed reward distributions of Japan and other countries have become hackneyed.

Animal Choice Behavior and the Evolution of Cognitive Architecture

Animals process sensory information according to specific computational rules and, subsequently, form representations of their environments that form the basis for decisions and choices. The specific computational rules used by organisms will often be evolutionarily adaptive by generating higher probabilities of survival, reproduction, and resource acquisition. Experiments with enclosed colonies of bumblebees constrained to foraging on artificial flowers suggest that the bumblebee's cognitive architecture is designed to efficiently exploit floral resources from spatially structured environments given limits on memory and the neuronal processing of information. A non-linear relationship between the biomechanics of nectar extraction and rates of net energetic gain by individual bees may account for sensitivities to both the arithmetic mean and variance in reward distributions in flowers. Heuristic rules that lead to efficient resource exploitation may also lead to subjective misperception of likelihoods. Subjective probability formation may then be viewed as a problem in pattern recognition subject to specific sampling schemes and memory constraints.

Risk-aversion, relative abundance of resources and foraging preference

Foraging theory depicts dietary choice as a function of prey quality and absolute abundance. Ecological processes, however, can depend on the way foragers respond to the relative abundances of available prey types; several models for frequency-dependent foraging adequately describe these responses. Our laboratory experiments with white-throated sparrows investigated preferential choice of two food rewards as we manipulated both reward quality and relative abundance. In any single experiment the two rewards provided the same mean food quantity, but the variances differed. Average energy budgets predicted risk-aversion, so that foraging preference should decrease as reward variance increases. We presented each two-reward pairing at availability ratios of 1:2, 1:1, and 2:1 for three consecutive days. By the third day risk-aversion exceeded preference for reward variance significantly. When relative abundances of the low and high variance rewards were not equal, the birds tended to prefer the rare over the common reward. This response began before the birds had thoroughly sampled the reward distributions. Preference for rarity apparently constrained the birds' economic response to reward variance levels.

Gender and reward distributions: A test of two hypotheses

This study was designed as a test of two competing explanations of gender differences in distributive justice: (a) the equity-equality hypothesis, which states that males endorse equitable distributions more than females and females endorse equal distributions more than males; and (b) the exploitation-accommodation hypothesis, which states that the sexes vary their norm endorsement according to self-favoring (males) or other-favoring (females) distribution outcomes. Preadolescent and college-aged subjects rated the fairness of reward distributions of vignette characters who had contributed either more or less than a co-worker in a task, and had subsequently divided the rewards either equitably or equally. The data provided no support for the equity-equality hypothesis, but did support the exploitation-accommodation hypothesis. Specifically, females rated equitable distributions of inferior workers as more fair than males did. Thus, the popular conclusion that males have a stronger commitment to equity than females must be rejected.

Caste‐specific patterns of flower visitation in bumble bees (Bombus kirbyellus) collecting nectar from Polemonium viscosum

ABSTRACT. Foraging routes of worker and queen bumble bees (Bombus kirbyellus Curtis) collecting nectar from flowers of the alpine sky pilot, Polemonium viscosum Nutt., were followed and the corolla tube length, corolla diameter, floral scent, and number of flowers on plants visited or bypassed by bees were monitored. Additionally, the number and proportion of flowers visited per inflorescence and distance flown from each to the next were recorded. Queens and workers differed significantly in choice of flowers. However, intra‐inflorescence visitation rates and departure distances were similar between castes. Castes differed in the extent to which visitation reflected patch quality versus individual floral traits. Both queens and workers failed to visit skunky‐flowered plants more often than they failed to visit sweet‐flowered ones, and preferred large over small inflorescences. However, queens visited large‐flowered plants more often than small‐flowered ones, while workers preferred flowers with shorter corolla tubes, regardless of their diameter. Although a number of studies have documented caste specialization on alternate species of host plants, ours is one of the first to show that morphological preferences promote comparable foraging differences between castes on monospecific plant resources. Queens, once on a plant, responded to floral traits by probing more flowers on large inflorescences, as well as on those with broader floral form. Workers did not alter intra‐inflorescence visitation rate in response to floral traits. For workers, no significant relationship was demonstrated between the likelihood of passing by a plant and the number of flowers probed on the previous inflorescence visited. Thus, workers appeared to accept or reject each plant of P. viscosum independently. However, queens passed by fewer plants when leaving rich inflorescences than poor ones. These results suggest that workers use only individual plant acceptability in choosing which plants to visit, whereas queens base plant choice on patch and individual attributes. Such differences between castes in foraging rules when exploiting the same floral resource have received little attention, and provide insights into the heterogeneity of harvestable reward distributions from the perspective of the forager population.

Contrasting movement patterns of nectar‐collecting and pollen‐collecting bumble bees (Bombus terricola) on fireweed (Chamaenerion angustifolium) inflorescences

Abstract. 1. Movements of nectar and pollen‐foraging bumble bees on inflorescences of Chamaenerion angustifolium (L.) J. Holub (fireweed or rosebay willow herb) were compared with predictions based on reward distributions and optimality principles. Observations suggest that nectar and pollen‐gathering bumble bees behave according to the same set of reward maximization criteria when foraging from flowers of this species.2. Both kinds of foragers matched their arrival points with the vertical positions on inflorescences in which the densities of their respective food resources were greatest. For nectar‐foragers, this point was located at the lowest tier of flowers, whereas for pollen‐foragers it was found in the middle of the inflorescences. Nectar and pollen‐foraging bees both moved upward on inflorescences following gradients from high to low reward availability.3. Nectar‐foragers responded to decreases in inflorescence size over the season by reducing the number of flower visits made on each raceme. Number of flowers visited by pollen‐foragers was low throughout and reflected the scarcity of male‐phase flowers on racemes. Flower revisitation rates were low for both kinds of workers, but were slightly higher for those collecting pollen.

Gaps and Glissandos: Inequality, Economic Development, and Social Mobility in 24 Countries

Intergenerational mobility has been seen as influenced by both level of economic development and political democracy. Here, with a sample of 24 countries, the first of these relationships is assessed. The observed effect of economic development (GNP/capita) on mobility we conclude to be a spurious consequence of the shape of the stratification system, indicated here by the shape of both reward distributions and occupational distributions. Some discussion precedes this analysis about how the shape of a stratification system should affect mobility. Some discussion follows about how the shape of the system must affect political democracy, and leads us to a partial reinterpretation of the findings of Rubinson and Quinlan (1977) on this topic.

Does Equity Describe the Results of Allocation Experiments?

Adams' (1963, 1965) theory of inequity has recently been extended to predict the results of allocation experiments. Most of these experiments have been limited to groups of size two. The present experiment had subjects distribute bonuses among groups of two, four, or six workers in which one key worker produced 10, 30, 50, 70, or 90% of the group's output. Mean bonus allocations provided strong support for an equity rule, with added trends toward overreward of 50% producers and underreward of 70 and 90% producers in four and six person groups. Closer examination of reward distributions revealed that equity was the average of a number of divergent strategies. The complexities of determining allocation strategies from post hoc examination of reward distributions are discussed.

Status Differentiation, External Consequences, and Alternative Reward Distributions

The question is raised as to the relative degree of determinance exerted on the distribution of rewards (egalitarian and differential) in groups by (1) the group's status structure and (2) the external consequences following the distributions. A longitudinal, laboratory experiment was designed requiring one-hour's participation of each group for ten days. The data indicated that, under the conditions of the experiment, external conquences could effect greater control of the group's reward distributions than the group's status structure.