Geometric Discounting Research Articles

Markov Decision Processes with Time-Varying Geometric Discounting

Canonical models of Markov decision processes (MDPs) usually consider geometric discounting based on a constant discount factor. While this standard modeling approach has led to many elegant results, some recent studies indicate the necessity of modeling time-varying discounting in certain applications. This paper studies a model of infinite-horizon MDPs with time-varying discount factors. We take a game-theoretic perspective – whereby each time step is treated as an independent decision maker with their own (fixed) discount factor – and we study the subgame perfect equilibrium (SPE) of the resulting game as well as the related algorithmic problems. We present a constructive proof of the existence of an SPE and demonstrate the EXPTIME-hardness of computing an SPE. We also turn to the approximate notion of epsilon-SPE and show that an epsilon-SPE exists under milder assumptions. An algorithm is presented to compute an epsilon-SPE, of which an upper bound of the time complexity, as a function of the convergence property of the time-varying discount factor, is provided.

Magnitude-sensitive reaction times reveal non-linear time costs in multi-alternative decision-making.

Optimality analysis of value-based decisions in binary and multi-alternative choice settings predicts that reaction times should be sensitive only to differences in stimulus magnitudes, but not to overall absolute stimulus magnitude. Yet experimental work in the binary case has shown magnitude sensitive reaction times, and theory shows that this can be explained by switching from linear to multiplicative time costs, but also by nonlinear subjective utility. Thus disentangling explanations for observed magnitude sensitive reaction times is difficult. Here for the first time we extend the theoretical analysis of geometric time-discounting to ternary choices, and present novel experimental evidence for magnitude-sensitivity in such decisions, in both humans and slime moulds. We consider the optimal policies for all possible combinations of linear and geometric time costs, and linear and nonlinear utility; interestingly, geometric discounting emerges as the predominant explanation for magnitude sensitivity.

Consumption Smoothing and Discounting in Infinite-Horizon, Discrete-Choice Problems

Suppose the consumption space is discrete. Our first contribution is a technical result showing that any continuous utility function of any stationary preference relation over infinite consumption streams has convex range, provided that the agent is sufficiently patient. Putting the result to use, we consider a model of endogenous discounting (a generalization of the standard model with geometric discounting) and show the uniqueness of the consumption-dependent discount factor as well as the cardinal uniqueness of utility. Comparative statics are then provided to substantiate the uniqueness. For instance, we show that, as in the more familiar case of an infinitely divisible good, the cardinal uniqueness of utility captures an agent’s desire to smooth consumption over time.

Algorithms for Stochastic Games With Perfect Monitoring

We study the pure‐strategy subgame‐perfect Nash equilibria of stochastic games with perfect monitoring, geometric discounting, and public randomization. We develop novel algorithms for computing equilibrium payoffs, in which we combine policy iteration when incentive constraints are slack with value iteration when incentive constraints bind. We also provide software implementations of our algorithms. Preliminary simulations indicate that they are significantly more efficient than existing methods. The theoretical results that underlie the algorithms also imply bounds on the computational complexity of equilibrium payoffs when there are two players. When there are more than two players, we show by example that the number of extreme equilibrium payoffs may be countably infinite.

Comment

Comment

Algorithms for Stochastic Games with Perfect Monitoring

We study the pure-strategy subgame-perfect Nash equilibria of stochastic games with perfect monitoring, geometric discounting, and public randomization. Novel algorithms are developed for calculating the discounted payoffs that can be attained in equilibrium. We also provide implementations of our algorithms. Preliminary simulations indicate that they are significantly more efficient than existing methods. The theoretical results that underlie the algorithms also yield a bound on the number of extremal equilibrium payoffs and on the computational complexity of equilibrium payoffs when there are two players.

Geometric service life modelling and discounting, a practical method for parametrised life cycle assessment

To evaluate the long-term advantage of reusing building elements, including reduced material consumption and waste production, life cycle assessments are purposeful. To translate these assessments in relevant design advices, it is necessary to model accurately the service life of the considered elements and acknowledge the related uncertainties. Practical methods to do this are nevertheless lacking. In reaction, this paper proposes a new assessment method: geometric service life modelling and discounting. The developed method is extensively parametric. Its formulas express an element’s service life in terms of a limited number of variables. This facilitates the evaluation of large series of elements as well as the automation of uncertainty analyses. Further, the method tackles different modelling complexities such as the interaction between replacements and refurbishments. Taking into account these complexities aligns the assessments with realistic service lives. For the presentation of the developed method, a focus on life cycle costing is chosen. In this paper, the outcomes of the newly developed method are compared to those of an existing calculation method and benchmarked with the manual modelling and assessment of 390 simplified building elements. This comparison is based on three indicators characterising the methods’ accuracy: the number of interventions, their individual impact and their resulting net present value. For each indicator, geometric discounting led to a considerable increase of accuracy compared to the existing method. From this comparison, it is concluded that geometric service life modelling and discounting offers not only a well-defined procedure for parametrised life cycle assessment studies, this method is also more accurate than the existing one. Moreover, the uncertainty analyses it facilitates illustrate how detailed assessment outcomes and relevant design advices about the effectiveness of element reuse can be obtained. Nevertheless, further research about the method’s practical implementation is required.

Rankings over Time

SummaryIn 2010, Kim Clijsters won the U.S. Open but had her world ranking drop from #3 to #5 by the Women′s Tennis Assocation (WTA). How can a tennis player win a tournament but drop in the rankings? The WTA uses a moving window to determine the rankings. Discounting older results in the window can prevent such counterintuitive behavior. We consider geometric and arithmetic discounting methods. We examine real data from the WTA and comment on discounting methods already in use by the Fédération Internationale de Football Association (FIFA) for ranking national teams for the World Cup and the Official World Golf Rankings.

Determination of the optimal sample size for a clinical trial accounting for the population size.

The problem of choosing a sample size for a clinical trial is a very common one. In some settings, such as rare diseases or other small populations, the large sample sizes usually associated with the standard frequentist approach may be infeasible, suggesting that the sample size chosen should reflect the size of the population under consideration. Incorporation of the population size is possible in a decision‐theoretic approach either explicitly by assuming that the population size is fixed and known, or implicitly through geometric discounting of the gain from future patients reflecting the expected population size. This paper develops such approaches. Building on previous work, an asymptotic expression is derived for the sample size for single and two‐arm clinical trials in the general case of a clinical trial with a primary endpoint with a distribution of one parameter exponential family form that optimizes a utility function that quantifies the cost and gain per patient as a continuous function of this parameter. It is shown that as the size of the population, N, or expected size, N∗ in the case of geometric discounting, becomes large, the optimal trial size is O(N1/2) or O(N∗1/2). The sample size obtained from the asymptotic expression is also compared with the exact optimal sample size in examples with responses with Bernoulli and Poisson distributions, showing that the asymptotic approximations can also be reasonable in relatively small sample sizes.

Investment Policy for Time-Inconsistent Discounters

This paper explores how a principal with time-inconsistent preferences invests optimally in technology or capital. If the current principal prefers her future self to save more, she can increase current investments complementary to future savings and decrease investments in the strategic substitutes, for example. To characterize the principal’s choices they are compared to a market equilibrium where the investors are private agents. Each investing agent applies the same discount factors as do the principal and he obtains full property rights to the future returns. With geometric discounting, there would be no need to regulate (subsidize/tax) these agents. With time-inconsistent preferences, however, the current principal benefits from subsidizing investments in capital (complementary to future savings) and tax investments in substitute capital such as brown technology and even adaptation technology. The paper can thus compare policies for different types of investments at the same level in the production hierarchy, but investments at different levels are also compared. With quasi-hyperbolic discounting, the optimal subsidy is unrelated to this level. With discount rates that are strictly decreasing in relative time, however, upstream investments (needed for downstream investments) will optimally be subsidized at a higher rate. When applied to environmental policy, the paper provides a new rationale for subsidizing green (and taxing brown) technology unrelated to the traditional motivation emphasizing public good aspects.

A 'Pencil Sharpening' Algorithm for Two Player Stochastic Games with Perfect Monitoring

We study the subgame perfect equilibria of two player stochastic games with perfect monitoring and geometric discounting. A novel algorithm is developed for calculating the discounted payoffs that can be attained in equilibrium. This algorithm generates a sequence of tuples of payoffs vectors, one payoff for each state, that move around the equilibrium payoff sets in a clockwise manner. The trajectory of these pivot payoffs asymptotically traces the boundary of the equilibrium payoff correspondence. We also provide an implementation of our algorithm, and preliminary simulations indicate that it is more efficient than existing methods. The theoretical results that underlie the algorithm also yield a bound on the number of extremal equilibrium payoffs.

Geometric Discounting in Discrete, Infinite-Horizon Choice Problems

The rate of time preference is traditionally defined as the marginal rate of substitution between current and future consumption. This definition is not applicable when outcomes are indivisible. Such is the case in all discrete-choice dynamic problems which arise, for example, in modeling housing or occupational decisions. Assuming an infinite horizon and a standard time-additive utility representation, this note shows that the discount factor can be uniquely recovered from the underlying preference order, provided that the decision-maker is sufficiently patient. Under the same conditions, the utility index over outcomes is cardinally unique. Finally, an algorithm for approximating the discount factor is provided which, at each stage, uses only finite-horizon data.

Measurements of risk in fisheries management

An important goal in many fisheries management problems is perceived to be minimisation of risk. This paper examines the problem of measuring risk by means of meaningful attributes or surrogate measures, for use in multiple criteria decision support systems. It is found that exponential utility functions, which are associated in this context with geometric discounting of the future, give a poor fit in many cases relative to power functions. The implication is that conventional mean-variance measures of risk may be less appropriate than cumulative probability measures.

Asymptotic Behavior of a Stochastic Discount Rate

The mean discount rate for a simple stochastic model behaves asymptotically roughly like $1/\sqrt{n}$ in contrast to the usual geometric discounting in a deterministic model.

Investment Policy for Time-Inconsistent Discounters

This paper explores how a principal with time-inconsistent preferences invests optimally in technology or capital. If the current principal prefers her future self to save more, she can increase current investments complementary to future savings and decrease investments in the strategic substitutes, for example. To characterize the principal’s choices they are compared to a market equilibrium where the investors are private agents. Each investing agent applies the same discount factors as do the principal and he obtains full property rights to the future returns. With geometric discounting, there would be no need to regulate (subsidize/tax) these agents. With time-inconsistent preferences, however, the current principal benefits from subsidizing investments in “green” capital (complementary to future savings) and tax investments in substitute capital such as “brown” technology and even adaptation technology. The paper can thus compare policies for different types of investments at the same level in the production hierarchy, but investments at different levels are also compared. With quasi-hyperbolic discounting, the optimal subsidy is unrelated to this level. With discount rates that are strictly decreasing in relative time, however, upstream investments (needed for downstream investments) will optimally be subsidized at a higher rate. When applied to environmental policy, the paper provides a new rationale for subsidizing green (and taxing brown) technology unrelated to the traditional motivation emphasizing public good aspects.

HOW WE VALUE THE FUTURE AFFECTS OUR DESIRE TO LEARN

Active adaptive management is increasingly advocated in natural resource management and conservation biology. Active adaptive management looks at the benefit of employing strategies that may be suboptimal in the near term but which may provide additional information that will facilitate better management in future years. However, when comparing management policies it is traditional to weigh future rewards geometrically (at a constant discount rate) which results in far-distant rewards making a negligible contribution to the total benefit. Under such a discounting scheme active adaptive management is rarely of much benefit, especially if learning is slow. A growing number of authors advocate the use of alternative forms of discounting when evaluating optimal strategies for long-term decisions which have a social component. We consider a theoretical harvested population for which the recovery rate from an unharvestably small population size is unknown and look at the effects on the benefit of experimental management when three different forms of discounting are employed. Under geometric discounting, with a discount rate of 5% per annum, managing to learn actively had little benefit. This study demonstrates that discount functions which weigh future rewards more heavily result in more conservative harvesting strategies, but do not necessarily encourage active learning. Furthermore, the optimal management strategy is not equivalent to employing geometric discounting at a lower rate. If alternative discount functions are made mandatory in calculating optimal management strategies for environmental management then this will affect the structure of optimal management regimes and change when and how much we are willing to invest in learning.

Generalized quasi-geometric discounting

This paper derives the ‘generalized Euler equation’ for an agent with multi-period deviations from geometric discounting. The functional equation that describes optimal consumption-savings decisions involves manipulation of future selves and indirect manipulation of more distant selves through intervening selves.

DISCOUNTING IN THE HISTORIC PRISONER'S DILEMMA

The standard spatial formulation of the Prisoner's Dilemma (PD) is ahistoric (memoryless): only results generated in the last round are taken into account to decide the next choice. In the standard historic model, all results coming from previous rounds are considered without discounting. The effect of geometric discounting in the historic spatial formulation of PD is assessed in this work. The fate of a single isolated defector surrounded by cooperators and that of some groups of cooperators surrounded by defectors are studied, varying both the discount factor (α) and the temptation (b, the parameter which characterizes the PD game). The effect of noise and stochastic rules are also assessed in the first scenario. The most important finding is that there are some range of parameter values (b, α) for which the evolution dynamics becomes unexpected.

A Note on Dirichlet One-Armed Bandits

One of the two independent stochastic processes (or ‘arms’) is selected and observed sequentially at each of n(≤ ∝) stages. Arm 1 yields observations identically distributed with unknown probability measure P with a Dirichlet process prior whereas observations from arm 2 have known probability measure Q. Future observations are discounted and at stage m, the payoff is a m(≥0) times the observation Z m at that stage. The objective is to maximize the total expected payoff. Clayton and Berry (1985) consider this problem when a m equals 1 for m ≤ n and 0 for m > n(< ∝) In this paper, the Clayton and Berry (1985) results are extended to the case of regular discount sequences of horizon n, which may also be infinite. The results are illustrated with numerical examples. In case of geometric discounting, the results apply to a bandit with many independent unknown Dirichlet arms.

Bayesian bandits in clinical trials

Suppose two treatments with binary responses are available for patients with some disease. Sequential allocation rules based on the theory of Bayesian bandit processes are examined. The patient horizon is assumed to be random and the objective is to maximize the total expected number of successes. This problem is equivalent to the classical two-armed bandit problem if we assume that the only information acquired during the trial about the patient horizon is whether or not it has so far been reached. When one treatment has a known success rate, the optimal allocation rule may sometimes be expressed in terms of dynamic allocation indices. The calculation of the inclices is discussed, and in particular error bounds for the accuracy of the calculation are given in terms of the starting point of the backward induction process. When both treatments have unknown success rates the use of dynamic allocation indices with geometric discounting is suggested. Simulation results indicate that this rule works well even when the distribution of the number of the patients is not geometrical, and that the choice of the discount factor is also not crucial.