Decision Theory Problem Research Articles

Optimality of greedy policy for a class of standard reward function of restless multi-armed bandit problem

In this paper,we consider the restless bandit problem, which is one of the most well-studied generalizations of the celebrated stochastic multi-armed bandit problem in decision theory. However, it is known be PSPACE-Hard to approximate to any non-trivial factor. Thus the optimality is very difficult to obtain due to its high complexity. A natural method is to obtain the greedy policy considering its stability and simplicity. However, the greedy policy will result in the optimality loss for its intrinsic myopic behavior generally. In this paper, by analyzing one class of so-called standard reward function, we establish the closed-form condition about the discounted factor \beta such that the optimality of the greedy policy is guaranteed under the discounted expected reward criterion, especially, the condition \beta = 1 indicating the optimality of the greedy policy under the average accumulative reward criterion. Thus, the standard form of reward function can easily be used to judge the optimality of the greedy policy without any complicated calculation. Some examples in cognitive radio networks are presented to verify the effectiveness of the mathematical result in judging the optimality of the greedy policy.

In This Issue

In This Issue

Approximation algorithms for restless bandit problems

The restless bandit problem is one of the most well-studied generalizations of the celebrated stochastic multi-armed bandit (MAB) problem in decision theory. In its ultimate generality, the restless bandit problem is known to be PSPACE-Hard to approximate to any nontrivial factor, and little progress has been made on this problem despite its significance in modeling activity allocation under uncertainty. In this article, we consider the Feedback MAB problem, where the reward obtained by playing each of n independent arms varies according to an underlying on/off Markov process whose exact state is only revealed when the arm is played. The goal is to design a policy for playing the arms in order to maximize the infinite horizon time average expected reward. This problem is also an instance of a Partially Observable Markov Decision Process (POMDP), and is widely studied in wireless scheduling and unmanned aerial vehicle (UAV) routing. Unlike the stochastic MAB problem, the Feedback MAB problem does not admit to greedy index-based optimal policies. We develop a novel duality-based algorithmic technique that yields a surprisingly simple and intuitive (2+ϵ)-approximate greedy policy to this problem. We show that both in terms of approximation factor and computational efficiency, our policy is closely related to the Whittle index , which is widely used for its simplicity and efficiency of computation. Subsequently we define a multi-state generalization, that we term Monotone bandits, which remains subclass of the restless bandit problem. We show that our policy remains a 2-approximation in this setting, and further, our technique is robust enough to incorporate various side-constraints such as blocking plays, switching costs, and even models where determining the state of an arm is a separate operation from playing it. Our technique is also of independent interest for other restless bandit problems, and we provide an example in nonpreemptive machine replenishment. Interestingly, in this case, our policy provides a constant factor guarantee, whereas the Whittle index is provably polynomially worse. By presenting the first O(1) approximations for nontrivial instances of restless bandits as well as of POMDPs, our work initiates the study of approximation algorithms in both these contexts.

Information Quantification Application to Management with Fuzzy Entropy and Similarity Measure

Verification of efficiency in data management fuzzy entropy and similarity measure were discussed and verified by applying reliable data selection problem and numerical data similarity evaluation. In order to calculate the certainty or uncertainty fuzzy entropy and similarity measure are designed and proved. Designed fuzzy entropy and similarity are considered as dissimilarity measure and similarity measure, and the relation between two measures are explained through graphical illustration. Obtained measures are useful to the application of decision theory and mutual information analysis problem. Extension of data quantification results based on the proposed measures are applicable to the decision making and fuzzy game theory.

A Bayesian approach for incorporating economic factors in sample size design for clinical trials of individual drugs and portfolios of drugs

Classical approaches to clinical trial design ignore economic factors that determine economic viability of a new drug. We address the choice of sample size in Phase III trials as a decision theory problem using a hybrid approach that takes a Bayesian view from the perspective of a drug company and a classical Neyman-Pearson view from the perspective of regulatory authorities. We incorporate relevant economic factors in the analysis to determine the optimal sample size to maximize the expected profit for the company. We extend the analysis to account for risk by using a 'satisficing' objective function that maximizes the chance of meeting a management-specified target level of profit. We extend the models for single drugs to a portfolio of clinical trials and optimize the sample sizes to maximize the expected profit subject to budget constraints. Further, we address the portfolio risk and optimize the sample sizes to maximize the probability of achieving a given target of expected profit.

Predicting the Event and Time Horizon of Bankruptcy Using Financial Ratios and the Maturity Schedule of Long-Term Debt

This study investigates the multi-period prediction of firm bankruptcy as a multi-alternative problem of Statistical Decision Theory. This approach enables a simultaneous assessment to be made of the prediction of bankruptcy and the time horizon at which the bankruptcy could occur. To illustrate the approach, using U.S. bankruptcy data, a comparative statistical analysis of various financial variables is undertaken to identify four relatively independent financial ratios that have the potential for multi-period bankruptcy forecasting. These ratios characterize the quantity and quality of debt, as well as the firm's ability to repay the debt. The study also investigates a new type of predictive information - the maturity schedule of a firm's long-term debt. Bayesian-type forecasting rules are developed that jointly use the financial ratios and maturity schedule factors. The rules noticeably enhance bankruptcy prediction compared with the familiar one-period (two-alternative) Z-score rules of Altman (1968) for bankruptcy within the first one, two or three years. Predictive factors derived from schedule information additionally enhance bankruptcy prediction at distant time horizons.

In Memory of Richard Jeffrey: Some Reminiscences and Some Reflections on The Logic of Decision

This paper is partly a tribute to Richard Jeffrey, partly a reflection on some of his writings, The Logic of Decision in particular. I begin with a brief biography and some fond reminiscences of Dick. I turn to some of the key tenets of his version of Bayesianism. All of these tenets are deployed in my discussion of his response to the St. Petersburg paradox, a notorious problem for decision theory that involves a game of infinite expectation. Prompted by that paradox, I conclude with some suggestions of avenues for future research.

Evidential Decision Theory and Medical Newcomb Problems

Huw Price ([1991]) has offered evidential decision theorists a defence against the charge that they make unintuitive recommendations for cases like Newcomb's Problem. He says that when conditional probabilities are assessed from the agent's point of view, evidential decision theory makes the same recommendation as intuition. I argue that calculating the probabilities in Price's way leads to no recommendation. It condemns the agent to perpetual oscillation between different options. 1. Price's Argument2. Instability3. Objections4. Conclusion

The Classical and Maximin Versions of the Two-Envelope Paradox

The Two-Envelope Paradox is classically presented as a problem in decision theory that turns on the use of probabilities in calculating expected utilities. I formulate a Maximin Version of the paradox, one that is decision-theoretic but omits considerations of probability. I investigate the source of the error in this new argument, and apply the insights thereby gained to the analysis of the classical version.

The Law of Large Demand for Information

An unresolved problem in Bayesian decision theory is how to value and price information. This paper resolves both problems assuming inexpensive information. Building on Large Deviation Theory, we produce a generically complete asymptotic order on samples of i.i.d. signals in finite-state, finite-action models. Computing the marginal value of an additional signal, we find it is eventually exponentially falling in quantity, and higher for lower quality signals. We provide a precise formula for the information demand, valid at low prices: asymptotically a constant times the log price, and falling in the signal quality for a given price.

A decision theory approach to fitness for purpose in analytical measurement.

The choice of an analytical procedure and the determination of an appropriate sampling strategy are here treated as a decision theory problem in which sampling and analytical costs are balanced against possible end-user losses due to measurement error. Measurement error is taken here to include both sampling and analytical variances, but systematic errors are not considered. The theory is developed in detail for the case exemplified by a simple accept or reject decision following an analytical measurement on a batch of material, and useful approximate formulae are given for this case. Two worked examples are given, one involving a batch production process and the other a land reclamation site.

Knowledge in action: logical foundations for specifying and implementing dynamical systems

Modeling and implementing dynamical systems is a central problem in artificial intelligence, robotics, software agents, simulation, decision and control theory, and many other disciplines. In recent years, a new approach to representing such systems, grounded in mathematical logic, has been developed within the AI knowledge-representation community. This book presents a comprehensive treatment of these ideas, basing its theoretical and implementation foundations on the situation calculus, a dialect of first-order logic. Within this framework, it develops many features of dynamical systems modeling, including time, processes, concurrency, exogenous events, reactivity, sensing and knowledge, probabilistic uncertainty, and decision theory. It also describes and implements a new family of high-level programming languages suitable for writing control programs for dynamical systems. Finally, it includes situation calculus specifications for a wide range of examples drawn from cognitive robotics, planning, simulation, databases, and decision theory, together with all the implementation code for these examples. This code is available on the book's Web site.

Advice on West Coast Rockfish Harvest Rates from Bayesian Meta-Analysis of Stock−Recruit Relationships

Abstract Over the past two decades, populations of rockfish Sebastes spp. off the U.S. West Coast have declined sharply, leading to heightened concern about the sustainability of current harvest policies for these populations. In this paper, I develop a hierarchical Bayesian model to jointly estimate the stock−recruit relationships of rockfish stocks in the northeastern Pacific Ocean. Stock−recruit curves for individual stocks are linked using a prior distribution for the “steepness” parameter of the Beverton–Holt stock−recruit curve, defined as the expected recruitment at 20% of unfished biomass relative to unfished recruitment. The choice of a spawning biomass per recruit (SPR) harvest rate is considered a problem in decision theory, in which different options are evaluated in the presence of uncertainty in the stock−recruit relationship. Markov chain Monte Carlo sampling is used to obtain the marginal distributions of variables of interest to management, such as the yield at a given SPR rate. A wide ra...

Discussion of The Interrelationship between Estimated Tax Payments and Taxpayer Compliance

CONTRIBUTION Feltham and Paquette (2002) (hereafter the paper) examine the effect of a taxpayer’s prepayment position on the taxpayer’s reporting strategy and the government’s auditing strategy in a tax compliance game. Chang and Schultz (1990) find a positive association between noncompliance and an underpayment position. Dusenbury (1994) asserts, ‘‘The prepayment level determines the additional payment due or the refund available, but logically should not influence the reported tax liability.’’ Dusenbury (1994) goes on to argue that the link between noncompliance and prepayment position can be explained using prospect theory. This paper offers a competing explanation. The authors argue that because a taxpayer’s prepayment position is endogenously determined, the prepayment itself provides information regarding the taxpayer’s true tax liability to the tax authority, which in turn affects the strategic interaction in the tax compliance game. It shows that behavior that appears to be illogical when tax compliance is framed as a single-person decision theory problem is optimal if one assumes that tax compliance is a multiplayer, strategic game.

The Law of Large Demand for Information

An unresolved problem in Bayesian decision theory is how to value and price information. This paper resolves both problems by assuming inexpensive information. Building on Large Deviation Theory, we produce a generically complete asymptotic order on samples of i.i.d. signals in finite-state, finite-action Bayesian models. We then extend this order from the 'total' to the 'marginal value of information' --- i.e. the value of an additional signal. We show that it is eventually exponentially falling in quantity, and is higher for lower quality signals. We exploit this result to provide a precise formula for the information demand, valid at low prices: q=[log p+(1/2)log (-log p)]C + D where the constants C and D depend on the underlying signal, and D also depends on preferences. So demand is logarithmic, falling in the price, and falling in the signal quality for a given price.

Decision Making: A Golden Rule

Click to increase image sizeClick to decrease image size Additional informationNotes on contributorsDimitris A. SardelisDIMITRIS A. SARDELIS received his Ph.D from Imperial College where he worked on quantum field theories under the supervision of Abdus Salam. He has been teaching at the American College of Greece since 1991. He is interested in the foundations of geometry, stochastic processes, and higher arithmetic.Theodoros M. ValahasTHEODOROS M. VALAHAS received his B.S. from Aristoteles University of Thessaloniki, Greece and his Ph.D. from Massachusetts Institute of Technology. After spending some years in industry he returned to the academic community. He co-ordinates the math and science group of the American College of Greece. His interests are classical fundamental problems in kinematics, dynamics, number theory, probabilities, and statistical decision theory.

Comment: Recruitment paradigms for fish stocks

Over the last 20 years, there has been a substantial shift in the importance most fisheries scientists ascribe to recruitment overfishing; it was widely assumed (Gulland 1983) that most marine fishes could not be recruitment overfished, and the greatest emphasis in fisheries management was on yield per recruit. However, a large number of fish stock collapses that showed reduced recruitment at low spawning stocks led to a general reappraisal, so that Hilborn and Walters (1992) stated that ithe most important and generally most difficult problem in biological assessment of fisheries is the relationship between stock and recruitment.i Myers et al. (1994) used a large data base on many marine fishes to confirm that most stocks showed reduced recruitment at low spawning stock. Gilbert (1997) offers a revisionist interpretation of the Myers et al. (1994) data and appears to threaten to turn back the clock to a time when fishery managers believed they did not have to worry about maintaining spawning stock biomass. Is this truly a backwards step? I have to answer no; Gilbertis paper is a positive step in our ongoing scientific investigation of stock and recruitment because Gilbert highlights the chicken-or-the-egg dilemma in most spawner recruit analysis. Did low spawning stock lead to low recruitment, or did low recruitment lead to low spawning stock? The growing recognition of the importance of environmental changes in fish recruitment (Francis and Hare 1994) emphasizes the difficulty in interpreting the causes of declines in recruitment, and we are essentially revisiting the ThompsonnBurkenroad debate of 50 years ago (Thompson 1950; Burkenroad 1948; Skud 1975) with more data in hand. Gilbertis paper challenges those of us who have advocated maintaining spawning stock biomass as a prudent form of management to tighten up our arguments. We need to develop statistical analyses that determine the relative probability of competing hypotheses about the cause of recruitment declines. The analysis of Myers et al. (1994) did not consider an environmental change hypothesis; Gilbertis analysis is of limited utility because it relies on acceptance and rejection of null hypotheses, a statistical format that is of no utility for decision making. While Gilbert makes us face the alternative hypothesis that recruitment declined because of environmental conditions, he stops short of the next step and asking what is the prudent management action given the uncertainty regarding the cause of the recruitment decline. If we accept that both a recruitment overfishing hypothesis and an environmental change hypothesis are consistent with an individual stockis history, then what should a manager do if faced with a stock showing both reduced recruitment and reduced spawning stock: continue fishing on the assumption that the environment changed, or reduce fishing in an effort to rebuild spawning stock size? This is a problem in statistical decision theory that requires understanding the probabilities of competing hypotheses and a manageris analysis of risk. Clearly, the costs of not reducing fishing pressure at low abundance, if the true cause is recruitment overfishing, are grave: the stock will simply fail to recover until fishing pressure is reduced. The cost of reducing fishing pressure if the cause of the decline is environmental is simply short-term lost catch. These options are summarized in Table 1: the prudent form of management would seem to be to act as if recruitment overfishing were the cause. However, the choice of management option depends upon the probabilities assigned to the alternative hypotheses and the risk preference of the decision maker. The challenge to fisheries science is now to develop methods to determine the probabilities of alternative hypotheses and perform the risk analysis.

Aspects of Uncertainty: A Tribute to D. V. Lindley.

Dennis Lindley: The First 70 Years The Operational Bayesian Approach Pivotal Inference Illustrated on the Darwin Maize Data Bayes Theory in Latent Variable Modelling Bayesian Sampling Schemes for Auditors Optimal Prediction with Hierarchical Models: Bayesian Clustering Inference for a Covariance Matrix The Role of Statistical Theory in Decision Aiding The Signature as a Covariate in Reliability and Biometry Residual Analysis and Outliers in Hierarchical Models The Island Problem: Coherent Use of Identification Evidence Utility: Probability's Younger Twin? Bayesian Hierarchical Analysis for Exponential Families Via Markov Chains Revising Exchangeable Beliefs: Subjectivist Foundations for the Inductive Argument On Steinian Shrinkage Estimators The Finite/Infinite Problem and Formalism in Probability and Statistics Bayesian Decision Theory and the Legal Structure Experimental Design From a Subjective Utilitarian Viewpoint The Bayesian Analysis of Categorical Data: A Selective Review Conflicting Information and a Class of Bivariate Heavy-Tailed Distributions Application of Lindley Information to the Design of Clinical Experiments On Two Classic Theorems Involving the Characteristic Function Hierarchical Priors and Mixture Models with Applications in Regression and Density Estimation.

Sequential Choice Under Ambiguity: Intuitive Solutions to the Armed-Bandit Problem

The process by which individuals learn from feedback when making recurrent choices among ambiguous alternatives is explored. We describe an experiment in which subjects solve a variant of the classic armed-bandit problem of dynamic decision theory, set in the context of airline choice. Subjects are asked to make repeated choices between two hypothetical airlines, one having an on-time departure probability which is known a priori, and the other has an ambiguous probability whose true value can only be discovered by making sample trips on the airline. Subjects attempt to make choices in such a way as to maximize the total number of one-time departures over a fixed planning horizon. We examine the extent to which actual choice patterns over time are consistent with those which would be made by a decision maker acting as an optimal Bernoulli sampler. The data offer support for a number of expected—and some unexpected—departures from optimality, including a tendency to underexperiment with promising options and overexperiment with unpromising options, and a tendency to increasingly switch between airlines as the average base rate of departures decreases. Implications of the work for the descriptive validity of normative dynamic decision models is explored, as well as for the generalizability of previous findings about choice under ambiguity to dynamic settings.

A computational theory of decision networks

This paper is about how to represent and solve decision problems in Bayesian decision theory (e.g. [6]). A general representation, called decision networks, is proposed based on influence diagrams [10]. This new representation incorporates the idea, from Markov decision processes (e.g. [5]), that a decision may be conditionally independent of certain pieces of available information. It also allows multiple cooperative agents and facilitates the exploitation of separability in the utility function. Decision networks inherit the advantages of both influence diagrams and Markov decision processes, which makes them a better representation framework for decision analysis, planning under uncertainty, and medical diagnosis and treatment. Influence diagrams are stepwise-solvable, in the sense that they can be evaluated by considering one decision at a time. However, the evaluation of a decision network requires, in general, simultaneous consideration of all the decisions. The theme of this paper is to seek the weakest condition under which decision networks are stepwise-solvable and to seek the best algorithms for evaluating stepwise-solvable decision networks. A concept of decomposability is introduced for decision networks, and it is shown that when a decision network is decomposable, a divide-and-conquer strategy can be utilized to aid its evaluation. In particular, when a decision network is stepwise-decomposable, it can be evaluated not only by considering one decision at a time, but also by considering one portion of the network at a time.