Contraction Mappings in the Theory Underlying Dynamic Programming

Abstract

Next article Contraction Mappings in the Theory Underlying Dynamic ProgrammingEric V. DenardoEric V. Denardohttps://doi.org/10.1137/1009030PDFBibTexSections ToolsAdd to favoritesExport CitationTrack CitationsEmail SectionsAbout[1] Richard Bellman, Dynamic programming, Princeton Univeristy Press, Princeton, N. J., 1957xxv+342 MR0090477 Google Scholar[2] David Blackwell, Discrete dynamic programming, Ann. Math. Statist., 33 (1962), 719–726 MR0149965 0133.12906 CrossrefISIGoogle Scholar[3] David Blackwell, Discounted dynamic programming, Ann. Math. Statist., 36 (1965), 226–235 MR0173536 0133.42805 CrossrefGoogle Scholar[4] A. Charnes and , R. G. Schroeder, On some tactical antisubmarine games, Systems Research Memorandum No. 131, The Technological Institute, Northwestern University, Evanston, Illinois, 1965 Google Scholar[5] E. V. Denardo, Masters Thesis, Sequential decision processes, Doctoral thesis, Northwestern University, Evanston, Illinois, 1965 Google Scholar[6] F. D'Epenoux, Sur un problème de production de stockage dans l'aleatoire, Rev. Française Recherche Operationelle, 14 (1960), 3–16 Google Scholar[7] Cyrus Derman, On sequential decisions and Markov chains, Management Sci., 9 (1962/1963), 16–24 MR0169685 0995.90621 CrossrefISIGoogle Scholar[8] Cyrus Derman and , Morton Klein, Some remarks on finite horizon Markovian decision models, Operations Res., 13 (1965), 272–278 MR0175636 0137.13901 CrossrefISIGoogle Scholar[9] J. H. Eaton and , L. A. Zadeh, Optimal pursuit strategies in discrete-state probabilistic systems, Trans. ASME Ser. D. J. Basic Engrg., 84 (1962), 23–29 MR0153510 CrossrefGoogle Scholar[10] L. È. Èlsgol'c, Qualitative methods in mathematical analysis, Translations of Mathematical Monographs, Vol. 12, American Mathematical Society, Providence, R.I., 1964vii+250, Trans. by A. A. Brown and J. M. Danskin MR0170048 0133.37102 CrossrefGoogle Scholar[11] B. Fox, Age replacement with discounting, Operations Res., to appear Google Scholar[12] Ronald A. Howard, Dynamic programming and Markov processes, The Technology Press of M.I.T., Cambridge, Mass., 1960viii+136 MR0118514 0091.16001 Google Scholar[13A] William S. Jewell, Markov-renewal programming. I. Formulation, finite return models, Operations Res., 11 (1963), 938–948 MR0163374 0126.15905 CrossrefISIGoogle Scholar[13B] William S. Jewell, Markov-renewal programming. II. Infinite return models, example, Operations Res., 11 (1963), 949–971 MR0163375 0126.15905 CrossrefISIGoogle Scholar[14] Samuel Karlin, The structure of dynamic programming models, Naval Res. Logist. Quart., 2 (1955), 285–294 (1956) MR0077850 CrossrefGoogle Scholar[15] L. G. Mitten, Composition principles for synthesis of optimal multistage processes, Operations Res., 12 (1964), 610–619 MR0180374 0127.36502 CrossrefISIGoogle Scholar[16] L. S. Shapley, Stochastic games, Proc. Nat. Acad. Sci. U. S. A., 39 (1953), 1095–1100 MR0061807 0051.35805 CrossrefISIGoogle Scholar[17] Lars Erik Zachrisson, M. Dresher, , L. S. Shapley and , A. W. Tucker, Markov gamesAdvances in game theory, Princeton Univ. Press, Princeton, N.J., 1964, 211–253 MR0170729 Google Scholar Next article FiguresRelatedReferencesCited byDetails Qauxi: Cooperative multi-agent reinforcement learning with knowledge transferred from auxiliary taskNeurocomputing, Vol. 504 Cross Ref Data-driven optimal control with a relaxed linear programAutomatica, Vol. 136 Cross Ref Markov Decision Processes with Discounted Costs: Improved Successive Over-Relaxation Method24 March 2022 Cross Ref Markov Decision Processes with Discounted Rewards: Improved Successive Over-Relaxation Method12 January 2022 Cross Ref Robust Speed Control of Ultrasonic Motors Based on Deep Reinforcement Learning of a Lyapunov FunctionIEEE Access, Vol. 10 Cross Ref Data-Driven Optimal Control of Affine Systems: A Linear Programming PerspectiveIEEE Control Systems Letters, Vol. 6 Cross Ref Stochastic Dynamic Programming with Non-linear Discounting23 December 2020 | Applied Mathematics & Optimization, Vol. 84, No. 3 Cross Ref On Constructive Extractability of Measurable Selectors of Set-Valued MapsIEEE Transactions on Automatic Control, Vol. 66, No. 8 Cross Ref On the convergence of reinforcement learning with Monte Carlo Exploring StartsAutomatica, Vol. 129 Cross Ref Successive Over-Relaxation ${Q}$ -LearningIEEE Control Systems Letters, Vol. 4, No. 1 Cross Ref Affine Monotonic and Risk-Sensitive Models in Dynamic ProgrammingIEEE Transactions on Automatic Control, Vol. 64, No. 8 Cross Ref Optimal forest management under financial risk aversion with discounted Markov decision process modelsCanadian Journal of Forest Research, Vol. 49, No. 7 Cross Ref Optimizing over pure stationary equilibria in consensus stopping games2 November 2018 | Mathematical Programming Computation, Vol. 11, No. 2 Cross Ref Robust shortest path planning and semicontractive dynamic programming8 August 2016 | Naval Research Logistics (NRL), Vol. 66, No. 1 Cross Ref On the reduction of total‐cost and average‐cost MDPs to discounted MDPs25 May 2017 | Naval Research Logistics (NRL), Vol. 66, No. 1 Cross Ref Optimal Liquidation in a Level-I Limit Order Book for Large-Tick StocksAntoine Jacquier and Hao Liu5 July 2018 | SIAM Journal on Financial Mathematics, Vol. 9, No. 3AbstractPDF (845 KB)An Average Polynomial Algorithm for Solving Antagonistic Games on Graphs2 March 2018 | Journal of Computer and Systems Sciences International, Vol. 57, No. 1 Cross Ref Dynamic Programming15 February 2018 Cross Ref Dynamic Programming and Markov Decision Processes15 February 2018 Cross Ref Long-Term Values in Markov Decision Processes, (Co)Algebraically20 September 2018 Cross Ref IDENTIFICATION OF DISCRETE CHOICE DYNAMIC PROGRAMMING MODELS WITH NONPARAMETRIC DISTRIBUTION OF UNOBSERVABLES21 March 2016 | Econometric Theory, Vol. 33, No. 3 Cross Ref Dynamic Programming, Numerical15 February 2017 Cross Ref Regular Policies in Abstract Dynamic ProgrammingDimitri P. Bertsekas17 August 2017 | SIAM Journal on Optimization, Vol. 27, No. 3AbstractPDF (510 KB)Optimal Liquidation in a Level-I Limit Order Book for Large Tick StocksSSRN Electronic Journal Cross Ref Easy Affine Markov Decision Processes: TheorySSRN Electronic Journal Cross Ref Optimality of the fastest available server policy1 October 2016 | Queueing Systems, Vol. 84, No. 3-4 Cross Ref A global shooting algorithm for the facility location and capacity acquisition problem on a line with dense demandComputers & Operations Research, Vol. 71 Cross Ref Optimality of the Fastest Available Server PolicySSRN Electronic Journal Cross Ref Approximation of two-person zero-sum continuous-time Markov games with average payoff criterionOperations Research Letters, Vol. 43, No. 1 Cross Ref On variable discounting in dynamic programming: applications to resource extraction and other economic models9 August 2011 | Annals of Operations Research, Vol. 220, No. 1 Cross Ref Valuing Customer Portfolios with Endogenous Mass and Direct Marketing Interventions Using a Stochastic Dynamic Programming DecompositionMarketing Science, Vol. 33, No. 5 Cross Ref Divergence Behaviour of the Successive Geometric Mean Method of Pairwise Comparison Matrix Generation for a Multiple Stage, Multiple Objective Optimization Problem20 December 2013 | Journal of Multi-Criteria Decision Analysis, Vol. 21, No. 3-4 Cross Ref Solving multichain stochastic games with mean payoff by policy iteration Cross Ref Discounting axioms imply risk neutrality8 February 2012 | Annals of Operations Research, Vol. 208, No. 1 Cross Ref (Approximate) iterated successive approximations algorithm for sequential decision processes8 February 2012 | Annals of Operations Research, Vol. 208, No. 1 Cross Ref The multi-armed bandit, with constraints13 November 2012 | Annals of Operations Research, Vol. 208, No. 1 Cross Ref Persistently Optimal Policies in Stochastic Dynamic Programming with Generalized DiscountingMathematics of Operations Research, Vol. 38, No. 1 Cross Ref A Dynamic Game of Reputation and Economic Performances in Nondemocratic Regimes15 June 2012 | Dynamic Games and Applications, Vol. 2, No. 4 Cross Ref Stochastic mutual induction computing in Het-CoMP empowered cellular networks Cross Ref SWITCHING AND SEQUENCING AVAILABLE THERAPIES SO AS TO MAXIMIZE A PATIENT'S EXPECTED TOTAL LIFETIME16 May 2012 | International Journal of Biomathematics, Vol. 05, No. 04 Cross Ref Multigrid methods for two-player zero-sum stochastic games17 January 2012 | Numerical Linear Algebra with Applications, Vol. 19, No. 2 Cross Ref Cooperative Access Class Barring for Machine-to-Machine CommunicationsIEEE Transactions on Wireless Communications, Vol. 11, No. 1 Cross Ref PARTIALLY OBSERVABLE MARKOV DECISION PROCESSES AND PERIODIC POLICIES WITH APPLICATIONS30 April 2012 | International Journal of Information Technology & Decision Making, Vol. 10, No. 06 Cross Ref Approximate policy iteration: a survey and some new methods19 July 2011 | Journal of Control Theory and Applications, Vol. 9, No. 3 Cross Ref Total Expected Discounted Reward MDPS: Existence of Optimal Policies15 February 2011 Cross Ref Stationary policies with Markov partition propertyJournal of Statistics and Management Systems, Vol. 13, No. 6 Cross Ref Myopic Solutions of Homogeneous Sequential Decision ProcessesOperations Research, Vol. 58, No. 4-part-2 Cross Ref Partially observable Markov decision model for the treatment of early Prostate Cancer13 October 2010 | OPSEARCH, Vol. 47, No. 2 Cross Ref Computable Markov-perfect industry dynamicsThe RAND Journal of Economics, Vol. 41, No. 2 Cross Ref Dynamic Allocation of Scarce Resources Under Supply UncertaintySSRN Electronic Journal Cross Ref Economically Efficient Constitutional GovernanceSSRN Electronic Journal Cross Ref Applications of Metric Coinduction16 September 2009 | Logical Methods in Computer Science, Vol. 5, No. 3 Cross Ref Probabilistic models for optimizing patients survival ratesJournal of Interdisciplinary Mathematics, Vol. 11, No. 5 Cross Ref A multi-period TSP with stochastic regular and urgent demandsEuropean Journal of Operational Research, Vol. 185, No. 1 Cross Ref Four Canadian Contributions to Stochastic Modeling18 January 2017 | INFOR: Information Systems and Operational Research, Vol. 46, No. 1 Cross Ref Dynamic Programming5 December 2016 Cross Ref Financial intermediary's choice of borrowingApplied Economics, Vol. 40, No. 2 Cross Ref Optimal prepayment behaviourApplied Economics Letters, Vol. 14, No. 15 Cross Ref A structured pattern matrix algorithm for multichain Markov decision processes6 February 2007 | Mathematical Methods of Operations Research, Vol. 66, No. 3 Cross Ref Incomplete markets, labor supply and capital accumulationJournal of Monetary Economics, Vol. 54, No. 8 Cross Ref VARIATIONS ON THE THEME OF CONNING IN MATHEMATICAL ECONOMICSJournal of Economic Surveys, Vol. 21, No. 3 Cross Ref Commercial loan borrower’s optimal borrowing and prepayment decisions under uncertaintyApplied Economics, Vol. 39, No. 8 Cross Ref Risk-Sensitive and Risk-Neutral Multiarmed BanditsMathematics of Operations Research, Vol. 32, No. 2 Cross Ref Computable Markov-Perfect Industry Dynamics: Existence, Purification, and MultiplicitySSRN Electronic Journal Cross Ref Semi-Markov information model for revenue management and dynamic pricing9 March 2006 | OR Spectrum, Vol. 29, No. 1 Cross Ref A Turnpike Theorem For A Risk-Sensitive Markov Decision Process with StoppingEric V. Denardo and Uriel G. Rothblum26 July 2006 | SIAM Journal on Control and Optimization, Vol. 45, No. 2AbstractPDF (189 KB)Discounting and Risk NeutralitySSRN Electronic Journal Cross Ref Myopic Solutions of Homogeneous Sequential Decision ProcessesSSRN Electronic Journal Cross Ref Limited Attention as a Bounded on RationalitySSRN Electronic Journal Cross Ref Approximation solution and suboptimality for discounted semi-markov decision problems with countable state spaceOptimization, Vol. 53, No. 4 Cross Ref Optimal threshold probability in undiscounted Markov decision processes with a target setApplied Mathematics and Computation, Vol. 149, No. 2 Cross Ref Index Policies for Stochastic Search in a Forest with an Application to R&D Project ManagementMathematics of Operations Research, Vol. 29, No. 1 Cross Ref Recursive methods in probability control Cross Ref Optimism and overconfidence in searchReview of Economic Dynamics, Vol. 7, No. 1 Cross Ref Nonclassical Brock-Mirman EconomiesSSRN Electronic Journal Cross Ref Optimal policies in continuous time inventory control models with limited supplyComputers & Mathematics with Applications, Vol. 46, No. 7 Cross Ref Existence and Uniqueness of Solutions to the Bellman Equation in the Unbounded CaseEconometrica, Vol. 71, No. 5 Cross Ref Dynamic Airline Revenue Management with Multiple Semi-Markov DemandOperations Research, Vol. 51, No. 1 Cross Ref Finite State and Action MDPS Cross Ref Dynamic Programming Cross Ref Incomplete Markets, Labor Supply and Capital AccumulationSSRN Electronic Journal Cross Ref Overconfidence in SearchSSRN Electronic Journal Cross Ref Constrained Discounted Semi-Markov Decision Processes Cross Ref Controlled Markov Chains with Utility Functions Cross Ref Total Reward Criteria Cross Ref Is There a Curse of Dimensionality for Contraction Fixed Points in the Worst Case?Econometrica, Vol. 70, No. 1 Cross Ref SET-VALUED CONTROL LAWS IN TEV-DC CONTROL PROBLEMSIFAC Proceedings Volumes, Vol. 35, No. 1 Cross Ref Dynamic economic management of soil erosion, nutrient depletion, and productivity in the north central USA1 January 2001 | Land Degradation & Development, Vol. 12, No. 4 Cross Ref On Markov Policies for Minimax Decision ProcessesJournal of Mathematical Analysis and Applications, Vol. 253, No. 1 Cross Ref Recursive method in stochastic optimization under compound criteria Cross Ref Kulatilaka '93: The Case of a Dual Fuel Boiler: A Review, Gauss Codes and Numerical ExamplesSSRN Electronic Journal Cross Ref Kulatilaka '88 as a CVP Analysis in a Real Option Framework: A Review, Gauss Codes and Numerical ExamplesSSRN Electronic Journal Cross Ref A stochastic programming approach to manufacturing flow controlIIE Transactions, Vol. 32, No. 10 Cross Ref Chapter 5 Numerical solution of dynamic economic models Cross Ref A Theory of Constitutional Standards and Civil LibertySSRN Electronic Journal Cross Ref The one-sector growth model with idiosyncratic shocks: Steady states and dynamicsJournal of Monetary Economics, Vol. 39, No. 3 Cross Ref Pansystems optimization, generalized principles of optimality, and fundamental equations of dynamic programmingKybernetes, Vol. 26, No. 3 Cross Ref Introduction Cross Ref Stochastic Inventory Models with Limited Production Capacity and Periodically Varying Parameters27 July 2009 | Probability in the Engineering and Informational Sciences, Vol. 11, No. 1 Cross Ref A Comparison of Policy Iteration Methods for Solving Continuous-State, Infinite-Horizon Markovian Decision Problems Using Random, Quasi-random, and Deterministic DiscretizationsSSRN Electronic Journal Cross Ref On the value function in constrained control of Markov chainsMathematical Methods of Operations Research, Vol. 44, No. 3 Cross Ref Models for capacity acquisition decisions considering operational costsInternational Journal of Flexible Manufacturing Systems, Vol. 8, No. 3 Cross Ref Charter value, minimum bank capital requirement and deposit insurance pricing in equilibriumJournal of Banking & Finance, Vol. 20, No. 2 Cross Ref Piecewise affine approximations for the control of a one-reservoir hydroelectric systemEuropean Journal of Operational Research, Vol. 89, No. 1 Cross Ref Chapter 14 Numerical dynamic programming in economics Cross Ref The bellman equation for vector-valued semi-markovian dyanmic programiingOptimization, Vol. 38, No. 1 Cross Ref A model of project evaluation with limited attentionEconomic Theory, Vol. 5, No. 1 Cross Ref Multiproduct production/inventory control under random demandsIEEE Transactions on Automatic Control, Vol. 40, No. 2 Cross Ref Mapping discounted and undiscounted Markov Decision Problems onto Hopfield neural networks8 June 2005 Cross Ref Differential Game Models of Global Environmental Management Cross Ref Learning to act using real-time dynamic programmingArtificial Intelligence, Vol. 72, No. 1-2 Cross Ref AUSTRALIAN ECONOMIC PAPERS DECEMBER, 1994: MONOPOLY INVESTMENT, PRICING AND PRODUCTION UNDER INTERTEMPORAL DEMAND UNCERTAINTYAustralian Economic Papers, Vol. 33, No. 63 Cross Ref Bibliography27 May 2008 Cross Ref Survey of linear programming for standard and nonstandard Markovian control problems. Part I: TheoryZOR - Methods and Models of Operations Research, Vol. 40, No. 1 Cross Ref Turnpikes and computation of piecewise open-loop equilibria in stochastic differential gamesJournal of Economic Dynamics and Control, Vol. 18, No. 2 Cross Ref Boundedly optimal control of piecewise deterministic systemsEuropean Journal of Operational Research, Vol. 73, No. 2 Cross Ref Chapter 51 Structural estimation of markov decision processes Cross Ref A generalized theorem of the maximumEconomic Theory, Vol. 3, No. 1 Cross Ref Some structured dynamic programs arising in economicsComputers & Mathematics with Applications, Vol. 24, No. 8-9 Cross Ref Policy iteration and Newton-Raphson methods for Markov decision processes under average cost criterionComputers & Mathematics with Applications, Vol. 24, No. 1-2 Cross Ref Optimal control of a facility with periodic interrupted demandJournal of Optimization Theory and Applications, Vol. 73, No. 3 Cross Ref A Review of Policy-Bounding Techniques in Fisheries Management Cross Ref An abstract topological approach to dynamic programmingJournal of Mathematical Economics, Vol. 21, No. 1 Cross Ref Turnpike properties for a class of piecewise deterministic systems arising in manufacturing flow controlAnnals of Operations Research, Vol. 29, No. 1 Cross Ref Optimal pricing of a product with periodic enhancementsEuropean Journal of Operational Research, Vol. 55, No. 2 Cross Ref Dynamic programming and maximum principle for discrete Goursat systemsJournal of Mathematical Analysis and Applications, Vol. 161, No. 1 Cross Ref An optimal one-way multigrid algorithm for discrete-time stochastic controlIEEE Transactions on Automatic Control, Vol. 36, No. 8 Cross Ref A turnpike improvement algorithm for piecewise deterministic controlOptimal Control Applications and Methods, Vol. 12, No. 1 Cross Ref Piecewise Deterministic and Piecewise Diffusion Differential Games with Modal Uncertainties Cross Ref Algorithms for Stochastic Games Cross Ref Relationships Between Various Markovian Decision Problem ClassesGary J. Koehler14 July 2006 | SIAM Journal on Control and Optimization, Vol. 28, No. 6AbstractPDF (856 KB)Equilibrium and adjustments in noncompetitive markets: Lost sales versus backlogging processesEngineering Costs and Production Economics, Vol. 19, No. 1-3 Cross Ref Recursive utility and the Ramsey problemJournal of Economic Theory, Vol. 50, No. 2 Cross Ref Fixed points for extrema of contractionsJournal of Mathematical Analysis and Applications, Vol. 146, No. 1 Cross Ref Chapter 8 Markov decision processes Cross Ref Optimal Bank Reorganization Policies and the Pricing of Federal Deposit Insurance30 April 2012 | The Journal of Finance, Vol. 44, No. 5 Cross Ref Controlled semi-markov models - the discounted caseJournal of Statistical Planning and Inference, Vol. 21, No. 3 Cross Ref Markov: A methodology for the solution of infinite time horizon markov decision processesApplied Stochastic Models and Data Analysis, Vol. 4, No. 4 Cross Ref Multiaction maintenance under markovian deterioration and incomplete state informationNaval Research Logistics, Vol. 35, No. 5 Cross Ref Sequential Stackelberg equilibria in two-person gamesJournal of Optimization Theory and Applications, Vol. 59, No. 1 Cross Ref Maximum Likelihood Estimation of Discrete Control ProcessesRust John1 August 2006 | SIAM Journal on Control and Optimization, Vol. 26, No. 5AbstractPDF (2387 KB)On efficiency of linear programming applied to discounted Markovian decision problemsOR Spektrum, Vol. 10, No. 3 Cross Ref Contraction mappings underlying undiscounted Markov decision problems—IIJournal of Mathematical Analysis and Applications, Vol. 132, No. 1 Cross Ref Solving Markovian decision processes by successive elimination of variablesJournal of Mathematical Analysis and Applications, Vol. 130, No. 2 Cross Ref The Social Costs of Monopoly and Regulation: A Game-Theoretic Analysis Cross Ref On the Existence of Sequential Equilibria in Markov Renewal Games Cross Ref Optimality conditions for continuous time systems with controlled jump Markov disturbances: Application to an FMS planning problem18 January 2006 Cross Ref Applications of fixed-point methods to discrete variational and quasi-variational inequalitiesNumerische Mathematik, Vol. 51, No. 6 Cross Ref On rational dynamic strategies in infinite horizon models where agents discount the futureJournal of Economic Behavior & Organization, Vol. 8, No. 3 Cross Ref Abstract Dynamic Programming Models under Commutativity ConditionsSergio Verdu and H. Vincent Poor14 July 2006 | SIAM Journal on Control and Optimization, Vol. 25, No. 4AbstractPDF (1846 KB)The Repair VS. Replacement problem: A stochastic control approach29 October 2007 | Optimal Control Applications and Methods, Vol. 8, No. 3 Cross Ref Asymptotic expansions for dynamic programming recursions with general nonnegative matricesJournal of Optimization Theory and Applications, Vol. 54, No. 1 Cross Ref Bounds on the fixed point of a monotone contraction operatorJournal of Mathematical Analysis and Applications, Vol. 123, No. 2 Cross Ref Iterative Bounds on the Equilibrium Distribution of a Finite Markov Chain27 July 2009 | Probability in the Engineering and Informational Sciences, Vol. 1, No. 1 Cross Ref Dynamic Programming and Markov Decision Processes11 November 2016 Cross Ref Optimal inspection policies for standby systemsCommunications in Statistics. Stochastic Models, Vol. 3, No. 2 Cross Ref On the indeterminacy of capital accumulation pathsJournal of Economic Theory, Vol. 40, No. 1 Cross Ref Variational characterizations in Markov decision processesJournal of Mathematical Analysis and Applications, Vol. 117, No. 2 Cross Ref Fixed point theorems for discounted finite markov decision processesJournal of Mathematical Analysis and Applications, Vol. 116, No. 2 Cross Ref Some new mathematical methods in dynamic programming over infinite horizonRivista di Matematica per le Scienze Economiche e Sociali, Vol. 9, No. 1 Cross Ref Approximation and bounds in discrete event dynamic programmingIEEE Transactions on Automatic Control, Vol. 31, No. 3 Cross Ref On the Computation of Equilibria in Discounted Stochastic Dynamic Games Cross Ref Optimal decisions over time and strange attractors: an analysis by the Bellman principleMathematical Modelling, Vol. 7, No. 2-3 Cross Ref Reward revision for partially observed Markov decision processes Cross Ref MARKOV DECISION PROCESSESStatistica Neerlandica, Vol. 39, No. 2 Cross Ref Computing optimal ( s, S ) policies in inventory models with continuous demands1 July 2016 | Advances in Applied Probability, Vol. 17, No. 2 Cross Ref Finite state approximation algorithms for average cost denumerable state Markov decision processes1 March 1985 | Operations-Research-Spektrum, Vol. 7, No. 1 Cross Ref Block-successive approximation for a discounted Markov decision modelStochastic Processes and their Applications, Vol. 19, No. 1 Cross Ref A survey on algortthmic aspects in preventive maintenanceMicroelectronics Reliability, Vol. 25, No. 2 Cross Ref A Fixed Point Approach to Undiscounted Markov Renewal ProgramsA. Federgrün and P. J. Schweitzer31 July 2006 | SIAM Journal on Algebraic Discrete Methods, Vol. 5, No. 4AbstractPDF (1282 KB)Truncated policy iteration methodsOperations Research Letters, Vol. 3, No. 5 Cross Ref Stochastic Production Planning with Production ConstraintsA. Bensoussan, S. P. Sethi, R. Vickson, and N. Derzko1 August 2006 | SIAM Journal on Control and Optimization, Vol. 22, No. 6AbstractPDF (1207 KB)Transient policies in discrete dynamic programming: Linear programming including suboptimality tests and additional constraintsMathematical Programming, Vol. 30, No. 1 Cross Ref Optimal admission pricing and service rate control of anM[x]/M/s queue with renegingNaval Research Logistics Quarterly, Vol. 30, No. 2 Cross Ref Vector-Valued Dynamic ProgrammingMordechai I. Henig17 February 2012 | SIAM Journal on Control and Optimization, Vol. 21, No. 3AbstractPDF (1059 KB)The Optimal Control of Partially Observable Semi-Markov Processes Over the Infinite Horizon: Discounted Costs Cross Ref Transformation of partially observable Markov decision processes into piecewise linear onesJournal of Mathematical Analysis and Applications, Vol. 91, No. 1 Cross Ref A welfare analysis of monopolistic R & DEconomics Letters, Vol. 12, No. 3-4 Cross Ref Optimization of STEOR networks via Markov renewal programmingZeitschrift für Operations Research, Vol. 26, No. 1 Cross Ref The variance of discounted Markov decision processes14 July 2016 | Journal of Applied Probability, Vol. 19, No. 04 Cross Ref The variance of discounted Markov decision processes14 July 2016 | Journal of Applied Probability, Vol. 19, No. 4 Cross Ref Calculating the variance in Markov-processes with random rewardTrabajos de Estadistica y de Investigacion Operativa, Vol. 33, No. 3 Cross Ref A multi-objective version of Bellman's inventory problemJournal of Mathematical Analysis and Applications, Vol. 87, No. 1 Cross Ref Conditions for characterizing the structure of optimal strategies in infinite-horizon dynamic programsJournal of Optimization Theory and Applications, Vol. 36, No. 3 Cross Ref Finite state approximations for denumerable state infinite horizon discounted Markov decision processes with unbounded rewardsJournal of Mathematical Analysis and Applications, Vol. 86, No. 1 Cross Ref Solving MDP functional equations by lexicographic optimization29 March 2011 | RAIRO - Operations Research, Vol. 16, No. 2 Cross Ref Computing the discounted return in markov and semi-markov chainsNaval Research Logistics Quarterly, Vol. 28, No. 4 Cross Ref Monotone optimal preventive maintenance policies for stochastically failing equipmentNaval Research Logistics Quarterly, Vol. 28, No. 3 Cross Ref Optimal control of price through restricted productionNaval Research Logistics Quarterly, Vol. 28, No. 3 Cross Ref A natural extension of the MacQueen extrapolationJournal of Mathematical Analysis and Applications, Vol. 83, No. 1 Cross Ref Dynamically optimized replacement with a Markovian renewal process14 July 2016 | Journal of Applied Probability, Vol. 18, No. 03 Cross Ref Dynamically optimized replacement with a Markovian renewal process14 July 2016 | Journal of Applied Probability, Vol. 18, No. 3 Cross Ref Isotone optimal policies for structured Markov decision processesEuropean Journal of Operational Research, Vol. 7, No. 4 Cross Ref Nonstationary Markov decision problems with converging parametersJournal of Optimization Theory and Applications, Vol. 34, No. 2 Cross Ref Countable-state average-cost regenerative stopping problems14 July 2016 | Journal of Applied Probability, Vol. 18, No. 02 Cross Ref Countable-state average-cost regenerative stopping problems14 July 2016 | Journal of Applied Probability, Vol. 18, No. 2 Cross Ref On the convergence of successive approximations in dynamic programming with non-zero terminal rewardZeitschrift für Operations Research, Vol. 25, No. 3 Cross Ref Bounds and elimination in generalized markov decisionsNaval Research Logistics Quarterly, Vol. 28, No. 1 Cross Ref Optimality in transient markov chains and linear programmingMathematical Programming, Vol. 18, No. 1 Cross Ref Economic aspects of appropriative water rightsJournal of Environmental Economics and Management, Vol. 7, No. 4 Cross Ref Optimal sequential selection and resource allocation under uncertainty1 July 2016 | Advances in Applied Probability, Vol. 12, No. 04 Cross Ref Stochastic optimal control: The discrete time case [Book reviews]IEEE Transactions on Automatic Control, Vol. 25, No. 6 Cross Ref Optimal sequential selection and resource allocation under uncertainty1 July 2016 | Advances in Applied Probability, Vol. 12, No. 4 Cross Ref Improved iterative computation of the expected discounted return in Markov and semi-Markov chainsZeitschrift für Operations Research, Vol. 24, No. 5 Cross Ref Discounted Stochastic Ratio GamesV. Aggarwal, R. Chandrasekaran, and K. P. K. Nair17 July 2006 | SIAM Journal on Algebraic Discrete Methods, Vol. 1, No. 2AbstractPDF (979 KB)Optimal maintenance-repair policies for the machine repair problemNaval Research Logistics Quarterly, Vol. 27, No. 1 Cross Ref Optimal admission pricing policies for M/Ek/1 queuesNaval Research Logistics Quarterly, Vol. 27, No. 1 Cross Ref Finite-state approximations for denumerable-state infinite-horizon discounted Markov decision processesJournal of Mathematical Analysis and Applications, Vol. 74, No. 1 Cross Ref Representation and Approximation of Noncooperative Sequential GamesWard Whitt18 July 2006 | SIAM Journal on Control and Optimization, Vol. 18, No. 1AbstractPDF (1819 KB)Über diskrete alternierende entseheidungsprozesse27 June 2007 | Mathematische Operationsforschung und Statistik. Series Optimization, Vol. 11, No. 1 Cross Ref A method of bisection for discounted Markov decision problemsZeitschrift für Operations Research, Vol. 23, No. 7 Cross Ref A priori bounds for approximations of Markov programsJournal of Mathematical Analysis and Applications, Vol. 71, No. 1 Cross Ref Steady State Policies for Deterministic Dynamic ProgramsJames Flynn12 July 2006 | SIAM Journal on Applied Mathematics, Vol. 37, No. 1AbstractPDF (2616 KB)Optimal state-dependent pricing policies for a class of stochastic multiunit service systemsNaval Research Logistics Quarterly, Vol. 26, No. 2 Cross Ref Value Convergence in a Generalized Markov Decision ProcessGary J. Koehler18 July 2006 | SIAM Journal on Control and Optimization, Vol. 17, No. 2AbstractPDF (744 KB)On theory and algorithms for Markov decision problems with the total reward criterion8 May 1979 | Operations-Research-Spektrum, Vol. 1, No. 1 Cross Ref Geometric convergence of value-iteration in multichain Markov decision problems1 July 2016 | Advances in Applied Probability, Vol. 11, No. 01 Cross Ref Geometric convergence of value-iteration in multichain Markov decision problems1 July 2016 | Advances in Applied Probability, Vol. 11, No. 1 Cross Ref A survey of algorithms for some restricted classes of Markov decision problems Cross Ref Successive approximations for Markov decision processes and Markov games with unbounded rewardsMathematische Operationsforschung und Statistik. Series Optimization, Vol. 10, No. 3 Cross Ref Markov decision processes and strongly excessive functionsStochastic Processes and their Applications, Vol. 8, No. 1 Cross Ref Contraction mappings underlying undiscounted Markov decision problemsJournal of Mathematical Analysis and Applications, Vol. 65, No. 3 Cross Ref A Stochastic Game Model of a Weapons Development CompetitionWayne Winston18 July 2006 | SIAM Journal on Control and Optimization, Vol. 16, No. 3AbstractPDF (1000 KB)Optimal control-limit policies for a zero-memory replacement problemInformation and Control, Vol. 37, No. 1 Cross Ref A zero-sum stochastic game model of duopolyInternational Journal of Game Theory, Vol. 7, No. 1 Cross Ref DISCOUNTED AND UNDISCOUNTED VALUE-ITERATION IN MARKOV DECISION PROBLEMS: A SURVEY Cross Ref THE ANALYTIC THEORY OF POLICY ITERATION11This research was partially supported by NRC Grant A3609. Cross Ref DYNAMIC PROGRAMMING IN BOREL SPACES11Work supported by Grant NSF ENG 74–19332. Cross Ref AFFINE DYNAMIC PROGRAMMING11This research was supported by NSF Grant ENG-76-15599. Cross Ref ON APPROXIMATE SOLUTIONS OF FINITE-STAGE DYNAMIC PROGRAMS Cross Ref AN OPERATOR-THEORETICAL TREATMENT OF NEGATIVE DYNAMIC PROGRAMMING Cross Ref REMOVED: References Cross Ref Successive approximation algorithms for stochastic games-Numerical comparisons Cross Ref Monotone Mappings with Application in Dynamic ProgrammingDimitri P. Bertsekas18 July 2006 | SIAM Journal on Control and Optimization, Vol. 15, No. 3AbstractPDF (2180 KB)Markov programming by successive approximations with respect to weighted supremum normsJournal of Mathematical Analysis and Applications, Vol. 58, No. 2 Cross Ref On the Optimality of Structured Policies in Countable Stage Decision Processes. II: Positive and Negative ProblemsDavid M. Kreps and Evan L. Porteus12 July 2006 | SIAM Journal on Applied Mathematics, Vol. 32, No. 2AbstractPDF (1264 KB)Stopping Times and Markov Programming Cross Ref Complementary Pivot Theory and Markovian Decision Chains Cross Ref A set of successive approximation methods for discounted Markovian decision problemsZeitschrift für Operations Research, Vol. 20, No. 5 Cross Ref The effect on optimal consumption of increased uncertainty in labor income in the multiperiod caseJournal of Economic Theory, Vol. 13, No. 1 Cross Ref On error bounds for successive approximation methodsIEEE Transactions on Automatic Control, Vol. 21, No. 3 Cross Ref The effect on optimal consumption of increased uncertainty in labor income in the multiperiod case21 May 2005 Cross Ref References Cross Ref A Survey of the Stete of the Art in Dynamic Programming16 July 2007 | A I I E Transactions, Vol. 8, No. 1 Cross Ref Convergence of discretization procedures in dynamic programmingIEEE Transactions on Automatic Control, Vol. 20, No. 3 Cross Ref Minimal representations of some classes of dynamic programmingInformation and Control, Vol. 27, No. 4 Cross Ref Brouwer's fixed point theorem and finite state space Markovian decision theoryJournal of Mathematical Analysis and Applications, Vol. 49, No. 3 Cross Ref Discounted semi-Markov decision processes: linear programming and policy iterationStatistica Neerlandica, Vol. 29, No. 1 Cross Ref On the Foundations of Dynamic Programming Cross Ref Introduction to Dynamic Programming**Adapted from E. V. Denardo and L. G. Mitten, “Elements of Sequential Decision Processes,” Journal of Industrial Engineering 18 (1967), 106-112. Cross Ref Zur Extrapolation in Markoffschen Entscheidungsmodellen mit DiskontierungZeitschrift für Operations Research, Vol. 18, No. 3 Cross Ref Optimal capital adjustment under uncertaintyJournal of Economic Theory, Vol. 8, No. 2 Cross Ref Classes of discrete optimization problems and their decision problemsJournal of Computer and System Sciences, Vol. 8, No. 1 Cross Ref A Class of Markovian Decision Processes Cross Ref Optimal Control of Queueing Systems Cross Ref Solvable classes of discrete dynamic programmingJournal of Mathematical Analysis and Applications, Vol. 43, No. 3 Cross Ref Continuous stochastic games14 July 2016 | Journal of Applied Probability, Vol. 10, No. 03 Cross Ref Continuous stochastic games14 July 2016 | Journal of Applied Probability, Vol. 10, No. 3 Cross Ref Optimal ordering policies for a product that perishes in two periods subject to stochastic demandNaval Research Logistics Quarterly, Vol. 20, No. 2 Cross Ref Evaluating information storage and retrieval system—A decision theory approachInformation Storage and Retrieval, Vol. 9, No. 5 Cross Ref Structured markovian decision problemsNaval Research Logistics Quarterly, Vol. 20, No. 1 Cross Ref Discretizing dynamic programsJournal of Optimization Theory and Applications, Vol. 11, No. 3 Cross Ref Solution of a Markovian decision problem by successive overrelaxationZeitschrift für Operations Research, Vol. 17, No. 1 Cross Ref Reducing the number of multiplikations in iterative processesActa Informatica, Vol. 3, No. 1 Cross Ref Representation theorems for equivalent optimization problemsInformation and Control, Vol. 21, No. 5 Cross Ref Finite-state approximations to denumerable-state dynamic programsJournal of Mathematical Analysis and Applications, Vol. 34, No. 3 Cross Ref On a set of optimal policies in continuous time Markovian decision problemJournal of Mathematical Analysis and Applications, Vol. 34, No. 1 Cross Ref Applications of Metric Coinduction Cross Ref Piecewise deterministic differential games Cross Ref Dynamic programming of stochastic activity networks with cycles Cross Ref Approximations and bounds in discrete stage markov decision processes Cross Ref An optimal multigrid algorithm for continuous state discrete time stochastic control Cross Ref Two-layer piecewise deterministic games Cross Ref New approach to optimization of discounted stochastic continuous-time discrete-event systems Cross Ref Finite state continuous time Markov decision processes with an infinite planning horizonJournal of Mathematical Analysis and Applications, Vol. 22, No. 3 Cross Ref Multichain Markov Renewal ProgramsE. V. Denardo and B. L. Fox12 July 2006 | SIAM Journal on Applied Mathematics, Vol. 16, No. 3AbstractPDF (2540 KB)Existence of Stationary Optimal Policies for Some Markov Renewal ProgramsBennett Fox18 July 2006 | SIAM Review, Vol. 9, No. 3AbstractPDF (448 KB)Markov Renewal Programming by Linear Fractional ProgrammingBennett Fox1 August 2006 | SIAM Journal on Applied Mathematics, Vol. 14, No. 6AbstractPDF (1321 KB) Volume 9, Issue 2| 1967SIAM Review History Submitted:19 August 1966Published online:18 July 2006 InformationCopyright © 1967 Society for Industrial and Applied MathematicsPDF Download Article & Publication DataArticle DOI:10.1137/1009030Article page range:pp. 165-177ISSN (print):0036-1445ISSN (online):1095-7200Publisher:Society for Industrial and Applied Mathematics