Finding Optimal Policies Research Articles

Stochastic Optimal Control of Risk Processes with Lipschitz Payoff Functions

This paper studies the stochastic optimal control problem of finding optimal dividend policies of an insurance company in discrete time with the use of general Lipschitz payoff functions involving indicators of profitability and risk. To construct positional optimal controls and to evaluate the performance indicators, the dynamic programming method is validated. The convergence rate of the successive approximation method in finding generally unbounded Bellman functions is estimated. The Pareto-optimal set of the problem is numerically approximated by so-called barrier-proportional control strategies.

Optimum inspection and maintenance policies for corroded structures using partially observable Markov decision processes and stochastic, physically based models

Stochastic control methods have a history of implementation in risk management and life-cycle cost procedures for civil engineering structures. The synergy of stochastic control methods and Bayesian principles can result in Partially Observable Markov Decision Processes (POMDPs) that allow consideration of uncertainty within the entire domain of the model and expand available policy options compared to other state-of-the art methods. The superior attributes of POMDPs enable optimum decisions which are based on the belief space or otherwise only on the best knowledge that a decision-maker can have at each time. In this work the effort is mostly based in modeling and solving the problem of finding optimal policies for the maintenance and management of aging structures through a POMDP framework with large state spaces that can adequately and sufficiently describe real-life problems. In order to form the POMDP framework, stochastic, physically based models can be used and their connection to the control process is explained in detail. Specific examples of a corroded existing structure are presented, based on non-stationary POMDPs, for both infinite and finite horizon cases with 332 and 14,009 states respectively. Results from both cases are compared and discussed and the capabilities of the method become apparent.

A particle swarm algorithm for optimising inspection policies in serial multistage production processes with uncertain inspection costs

Since the quality of products is a major concern in production systems, implementing efficient inspection policies is of great importance to reduce quality-related costs. This article addresses the problem of finding optimal inspection policies in serial multistage production processes to minimise total inspection cost where the cost components are described by the use of fuzzy numbers. The type of inspection in each stage and the acceptance limits are considered as decision variables. The size of sample to inspect, where sampling inspection is adopted, is also incorporated into the model as the third decision parameter. The case in which the inspectors are not error-free is examined, as well. A solution algorithm is proposed based on particle swarm optimisation, and simulation is used to provide better insight into the optimal solution. Numerical experiments discussed in different settings illustrate the applicability and efficiency of the proposed approach.

Bootstrapping Simulation-Based Algorithms with a Suboptimal Policy

Finding optimal policies for Markov Decision Processes with large state spaces is in general intractable. Nonetheless, simulation-based algorithms inspired by Sparse Sampling (SS) such as Upper Confidence Bound applied in Trees (UCT) and Forward Search Sparse Sampling (FSSS) have been shown to perform reasonably well in both theory and practice, despite the high computational demand. To improve the efficiency of these algorithms, we adopt a simple enhancement technique with a heuristic policy to speed up the selection of optimal actions. The general method, called Aux, augments the look-ahead tree with auxiliary arms that are evaluated by the heuristic policy. In this paper, we provide theoretical justification for the method and demonstrate its effectiveness in two experimental benchmarks that showcase the faster convergence to a near optimal policy for both SS and FSSS. Moreover, to further speed up the convergence of these algorithms at the early stage, we present a novel mechanism to combine them with UCT so that the resulting hybrid algorithm is superior to both of its components.

AN OPTIMAL DIVIDEND POLICY WITH DELAYED CAPITAL INJECTIONS

AbstractThis work focuses on finding optimal dividend payment and capital injection policies to maximize the present value of the difference between the cumulative dividend payment and the possible capital injections with delays. Starting from the classical Cramér–Lundberg process, using the dynamic programming approach, the value function obeys a quasi-variational inequality. With delays in capital injections, the company will be exposed to the risk of financial ruin during the delay period. In addition, the optimal dividend payment and capital injection strategy should balance the expected cost of the possible capital injections and the time value of the delay period. In this paper, the closed-form solution of the value function and the corresponding optimal policies are obtained. Some limiting cases are also discussed. A numerical example is presented to illustrate properties of the solution. Some economic insights are also given.

Analysis of cross-price effects on markdown policies by using function approximation techniques

Markdown policies for product groups having significant cross-price elasticity among each other should be jointly determined. However, finding optimal policies for product groups becomes computationally intractable as the number of products increases. Therefore, we formulate the problem as a Markov decision process and use approximate dynamic programming approach to solve it. Since the state space is multidimensional and very large, the number of iterations required to learn the state values is enormous. Therefore, we use aggregation and neural networks in order to approximate the value function and to determine the optimal markdown policies approximately. In a numerical study, we provide insights on the behavior of markdown policies when one product is expensive, the other is cheap and both have the same price. We also provide insights and compare the markdown policies for the cases in which there is a substitution effect between products and the products are independent.

Solution time reduction techniques of a stochastic dynamic programming approach for hazardous material route selection problem

We use a stochastic dynamic programming (SDP) approach to solve the problem of determining the optimal routing policies in a stochastic dynamic network. Due to its long time for solving SDP, we propose three techniques for pruning stochastic dynamic networks to expedite the process of obtaining optimal routing policies. The techniques include: (1) use of static upper/lower bounds, (2) pre-processing the stochastic dynamic networks by using the start time and origin location of the vehicle, and (3) a mix of pre-processing and upper/lower bounds. Our experiments show that while finding optimal routing policies in stochastic dynamic networks, the last two of the three strategies have a significant computational advantage over conventional SDP. Our main observation from these experiments was that the computational advantage of the pruning strategies that depend on the start time of the vehicle varies according to the time input to the problem. We present the results of this variation in the experiments section. We recommend that while comparing the computational performances of time-dependent techniques, it is very important to test the performance of such strategies at various time inputs.

Q-Learning Based Energy Management Policies for a Single Sensor Node with Finite Buffer

In this paper, we consider the problem of finding optimal energy management policies in the presence of energy harvesting sources to maximize network performance. We formulate this problem in the discounted cost Markov decision process framework and apply two reinforcement learning algorithms. Prior work obtains optimal policy in the case when the conversion function mapping energy to data transmitted is linear and provides heuristic policies in the case when the same is nonlinear. Our algorithms, however, provide optimal policies regardless of the form of the conversion function. Through simulations, our policies are seen to outperform those of in the nonlinear case.

Numerical Methods for Optimal Dividend Payment and Investment Strategies of Markov-Modulated Jump Diffusion Models with Regular and Singular Controls

This work focuses on numerical methods for finding optimal dividend payment and investment policies to maximize the present value of the cumulative dividend payment until ruin; the surplus is modeled by a regime-switching jump diffusion process subject to both regular and singular controls. Using the dynamic programming principle, the optimal value function obeys a coupled system of nonlinear integro-differential quasi-variational inequalities. Since the closed-form solutions are virtually impossible to obtain, we use Markov chain approximation techniques to approximate the value function and optimal controls. Convergence of the approximation algorithms are proved. Examples are presented to illustrate the applicability of the numerical methods.

BOUNDARY EVOLUTION EQUATIONS FOR AMERICAN OPTIONS

We consider the problem of finding optimal exercise policies for American options, both under constant and stochastic volatility settings. Rather than work with the usual equations that characterize the price exclusively, we derive and use boundary evolution equations that characterize the evolution of the optimal exercise boundary. Using these boundary evolution equations we show how one can construct very efficient computational methods for pricing American options that avoid common sources of error. First, we detail a methodology for standard static grids and then describe an improvement that defines a grid that evolves dynamically while solving the problem. When integral representations are available, as in the Black–Scholes setting, we also describe a modified integral method that leverages on the representation to solve the boundary evolution equations. Finally we compare runtime and accuracy to other popular numerical methods. The ideas and methodology presented herein can easily be extended to other optimal stopping problems.

Optimal search, learning and implementation

We characterize the incentive compatible, constrained efficient policy (“second-best”) in a dynamic matching environment, where impatient, privately informed agents arrive over time, and where the designer gradually learns about the distribution of agentsʼ values. We also derive conditions on the learning process ensuring that the complete-information, dynamically efficient allocation of resources (“first-best”) is incentive compatible. Our analysis reveals and exploits close, formal relations between the problem of ensuring implementable allocation rules in our dynamic allocation problems with incomplete information and learning, and between the classical problem, posed by Rothschild (1974) [20], of finding optimal stopping policies for search that are characterized by a reservation price property.

Computing ( r, Q) policy for an inventory system with limited sharable resource

This paper deals with inventory systems with limited resource for a single item or multiple items under continuous review ( r, Q) policies. For the single-item system with a stochastic demand and limited resource, it is shown that an existing algorithm can be applied to find an optimal ( r, Q) policy that minimizes the expected system costs. For the multi-item system with stochastic demands and limited resource commonly shared among all items, an optimization problem is formulated for finding optimal ( r, Q) policies for all items, which minimize the expected system costs. Bounds on the parameters (i.e., r and Q) of the optimal policies and bounds on the minimum expected system costs are obtained. Based on the bounds, an algorithm is developed for finding an optimal or near-optimal solution. A method is proposed for evaluating the quality of the solution. It is shown that the algorithm proposed in this paper finds a solution that is (i) optimal/near-optimal and/or (ii) significantly better than the optimal solution with unlimited resource.

Boundary Evolution Equations for American Options

We consider the problem of finding optimal exercise policies for American options, both under constant and stochastic volatility settings. Rather than work with the usual equations that characterize the price exclusively, we derive and use boundary evolution equations that characterize the evolution of the optimal exercise boundary. Using these boundary evolution equations we show how one can construct very efficient computational methods for pricing American options that avoid common sources of error. First, we detail a methodology for standard static grids and then describe an improvement that defines a grid that evolves dynamically while solving the problem. When integral representations are available, as in the Black–Scholes setting, we also describe a modified integral method that leverages on the representation to solve the boundary evolution equations. Finally we compare runtime and accuracy to other popular numerical methods. The ideas and methodology presented herein can easily be extended to other optimal stopping problems.

On Optimal Cooperator Selection Policies for Multi-Hop Ad Hoc Networks

In this paper we consider wireless cooperative multihop networks, where nodes that have decoded the message at the previous hop cooperate in the transmission toward the next hop, realizing a distributed space-time coding scheme. Our objective is finding optimal cooperator selection policies for arbitrary topologies with links affected by path loss and multipath fading. To this end, we model the network behavior through a suitable Markov chain and we formulate the cooperator selection process as a stochastic shortest path problem (SSP). Further, we reduce the complexity of the SSP through a novel pruning technique that, starting from the original problem, obtains a reduced Markov chain which is finally embedded into a solver based on focused real time dynamic programming (FRTDP). Our algorithm can find cooperator selection policies for large state spaces and has a bounded (and small) additional cost with respect to that of optimal solutions. Finally, for selected network topologies, we show results which are relevant to the design of practical network protocols and discuss the impact of the set of nodes that are allowed to cooperate at each hop, the optimization criterion and the maximum number of cooperating nodes.

Modelling the strategies for age specific vaccination scheduling during influenza pandemic outbreaks

Finding optimal policies to reduce the morbidity and mortality of the ongoing pandemic is a top public health priority. Using a compartmental model with age structure and vaccination status, we examined the effect of age specific scheduling of vaccination during a pandemic influenza outbreak, when there is a race between the vaccination campaign and the dynamics of the pandemic. Our results agree with some recent studies on that age specificity is paramount to vaccination planning. However, little is known about the effectiveness of such control measures when they are applied during the outbreak. Comparing five possible strategies, we found that age specific scheduling can have a huge impact on the outcome of the epidemic. For the best scheme, the attack rates were up to 10% lower than for other strategies. We demonstrate the importance of early start of the vaccination campaign, since ten days delay may increase the attack rate by up to 6%. Taking into account the delay between developing immunity and vaccination is a key factor in evaluating the impact of vaccination campaigns. We provide a general framework which will be useful for the next pandemic waves as well.

Learning automata based multi‐agent system algorithms for finding optimal policies in Markov games

AbstractMarkov games, as the generalization of Markov decision processes to the multi‐agent case, have long been used for modeling multi‐agent systems (MAS). The Markov game view of MAS is considered as a sequence of games having to be played by multiple players while each game belongs to a different state of the environment. In this paper, several learning automata based multi‐agent system algorithms for finding optimal policies in Markov games are proposed. In all of the proposed algorithms, each agent residing in every state of the environment is equipped with a learning automaton. Every joint‐action of the set of learning automata in each state corresponds to moving to one of the adjacent states. Each agent moves from one state to another and tries to reach the goal state. The actions taken by learning automata along the path traversed by the agent are then rewarded or penalized based on the comparison of the average reward received by agent per move along the path with a dynamic threshold. In the second group of the proposed algorithms, the concept of entropy has been imported into learning automata based multi‐agent systems to improve the performance of the algorithms. To evaluate the performance of the proposed algorithms, computer experiments have been conducted. The results of experiments have shown that the proposed algorithms perform better than the existing algorithms in terms of speed and accuracy of reaching the optimal policy.Copyright © 2010 John Wiley and Sons Asia Pte Ltd and Chinese Automatic Control Society

A multi-objective approach to simultaneous determination of spare part numbers and preventive replacement times

Different models have been proposed in the field of preventive maintenance planning for finding optimal age replacement policies. While previous studies have focused mainly on classical cost objectives, this paper presents a novel multi-objective model for preventive replacement of a part over a planning horizon. The proposed model considers different objectives and practical issues, such as corrective replacement and its consequences, residual lifetime objective, and kind of productivity index. Also, the model determines number of spare parts, required for replacement with the defected part, to be provided at the beginning of the planning horizon. The multi-objective model is applicable for machines or equipments which are repaired through replacing their defected part with new spare part. For solving the multi-objective model, regarding to ability of ε-constraint method to generate different pareto-optimal solutions, a procedure is developed based on this method. The procedure shows how the ε-constraint method can be used for finding preferred solution in situations where there is no access to decision maker. The model and solution procedure are illustrated by a numerical example.

Sensitivity analysis and optimal ultimately stationary deterministic policies in some constrained discounted cost models

We consider a discrete time Markov Decision Process (MDP) under the discounted payoff criterion in the presence of additional discounted cost constraints. We study the sensitivity of optimal Stationary Randomized (SR) policies in this setting with respect to the upper bound on the discounted cost constraint functionals. We show that such sensitivity analysis leads to an improved version of the Feinberg–Shwartz algorithm (Math Oper Res 21(4):922–945, 1996) for finding optimal policies that are ultimately stationary and deterministic.

Optimal Search, Learning, and Implementation

We derive conditions on the learning environment - which encompasses both Bayesian and non-Bayesian processes - ensuring that an efficient allocation of resources is achievable in a dynamic allocation environment where impatient, privately informed agents arrive over time, and where the designer gradually learns about the distribution of agents' values. There are two main kind of conditions: 1) Higher observations should lead to more optimistic beliefs about the distribution of future values; 2) The allowed optimism associated with higher observations needs to be carefully bounded. Our analysis reveals and exploits close, formal relations between the problem of ensuring monotone - and hence implementable - allocation rules in our dynamic allocation problems with incomplete information and learning, and between the classical problem of finding optimal stopping policies for search that are characterized by a reservation price property.

Optimal parameter trajectory estimation in parameterized SDEs

We consider the problem of estimating the optimal parameter trajectory over a finite time interval in a parameterized stochastic differential equation (SDE), and propose a simulation-based algorithm for this purpose. Towards this end, we consider a discretization of the SDE over finite time instants and reformulate the problem as one of finding an optimal parameter at each of these instants. A stochastic approximation algorithm based on the smoothed functional technique is adapted to this setting for finding the optimal parameter trajectory. A proof of convergence of the algorithm is presented and results of numerical experiments over two different settings are shown. The algorithm is seen to exhibit good performance. We also present extensions of our framework to the case of finding optimal parameterized feedback policies for controlled SDE and present numerical results in this scenario as well.