Optimal Reward Function Research Articles

An optimal maintenance model using a number of different actions

In this paper, we study an optimal maintenance model. The state of a system is determined by the distribution of its operating time. Whenever the system fails, a number of actions can be chosen, including the repair actions, the replacement actions and the action of discarding the system. The objective of this paper is to determine an optimal policy which maximizes the expected total discounted reward. By using the semi-Markov decision process approach, the method of successive approximations is suggested for determining the optimal reward function and the corresponding optimal policy.

The Optimal Reward Operator in Negative Dynamic Programming

We consider the negative dynamic programming model of Strauch [12] and prove that the optimal reward function can be obtained by a transfinite iteration of the optimal reward operator. We show that a player loses nothing by restricting himself to measurable policies, if the returns from nonmeasurable policies are evaluated by lower integrals.

On the Optimal Reward Function of the Continuous Time Multiarmed Bandit Problem

The optimal reward function associated with the so-called “multiarmed bandit problem” for general Markov–Feller processes is considered. It is shown that this optimal reward function has a simple expression (product form) in terms of individual stopping problems, without any smoothness properties of the optimal reward function neither for the global problem nor for the individual stopping problems. Some results relative to a related problem with switching cost are obtained.

Error bounds for rolling horizon policies in discrete-time Markov control processes

Error bounds are presented for rolling horizon (RH) policies in general, stationary and nonstationary, (Borel) Markov control problems with both discounted and average reward criteria. In each of these cases, conditions are given under which the reward of the rolling horizon policy converges geometrically to the optimal reward function, uniformly in the initial state, as the length of the rolling horizon increases. A description of the control model and the general assumptions are given. The approach is based on extending the results of J.M. Alden and A.R.L. Smith (1988) on nonstationary processes with finite state and action spaces. However the proofs presented are simpler. This is because, when stationary models are analyzed first, the error bounds follow more or less directly from well-known value iteration results. The corresponding error bounds for nonstationary models are obtained by reducing these models to stationary ones. >

Nonparametric adaptive control of discrete-time partially observable stochastic systems

We consider discrete-time, partially observable stochastic control systems x t + 1 = F( x t , u t , v t ), t = 0,1,…, y t = G( u t − 1 , x t , w t ), t = 1,2,…, y 0 = G 0( x 0, w 0), where x t , u t , and y t are, respectively, the state, control, and observation at time t, and the state and observation disturbance processes { v t } and { w t } are sequences of i.i.d. random elements with distributions ν and μ, respectively. We study the non-parametric adaptive control problem of maximizing the infinite-horizon discounted reward criterion when ν and μ are unknown and determine conditions for several adaptive policies to be asymptotically optimal and for almost-surely uniform approximations of the optimal reward function. Our approach combines convergence results for empirical processes and recent results on parameter-adaptive stochastic control problems.

Continuous dependence of stochastic control models on the noise distribution

Given a discrete-time stochastic control systemx t+1 =F(x t ,a t ,ξ t ),t=0, 1,.., N (N≤∞), where the noise process {ξ t } is a sequence of i.i.d. random elements with distributionμ, letυ μ N (x) be the optimal reward function when the initial state isx and the planning horizon isN. We give conditions under whichv μ N is a continuous function inμ for several reward criteria. The applicability of these results to nonparametric adaptive control of stochastic systems is briefly discussed.

Adaptive policies for discrete-time stochastic control systems with unknown disturbance distribution

We introduce adaptive policies for discrete-time, infinite horizon, stochastic control systems x 1+1 = F( x 1, a 1, ξ 1, t =0, 1, …, with discounted reward criterion, where the disturbance process ξ 1 is a sequence of i.i.d. random elements with unknown distribution. These policies are shown to be asymptotically optimal and for each of them we obtain (almost surely) uniform approximations of the optimal reward function.

Adaptive control of Markov processes with incomplete state information and unknown parameters

Recent results for parameter-adaptive Markov decision processes (MDP's) are extended to partially observed MDP's depending on unknown parameters. These results include approximations converging uniformly to the optimal reward function and asymptotically optimal adaptive policies.

The Agent-Agents Problem: Payment by Relative Output

This paper begins an investigation into the incentive problems at a firm when the effort of the employer and the employees combines to determine final output. The major conclusion is that optimal reward functions in this circumstance will in general value the performance of a worker relative to that of his peers. The paper thus joins a growing literature associated with the work of Lazear and Rosen on rank-order tournaments.

逆制御過程と逆配分過程

An inverse theory of sequential decision processes, including the standard control process and allocation process, is developed. A fInite-stage deterministic invertible (main) dynamic program (DP) whose reward functions depend not only on action but also on state is formulated a!i a sequential decision process. The main DP is transformed into an equivalent inverse DP by an algebraic inversion. Th(: main DP maximizes a generalized total reward, while the inverse DP minimizes a generalized total state. An inverse theorem is established. It characterizes optimal solutions (optimal reward functions and optimal policy) of inverse DP by those of main DP through inverse and composition. The main DP includl~s a linear equation and quadratic crit(:rion (main) control process on the half-line and a typical multi-stage (main) allocation process. Therefore, the inverse DP generates an inverse control process and an inverse allocation process, respectively. Not solving the recursive equation directly but applying the inverse theorem, optimal solutions of both inv(:rse processes are easily calculated by use of those of the corresponding main processes. 1. Introductiion

Structural Results for Partially Observable Markov Decision Processes

This paper examines monotonicity results for a fairly general class of partially observable Markov decision processes. When there are only two actual states in the system and when the actions taken are primarily intended to improve the system, rather than to inspect it, we give reasonable conditions which ensure that the optimal reward function and the optimal action are both monotone in the current state of information. Examples of maintenance systems and advertising systems for which our results hold are given. Finally, we examine the case where there are three or more actual states and indicate the difficulties encountered when we attempt to extend the monotonicity results to this situation.

INVERSE DYNAMIC PROGRAMMING II

This paper treats a class of finite-stage dynamic programming (DP) problems whose state space is an interval in the one-dimensional Euclidean space R1. By introducing a notion of “reward space” we can define, in this class, an inverse problem (called the inverse DP) to a given DP (called the main DP), because of the monotonicity and continuity of the reward functions and state transformations. Roughly speaking the inverse DP to the main DP is defined by replacing “max”, “reward functions” and “state transformations” with “min”, “inverse (in some sense) to the state transformations” and “inverse (in some sense) to the reward functions”, respectively, by replacing “terminal reward function” with “inverse to itself”, and by alternating “reward spaces” with “state spaces”, but by letting “action spaces” remain as they are. According to the monotonicity of the reward functions and state transformations we clarify the relation between the main DP and the inverse DP separately for four cases. Our main result is stated as follows. The pair of optimal reward functions and optimal policy for the main DP characterizes, in an inverse sense, the pair of optimal reward functions and optimal policy for the inverse DP, and vice versa (Section 2). Further we show that our DP represents a mathematical programming problem which has the property “recursiveness with monotonicity” in [5] and conversely such a problem can also be represented by our DP (Section 3). The last section is devoted to the illustration of several examples.

INVERSE DYNAMIC PROGRAMMING

This paper treats a class of finite-stage dynamic programming (DP) problems whose state space is an interval in the one-dimensional Euclidean space R1. By introducing a notion of “reward space” we can define, in this class, an inverse problem (called the inverse DP) to a given DP (called the main DP), because of the monotonicity and continuity of the reward functions and state transformations. Roughly speaking the inverse DP to the main DP is defined by replacing “max”, “reward functions” and “state transformations” with “min”, “inverse (in some sense) to the state transformations” and “inverse (in some sense) to the reward functions”, respectively, by replacing “terminal reward function” with “inverse to itself”, and by alternating “reward spaces” with “state spaces”, but by letting “action spaces” remain as they are. According to the monotonicity of the reward functions and state transformations we clarify the relation between the main DP and the inverse DP separately for four cases. Our main result is stated as follows. The pair of optimal reward functions and optimal policy for the main DP characterizes, in an inverse sense, the pair of optimal reward functions and optimal policy for the inverse DP, and vice versa (Section 2). Further we show that our DP represents a mathematical programming problem which has the property “recursiveness with monotonicity” in [5] and conversely such a problem can also be represented by our DP (Section 3). The last section is devoted to the illustration of several examples.