Dynamic programming approach to a final-value control system with a random variable having an unknown distribution function

Abstract

As described in the introduction of this paper, the state vector x <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</inf> , representing the present state of a sampling control system, is assumed to satisfy a difference equation <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">x_{n+1} = T(x_{n}, r_{n}, \upsilon_{n})</tex> , where <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">\upsilon_{n}</tex> is the control vector of the system subject to random disturbances r <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</inf> . The random variables r <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</inf> are assumed to be independent of each other and to be defined in a parameter space in the following manner: Nature is assumed to be in one of a finite number, <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">q</tex> , of possible states and each state has its own parameter value, thus specifying uniquely and unequivocally the distribution function of r <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</inf> . It is further assumed that we are given the a priori probability <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">z =(z_{1}, . . . , z_{q})</tex> of each possible state of nature, <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i= 1,2, . . . , q,\sum\min{i=1}\max{q} z_{i}=1, z_{i} \geq 0</tex> . Given a criterion of performance, the duration of the process and the domain of the control variable, a sequence of control variables { <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">\upsilon_{n}</tex> } is to be determined as a function of the state vector of the system and time, so as to optimize the performance. The sequential nature of the determination of <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">\upsilon_{n}</tex> is evident here, since the stochastic nature of the problem prevents specification of such a sequence of <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">\upsilon</tex> 's as a function of the initial state and time. By means ot the functional equation technique of dynamic programming, a recurrence relation of the criterion function of the process, <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">k_{n}(x, z)</tex> , is derived, where <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">x</tex> is the state variable of the system when there remain <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</tex> control stages. When <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">q</tex> is taken to be two and the criterion of performance to he <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">x_{N}^{2}</tex> with the constraint on the control variables <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">\sum\min{i=0}\max{N-1} \upsilon_{i}^{2} \leq K</tex> , where x <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</inf> is the final state vector of the system, it is shown that <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">k_{n}(x, z)</tex> is the form <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">w_{n}(z)x^{2}</tex> and the optimal <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">\upsilon_{n}(x, z)</tex> is linear in <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">x</tex> , as might be expected. The dependence of <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">k_{n}(x, z)</tex> and <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">\upsilon_{n}(x, z)</tex> on <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">z</tex> is investigated further, and <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">w_{n}(z)</tex> is found to he concave in <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">z</tex> . Explicit quadratic forms for <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">w_{n}(z)</tex> are obtained in the neighborhood of <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">z = 0</tex> and 1. The optimal <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">\upsilon_{n}(x, z)</tex> is found to be monotonically decreasing in <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">z</tex> . When the domain of <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">\upsilon_{n}</tex> is restricted to a finite set of values, as in contactor servo systems, no explicit expressions for <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">k_{n}(x, z)</tex> and <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">\upsilon_{n}(x, z)</tex> are available. However, <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">k_{n}(x, z)</tex> is still concave in <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">z</tex> and approximately given by <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">zk_{n}(x, 1) + (1 - z)k_{n}(x, 0)</tex> . By solving the recurrence relation numerically, this approximation is found to be very good for moderately large n, say 10. This means that if one has explicit solutions for <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">p =p_{1}</tex> and <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">p = P_{2}</tex> , then one can approximate the criterion function for the adaptive case where <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Pr(p =p_{1})= z</tex> by a linear function in <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">z</tex> as shown above. This provides fairly good lower bound on <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">k_{n}(x, z)</tex> , and in turn serves to determine an initial approximate policy. A concept of suboptimal policy is introduced and numerical experiments are performed to test suboptimal policy suggested from numerical solution. The system behavior under the suboptimal policy is also discussed.