Adaptive Control and Cost Sensitivity of Stochastic Systems with Continuous State and Parameters, and Logical Controls

Abstract

We consider event-driven stochastic control problems, in which the state {X <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">t</sub> } takes values in a continuum, and the control actions {U <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">t</sub> } are selected from a finite (event) set. Furthermore, lack of knowledge about certain quantities in the model, e.g. transition probability matrices, is accounted for via a parameterization by a quantity θ, which also takes values in a continuum. More specifically, the model for the stochastic systems that we consider is that of a discrete-time controlled Markov process (CMP), parameterized by the quantity θ. This model is described by the quadruplet (U, U, θ P), where the state space X and parameter set θ are complete and separable metric spaces, the event (or control) set U is a finite set, and P(· | x, u; θ) is the transition (stochastic) kernel that gives the distribution of X <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">t+1</sub> , when X <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">t</sub> = x, U <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">t</sub> = u, and the (unknown) value for the parameter is ∞; see [4], [6], [9], for details on CMP. The characteristic of special interest in our model is that while the state and unknown parameters take values in a continuum, the control actions are drawn from a finite set, i.e., these are logical or Boolean quantities. This situation arises in many problems of interest.