Observer and control design in partially observable finite Markov chains

Abstract

The controllable Partially Observable Markov Decision Process (POMDP) framework has proven to be useful in different domains where one is constrained to provide incomplete information of the structure and the parameters of the problem. Sometimes, it is not easy to track and measure accurately some state variables, and it may be more effective to make decisions based on imprecise information. This paper is focused on the design of an observer (which is unknown) for a class of ergodic homogeneous finite Markov chains with partially observable states. The main goal of the proposed method is the derivation of formulas for computing an observer, and as a result, on optimal control policy. For solving the problem, we introduce a new variable, which involves the product of the policy, the observation kernel and the distribution vector. We derive the formulas to recover the variables of interest. This work considers a dynamic environment for learning the parameters of the POMDP model. The construction of the adaptive policies is based on an identification approach, where we estimate the elements of the transition matrices and utility matrices by counting the number of unobserved experiences. A numerical example is presented to illustrate the practical implications of the theoretical issues applied to a portfolio optimization problem. These findings are important and new to the literature.