Markov decision processes with controlled observations and the sensor querying problem

Abstract

This paper studies a class of discrete-time stochastic control problems with controlled observations. This problem is motivated by the wide deployment of networked control systems and data fusion, where querying all sensors is not possible at each time instant. Since only part of the observation information is available at each time step, the controller has to balance the system performance with the penalty of the requested information (query). We first formulate the problem for a general partially observed Markov decision process model and provide a sufficient condition for existence of ergodic control with hierarchical information structure. Then we specialize to the linear stochastic control problem, where we show that the separation principle partially holds. We also show that distributing sensors does not affect the stability of the system: if the system can be stabilized with all sensor data available at the controller, it can also be stabilized with sensor querying. A sharp condition is provided for the existence of ergodic control for LQG control problem with sensor querying.