Discounted linear Q-learning control with novel tracking cost and its stability

Abstract

In this article, in order to achieve optimal tracking control of unknown linear discrete systems, a model-free scheme based on Q-learning is established online. First, we introduce an innovative performance index function, so as to eliminate the tracking error and avert the calculation for stable control policies of the reference trajectory. Taking value iteration and policy iteration into consideration, the corresponding model-based approaches are derived. Then, the Q-function is developed and the model-free algorithm utilizing Q-learning is given for the sake of dealing with the linear quadratic tracking (LQT) problem online without relying on system dynamics information. In addition, novel stability analysis based on Q-learning is provided for the discounted LQT control issue and the probing noise is demonstrated that it does not result in any excitation noise bias. Finally, by means of conducting numerical simulation, the proposed Q-learning algorithm is demonstrated to be effective and practicable.