Reinforcement Learning-Based Tracking Control under Stochastic Noise and Unmeasurable State for Tip–Tilt Mirror Systems

Abstract

The tip–tilt mirror (TTM) is an important component of adaptive optics (AO) to achieve beam stabilization and pointing tracking. In many practical applications, the information of accurate TTM dynamics, complete system state, and noise characteristics is difficult to achieve due to the lack of sufficient sensors, which then restricts the implementation of high precision tracking control for TTM. To this end, this paper proposes a new method based on noisy-output feedback Q-learning. Without relying on neural networks or additional sensors, it infers the dynamics of the controlled system and reference jitter using only noisy measurements, thereby achieving optimal tracking control for the TTM system. We have established a modified Bellman equation based on estimation theory, directly linking noisy measurements to system performance. On this basis, a fast iterative learning of the control law is implemented through the adaptive transversal predictor and experience replay technique, making the algorithm more efficient. The proposed algorithm has been validated with an application to a TTM tracking control system, which is capable of quickly learning near-optimal control law under the interference of random noise. In terms of tracking performance, the method reduces the tracking error by up to 98.7% compared with the traditional integral control while maintaining a stable control process. Therefore, this approach may provide an intelligent solution for control issues in AO systems.