Disturbance-aware reinforcement learning for rejecting excessive disturbances

Abstract

This paper presents a disturbance-aware Reinforcement Learning (RL) approach for stabilizing a free-floating platform under excessive external disturbances. In particular, we consider the scenarios where disturbances frequently exceed actuator limits and largely affect the dynamics characterizing the disturbed platform. This stabilization problem is better described by a set of Unknown Partially Observable Markovian Decision Processes (POMDPs), as opposed to a single-POMDP formulation, making online disturbance awareness necessary. This paper proposes a new Disturbance-Observer network (DO-net) that mimics prediction procedures through an auxiliary Gated Recurrent Unit (GRU), for the purpose of estimating and encoding the disturbance states and the disturbance transition functions, respectively. Then the controller subnetwork is trained with joint optimization of the observer subnetwork in an RL manner for mutual robustness and runtime efficiency. Numerical simulations on position regulation tasks have demonstrated that the DO-net outperforms the DOB-net and reduces the gap with an ideal performance estimate, the latter of which is obtained by a commercial solver given precise disturbance knowledge.