Meta-Reinforcement Learning for Spacecraft Proximity Operations Guidance and Control in Cislunar Space

Abstract

Innovative lightweight and model-free guidance algorithms are essential to achieve full autonomy of spacecraft and address future space exploration challenges. In the near future, this type of technology will be essential for cislunar space proximity operations, such as NASA’s Artemis program and its Lunar Gateway project. In this scenario, a meta-reinforcement learning algorithm is employed to address the real-time optimal guidance problem of a spacecraft in the cislunar environment. Non-Keplerian orbits pose complex dynamics, where classic control theory may be less adaptable and more computationally expensive compared to machine learning approaches. Meta-RL stands out for its ability to learn how to learn through experience, training on various tasks to enhance efficiency, and effectiveness in tackling new ones. By modeling a stochastic optimal control problem within the circular restricted three-body problem framework as a Markov decision process, an LSTM-based network agent is trained using proximal policy optimization, considering operational constraints and stochastic effects for safety and robustness evaluation. Additionally, an MLP-based agent and an optimal control pseudo-spectral solution are assessed for comparison. The resulting tool autonomously guides spacecraft in cislunar proximity operations, approximating the optimal control solution with a versatile algorithmic framework. This ensures both robustness and computational efficiency.