Generation of Spacecraft Operations Procedures Using Deep Reinforcement Learning

Abstract

The high cost of space mission operations has motivated several space agencies to prioritize the development of autonomous spacecraft command and control technologies. Deep reinforcement learning (DRL) techniques present one promising domain for the creation of autonomous agents for complex, multifaceted operations problems. This work examines the feasibility of adapting DRL-driven policy generation algorithms to problems in spacecraft decision-making, including strategies for framing spacecraft decision-making problems such as Markov decision processes, avenues for dimensionality reduction, and simplification using expert domain knowledge, sensitivity to hyperparameters, and robustness in the face of mismodeled environmental dynamics. In addition, consideration is given to ensuring the safety of these approaches by hybridizing them with correct-by-construction control techniques in a novel adaptation of shielded deep reinforcement learning. These strategies are demonstrated against a prototypical low-fidelity stationkeeping scenario and a high-fidelity attitude mode management scenario involving flight heritage attitude control and momentum management algorithms. DRL techniques are found to compare favorably to other black-box optimization tools or heuristic solutions for these problems and to require similar network sizes and training durations as widely used testing datasets in the deep learning community.