Abstract
In this paper, we propose a general Riemannian proximal optimization algorithm with guaranteed convergence to solve Markov decision process (MDP) problems. To model policy functions in MDP, we employ Gaussian mixture model (GMM) and formulate it as a non-convex optimization problem in the Riemannian space of positive semidefinite matrices. For two given policy functions, we also provide its lower bound on policy improvement by using bounds derived from the Wasserstein distance of GMMs. Preliminary experiments show the efficacy of our proposed Riemannian proximal policy optimization algorithm.
Highlights
Reinforcement learning studies how agents explore/exploit environment, and take actions to maximize long-term reward
For two given policy functions, we provide its lower bound on policy improvement by using bounds derived from the Wasserstein distance of Gaussian mixture model (GMM)
To optimize GMM and learn the optimal policy functions efficiently, we formulate it as a non-convex optimization problem in the Riemannian space
Summary
Reinforcement learning studies how agents explore/exploit environment, and take actions to maximize long-term reward. TRPO, PPO and CPO have shown promising performance on complex decision-making problems, such as continuous-control tasks and playing Atari, as other neural network based models, they face two typical challenges: the lack of interpretability, and difficult to converge due to the nature of non-convex optimization in high dimensional parameter space. In this study we choose GMM due to its good analytical characteristics, universal representation power and low computational cost compared with neural networks It is well-known that the covariance matrices of GMM lie in a Riemannian manifold of positive semidefinite matrices. To optimize GMM and learn the optimal policy functions efficiently, we formulate it as a non-convex optimization problem in the Riemannian space By this way, our method gains advantages in improving both interpretability and speed of convergence. It suffers from the headache of Q-learning that it can hardly handle problems with large continuous state-action space
Sign-in/Register to view highlights & summary Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
