EPPE: An Efficient Progressive Policy Enhancement framework of deep reinforcement learning in path planning

Abstract

Path planning is a key process in robotics, playing an important role in fields such as autonomous driving and logistic delivery. Our work addresses the dual challenges of training efficiency and composite optimization in path planning using Deep Reinforcement Learning (DRL). We introduce the Efficient Progressive Policy Enhancement (EPPE) framework, which integrates the advantages of sparse rewards, aimed at achieving a globally optimal policy for the agent, with process rewards that provide real-time feedback for the agent’s policy adjustment. This framework not only significantly enhances policy learning efficiency but also effectively resolves the reward coupling issues introduced by process rewards, thereby ensuring the achievement of a globally optimal policy. Within this framework, the initial reward structure incorporates guiding rewards, which are a type of process reward based on conventional path planning algorithms, and assigns significant weights to provide real-time feedback, thereby effectively enhancing the training efficiency. Additionally, the Incremental Reward Adjustment (IRA) model is proposed to progressively increase the reward weights in the composite optimization part. The Fine-tuning Policy Optimization (FPO) model, supporting the IRA model, makes gradual adjustments to the learning rate throughout the entire process. Simulated experiments demonstrate the advantage of our framework in path composite optimization. In static obstacle environments, compared to seven benchmark algorithms, the time and distance to reach the target are improved by at least 10.4%. In mixed obstacle environments, these improvements are at least 19.1% and 18.2%. Additionally, our framework also significantly enhances the training efficiency of DRL.