Sparse online maximum entropy inverse reinforcement learning via proximal optimization and truncated gradient

Abstract

Maximum entropy (ME) algorithms have been broadly studied towards the learning of rewards and obtaining of optimal policies for inverse reinforcement learning (IRL) problems. However, the tendency for the avoidance of maximum entropy IRL (ME IRL) algorithms, due to issues such as complicated computations, overfitting, and low convergence, has inspired studies aimed at the development of newer ME IRL algorithms of improved performance. Here, a new maximum entropy IRL based on proximal optimization is proposed. The follow-the-proximally-regularized-leader (FTPRL) method, with its good sparse solutions, is taken as proximal optimization to improve the generalization performance of ME IRL algorithm, resulting in ME-FTPRL IRL. With the help of l1/l22-regularization and adaptive per-state learning rates, our proposed algorithm can select features and correct the update direction of reward weights to reduce model complexity and avoid overfitting, which also speeds up convergence. During each iteration, the truncated gradient (TG) method is applied for the ME-FTPRL IRL (named ME-TFTPRL IRL) to update reward weights. This avoids the floating-point problem of the FTPRL method. Q-learning algorithm is then used to obtain the optimal strategies with the learned rewards. Subsequently, the sparsity and convergence of ME-TFTPRL IRL are proven based on regularization, TG method and regret bound. Simulation results demonstrate that our proposed ME-TFTPRL IRL has better sparsity, better generalization, and faster convergence than either existing or our own initially proposed ME IRL algorithms.