Incremental Receptive Field Weighted Actor-Critic

Abstract

In this paper, a novel actor-critic method using an incrementally constructed radial basis function network is developed to deal with continuous state and action problems. There exists one local model for each basis function, and the number of local models is increased as the basis function network grows. The normalized weighted sum of their outputs is used to estimate the value function for the critic, and the models are updated with the local temporal difference error in the receptive field of the corresponding basis function. A Gaussian policy is used for continuous action, and it is parameterized by the mean and the standard deviation. The parameters are determined by the normalized weighed sum of the corresponding sub-parameters assigned to the basis functions, and the regular policy gradient method is used for their update process. A new error is introduced for the online shape adaptation of the basis functions. Reducing this error prevents some of the basis functions from dominating the value function approximation and the policy, and improves the performance when the incrementally constructed basis function network is used. Simulation results for three benchmark problems show the performance and effectiveness of the proposed method in comparison to other methods.