A Sparse Kernel-Based Least-Squares Temporal Difference Algorithm for Reinforcement Learning

Abstract

Reinforcement learning algorithms with function approximation have attracted many research interests since most real-world problems have large or continuous state spaces. To improve the generalization ability of function approximation, kernel-based reinforcement learning becomes one of the most promising methods in recent years. But one main difficulty in kernel methods is the computational and storage costs of kernel matrix whose dimension is equal to the number of data samples. In this paper, a novel sparse kernel-based least-squares temporal-difference (TD) algorithm for reinforcement learning is presented, where a kernel sparsification procedure using approximately linear dependent (ALD) analysis is used to reduce the kernel matrix dimension efficiently. The solution of the kernel-based LS-TD(λ) learning algorithm is derived by a least-squares regression in the kernel-induced high-dimensional feature space and its sparsity is guaranteed by the ALD-based sparsification procedure. Compared to the previous linear TD(λ) methods, the proposed method not only has good performance in nonlinear approximation ability but also has sparse solutions with low computational costs. Experimental results on learning prediction of a Markov chain illustrate the effectiveness of the proposed method.