Temporal difference learning for heuristic search and game playing

Abstract

Temporal difference (TD) learning is a natural method of reinforcement learning that is particularly appropriate for learning in heuristic search and game playing. Sutton [Machine Learning 3 (1988) 9–44] introduced the TD( λ) method which is an elegant integration of supervised learning with TD learning. TD( λ) enabled Tesauro’s backgammon program to reach world championship standard. But it can be slow. Tesauro’s program was trained on 1 500 000 games. Recent work [D.F. Beal, M.C. Smith, Temporal coherence and prediction decay in temporal difference learning. Technical Report no. 756, Department of Computer Science, Queen Mary and Westfield College, University of London, 1998] has described a significant algorithmic improvement (temporal coherence) that controls learning rates, and produces more stable final values. Results from a random walk task, and two complex real-world games are presented here to show that temporal coherence (TC) produces faster learning than earlier methods, and that TD learning can produce values that are superior to standard values for specified search regimes, without any domain-specific information or human assistance. In the chess domain, we also describe the emergence of classic human elementary knowledge in the pattern of weights learnt from self-play, starting with weights initialised to zero.