Chapter 22 - Predicting risk in a multiple stimulus-reward environment

Abstract

Recently, neurobiological evidence has emerged of risk prediction along the lines of simple reinforcement learning. This evidence, however, is limited to one-step-ahead risk, namely, the uncertainty involved in the next forecast or reward. However, it appears that multistep prediction risk is more relevant. Multiple stimuli may all predict the same future reward or a sequence of future rewards, and only the risk of the total reward is relevant. It is known that subjects are indeed interested in predicting the total reward (sum of the discounted future rewards), and that learning of the expected total reward accords with the temporal differencing algorithm. Complex extensions of simple reinforcement learning such as temporal difference (TD) learning have been proposed for situations where there are multiple stimuli and rewards; in those cases, the object to be learned is the expected total reward, namely, the expected value of the sum of discounted future rewards. This paper provides a mathematical rationale for the encoding of one-step-ahead prediction risks. It does so by exploring how TD learning could be implemented to learn risk in multiple-stimuli, multiple-rewards setting. The chapter illustrates total reward risk learning in a simple, but generic example, which is called the multistep risk example, and proposes how TD learning could be used to learn total reward risk. It is shown that TD learning is an effective way to learn one-step-ahead prediction risks, and from estimates of the latter, total reward prediction risk can be learned. Simulations illustrate how the proposed learning algorithm works.