Experience Replay-based Deep Reinforcement Learning for Dialogue Management Optimisation

Abstract

Dialogue policy is a crucial component in task-oriented Spoken Dialogue Systems (SDSs). As a decision function, it takes the current dialogue state as input and generates appropriate system’s response. In this paper, we explore the reinforcement learning approaches to solve this problem in an Indic language scenario. Recently, Deep Reinforcement Learning (DRL) has been used to optimise the dialogue policy. However, many DRL approaches are not sample-efficient. Hence, particular attention is given to actor-critic methods based on off-policy reinforcement learning that utilise the Experience Replay (ER) technique for reducing the bias and variance to achieve high sample efficiency. ER based actor-critic methods, such as Advantage Actor-Critic Experience Replay (A2CER) are proven to deliver competitive results in gaming environments that are fully observable and have a very small action-set. While, in SDSs, the states are not fully observable and often have to deal with the large action space. Describing the limitations of traditional methods, i.e., value-based and policy-based methods, such as high variance, low sample-efficiency, and often converging to local optima, we firstly explore the use of A2CER in dialogue policy learning. It is shown to beat the current state-of-the-art deep learning methods for SDS. Secondly, to handle the issues of early-stage performance, we utilise a demonstration corpus to pre-train the models prior to on-line policy learning. We thus experiment with the A2CER on a larger action space and find it significantly faster than the current state-of-the-art. Combining both approaches, we present a novel DRL based dialogue policy optimisation method, A2CER and its effectiveness for a task-oriented SDS in the Indic language.