Real Users and Real Dialog Systems: The Hard Challenge for SDS

Abstract

Much of the research done in our community is based on developing spoken dialog systems and testing various techniques within those dialog systems. Because it makes it easier to deal with our experimental conditions, many of our tests and studies involve controlled (paid or volunteered) users. However, we have seen in a number of studies that these controlled users do not use the system in the same way as those for whom the system was actually designed. Sometimes the difference between the real user, who wants the information the spoken dialog system is providing or who wants to give information to it, and the controlled user, who is acting under some direction, is not that different. Certainly in some circumstances it is necessary to use the latter. But, since state-of-the-art systems have become increasingly reliant on large amounts of user data to train their models of behavior, it is critical that the user behavior we train on is real user behavior. This paper describes the issues that arise when building a spoken dialog system for real users. The goal is to provide both a service to the user and a realistic spoken dialog system (SDS) research platform.