R-Flow: An Extensible XML Based Multimodal Dialog System Architecture

Abstract

This paper presents an approach for an extensible multimodal interaction dialogue system, R-Flow, based on a recursive application of Model-View-C'ontroller (MVC) design patterns to derive system components and interfaces. This approach leads to a clear separation of three self-contained functional layers in a multimodal dialogue system: modality independent dialog control, synchronization of logical modalities, and physical presentation. These layers are codified and weaved together through standard based XML languages. In particular, it utilizes the standard State-Chart XML (SCXML) for dialog control, SMIL and EMMA based XM-Flow for modality synchronization and interpretation, and a generic XML based binding mechanism to map logical modalities to physical presentations. A prototype system has been implemented for multimodal (e.g. speech, text, and mouse) manipulation of Google map. Our experimental results indicated that such layered and component-based XML MMI system is feasible and the performance of such MMI system is studied and measured.