Comparative evaluation and shared tasks for NLG in interactive systems

Abstract

Introduction Natural Language Generation (NLG) has strong evaluation traditions, in particular in the area of user evaluation of NLG-based application systems, as conducted for example in the M-PIRO (Isard et al ., 2003), COMIC (Foster and White, 2005), and SumTime (Reiter and Belz, 2009) projects. There are also examples of embedded evaluation of NLG components compared to non-NLG baselines, including, e.g., the DIAG (Di Eugenio et al ., 2002), STOP (Reiter et al ., 2003b), and SkillSum (Williams and Reiter, 2008) evaluations, and of different versions of the same component, e.g., in the ILEX (Cox et al ., 1999), SPoT (Rambow et al ., 2001), and CLASSiC (Janarthanam et al ., 2011) projects. Starting with Langkilde and Knight's work (Knight and Langkilde, 2000), automatic evaluation against reference texts also began to be used, especially in surface realization. What was missing, until 2006, were comparative evaluation results for directly comparable, but independently developed, NLG systems. In 1981, Sparck Jones wrote that information retrieval (IR) lacked consolidation and the ability to progress collectively, and that this was substantially because there was no commonly agreed framework for describing and evaluating systems (Sparck Jones, 1981, p. 245). Since then, various sub-disciplines of natural language processing (NLP) and speech technology have consolidated results and progressed collectively through developing common task definitions and evaluation frameworks, in particular in the context of shared-task evaluation campaigns (STECs), and have achieved successful commercial deployment of a range of technologies (e.g. speech recognition software, document retrieval, and dialogue systems).