Dialogue Annotation Research Articles

Human–robot dialogue annotation for multi-modal common ground

Human–robot dialogue annotation for multi-modal common ground

An Investigation Into English Learners’ Speech Acts Using a Semi-Automatic Annotation Tool

This paper explores the speech acts most frequently performed by learners of English as a foreign language (EFL) through the Dialogue Annotation and Research Tool (DART), a semi-automatic annotation tool. The study used learner corpus research and an English as the first language (L1) speaker corpus as a reference to compare the speech acts performed by the EFL learners. The study involved 90 EFL learners of four nationalities. A spoken learner corpus was built with the dyadic interlanguage English conversation register. The most frequently performed speech acts were generally different for the English L1 speakers and the EFL learners. The speech acts labeled as correctSelf, referProcess, expressNonAwareness, and stateReason were in the top 10 for EFL learners, whereas they were not in the top 10 for the English L1 speakers. This difference was caused by three factors: the English proficiency of the EFL learners, the frequently used formulaic phrases, and the task requirements. Moreover, the corpus annotation has a problem with unrecognized speech acts, primarily due to the redundancies and fragments in the learner discourses. The findings of this study reveal the complementary relationship between grammatical and pragmatic performances.

The Use of Semi-automatic Annotation in Speech Acts Performed by Learners of English

Since rare research studied speech acts (SAs) from a macro perspective, this study used a semi-automatic annotation tool named Dialogue Annotation and Research Tool (DART) to examine the most frequently performed SAs by Chinese and Thai learners of English as a foreign language (EFL). It also attempted to reveal the reasons for the similarities and differences in the SA performances of EFL learners from discrete linguacultural backgrounds. This study involved 30 Chinese and 30 Thai EFL learners, totaling 60 participants. A total of 30 dyadic English interlanguage conversations were collected for this study. The learner corpus research with the contrastive interlanguage analysis was adopted for the analysis. The DART annotation revealed that both Chinese and Thai EFL learners most frequently performed the same six SAs. These SAs were labeled by DART as state, hesitate, reqInfo, answer, expressOpinion, and stateReason. Of the six listed SAs, both Chinese and Thai EFL learners most frequently performed the SA labeled state; both cohorts also presented the SA stateReason least often. Task requirements, the frequent use of certain types of formulaic language, and English proficiency levels were ascertained as factors causing the identical presentation of the six identified SAs. The English proficiency levels of Chinese EFL learners were determined as the principal reason for their discrete performance of SAs. Conversely, the influence of the Thai linguacultural background denoted the primary factor for discrepancies in the SA performance of Thai EFL learners.

Creation, Analysis and Evaluation of AnnoMI, a Dataset of Expert-Annotated Counselling Dialogues

Research on the analysis of counselling conversations through natural language processing methods has seen remarkable growth in recent years. However, the potential of this field is still greatly limited by the lack of access to publicly available therapy dialogues, especially those with expert annotations, but it has been alleviated thanks to the recent release of AnnoMI, the first publicly and freely available conversation dataset of 133 faithfully transcribed and expert-annotated demonstrations of high- and low-quality motivational interviewing (MI)—an effective therapy strategy that evokes client motivation for positive change. In this work, we introduce new expert-annotated utterance attributes to AnnoMI and describe the entire data collection process in more detail, including dialogue source selection, transcription, annotation, and post-processing. Based on the expert annotations on key MI aspects, we carry out thorough analyses of AnnoMI with respect to counselling-related properties on the utterance, conversation, and corpus levels. Furthermore, we introduce utterance-level prediction tasks with potential real-world impacts and build baseline models. Finally, we examine the performance of the models on dialogues of different topics and probe the generalisability of the models to unseen topics.

Zero-shot language extension for dialogue state tracking via pre-trained models and multi-auxiliary-tasks fine-tuning

Dialogue state tracking (DST), a crucial component of the task-oriented dialogue system (TOD), is designed to track the user’s goal. Existing DST models mainly focus on monolingual dialogue input, failing to meet the growing needs of a TOD to provide multilingual services. Therefore, this paper proposes a novel Zero-shot Language Extension scenario for DST, extending the monolingual DST to multilingual DST without extra high-cost dialogue data annotation. In this scenario, the multilingual DST only needs a single shared model to handle multilingual input and generate a unified dialogue state. This setting makes deploying a complete multilingual TOD easy since it could be reused by the downstream components from existing monolingual TOD. Specifically, we achieve the language extension by multi-auxiliary-tasks fine-tuning of multilingual pre-trained models, where five relevant auxiliary tasks are jointly designed, including monolingual DST, cross-lingual DST, forward word translation, utterance recovery, and semantic similarity. The extended multilingual DST model can be enhanced through joint optimization with all the auxiliary tasks by capturing multilingual context understanding and cross-lingual alignment characteristics. Comprehensive experiments on Multilingual WOZ dataset (English → German and English → Italian) and cross-lingual MultiWOZ dataset (English → Chinese and Chinese → English) demonstrate the effectiveness and superiority of the proposed method.

The DART annotation scheme: form, appliability & application

ABSTRACTThe Dialogue Annotation and Research Tool (DART) is a tool designed to facilitate large-scale corpus-based research into pragmatics-related aspects, including syntax, pragmatics (speech-acts), semantico-pragmatics (Searle’s ‘IFIDS’), as well as other interaction-relevant features. This chapter aims at providing a detailed description and discussion of the DART annotation scheme, which has already been successfully applied to a number of corpora from various domains, and provide examples of its application and appliability to various types of spoken data, illustrating e.g. the ability to create speaker profiles that allow the researcher to investigate features of (in)directness & politeness, initiative, etc. (cf. Weisser 2016d). In doing so, I shall draw on materials from task-oriented corpora (SPAADIA; SRI Amex), unconstrained dialogue (Switchboard), and ‘English as a lingua franca (ELF)’ data from ICE-HK, at the same time illustrating the advantages of the scheme over prior existing schemes.

Illocutionary Force and Perlocutionary Effect Recognition in the (Semi-)Automated Pragmatic Annotation of ELF Spoken Discourse in Migration Domains

The adoption of English as a Lingua Franca (henceforth ELF; cf. Seidlhofer 2001; Guido 2008) for mutual understanding by and among interactants with different lingua-cultural backgrounds has become a widespread and well-documented phenomenon and scholars in the field have focused their attention on numerous aspects of its use (business transactions; attitudes of speakers; Facebook interactions; pedagogy; sociolinguistics; discourse cohesion; migration domains; respectively Cogo et al. 2011; Centonze 2013, 2015c, 2016b, forthcoming). The present contribution is part of a research project (Centonze 2017, forthcoming) that aims at introducing a novel approach to spoken discourse in ELF which combines corpus pragmatics (Aijmer and Ruhlemann 2015; cf. Aijmer 2002, 2013; Anderson and Corbett 2009) - a relatively new research area in the field of language and discourse studies - with the most recent techniques of quantitative/qualitative analysis and corpus annotation by means of semi-automated software. More specifically, the present research study will focus on the annotation of speech acts from an ELF perspective and on the analysis of speech acts in their frequencies and collocations in a study corpus by means of DART (the Dialogue Annotation Research Tool v 1.1, Weisser 2015), i.e. a research tool which, among other things, includes the pragmatic annotation of spoken discourse. The interest in speech act annotation in the present PhD research project takes as a bedrock for analysis Austin’s (1962) and Searle’s (1975) speech act taxonomy and a critical review of it, which highlights both strengths and weaknesses of such an approach with a consequent need for the implementation of a new categorization system which would be fit for the purposes of the analysis of speech acts in a corpus in English as a Lingua Franca in migration encounters; hence, in the present research project speech acts are not perceived as pre-constructed categories on which to base further studies on discourse but rather undergo a process of resemanticization, re formulation as well as re-contextualization within the framework of a corpusdriven approach applied to ELF. The corpus which is being taken into consideration will be referred to as the ELF MiDo Corpus (English as a Lingua Franca in Migration DOmains corpus) and consists of over 50,000 words of conversation between asylum seekers and intercultural mediators in symmetrical contexts. All the different corpus interviews and interactions are transcribed according to a basic mark-up format (.XML) which proved to be a necessary condition for the whole corpus to be properly scanned for analysis through the DART interface. Corpora subsections are also differentiated according to the typology of speech event and interactant (e.g. informal conversation; meeting; intercultural mediator vs. asylum seeker) - analysis of speech acts is also carried out accordingly.

DART – The dialogue annotation and research tool

AbstractCorpus-based research into pragmatics is suffering from a distinct lack of suitably annotated corpora. This dilemma has so far generally forced researchers in corpus-based pragmatics to focus on well-known fixed expressions (e. g. discourse markers, politeness formulae, etc.) in their research, rather than being able to investigate interaction on the level of speech acts and other pragmatics-relevant features on a larger scale. This article describes a research environment that aims at remedying this problem (currently for English only) by making large-scale annotation of, and research into, speech acts and other linguistic levels possible in an efficient manner, at the same time discussing the difficulties and complexities inherent in such an endeavour. It then goes on to illustrate the efficiency of the approach, and how the resulting annotations represent an improvement over existing models in the form of a brief case study. The latter includes an illustrative discussion of the performance of the tool in annotating a subset of 100 files from the Switchboard corpus, plus a more detailed comparison of the automatically annotated version of one of the files with its original, manually annotated, version.

Unsegmented Dialogue Act Annotation and Decoding with N-Gram Transducers

Most studies on dialogue corpora, as well as most dialogue systems, employ dialogue acts as the basic units for interpreting discourse structure, user input and system actions. The definition of the discourse structure and the dialogue strategy consequently require the tagging of dialogue corpora in terms of dialogue acts. The tagging problem presents two basic variants: a batch variant (annotation of whole dialogues, in order to define dialogue strategy or study discourse structure) and an online variant (decoding of the dialogue act sequence of a given turn, in order to interpret user intentions). In the two variants is unusual having the segmentation of each turn into the dialogue meaningful units (segments) to which a dialogue act is assigned. In this paper we present the use of the N-Gram Transducer technique for tagging dialogues, without needing to provide a prior segmentation, in these two different variants (dialogue annotation and turn decoding). Experiments were performed in two corpora of different nature and results show that N-Gram Transducer models are suitable for these tasks and provide good performance.

Does History Help? An Experiment on How Context Affects Crowdsourcing Dialogue Annotation

Crowds of people can potentially solve some problems faster than individuals. Crowd sourced data can be leveraged to benefit the crowd by providing information or solutions faster than traditional means. Many tasks needed for developing dialogue systems such as annotation can benefit from crowdsourcing as well. We investigate how to outsource dialogue data annotation through Amazon Mechanical Turk. We are in particular interested in empirically analyzing how much context from previous parts of the dialogue (e.g. previous dialogue turns) is needed to be provided before the target part (dialogue turn) is presented to the annotator. The answer to this question is essentially important for leveraging crowd sourced data for appropriate and efficient response and coordination. We study the effect of presenting different numbers of previous data (turns) to the Turkers in annotating sentiments of dyadic negotiation dialogs on the inter annotator reliability and comparison to the gold standard.

Annotation of negotiation processes in joint-action dialogues

Situated dialogic corpora are invaluable resources for understanding the complex relationship between language, perception, and action as they are based on naturalistic dialogue situations in which the interactants are given shared goals to be accomplished in the real world. In such situations, verbal interactions are intertwined with actions, and shared goals can only be achieved via dynamic negotiation processes based on common ground constructed from discourse history as well as the interactants' knowledge about the status of actions. In this paper, we propose four major dimensions of collaborative tasks that affect the negotiation processes among interactants, and, hence, the structure of the dialogue. Based on a review of available dialogue corpora and annotation manuals, we show that existing annotation schemes so far do not adequately account for the complex dialogue processes in situated task-based scenarios. We illustrate the effects of specific features of a scenario using annotated samples of dialogue taken from the literature as well as our own corpora, and end with a brief discussion of the challenges ahead.

Developing a reciprocal teaching/learning system for college remedial reading instruction

In face-to-face instruction of Reciprocal Teaching (RT), students’ reading processes and dialogues with their peers are hardly observed. As a result, the teacher has few clues to identify students’ learning difficulties and provide further scaffoldings. To record students’ reading processes and enhance their comprehension, this study reports on the design of an online reciprocal teaching and learning system to support teachers and students in college remedial reading instruction. A sample of 129 under-prepared college students voluntarily signed up to participate in a remedial reading program. They were encouraged to use multiple strategies such as predicting, clarifying, questioning, and summarizing, which were supported by the functionalities of dialogue box, chat room, discussion forum, and annotation tool in the system. In this study, it was observed that students employed the multiple strategies to enhance their reading comprehension, as revealed by the students’ reading processes recorded in the system. When encountering difficulties in using these multiple strategies, students expressed that they observed and learned from the teacher’s or their peers’ externalization of strategy usage. Students’ reading progress in the remedial instruction incorporating the RT system was also identified by the pre- and post-tests. This study suggests that there may be benefits for teachers in encouraging students to interact with others in order to clarify and discuss comprehension questions and constantly monitor and regulate their own reading.