Abstract

The CoNLL-2015 Shared Task is on Shallow Discourse Parsing, a task focusing on identifying individual discourse relations that are present in a natural language text. A discourse relation can be expressed explicitly or implicitly, and takes two arguments realized as sentences, clauses, or in some rare cases, phrases. Sixteen teams from three continents participated in this task. For the first time in the history of the CoNLL shared tasks, participating teams, instead of running their systems on the test set and submitting the output, were asked to deploy their systems on a remote virtual machine and use a web-based evaluation platform to run their systems on the test set. This meant they were unable to actually see the data set, thus preserving its integrity and ensuring its replicability. In this paper, we present the task definition, the training and test sets, and the evaluation protocol and metric used during this shared task. We also summarize the different approaches adopted by the participating teams, and present the evaluation results. The evaluation data sets and the scorer will serve as a benchmark for future research on shallow discourse parsing.

Highlights

  • The shared task for the Nineteenth Conference on Computational Natural Language Learning (CoNLL-2015) is on Shallow Discourse Parsing (SDP)

  • The training data for the CoNLL-2015 Shared Task was adapted from the Penn Discourse TreeBank 2.0. (PDTB-2.0.) (Prasad et al, 2008; Prasad et al, 2014), annotated over the one million word Wall Street Journal (WSJ) corpus that has been annotated with syntactic structures (Marcus et al, 1993) and propositions (Palmer et al, 2005)

  • A review of the guidelines was followed by double blind annotation of a small number of WSJ texts not previously annotated in the PDTB, and differences were compared and discussed

Read more

Summary

Introduction

The shared task for the Nineteenth Conference on Computational Natural Language Learning (CoNLL-2015) is on Shallow Discourse Parsing (SDP). The release of the RST-DT and PDTB has attracted a significant amount of research on discourse parsing (Pitler et al, 2008; Duverle and Prendinger, 2009; Lin et al, 2009; Pitler et al, 2009; Subba and Di Eugenio, 2009; Zhou et al, 2010; Feng and Hirst, 2012; Ghosh et al, 2012; Park and Cardie, 2012; Wang et al, 2012; Biran and McKeown, 2013; Lan et al, 2013; Feng and Hirst, 2014; Ji and Eisenstein, 2014; Li and Nenkova, 2014; Li et al, 2014; Lin et al, 2014; Rutherford and Xue, 2014), and the momentum is building Almost all of these recent attempts at discourse parsing use machine learning techniques, which is consistent with the theme of the CoNLL conference.

Task Definition
Training and Development
Test Data
Data Selection and Post-processing
Annotations
Inter-annotator Agreement
Adapting the PDTB Annotation for the shared task
Closed and open tracks
Evaluation Platform
Evaluation metrics and scorer
Approaches
Results
Conclusions
Full Text
Paper version not known

Talk to us

Join us for a 30 min session where you can share your feedback and ask us any queries you have

Schedule a call