Annotating topics, stance, argumentativeness and claims in Dutch social media comments: A pilot study

Abstract

Habernal and Gurevych (2016) in their feasibility study of Arfgument Mining on User Generated Content (UGC) indicated the major challenge facing the field: there seems to be a big gap between the substantial argumentation research and the limited use NLP applications seem to make of these. This also leads to a myriad of annotation approaches, which either start from scratch (with a proprietory scheme which is rarely motivated in terms of the theories underlying it) or apply an existing scheme (Stephen Toulmin being a popular choice for many different text genres, ranging from more formal texts (legal documents) to more informal data like forum posts). The pilot study into the annotation of Dutch Facebook comments (for topics, aspects, stance, argumentativeness and claim) we present was motivated by our hypothesis that the divide between the unpredictable nature of the data and the traditionally restrictive definitions of concepts like argumentativeness may be bridged by gaining more knowledge on the great variety of linguistic expressions of argumentation in user comments. We motivate some of the theoretical principles underlying our annotation guidelines as well as illustrate some of the annotation difficulties which have prompted the adjustments of our initial guidelines.