Modelling Speaker Attribution in Narrative Texts With Biased and Bias-Adjustable Neural Networks.

Tillmann Dönicke,Hanna Varachkina,Benjamin Gittel,Anna Mareike Weimer,Florian Barth,Caroline Sporleder,Anke Holler,Luisa Gödeke

doi:10.3389/frai.2021.725321

Abstract

Literary narratives regularly contain passages that different readers attribute to different speakers: a character, the narrator, or the author. Since literary narratives are highly ambiguous constructs, it is often impossible to decide between diverging attributions of a specific passage by hermeneutic means. Instead, we hypothesise that attribution decisions are often influenced by annotator bias, in particular an annotator's literary preferences and beliefs. We present first results on the correlation between the literary attitudes of an annotator and their attribution choices. In a second set of experiments, we present a neural classifier that is capable of imitating individual annotators as well as a common-sense annotator, and reaches accuracies of up to 88% (which improves the majority baseline by 23%).

Highlights

Humans have different habits when it comes to reading and interpreting literature
As we are interested in investigating annotator bias for this task, we provide annotation guidelines which give annotators a lot of freedom to use their own judgement when attributing a passage
It can be seen that the label “author” shows a high correlation with more questions (13 questions) than the label “character” (8 questions), which in turn shows a high correlation with more questions than the label “narrator” (4 questions). This is in line with what we would expect: Whether the author is interpreted as taking responsibility for the information conveyed in a passage is typically not explicitly signalled in the text and, as such, it leaves much room for diverging annotation decisions based on the annotator’s preferences and beliefs

Summary

Introduction

Humans have different habits when it comes to reading and interpreting literature. human annotators bring different assumptions and beliefs to the annotation task and introduce an annotator bias (e.g., Geva et al, 2019) in the data creation. Annotator bias has been studied in regard to several phenomena of natural language understanding (see Geva et al, 2019; Akhtar et al, 2020; Kuwatly et al, 2020), but not yet in the literary domain This is surprising, because a basic problem in literary studies is the inability to hermeneutically examine the influence of the recipients’ world knowledge on the reception process. We will show that disagreements regarding the annotation of literary phenomena are not randomly distributed but fall into discernible patterns that can be attributed to differing literature-specific preferences and implicit beliefs of readers. To this end, we compute correlations between questionnaire-based data on readers’ preferences and beliefs, and the annotations produced by them. We develop a bias-adjusted (and biasadjustable) classifier, that takes into account literature-specific attitudes of annotators and show that this outperforms an annotator-agnostic classifier

Objectives

Methods

Results

Conclusion