Construction and Evaluation of Text-Dialog Corpus with Emotion Tags Focusing on Facial Expression in Comics

Abstract

Large-scale text-dialog corpora with emotion tags are required to generate a knowledge base for emotional reasoning from text. Annotating emotion tags is known to suffer from problems with instability. These are caused by the lack of non-linguistic expressions (e.g. speech and facial expressions) in the text dialog. We aimed to construct a stable, usable text-dialog corpus with emotion tags. We first focused on facial expression in comics. Some comics contain many text dialogs that are similar to everyday conversation, and it is worth analyzing their text. We therefore extracted 29,538 sentences from 10 comic books and annotated face tags and emotion tags. Two annotators independently placed “temporary face/emotion tags” on stories and then decided what the “correct face/emotion tags” were by discussing them with each other. They acquired 16,635 correct emotion tags as a result. We evaluated the stability and usability of the corpus. We evaluated the correspondence between temporary and correct tags to assess stability, and found precision was 83.8% and recall was 78.8%. These were higher than for annotation without facial expressions (precision = 56.2%, recall = 51.5%). We extracted emotional suffix expressions from the corpus using a probabilistic method to evaluate usability. We could thus construct a text-dialog corpus with emotion tags and confirm its stability and usability.